Reasoning in Large Language Models (Slides)

<h1 style="color: #41474e; font-size: 5.6rem; font-weight: 800; line-height: 1.1; margin: 0 0 2.6rem; text-transform: none; letter-spacing: -0.01em;">Reasoning in Large Language Models</h1>

<div style="display: flex; gap: 0.9rem; flex-wrap: wrap; justify-content: center; margin: 0 0 2.6rem;">
<a href="#/chain-of-thought" style="padding: 0.4em 1.2em; border-radius: 999px; background: rgba(210,104,120,0.12); color: #b1495a; border: 1px solid rgba(210,104,120,0.34); font-size: 1.7rem; font-weight: 600; text-decoration: none; cursor: pointer;">Chain-of-Thought</a>
<a href="#/retrieval-cot" style="padding: 0.4em 1.2em; border-radius: 999px; background: rgba(210,104,120,0.12); color: #b1495a; border: 1px solid rgba(210,104,120,0.34); font-size: 1.7rem; font-weight: 600; text-decoration: none; cursor: pointer;">Retrieval CoT</a>
<a href="#/beyond-reasoning" style="padding: 0.4em 1.2em; border-radius: 999px; background: rgba(210,104,120,0.12); color: #b1495a; border: 1px solid rgba(210,104,120,0.34); font-size: 1.7rem; font-weight: 600; text-decoration: none; cursor: pointer;">Beyond Reasoning</a>
<a href="#/interpretability" style="padding: 0.4em 1.2em; border-radius: 999px; background: rgba(210,104,120,0.12); color: #b1495a; border: 1px solid rgba(210,104,120,0.34); font-size: 1.7rem; font-weight: 600; text-decoration: none; cursor: pointer;">Interpretability</a>
</div>

<p style="color: #41474e; font-size: 2.1rem; font-weight: 600; margin: 0 0 1.8rem;">Mohsen Hariri <span style="color: #9aa0a6;">&middot;</span> September 2024</p>

<div style="display: flex; gap: 1rem; flex-wrap: wrap; justify-content: center;">
<a href="mailto:mxh1029@case.edu" style="padding: 0.5em 1.6em; border-radius: 999px; background: #5690af; color: #ffffff; font-size: 1.7rem; font-weight: 600; text-decoration: none;">Email</a>
<a href="https://mohsenhariri.github.io" style="padding: 0.5em 1.6em; border-radius: 999px; background: transparent; color: #5690af; border: 1px solid #5690af; font-size: 1.7rem; font-weight: 600; text-decoration: none;">Website</a>
</div>

</div>

---

# Chain-of-Thought

Reasoning elicited from model weights.

---

## Elicit Reasoning

[Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)

- **Chain of Thoughts (CoT)**
  - A series of intermediate reasoning steps.
  - Example:
    - Task: Question Answering
    - input
    - [intermediate steps will be generated]
    - [answer will be generated]

### Elicit Reasoning — a wide range of tasks

### Elicit Reasoning — contribution

- Chain of thoughts (CoT) = a series of intermediate reasoning steps
- Instead of asking a question directly, provide a series of intermediate reasoning steps.

---

## Zero-shot CoT (zero-shot reasoning)

[Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916)

- **Zero-shot**
  - Model has never trained on the task. (as far as we know!)
  - Questions are asked LLMs without any context.
  - Example:
    - Task: Question Answering
    - input
    - [answer will be generated]
- **Few-shot**
  - Again, model has never trained on the task.
  - But, model is given a few examples of the task as follows:
    - Task: Question Answering
    - Examples: Q1 → A1 → … → input → [answer will be generated]

### Zero-shot CoT — few-shot-CoT vs zero-shot-CoT

- **Few-shot-CoT**
  - In each exemplar, model is also given intermediate steps to solve the task.
  - Example: Q1 → intermediate steps to solve Q1 → A1 → … → input → [intermediate steps] → [answer]
- **Zero-shot-CoT**
  - Unlike few-shot-CoT, model is not given any intermediate steps.
  - The prompt encourages the model to generate intermediate steps by adding "Let's think step by step" at the beginning of the prompt.
  - Example: input → "Let's think step by step." → [intermediate steps] → [answer]

### Zero-shot CoT

### Zero-shot CoT — contributions

- Let's think step by step.
- It can be complementary to the CoT. E.g., reasoning steps start with "Let's think step by step" and continue with CoT.

---

## Re-Reading Prompt

[Re-Reading Improves Reasoning in Large Language Models](https://arxiv.org/abs/2309.06275)

- **Zero-shot-CoT**
  - Let's think step by step.
- **Zero-shot-CoT + Re-Reading**
  - Let's think step by step.
  - Re-read the prompt.
  - "bidirectional" understanding of the prompt.

### Re-Reading Prompt — contributions

- Provide a prompt twice.

---

## Self-Ask

[Measuring and Narrowing the Compositionality Gap in Language Models](https://arxiv.org/abs/2210.03350)

Compositional Reasoning:

- Cognitive process of understanding complex concepts or systems by breaking them down into their constituent parts and understanding the relationships between these parts.

### Self-Ask — with retrieval

- This method can be used with retrieval information.

---

## Rephrase and Respond

[Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://arxiv.org/abs/2311.04205)

Motivating Example

### Rephrase and Respond

- **One-step RaR**
  - {Question}
  - Rephrase and expand the question, and respond.
- **Two-step RaR**
  - Let Stronger LLMs Rephrase for Weaker LLMs to Respond.
  - step 1: {Question} → "Given the above question, rephrase and expand it to help you do better answering. Maintain all information in the original question."
  - step 2: {Question} → {Rephrased Question} → "Use your answer to the rephrased question to answer the original question."

### Rephrase and Respond — few-shot exemplars

Learning from few-shot exemplars:

### Rephrase and Respond — contributions

- Let LLMs ask better questions for themselves by rephrasing and expanding the question.
- "Rephrase and Respond"
- **One-step RaR** and **Two-step RaR**

---

## Recursive Decomposition

[Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://arxiv.org/abs/2205.10625)

- Easy to hard generalization
- Least to most prompting
- In contrast to CoT, least-to-most prompting starts with the simplest form of the task and gradually increases the complexity of the task by asking questions.

### Least-to-Most Prompting

1. **Decomposition.** The prompt in this stage contains constant examples that demonstrate the decomposition, followed by the specific question to be decomposed.
2. **Subproblem solving.** The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions; and (3) the question to be answered next.

---

## Contrastive Reasoning

[Contrastive Chain-of-Thought Prompting](https://arxiv.org/abs/2311.09277)

---

## Self-consistency

[Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171)

---

## Other Approaches

---

## References — Chain-of-Thought

1. [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)
2. [Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916)
3. [Re-Reading Improves Reasoning in Large Language Models](https://arxiv.org/abs/2309.06275)
4. [Measuring and Narrowing the Compositionality Gap in Language Models](https://arxiv.org/abs/2210.03350)
5. [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://arxiv.org/abs/2311.04205)
6. [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://arxiv.org/abs/2205.10625)
7. [Contrastive Chain-of-Thought Prompting](https://arxiv.org/abs/2311.09277)
8. [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171)

---

# Retrieval-Augmented Reasoning

Reasoning with retrieval information in large language models.

---

## Rethinking with Retrieval

[Rethinking with Retrieval: Faithful Large Language Model Inference](https://arxiv.org/abs/2301.00303)

- Out-of-date, incorrect, or missing relevant knowledge in the pre-training corpus
- Incorrect memorization of relevant knowledge during pre-training.
- Incorrect retrieval of relevant knowledge during inference stage.

---

## IR-CoT

[Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://arxiv.org/abs/2212.10509)

---

# Beyond Reasoning

- Meta reasoning
- Bounded rationality
- Elements of reasoning in LLMs

---

## But what is reasoning?

<div class="cols">
<div>
<p>What is <strong>reasoning</strong>?</p>
<ul>
<li><strong>Reasoning</strong> is an English word that comes from the Latin word "ratio". "Ratio" comes from the Latin word "reri", and "reri" means "to think".</li>
</ul>
</div>
<div>
<p>Reasoning in LLMs</p>
<img src="/assets/slides/2024-09-15-reasoning-in-llms/beyond/image-5.webp" alt="Reasoning in LLMs" class="slide-fig fig-lg">
</div>
</div>

---

## So what?

- What is **reasoning**?

<p style="text-align: justify; font-size: 12px;">
Reasoning is one of the most fundamental cognitive processes that distinguishes human intelligence from other forms of life. It refers to the capacity to draw inferences or conclusions from premises, evidence, or principles. In a broad sense, reasoning involves the mental activity of organizing information and beliefs in order to solve problems, make decisions, and establish beliefs based on logic, experience, and the available information. Reasoning can take many forms, including deductive, inductive, abductive, analogical, and causal reasoning, each of which serves different purposes in thinking and problem-solving. Deductive reasoning allows us to derive specific conclusions from general premises with certainty, provided the premises are true, while inductive reasoning leads to general conclusions based on patterns or repeated observations, though these conclusions are only probable rather than certain. Abductive reasoning generates the most plausible explanation for a set of observations, especially when faced with incomplete information, helping us form hypotheses or quick judgments about real-world situations. Analogical reasoning allows individuals to make comparisons between two domains or systems, applying insights from one to the other, and is commonly used in creative problem-solving and scientific innovation. Causal reasoning, on the other hand, helps individuals understand cause-and-effect relationships, allowing for predictions about how one event might influence another. Various cognitive processes are involved in reasoning, including attention, which focuses on relevant information, memory, which recalls previous knowledge, and language, which structures thoughts and communication. Visual representations and symbols often aid reasoning, especially in complex tasks like mathematical problem-solving. In everyday life, reasoning is crucial for decision-making, whether it's about simple choices like what to eat or major life decisions about careers or finances. It is equally important in education and learning, as reasoning helps students understand new concepts and solve problems in subjects like mathematics and science. Ethical reasoning allows people to navigate moral dilemmas and interpersonal conflicts by balancing logic, emotion, and social values. Reasoning also plays a critical role in social cognition, helping individuals interpret others' behavior, intentions, and emotions to form relationships or resolve conflicts. In technical fields like science, medicine, and engineering, reasoning is indispensable for analyzing data, solving complex problems, and innovating. For example, in scientific inquiry, researchers rely on inductive reasoning to form hypotheses and on deductive reasoning to test them. Medical professionals use abductive reasoning to make diagnoses based on symptoms and patient history. However, reasoning is not infallible and is prone to errors, such as logical fallacies, which can lead to faulty conclusions. Fallacies like ad hominem attacks, slippery slopes, and false dichotomies distort reasoning by misrepresenting or oversimplifying arguments. Cultivating critical thinking skills and being aware of these potential pitfalls can improve the quality of one's reasoning. Ultimately, reasoning is an essential cognitive function that enables humans to navigate complex environments, solve problems, and make informed decisions. It allows for understanding relationships between ideas, predicting outcomes, and forming sound judgments, making it a fundamental aspect of human thought and behavior.
</p>

---

## Context Matters!

<p style="text-align: left; font-size: 2rem;">
    <strong>Context:</strong> It is obvious (to us) that the presenter should talk about meaning of reasoning in a scientific context.<br><br>
    This is what <strong>Chain of Thought</strong> is about in language modeling.<br><br>
    It gives the <strong>context</strong> to LLMs, such that the probability of each generated token is conditioned on the chain of previous tokens (context).<br><br>
    This is the <strong>reasoning</strong> that LLMs do.
</p>

---

## But is this really reasoning in LLMs?

- Reasoning, reciting, or regurgitating?

---

## Reasoning in LLMs

<p style="text-align: left; font-size: 2rem;">
  <strong>Reasoning</strong> is <em>the cognitive process of drawing conclusions, making decisions, and solving problems</em> based on logical thinking, evidence, and analysis. It involves critical thinking, as well as the use of deduction and induction, to arrive at sound judgments.
</p>
<p style="text-align: left; font-size: 2rem;">
  <strong>Reasoning</strong> utilizes <em>existing knowledge</em>, <em>patterns</em>, and principles to form coherent and justified conclusions.
</p>

---

## Factual or Counterfactual?

---

## Factual or Counterfactual?

---

## Reversal Curse

### Mathematical tools to explore reasoning in LLMs

- Linear Probes
- Sampling (Decoding)
- LogitLens
- Influence Functions

---

## Meta Reasoning

- Can we reason without words?

### Language and Thought

I-Language and E-Language

---

## Let's break it down

What is a (causal) language model?

- A language model is a model that can generate human-like text.
  - It is trained on a large corpus of text data. (Seeing and finding patterns in the data)
  - It can generate text by predicting the next word in a sentence. (Sampling from the learned patterns)

---

## Let's break it down

<div class="cols">
<div>
<p>Good at Deductive Reasoning</p>
<ul>
<li>Good at memorizing facts, More parameters, better weights</li>
<li>Good at deducing facts, Better sampling (Research Gap)</li>
</ul>
</div>
<div>
<img src="/assets/slides/2024-09-15-reasoning-in-llms/beyond/deductive-reasoning.webp" alt="Deductive reasoning" class="slide-fig fig-md">
</div>
</div>

---

## Bounded Rationality

- Why are we satisfied by "good enough?"
  - Omniscient
  - Omnipotent

## Bounded Rationality

<p style="text-align: left; font-size: 2.2rem;">
    <strong>Bounded rationality</strong> is the idea that when individuals make decisions, their rationality is limited by the information they have, the cognitive limitations of their minds, and the finite amount of time they have to make a decision. It was proposed by <strong>Herbert Simon</strong> as an alternative to the classical theory of rationality, which assumes that people make decisions by optimizing.
</p>

---

## Need more time?

- NO!
- One more reason to be satisfied by "good enough."

---

## LLMs and Cognitive Biases

- Human survival
- Survivorship bias
- Bounded rationality
- Mental shortcuts

---

## Lost in the middle

Serial position effect

- Primacy effect
- Recency effect

---

## Biases, Safety, and Ethics

- Now I need more time!
- Constrained decoding

---

## Cost

---

## Research in LLMs and DL

- Herbert A. Simon: "Theories of bounded rationality"

## Research in LLMs and DL

- Bojack Horseman: "Okay, okay. Here's the secret to doing research in LLMs. Just pretend you are doing research, and eventually, you'll forget you're pretending."

---

## References

- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://arxiv.org/pdf/2307.02477)
- [Bounded Rationality](https://en.wikipedia.org/wiki/Bounded_rationality)
- [Why are we satisfied by "good enough?"](https://thedecisionlab.com/biases/bounded-rationality)
- [Information-Theoretic Bounded Rationality](https://arxiv.org/pdf/1512.06789)
- [Reasoning skills of large language models are often overestimated](https://news.mit.edu/2024/reasoning-skills-large-language-models-often-overestimated-0711)
- Bojack Horseman Season 4, Episode 7 titled "Underground."
- Background has been taken from [LessWrong](https://www.youtube.com/watch?v=8kX62n6yNXA)
- [Noam Chomsky — Thought Without Language](https://www.youtube.com/watch?v=5YXXGHwmogU)

---

# Interpretability and Explainability

- LogitLens
- Embedding Projectors
- Inductive Reasoning

---

## Links

- [Learning resources](https://github.com/mohsenhariri/common-research)
- [LessWrong](https://www.lesswrong.com/)
- [Transformer Circuit](https://transformer-circuits.pub/)

Navigation