Psychometric Modeling of LLM Evaluation Datasets (Slides)

# Psychometric Modeling of LLM Evaluation Datasets

**Mohsen Hariri**  
*Case Western Reserve University*  
October 27, 2025

</div>

---

## What are we talking about?

**Background**. What are LLMs? What is reasoning in LLMs? What are the challenges of reasoning in LLMs?

**Conclusion**. LLMs might not build in such way that human biological cognitive functions are built. But they can mimic some of these functions to a good extent. So, it's valid to use psychometric modeling techniques to **evaluate** LLMs on **reasoning tasks**.

**Evaluation of LLMS**. A Bayesian framework with posterior estimates, credible intervals, and stable rankings for LLM evaluation.

**Evaluation of Datasets**. Psychometric modeling techniques to **analyze** LLM evaluation datasets.

**Question**. How can we determine whether an example is suitable to use as a criterion for evaluating LLMs on reasoning tasks?

---

## What Is An LLM?

- Autoregressive transformer trained on large corpora
- Learns latent world models via next‑token loss
- Instruction‑tuned to follow tasks; RLHF to reduce unsafe/undesired outputs
- With tools, it can plan, call functions, verify, and iterate

---

## The Reasoning Shift (2022 → 2025)

- Chain‑of‑Thought (CoT): ask for steps — Wei et al., 2022  
  - https://arxiv.org/abs/2201.11903  
- Self‑Consistency: sample multiple CoTs, vote — Wang et al., 2023  
  - https://arxiv.org/abs/2203.11171  
- Tree‑of‑Thoughts: branch + look‑ahead/backtrack — Yao et al., 2023  
  - https://arxiv.org/abs/2305.10601  
- Reflexion: verbal feedback + memory — Shinn et al., 2023  
  - https://arxiv.org/abs/2303.11366  
- DeepSeek‑R1 (2025): RL‑driven reasoning behaviors; distilled variants  
  - https://github.com/deepseek-ai/DeepSeek-R1

```text
predict → reflect → verify → revise → finalize
```

---

## Are they human-like thinkers or just stochastic parrots?

---

## But is this really reasoning in LLMs?

- Reasoning, reciting, or regurgitating?

---

## Reasoning in LLMs
<p style="text-align: left; font-size: 2rem;">
  <strong>Reasoning</strong> is <em>the cognitive process of drawing conclusions, making decisions, and solving problems</em> based on logical thinking, evidence, and analysis. It involves critical thinking, as well as the use of deduction and induction, to arrive at sound judgments.
</p>
<p style="text-align: left; font-size: 2rem;">
  <strong>Reasoning</strong> utilizes <em>existing knowledge</em>, <em>patterns</em>, and principles to form coherent and justified conclusions.
</p>

---

## Factual or Counterfactual?
<img src="/assets/slides/2025-10-27-psychometric-modeling/image-7.webp" alt="Alt text" style="width:30%; height:auto; display:block; margin-left:auto; margin-right:auto;">

---

## Factual or Counterfactual?
<img src="/assets/slides/2025-10-27-psychometric-modeling/image-8.webp" alt="Alt text" style="width:60%; height:auto; display:block; margin-left:auto; margin-right:auto;">

---

## Reversal Curse
<img src="/assets/slides/2025-10-27-psychometric-modeling/image-9.webp" alt="Alt text" style="width:100%; height:auto; display:block; margin-left:auto; margin-right:auto;">

---

## Mathematical tools to explore reasoning in LLMs
* Linear Probes
* Sampling (Decoding)
* LogitLens
* Influence Functions

---

## Meta Reasoning

- Can we reason without words?

### Language and Thought

**I-Language** and **E-Language**

---

## Let's break it down

What is a (causal) language model?
- A language model is a model that can generate human-like text.
  - It is trained on a large corpus of text data. (Seeing and finding patterns in the data)
  - It can generate text by predicting the next word in a sentence.(Sampling from the learned patterns)

---

## Let's break it down

<div>

Good at Deductive Reasoning
- Good at memorizing facts, More parameters, better weights
- Good at deducing facts, Better sampling (Research Gap)

</div>

<div>

</div>

---

## Bounded Rationality

<div>

- Why are we satisfied by “good enough?”
  - Omniscient
  - Omnipotent

<p style="text-align: left; font-size: 2.2rem;">
    <strong>Bounded rationality</strong> is the idea that when individuals make decisions, their rationality is limited by the information they have, the cognitive limitations of their minds, and the finite amount of time they have to make a decision. It was proposed by <strong>Herbert Simon</strong> as an alternative to the classical theory of rationality, which assumes that people make decisions by optimizing.
</p>

</div>

<div>

</div>

---

## Need more time?
* NO!
* One more reason to be satisfied by "good enough."

---

## LLMs and Cognitive Biases
* Human survival
* Survivorship bias
* Bounded rationality
* Mental shortcuts

---

## Lost in the middle
Serial position effect
* Primacy effect
* Recency effect
<img src="/assets/slides/2025-10-27-psychometric-modeling/image-12.webp" alt="Alt text" style="width:50%; height:auto; display:block; margin-left:auto; margin-right:auto;">

---

## Biases, Safety, and Ethics

<div>

- Now I need more time!
- Constrained decoding

</div>

<div>

</div>

---

## Cost

---

## What are we talking about?

<section>
  <p><strong>Background.</strong> What are LLMs? What is reasoning in LLMs? What are the challenges of reasoning in LLMs?</p>
</section>

<section>
  <p><strong>Conclusion.</strong> LLMs might not be built in the same way as human biological cognitive functions, but they can mimic some of these functions to a good extent. Therefore, it is valid to use psychometric modeling techniques to <strong>evaluate</strong> LLMs on <strong>reasoning tasks</strong>.</p>
</section>

<section>
  <p><strong>Evaluation of LLMs.</strong> A Bayesian framework with posterior estimates, credible intervals, and stable rankings for LLM evaluation.</p>
</section>

---

## The Evaluation Crisis

The problem with current LLM evaluation:

- **Deployment accelerates** Students, developers, analysts rely on LLMs 
- **Trust matters** Billions invested based on benchmark rankings 
- **Evaluation is the weakest link** Fragile metrics, unstable rankings

</div>

**The core issue**: Pass@k and single-run accuracy produce unstable rankings, are sensitive to decoding choices, and provide **no guidance on whether observed gaps are meaningful or noise**

</div>

Note:
  LLMs have moved from research artifacts to critical infrastructure.
  But our evaluation methods haven't kept pace.
  Pass@k was convenient but is fundamentally flawed for ranking models.

---

## What is Pass@k?

The current standard for code and reasoning tasks:

$$
\text{Pass@}k = \mathbb{E}_{\text{problems}} \left[ 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}} \right]
$$

- Generate $k$ samples per problem 
- Task is "solved" if **any** sample passes 
- Estimate the overall success rate

</div>

**Sounds reasonable... but** 😬

</div>

Note:
  Pass@k estimates the probability that at least one of k attempts succeeds.
  Introduced for HumanEval code generation, now used everywhere.
  But it has serious problems we'll uncover.

---

## Pass@k Problems

### ⚠️ Instability

High variance when $k \approx n$

Tiny fluctuations → distorted comparisons

</div>

### ❌ No Uncertainty

No closed-form variance

Requires expensive bootstrapping

</div>

### 🎯 Binary Only

0/1 correctness only

Discards format, reasoning, efficiency

</div>

### 🤷 When to Stop?

No principled stopping rule

Unclear if differences are real

</div>

Note:
  Four critical problems make Pass@k unreliable.
  Especially problematic on small, expensive benchmarks like AIME.
  We need something better - something principled.

---

## Enter Bayesian Thinking

Instead of point estimates, let's ask:

> **"What is the distribution of a model's underlying capability, given the observed trials?"**

</div>

This shifts evaluation from **statistical estimation** to **statistical inference**

</div>

✨ **Key insight**: Treat each question's success probability as a latent variable with a posterior distribution

</div>

Note:
  The paradigm shift: evaluation IS inference.
  We observe finite trials and want to infer underlying capability.
  Bayesian machinery gives us exactly this, plus uncertainty for free.

---

## Validation: Biased Coins 🪙

**Simulation setup**: 11 "LLM mimics" with known success probabilities

- $M=30$ questions, $\bar{\pi} = [0.233, 0.255, 0.360, \ldots, 0.733]$
- Includes one intentional tie: $\bar{\pi}_4 = \bar{\pi}_5 = 0.364$
- Compare rankings vs. **known ground truth**

**Metric**: Kendall's $\tau$ rank correlation (1 = perfect agreement)

</div>

</div>

Note:
  This is the gold standard validation: we KNOW the true ranking.
  Bayes@N converges faster and more reliably than all Pass@k variants.
  The tie is correctly handled with uncertainty quantification.

---

## Real Experiments: Math Reasoning

**Four challenging benchmarks**:

- **AIME 2024** & **AIME 2025**: Mathematical Association of America, 15 problems each
- **HMMT Feb 2025**: Harvard-MIT Math Tournament
- **BrUMO 2025**: Brown University Math Olympiad

**11 state-of-the-art reasoning models** (as of Oct 2025):

**Setup**: 80 trials per dataset/model, uniform prompting & decoding

</div>

Note:
  These are HARD benchmarks - competition math problems.
  The model cohort spans sizes from 1.2B to 32B parameters.
  All with latest reasoning capabilities (chain-of-thought, etc).

---

## Framework Comparison

| **Evaluation Method** | **Convergence** | **Confidence Interval** | **Prior Knowledge** | **Categorical** |
|:---------------------|:---------------:|:-----------------------:|:-------------------:|:---------------:|
| Pass@k (& variants) | ❌ | ❌ | ❌ | ❌ |
| avg@N | ✅ | ⚠️ Limited (bootstrap) | ❌ | ❌ |
| **Bayes@N** | ✅  | ✅ **Analytic** | ✅ **Yes** | ✅ **Yes** |

</div>

**Bayes@N dominates across all criteria**

- Faster convergence (30-40% fewer trials)
- Closed-form uncertainty (no expensive bootstrapping)
- Supports informative priors (reuse evidence)
- Handles categorical rubrics (beyond 0/1)

</div>

Note:
  This table summarizes everything we've seen.
  Bayes@N is strictly better than existing methods.
  You get all the benefits with no downsides.

---

## Don't Just Evaluate... **Infer!**

**"Evaluation is inference."**

- Treat LLM capabilities as latent variables  
- Report posteriors, not point estimates  
- Make uncertainty explicit  
- Stop over-interpreting noise

</div>

---

## What are we talking about?

<section>
  <p><strong>Background.</strong> What are LLMs? What is reasoning in LLMs? What are the challenges of reasoning in LLMs?</p>

<p><strong>Conclusion.</strong> LLMs might not be built in the same way as human biological cognitive functions, but they can mimic some of these functions to a good extent. Therefore, it is valid to use psychometric modeling techniques to <strong>evaluate</strong> LLMs on <strong>reasoning tasks</strong>.</p>

<p><strong>Evaluation of LLMs.</strong> A Bayesian framework with posterior estimates, credible intervals, and stable rankings for LLM evaluation.</p>
</section>

<section>
  <p><strong>Evaluation of Datasets.</strong> Psychometric modeling techniques to <strong>analyze</strong> LLM evaluation datasets.</p>
</section>

---

## Kuzco wants to take a test; How to model it?

<figure style="flex: 1; margin: 0; text-align: center; display: flex; flex-direction: column;">
  <img src="/assets/slides/2025-10-27-psychometric-modeling/k_e.webp" alt="Easy" style="width: 100%; height: 50vh; object-fit: contain;" />
  <figcaption style="padding-top: 0.5rem; font-size: 0.75em;">
    <strong style="color: #34d399;">Easy question</strong><br/>θ ≫ b
  </figcaption>
</figure>

<figure style="flex: 1; margin: 0; text-align: center; display: flex; flex-direction: column;">
  <img src="/assets/slides/2025-10-27-psychometric-modeling/k_m.webp" alt="Medium" style="width: 100%; height: 50vh; object-fit: contain;" />
  <figcaption style="padding-top: 0.5rem; font-size: 0.75em;">
    <strong style="color: #fbbf24;">Medium question</strong><br/>θ ≈ b
  </figcaption>
</figure>

<figure style="flex: 1; margin: 0; text-align: center; display: flex; flex-direction: column;">
  <img src="/assets/slides/2025-10-27-psychometric-modeling/k_d.webp" alt="Hard" style="width: 100%; height: 50vh; object-fit: contain;" />
  <figcaption style="padding-top: 0.5rem; font-size: 0.75em;">
    <strong style="color: #ef4444;">Hard question</strong><br/>θ ≪ b
  </figcaption>
</figure>

</div>

**Key insight**: Success probability depends on the **distance** between ability and difficulty: $\theta - b$

</div>

Note:
  Visual representation of how ability vs difficulty affects outcomes.
  When ability exceeds difficulty, success is likely.
  When difficulty exceeds ability, failure is likely.
  The S-shaped curve in between is what IRT models.

---

## The Rasch Model (1PL)

**Simplest IRT model**: Only ability $\theta$ and difficulty $b$

$$P(Y_j = 1 \mid \theta, b_j) = \sigma(\theta - b_j) = \frac{1}{1 + e^{-(\theta - b_j)}}$$

**Example**: Kuzco with $\theta = 1.2$

- Question 5: $b_5 = -0.5$ → difference = $1.7$ → $P(\text{correct}) \approx 0.85$ ✅ 
- Question 12: $b_{12} = 2.0$ → difference = $-0.8$ → $P(\text{correct}) \approx 0.31$ ❌

</div>

When $\theta = b$: $P = \sigma(0) = 0.5$ — a coin flip!

</div>

Note:
  The Rasch model is elegant: only one parameter per item (difficulty).
  All items discriminate equally well.
  The model explains patterns of which questions get answered correctly.

---

## From Students to LLMs

**The transformation**: Kuzco → Llama 🦙

- Students taking tests → **Models solving problems** 
- Ability $\theta$ → **Model capability** 
- Question difficulty $b$ → **Problem difficulty** 
- Multiple students → **Multiple models** ($L$ LLMs) 
- Test questions → **Evaluation dataset** ($M$ problems) 
- Multiple attempts → **Stochastic sampling** ($N$ runs)

</div>

Note:
  The mapping from psychometrics to LLM evaluation is natural.
  Everything we learned about testing humans applies to testing models.

---

## What are we talking about?

<section>
  <p><strong>Background.</strong> What are LLMs? What is reasoning in LLMs? What are the challenges of reasoning in LLMs?</p>

<p><strong>Evaluation of LLMs.</strong> A Bayesian framework with posterior estimates, credible intervals, and stable rankings for LLM evaluation.</p>

<p><strong>Evaluation of Datasets.</strong> Psychometric modeling techniques to <strong>analyze</strong> LLM evaluation datasets.</p>
</section>

<section>
  <p><strong>Question.</strong> How can we determine whether an example is suitable to use as a criterion for evaluating LLMs on reasoning tasks?</p>
</section>

---

## Questions?

**Thank you!**

</div>

<figure style="flex:1; margin:0; text-align:center;">
  <a href="https://mohsenhariri.github.io/research/2025-10-21-bayes-at-n.html">
    <img src="/assets/slides/2025-10-27-psychometric-modeling/bayes.webp" alt="Don’t Pass@𝑘: A Bayesian Framework for Large Language Model Evaluation" style="width:80%; height:auto; object-fit:contain; border-radius:8px;"/>
  </a>
  <figcaption style="padding-top:0.5rem; font-size:0.85em;">
    <a href="https://mohsenhariri.github.io/research/2025-10-21-bayes-at-n.html">Don’t Pass@𝑘: A Bayesian Framework for Large Language Model Evaluation</a>
  </figcaption>
</figure>

<figure style="flex:1; margin:0; text-align:center;">
  <a href="https://mohsenhariri.github.io/posts/2025-10-22-simulating-llm-answers-to-eval-datasets.html">
    <img src="/assets/slides/2025-10-27-psychometric-modeling/ds.webp" alt="Simulating LLM Answers to Evaluation Datasets" style="width:80%; height:auto; object-fit:contain; border-radius:8px;"/>
  </a>
  <figcaption style="padding-top:0.5rem; font-size:0.85em;">
    <a href="https://mohsenhariri.github.io/posts/2025-10-22-simulating-llm-answers-to-eval-datasets.html">Simulating LLM Answers to Evaluation Datasets</a>
  </figcaption>
</figure>

<figure style="flex:1; margin:0; text-align:center;">
  <a href="https://mohsenhariri.github.io/posts/2025-10-22-item-response-theory-llms.html">
    <img src="/assets/slides/2025-10-27-psychometric-modeling/irt.webp" alt="Simulating LLM Evaluation Datasets Using Psychometric Models" style="width:80%; height:auto; object-fit:contain; border-radius:8px;"/>
  </a>
  <figcaption style="padding-top:0.5rem; font-size:0.85em;">
    <a href="https://mohsenhariri.github.io/posts/2025-10-22-item-response-theory-llms.html">Simulating LLM Evaluation Datasets Using Psychometric Models</a>
  </figcaption>
</figure>

</div>

Please feel free to contact **mohsen.hariri@case.edu** with any questions or feedback, especially if you’d like to collaborate on this research!

Navigation