Slide
Test-Time Scaling Under Budget
M.Sc. thesis defense on evaluating and ranking reasoning LLMs under a fixed test-time sampling budget, covering Pass@k's instability, Bayesian alternatives, and how many samples a reliable ranking actually needs.
Reasoning
Tag: Reasoning
5 items tagged with "Reasoning"
Slide
Psychometric Modeling of LLM Evaluation Datasets
A principled Bayesian framework that replaces Pass@k with posterior estimates, credible intervals, and stable rankings for LLM evaluation
Post
Simulating LLM Evaluation Datasets Using Psychometric Models
Explore how Item Response Theory (IRT) and other psychometric models can simulate and analyze LLM evaluation datasets. Learn how difficulty, discrimination, and guessing parameters reveal model reasoning patterns, with interactive examples across multiple reading levels.
Paper
Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, Vipin Chaudhary
A Bayesian framework for evaluating large language models that replaces unstable Pass@k metrics with posterior estimates and credible intervals. The method improves sample efficiency, supports graded outcomes, and enables statistically sound model comparisons.
Slide
Reasoning in Large Language Models
Reasoning in Large Language Models: chain-of-thought prompting (zero-shot CoT, re-reading, self-ask, least-to-most, self-consistency), retrieval-augmented reasoning, reasoning vs. reciting, bounded rationality, and interpretability.