Post
Scorio Datasets for Test-Time Scaling
Scorio Trace, Lite, Math, and GPQA contain 1.4 million repeated reasoning attempts for evaluation, ranking, answer selection, and token-level confidence analysis.
Reasoning
Tag: Reasoning
6 items tagged with "Reasoning"
Slide
Test-Time Scaling Under Budget
M.Sc. thesis defense on evaluating and ranking reasoning LLMs under a fixed test-time sampling budget, covering Pass@k's instability, Bayesian alternatives, and how many samples a reliable ranking actually needs.
Slide
Psychometric Modeling of LLM Evaluation Datasets
A principled Bayesian framework that replaces Pass@k with posterior estimates, credible intervals, and stable rankings for LLM evaluation
Post
Simulating LLM Evaluation Datasets Using Psychometric Models
Explore how Item Response Theory (IRT) and other psychometric models can simulate and analyze LLM evaluation datasets. Learn how difficulty, discrimination, and guessing parameters reveal model reasoning patterns, with interactive examples across multiple reading levels.
Paper
Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, Vipin Chaudhary
A Bayesian framework for evaluating large language models that replaces unstable Pass@k metrics with posterior estimates and credible intervals. The method improves sample efficiency, supports graded outcomes, and enables statistically sound model comparisons.
Slide
Reasoning in Large Language Models
Reasoning in Large Language Models: chain-of-thought prompting (zero-shot CoT, re-reading, self-ask, least-to-most, self-consistency), retrieval-augmented reasoning, reasoning vs. reciting, bounded rationality, and interpretability.