Scorio Datasets for Test-Time Scaling

We collected repeated runs from reasoning models for several papers on evaluation and test-time scaling. Each model answered each question 80 times. We kept the response, parsed answer, grade, token counts, verifier scores and, in the larger releases, token distributions.

The 80 attempts for one model and one question form a candidate pool. With a pool, we can measure how often the model solves a problem, whether majority vote helps, whether a confidence score picks the right answer, and how the model ranking changes as more samples are used. None of that is visible from a single run.

We published the data as Scorio Trace, Scorio Lite, Scorio Math, and Scorio GPQA. The first two are regular Hugging Face datasets. Math and GPQA are Storage Buckets because the top-20 token distributions make them much larger. Lite contains smaller copies of the attempts in those two Buckets.

The attempt counts in the table are not meant to be added. Scorio Lite repeats the attempts stored in the Math and GPQA Buckets, with fewer token details. There are 1,403,520 distinct sampled attempts: 192,000 in Scorio Trace and 1,211,520 shared by Lite, Math, and GPQA.

Scorio Trace

Scorio Trace is the broad model comparison. It has 20 model configurations evaluated on AIME 2024, AIME 2025, the February 2025 HMMT, and BrUMO 2025. Each benchmark has 30 questions. With 80 attempts per question, the dataset has 192,000 rows.

Each row includes the prompt, generated response, extracted answer, reference answer, correctness label, finish reason, token counts, and scores from external verifiers. It also records aggregate prompt and completion log probabilities. The per-model configurations go further and store the sampled token, its log probability, and its vocabulary rank at every position.

There are two ways to load the same attempts. The default meta configuration puts all 20 models together and leaves out the long per-token arrays. At about 1.36 GiB, I use it for accuracy tables, Pass@$k$, model ranking, and verifier studies. The 20 per-model configurations total about 11.72 GiB and are needed when a calculation uses the token sequence itself.

This is the dataset to use when model coverage matters more than detailed top-$k$ token distributions. It is also a good place to study low-budget ranking because every model saw the same 120 questions and each candidate pool has the same 80 seeds.

Scorio Lite

Scorio Lite is where I would start with the newer collection. It has four configurations: Qwen3.6-35B-A3B and the low, medium, and high reasoning settings of gpt-oss-20b.

The math side contains 59,520 attempts on five competition sets: AIME 2026, CMIMC 2025, HMMT February 2026, HMMT November 2025, and SMT 2025. The broad-knowledge side contains 1,152,000 attempts on a fixed sample of 3,600 SuperGPQA questions. Those questions span 72 fields, with 50 questions from each field.

Lite keeps the fields used in most experiments: responses, parsed answers, correctness, verifier outputs, token counts, and aggregate log-probability statistics. Its per-model configurations also keep arrays for the tokens the model actually produced, together with their log probabilities and vocabulary ranks. What it leaves out is the probability distribution over alternative tokens at each step. That omission is why the whole release fits in about 35 GiB instead of hundreds.

The meta-math and meta-gpqa configurations are smaller again. They drop the token arrays while keeping the rows needed for evaluation and selection work. If you want to compare accuracy, draw learning curves over $N$, test majority voting, or rank the four configurations, start there.

One detail matters when joining the SuperGPQA records. Its local data_id repeats across stages. Use full_data_id, uuid, selection_hash, or the pair (stage, data_id) as the question key.

Scorio Math

Scorio Math contains the same 59,520 math attempts found in Lite. The difference is what was saved during generation.

At every prompt and completion position, the Bucket stores the top 20 candidate tokens and their log probabilities. That makes it possible to compute token entropy, varentropy, self-certainty, maximum probability, and the margin between likely alternatives. Lite knows how confident the model was in the token it chose. Math also knows what else the model nearly chose.

That extra information is expensive. The release has 744 Parquet files and occupies about 166.77 GiB. Each file is one complete candidate pool: one model, one question, 80 attempts in seed order. You can read only the columns you need, so an accuracy experiment does not have to pull every nested probability list across the network.

Use this Bucket for questions about confidence and answer selection on competition math. If the experiment only needs correctness, extracted answers, or realized-token log probabilities, the math configurations in Lite are much easier to handle.

Scorio GPQA

Scorio GPQA is the large one. It contains 3,600 SuperGPQA questions, four model configurations, and 80 attempts per model and question. That comes to 1,152,000 attempts stored in 14,400 Parquet files, one candidate pool per file. The Parquet data occupies about 679.38 GiB.

Like the Math Bucket, GPQA saves the top 20 alternatives at each token position. Its 72 fields let us check whether a confidence or selection rule that works on Olympiad problems also works in science, engineering, medicine, law, and the humanities without changing the data format.

At 679.38 GiB, this is not a dataset to load all at once. The companion notebook starts with one question or one field and requests a short column list when the token distributions are not needed. It builds the 72-field catalog from one small marker in each block, then fetches only the selected fields.

Notebooks

The companion notebook directory has one folder for each release: trace, lite, math, and gpqa. Each folder follows the same pattern.

The notebook named after the dataset explains the schema and loads representative records. eval.ipynb treats the 80 attempts as repeated evidence and estimates performance with uncertainty. rank.ipynb compares model orderings as the trial budget changes. aggregate.ipynb takes several candidate answers and tests ways to return one answer, including voting and score-based selection.

I usually open the dataset notebook first and inspect one candidate pool before running the evaluation, ranking, or aggregation notebooks. It is an easy way to catch mixed model IDs, question IDs, or seed order before they reach a metric.

The two Bucket notebooks require datasets>=5.0.0 and huggingface_hub>=1.5.0. A Bucket is loaded with a name such as buckets/harimo/scorio-math plus an explicit file path. The standard Scorio Trace and Scorio Lite repositories use the familiar load_dataset() interface. In either case, streaming and PyArrow column projection are useful when you only need a slice.

Papers using the data

This collection grew over several projects. Four papers use the repeated-run data or the evaluation setup built around it.

Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation, published at ICLR 2026, came first. We used repeated runs on AIME 2024, AIME 2025, HMMT 2025, and BrUMO 2025 to study how evaluation scores and model rankings change with the number of trials. The paper introduced Bayes@$N$, which reports posterior uncertainty instead of treating a finite set of samples as an exact score.

Ranking Reasoning LLMs under Test-Time Scaling, published at ACL 2026, expanded the comparison to 20 model configurations. It compares 72 ranking methods on the four math benchmarks. That 20-model, 80-trial grid is the main content of Scorio Trace.

Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility, currently under review at TMLR, is the paper most directly connected to the full release. It uses the math collection and the newer SuperGPQA runs to compare sampling, answer selection, and evaluation. One result is the selection gap: drawing more answers often puts a correct answer somewhere in the pool, but voting, model confidence, or a verifier may still select a wrong one.

The fourth paper is Geom@k: Stable Evaluation and Fast Rank Recovery for LLM Reasoning, currently under review. It returns to the 20-model math data and combines two properties that Pass@$k$ keeps separate: finding at least one correct answer and producing that answer repeatedly. In those experiments, Geom@$2$ reaches the full-budget ranking with fewer trials than the other fixed-budget metrics we tested.

The papers ask different questions of the same basic object: an 80-attempt candidate pool. That is why the releases include raw responses and grades as well as token probabilities and verifier scores.

Which release to use

Start with Scorio Lite if you are unsure. It covers math and broad knowledge, works as a normal Hugging Face dataset, and contains enough information for most evaluation, ranking, and aggregation experiments.

Choose Scorio Trace for a wider, 20-model comparison. Move to the Math or GPQA Bucket only when the experiment needs the top-20 token distributions. The Buckets answer questions that Lite cannot, but they are a costly way to read columns that Lite already has.

All four releases use the MIT License for the artifact itself. The benchmark questions keep their original terms. In particular, the SuperGPQA content is distributed under ODC-BY 1.0 and requires credit to the SuperGPQA benchmark and paper.