Ranking Reasoning LLMs under Test-Time Scaling

Introduction

Large language models (LLMs) are increasingly used as general-purpose reasoning systems for tasks such as programming and mathematical problem solving [1, 2]. Reliable evaluation is therefore essential. In many settings, what matters is not only an absolute score but also a ranking that supports model selection, deployment, and scientific comparison. This need is amplified by test-time scaling, which allocates additional inference compute by sampling multiple outputs per prompt and aggregating them, turning evaluation into a repeated-sampling problem [2, 3, 4].

Statistical ranking methods underpin two common LLM workflows. First, preference-based learning and alignment pipelines rely on human or model preferences over alternative responses, where the primitive observations are paired comparisons and downstream optimization depends on how those preferences are modeled and aggregated [5, 6]. Second, model comparisons are often communicated through leaderboards. Crowdsourced paired-comparison platforms such as Chatbot Arena collect head-to-head judgments and fit rating or paired-comparison models to produce public rankings [7], while benchmark-style evaluations rank models by task performance metrics such as Pass@$k$ [1]. Recent work has revisited the statistical foundations of LLM ranking in both preference-based settings [8] and benchmark settings, including IRT-style benchmarking [9]. Different ranking methods can produce noticeably different model orderings, and their agreement can vary with benchmark difficulty (figure 1).

A key practical distinction between these regimes is the representation of the data used for ranking. Preference-based evaluation typically yields a sparse and evolving comparison graph because only a subset of model pairs are compared and the model pool changes over time [7]. In contrast, benchmark evaluations produce dense outcomes for every model - question pair. For a fixed set of $L$ models and $M$ questions, we observe an outcome for every pair. Under test-time scaling, each model - question pair is evaluated with $N$ independent trials, producing a response tensor $\mathbf{R}\in\{0,1\}^{L\times M\times N}$. This dense repeated-trial setting raises new methodological questions: Which ranking rule should be used when $N$ is small? How quickly do different ranking methods stabilize as $N$ grows? How do priors and uncertainty estimates affect ranking robustness?

In this work, we study performance-based ranking under test-time scaling. We formalize the dense benchmark setting through the response tensor $\mathbf{R}$, evaluate ranking methods by their low-budget stability and convergence as the test-time budget increases, and implement the studied methods in Scorio.

We summarize our contributions as follows:

• We formalize dense benchmark ranking under test-time scaling via $\mathbf{R}\in\{0,1\}^{L\times M\times N}$ and connect common ranking families through pointwise, pairwise, and setwise transformations of $\mathbf{R}$.

• We propose an evaluation protocol based on low-budget stability (agreement between rankings computed from subsampled trials and reference rankings) and convergence with increasing numbers of trials.

• We compare a broad suite of ranking methods across $20$ reasoning models and four Olympiad-style math benchmarks (up to $N=80$ trials), characterizing where method families agree and where they diverge.

• We analyze Bayesian and uncertainty-aware ranking choices, including priors and conservative (quantile-based) scoring, and quantify their bias - variance trade-offs in low-trial regimes.

• We release Scorio, a library implementing the ranking methods and Bayesian options.

Ranking Problem and Test-time Scaling

In classical statistical settings, there is no canonical theoretical ground truth or empirical gold standard against which competing ranking rules can be judged. Choosing among methods therefore usually requires additional modeling assumptions. Test-time scaling offers a useful alternative: because each model - question pair can be sampled repeatedly, it lets us evaluate ranking methods by how stable they are in low-budget settings and how quickly they converge as more trials are observed.

Statistical ranking methods are widely used in domains such as sports competitions (e.g., paired-comparison models and rating systems for head-to-head games) [10, 11, 12] and voting or collective decision-making [13, 14, 15]. In such settings, there are $L$ entities to be ranked (e.g., players, items, or models) over $M$ tasks (e.g., matches, questions, or instances). Test-time scaling adds a third dimension: $N$, the number of i.i.d. samples generated for a fixed question $m\in\{1,\dots,M\}$. Repeated sampling lets us study two complementary properties. First, low-budget stability asks whether a ranking computed from a small number of trials agrees with a high-budget reference ranking. In our experiments, the low-budget case is $N=1$: we subsample one trial per question, compute the ranking, repeat this over the available single-trial draws, and compare each ranking either with an empirical gold standard or with the same method's full-trial ranking. Second, convergence asks how quickly rankings computed from $n$ trials approach the full-trial ordering as $n$ increases from $1$ to $N$.

Gold Standard Rankings

Evaluation metrics widely used in test-time scaling, such as Pass@$k$ and Bayes@$N$, can be analyzed through statistical properties such as bias. For instance, [1] derive an unbiased estimator for Pass@$k$. As the number of trials $N$ grows, empirical estimates of these metrics concentrate around their population values, making metric-based rankings increasingly stable. In particular, for binary outcomes, $\mathrm{Bayes}_{\mathcal{U}}@N$ is order-equivalent to mean accuracy avg@$N$ [16], which motivates our use of the full-trial $\mathrm{Bayes}_{\mathcal{U}}@N$ ranking as an empirical accuracy-based gold standard.

This reasoning does not extend automatically to all ranking methods. Even as the number of questions $M$ or trials $N$ increases, different ranking methods need not converge to a unique limiting ordering, such as the one induced by average accuracy (Appendix C.1). Unlike evaluation metrics, ranking algorithms can emphasize different aspects of performance across tasks, players, or items. In section 3, we show that rankings induced by probabilistic models (e.g., Bradley - Terry) can differ from those induced by expected-performance metrics (e.g., mean accuracy or Bayesian estimates).

Given the absence of a universal gold standard for ranking methods, we use two target rankings for comparison. First, we define an empirical gold standard based on average performance over all trials with a large sample size (e.g., $N=80$). This target captures aggregate performance across tasks and trials while allowing ties. This choice is justified for several reasons: (a) the ranking induced by average performance is order-equivalent to the ranking induced by Bayesian estimation with a uniform prior ($\mathrm{Bayes}_{\mathcal{U}}@N$); (b) when $N$ is large, average performance is among the most stable ranking rules relative to the alternatives (section 3.1); and (c) it is easy to interpret, widely used in practice, and yields absolute performance values.

The second target ranking is the ordering produced by a method itself (method@$80$) when all available trials are aggregated. This target lets us assess a method's self-consistency and convergence as more data become available.

Representation

We consider $L$ models evaluated on a benchmark of $M$ questions under test-time scaling, generating $N$ i.i.d. trials per model - question pair. Let $\mathcal{L}=\{1,\dots,L\}$ index models and $\mathcal{Q}=\{1,\dots,M\}$ questions; for each question we observe $N$ independent trials indexed by $n\in\{1,\dots,N\}$. For each $(l,m,n)\in \mathcal{L}\times\mathcal{Q}\times\{1,\dots,N\}$ we observe a binary outcome

where $R_{lmn}=1$ if model $l$ solves question $m$ on trial $n$. We collect these outcomes in a response tensor $\mathbf{R}\in\{0,1\}^{L\times M\times N}$. When $N=1$, this reduces to the standard single-run benchmark setting. Unlike crowdsourced paired-comparison datasets (e.g., Chatbot Arena [7]), where the primitive observations are model - model outcomes on a possibly sparse comparison graph, our benchmark setting produces outcomes for every model - question pair. We discuss the Arena sparse-comparison counterpart in appendix G. We therefore take $\mathbf{R}$ as the primitive object; all ranking methods we study use $\mathbf{R}$ as input, but they differ in the representations on which they operate after transforming or aggregating it.

Pointwise (model - question) representation.

Define the per-question solve rate

and the overall mean accuracy $\widehat{p}_{l} := \frac{1}{M}\sum_{m=1}^M \widehat{p}_{lm}$. Pointwise and IRT-style methods operate on the matrix $\widehat{\mathbf{P}}=[\widehat{p}_{lm}]\in[0,1]^{L\times M}$ (or on its row means), optionally reweighting questions (e.g., inverse-difficulty weighting [17]). Classical IRT models infer latent abilities from this representation [18, 19], and have recently been applied to LLM benchmarking [9]. When $N>1$, the trial axis corresponds to repeated Bernoulli observations; likelihood-based models (including IRT) can equivalently work with the sufficient statistic $k_{lm}:=\sum_{n}R_{lmn}$, yielding a binomial-response formulation [20, 21]. Related repeated-measures and longitudinal IRT extensions are also well studied [22, 23]. Evaluation-metric rankings (e.g., Pass@$k$ and Bayes@$N$) additionally use the per-question trial multiset $\{R_{lm1},\dots,R_{lmN}\}$ (equivalently the count $\sum_n R_{lmn}$) to compute per-question metrics before aggregating across $m$ [1].

Pairwise (win/tie) representation.

Many classical ranking methods reduce $\mathbf{R}$ to pairwise outcomes. For a pair of models $(i,j)\in\mathcal{L}^2$ we define win and tie counts

so that, in our fully observed setting, $W_{ij}+W_{ji}+T_{ij}=MN$ for all $i\neq j$. Equivalently, we can form an undirected comparison graph $G=(V,E)$ with vertex set $V=\mathcal{L}$ and edge set $E=\{\{i,j\}:W_{ij}+W_{ji}+T_{ij}>0\}$, and store $(W_{ij},W_{ji},T_{ij})$ on each edge. In our benchmark setting $E$ is the complete graph (every pair is compared $MN$ times), whereas in interactive evaluation settings $E$ is typically sparse and one assumes $G$ is connected. The matrices $\mathbf{W}=[W_{ij}]$ and $\mathbf{T}=[T_{ij}]$ define a weighted comparison graph over models. Probabilistic paired-comparison models (e.g., Bradley - Terry and tie extensions [10, 24, 25]) and voting rules (e.g., Borda and Copeland [13, 26]) use these aggregated counts; graph- and spectral-based methods (e.g., PageRank, Rank Centrality, HodgeRank, SerialRank, AlphaRank, and Nash-based ranking [27, 28, 29, 30, 31, 32]) further transform $(\mathbf{W},\mathbf{T})$ into Markov chains or skew-symmetric edge flows, typically via edge weights based on empirical win rates such as $\widehat{P}_{i\succ j}=(W_{ij}+\tfrac{1}{2}T_{ij})/(W_{ij}+W_{ji}+T_{ij})$. Sequential rating systems (e.g., Elo and TrueSkill [11, 33]) instead process the underlying stream of pairwise "matches" induced by each question - trial $(m,n)$.

Listwise or setwise representation.

For each question - trial $(m,n)$ we define the winning set $U_{mn}:=\{l\in\mathcal{L} : R_{lmn}=1\}$ and the losing set $\mathcal{L}\setminus U_{mn}$, which induces a two-level partial order: all winners tie above all losers. Setwise or listwise models (e.g., Plackett - Luce [34, 35] and Davidson - Luce [36]) operate directly on the collection of events $\{(U_{mn},\mathcal{L}\setminus U_{mn})\}_{m,n}$, discarding degenerate events with $U_{mn}=\emptyset$ or $U_{mn}=\mathcal{L}$. In our binary two-level setting, Plackett - Luce likelihoods collapse to functions of pairwise win counts (cf. the MM formulation for generalized Bradley - Terry and Plackett - Luce likelihoods [37]), whereas Davidson - Luce explicitly models within-set ties.

Bayesian Approaches in Ranking

Many ranking methods can be viewed as probabilistic models with latent parameters $\theta$ (e.g., model strength and, optionally, question difficulty). Given observations $\mathbf{R}$ (or derived representations such as pairwise counts; section 2.2), inference reduces to estimating $\theta$ from a likelihood $p(\mathbf{R}\mid\theta)$. We consider maximum likelihood estimation (MLE), maximum a posteriori (MAP), and expected a posteriori (EAP), and discuss how uncertainty can be propagated to rankings [38]. Although MLE is not Bayesian, we include it as a standard baseline for likelihood-based ranking models.

Maximum likelihood estimation (MLE).

The maximum likelihood estimate is

which yields a point estimate without requiring a prior. MLE is attractive for its simplicity, but in paired-comparison and IRT-like models it can be unstable under (near-)separation or weak identification, which motivates priors in MAP and EAP.

Maximum a posteriori (MAP).

MAP incorporates prior information $p(\theta)$ and estimates the posterior mode:

Equivalently, MAP is a penalized MLE in which $-\log p(\theta)$ acts as a regularizer; priors can improve stability in paired-comparison and IRT-style models [39, 40]. We can also construct empirical priors from auxiliary evaluation runs. For example, a prior outcome tensor $\mathbf{R}_0$ (e.g., one greedy decode per question) can be used to regularize stochastic trials (EmpiricalPrior in Scorio) [16].

Expected a posteriori (EAP).

EAP uses the posterior mean as the point estimate:

which is Bayes-optimal under squared-error loss [38]. Compared with MAP, EAP accounts for posterior mass beyond the mode and typically requires approximation or sampling. EAP is common in latent-trait settings such as IRT and adaptive testing [41].

Interval estimates and conservative ranking.

Bayesian methods naturally yield credible intervals (posterior quantiles) for each $\theta_l$, while frequentist analyses can produce approximate confidence intervals for $\hat{\theta}_{\mathrm{MLE}}$ via bootstrap resampling of questions or trials. Interval estimates are especially useful because ranking is sensitive to near ties: rather than ranking by point estimates alone, one can rank conservatively using a lower credible or confidence bound (LCB), or report pairwise superiority probabilities $\Pr(\theta_i > \theta_j \mid \mathbf{R})$. Metric-level Bayesian estimators such as Bayes@$N$ provide both a posterior mean and uncertainty, enabling rankings by posterior mean or by a chosen posterior quantile. Bayes@$N$ also supports incorporating prior outcomes $\mathbf{R}_0$ (e.g., one greedy decode per question) as pseudo-counts in the posterior, which is complementary to using $\mathbf{R}_0$ to define empirical priors for MAP in parametric ranking models. Our implementation in Scorio supports both credible-interval ranking via Bayes@$N$ and empirical priors via EmpiricalPrior for MAP estimation.

Experiments

We evaluate $72$ ranking methods (Appendix J.2) on four Olympiad-style math benchmarks: AIME'24, AIME'25, HMMT'25, and BrUMO'25, each with $M=30$ questions. We use $L=20$ reasoning LLMs (full list in table 22). For each model - question pair, we collect $N=80$ independent trials via top-$p$ sampling, yielding a response tensor $\mathbf{R}\in\{0,1\}^{20\times 30\times 80}$. We also collect a single greedy-decoding output per question ($\mathbf{R}_0$) to serve as an empirical prior. Detailed generation, sampling, and reproducibility settings appear in appendix I; the library API is documented in appendix J.

Gold Standard Ranking

Following section 2.1, we define the gold-standard ranking as $\mathrm{Bayes}_{\mathcal{U}}@80$, the Bayesian posterior-mean estimator with a uniform prior computed from all $N=80$ trials. This choice is order-equivalent to avg@$80$ (mean correctness over all $M$ questions and all $N=80$ trials, with ties allowed) and yields an interpretable accuracy-based target. Empirically, when each of our $72$ ranking methods is computed using all $80$ trials, the resulting orderings agree closely with $\mathrm{Bayes}_{\mathcal{U}}@80$ (table 1): across benchmarks, the average Kendall's $\tau_b$ between $\mathrm{Bayes}_{\mathcal{U}}@80$ and the other methods is $0.93$ - $0.95$ (median $0.95$ - $0.99$), and $19$ - $34$ methods recover exactly the same ordering ($\tau_b=1$). The largest deviations come from a small set of voting rules (e.g., minimax and Nanson variants) and difficulty-weighted baselines, with minimum $\tau_b$ values of $0.68$ - $0.79$ depending on the benchmark. Although $\mathrm{Bayes}_{\mathcal{U}}@N$ is order-equivalent to avg@$N$, we prefer the Bayesian formulation because it supports priors (e.g., $\mathrm{Bayes}_{\mathbf{R}_0}@N$) and uncertainty estimates.

Ranking-Method Stability

To compare ranking methods in the low-budget regime, we set $N=1$ by subsampling one of the $80$ trials per question and recomputing the rankings. For each method, we report Kendall's $\tau_b$ averaged over the $80$ single-trial draws (mean $\pm$ std). Since the Pass@$k$ family requires at least two trials to differ from mean accuracy, the $N=1$ comparisons below cover the remaining $69$ methods.

Gold-standard agreement.

We first rank methods by agreement with the empirical gold standard ($\mathrm{Bayes}_{\mathcal{U}}@80$). Across AIME'24, AIME'25, and BrUMO'25, $\mathrm{Bayes}_{\mathbf{R}_0}@N$ performs best, achieving $\tau_b=0.779\pm0.034$, $0.798\pm0.045$, and $0.858\pm0.028$, respectively (table 2). On HMMT'25, the hardest benchmark (see appendix B), the greedy prior no longer helps, and the best score is shared by a $21$-method equivalence class ($\mathrm{Bayes}_{\mathcal{U}}@N$ and several graph- and voting-based methods), with $\tau_b=0.790\pm0.053$. When all benchmarks are pooled (Combined), the same $21$-method class attains $\tau_b=0.865\pm0.049$, while $\mathrm{Bayes}_{\mathbf{R}_0}@N$ drops to $\tau_b=0.786\pm0.031$ (table 17).

Self-consistency and convergence.

Next, we evaluate each method against its own full-trial ranking (method@$80$), which summarizes convergence from $N=1$ to $N=80$. Rasch MML with LCB scoring is the most self-consistent on AIME'24, AIME'25, and HMMT'25, with $\tau_b=0.804\pm0.051$, $0.834\pm0.054$, and $0.810\pm0.056$ (table 2); BrUMO'25 again favors $\mathrm{Bayes}_{\mathbf{R}_0}@N$ ($0.858\pm0.028$). On the Combined benchmark, the most self-consistent method is Nanson's rule with tie averaging ($0.892\pm0.050$), followed by Rasch MML (LCB) ($0.883\pm0.037$), whereas several minimax variants are among the least self-consistent (down to $0.765\pm0.045$; table 18). High self-consistency does not imply strong agreement with the gold standard: Nanson (avg ties) ranks first in self-consistency on Combined but has substantially lower gold-standard agreement ($0.807\pm0.036$; table 17).

Bootstrapped Model-Pool Robustness

The preceding $N=1$ results use the full set of $20$ models. To test whether those conclusions depend on the evaluation pool, we repeat the low-budget analysis on bootstrapped model pools of size $5$, $10$, and $15$. For each bootstrap subset, we recompute the full-trial rankings, use the subset-specific avg@$80$ ordering as the gold-standard target, and compare each method's $80$ single-trial rankings against two references: (i) the subset-specific avg@$80$ ordering and (ii) its own subset-specific full-trial ranking (method@$80$). We aggregate $1000$ bootstrap subsets for each benchmark - size setting.

Easy and medium benchmarks preserve the original winner.

On AIME'24, AIME'25, and BrUMO'25, $\mathrm{Bayes}_{\mathbf{R}_0}@N$ remains the best representative method under both targets at all three model-pool sizes (table 3). The mean score changes only slightly with pool size: on AIME'24, gold-standard agreement moves from $0.769$ to $0.780$ and self-consistency from $0.773$ to $0.785$ as the pool size increases from $5$ to $15$ models; on AIME'25, the corresponding ranges are $0.797$ - $0.802$ and $0.803$ - $0.809$; on BrUMO'25, $\mathrm{Bayes}_{\mathbf{R}_0}@N$ stays near $0.854$ - $0.858$ for both targets. On BrUMO'25, this advantage also becomes more decisive as the pool grows: the fraction of subsets where $\mathrm{Bayes}_{\mathbf{R}_0}@N$ is the top-scoring method rises from about $0.69$ at $k=5$ to $0.98$ - $0.99$ at $k=15$.

Harder benchmarks remain tie-rich.

The harder settings behave differently. On HMMT'25 and on the Combined benchmark, the top score is not unique: for agreement with avg@$80$, an equivalence class of $29$ - $30$ methods shares the best mean, while for method@$80$ the tied class still contains $13$ - $14$ methods. We report avg (avg@$N$, order-equivalent to $\mathrm{Bayes}_{\mathcal{U}}@N$) as a representative member of these tied classes. The tied optimum is essentially flat across pool size, staying near $0.788$ - $0.790$ on HMMT'25 and $0.863$ - $0.866$ on Combined. This mirrors the full-model analysis in section 3.2: once the benchmark is difficult or pooled across heterogeneous tasks, many pointwise, voting, and graph-based methods become empirically indistinguishable.

Larger pools mainly reduce between-subset variance.

The primary effect of increasing the model-pool size is to reduce dispersion across subsets rather than to shift the mean systematically (table 3). For the best method under the avg@$80$ target, the across-subset standard deviation falls from $0.209$ to $0.057$ on AIME'24, from $0.144$ to $0.038$ on AIME'25, from $0.114$ to $0.033$ on HMMT'25, from $0.136$ to $0.032$ on BrUMO'25, and from $0.084$ to $0.023$ on Combined when moving from $5$ to $15$ models. Thus, the qualitative recommendation is stable under moderate changes to the model pool: larger pools mainly make the same conclusion more certain.

Effect of Empirical Priors

Empirical priors use auxiliary evaluation signals to stabilize low-budget rankings. In our setting, the signal is a single greedy decode, $\mathbf{R}_0$. We incorporate $\mathbf{R}_0$ into Bayes@$N$, yielding $\mathrm{Bayes}_{\mathbf{R}_0}@N$, and compare it with the uniform-prior variant $\mathrm{Bayes}_{\mathcal{U}}@N$. We evaluate both variants by their agreement with the gold-standard ranking $\mathrm{Bayes}_{\mathcal{U}}@80$. For each $N$, we compute Kendall's $\tau_b$ between the induced model ranking and $\mathrm{Bayes}_{\mathcal{U}}@80$ and report the mean and standard deviation over $50$ resampled datasets.

Empirical priors reduce variance at low $N$.

Across all benchmarks, $\mathrm{Bayes}_{\mathbf{R}_0}@N$ yields more stable low-$N$ rankings than $\mathrm{Bayes}_{\mathcal{U}}@N$. At $N=1$, the standard deviation of $\tau_b$ decreases by $16$ - $52\%$ depending on the benchmark (table 4, figure 7). This advantage shrinks quickly as $N$ increases (figure 2), consistent with the prior contributing only $O(1)$ pseudo-counts per question.

The mean effect depends on greedy - sampling alignment.

Variance reduction does not guarantee improved agreement with $\mathrm{Bayes}_{\mathcal{U}}@80$. The greedy prior increases mean $\tau_b$ on AIME'24, AIME'25, and BrUMO'25, but decreases it on HMMT'25 (table 4). At $N=1$, when all benchmarks are pooled, this negative shift is substantially larger (table 17), indicating that an empirical prior can introduce systematic bias when greedy and sampling behave differently across datasets.

We summarize this diagnostic via greedy - sampling alignment $\tau_{\text{G-S}}$, defined as Kendall's $\tau_b$ between the model rankings induced by greedy decoding and by stochastic sampling at $N=80$. In our results, higher $\tau_{\text{G-S}}$ coincides with a more positive $\Delta\tau$ (appendix E, figure 6), suggesting that the empirical prior is most likely to help when greedy is a faithful proxy for the sampling-induced ordering. While this evidence is limited to four benchmarks, the trend is consistent with $\mathrm{Bayes}_{\mathbf{R}_0}@N$ acting as shrinkage toward the greedy ordering.

Implications.

$\mathrm{Bayes}_{\mathbf{R}_0}@N$ behaves as a shrinkage estimator toward the greedy ordering: it is helpful when greedy decoding is a faithful proxy for the sampling-induced ranking, and harmful when the two disagree. Because $\mathbf{R}_0$ is generated under a different decoding policy, incorporating it effectively biases the estimate toward greedy behavior. This can be desirable for variance reduction, but it changes the implied evaluation target. A plausible source of disagreement is that greedy decoding may under-explore on hard instances, while stochastic sampling can recover alternative successful reasoning paths. In practice, empirical priors are most attractive when $N$ is very small and greedy - sampling alignment has been checked on a small pilot sample; otherwise, $\mathrm{Bayes}_{\mathcal{U}}@N$ provides a safer default.

Bias - variance trade-off.

Figure 7 visualizes the trade-off induced by empirical priors: in our benchmarks, the greedy prior reduces variability (narrower distributions) but can introduce bias (shifted means), with the net effect governed by greedy - sampling alignment.

Categorical Ranking

We extend the Bayesian framework to categorical outcomes: each completion is mapped to one of $C+1$ ordered categories based on signals such as answer format (boxed vs. unboxed), model confidence (completion bits per token), token efficiency, and external verifier judgments. Each scheme defines a categorical mapping and a utility weight vector $\mathbf{w}=(w_0,\dots,w_C)$; Bayesian estimation then proceeds with a Dirichlet - multinomial model rather than a Beta - binomial model (details and scheme definitions are given in appendix F).

We select eight non-redundant representative schemes. Using the $N=1$ subsampling protocol on the Combined benchmark (the first $L=11$ models of table 22, $M=120$ questions pooled across all four datasets), we measure Kendall's $\tau_b$ against three references (table 5).

Self-consistency vs. gold-standard trade-off.

Signal-rich schemes achieve the highest self-consistency: Verifier-only ($\tau_{\text{Self}} = 0.897$) and OOD-robust ($0.892$) rank first and second (figure 4). Yet these schemes have the lowest agreement with the gold standard ($\tau_{\text{GS}} = 0.824$ and $0.840$, respectively), extending the finding from section 3.2 that high self-consistency does not imply closeness to the gold standard. The negative correlation between $\tau_{\text{GS}}$ and $\tau_{\text{Self}}$ across schemes (figure 4) suggests that auxiliary signals introduce systematic biases away from the correctness-based ordering while stabilizing single-trial rankings.

Greedy-prior alignment.

All eight schemes correlate more strongly with $\mathrm{Bayes}_{\mathbf{R}_0}@80$ than with $\mathrm{Bayes}_{\mathcal{U}}@80$; the gap is largest for Verifier-only ($\Delta\tau = +0.046$) and OOD-robust ($+0.031$), consistent with the mechanism in section 3.4: verifier and OOD signals encode information partially aligned with greedy-decoding behavior. Per-dataset results (appendix F) show that scheme differentiation widens on harder benchmarks (HMMT'25, BrUMO'25), where Verifier-only drops to $\tau_{\text{GS}}=0.753$ and $0.734$, while correctness-driven schemes remain stable ($\tau_{\text{GS}}\ge 0.80$).

Related Work

Test-time scaling samples multiple solutions per prompt and aggregates them [2, 3, 4]; because stochastic reasoning varies across runs [42], we study how this variability affects rankings as budget changes. Preference evaluation and alignment learn from paired comparisons [5, 6] and underpin leaderboards such as Chatbot Arena [7, 8]. Benchmark leaderboards often rank models by task metrics such as Pass@$k$ [1], but item-level difficulty and discrimination affect reliability [43]; recent work adds Bayesian uncertainty and IRT-style modeling [16, 9]. We extend this literature to dense repeated-trial benchmarks and compare ranking methods by stability and convergence; appendix H gives background.

Conclusion & Future Directions

Test-time scaling turns LLM benchmarking into a repeated-sampling problem, so model rankings must be estimated from stochastic trials rather than from a single run. We formalize this setting and compare a broad collection of ranking methods within a common framework. When many trials are available, most reasonable ranking families induce nearly identical orderings, making $\mathrm{Bayes}_{\mathcal{U}}@N$ a simple and interpretable default. The main differences appear in the low-budget regime. There, uncertainty-aware estimators can improve stability, and the greedy prior $\mathrm{Bayes}_{\mathbf{R}_0}@N$ acts as a shrinkage estimator: it reduces variance when greedy and stochastic sampling align, but can bias rankings when they diverge.

In practice, $\mathrm{Bayes}_{\mathcal{U}}@N$ is a strong default, whereas $\mathrm{Bayes}_{\mathbf{R}_0}@N$ is best used after checking greedy - sampling alignment on a small pilot sample. Our experiments focus on binary correctness; extending the analysis to partial credit, rubric-based scoring, and other categorical evaluation settings is a natural next step.

Limitations

Our experiments focus on mathematical reasoning benchmarks. We do not evaluate partial credit, or open-ended outputs, where outcome categories are less clear and annotation or verification noise may be larger. More generally, when informative priors are used - especially priors derived from auxiliary signals other than greedy decoding - the prior source and specification should be reported explicitly, since the prior can introduce systematic bias if it is misaligned with the stochastic evaluation regime.

Acknowledgments

This research was supported in part by NSF awards 2117439 and 2320952.

sectionappendix appendixAppendixAppendices appendixAppendixAppendices lstlistingListingListings lstlistingListingListings

Notation and Definitions

Throughout the paper, we use the following notation.

Data and Basic Quantities

• $L$: number of models being ranked.

• $M$: number of questions in a benchmark.

• $N$: number of independent stochastic trials per model - question pair under test-time scaling.

• $\mathbf{R}\in\{0,1\}^{L\times M\times N}$: response tensor, where $R_{lmn}=1$ if model $l$ solves question $m$ on trial $n$.

• $\mathbf{R}_0$: optional prior outcomes used by Bayesian estimators. In this paper, greedy decoding yields a shared prior matrix $\mathbf{R}_0\in\{0,1\}^{M\times D}$ with $D=1$, but the notation can also accommodate model-specific prior tensors.

• $\widehat{p}_{lm} := \frac{1}{N}\sum_{n=1}^N R_{lmn}$: per-question solve rate for model $l$ on question $m$.

• $k_{lm} := \sum_{n=1}^N R_{lmn}$: number of successful trials for model $l$ on question $m$.

Metric Shorthand

• Bayes@$N$: Bayesian posterior-mean estimate at $N$ trials under a specified prior.

• $\mathrm{Bayes}_{\mathcal{U}}@N$: Bayesian estimate with a uniform Dirichlet prior, denoted $\mathrm{Bayes}_{\mathcal{U}}@N$.

• $\mathrm{Bayes}_{\mathbf{R}_0}@N$: Bayesian estimate with a greedy empirical prior, denoted $\mathrm{Bayes}_{\mathbf{R}_0}@N$.

• Pass@$k$: probability that at least one of $k$ sampled completions is correct.

• avg@$N$: mean accuracy over all $M$ questions and $N$ trials. For binary outcomes, it is order-equivalent to $\mathrm{Bayes}_{\mathcal{U}}@N$.

Ranking-Method Families

• Pointwise methods: aggregate per-question performance to produce model scores (e.g., mean accuracy, inverse-difficulty weighting).

• Pairwise methods: transform outcomes into win/tie counts between model pairs and fit paired-comparison models (e.g., Bradley - Terry, Elo, Glicko).

• Listwise, setwise methods: operate on winner and loser sets for each question - trial (e.g., Plackett - Luce, Davidson - Luce).

• Voting rules: treat questions as voters that rank models and then aggregate those preferences (e.g., Borda, Copeland, Schulze, Kemeny - Young).

• Graph/spectral methods: construct comparison graphs and compute centrality- or flow-based scores (e.g., PageRank, Rank Centrality, HodgeRank, $\alpha$-Rank).

• IRT-inspired methods: estimate latent model abilities and item difficulties (e.g., Rasch, 2PL, 3PL, dynamic IRT).

Evaluation Criteria

• Kendall's $\tau_b$: rank-correlation coefficient that accounts for ties; it ranges from $-1$ (perfect disagreement) to $+1$ (perfect agreement).

• Gold-standard agreement: agreement between a low-budget ranking and the empirical gold standard, typically $\mathrm{Bayes}_{\mathcal{U}}@80$ in this paper.

• Self-consistency: agreement between a low-budget ranking and the same method's all-trial ranking.

• Convergence: the rate at which a method's ranking approaches its full-trial ordering as the number of trials increases.

• Greedy - sampling alignment ($\tau_{\text{G-S}}$): Kendall's $\tau_b$ between the ranking induced by greedy decoding and the ranking induced by stochastic sampling at high budget.

Inference Terminology

• MLE (maximum likelihood estimation): point estimate that maximizes $p(\mathbf{R}\mid\theta)$.

• MAP (maximum a posteriori): point estimate that maximizes $p(\mathbf{R}\mid\theta)p(\theta)$.

• EAP (expected a posteriori): posterior mean estimate $\mathbb{E}[\theta\mid\mathbf{R}]$.

• MML (marginal maximum likelihood): likelihood-based estimation that integrates over a latent population distribution, commonly used in IRT.

• Credible intervals (CrI): Bayesian posterior intervals used for uncertainty quantification; we use lower credible bounds (LCBs) for conservative ranking.

Accuracy of Models

Table 6, Table 8, Table 7, Table 9 report detailed accuracy statistics for all $L=20$ models, including greedy accuracy and stochastic-sampling statistics (minimum, mean, maximum, and standard deviation) over $N=80$ trials. HMMT'25 is the most difficult benchmark (mean accuracies $0.080$ - $0.554$), whereas AIME'24 and BrUMO'25 are less difficult. Figure 5 visualizes these distributions across benchmarks and highlights the heterogeneity in model performance and sampling variance that motivates our ranking-stability analysis.

Gold Standard Agreement

To justify our use of $\mathrm{Bayes}_{\mathcal{U}}@80$ as the gold standard, we compare the full-trial rankings produced by all methods at $N=80$. table 1 summarizes Kendall's $\tau_b$ between $\mathrm{Bayes}_{\mathcal{U}}@80$ and each competing method. These results indicate that $\mathrm{Bayes}_{\mathcal{U}}@80$ is also a high-consensus ordering: by average agreement with all other methods, it ranks first on AIME'25, HMMT'25, and the Combined benchmark and second on AIME'24 and BrUMO'25 within $5\times 10^{-4}$ of the best (table 10). Dataset-level consensus tables appear in table 12, table 12, table 13, table 14, table 15. Many methods recover the same ordering exactly (table 16), and the remaining disagreement is concentrated in a small low-agreement tail (table 11).

Convergence of Ranking Methods

As the number of trials $N$ (or questions $M$) increases, evaluation metrics such as avg@$N$, Bayes@$N$, and Pass@$k$ need not induce the same limiting ordering as ranking methods. The reason is that they target different population quantities.

We illustrate this distinction for two canonical choices used throughout the paper: the average-accuracy ranking and the Bradley - Terry (BT) model.

Large-Budget Limits: Each Method Converges, but Generally to a Different Target

To discuss $M\to\infty$ (or $N\to\infty$) formally, we introduce an i.i.d. sampling model at the level of question - trial pairs. Assume $(X_{mn})_{m\in[M],n\in[N]}$ are i.i.d. draws from some distribution $P$ on $\{0,1\}^L$. Let

Here, $p_\ell$ depends only on the marginal of model $\ell$, whereas $w_{ij}$ depends on the joint distribution of $(X_i,X_j)$.

Average targets marginal accuracy.

By the law of large numbers,

Likewise, $\mathrm{Bayes}_{\mathcal{U}}@N$ converges to the same $p_\ell$; for binary outcomes it differs from $\widehat{p}^{\text{avg}}_\ell$ only by $O((MN)^{-1})$ smoothing.

Bradley - Terry targets a pairwise decisive-win functional.

The empirical win frequencies converge:

Define the BT log-likelihood

Then the BT-ML estimator is an $M$-estimator: maximizing (8) with $W_{ij}$ is equivalent to maximizing the scaled objective $(MN)^{-1}\ell(\pi;W)$. Under mild regularity and connectivity conditions (ensuring strict concavity in $\log\pi$ and uniqueness up to scale), $\widehat{\pi}$converges to the unique (up to scale) maximizer of the population objective

The limiting objects $(p_\ell)_{\ell=1}^L$ and $\pi^\star$ are generally not linked by any monotone transform: $p_\ell$ depends only on marginal correctness, while $\pi^\star$ depends on the full matrix $(w_{ij})_{i\neq j}$. Therefore, without additional assumptions on $P$ (e.g., that $P$ is generated by a BT choice model at the level of decisive comparisons), there is no reason to expect the induced orderings to coincide as $MN\to\infty$. The following counterexample demonstrates this non-equivalence.

A Counterexample: Average accuracy and Bradley - Terry disagree even at infinite budget

We construct a distribution $P$ (equivalently, a finite pattern that can be repeated) for which the average ranking and the BT-ML ranking disagree. The construction uses $L=3$ models. For notational convenience, we label them $0,1,2$.

Outcome patterns.

Consider the following three outcome vectors in $\{0,1\}^3$:

Let $P$ place mass

Equivalently, one may take a deterministic dataset with $M=8$ questions and $N=1$ trial, containing exactly $2$ questions of Type A, $3$ of Type B, and $3$ of Type C; repeating this block preserves both rankings, as established by the derivation.

The marginal success probabilities are

so the average method ranks

For these three types, the decisive-win probabilities $w_{ij}=\mathbb{P}(X_i=1,X_j=0)$ are:

For the finite $M=8,N=1$ realization, the corresponding win counts are $W_{ij}=8w_{ij}$, i.e.,

It remains to show that BT-ML ranks $1>0>2$ for (10), thereby disagreeing with the average ranking.

A convenient characterization of the BT-ML optimum is the standard first-order condition equating observed wins to model-implied expected wins: for each $i$,

(These equations follow by differentiating (8) with respect to $\log\pi_i$.)