Don’t Pass@𝑘: A Bayesian Framework for Large Language Model Evaluation

Consider a dataset with $M$ questions. An LLM is used to generate $N$ independent trials (samples) per question and evaluate each trial using a rubric that assigns it to one of $C + 1$ categories: $(0, 1, \ldots, C)$. The results are stored in an $M \times N$ matrix $R$, where $R_{\alpha i}$ denotes the category of the $i$-th trial for question $\alpha$. Optionally, you may include prior runs in a matrix $R^{0}$ with $D$ trials per question.

The simplest and most common rubric is binary ($C = 1$), where each trial is either correct (1) or incorrect (0). In this case, the results matrix $R$ contains only 0s and 1s. Suppose the dataset has $M$ questions and you run $N$ trials per question, resulting in an $M \times N$ matrix $R$. See the example below:

from scorio import eval
 
w = [0, 1]  # 0 for incorrect, 1 for correct
mu, sigma = bayes(R, w)
 
# if you have prior runs, let's say you run N top_p sampling and one greedy decoding (R0)
mu, sigma = bayes(R, w, R0)

using Scorio
 
w = [0, 1]
mu, sigma = bayes(R, w)

Abstract

Pass@k is widely used to report performance for LLM reasoning, but it often yields unstable and misleading rankings, especially when the number of trials (samples) is limited and compute is constrained. We present a principled Bayesian evaluation framework that replaces Pass@k and average accuracy over N trials with posterior estimates of a model's underlying success probability and credible intervals, yielding stable rankings and a transparent decision rule for differences. Evaluation outcomes are modeled as categorical (not just 0/1) with a Dirichlet prior, giving closed-form expressions for the posterior mean and uncertainty of any weighted rubric and enabling the use of prior evidence when appropriate. Theoretically, under a uniform prior, the Bayesian posterior mean is order-equivalent to average accuracy (Pass@1), explaining its empirical robustness while adding principled uncertainty. Empirically, in simulations with known ground-truth success rates and on AIME’24/’25, HMMT’25, and BrUMO’25, the Bayesian/avg procedure achieves faster convergence and greater rank stability than Pass@k and recent variants, enabling reliable comparisons at far smaller sample counts. The framework clarifies when observed gaps are statistically meaningful (non-overlapping credible intervals) versus noise, and it naturally extends to graded, rubric-based evaluations. Together, these results recommend replacing Pass@k for LLM evaluation and ranking with a posterior-based, compute-efficient protocol that unifies binary and non-binary evaluation while making uncertainty explicit. Source code is available at https://mohsenhariri.github.io/bayes-kit.

Algorithm

In principle, we could estimate $\boldsymbol{\pi}_\alpha$ by running an arbitrarily large number of trials with the LLM, yielding an accurate estimate of $\bar{\pi}$. However, we are typically constrained to small $N$ due to limited computational resources. Our goal is to develop a Bayesian approach to estimate $\bar{\pi}$ and its associated uncertainty given a finite $N$.

The first step is to construct the posterior distribution

the probability of $\boldsymbol{\pi}_\alpha$ given the $\alpha$-th row of the matrix $R$, denoted $\boldsymbol{R}_\alpha$. This posterior depends on the observed data $\boldsymbol{R}_\alpha$ and a chosen prior distribution $\mathcal{P}(\boldsymbol{\pi}_\alpha)$ for the unknown underlying probability vector $\boldsymbol{\pi}_\alpha$. The prior could be uniform (assuming no prior information) or incorporate previously gathered evidence about the LLM's performance.

The Bayesian framework focuses on two key quantities:

1. Posterior mean of $\bar{\pi}$, denoted $\mu(R)$, which is the mean of $\bar{\pi}$ over the joint posterior for all questions:

   $$\mu(R) = \mathbb{E}_{\{\boldsymbol{\pi}_\alpha\} \mid R}\left[ \bar{\pi} \right].$$

   This is the Bayesian optimal estimator, minimizing the quadratic loss function

   $$\mathcal{L}(\bar{\pi}^{\mathrm{est}}) = \mathbb{E}_{R, \boldsymbol{\pi}_\alpha} \left[ \bar{\pi}^{\mathrm{est}}(R) - \bar{\pi} \right]^2,$$

   over all possible estimators $\bar{\pi}^{\mathrm{est}}(R)$, where the expectation is taken over all possible $\boldsymbol{\pi}_\alpha$ and realizations of $R$.

2. Posterior variance, denoted $\sigma^2(R)$, which quantifies the uncertainty of the $\mu$ estimate:

   $$\sigma^2(R) = \mathrm{Var}_{\{\boldsymbol{\pi}_\alpha\} \mid R}\left[ \bar{\pi} \right].$$

Both $\mu(R)$ and $\sigma^2(R)$ have exact closed-form expressions, derived in Appendix A of the paper, and can be efficiently computed for any $R$ using the Algorithm below.