Scorio

A Bayesian framework for Large Language Model evaluation and ranking. Scorio implements test-time scaling metrics, including Bayes@N, with uncertainty quantification for evaluating and ranking models.

Bayes@N arXiv paper Ranking arXiv paper Scorio.jl arXiv paper MIT License Python 3.10+ Julia 1.6+

Papers

Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation ICLR 2026 arXiv:2510.04265 Ranking Reasoning LLMs under Test-Time Scaling ACL 2026 Main arXiv:2603.10960 Scorio.jl: A Julia package for ranking stochastic responses JuliaCon 2026 arXiv:2603.14103