10 items tagged with "Benchmarking"
MICCAI 2026 presentation on MIS-Ground, a factorial benchmark that isolates the language-side brittleness behind 3D medical spatial grounding, and MIS-SemSam, a training-free semantic-sampling decode rule that lifts Qwen3-VL-32B by 13.06% to 66.5% overall, the best open-weights result and above Gemini 3 Flash.
MICCAI 2026 poster: MIS-Ground measures 3D anatomical spatial grounding in vision-language models under a controlled factorial design, and MIS-SemSam fixes language-side brittleness at decode time for free.
ACL 2026 presentation on ranking reasoning LLMs under test-time scaling: dense repeated-trial evaluation, Bayes@N as a practical default, low-budget priors, categorical ranking, and the Scorio toolkit.
Serving reasoning LLMs efficiently and reliably: lossless DFloat11 compression, KV-cache quantization, and Bayes@N evaluation and ranking under test-time scaling.
Serving reasoning LLMs efficiently and reliably: lossless DFloat11 compression, KV-cache quantization, and Bayes@N evaluation and ranking under test-time scaling.
MIS-Ground is a controlled factorial benchmark that isolates the language-side brittleness behind 3D medical spatial grounding in vision-language models. MIS-SemSam, a training-free semantic-sampling decode rule, lifts Qwen3-VL-32B by 13.06% to 66.5% overall, the best open-weights result and above Gemini 3 Flash.
Ranking reasoning LLMs under test-time scaling. We compare 72 ranking methods (Bayes@N, Bradley-Terry, Elo, IRT, voting, graph/spectral) across 20 models and four Olympiad-style math benchmarks. At full budget they mostly agree (Kendall's tau_b 0.93-0.95); at one trial a greedy prior cuts variance 16-52% but can bias the ranking. Packaged in the Scorio toolkit.
Ranking reasoning LLMs under repeated sampling, comparing 72 ranking methods across four Olympiad-style math benchmarks and packaging them in Scorio.
ICLR 2026 presentation on Don't Pass@k: a Bayesian evaluation framework (Bayes@N) with Dirichlet posteriors, credible intervals, a non-overlap decision rule, categorical rubric scoring, and the Scorio toolkit.
Proposed a Bayesian framework that estimates models' success probabilities with quantified uncertainty, yielding more reliable rankings and enabling categorical evaluation of LLMs.