Scorio Datasets for Test-Time Scaling
Scorio Trace, Lite, Math, and GPQA contain 1.4 million repeated reasoning attempts for evaluation, ranking, answer selection, and token-level confidence analysis.
Posts
Entropy of bfloat16 During Training: How Optimizers Shape Weight Distributions
During training, bfloat16 exponent bits evolve differently depending on the optimizer. Adam increases entropy, SGD decreases it, while AdamW consistently produces the ~2.6 bits observed in trained LLMs.
Entropy of bfloat16: 8 Bits Are Doing 2.6 Bits of Work
BFloat16 uses 8 bits to store exponents, but those 8 bits carry only about 2.6 bits of actual information in trained neural networks. Regardless of the initialization and training recipe.
Simulating LLM Evaluation Datasets Using Psychometric Models
Explore how Item Response Theory (IRT) and other psychometric models can simulate and analyze LLM evaluation datasets. Learn how difficulty, discrimination, and guessing parameters reveal model reasoning patterns, with interactive examples across multiple reading levels.
Simulating LLM Answers to Evaluation Datasets
Explore how simulating LLM responses to evaluation datasets with stochastic sampling is like flipping biased coins—revealing variability, bias, and the importance of multiple trials for reliable benchmarking.