Serving Reasoning LLMs Efficiently and Reliably [No Anime] (Slides)

.reveal .slides {
    text-align: left;
  }

.reveal section.conf-slide {
    box-sizing: border-box;
    width: 100%;
    height: 100%;
    padding: 54px 70px 44px;
    color: var(--conf-ink);
    background:
      linear-gradient(135deg, rgba(37, 99, 235, 0.055), rgba(255, 255, 255, 0) 32%),
      linear-gradient(225deg, rgba(15, 118, 110, 0.055), rgba(255, 255, 255, 0) 36%),
      #f8fafc;
    overflow: hidden;
  }

.reveal section.conf-title {
    padding: 64px 82px 54px;
  }

.reveal .conf-slide h1,
  .reveal .conf-slide h2,
  .reveal .conf-slide h3 {
    color: var(--conf-ink);
    letter-spacing: 0;
    text-transform: none;
  }

.reveal .conf-slide h1 {
    font-size: 76px;
    line-height: 1.03;
    margin: 0;
  }

.reveal .conf-slide h2 {
    font-size: 46px;
    line-height: 1.08;
    margin: 0 0 26px;
  }

.reveal .conf-slide h3 {
    font-size: 25px;
    line-height: 1.18;
    margin: 0 0 10px;
  }

.reveal .conf-slide p,
  .reveal .conf-slide li {
    color: var(--conf-muted);
    font-size: 24px;
    line-height: 1.36;
  }

.reveal .conf-slide strong {
    color: var(--conf-ink);
    font-weight: 720;
  }

.reveal .conf-kicker {
    color: var(--conf-teal);
    font-size: 18px;
    font-weight: 760;
    letter-spacing: 0.08em;
    margin-bottom: 14px;
    text-transform: uppercase;
  }

.reveal .conf-subtitle {
    max-width: 950px;
    color: var(--conf-muted);
    font-size: 31px;
    line-height: 1.32;
    margin-top: 26px;
  }

.reveal .conf-title-grid {
    display: grid;
    grid-template-columns: minmax(0, 1100px) 286px;
    gap: 42px;
    align-content: center;
    align-items: center;
    height: 100%;
    max-width: none;
  }

.reveal .conf-title-main {
    min-width: 0;
  }

.reveal .conf-title-qr {
    box-sizing: border-box;
    justify-self: end;
    width: 286px;
    padding: 18px;
    text-align: center;
  }

.reveal .conf-title-qr img {
    display: block;
    width: 238px;
    height: 238px;
    max-width: none;
    max-height: none;
    object-fit: contain;
    margin: 0 auto 13px;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .conf-title-qr strong {
    display: block;
    color: var(--conf-ink);
    font-size: 21px;
    line-height: 1.15;
  }

.reveal .conf-acl-logo {
    display: block;
    width: 300px;
    max-width: none;
    height: auto;
    margin: 0 0 48px;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .conf-meta {
    color: var(--conf-muted);
    font-size: 21px;
    line-height: 1.45;
    margin-top: 24px;
  }

.reveal .conf-presenter {
    display: block;
    color: var(--conf-ink);
    font-size: 33px;
    font-weight: 740;
    line-height: 1.14;
    margin-bottom: 5px;
  }

.reveal .conf-event-line {
    color: var(--conf-teal);
    font-size: 23px;
    font-weight: 760;
    line-height: 1.25;
    margin-top: 18px;
  }

.reveal .conf-logo-row {
    display: flex;
    align-items: center;
    gap: 24px;
    margin-top: 40px;
    flex-wrap: nowrap;
  }

.reveal .conf-logo-badge {
    display: flex;
    align-items: center;
    justify-content: center;
    height: 84px;
    padding: 10px 18px;
    border-radius: 14px;
    background: rgba(255, 255, 255, 0.92);
    box-shadow: 0 18px 46px rgba(15, 23, 42, 0.12);
  }

.reveal .conf-logo-badge.nsf {
    width: 104px;
    height: 92px;
    padding: 7px;
  }

.reveal .conf-logo-badge.scorio {
    width: 250px;
  }

.reveal .conf-logo-row img {
    display: block;
    width: auto;
    max-width: none;
    height: 100%;
    max-height: none;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-panel {
    border: 1px solid rgba(15, 23, 42, 0.14);
    border-radius: 8px;
    background: var(--conf-panel);
    box-shadow: 0 18px 46px rgba(15, 23, 42, 0.08);
  }

.reveal .conf-title-card {
    padding: 24px;
  }

.reveal .conf-title-card img {
    width: 100%;
    height: 370px;
    object-fit: contain;
    display: block;
  }

.reveal .conf-title-metric {
    display: grid;
    grid-template-columns: repeat(2, minmax(0, 1fr));
    gap: 12px;
    margin-top: 20px;
  }

.reveal .conf-title-metric p,
  .reveal .conf-metric p {
    margin: 0;
  }

.reveal .conf-title-metric strong,
  .reveal .conf-metric strong {
    display: block;
    color: var(--conf-blue);
    font-size: 34px;
    line-height: 1;
  }

.reveal .conf-title-metric span,
  .reveal .conf-metric span {
    display: block;
    color: var(--conf-muted);
    font-size: 16px;
    line-height: 1.22;
    margin-top: 7px;
  }

.reveal .conf-chip-row {
    display: flex;
    flex-wrap: wrap;
    gap: 10px;
    margin-top: 18px;
  }

.reveal .conf-chip {
    display: inline-flex;
    align-items: center;
    border: 1px solid rgba(15, 23, 42, 0.14);
    border-radius: 999px;
    color: var(--conf-muted);
    background: rgba(255, 255, 255, 0.74);
    font-size: 16px;
    font-weight: 720;
    line-height: 1;
    padding: 9px 13px;
  }

.reveal .conf-chip.blue {
    color: var(--conf-blue);
    background: var(--conf-blue-soft);
    border-color: rgba(37, 99, 235, 0.24);
  }

.reveal .conf-chip.teal {
    color: var(--conf-teal);
    background: var(--conf-teal-soft);
    border-color: rgba(15, 118, 110, 0.24);
  }

.reveal .conf-chip.amber {
    color: var(--conf-amber);
    background: var(--conf-amber-soft);
    border-color: rgba(180, 83, 9, 0.24);
  }

.reveal .conf-grid {
    display: grid;
    gap: 18px;
  }

.reveal .conf-grid.two {
    grid-template-columns: repeat(2, minmax(0, 1fr));
  }

.reveal .conf-grid.three {
    grid-template-columns: repeat(3, minmax(0, 1fr));
  }

.reveal .conf-grid.four {
    grid-template-columns: repeat(4, minmax(0, 1fr));
  }

.reveal .conf-card {
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.78);
    padding: 20px 22px;
    min-height: 138px;
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-card p {
    margin: 0;
    font-size: 20px;
  }

.reveal .conf-num {
    color: var(--conf-blue);
    font-size: 18px;
    font-weight: 780;
    margin-bottom: 12px;
  }

.reveal .conf-callout {
    border-left: 5px solid var(--conf-blue);
    border-radius: 8px;
    background: var(--conf-blue-soft);
    padding: 18px 22px;
    color: var(--conf-ink);
    font-size: 25px;
    line-height: 1.34;
  }

.reveal .conf-callout.teal {
    border-left-color: var(--conf-teal);
    background: var(--conf-teal-soft);
  }

.reveal .conf-callout.amber {
    border-left-color: var(--conf-amber);
    background: var(--conf-amber-soft);
  }

.reveal .conf-callout.rose {
    border-left-color: var(--conf-rose);
    background: var(--conf-rose-soft);
  }

.reveal .conf-big-question {
    margin-top: 24px;
    padding: 26px 30px;
    color: var(--conf-ink);
    font-size: 35px;
    line-height: 1.2;
  }

.reveal .conf-equation {
    display: flex;
    align-items: center;
    justify-content: center;
    box-sizing: border-box;
    min-width: 0;
    min-height: 112px;
    max-width: 100%;
    overflow-x: auto;
    border: 1px solid rgba(15, 23, 42, 0.12);
    border-radius: 8px;
    background: rgba(15, 23, 42, 0.04);
    color: var(--conf-ink);
    font-size: 31px;
    line-height: 1.25;
    padding: 18px 22px;
  }

/* Keep display math inside its panel instead of bleeding over the next
     column. Wide formulas still read best one size down: add `conf-equation sm`
     or give the equation a full-width row (see slide-patterns.md). */
  .reveal .conf-equation .katex { max-width: 100%; }

.reveal .conf-equation.sm {
    font-size: 24px;
  }

.reveal .conf-figure {
    border: 1px solid rgba(15, 23, 42, 0.12);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.84);
    padding: 16px;
  }

.reveal .conf-figure img {
    display: block;
    width: 100%;
    height: 100%;
    object-fit: contain;
  }

.reveal .conf-figure figcaption {
    color: var(--conf-muted);
    font-size: 16px;
    line-height: 1.25;
    margin-top: 10px;
  }

.reveal .conf-fig-short {
    height: 430px;
  }

.reveal .conf-fig-mid {
    height: 540px;
  }

.reveal .conf-fig-wide {
    height: 465px;
  }

.reveal .conf-fig-tall {
    height: 610px;
  }

.reveal .conf-split {
    display: grid;
    grid-template-columns: minmax(0, 0.98fr) minmax(420px, 0.82fr);
    gap: 30px;
    align-items: stretch;
  }

.reveal .conf-split.reverse {
    grid-template-columns: minmax(420px, 0.76fr) minmax(0, 1fr);
  }

.reveal .conf-metric {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 14px;
    margin-top: 18px;
  }

.reveal .conf-metric .conf-card {
    min-height: 104px;
  }

.reveal .conf-mini-table,
  .reveal .conf-table {
    width: 100%;
    border-collapse: collapse;
  }

.reveal .conf-mini-table {
    font-size: 18px;
  }

.reveal .conf-table {
    font-size: 20px;
  }

.reveal .conf-mini-table th,
  .reveal .conf-mini-table td,
  .reveal .conf-table th,
  .reveal .conf-table td {
    border-bottom: 1px solid rgba(15, 23, 42, 0.12);
    padding: 11px 10px;
    text-align: right;
    color: var(--conf-muted);
  }

.reveal .conf-mini-table th:first-child,
  .reveal .conf-mini-table td:first-child,
  .reveal .conf-table th:first-child,
  .reveal .conf-table td:first-child {
    text-align: left;
  }

.reveal .conf-mini-table th,
  .reveal .conf-table th {
    color: var(--conf-ink);
    font-weight: 760;
  }

.reveal .conf-win {
    color: var(--conf-green) !important;
    font-weight: 760;
    background: rgba(21, 128, 61, 0.08);
  }

.reveal .conf-loss {
    color: var(--conf-rose) !important;
  }

.reveal .conf-flow {
    display: grid;
    grid-template-columns: repeat(4, minmax(0, 1fr));
    gap: 12px;
    margin-top: 18px;
  }

.reveal .conf-flow .conf-card {
    min-height: 148px;
  }

.reveal .conf-flow h3 {
    font-size: 23px;
  }

.reveal .conf-list {
    margin: 0;
    padding-left: 1.1em;
  }

.reveal .conf-list li {
    margin: 0 0 12px;
  }

.reveal .conf-footer-note {
    position: absolute;
    left: 70px;
    right: 70px;
    bottom: 28px;
    color: rgba(71, 85, 105, 0.78);
    font-size: 16px;
  }

.reveal .conf-takeaway-header {
    display: flex;
    align-items: center;
    gap: 24px;
    margin: 0 0 34px;
  }

.reveal .conf-takeaway-icon {
    display: block;
    width: 104px;
    height: 104px;
    max-width: none;
    max-height: none;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-takeaway-title {
    color: var(--conf-ink);
    font-size: 62px;
    line-height: 0.98;
    margin: 0;
  }

.reveal .conf-takeaway-subtitle {
    color: var(--conf-teal);
    font-size: 28px;
    font-weight: 760;
    line-height: 1.12;
    margin-top: 12px;
  }

.reveal .conf-takeaway-grid {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 22px;
  }

.reveal .conf-takeaway-card {
    position: relative;
    min-height: 250px;
    padding: 30px 30px 28px;
    overflow: hidden;
    border-top: 5px solid var(--conf-teal);
    background:
      linear-gradient(180deg, rgba(15, 118, 110, 0.08), rgba(255, 255, 255, 0.82) 42%),
      rgba(255, 255, 255, 0.84);
    box-shadow: 0 18px 42px rgba(15, 23, 42, 0.09);
  }

.reveal .conf-takeaway-card:nth-child(2) {
    border-top-color: var(--conf-blue);
    background:
      linear-gradient(180deg, rgba(37, 99, 235, 0.08), rgba(255, 255, 255, 0.82) 42%),
      rgba(255, 255, 255, 0.84);
  }

.reveal .conf-takeaway-card:nth-child(3) {
    border-top-color: var(--conf-amber);
    background:
      linear-gradient(180deg, rgba(180, 83, 9, 0.08), rgba(255, 255, 255, 0.82) 42%),
      rgba(255, 255, 255, 0.84);
  }

.reveal .conf-takeaway-num {
    display: inline-grid;
    place-items: center;
    width: 42px;
    height: 42px;
    margin-bottom: 20px;
    border-radius: 50%;
    background: var(--conf-teal-soft);
    color: var(--conf-teal);
    font-size: 22px;
    font-weight: 820;
  }

.reveal .conf-takeaway-card:nth-child(2) .conf-takeaway-num {
    background: var(--conf-blue-soft);
    color: var(--conf-blue);
  }

.reveal .conf-takeaway-card:nth-child(3) .conf-takeaway-num {
    background: var(--conf-amber-soft);
    color: var(--conf-amber);
  }

.reveal .conf-takeaway-card h3 {
    font-size: 30px;
    line-height: 1.08;
    min-height: 42px;
    margin-bottom: 16px;
  }

.reveal .conf-takeaway-card p {
    font-size: 25px;
    line-height: 1.34;
    min-height: 128px;
  }

.reveal .conf-closing-layout {
    display: grid;
    grid-template-rows: auto minmax(0, 1fr) auto;
    gap: 26px;
    align-items: stretch;
    height: 100%;
  }

.reveal .conf-closing-main {
    display: grid;
    justify-items: center;
    align-content: start;
    text-align: center;
    padding-top: 8px;
  }

.reveal .conf-closing-title {
    font-size: 72px;
    line-height: 1.03;
    margin: 0 0 18px;
  }

.reveal .conf-closing-main .conf-subtitle {
    max-width: none;
    margin-top: 0;
  }

.reveal .conf-closing-authors {
    color: var(--conf-muted);
    font-size: 22px;
    line-height: 1.25;
    margin-top: 12px;
  }

.reveal .conf-qr-grid {
    display: grid;
    grid-template-columns: repeat(4, 260px);
    justify-content: center;
    gap: 24px;
    width: min(1120px, 100%);
    margin: 32px auto 0;
  }

.reveal .conf-qr-item {
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.82);
    padding: 18px;
    text-align: center;
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-qr-item img {
    display: block;
    width: 210px;
    height: 210px;
    max-width: none;
    max-height: none;
    object-fit: contain;
    margin: 0 auto 12px;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .conf-qr-item strong {
    display: block;
    color: var(--conf-ink);
    font-size: 20px;
    line-height: 1.1;
  }

.reveal .conf-contact-line {
    max-width: 1220px;
    margin: 18px auto 0;
    color: var(--conf-muted);
    font-size: 20px;
    line-height: 1.25;
  }

.reveal .conf-contact-email {
    color: var(--conf-blue);
    font-weight: 780;
    white-space: nowrap;
  }

.reveal .conf-ack-stack {
    display: grid;
    grid-template-columns: 450px 760px;
    justify-content: center;
    align-items: center;
    gap: 34px;
    width: 100%;
    margin: 0 auto;
    padding: 0;
  }

.reveal .conf-ack-item {
    display: grid;
    grid-template-columns: auto minmax(0, 1fr);
    gap: 22px;
    align-items: center;
    border: 0;
    border-radius: 0;
    background: transparent;
    padding: 0;
    box-shadow: none;
  }

.reveal .conf-ack-cwru {
    grid-template-columns: 290px minmax(0, 1fr);
  }

.reveal .conf-ack-item img {
    display: block;
    width: auto;
    max-width: none;
    max-height: none;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-ack-nsf img {
    width: 84px;
    height: 84px;
  }

.reveal .conf-ack-cwru img {
    width: 270px;
    height: auto;
  }

.reveal .conf-ack-item h3 {
    font-size: 26px;
    line-height: 1.08;
    margin: 0 0 8px;
    white-space: nowrap;
  }

.reveal .conf-ack-item p {
    font-size: 22px;
    line-height: 1.18;
    margin: 0;
    white-space: nowrap;
  }

.reveal .slides section.conf-slide .conf-acl-logo {
    width: 300px;
    max-width: none;
    height: auto;
    margin: 0 0 48px;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .slides section.conf-slide .conf-logo-row img {
    width: auto;
    max-width: none;
    height: 64px;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .slides section.conf-slide .conf-logo-badge.nsf img {
    width: 76px;
    height: 76px;
  }

.reveal .slides section.conf-slide .conf-logo-badge.scorio img {
    width: 214px;
    height: auto;
  }

.reveal .slides section.conf-slide .conf-qr-item img {
    width: 210px;
    height: 210px;
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0 auto 12px;
  }

.reveal .slides section.conf-slide .conf-title-qr img {
    width: 238px;
    height: 238px;
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0 auto 13px;
  }

.reveal .slides section.conf-slide .conf-ack-item img {
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .slides section.conf-slide .conf-ack-nsf img {
    width: 84px;
    height: 84px;
  }

.reveal .slides section.conf-slide .conf-ack-cwru img {
    width: 270px;
    height: auto;
  }

.reveal .conf-title-thesis {
    max-width: 1080px;
    color: var(--conf-muted);
    font-size: 29px;
    line-height: 1.28;
    margin-top: 28px;
  }

.reveal .conf-slide .conf-lead {
    max-width: 1210px;
    color: var(--conf-muted);
    font-size: 30px;
    line-height: 1.28;
    margin: -8px 0 28px;
  }

.reveal .conf-slide .conf-small {
    color: var(--conf-muted);
    font-size: 18px;
    line-height: 1.3;
  }

.reveal .conf-stat-row {
    display: grid;
    grid-template-columns: repeat(4, minmax(0, 1fr));
    gap: 14px;
    margin-top: 20px;
  }

.reveal .conf-stat {
    border: 1px solid rgba(15, 23, 42, 0.12);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.78);
    padding: 18px 18px 16px;
    min-height: 112px;
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-stat strong {
    display: block;
    color: var(--conf-blue);
    font-size: 43px;
    line-height: 1;
  }

.reveal .conf-stat span {
    display: block;
    color: var(--conf-muted);
    font-size: 17px;
    line-height: 1.22;
    margin-top: 9px;
  }

.reveal .conf-slide .conf-table.compact {
    font-size: 17px;
  }

.reveal .conf-slide .conf-table.compact th,
  .reveal .conf-slide .conf-table.compact td {
    padding: 8px 8px;
  }

.reveal .conf-slide .conf-table.wide-text td,
  .reveal .conf-slide .conf-table.wide-text th {
    text-align: left;
  }

.reveal .conf-slide .conf-table.wide-text td:last-child,
  .reveal .conf-slide .conf-table.wide-text th:last-child {
    text-align: right;
  }

.reveal .conf-slide .conf-table .conf-method-cell {
    color: var(--conf-ink);
    font-weight: 720;
  }

.reveal .conf-two-thirds {
    display: grid;
    grid-template-columns: minmax(0, 1.05fr) minmax(420px, 0.72fr);
    gap: 28px;
    align-items: stretch;
  }

.reveal .conf-figure-grid {
    display: grid;
    grid-template-columns: minmax(0, 1fr) minmax(390px, 0.52fr);
    gap: 24px;
    align-items: stretch;
  }

.reveal .conf-figure-grid.reverse {
    grid-template-columns: minmax(390px, 0.52fr) minmax(0, 1fr);
  }

.reveal .conf-figure-grid .conf-figure {
    height: 590px;
  }

.reveal .conf-figure-grid .conf-figure.short {
    height: 500px;
  }

.reveal .conf-stack {
    display: grid;
    gap: 16px;
    align-content: start;
  }

.reveal .conf-target-card {
    min-height: 210px;
  }

.reveal .conf-target-card .conf-equation {
    min-height: 82px;
    margin-bottom: 16px;
    font-size: 27px;
  }

.reveal .conf-method-map {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 18px;
    margin-top: 20px;
  }

.reveal .conf-method-map .conf-card {
    min-height: 255px;
  }

.reveal .conf-method-map .conf-equation {
    min-height: 76px;
    font-size: 23px;
    margin-bottom: 14px;
  }

.reveal .conf-rule-grid {
    display: grid;
    grid-template-columns: repeat(5, minmax(0, 1fr));
    gap: 13px;
    margin-top: 26px;
  }

.reveal .conf-rule-grid .conf-card {
    min-height: 190px;
    padding: 18px 18px;
  }

.reveal .conf-rule-grid h3 {
    font-size: 22px;
  }

.reveal .conf-code {
    box-sizing: border-box;
    max-width: 100%;
    overflow-x: auto;
    border: 1px solid rgba(148, 163, 184, 0.2);
    border-radius: 12px;
    background: linear-gradient(180deg, #111c33, #0b1424);
    color: #dbe3f1;
    font-family: ui-monospace, "SF Mono", "SFMono-Regular", "Menlo", "JetBrains Mono", "Cascadia Code", "Consolas", monospace;
    font-size: 18px;
    line-height: 1.5;
    letter-spacing: -0.01em;
    margin: 0;
    padding: 20px 22px;
    white-space: pre;
    tab-size: 2;
    -webkit-font-smoothing: antialiased;
    text-rendering: optimizeLegibility;
    box-shadow: 0 16px 34px rgba(8, 15, 30, 0.4), inset 0 1px 0 rgba(255, 255, 255, 0.05);
  }

.reveal .conf-code .cm {
    color: #6b7a92;
    font-style: italic;
  }

.reveal .conf-code .kw {
    color: #7dd3fc;
  }

.reveal .conf-code .fn {
    color: #86efac;
  }

.reveal .conf-code .arg {
    color: #fcd34d;
  }

.reveal .conf-code .str {
    color: #fca5a5;
  }

.reveal .conf-code .num {
    color: #c4b5fd;
  }

.reveal .conf-qrs-small {
    display: grid;
    grid-template-columns: repeat(2, 180px);
    gap: 18px;
    justify-content: start;
    margin-top: 16px;
  }

.reveal .conf-qrs-small .conf-qr-item {
    padding: 14px;
  }

.reveal .conf-qrs-small .conf-qr-item img {
    width: 144px;
    height: 144px;
  }

.reveal .conf-scorio-header {
    display: grid;
    grid-template-columns: 250px minmax(0, 1fr);
    gap: 24px;
    align-items: center;
    margin-bottom: 18px;
  }

.reveal .conf-scorio-logo-panel {
    display: flex;
    align-items: center;
    justify-content: flex-start;
    height: 104px;
  }

.reveal .conf-scorio-logo-panel img {
    display: block;
    width: 202px;
    height: auto;
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-scorio-header h2 {
    font-size: 40px;
    line-height: 1.08;
    margin: 0;
  }

.reveal .conf-scorio-heading {
    display: grid;
    gap: 9px;
    align-content: center;
  }

.reveal .conf-scorio-docs {
    justify-self: start;
    display: inline-flex;
    align-items: center;
    gap: 7px;
    border: 1px solid rgba(15, 118, 110, 0.3);
    border-radius: 999px;
    background: var(--conf-teal-soft);
    color: var(--conf-teal);
    font-size: 18px;
    font-weight: 780;
    line-height: 1;
    padding: 9px 15px;
    text-decoration: none;
  }

.reveal .conf-scorio-docs:hover {
    background: rgba(15, 118, 110, 0.17);
  }

.reveal .conf-scorio-app {
    border: 1px solid rgba(15, 118, 110, 0.22);
    border-radius: 999px;
    background: var(--conf-teal-soft);
    color: var(--conf-teal);
    font-size: 17px;
    font-weight: 780;
    line-height: 1;
    padding: 11px 15px;
    text-align: center;
  }

.reveal .conf-scorio-layout {
    display: grid;
    grid-template-columns: minmax(0, 0.78fr) minmax(0, 1.22fr);
    gap: 18px;
    align-items: stretch;
  }

.reveal .conf-scorio-panel,
  .reveal .conf-api-board {
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.8);
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-scorio-panel {
    padding: 18px;
  }

.reveal .conf-scorio-panel h3,
  .reveal .conf-api-board h3 {
    font-size: 23px;
    line-height: 1.08;
    margin: 0 0 12px;
  }

.reveal .conf-scorio-code {
    font-size: 15px;
    line-height: 1.5;
    padding: 16px 18px;
  }

.reveal .conf-scorio-panel .conf-scorio-code {
    font-size: 14px;
    line-height: 1.55;
  }

.reveal .conf-scorio-note {
    color: var(--conf-muted);
    font-size: 18px;
    line-height: 1.26;
    margin-top: 14px;
  }

.reveal .conf-scorio-note strong {
    color: var(--conf-ink);
  }

.reveal .conf-scorio-use {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 8px;
    margin-top: 14px;
  }

.reveal .conf-scorio-use .conf-scorio-app {
    display: grid;
    place-items: center;
    min-height: 52px;
    border: 1px solid rgba(15, 118, 110, 0.22);
    border-radius: 999px;
    background: var(--conf-teal-soft);
    color: var(--conf-teal);
    font-size: 15px;
    font-weight: 780;
    line-height: 1.08;
    padding: 10px 12px;
    text-align: center;
  }

.reveal .conf-api-board {
    padding: 16px 18px 14px;
  }

.reveal .conf-api-board h3 {
    display: flex;
    align-items: baseline;
    justify-content: space-between;
    gap: 16px;
  }

.reveal .conf-api-board h3 span {
    color: var(--conf-muted);
    font-size: 15px;
    font-weight: 680;
  }

.reveal .conf-api-group {
    padding: 8px 0;
    border-top: 1px solid rgba(15, 23, 42, 0.11);
  }

.reveal .conf-api-head {
    display: flex;
    align-items: baseline;
    gap: 12px;
    margin-bottom: 4px;
  }

.reveal .conf-api-head code {
    color: var(--conf-ink);
    font-size: 16px;
    font-weight: 820;
  }

.reveal .conf-api-head span {
    color: var(--conf-teal);
    font-size: 14px;
    font-weight: 760;
  }

.reveal .conf-api-columns {
    display: grid;
    grid-template-columns: repeat(2, minmax(0, 1fr));
    column-gap: 16px;
    row-gap: 2px;
  }

.reveal .conf-slide .conf-api-line {
    color: var(--conf-muted);
    font-size: 13px;
    line-height: 1.22;
    margin: 0 0 5px;
  }

.reveal .conf-slide .conf-api-line strong {
    display: inline;
    color: var(--conf-ink);
    font-size: 13px;
    font-weight: 800;
  }

.reveal .conf-slide .conf-api-line code {
    display: inline-block;
    border-radius: 5px;
    background: rgba(15, 23, 42, 0.055);
    color: var(--conf-ink);
    font-size: 11.5px;
    line-height: 1.18;
    margin: 0 2px 3px 0;
    padding: 2px 5px;
  }

.reveal .conf-slide .conf-fit-list li {
    font-size: 21px;
    line-height: 1.28;
    margin-bottom: 10px;
  }

.reveal .conf-slide .conf-tight-list li {
    font-size: 19px;
    line-height: 1.24;
    margin-bottom: 8px;
  }

.reveal .conf-banner {
    display: grid;
    grid-template-columns: auto minmax(0, 1fr);
    gap: 18px;
    align-items: center;
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.78);
    padding: 18px 22px;
    margin-top: 20px;
  }

.reveal .conf-banner-number {
    color: var(--conf-blue);
    font-size: 54px;
    font-weight: 780;
    line-height: 1;
  }

.reveal .conf-banner p {
    margin: 0;
    font-size: 23px;
    line-height: 1.26;
  }

.reveal .conf-label {
    display: inline-flex;
    align-items: center;
    border-radius: 999px;
    background: rgba(15, 23, 42, 0.08);
    color: var(--conf-ink);
    font-size: 15px;
    font-weight: 760;
    line-height: 1;
    padding: 8px 10px;
    text-transform: uppercase;
  }

.reveal .conf-figure img.no-stretch {
    height: auto;
    max-height: 100%;
  }

/* ---- kvq component classes (reused KV-cache slides) ---- */
  .reveal .conf-quanto-layout { align-items: start; }
  .reveal .conf-metric.conf-quanto-metrics {
    grid-template-columns: minmax(0, 1fr);
    grid-template-rows: repeat(3, minmax(0, 1fr));
    gap: 10px;
    height: 246px;
    margin-top: 0;
  }
  .reveal .conf-metric.conf-quanto-metrics .conf-card {
    display: grid; align-content: center; min-height: 0; padding: 10px 20px;
  }
  .reveal .conf-metric.conf-quanto-metrics strong { font-size: 28px; line-height: 1.12; }
  .reveal .conf-metric.conf-quanto-metrics p > span { font-size: 15px; line-height: 1.15; margin-top: 4px; }

.reveal .conf-memory-layout {
    display: grid;
    grid-template-columns: minmax(0, 0.98fr) minmax(420px, 0.82fr);
    gap: 24px 30px; align-items: stretch;
  }
  .reveal .conf-memory-context { display: grid; gap: 14px; }
  .reveal .conf-need-box {
    grid-column: 1 / -1; min-height: 134px;
    border-left: 5px solid var(--conf-teal); background: var(--conf-teal-soft); padding: 24px 30px;
  }
  .reveal .conf-need-box p { font-size: 28px; }

.reveal .conf-accuracy-table {
    table-layout: fixed; overflow: hidden; border-collapse: separate; border-spacing: 0;
    background: rgba(255, 255, 255, 0.9); box-shadow: 0 18px 38px rgba(15, 23, 42, 0.09);
  }
  .reveal .conf-accuracy-table th, .reveal .conf-accuracy-table td {
    padding: 16px 20px; border-bottom: 1px solid rgba(15, 23, 42, 0.1); font-size: 21px;
  }
  .reveal .conf-accuracy-table th {
    background: linear-gradient(180deg, rgba(15, 23, 42, 0.055), rgba(15, 23, 42, 0.015)); font-size: 20px;
  }
  .reveal .conf-accuracy-table th:first-child, .reveal .conf-accuracy-table td:first-child { width: 42%; }
  .reveal .conf-accuracy-table td:first-child { color: var(--conf-ink); font-weight: 720; }
  .reveal .conf-accuracy-table th:not(:first-child), .reveal .conf-accuracy-table td:not(:first-child) {
    width: 29%; font-variant-numeric: tabular-nums;
  }
  .reveal .conf-accuracy-table tbody tr:nth-child(even) td:not(.conf-win) { background: rgba(15, 23, 42, 0.025); }
  .reveal .conf-accuracy-table tbody tr:last-child td { border-bottom: 0; }
  .reveal .conf-accuracy-table .conf-win { background: rgba(21, 128, 61, 0.13); color: var(--conf-green) !important; font-weight: 820; }

/* ---- merged-deck closing layout (two children: main + acks) ---- */
  .reveal .conf-closing-layout { grid-template-rows: auto auto; align-content: center; gap: 44px; height: 100%; }

/* ---- per-paper section dividers (paper headline screenshot) ---- */
  .reveal .conf-paper-divider {
    display: grid;
    grid-template-rows: auto auto auto auto;
    align-content: center;
    gap: 22px;
    height: 100%;
  }
  .reveal .conf-paper-divider .conf-kicker { margin: 0; }
  .reveal .conf-paper-divider h2 { margin: 0; }
  .reveal .conf-paper-shot {
    box-sizing: border-box;
    display: grid;
    place-items: center;
    margin: 0;
    padding: 24px 32px;
    background: #ffffff;
    border: 1px solid rgba(15, 23, 42, 0.14);
    border-radius: 8px;
    box-shadow: 0 18px 46px rgba(15, 23, 42, 0.08);
  }
  .reveal .conf-paper-shot img {
    display: block;
    width: auto;
    max-width: 100%;
    height: auto;
    max-height: 432px;
    margin: 0 auto;
    border-radius: 0;
    box-shadow: none;
  }
  .reveal .conf-paper-foot {
    margin: 0;
    color: var(--conf-muted);
    font-size: 22px;
    line-height: 1.32;
  }
  .reveal .conf-paper-foot strong { color: var(--conf-ink); }

/* venue name caption under each logo on the title slide */
  .reveal .conf-venue-row { align-items: flex-start; }
  .reveal .conf-venue-item {
    display: flex;
    flex-direction: column;
    align-items: center;
    gap: 12px;
  }
  .reveal .conf-venue-name {
    color: var(--conf-muted);
    font-size: 18px;
    font-weight: 720;
    letter-spacing: 0.03em;
  }

/* venue chip (logo + name) instead of a kicker on each paper divider */
  .reveal .conf-paper-venue {
    display: flex;
    align-items: center;
    gap: 16px;
  }
  .reveal .conf-paper-venue .conf-venue-chip {
    display: flex;
    align-items: center;
    justify-content: center;
    height: 56px;
    padding: 8px 16px;
    border-radius: 12px;
    background: rgba(255, 255, 255, 0.92);
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.1);
  }
  .reveal .conf-paper-venue .conf-venue-chip img {
    display: block;
    width: auto;
    height: 38px;
    max-width: none;
    max-height: none;
    margin: 0;
    border-radius: 0;
    box-shadow: none;
    object-fit: contain;
  }
  .reveal .conf-paper-venue-name {
    color: var(--conf-teal);
    font-size: 20px;
    font-weight: 760;
    letter-spacing: 0.06em;
    text-transform: uppercase;
  }

/* four-up takeaways (one card per paper) */
  .reveal .conf-takeaway-grid.quad { grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 18px; }
  .reveal .conf-takeaway-grid.quad .conf-takeaway-card { min-height: 322px; padding: 26px 24px 24px; }
  .reveal .conf-takeaway-grid.quad .conf-takeaway-card h3 { font-size: 24px; line-height: 1.12; min-height: 58px; margin-bottom: 12px; }
  .reveal .conf-takeaway-grid.quad .conf-takeaway-card p { font-size: 19px; line-height: 1.34; min-height: 0; }
  .reveal .conf-takeaway-grid.quad .conf-takeaway-num { width: 38px; height: 38px; margin-bottom: 16px; font-size: 20px; }
  .reveal .conf-takeaway-card:nth-child(4) {
    border-top-color: var(--conf-violet);
    background:
      linear-gradient(180deg, rgba(124, 58, 237, 0.08), rgba(255, 255, 255, 0.82) 42%),
      rgba(255, 255, 255, 0.84);
  }
  .reveal .conf-takeaway-card:nth-child(4) .conf-takeaway-num {
    background: var(--conf-violet-soft);
    color: var(--conf-violet);
  }
</style>

<div class="conf-title-grid">
  <div class="conf-title-main">
    <h1>Serving Reasoning LLMs Efficiently and Reliably</h1>
    <div class="conf-logo-row conf-venue-row">
      <div class="conf-venue-item">
        <span class="conf-logo-badge"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/neurips.svg" alt="NeurIPS 2025" /></span>
        <span class="conf-venue-name">NeurIPS 2025</span>
      </div>
      <div class="conf-venue-item">
        <span class="conf-logo-badge"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/iclr.svg" alt="ICLR 2026" /></span>
        <span class="conf-venue-name">ICLR 2026</span>
      </div>
      <div class="conf-venue-item">
        <span class="conf-logo-badge"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/acl.svg" alt="ACL 2026" /></span>
        <span class="conf-venue-name">ACL 2026</span>
      </div>
    </div>
    <div class="conf-meta">
      <span class="conf-presenter">Vipin Chaudhary</span>
      Case Western Reserve University
    </div>
    <div class="conf-logo-row">
      <span class="conf-logo-badge"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/cwru.svg" alt="Case Western Reserve University" /></span>
      <span class="conf-logo-badge nsf"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/nsf.svg" alt="National Science Foundation" /></span>
    </div>
  </div>
  <aside class="conf-title-qr conf-panel" aria-label="Interactive slides QR code">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/slide_qr.svg" alt="QR code for the interactive slides" />
    <strong>Scan for interactive slides</strong>
  </aside>
</div>

Note:
  One sentence for the whole talk: reasoning models pushed the cost of intelligence from training to inference, and these four papers — across NeurIPS, ICLR, and ACL — make that shift both affordable to run and rigorous to measure. The discipline that ties them together is right in the title: be exactly lossless wherever a deviation would be unacceptable, and wherever you can't be, quantify the error you're left with. Two papers go after the memory bill, two after the measurement bill.

---

<div class="conf-kicker">Motivation</div>
<h2>Reasoning moved the bill from training to inference</h2>
<p class="conf-lead">A reasoning model spends its compute at inference time &mdash; long chains of thought, and many sampled attempts per question.</p>

<div class="conf-flow">
  <div class="conf-card"><div class="conf-num">01</div><h3>Think longer</h3><p>One answer becomes a long chain of thought; tokens per query rise by 10&ndash;100&times;.</p></div>
  <div class="conf-card"><div class="conf-num">02</div><h3>Sample more</h3><p>Pass@k, self-consistency, best-of-N &mdash; we draw many trials per question, not one.</p></div>
  <div class="conf-card"><div class="conf-num">03</div><h3>Inference dominates</h3><p>The cost and the memory of an LLM increasingly live at serving time, not training time.</p></div>
  <div class="conf-card"><div class="conf-num">04</div><h3>Two bills arrive</h3><p>Running it costs memory. Trusting it costs measurement. Both get harder as N grows.</p></div>
</div>

<div class="conf-callout teal" style="margin-top: 26px;">Test-time scaling buys accuracy with inference compute &mdash; and the bill comes due twice.</div>

Note:
  The shift everyone is living through: a reasoning model doesn't answer in one shot — it writes a long chain of thought, so tokens per query jump by ten to a hundred times, and then we sample it many times per question for pass@k, self-consistency, best-of-N. Training used to be the expensive part; now the cost and the memory live at serving time. That hands us two bills, and both get worse as the sampling budget N grows: running the model costs memory, and trusting its score costs measurement. The rest of the talk is those two bills.

---

<div class="conf-kicker">The argument</div>
<h2>Lossless where you must, calibrated where you can't</h2>
<p class="conf-lead">Test-time scaling sends two bills: one to run the model, one to trust the numbers.</p>

<div class="conf-grid two">
  <div class="conf-card">
    <h3>The memory bill &mdash; run it cheaply</h3>
    <ul class="conf-list conf-fit-list">
      <li><strong>DFloat11</strong> &mdash; losslessly compress the weights <span class="conf-small">(NeurIPS 2025)</span></li>
      <li><strong>More keys, less values</strong> &mdash; quantize the KV cache by information density <span class="conf-small">(ACL 2026)</span></li>
    </ul>
  </div>
  <div class="conf-card">
    <h3>The measurement bill &mdash; measure it rigorously</h3>
    <ul class="conf-list conf-fit-list">
      <li><strong>Don't Pass@k</strong> &mdash; a Bayesian posterior with credible intervals <span class="conf-small">(ICLR 2026)</span></li>
      <li><strong>Ranking under test-time scaling</strong> &mdash; which ranking to trust at a given budget <span class="conf-small">(ACL 2026)</span></li>
    </ul>
  </div>
</div>

<div class="conf-callout teal" style="margin-top: 26px;">One discipline throughout: spend bits where the information is, then report the uncertainty you can't remove.</div>

Note:
  Here's the whole talk on one slide. Two papers make reasoning models cheaper to run — DFloat11 losslessly compresses the weights, and our KV-cache work spends a tight bit budget where the information actually is. Two make them trustworthy to measure — Don't Pass@k turns a score into a posterior with error bars, and the ranking paper tells you which leaderboard to trust at a given budget. The title is the discipline connecting all four: be exactly lossless when any deviation is unacceptable, and when you can't be, quantify the error you can't remove.

---

<div class="conf-kicker">The memory bill</div>
<h2>Long thinking is memory-bound &mdash; and not all bits are equal</h2>
<p class="conf-lead">Two things fill the GPU at serving time. Both shrink once you stop spending equal precision everywhere.</p>

<div class="conf-grid two">
  <div class="conf-card">
    <div class="conf-num">WEIGHTS</div>
    <h3>Static, and mostly redundant</h3>
    <p>BFloat16 stores every weight in 16 bits, but its exponent field is nearly empty. DFloat11 reclaims it &mdash; losslessly.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">KV CACHE</div>
    <h3>Dynamic, and grows with the chain</h3>
    <p>Every reasoning step appends keys and values. Quantize them &mdash; but keys and values are not equal. More for keys.</p>
  </div>
</div>

<div class="conf-callout" style="margin-top: 26px;">The shared move: find where the information actually lives, and spend precision there.</div>

Note:
  At serving time two things fill the GPU: the weights, which are static, and the KV cache, which grows with every token the model thinks. The first paper shrinks the weights, the second shrinks the cache, and they share one move — find where the bits actually carry information, and only spend precision there. For the weights, the slack is hiding in the floating-point exponent; for the cache, it's the asymmetry between keys and values.

---

<div class="conf-paper-divider">
  <div class="conf-paper-venue">
    <span class="conf-venue-chip"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/neurips.svg" alt="NeurIPS 2025" /></span>
    <span class="conf-paper-venue-name">NeurIPS 2025</span>
  </div>
  <h2>First, the weights: 30% smaller, every output identical</h2>
  <figure class="conf-paper-shot">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/header_df11.webp" alt="Paper headline: 70% Size, 100% Accuracy — Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float. Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, Anshumali Shrivastava." />
  </figure>
  <p class="conf-paper-foot">Bit-for-bit identical &mdash; safe even where any deviation is disqualifying.</p>
</div>

Note:
  The first paper, at NeurIPS this year, attacks the static cost — the weights. The title is the whole pitch: seventy percent of the size, a hundred percent of the accuracy. About a third smaller, and not a single output bit changes. That last part is what makes it usable in finance or healthcare, where an answer that merely differs from the original model is already a non-starter.

---

<div class="conf-kicker">Lossy compression</div>
<h2>Quantization shrinks models, but the loss is real and hard to see</h2>

<div class="conf-flow">
  <div class="conf-card">
    <div class="conf-num">01</div>
    <h3>Accuracy drops</h3>
    <p>8-bit SmoothQuant on a distilled Qwen 1.5B loses 9.09% average accuracy on reasoning tasks.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">02</div>
    <h3>Answers flip</h3>
    <p>W8A16 GPTQ Qwen2 1.5B moves only 0.3% on GSM8K, yet 6.37% of answers flip correctness.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">03</div>
    <h3>Benchmarks hide it</h3>
    <p>Perplexity and MMLU are narrow; the gap an end-user feels rarely shows up there.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">04</div>
    <h3>Compliance breaks</h3>
    <p>In finance or healthcare, an output that differs from the original is a non-starter.</p>
  </div>
</div>

<div class="conf-callout amber" style="margin-top: 28px;">
  Lossy quantization adds an uncontrolled variable you have to re-validate for every model, task, and bit-width.
</div>

Note:
  Quantization is the default, and at eight bits it looks safe on the usual leaderboards. But the loss is real and it hides. Eight-bit SmoothQuant on a distilled Qwen-1.5B drops average reasoning accuracy by nine percent. Worse, GPTQ on Qwen2-1.5B moves GSM8K by only three tenths of a percent — yet more than six percent of individual answers flip correctness, so the benchmark looks fine while behavior drifts underneath it. Perplexity and MMLU just don't surface that, and in a regulated setting any deviation is disqualifying. So the question that drives the paper: can we get the memory savings with exactly zero loss?

---

<div class="conf-kicker">BFloat16 entropy</div>
<h2>BFloat16 spends 8 bits on an exponent worth about 2.6</h2>

<div class="conf-split">
  <figure class="conf-figure conf-fig-mid">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/entropy.webp" alt="Shannon entropy of the sign, exponent, and mantissa bits of BFloat16 weights across five LLMs" />
    <figcaption>Shannon entropy of each BFloat16 field across LLMs. Sign (≈1 bit) and mantissa (≈7 bits) are near their widths; the 8-bit exponent carries only ≈2.6 bits.</figcaption>
  </figure>
  <div>
    <div class="conf-card">
      <h3>The exponent is the slack</h3>
      <p>Sign and mantissa are essentially uniform, so they barely compress. The exponent is wildly skewed: only ~40 of 256 possible values ever appear.</p>
    </div>
    <div class="conf-card" style="margin-top: 16px;">
      <h3>Same story everywhere</h3>
      <p>Llama, Gemma, Qwen, Mistral — the ~2.6-bit exponent holds across families. This is a property of the format, not one model.</p>
    </div>
    <div class="conf-callout teal" style="margin-top: 16px;">
      ≈5 bits per weight are pure redundancy — reclaimable losslessly.
    </div>
  </div>
</div>

Note:
  This figure is the whole idea. Measure the actual information content — the Shannon entropy — of each field of a BFloat16 weight. The sign is about one bit and the mantissa about six-point-eight of its seven, so both are essentially random and there's nothing to compress. The exponent is the outlier: only about forty of its 256 possible values ever appear, so its real content is about 2.6 bits, not eight. And this holds across Llama, Gemma, Qwen, Mistral — it's a property of the format, not of any one model. That's roughly five free bits per weight, if we encode the exponent by how often each value occurs.

---

<div class="conf-kicker">Dynamic-Length Float</div>
<h2>DF11: Huffman-code the exponent, leave the rest alone</h2>

<div class="conf-equation" style="margin-bottom: 22px;">
  \[ x = (-1)^{\text{sign}} \cdot 2^{\,\text{exponent} - 127} \cdot (1.\text{mantissa}) \]
</div>

<div class="conf-split">
  <figure class="conf-figure conf-fig-short">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/DFloat11.webp" alt="The DFloat11 format: variable-length Huffman-coded exponents plus fixed sign and mantissa bits, decoded through a Huffman tree" />
    <figcaption>Exponents become variable-length Huffman codes (avg ~2.6 bits); sign (1 bit) and mantissa (7 bits) stay fixed and are decoded by a tree walk.</figcaption>
  </figure>
  <div>
    <div class="conf-callout teal">
      Frequent exponents get short codes, rare ones long — near the information-theoretic limit, with no rounding.
    </div>
    <ul class="conf-list conf-fit-list" style="margin-top: 18px;">
      <li><strong>16 → ~11 bits</strong> average weight width</li>
      <li><strong>~70%</strong> of the original model size</li>
      <li><strong>Bit-for-bit identical</strong> weights after decompression</li>
    </ul>
  </div>
</div>

Note:
  The format itself is simple. Build one Huffman tree over the exponent values across all the linear-layer weights, replace each fixed eight-bit exponent with its variable-length code — frequent values get short codes, rare ones long — and leave the sign and mantissa completely untouched. The average weight drops from sixteen bits to about eleven, so roughly thirty percent smaller. And because we only re-encode bits that were already there and round nothing, decompression returns the original weights bit-for-bit. It's compression right up against the information-theoretic limit, with no approximation.

---

<div class="conf-kicker">Online decompression</div>
<h2>Variable-length codes are murder on a GPU</h2>

<div class="conf-grid three">
  <div class="conf-card">
    <h3>Can't multiply compressed</h3>
    <p>Entropy-coded weights aren't matrices. Each one must be decoded back to BFloat16 on the fly, used, then discarded.</p>
  </div>
  <div class="conf-card">
    <h3>Huffman decode is serial</h3>
    <p>Classic decoding walks the tree bit by bit — inherently sequential, the opposite of what a GPU wants.</p>
  </div>
  <div class="conf-card">
    <h3>One thread per weight stalls</h3>
    <p>Naive per-element threads leave the GPU mostly idle and latency high. The packed stream has no obvious split points.</p>
  </div>
</div>

<div class="conf-callout" style="margin-top: 28px;">
  The fix: SRAM lookup tables instead of a tree walk, a two-pass split that turns the serial stream into parallel work, and decoding a whole transformer block at once — so the overhead amortizes to near-free.
</div>

Note:
  Compression was the easy half; making it fast is the hard half. Entropy-coded weights aren't matrices, so every weight has to be decoded back to BFloat16 right before it's used — online, then discarded. And classic Huffman decoding is a bit-by-bit walk down a tree, completely serial, which is exactly what a GPU hates; one thread per weight leaves it idle. So we make the decode parallel with three hardware-aware moves: small lookup tables that live in fast SRAM instead of a tree walk, a two-pass kernel that first counts then writes so threads know where their output goes, and decoding a whole transformer block at once so the fixed cost amortizes. The payoff: decompression runs about thirty-five times faster than just streaming the weights in from CPU, and around twenty times faster than NVIDIA's own nvCOMP — fast enough to be nearly free at inference.

---

<div class="conf-kicker">Results · losslessness</div>
<h2>Same bits out — and 405B fits on one node</h2>

<div class="conf-split">
  <table class="conf-table conf-panel" aria-label="Accuracy and perplexity, BF16 versus DF11, Llama 3.1 8B Instruct">
    <thead>
      <tr><th>Llama 3.1 8B</th><th>BF16</th><th>DF11</th></tr>
    </thead>
    <tbody>
      <tr><td>MMLU (acc)</td><td>68.010</td><td class="conf-win">68.010</td></tr>
      <tr><td>TruthfulQA (acc)</td><td>36.965</td><td class="conf-win">36.965</td></tr>
      <tr><td>WikiText (ppl)</td><td>8.649</td><td class="conf-win">8.649</td></tr>
      <tr><td>C4 (ppl)</td><td>21.677</td><td class="conf-win">21.677</td></tr>
    </tbody>
  </table>
  <div>
    <div class="conf-banner">
      <div class="conf-banner-number">405B</div>
      <p>Llama 3.1 405B runs losslessly on a single 8×80 GB node — half the hardware, none of the loss.</p>
    </div>
    <div class="conf-callout teal" style="margin-top: 18px;">
      Not "within noise" — identical. Every decompressed weight matches the original bit-for-bit.
    </div>
  </div>
</div>

Note:
  Losslessness here isn't "no measurable difference on a benchmark." It's identical. Accuracy and perplexity match to the digit — MMLU, TruthfulQA, WikiText, C4 — because every decompressed weight is bit-for-bit the original; even diffusion images come out pixel-identical at the same seed. The practical headline: Llama-3.1 405B is 810 gigabytes in BFloat16 and needs two eight-GPU nodes; in DFloat11 it's about 551 gigabytes and fits on one. Half the hardware, none of the loss.

---

<div class="conf-kicker">Results · memory</div>
<h2>The freed memory becomes 5.7–14.9× longer generation</h2>

<figure class="conf-figure conf-fig-short">
  <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/memory_usage.webp" alt="GPU memory consumption versus decoded tokens for BF16 and DF11 models" />
  <figcaption>Same GPU budget, batch size 1. DF11's smaller footprint leaves room for the KV cache; BF16 hits out-of-memory far sooner.</figcaption>
</figure>

<div class="conf-metric" style="margin-top: 32px;">
  <div class="conf-card"><p><strong>5.70–14.86×</strong><span>more tokens before out-of-memory</span></p></div>
  <div class="conf-card"><p><strong>KV cache</strong><span>the saved memory directly buys context length</span></p></div>
  <div class="conf-card"><p><strong>−28% / +5%</strong><span>diffusion memory saved at a few percent more latency</span></p></div>
</div>

Note:
  And the memory you save isn't just a lower GPU count — it directly buys context length. During decoding it's the KV cache that runs you out of memory, and DFloat11's smaller weights leave between roughly six and fifteen times more room for that cache before you hit the limit, at batch size one on the same GPU. The same trick carries over to image models: about twenty-eight percent less memory on Stable Diffusion and FLUX, for only a few percent more latency. So lossless compression on the weights is what funds the longer generations that reasoning needs.

---

<div class="conf-paper-divider">
  <div class="conf-paper-venue">
    <span class="conf-venue-chip"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/acl.svg" alt="ACL 2026" /></span>
    <span class="conf-paper-venue-name">ACL 2026</span>
  </div>
  <h2>Then the KV cache: more bits for keys, fewer for values</h2>
  <figure class="conf-paper-shot">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/header_kvq.webp" alt="Paper headline: Quantize What Counts — More for Keys, Less for Values. Mohsen Hariri, Alan Luo, Weicong Chen, Tianyi Zhang, Qifan Wang, Xiaotian Han, Vipin Chaudhary." />
  </figure>
  <p class="conf-paper-foot">A short theorem says why: <strong>key</strong> weights dominate <strong>value</strong> weights in norm.</p>
</div>

Note:
  That sets up the second paper, at ACL. The weights are static, but the KV cache grows with every token the model thinks, so for long reasoning chains it becomes the dominant memory cost. The standard move is to quantize it — and the standard default spends equal precision on keys and values. This paper says that even split is the wrong default: keys deserve more bits than values, and there's a clean theorem for why.

---

<h2>KV cache dominates long-context inference memory</h2>

<div class="conf-memory-layout">
  <div>
    <div class="conf-equation">
      \[
      M_{\mathrm{KV}} =
      L\,n_{\mathrm{layers}}\,n_{\mathrm{KV}}\,d_{\mathrm{head}}\,
      \frac{b_K + b_V}{8}
      \]
    </div>
    <div class="conf-big-question conf-panel">
      If the budget is fixed, should a bit go to \(K\) or to \(V\)?
    </div>
  </div>
  <div class="conf-memory-context">
    <div class="conf-card">
      <h3>Common default</h3>
      <p>Use the same precision for key and value caches: \(K_4V_4\), \(K_2V_2\), and so on.</p>
    </div>
    <div class="conf-card">
      <h3>Existing mixed precision</h3>
      <p>Often tunes bit splits through grid search or runtime cache statistics.</p>
    </div>
  </div>
  <div class="conf-card conf-need-box">
    <h3>Need</h3>
    <p>A principled rule that generalizes across models, prompts, and quantization backends.</p>
  </div>
</div>

Note:
  Start with where the memory goes. This formula is just bookkeeping — the cache grows with sequence length, layers, heads, head dimension, and the per-token bit width of keys plus values — but the point is that in long-context decoding it's the KV cache, not the weights, that dominates. Quantization shrinks the total bit budget, but it still leaves an allocation question the formula makes explicit: with a fixed budget, does the next bit go to K or to V? Equal precision is the convenient answer and what most systems ship. The alternatives — grid search over splits, or tuning from runtime cache statistics — are expensive and don't transfer across models. What we want instead is a principled rule that comes from the model itself and generalizes across models, prompts, and quantization backends.

---

<h2>The extra bits should go to keys</h2>

<div class="conf-split">
  <figure class="conf-figure conf-fig-short">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/fig1_key_value_statistics.webp" alt="Spectral norm plot and GSM8K K2V4 versus K4V2 accuracy table" />
    <figcaption>Key cache spectral norms are larger, and \(K_4V_2\) beats \(K_2V_4\) at the same average bit budget.</figcaption>
  </figure>
  <div>
    <table class="conf-mini-table conf-accuracy-table conf-panel" aria-label="GSM8K accuracy for K2V4 and K4V2">
      <thead>
        <tr>
          <th>Model size</th>
          <th>\(K_2V_4\)</th>
          <th>\(K_4V_2\)</th>
        </tr>
      </thead>
      <tbody>
        <tr><td>1B</td><td>0.06</td><td class="conf-win">0.34</td></tr>
        <tr><td>8B</td><td>0.55</td><td class="conf-win">0.75</td></tr>
        <tr><td>14B</td><td>0.78</td><td class="conf-win">0.91</td></tr>
        <tr><td>70B</td><td>0.76</td><td class="conf-win">0.87</td></tr>
      </tbody>
    </table>
    <div class="conf-metric">
      <div class="conf-card"><p><strong>Same budget</strong><span>K<sub>2</sub>V<sub>4</sub> and K<sub>4</sub>V<sub>2</sub> both average 3 bits per cache tensor.</span></p></div>
      <div class="conf-card"><p><strong>Different accuracy</strong><span>The key-favored split is consistently better.</span></p></div>
      <div class="conf-card"><p><strong>Core question</strong><span>What property of attention makes this happen?</span></p></div>
    </div>
  </div>
</div>

Note:
  This first figure is the entire paper in miniature. Take two mixed-precision schemes at the exact same average budget — K2V4 and K4V2 both average three bits per cache tensor — and just swap which side gets the extra bits. The accuracy doesn't move a little, it moves a lot: on GSM8K the one-billion model goes from six percent to thirty-four, the eight-billion from fifty-five to seventy-five, and the same gap holds at fourteen and seventy billion. Same budget, very different accuracy, and the key-favored split wins every time. On the left you can already see why it might — the key cache has systematically larger spectral norms than the value cache. So the asymmetry looks structural, not like an implementation artifact, which raises the real question: what property of attention forces it?

---

<div class="conf-kicker">Theorem 1</div>
<h2>Key weights dominate value weights in norm</h2>

<div class="conf-grid two">
  <div class="conf-card">
    <h3>Key-Value Norm Disparity</h3>
    <div class="conf-equation">
      \[
      \mathbb{E}\left[\lVert W^K\rVert_F\right] >
      \mathbb{E}\left[\lVert W^V\rVert_F\right]
      \]
    </div>
  </div>
  <div class="conf-card">
    <h3>Why the key path grows</h3>
    <ul class="conf-list">
      <li>\(W^K\) shapes the attention map through \(QK^\top\).</li>
      <li>Query growth sharpens attention and amplifies gradients into \(W^K\).</li>
      <li>\(W^V\) mainly affects post-attention retrieved content.</li>
    </ul>
  </div>
</div>

<div class="conf-callout teal" style="margin-top: 28px;">
  The proof starts from equal expected norms at Xavier initialization and tracks SGD updates. The architectural asymmetry gives keys systematically larger norm growth.
</div>

Note:
  The answer is a statement about the projection weights themselves, independent of any prompt. At Xavier initialization the key and value matrices have the same expected norm. But during training the gradients aren't symmetric: the key matrix sits inside the attention score — the Q-K-transpose term — so as queries grow and attention sharpens, those gradients get amplified and the key weights accumulate more norm, while the value matrix only feeds the content retrieved after attention and grows more slowly. That's Theorem 1: keys end up larger in norm. Then comes the bridge to bits, which is the second result — quantization error scales with the norm of what you're quantizing, so the larger-norm tensor is the one that hurts most when you starve it. Keys are larger, therefore keys are the binding constraint, therefore keys get the bits. The figure on the previous slide was just this theorem showing up in the data.

---

<div class="conf-kicker">Downstream accuracy: Optimum Quanto</div>
<h2>\(K_4V_2\) nearly matches \(K_4V_4\) while \(K_2V_4\) falls behind</h2>

<div class="conf-split reverse conf-quanto-layout">
  <div>
    <table class="conf-table conf-panel" aria-label="GSM8K 1-shot Optimum Quanto results">
      <thead>
        <tr>
          <th>GSM8K 1-shot</th>
          <th>\(K_2V_4\)</th>
          <th>\(K_4V_2\)</th>
          <th>\(K_4V_4\)</th>
        </tr>
      </thead>
      <tbody>
        <tr><td>Llama 3.2-1B</td><td>0.035</td><td class="conf-win">0.338</td><td>0.357</td></tr>
        <tr><td>Llama 3.1-8B</td><td>0.547</td><td class="conf-win">0.752</td><td>0.754</td></tr>
        <tr><td>Phi-4-14B</td><td>0.783</td><td class="conf-win">0.913</td><td>0.923</td></tr>
        <tr><td>DeepSeek R1Q-14B</td><td>0.775</td><td class="conf-win">0.865</td><td>0.867</td></tr>
      </tbody>
    </table>
  </div>
  <div>
    <div class="conf-metric conf-quanto-metrics">
      <div class="conf-card"><p><strong>~94%</strong><span>average full-precision baseline recovery for K4V2 with Optimum Quanto.</span></p></div>
      <div class="conf-card"><p><strong>+30 pp</strong><span>1B GSM8K gain for K4V2 over K2V4.</span></p></div>
      <div class="conf-card"><p><strong>near K4V4</strong><span>Higher key precision preserves downstream performance while values use fewer bits.</span></p></div>
    </div>
  </div>
</div>

Note:
  Now the downstream test, on one-shot GSM8K through the Optimum Quanto backend. Read across each row: K4V2 lands essentially on top of the full four-bit K4V4 baseline — eight-billion is 0.752 versus 0.754, fourteen-billion is in the same neighborhood — while K2V4, the same budget spent the other way, collapses, most dramatically on the small model where it falls from thirty-four percent to three. Averaged over these models K4V2 recovers about ninety-four percent of the full-precision baseline. The conclusion isn't that values are irrelevant; it's that once the budget is tight, keys are the binding constraint, so that's where the precision should sit.

---

<div class="conf-kicker">Orthogonal integration</div>
<h2>Rotation helps most when it targets keys</h2>

<div class="conf-split">
  <figure class="conf-figure conf-fig-tall">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/rotation_impact_heatmap.webp" alt="Heatmaps showing rotation effects with different mixed precision KV cache configurations" />
    <figcaption>Downstream accuracy under four rotation scopes and four bit allocations, fixed group size 64.</figcaption>
  </figure>
  <div>
    <div class="conf-card">
      <h3>Rotation redistributes outliers</h3>
      <p>QuaRot-style Hadamard rotations make activation distributions easier to quantize.</p>
    </div>
    <div class="conf-card" style="margin-top: 16px;">
      <h3>Key-only is the main gain</h3>
      <p>Applying rotation to keys consistently improves low-bit settings more than rotating values.</p>
    </div>
    <div class="conf-card" style="margin-top: 16px;">
      <h3>Synergy</h3>
      <p>\(K_4V_2\) plus key-only rotation closely tracks the \(K_4V_4\) baseline and can outperform it by 4.4-18% across tasks.</p>
    </div>
  </div>
</div>

Note:
  One more result, because in practice people don't just quantize — they also rotate. QuaRot-style Hadamard rotations smear out the outliers that make a tensor hard to quantize, and that's orthogonal to our question: rotation reshapes the distribution, bit allocation decides where the precision goes. The finding is that rotation pays off most when you aim it at the keys — key-only rotation captures essentially all of the benefit. Stack it on K4V2 and you don't just approach the full K4V4 baseline, you can beat it, by four to eighteen percent across tasks, while keeping values at two bits; across the suite K4V2 holds about ninety-eight percent of full-cache accuracy. Same story from a third angle — keys are the high-sensitivity channel.

---

<div class="conf-kicker">The measurement bill</div>
<h2>Sample it many times, and the leaderboard becomes an estimate</h2>
<p class="conf-lead">The memory we just bought back gets spent on a bigger sampling budget N. That same N turns evaluation into statistics.</p>

<div class="conf-flow">
  <div class="conf-card"><div class="conf-num">01</div><h3>One run is one draw</h3><p>A single Pass@1 or Pass@k is a noisy sample, not the model's true success rate.</p></div>
  <div class="conf-card"><div class="conf-num">02</div><h3>Rankings wobble</h3><p>On small, costly benchmarks &mdash; tens of AIME problems &mdash; the order swings with sampling noise.</p></div>
  <div class="conf-card"><div class="conf-num">03</div><h3>The metric is a choice</h3><p>Pointwise, pairwise, voting, Bayesian: many rankings, and they disagree at low budget.</p></div>
  <div class="conf-card"><div class="conf-num">04</div><h3>So quantify it</h3><p>Estimate the success probability, report the interval, and don't call a winner on overlap.</p></div>
</div>

<div class="conf-callout teal" style="margin-top: 24px;">You can't make sampling noise lossless &mdash; so measure it.</div>

Note:
  Now the pivot to the second half of the talk. All that memory we just bought back gets spent on a bigger sampling budget — more trials per question — because that's how test-time scaling buys accuracy. But the very same N turns evaluation into statistics: a single pass@1 or pass@k is one noisy draw, not the model's true success rate, and on small, expensive benchmarks like the few dozen AIME problems the leaderboard order genuinely wobbles with sampling noise. You can't make that noise lossless — so the honest move is to measure it. The next two papers are about reading the leaderboard with its error bars attached.

---

<div class="conf-paper-divider">
  <div class="conf-paper-venue">
    <span class="conf-venue-chip"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/iclr.svg" alt="ICLR 2026" /></span>
    <span class="conf-paper-venue-name">ICLR 2026</span>
  </div>
  <h2>Now the measurement: report a posterior, not a point</h2>
  <figure class="conf-paper-shot">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/header_bayes.webp" alt="Paper headline: Don't Pass@k — A Bayesian Framework for Large Language Model Evaluation. Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, Vipin Chaudhary." />
  </figure>
  <p class="conf-paper-foot">A Beta&ndash;Binomial posterior with credible intervals &mdash; and a rule for when a gap is real.</p>
</div>

Note:
  The third paper, at ICLR, makes that concrete for a single model's score. Don't Pass@k reframes scoring as posterior inference: instead of reporting a point number, you report a Beta-Binomial posterior over the model's true success probability, with a credible interval — and, crucially, a rule for deciding when a gap between two models is real rather than just sampling noise.

---

<div class="conf-kicker">Motivation</div>
<h2>Pass@<i>k</i> ranks the wrong quantity</h2>
<p class="conf-lead">
  Pass@\(k\) estimates the chance of at least one success in \(k\) tries. That is not a model's underlying success probability, and as a ranking signal it is fragile.
</p>

<div class="conf-grid three">
  <div class="conf-card">
    <div class="conf-num">01</div>
    <h3>Unstable at low budget</h3>
    <p>On small or costly benchmarks (tens of AIME problems), rankings swing with decoding and sampling noise.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">02</div>
    <h3>No closed-form uncertainty</h3>
    <p>Variance needs bootstrapping; there is no transparent rule for whether a gap is meaningful or noise.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">03</div>
    <h3>Binary only</h3>
    <p>It does not extend to partial credit, refusals, or rubric-defined outcomes without ad hoc aggregation.</p>
  </div>
</div>

<div class="conf-callout" style="margin-top: 28px;">
  Treat evaluation as statistical inference: estimate the success probability and report how sure you are.
</div>

Note:
  Start from the pain. Pass@k answers "can the model get this in k attempts" — that's a coverage question, the probability of at least one success, not an estimate of how good the model actually is. Three problems follow. At small budgets the estimator has high variance, so leaderboards reshuffle for reasons that have nothing to do with the models. It has no closed-form uncertainty — you're bootstrapping to get an error bar, with no clean rule for whether a gap is meaningful. And it's binary only: no natural room for partial credit, refusals, or rubric scores. The fix is to treat evaluation as what it is — statistical inference — estimate the success probability and report how sure you are.

---

<div class="conf-kicker">Method</div>
<h2>Bayes@<i>N</i> turns scoring into posterior inference</h2>

<div class="conf-rule-grid">
  <div class="conf-card">
    <div class="conf-num">1</div>
    <h3>Sample</h3>
    <p>Run \(N\) attempts per question.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">2</div>
    <h3>Score</h3>
    <p>Label each attempt: binary or a rubric category \(0,\dots,C\).</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">3</div>
    <h3>Tally</h3>
    <p>Count category hits \(n_{\alpha k}\) per question \(\alpha\).</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">4</div>
    <h3>Posterior</h3>
    <p>Form a Dirichlet posterior over outcome probabilities \(\boldsymbol{\pi}_\alpha\).</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">5</div>
    <h3>Report</h3>
    <p>Output mean \(\mu\), uncertainty \(\sigma\), and credible intervals.</p>
  </div>
</div>

<div class="conf-callout teal" style="margin-top: 26px;">
  The unit of evidence is the results matrix \( R \in \{0,\dots,C\}^{M\times N} \): \(M\) questions, \(N\) repeated trials, one rubric label per cell.
</div>

Note:
  The method is five steps, and the reassuring part is that none of them needs a new decoder or a new benchmark — it's a scoring layer on top of the trials you already ran. Sample N attempts per question, score each one — binary, or into rubric categories zero through C — tally the counts per question, and turn those counts into a Dirichlet posterior over the outcome probabilities; for the binary case that's just a Beta posterior over the success rate. Then you report the posterior mean, its standard deviation, and a credible interval, instead of a bare number. The unit of evidence is the whole results matrix R — M questions by N trials, one label per cell. And a nice property: with a uniform prior the posterior mean is order-equivalent to plain average accuracy, so you lose nothing relative to pass@1 and gain the uncertainty for free.

---

<div class="conf-kicker">Result · convergence</div>
<h2>Bayes@<i>N</i> reaches the stable ranking with fewer trials</h2>

<div class="conf-split">
  <figure class="conf-figure conf-fig-mid">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/convergence.webp" alt="Convergence@n probability mass functions comparing Bayes@N against Pass@2/4/8 across four math benchmarks, with example ranking traces" />
    <figcaption>Convergence@\(n\): the smallest trial budget after which the ranking matches the \(N=80\) gold standard and stays put.</figcaption>
  </figure>
  <div class="conf-stack">
    <div class="conf-card">
      <h3>Pass@<i>k</i> can fail to converge</h3>
      <p>On AIME'24 and AIME'25, the Pass family often never settles within the budget; Bayes@\(N\) does.</p>
    </div>
    <div class="conf-card">
      <h3>And converges slower when it does</h3>
      <p>On HMMT'25 and BrUMO'25, mean convergence@\(n\) is \(\approx 44.2\) and \(\approx 27.1\) for Bayes@\(N\) versus \(\approx 69.5\) and \(\approx 48.5\) for Pass@\(k\).</p>
    </div>
    <div class="conf-callout">
      Same trials, same data. Only the scoring rule changes.
    </div>
  </div>
</div>

Note:
  The first result answers the practical question: how many samples before the leaderboard stops moving? We define convergence-at-n as the smallest budget after which the ranking matches the eighty-trial gold standard and then stays put. On the harder benchmarks the Pass family either never settles inside the budget — that's AIME'24 and '25 — or it settles late: on HMMT and BrUMO it needs about seventy and about forty-nine trials, where Bayes@N gets there in about forty-four and twenty-seven. Same trials, same data, same questions — the only thing that changed is the scoring rule, and the Bayesian estimator stabilizes much earlier.

---

<div class="conf-kicker">Result · uncertainty</div>
<h2>Close models need far more than 80 trials to separate</h2>

<div class="conf-split">
  <figure class="conf-figure conf-fig-mid">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/decision-rule.webp" alt="Probability and z-score of correctly ranking two close models as a function of the number of trials" />
    <figcaption>Probability and \(z\)-score of correctly ordering two near-tied models (\(\bar\pi = 0.608\) vs \(0.6213\)) versus \(N\).</figcaption>
  </figure>
  <div class="conf-stack">
    <div class="conf-metric" style="grid-template-columns: repeat(3, minmax(0,1fr));">
      <div class="conf-card"><p><strong>83.7%</strong><span>correct order at N = 80</span></p></div>
      <div class="conf-card"><p><strong>94.7%</strong><span>N = 199 · z = 1.645</span></p></div>
      <div class="conf-card"><p><strong>96.9%</strong><span>N = 285 · z = 1.96</span></p></div>
    </div>
    <div class="conf-callout amber">
      A typical \(N=80\) evaluation is not enough to confidently separate two models that differ by \(\approx 1.3\) accuracy points.
    </div>
    <div class="conf-card">
      <h3>The interval makes this visible</h3>
      <p>Without a credible interval, the point estimate silently picks a winner the data does not support.</p>
    </div>
  </div>
</div>

Note:
  The second result is about when a gap is actually real. Take two models about one-point-three accuracy points apart — true success rates of 0.608 and 0.621. At a typical eighty-trial evaluation you'll rank them correctly only about eighty-four percent of the time. To get to ninety-five percent you need about two hundred trials; to get to ninety-seven, about two hundred eighty-five. So the number to walk out with: separating two models a point apart, with confidence, costs on the order of two hundred samples per question — and the credible interval is exactly what makes that visible. A bare pass@k or average silently picks a winner the data doesn't support.

---

<div class="conf-kicker">Rubrics</div>
<h2>The same machinery scores beyond right-or-wrong</h2>

<div class="conf-split reverse">
  <div class="conf-stack">
    <div class="conf-card">
      <h3>Outcomes are categories</h3>
      <p>Correct, partial credit, format errors, refusals, verifier signals, efficiency penalties — all live in one Dirichlet posterior.</p>
    </div>
    <div class="conf-card">
      <h3>Weights encode the goal</h3>
      <p>Different \(w\) vectors express different evaluation priorities while the uncertainty stays explicit and comparable.</p>
    </div>
    <div class="conf-callout teal">
      Across categorical schemes, Qwen3-Thinking stays first; the rubric choice mainly reshuffles the middle of the pack.
    </div>
  </div>
  <figure class="conf-figure conf-fig-tall">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/categorical.webp" alt="Ranking traces of 11 models across several categorical scoring schemes" />
    <figcaption>Model rank across categorical schemes; top models are stable while mid-pack order depends on the rubric.</figcaption>
  </figure>
</div>

Note:
  And because the posterior is over categories, not just right-or-wrong, the same machinery scores everything a grader actually sees — partial credit, format errors, refusals, verifier signals, even efficiency penalties — all in one Dirichlet posterior, with a weight vector that encodes what you care about. Binary scoring throws most of that away. We ran about a dozen different rubric schemes over eleven models, and the reassuring finding is that the top of the leaderboard is stable — Qwen3-Thinking stays first throughout — while the rubric mostly reshuffles the middle of the pack. So the choice of rubric changes the nuance, not the headline.

---

<div class="conf-paper-divider">
  <div class="conf-paper-venue">
    <span class="conf-venue-chip"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/acl.svg" alt="ACL 2026" /></span>
    <span class="conf-paper-venue-name">ACL 2026</span>
  </div>
  <h2>And the leaderboard: rank the distribution, not one sample</h2>
  <figure class="conf-paper-shot">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/header_rank.webp" alt="Paper headline: Ranking Reasoning LLMs under Test-Time Scaling. Mohsen Hariri, Michael Hinczewski, Jing Ma, Vipin Chaudhary." />
  </figure>
  <p class="conf-paper-foot">The benchmark is a response tensor &mdash; which ranking you trust depends on the budget.</p>
</div>

Note:
  The fourth paper, also at ACL, scales that idea up from one model's score to a whole leaderboard. Under test-time scaling every model-question pair is no longer one number but a distribution of attempts, so ranking becomes a statistical question with two parts: which ordering actually survives the noise, and — when you can only afford one or two trials per prompt — which ranking method should you even trust?

---

<div class="conf-kicker">Motivation</div>
<h2>A one-sample leaderboard hides the object we are ranking</h2>
<p class="conf-lead">
  Test-time scaling turns each model-question outcome into a distribution of sampled attempts. The ranking question becomes statistical: which ordering survives the sampling noise?
</p>

<div class="conf-two-thirds">
  <div class="conf-stack">
    <div class="conf-callout">
      The unit of evidence is no longer one score per model. It is a repeated-trial response tensor that can be sliced by model, question, and trial budget.
    </div>
    <div class="conf-grid three">
      <div class="conf-card">
        <div class="conf-num">01</div>
        <h3>More samples</h3>
        <p>Stochastic decodes expose variance, near ties, and ranking reversals.</p>
      </div>
      <div class="conf-card">
        <div class="conf-num">02</div>
        <h3>Many ranking rules</h3>
        <p>Pointwise, pairwise, voting, graph, IRT, and Bayesian methods can disagree.</p>
      </div>
      <div class="conf-card">
        <div class="conf-num">03</div>
        <h3>Target matters</h3>
        <p>Accuracy agreement and self-consistency reward different behavior.</p>
      </div>
    </div>
  </div>
  <div class="conf-panel conf-title-card">
    <h3>Core question</h3>
    <div class="conf-big-question">
      If we can afford only one or two trials per prompt, which ranking should we trust?
    </div>
    <div class="conf-chip-row">
      <span class="conf-chip blue">low-budget stability</span>
      <span class="conf-chip teal">full-budget agreement</span>
      <span class="conf-chip amber">prior bias</span>
    </div>
  </div>
</div>

Note:
  Start from the practical problem. Once you sample each model repeatedly, the leaderboard isn't a fact, it's an estimator — and a one-sample leaderboard is just the N-equals-one slice of a much richer object, hiding all the variance, the near-ties, and the ranking reversals underneath. Three things make this hard: more samples expose that variance; there are many legitimate ranking rules — pointwise, pairwise, voting, graph-based, Bayesian — and at low budget they genuinely disagree; and the right answer depends on your target, because rewarding raw accuracy and rewarding self-consistency are different goals. So the question on the right is the real one: if you can afford only one or two trials per prompt, which ranking do you trust?

---

<div class="conf-kicker">Formalization</div>
<h2>The benchmark is a tensor, not a table</h2>

<div class="conf-two-thirds">
  <div>
    <div class="conf-equation">
      <span><em>R</em><sub>lmn</sub> = 1 &nbsp; if model <em>l</em> solves question <em>m</em> on trial <em>n</em></span>
    </div>
    <div class="conf-stat-row">
      <div class="conf-stat">
        <strong>20</strong>
        <span>reasoning LLMs</span>
      </div>
      <div class="conf-stat">
        <strong>30</strong>
        <span>questions per benchmark</span>
      </div>
      <div class="conf-stat">
        <strong>80</strong>
        <span>top-p trials per pair</span>
      </div>
      <div class="conf-stat">
        <strong>1</strong>
        <span>greedy prior draw R<sub>0</sub></span>
      </div>
    </div>
    <div class="conf-callout teal" style="margin-top: 22px;">
      The single-run benchmark is only the \(N=1\) slice. Repeated trials let us ask how rankings move as the trial budget grows.
    </div>
  </div>
  <div class="conf-panel conf-title-card">
    <h3>One tensor supports multiple views</h3>
    <ul class="conf-list conf-fit-list">
      <li>Per-question solve rates \( \widehat{p}_{lm} \)</li>
      <li>Pairwise win/tie counts \( W_{ij}, T_{ij} \)</li>
      <li>Winner sets \( U_{mn} \) for listwise methods</li>
      <li>Posterior distributions for uncertainty-aware ranks</li>
    </ul>
  </div>
</div>

Note:
  The formalization is simple but it's the key move. Instead of a model-by-score table, the benchmark is a three-way tensor: model l, question m, trial n, with a one if that model solved that question on that trial. Concretely, twenty reasoning models, thirty questions, eighty stochastic trials each, plus one greedy decode we keep as a prior. The ordinary single-run benchmark is just the N-equals-one slice of this. And because every model attempts every question many times, this is a dense tensor — unlike a sparse preference graph like Chatbot Arena where only some pairs ever meet — which is exactly what lets us slice it into per-question rates, pairwise win counts, or winner sets, and ask how the ranking moves as the budget grows.

---

<div class="conf-kicker">Method families</div>
<h2>Ranking rules are different projections of the same tensor</h2>

<div class="conf-method-map">
  <div class="conf-card">
    <div class="conf-label">Pointwise</div>
    <div class="conf-equation">\( \widehat{p}_{lm} = \frac{1}{N}\sum_n R_{lmn} \)</div>
    <h3>Model-question performance</h3>
    <p>Average accuracy, Bayes@N, Pass@k, difficulty weighting, and IRT-style models.</p>
  </div>
  <div class="conf-card">
    <div class="conf-label">Pairwise</div>
    <div class="conf-equation">W<sub>ij</sub>, W<sub>ji</sub>, T<sub>ij</sub></div>
    <h3>Model-model comparisons</h3>
    <p>Bradley-Terry, Elo, TrueSkill, Borda, Copeland, PageRank, Rank Centrality, and HodgeRank.</p>
  </div>
  <div class="conf-card">
    <div class="conf-label">Setwise</div>
    <div class="conf-equation">U<sub>mn</sub> = { l : R<sub>lmn</sub> = 1 }</div>
    <h3>Winner and loser sets</h3>
    <p>Plackett-Luce and Davidson-Luce operate on two-level partial orders per question-trial.</p>
  </div>
</div>

<div class="conf-callout amber" style="margin-top: 24px;">
  These methods can be equally reasonable while optimizing different notions of rank. The evaluation target has to be named before declaring a winner.
</div>

Note:
  And here's the unifying idea: this isn't Bayes versus everyone else. All seventy-plus ranking methods we compare are really just three different projections of that same tensor. Collapse over trials and you get pointwise data — per-question solve rates — which feeds average accuracy, Bayes@N, pass@k, IRT. Collapse into head-to-head win and tie counts and you get pairwise data — Bradley-Terry, Elo, TrueSkill, the voting and graph methods. Or keep each trial's winner set and you get setwise data for Plackett-Luce-style models. Each projection keeps some information and discards some, and all of them can be perfectly reasonable while optimizing different notions of rank — which is exactly why you have to name your evaluation target before you can call any one method the winner.

---

<div class="conf-kicker">Result 1: high-budget consensus</div>
<h2>With 80 trials, most reasonable ranking families agree</h2>

<div class="conf-figure-grid">
  <figure class="conf-figure">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/rank_bump_chart.webp" alt="Bump charts showing agreement between full-trial ranking methods and Bayes U at N=80" />
    <figcaption>Full-trial rankings against Bayes<sub>U</sub>@80 on BrUMO'25 and HMMT'25.</figcaption>
  </figure>
  <div class="conf-stack">
    <div class="conf-callout">
      Mean agreement with Bayes<sub>U</sub>@80 is &tau;<sub>b</sub>=0.93 to 0.95 across benchmarks.
    </div>
    <table class="conf-table compact">
      <thead>
        <tr><th>Benchmark</th><th>Mean</th><th>Exact</th><th>≥0.95</th></tr>
      </thead>
      <tbody>
        <tr><td>AIME'24</td><td>0.941</td><td>20</td><td>40</td></tr>
        <tr><td>AIME'25</td><td>0.934</td><td>19</td><td>29</td></tr>
        <tr><td>HMMT'25</td><td>0.950</td><td>34</td><td>44</td></tr>
        <tr><td>BrUMO'25</td><td>0.954</td><td>26</td><td>49</td></tr>
        <tr><td>Combined</td><td>0.962</td><td>22</td><td>53</td></tr>
      </tbody>
    </table>
    <p class="conf-small">Statistics are over the other 71 methods computed from all 80 trials.</p>
  </div>
</div>

Note:
  Two results. The first is the reassuring one: when you actually have the full eighty trials, the choice of method barely matters. Measured against a Bayesian uniform-prior reference at eighty trials, the average Kendall agreement across all the other seventy-one methods is about 0.93 to 0.95 on every benchmark, and on the combined set more than fifty methods agree to within 0.95. So at high budget, just use the simple, interpretable thing — average accuracy or Bayes@80 — and don't agonize over it. The interesting disagreements only show up when you cut the budget.

---

<div class="conf-kicker">Result 2: low-budget ranking</div>
<h2>At \(N=1\), the best method depends on the target</h2>

<table class="conf-table compact wide-text">
  <thead>
    <tr>
      <th>Benchmark</th>
      <th>Best vs. Bayes<sub>U</sub>@80</th>
      <th>&tau;<sub>b</sub></th>
      <th>Best self-consistency</th>
      <th>&tau;<sub>b</sub></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>AIME'24</td>
      <td class="conf-method-cell">Bayes<sub>R0</sub>@1</td>
      <td>0.779 ± 0.034</td>
      <td class="conf-method-cell">Rasch MML LCB</td>
      <td>0.804 ± 0.051</td>
    </tr>
    <tr>
      <td>AIME'25</td>
      <td class="conf-method-cell">Bayes<sub>R0</sub>@1</td>
      <td>0.798 ± 0.045</td>
      <td class="conf-method-cell">Rasch MML LCB</td>
      <td>0.834 ± 0.054</td>
    </tr>
    <tr>
      <td>HMMT'25</td>
      <td class="conf-method-cell">Bayes<sub>U</sub>@1 and 20 tied methods</td>
      <td>0.790 ± 0.053</td>
      <td class="conf-method-cell">Rasch MML LCB</td>
      <td>0.810 ± 0.056</td>
    </tr>
    <tr>
      <td>BrUMO'25</td>
      <td class="conf-method-cell">Bayes<sub>R0</sub>@1</td>
      <td>0.858 ± 0.028</td>
      <td class="conf-method-cell">Bayes<sub>R0</sub>@1</td>
      <td>0.858 ± 0.028</td>
    </tr>
    <tr>
      <td>Combined</td>
      <td class="conf-method-cell">Bayes<sub>U</sub>@1 and 20 tied methods</td>
      <td>0.865 ± 0.049</td>
      <td class="conf-method-cell">Nanson avg ties</td>
      <td>0.892 ± 0.050</td>
    </tr>
  </tbody>
</table>

<div class="conf-callout teal" style="margin-top: 22px;">
  Bayes<sub>R0</sub>@1 is strongest when greedy decoding is aligned with stochastic sampling. Self-consistency winners can be different.
</div>

Note:
  And here's where it gets interesting: at one trial per prompt, the best method depends on what you're optimizing for. If your target is matching the full-budget ranking, the Bayesian method with the greedy decode as a prior — Bayes-R-zero at one — tends to win, because that greedy draw cuts the variance by anywhere from sixteen to over fifty percent. But it only helps when greedy decoding actually agrees with stochastic sampling; on a benchmark where they diverge, the prior pulls you the wrong way and a plain uniform-prior method — or a whole tie of voting methods like Borda, Copeland, PageRank — does just as well. And if your target is self-consistency instead, the repeatability of the method with itself, an IRT method, Rasch with a credible bound, is the steadier choice. So the practical message isn't "method X always wins"; it's name whether you care about accuracy agreement or repeatability first, then pick.

---

<div class="conf-kicker">Synthesis</div>
<h2>One object, two questions: fill R cheaply, read R rigorously</h2>

<div class="conf-split">
  <div>
    <div class="conf-equation"><span style="font-size: 40px;"><em>R</em> &isin; {0,1}<sup><em>L</em> &times; <em>M</em> &times; <em>N</em></sup></span></div>
    <p class="conf-lead" style="margin-top: 18px;">One response tensor: <strong>L</strong> models &times; <strong>M</strong> questions &times; <strong>N</strong> trials, each entry a 0/1 &mdash; did this model solve this question on this trial. Test-time scaling just grows <strong>N</strong>.</p>
    <div class="conf-callout teal" style="margin-top: 18px;">Two papers shrink the cost of <strong>filling</strong> R; two govern what you may <strong>read</strong> off it.</div>
  </div>
  <div class="conf-stack">
    <div class="conf-card">
      <h3>Fill R cheaply &mdash; the memory bill</h3>
      <ul class="conf-list conf-fit-list">
        <li><strong>DFloat11</strong> &mdash; lossless weights, so every entry of R is bit-for-bit the full model's answer.</li>
        <li><strong>More keys, less values</strong> &mdash; a tight KV-cache budget spent where information is densest (the keys).</li>
      </ul>
    </div>
    <div class="conf-card">
      <h3>Read R rigorously &mdash; the measurement bill</h3>
      <ul class="conf-list conf-fit-list">
        <li><strong>Don't Pass@k</strong> &mdash; a posterior with credible intervals: is a gap real, or sampling noise?</li>
        <li><strong>Ranking under test-time scaling</strong> &mdash; which ordering to trust at a given trial budget.</li>
      </ul>
    </div>
  </div>
</div>

Note:
  Keep one picture in your head: the response tensor R. Each entry is a single zero or one — did model l solve question m on trial n — and test-time scaling is just the regime where N gets large, so R becomes enormous and expensive to fill. The first two papers lower the cost of filling it: DFloat11 makes every entry bit-for-bit what the full-precision model would have produced, and more-keys-less-values spends a tight cache budget where the information is densest. The last two govern what you're allowed to read off it: Don't Pass@k attaches a credible interval to each model's success rate, and the ranking paper tells you which ordering survives at a small budget. One object, two questions — fill it cheaply, read it honestly.

---

<h2>Both methods are released as installable packages</h2>

<div class="conf-grid two">
  <div class="conf-card">
    <div class="conf-num">DFloat11</div>
    <h3>vllm-df11: DFloat11 weights in vLLM</h3>
    <pre class="conf-code conf-scorio-code" data-no-copy><span class="kw">pip install</span> vllm-df11
<span class="kw">import</span> os
os.environ[<span class="str">"VLLM_PLUGINS"</span>] = <span class="str">"df11"</span>
<span class="kw">from</span> vllm.plugins <span class="kw">import</span> load_general_plugins
<span class="fn">load_general_plugins</span>()
<span class="kw">from</span> vllm <span class="kw">import</span> LLM, SamplingParams
llm = <span class="fn">LLM</span>(
    <span class="arg">model</span>=<span class="str">"/path/to/llama-3.1-8b-it-df11"</span>,
    <span class="arg">load_format</span>=<span class="str">"df11"</span>,
    <span class="arg">dtype</span>=<span class="str">"bfloat16"</span>,
)
outputs = llm.<span class="fn">generate</span>(prompts, sampling_params)</pre>
  </div>
  <div class="conf-card">
    <div class="conf-num">More Keys, Less Values</div>
    <h3>kvq: non-uniform KV-cache quantization</h3>
    <pre class="conf-code conf-scorio-code" data-no-copy><span class="kw">pip install</span> kvq
<span class="kw">from</span> kvq <span class="kw">import</span> KVQConfig, KVQ
config = <span class="fn">KVQConfig</span>(
    <span class="arg">budget</span>=<span class="num">4</span>,
    <span class="arg">model</span>=<span class="str">"meta-llama/Llama-3.1-8B-Instruct"</span>,
    <span class="arg">residual_length</span>=<span class="num">32</span>,
    <span class="arg">group_size</span>={<span class="str">"k"</span>: <span class="num">64</span>, <span class="str">"v"</span>: <span class="num">64</span>},
    <span class="arg">axis</span>={<span class="str">"k"</span>: <span class="num">0</span>, <span class="str">"v"</span>: <span class="num">0</span>},
)
kv_cache = <span class="fn">KVQ</span>(config)
outputs = model.<span class="fn">generate</span>(
    **inputs,
    <span class="arg">past_key_values</span>=kv_cache,
    <span class="arg">use_cache</span>=<span class="kw">True</span>,
)</pre>
  </div>
</div>

Note:
  And both of these methods ship. vllm-df11 registers DFloat11 as a vLLM plugin — set the plugin variable, point load_format at a DFloat11 checkpoint, and vLLM decompresses the weights losslessly during inference, with the rest of your serving stack untouched. kvq gives you a config and a drop-in KVQ cache object that replaces the standard past-key-values and applies the key-heavy bit allocation from the paper. In both cases the surrounding inference code doesn't change — you opt into the memory savings with a couple of lines.

---

<div class="conf-scorio-header">
  <div class="conf-scorio-logo-panel">
    <img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/scorio.svg" alt="Scorio" />
  </div>
  <div class="conf-scorio-heading">
    <h2>Scorio: a Bayesian evaluation and ranking toolkit</h2>
    <a class="conf-scorio-docs" href="https://mohsenhariri.github.io/scorio/">Documentation: mohsenhariri.github.io/scorio ↗</a>
  </div>
</div>

<div class="conf-scorio-layout">
  <div class="conf-scorio-panel">
    <pre class="conf-code conf-scorio-code" data-no-copy><span class="kw">import</span> numpy <span class="kw">as</span> np
<span class="kw">from</span> scorio <span class="kw">import</span> eval, rank
<span class="cm"># Eval (Don't Pass@k): score an M×N matrix</span>
R = np.<span class="fn">array</span>([[0, 1, 1, 0, 1],
              [1, 1, 0, 1, 1]])
mu, sd, lo, hi = eval.<span class="fn">bayes_ci</span>(R, <span class="arg">confidence</span>=<span class="num">0.95</span>)
p,  sd, lo, hi = eval.<span class="fn">pass_at_k_ci</span>(R, <span class="arg">k</span>=<span class="num">1</span>)
<span class="cm"># Rank L models from an L×M×N tensor</span>
S = np.random.<span class="fn">randint</span>(0, 2, (20, 30, 8))
bayes_ranks = rank.<span class="fn">bayes</span>(S)
bt_ranks    = rank.<span class="fn">bradley_terry</span>(S)
graph_ranks = rank.<span class="fn">pagerank</span>(S)</pre>
    <div class="conf-scorio-note">
      <strong>eval</strong> scores a results matrix and returns a mean with a credible interval; <strong>rank</strong> orders \(L\) models from a response tensor. Both take the trial data already collected for Pass@\(k\).
    </div>
    <div class="conf-scorio-use" aria-label="Modules">
      <span class="conf-scorio-app">Evaluation</span>
      <span class="conf-scorio-app">Ranking</span>
      <span class="conf-scorio-app">Credible intervals</span>
    </div>
  </div>
  <div class="conf-api-board">
    <h3>Public API map <span>scorio.eval / scorio.rank / scorio.utils</span></h3>
    <div class="conf-api-group">
      <div class="conf-api-head"><code>scorio.eval</code><span>metrics and uncertainty</span></div>
      <div class="conf-api-columns">
        <p class="conf-api-line"><strong>Bayes/avg</strong> <code>bayes</code><code>bayes_ci</code><code>avg</code><code>avg_ci</code></p>
        <p class="conf-api-line"><strong>Pass/AUC/majority/max</strong> <code>pass_at_k</code><code>pass_hat_k</code><code>pass_at_k_ci</code><code>pass_hat_k_ci</code><code>auc_at_k</code><code>auc_at_k_ci</code><code>maj_at_k</code><code>maj_at_k_ci</code><code>max_at_k</code><code>max_at_k_ci</code></p>
        <p class="conf-api-line"><strong>Generalized pass</strong> <code>g_pass_at_k</code><code>g_pass_at_k_tau</code><code>mg_pass_at_k</code><code>g_pass_at_k_ci</code><code>g_pass_at_k_tau_ci</code><code>mg_pass_at_k_ci</code></p>
        <p class="conf-api-line"><strong>GeoSpectrum</strong> <code>geom_at_k</code><code>geom_at_k_ci</code><code>geom_ds_at_k</code><code>geom_ds_at_k_ci</code><code>geo_spectrum_at_k</code><code>geo_spectrum_at_k_ci</code><code>geo_spectrum_star_at_k</code><code>geo_spectrum_star_at_k_ci</code><code>threshold_spectrum_at_k</code><code>threshold_spectrum_at_k_ci</code></p>
      </div>
    </div>
    <div class="conf-api-group">
      <div class="conf-api-head"><code>scorio.rank</code><span>model ranking families</span></div>
      <div class="conf-api-columns">
        <p class="conf-api-line"><strong>Priors</strong> <code>Prior</code><code>EmpiricalPrior</code><code>GaussianPrior</code><code>LaplacePrior</code><code>CauchyPrior</code><code>UniformPrior</code><code>CustomPrior</code></p>
        <p class="conf-api-line"><strong>Eval rankers</strong> <code>avg</code><code>bayes</code><code>pass_at_k</code><code>pass_hat_k</code><code>g_pass_at_k_tau</code><code>mg_pass_at_k</code></p>
        <p class="conf-api-line"><strong>Point/graph/structure</strong> <code>inverse_difficulty</code><code>rank_centrality</code><code>hodge_rank</code><code>serial_rank</code><code>pagerank</code><code>spectral</code><code>alpharank</code><code>nash</code></p>
        <p class="conf-api-line"><strong>Pairwise/probabilistic</strong> <code>elo</code><code>trueskill</code><code>glicko</code><code>bradley_terry</code><code>bradley_terry_map</code><code>bradley_terry_davidson</code><code>bradley_terry_davidson_map</code><code>rao_kupper</code><code>rao_kupper_map</code></p>
        <p class="conf-api-line"><strong>Bayesian/IRT</strong> <code>thompson</code><code>bayesian_mcmc</code><code>rasch</code><code>rasch_map</code><code>rasch_mml</code><code>rasch_mml_credible</code><code>rasch_2pl</code><code>rasch_2pl_map</code><code>rasch_3pl</code><code>rasch_3pl_map</code><code>dynamic_irt</code></p>
        <p class="conf-api-line"><strong>Voting/Luce</strong> <code>borda</code><code>copeland</code><code>win_rate</code><code>minimax</code><code>schulze</code><code>ranked_pairs</code><code>kemeny_young</code><code>nanson</code><code>baldwin</code><code>majority_judgment</code><code>plackett_luce</code><code>plackett_luce_map</code><code>davidson_luce</code><code>davidson_luce_map</code><code>bradley_terry_luce</code><code>bradley_terry_luce_map</code></p>
      </div>
    </div>
    <div class="conf-api-group">
      <div class="conf-api-head"><code>scorio.utils</code><span>ranking helpers</span></div>
      <p class="conf-api-line"><code>rank_scores</code><code>compare_rankings</code><code>ranking_hash</code><code>unhash_ranking</code><code>lehmer_hash</code><code>lehmer_unhash</code></p>
    </div>
  </div>
</div>

Note:
  The evaluation half ships as one package too, called Scorio. The eval module is the Don't Pass@k scoring — hand it your M-by-N results matrix and it returns a mean with a credible interval. The rank module is the ranking paper — hand it the L-by-M-by-N tensor and it gives you any of the ranking families we compared, from Bayes to Bradley-Terry to PageRank. The input is exactly the trial data you already collected for pass@k, so adopting it costs nothing extra. The board on the right is just the public API — the full menu of estimators and ranking methods.

---

<div class="conf-takeaway-header">
  <img class="conf-takeaway-icon" src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/takeaway.webp" alt="" aria-hidden="true" />
  <div>
    <h2 class="conf-takeaway-title">Takeaways</h2>
    
  </div>
</div>

<div class="conf-takeaway-grid quad">
  <div class="conf-card conf-takeaway-card">
    <div class="conf-takeaway-num">1</div>
    <h3>DFloat11: shrink weights, change nothing</h3>
    <p>Huffman-code BFloat16's near-empty exponent: ~30% smaller, bit-identical outputs. Fits 405B on one node and buys 5.7&ndash;14.9&times; longer generation.</p>
  </div>
  <div class="conf-card conf-takeaway-card">
    <div class="conf-takeaway-num">2</div>
    <h3>More keys, less values</h3>
    <p>Key weights dominate value weights in norm, so give the KV budget to keys. K4V2 keeps 98.3% accuracy while K2V4 falls behind.</p>
  </div>
  <div class="conf-card conf-takeaway-card">
    <div class="conf-takeaway-num">3</div>
    <h3>Don't Pass@k &mdash; score with a posterior</h3>
    <p>Bayes@N replaces point Pass@k with credible intervals &mdash; and a rule for when a gap is real, not sampling noise.</p>
  </div>
  <div class="conf-card conf-takeaway-card">
    <div class="conf-takeaway-num">4</div>
    <h3>Rank the distribution, not one sample</h3>
    <p>The benchmark is a response tensor. At full budget rankings agree; at N=1 the method you trust depends on the target.</p>
  </div>
</div>

Note:
  Four things to walk out with, one per paper. One: you can compress the weights losslessly — Huffman-code that near-empty exponent for about thirty percent smaller and bit-identical outputs. Two: under a fixed KV budget, give the bits to the keys, because keys dominate values in norm. Three: don't report a point pass@k — report a posterior with a credible interval, and a rule for when a gap is real. Four: rank the whole distribution, and match the ranking method to your trial budget. And underneath all four, one discipline: spend precision where the information is, and quantify the error you can't remove.

---

<div class="conf-closing-layout">
<div class="conf-closing-main">
<h1 class="conf-closing-title">Thank you! Questions?</h1>
<div class="conf-subtitle">Serving reasoning LLMs efficiently and reliably.</div>
<div class="conf-qr-grid" aria-label="Project links">
<div class="conf-qr-item"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/qr_df11.svg" alt="QR code for the DFloat11 paper" /><strong>DFloat11</strong></div>
<div class="conf-qr-item"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/qr_kvq.svg" alt="QR code for the More Keys, Less Values paper" /><strong>More keys, less values</strong></div>
<div class="conf-qr-item"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/qr_bayes.svg" alt="QR code for the Don't Pass@k paper" /><strong>Don't Pass@k</strong></div>
<div class="conf-qr-item"><img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/qr_rank.svg" alt="QR code for the Ranking under Test-Time Scaling paper" /><strong>Ranking &amp; Scorio</strong></div>
</div>
</div>
<div class="conf-ack-stack" aria-label="Acknowledgments">
<div class="conf-ack-item conf-ack-nsf">
<img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/nsf.svg" alt="National Science Foundation" />
<div><h3>NSF Awards</h3><p>2117439 &middot; 2112606 &middot; 2320952</p></div>
</div>
<div class="conf-ack-item conf-ack-cwru">
<img src="/assets/slides/2026-06-19-serving-reasoning-llms-efficiently-and-reliably-no-anime/cwru_hpc_logo.webp" alt="Case Western Reserve University" />
<div><h3>High Performance Computing (HPC)</h3><p>Center at CWRU</p></div>
</div>
</div>
</div>

Note:
  Thank you. Each QR code goes to one of the four papers, and the code for the evaluation half is the Scorio library. I'm happy to take questions — and especially happy to talk about collaborations.

Navigation