Ranking Reasoning LLMs under Test-Time Scaling (Slides)

.reveal .slides {
    text-align: left;
  }

.reveal section.conf-slide {
    box-sizing: border-box;
    width: 100%;
    height: 100%;
    padding: 54px 70px 44px;
    color: var(--conf-ink);
    background:
      linear-gradient(135deg, rgba(37, 99, 235, 0.055), rgba(255, 255, 255, 0) 32%),
      linear-gradient(225deg, rgba(15, 118, 110, 0.055), rgba(255, 255, 255, 0) 36%),
      #f8fafc;
    overflow: hidden;
  }

.reveal section.conf-title {
    padding: 64px 82px 54px;
  }

.reveal .conf-slide h1,
  .reveal .conf-slide h2,
  .reveal .conf-slide h3 {
    color: var(--conf-ink);
    letter-spacing: 0;
    text-transform: none;
  }

.reveal .conf-slide h1 {
    font-size: 76px;
    line-height: 1.03;
    margin: 0;
  }

.reveal .conf-slide h2 {
    font-size: 46px;
    line-height: 1.08;
    margin: 0 0 26px;
  }

.reveal .conf-slide h3 {
    font-size: 25px;
    line-height: 1.18;
    margin: 0 0 10px;
  }

.reveal .conf-slide p,
  .reveal .conf-slide li {
    color: var(--conf-muted);
    font-size: 24px;
    line-height: 1.36;
  }

.reveal .conf-slide strong {
    color: var(--conf-ink);
    font-weight: 720;
  }

.reveal .conf-kicker {
    color: var(--conf-teal);
    font-size: 18px;
    font-weight: 760;
    letter-spacing: 0.08em;
    margin-bottom: 14px;
    text-transform: uppercase;
  }

.reveal .conf-subtitle {
    max-width: 950px;
    color: var(--conf-muted);
    font-size: 31px;
    line-height: 1.32;
    margin-top: 26px;
  }

.reveal .conf-title-grid {
    display: grid;
    grid-template-columns: minmax(0, 1100px) 286px;
    gap: 42px;
    align-content: center;
    align-items: center;
    height: 100%;
    max-width: none;
  }

.reveal .conf-title-main {
    min-width: 0;
  }

.reveal .conf-title-qr {
    box-sizing: border-box;
    justify-self: end;
    width: 286px;
    padding: 18px;
    text-align: center;
  }

.reveal .conf-title-qr img {
    display: block;
    width: 238px;
    height: 238px;
    max-width: none;
    max-height: none;
    object-fit: contain;
    margin: 0 auto 13px;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .conf-title-qr strong {
    display: block;
    color: var(--conf-ink);
    font-size: 21px;
    line-height: 1.15;
  }

.reveal .conf-acl-logo {
    display: block;
    width: 300px;
    max-width: none;
    height: auto;
    margin: 0 0 48px;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .conf-meta {
    color: var(--conf-muted);
    font-size: 21px;
    line-height: 1.45;
    margin-top: 24px;
  }

.reveal .conf-event-line {
    color: var(--conf-teal);
    font-size: 23px;
    font-weight: 760;
    line-height: 1.25;
    margin-top: 18px;
  }

.reveal .conf-logo-row {
    display: flex;
    align-items: center;
    gap: 24px;
    margin-top: 40px;
    flex-wrap: nowrap;
  }

.reveal .conf-logo-badge {
    display: flex;
    align-items: center;
    justify-content: center;
    height: 84px;
    padding: 10px 18px;
    border-radius: 14px;
    background: rgba(255, 255, 255, 0.92);
    box-shadow: 0 18px 46px rgba(15, 23, 42, 0.12);
  }

.reveal .conf-logo-badge.nsf {
    width: 104px;
    height: 92px;
    padding: 7px;
  }

.reveal .conf-logo-badge.scorio {
    width: 250px;
  }

.reveal .conf-logo-row img {
    display: block;
    width: auto;
    max-width: none;
    height: 100%;
    max-height: none;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-panel {
    border: 1px solid rgba(15, 23, 42, 0.14);
    border-radius: 8px;
    background: var(--conf-panel);
    box-shadow: 0 18px 46px rgba(15, 23, 42, 0.08);
  }

.reveal .conf-title-card {
    padding: 24px;
  }

.reveal .conf-title-card img {
    width: 100%;
    height: 370px;
    object-fit: contain;
    display: block;
  }

.reveal .conf-title-metric {
    display: grid;
    grid-template-columns: repeat(2, minmax(0, 1fr));
    gap: 12px;
    margin-top: 20px;
  }

.reveal .conf-title-metric p,
  .reveal .conf-metric p {
    margin: 0;
  }

.reveal .conf-title-metric strong,
  .reveal .conf-metric strong {
    display: block;
    color: var(--conf-blue);
    font-size: 34px;
    line-height: 1;
  }

.reveal .conf-title-metric span,
  .reveal .conf-metric span {
    display: block;
    color: var(--conf-muted);
    font-size: 16px;
    line-height: 1.22;
    margin-top: 7px;
  }

.reveal .conf-chip-row {
    display: flex;
    flex-wrap: wrap;
    gap: 10px;
    margin-top: 18px;
  }

.reveal .conf-chip {
    display: inline-flex;
    align-items: center;
    border: 1px solid rgba(15, 23, 42, 0.14);
    border-radius: 999px;
    color: var(--conf-muted);
    background: rgba(255, 255, 255, 0.74);
    font-size: 16px;
    font-weight: 720;
    line-height: 1;
    padding: 9px 13px;
  }

.reveal .conf-chip.blue {
    color: var(--conf-blue);
    background: var(--conf-blue-soft);
    border-color: rgba(37, 99, 235, 0.24);
  }

.reveal .conf-chip.teal {
    color: var(--conf-teal);
    background: var(--conf-teal-soft);
    border-color: rgba(15, 118, 110, 0.24);
  }

.reveal .conf-chip.amber {
    color: var(--conf-amber);
    background: var(--conf-amber-soft);
    border-color: rgba(180, 83, 9, 0.24);
  }

.reveal .conf-grid {
    display: grid;
    gap: 18px;
  }

.reveal .conf-grid.two {
    grid-template-columns: repeat(2, minmax(0, 1fr));
  }

.reveal .conf-grid.three {
    grid-template-columns: repeat(3, minmax(0, 1fr));
  }

.reveal .conf-grid.four {
    grid-template-columns: repeat(4, minmax(0, 1fr));
  }

.reveal .conf-card {
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.78);
    padding: 20px 22px;
    min-height: 138px;
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-card p {
    margin: 0;
    font-size: 20px;
  }

.reveal .conf-num {
    color: var(--conf-blue);
    font-size: 18px;
    font-weight: 780;
    margin-bottom: 12px;
  }

.reveal .conf-callout {
    border-left: 5px solid var(--conf-blue);
    border-radius: 8px;
    background: var(--conf-blue-soft);
    padding: 18px 22px;
    color: var(--conf-ink);
    font-size: 25px;
    line-height: 1.34;
  }

.reveal .conf-callout.teal {
    border-left-color: var(--conf-teal);
    background: var(--conf-teal-soft);
  }

.reveal .conf-callout.amber {
    border-left-color: var(--conf-amber);
    background: var(--conf-amber-soft);
  }

.reveal .conf-callout.rose {
    border-left-color: var(--conf-rose);
    background: var(--conf-rose-soft);
  }

.reveal .conf-big-question {
    margin-top: 24px;
    padding: 26px 30px;
    color: var(--conf-ink);
    font-size: 35px;
    line-height: 1.2;
  }

.reveal .conf-equation {
    display: flex;
    align-items: center;
    justify-content: center;
    box-sizing: border-box;
    min-width: 0;
    min-height: 112px;
    max-width: 100%;
    overflow-x: auto;
    border: 1px solid rgba(15, 23, 42, 0.12);
    border-radius: 8px;
    background: rgba(15, 23, 42, 0.04);
    color: var(--conf-ink);
    font-size: 31px;
    line-height: 1.25;
    padding: 18px 22px;
  }

.reveal .conf-equation .katex { max-width: 100%; }

.reveal .conf-equation.sm {
    font-size: 24px;
  }

.reveal .conf-figure {
    border: 1px solid rgba(15, 23, 42, 0.12);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.84);
    padding: 16px;
  }

.reveal .conf-figure img {
    display: block;
    width: 100%;
    height: 100%;
    object-fit: contain;
  }

.reveal .conf-figure figcaption {
    color: var(--conf-muted);
    font-size: 16px;
    line-height: 1.25;
    margin-top: 10px;
  }

.reveal .conf-fig-short {
    height: 430px;
  }

.reveal .conf-fig-mid {
    height: 540px;
  }

.reveal .conf-fig-wide {
    height: 465px;
  }

.reveal .conf-fig-tall {
    height: 610px;
  }

.reveal .conf-split {
    display: grid;
    grid-template-columns: minmax(0, 0.98fr) minmax(420px, 0.82fr);
    gap: 30px;
    align-items: stretch;
  }

.reveal .conf-split.reverse {
    grid-template-columns: minmax(420px, 0.76fr) minmax(0, 1fr);
  }

.reveal .conf-metric {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 14px;
    margin-top: 18px;
  }

.reveal .conf-metric .conf-card {
    min-height: 104px;
  }

.reveal .conf-mini-table,
  .reveal .conf-table {
    width: 100%;
    border-collapse: collapse;
  }

.reveal .conf-mini-table {
    font-size: 18px;
  }

.reveal .conf-table {
    font-size: 20px;
  }

.reveal .conf-mini-table th,
  .reveal .conf-mini-table td,
  .reveal .conf-table th,
  .reveal .conf-table td {
    border-bottom: 1px solid rgba(15, 23, 42, 0.12);
    padding: 11px 10px;
    text-align: right;
    color: var(--conf-muted);
  }

.reveal .conf-mini-table th:first-child,
  .reveal .conf-mini-table td:first-child,
  .reveal .conf-table th:first-child,
  .reveal .conf-table td:first-child {
    text-align: left;
  }

.reveal .conf-mini-table th,
  .reveal .conf-table th {
    color: var(--conf-ink);
    font-weight: 760;
  }

.reveal .conf-win {
    color: var(--conf-green) !important;
    font-weight: 760;
    background: rgba(21, 128, 61, 0.08);
  }

.reveal .conf-loss {
    color: var(--conf-rose) !important;
  }

.reveal .conf-flow {
    display: grid;
    grid-template-columns: repeat(4, minmax(0, 1fr));
    gap: 12px;
    margin-top: 18px;
  }

.reveal .conf-flow .conf-card {
    min-height: 148px;
  }

.reveal .conf-flow h3 {
    font-size: 23px;
  }

.reveal .conf-list {
    margin: 0;
    padding-left: 1.1em;
  }

.reveal .conf-list li {
    margin: 0 0 12px;
  }

.reveal .conf-footer-note {
    position: absolute;
    left: 70px;
    right: 70px;
    bottom: 28px;
    color: rgba(71, 85, 105, 0.78);
    font-size: 16px;
  }

.reveal .conf-takeaway-header {
    display: flex;
    align-items: center;
    gap: 24px;
    margin: 0 0 34px;
  }

.reveal .conf-takeaway-icon {
    display: block;
    width: 104px;
    height: 104px;
    max-width: none;
    max-height: none;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-takeaway-title {
    color: var(--conf-ink);
    font-size: 62px;
    line-height: 0.98;
    margin: 0;
  }

.reveal .conf-takeaway-subtitle {
    color: var(--conf-teal);
    font-size: 28px;
    font-weight: 760;
    line-height: 1.12;
    margin-top: 12px;
  }

.reveal .conf-takeaway-grid {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 22px;
  }

.reveal .conf-takeaway-card {
    position: relative;
    min-height: 250px;
    padding: 30px 30px 28px;
    overflow: hidden;
    border-top: 5px solid var(--conf-teal);
    background:
      linear-gradient(180deg, rgba(15, 118, 110, 0.08), rgba(255, 255, 255, 0.82) 42%),
      rgba(255, 255, 255, 0.84);
    box-shadow: 0 18px 42px rgba(15, 23, 42, 0.09);
  }

.reveal .conf-takeaway-card:nth-child(2) {
    border-top-color: var(--conf-blue);
    background:
      linear-gradient(180deg, rgba(37, 99, 235, 0.08), rgba(255, 255, 255, 0.82) 42%),
      rgba(255, 255, 255, 0.84);
  }

.reveal .conf-takeaway-card:nth-child(3) {
    border-top-color: var(--conf-amber);
    background:
      linear-gradient(180deg, rgba(180, 83, 9, 0.08), rgba(255, 255, 255, 0.82) 42%),
      rgba(255, 255, 255, 0.84);
  }

.reveal .conf-takeaway-num {
    display: inline-grid;
    place-items: center;
    width: 42px;
    height: 42px;
    margin-bottom: 20px;
    border-radius: 50%;
    background: var(--conf-teal-soft);
    color: var(--conf-teal);
    font-size: 22px;
    font-weight: 820;
  }

.reveal .conf-takeaway-card:nth-child(2) .conf-takeaway-num {
    background: var(--conf-blue-soft);
    color: var(--conf-blue);
  }

.reveal .conf-takeaway-card:nth-child(3) .conf-takeaway-num {
    background: var(--conf-amber-soft);
    color: var(--conf-amber);
  }

.reveal .conf-takeaway-card h3 {
    font-size: 30px;
    line-height: 1.08;
    min-height: 42px;
    margin-bottom: 16px;
  }

.reveal .conf-takeaway-card p {
    font-size: 25px;
    line-height: 1.34;
    min-height: 128px;
  }

.reveal .conf-closing-layout {
    display: grid;
    grid-template-rows: auto minmax(0, 1fr) auto;
    gap: 26px;
    align-items: stretch;
    height: 100%;
  }

.reveal .conf-closing-main {
    display: grid;
    justify-items: center;
    align-content: start;
    text-align: center;
    padding-top: 8px;
  }

.reveal .conf-closing-title {
    font-size: 72px;
    line-height: 1.03;
    margin: 0 0 18px;
  }

.reveal .conf-closing-main .conf-subtitle {
    max-width: none;
    margin-top: 0;
  }

.reveal .conf-closing-authors {
    color: var(--conf-muted);
    font-size: 22px;
    line-height: 1.25;
    margin-top: 12px;
  }

.reveal .conf-qr-grid {
    display: grid;
    grid-template-columns: repeat(4, 260px);
    justify-content: center;
    gap: 24px;
    width: min(1120px, 100%);
    margin: 32px auto 0;
  }

.reveal .conf-qr-item {
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.82);
    padding: 18px;
    text-align: center;
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-qr-item img {
    display: block;
    width: 210px;
    height: 210px;
    max-width: none;
    max-height: none;
    object-fit: contain;
    margin: 0 auto 12px;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .conf-qr-item strong {
    display: block;
    color: var(--conf-ink);
    font-size: 20px;
    line-height: 1.1;
  }

.reveal .conf-contact-line {
    max-width: 1220px;
    margin: 18px auto 0;
    color: var(--conf-muted);
    font-size: 20px;
    line-height: 1.25;
  }

.reveal .conf-contact-email {
    color: var(--conf-blue);
    font-weight: 780;
    white-space: nowrap;
  }

.reveal .conf-ack-stack {
    display: grid;
    grid-template-columns: 450px 760px;
    justify-content: center;
    align-items: center;
    gap: 34px;
    width: 100%;
    margin: 0 auto;
    padding: 0;
  }

.reveal .conf-ack-item {
    display: grid;
    grid-template-columns: auto minmax(0, 1fr);
    gap: 22px;
    align-items: center;
    border: 0;
    border-radius: 0;
    background: transparent;
    padding: 0;
    box-shadow: none;
  }

.reveal .conf-ack-cwru {
    grid-template-columns: 290px minmax(0, 1fr);
  }

.reveal .conf-ack-item img {
    display: block;
    width: auto;
    max-width: none;
    max-height: none;
    object-fit: contain;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-ack-nsf img {
    width: 84px;
    height: 84px;
  }

.reveal .conf-ack-cwru img {
    width: 270px;
    height: auto;
  }

.reveal .conf-ack-item h3 {
    font-size: 26px;
    line-height: 1.08;
    margin: 0 0 8px;
    white-space: nowrap;
  }

.reveal .conf-ack-item p {
    font-size: 22px;
    line-height: 1.18;
    margin: 0;
    white-space: nowrap;
  }

.reveal .slides section.conf-slide .conf-acl-logo {
    width: 300px;
    max-width: none;
    height: auto;
    margin: 0 0 48px;
    border-radius: 0;
    box-shadow: none;
  }

.reveal .slides section.conf-slide .conf-logo-row img {
    width: auto;
    max-width: none;
    height: 64px;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .slides section.conf-slide .conf-logo-badge.nsf img {
    width: 76px;
    height: 76px;
  }

.reveal .slides section.conf-slide .conf-logo-badge.scorio img {
    width: 214px;
    height: auto;
  }

.reveal .slides section.conf-slide .conf-qr-item img {
    width: 210px;
    height: 210px;
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0 auto 12px;
  }

.reveal .slides section.conf-slide .conf-title-qr img {
    width: 238px;
    height: 238px;
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0 auto 13px;
  }

.reveal .slides section.conf-slide .conf-ack-item img {
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .slides section.conf-slide .conf-ack-nsf img {
    width: 84px;
    height: 84px;
  }

.reveal .slides section.conf-slide .conf-ack-cwru img {
    width: 270px;
    height: auto;
  }

.reveal .conf-title-thesis {
    max-width: 1080px;
    color: var(--conf-muted);
    font-size: 29px;
    line-height: 1.28;
    margin-top: 28px;
  }

.reveal .conf-slide .conf-lead {
    max-width: 1210px;
    color: var(--conf-muted);
    font-size: 30px;
    line-height: 1.28;
    margin: -8px 0 28px;
  }

.reveal .conf-slide .conf-small {
    color: var(--conf-muted);
    font-size: 18px;
    line-height: 1.3;
  }

.reveal .conf-stat-row {
    display: grid;
    grid-template-columns: repeat(4, minmax(0, 1fr));
    gap: 14px;
    margin-top: 20px;
  }

.reveal .conf-stat {
    border: 1px solid rgba(15, 23, 42, 0.12);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.78);
    padding: 18px 18px 16px;
    min-height: 112px;
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-stat strong {
    display: block;
    color: var(--conf-blue);
    font-size: 43px;
    line-height: 1;
  }

.reveal .conf-stat span {
    display: block;
    color: var(--conf-muted);
    font-size: 17px;
    line-height: 1.22;
    margin-top: 9px;
  }

.reveal .conf-slide .conf-table.compact {
    font-size: 17px;
  }

.reveal .conf-slide .conf-table.compact th,
  .reveal .conf-slide .conf-table.compact td {
    padding: 8px 8px;
  }

.reveal .conf-slide .conf-table.wide-text td,
  .reveal .conf-slide .conf-table.wide-text th {
    text-align: left;
  }

.reveal .conf-slide .conf-table.wide-text td:last-child,
  .reveal .conf-slide .conf-table.wide-text th:last-child {
    text-align: right;
  }

.reveal .conf-slide .conf-table .conf-method-cell {
    color: var(--conf-ink);
    font-weight: 720;
  }

.reveal .conf-two-thirds {
    display: grid;
    grid-template-columns: minmax(0, 1.05fr) minmax(420px, 0.72fr);
    gap: 28px;
    align-items: stretch;
  }

.reveal .conf-figure-grid {
    display: grid;
    grid-template-columns: minmax(0, 1fr) minmax(390px, 0.52fr);
    gap: 24px;
    align-items: stretch;
  }

.reveal .conf-figure-grid.reverse {
    grid-template-columns: minmax(390px, 0.52fr) minmax(0, 1fr);
  }

.reveal .conf-figure-grid .conf-figure {
    height: 590px;
  }

.reveal .conf-figure-grid .conf-figure.short {
    height: 500px;
  }

.reveal .conf-stack {
    display: grid;
    gap: 16px;
    align-content: start;
  }

.reveal .conf-target-card {
    min-height: 210px;
  }

.reveal .conf-target-card .conf-equation {
    min-height: 82px;
    margin-bottom: 16px;
    font-size: 27px;
  }

.reveal .conf-method-map {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 18px;
    margin-top: 20px;
  }

.reveal .conf-method-map .conf-card {
    min-height: 255px;
  }

.reveal .conf-method-map .conf-equation {
    min-height: 76px;
    font-size: 23px;
    margin-bottom: 14px;
  }

.reveal .conf-rule-grid {
    display: grid;
    grid-template-columns: repeat(5, minmax(0, 1fr));
    gap: 13px;
    margin-top: 26px;
  }

.reveal .conf-rule-grid .conf-card {
    min-height: 190px;
    padding: 18px 18px;
  }

.reveal .conf-rule-grid h3 {
    font-size: 22px;
  }

.reveal .conf-code {
    box-sizing: border-box;
    max-width: 100%;
    overflow-x: auto;
    border: 1px solid rgba(15, 23, 42, 0.14);
    border-radius: 8px;
    background: #111827;
    color: #e5e7eb;
    font-size: 20px;
    line-height: 1.42;
    margin: 0;
    padding: 24px 26px;
    white-space: pre;
  }

.reveal .conf-code .kw {
    color: #67e8f9;
  }

.reveal .conf-code .fn {
    color: #a7f3d0;
  }

.reveal .conf-code .arg {
    color: #fbbf24;
  }

.reveal .conf-qrs-small {
    display: grid;
    grid-template-columns: repeat(2, 180px);
    gap: 18px;
    justify-content: start;
    margin-top: 16px;
  }

.reveal .conf-qrs-small .conf-qr-item {
    padding: 14px;
  }

.reveal .conf-qrs-small .conf-qr-item img {
    width: 144px;
    height: 144px;
  }

.reveal .conf-scorio-header {
    display: grid;
    grid-template-columns: 250px minmax(0, 1fr);
    gap: 24px;
    align-items: center;
    margin-bottom: 18px;
  }

.reveal .conf-scorio-logo-panel {
    display: flex;
    align-items: center;
    justify-content: flex-start;
    height: 104px;
  }

.reveal .conf-scorio-logo-panel img {
    display: block;
    width: 202px;
    height: auto;
    max-width: none;
    max-height: none;
    border-radius: 0;
    box-shadow: none;
    margin: 0;
  }

.reveal .conf-scorio-header h2 {
    font-size: 40px;
    line-height: 1.08;
    margin: 0;
  }

.reveal .conf-scorio-app {
    border: 1px solid rgba(15, 118, 110, 0.22);
    border-radius: 999px;
    background: var(--conf-teal-soft);
    color: var(--conf-teal);
    font-size: 17px;
    font-weight: 780;
    line-height: 1;
    padding: 11px 15px;
    text-align: center;
  }

.reveal .conf-scorio-layout {
    display: grid;
    grid-template-columns: minmax(0, 0.66fr) minmax(0, 1.34fr);
    gap: 18px;
    align-items: stretch;
  }

.reveal .conf-scorio-panel,
  .reveal .conf-api-board {
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.8);
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.055);
  }

.reveal .conf-scorio-panel {
    padding: 18px;
  }

.reveal .conf-scorio-panel h3,
  .reveal .conf-api-board h3 {
    font-size: 23px;
    line-height: 1.08;
    margin: 0 0 12px;
  }

.reveal .conf-scorio-code {
    font-size: 15px;
    line-height: 1.34;
    padding: 16px 18px;
  }

.reveal .conf-scorio-note {
    color: var(--conf-muted);
    font-size: 18px;
    line-height: 1.26;
    margin-top: 14px;
  }

.reveal .conf-scorio-note strong {
    color: var(--conf-ink);
  }

.reveal .conf-scorio-use {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 8px;
    margin-top: 14px;
  }

.reveal .conf-scorio-use .conf-scorio-app {
    display: grid;
    place-items: center;
    min-height: 52px;
    border: 1px solid rgba(15, 118, 110, 0.22);
    border-radius: 999px;
    background: var(--conf-teal-soft);
    color: var(--conf-teal);
    font-size: 15px;
    font-weight: 780;
    line-height: 1.08;
    padding: 10px 12px;
    text-align: center;
  }

.reveal .conf-api-board {
    padding: 16px 18px 14px;
  }

.reveal .conf-api-board h3 {
    display: flex;
    align-items: baseline;
    justify-content: space-between;
    gap: 16px;
  }

.reveal .conf-api-board h3 span {
    color: var(--conf-muted);
    font-size: 15px;
    font-weight: 680;
  }

.reveal .conf-api-group {
    padding: 8px 0;
    border-top: 1px solid rgba(15, 23, 42, 0.11);
  }

.reveal .conf-api-head {
    display: flex;
    align-items: baseline;
    gap: 12px;
    margin-bottom: 4px;
  }

.reveal .conf-api-head code {
    color: var(--conf-ink);
    font-size: 16px;
    font-weight: 820;
  }

.reveal .conf-api-head span {
    color: var(--conf-teal);
    font-size: 14px;
    font-weight: 760;
  }

.reveal .conf-api-columns {
    display: grid;
    grid-template-columns: repeat(2, minmax(0, 1fr));
    column-gap: 16px;
    row-gap: 2px;
  }

.reveal .conf-slide .conf-api-line {
    color: var(--conf-muted);
    font-size: 13px;
    line-height: 1.22;
    margin: 0 0 5px;
  }

.reveal .conf-slide .conf-api-line strong {
    display: inline;
    color: var(--conf-ink);
    font-size: 13px;
    font-weight: 800;
  }

.reveal .conf-slide .conf-api-line code {
    display: inline-block;
    border-radius: 5px;
    background: rgba(15, 23, 42, 0.055);
    color: var(--conf-ink);
    font-size: 11.5px;
    line-height: 1.18;
    margin: 0 2px 3px 0;
    padding: 2px 5px;
  }

.reveal .conf-slide .conf-fit-list li {
    font-size: 21px;
    line-height: 1.28;
    margin-bottom: 10px;
  }

.reveal .conf-slide .conf-tight-list li {
    font-size: 19px;
    line-height: 1.24;
    margin-bottom: 8px;
  }

.reveal .conf-banner {
    display: grid;
    grid-template-columns: auto minmax(0, 1fr);
    gap: 18px;
    align-items: center;
    border: 1px solid rgba(15, 23, 42, 0.13);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.78);
    padding: 18px 22px;
    margin-top: 20px;
  }

.reveal .conf-banner-number {
    color: var(--conf-blue);
    font-size: 54px;
    font-weight: 780;
    line-height: 1;
  }

.reveal .conf-banner p {
    margin: 0;
    font-size: 23px;
    line-height: 1.26;
  }

.reveal .conf-label {
    display: inline-flex;
    align-items: center;
    border-radius: 999px;
    background: rgba(15, 23, 42, 0.08);
    color: var(--conf-ink);
    font-size: 15px;
    font-weight: 760;
    line-height: 1;
    padding: 8px 10px;
    text-transform: uppercase;
  }

.reveal .conf-figure img.no-stretch {
    height: auto;
    max-height: 100%;
  }
</style>

<div class="conf-title-grid">
  <div class="conf-title-main">
    <img class="conf-acl-logo" src="/assets/slides/2026-06-12-ranking-reasoning/ACL.webp" alt="ACL 2026" />
    <h1>Ranking Reasoning LLMs<br>under Test-Time Scaling</h1>
    <div class="conf-event-line">Coronado | July 6, 2026 | 9:00 AM - 10:30 AM</div>
    <div class="conf-meta">
      <strong>Mohsen Hariri</strong>, Michael Hinczewski, Jing Ma, Vipin Chaudhary
    </div>
    <div class="conf-logo-row">
      <span class="conf-logo-badge"><img src="/assets/slides/2026-06-12-ranking-reasoning/cwru.svg" alt="Case Western Reserve University" /></span>
      <span class="conf-logo-badge scorio"><img src="/assets/slides/2026-06-12-ranking-reasoning/logo/scorio.svg" alt="Scorio" /></span>
      <span class="conf-logo-badge nsf"><img src="/assets/slides/2026-06-12-ranking-reasoning/nsf.svg" alt="National Science Foundation" /></span>
    </div>
  </div>
  <aside class="conf-title-qr conf-panel" aria-label="Interactive slides QR code">
    <img src="/assets/slides/2026-06-12-ranking-reasoning/slide_qr.webp" alt="QR code for the interactive slides" />
    <strong>Scan for interactive slides</strong>
  </aside>
</div>

Note:
  The main message is that repeated sampling changes how we should rank reasoning models.
  The deck compares ranking methods under test-time scaling and motivates Scorio as a practical toolkit.

---

<div class="conf-kicker">Motivation</div>
<h2>A one-sample leaderboard hides the object we are ranking</h2>
<p class="conf-lead">
  Test-time scaling turns each model-question outcome into a distribution of sampled attempts. The ranking question becomes statistical: which ordering survives the sampling noise?
</p>

<div class="conf-two-thirds">
  <div class="conf-stack">
    <div class="conf-callout">
      The unit of evidence is no longer one score per model. It is a repeated-trial response tensor that can be sliced by model, question, and trial budget.
    </div>
    <div class="conf-grid three">
      <div class="conf-card">
        <div class="conf-num">01</div>
        <h3>More samples</h3>
        <p>Stochastic decodes expose variance, near ties, and ranking reversals.</p>
      </div>
      <div class="conf-card">
        <div class="conf-num">02</div>
        <h3>Many ranking rules</h3>
        <p>Pointwise, pairwise, voting, graph, IRT, and Bayesian methods can disagree.</p>
      </div>
      <div class="conf-card">
        <div class="conf-num">03</div>
        <h3>Target matters</h3>
        <p>Accuracy agreement and self-consistency reward different behavior.</p>
      </div>
    </div>
  </div>
  <div class="conf-panel conf-title-card">
    <h3>Core question</h3>
    <div class="conf-big-question">
      If we can afford only one or two trials per prompt, which ranking should we trust?
    </div>
    <div class="conf-chip-row">
      <span class="conf-chip blue">low-budget stability</span>
      <span class="conf-chip teal">full-budget agreement</span>
      <span class="conf-chip amber">prior bias</span>
    </div>
  </div>
</div>

Note:
  Start from the practical problem. Evaluation under test-time scaling is not just a bigger pass@k table.
  Once outputs are sampled repeatedly, the leaderboard is an estimator, and we can ask how stable that estimator is.

---

<div class="conf-kicker">Overview</div>
<h2>The paper makes ranking under repeated sampling explicit</h2>

<div class="conf-flow">
  <div class="conf-card">
    <div class="conf-num">1</div>
    <h3>Represent the benchmark</h3>
    <p>Use \( \mathbf{R}\in\{0,1\}^{L\times M\times N} \) as the primitive data object.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">2</div>
    <h3>Compare ranking families</h3>
    <p>Study 72 methods spanning metrics, Bayesian estimators, IRT, voting, graph, and paired-comparison models.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">3</div>
    <h3>Choose evaluation targets</h3>
    <p>Measure agreement with \( \mathrm{Bayes}_{\mathcal{U}}@80 \) and convergence to each method's own full-trial rank.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">4</div>
    <h3>Turn results into guidance</h3>
    <p>Use \( \mathrm{Bayes}_{\mathcal{U}}@N \) by default; use empirical priors only after alignment checks.</p>
  </div>
</div>

<div class="conf-banner">
  <div class="conf-banner-number">72</div>
  <p>ranking methods are compared on the same dense repeated-trial evidence, so differences come from the ranking rule rather than data availability.</p>
</div>

Note:
  The framing that matters: the paper is not proposing one more exotic leaderboard. It builds a common evaluation harness for ranking rules under test-time scaling.

---

<div class="conf-kicker">Formalization</div>
<h2>The benchmark is a tensor, not a table</h2>

<div class="conf-two-thirds">
  <div>
    <div class="conf-equation">
      \( R_{lmn}=1 \) if model \(l\) solves question \(m\) on trial \(n\)
    </div>
    <div class="conf-stat-row">
      <div class="conf-stat">
        <strong>20</strong>
        <span>reasoning LLMs</span>
      </div>
      <div class="conf-stat">
        <strong>30</strong>
        <span>questions per benchmark</span>
      </div>
      <div class="conf-stat">
        <strong>80</strong>
        <span>top-p trials per pair</span>
      </div>
      <div class="conf-stat">
        <strong>1</strong>
        <span>greedy prior draw \( \mathbf{R}_0 \)</span>
      </div>
    </div>
    <div class="conf-callout teal" style="margin-top: 22px;">
      The single-run benchmark is only the \(N=1\) slice. Repeated trials let us ask how rankings move as the trial budget grows.
    </div>
  </div>
  <div class="conf-panel conf-title-card">
    <h3>One tensor supports multiple views</h3>
    <ul class="conf-list conf-fit-list">
      <li>Per-question solve rates \( \widehat{p}_{lm} \)</li>
      <li>Pairwise win/tie counts \( W_{ij}, T_{ij} \)</li>
      <li>Winner sets \( U_{mn} \) for listwise methods</li>
      <li>Posterior distributions for uncertainty-aware ranks</li>
    </ul>
  </div>
</div>

Note:
  The notation is simple but important.
  Every model-question pair is observed repeatedly, so we have a dense tensor. This differs from sparse preference graphs like Chatbot Arena.

---

<div class="conf-kicker">Method families</div>
<h2>Ranking rules are different projections of the same tensor</h2>

<div class="conf-method-map">
  <div class="conf-card">
    <div class="conf-label">Pointwise</div>
    <div class="conf-equation">\( \widehat{p}_{lm} = \frac{1}{N}\sum_n R_{lmn} \)</div>
    <h3>Model-question performance</h3>
    <p>Average accuracy, \( \mathrm{Bayes}@N \), \( \mathrm{Pass}@k \), difficulty weighting, and IRT-style models.</p>
  </div>
  <div class="conf-card">
    <div class="conf-label">Pairwise</div>
    <div class="conf-equation">\( W_{ij}, W_{ji}, T_{ij} \)</div>
    <h3>Model-model comparisons</h3>
    <p>Bradley-Terry, Elo, TrueSkill, Borda, Copeland, PageRank, Rank Centrality, and HodgeRank.</p>
  </div>
  <div class="conf-card">
    <div class="conf-label">Setwise</div>
    <div class="conf-equation">\( U_{mn}=\{l:R_{lmn}=1\} \)</div>
    <h3>Winner and loser sets</h3>
    <p>Plackett-Luce and Davidson-Luce operate on two-level partial orders per question-trial.</p>
  </div>
</div>

<div class="conf-callout amber" style="margin-top: 24px;">
  These methods can be equally reasonable while optimizing different notions of rank. The evaluation target has to be named before declaring a winner.
</div>

Note:
  This is not Bayes versus everything else. The same tensor reduces to pointwise, pairwise, or setwise data, and each reduction keeps or discards different information.

---

<div class="conf-kicker">Evaluation targets</div>
<h2>There is no universal gold standard for ranking methods</h2>

<div class="conf-grid two">
  <div class="conf-card conf-target-card">
    <div class="conf-label">Target A</div>
    <div class="conf-equation">\( \mathrm{Bayes}_{\mathcal{U}}@80 \equiv \mathrm{avg}@80 \)</div>
    <h3>Accuracy-based gold standard</h3>
    <p>Uses all 80 stochastic trials and a uniform prior. It is interpretable, stable at high budget, and allows ties.</p>
  </div>
  <div class="conf-card conf-target-card">
    <div class="conf-label">Target B</div>
    <div class="conf-equation">\( \mathrm{method}@80 \)</div>
    <h3>Self-consistency target</h3>
    <p>Asks whether a method at low budget converges to its own full-trial ordering, regardless of accuracy alignment.</p>
  </div>
</div>

<div class="conf-callout rose" style="margin-top: 24px;">
  A method can be highly self-consistent and still move away from the correctness-based ordering. The paper reports both views.
</div>

Note:
  Bayes_U@80 is a gold standard only for an accuracy-based target. We are careful not to claim that every ranking method must converge to this ordering.

---

<div class="conf-kicker">Experiment setup</div>
<h2>Dense repeated trials on Olympiad-style math benchmarks</h2>

<div class="conf-stat-row">
  <div class="conf-stat">
    <strong>4</strong>
    <span>AIME'24, AIME'25, HMMT'25, BrUMO'25</span>
  </div>
  <div class="conf-stat">
    <strong>20</strong>
    <span>reasoning models</span>
  </div>
  <div class="conf-stat">
    <strong>80</strong>
    <span>independent stochastic trials</span>
  </div>
  <div class="conf-stat">
    <strong>69</strong>
    <span>N=1 methods after excluding Pass@k variants</span>
  </div>
</div>

<div class="conf-grid two" style="margin-top: 24px;">
  <div class="conf-card">
    <h3>Low-budget protocol</h3>
    <ul class="conf-list conf-fit-list">
      <li>Subsample one of the 80 trials per question.</li>
      <li>Recompute each ranking method.</li>
      <li>Repeat over all single-trial draws.</li>
      <li>Report mean \( \tau_b \pm \) standard deviation.</li>
    </ul>
  </div>
  <div class="conf-card">
    <h3>Prior diagnostic</h3>
    <ul class="conf-list conf-fit-list">
      <li>Collect one greedy-decoding run \( \mathbf{R}_0 \).</li>
      <li>Compare \( \mathrm{Bayes}_{\mathcal{U}}@N \) with \( \mathrm{Bayes}_{\mathbf{R}_0}@N \).</li>
      <li>Measure greedy-sampling rank alignment before trusting the prior.</li>
    </ul>
  </div>
</div>

Note:
  The experimental design makes ranking stability measurable.
  At N=1, pass@k variants are excluded because they need at least two trials to differ from mean accuracy.

---

<div class="conf-kicker">Result 1: high-budget consensus</div>
<h2>With 80 trials, most reasonable ranking families agree</h2>

<div class="conf-figure-grid">
  <figure class="conf-figure">
    <img src="/assets/slides/2026-06-12-ranking-reasoning/figures/main_bump_chart_rankings_show_case.webp" alt="Bump charts showing agreement between full-trial ranking methods and Bayes U at N=80" />
    <figcaption>Full-trial rankings against \( \mathrm{Bayes}_{\mathcal{U}}@80 \) on BrUMO'25 and HMMT'25.</figcaption>
  </figure>
  <div class="conf-stack">
    <div class="conf-callout">
      Mean agreement with \( \mathrm{Bayes}_{\mathcal{U}}@80 \) is \( \tau_b=0.93 \) to \(0.95\) across benchmarks.
    </div>
    <table class="conf-table compact">
      <thead>
        <tr><th>Benchmark</th><th>Mean</th><th>Exact</th><th>&ge;0.95</th></tr>
      </thead>
      <tbody>
        <tr><td>AIME'24</td><td>0.941</td><td>20</td><td>40</td></tr>
        <tr><td>AIME'25</td><td>0.934</td><td>19</td><td>29</td></tr>
        <tr><td>HMMT'25</td><td>0.950</td><td>34</td><td>44</td></tr>
        <tr><td>BrUMO'25</td><td>0.954</td><td>26</td><td>49</td></tr>
        <tr><td>Combined</td><td>0.962</td><td>22</td><td>53</td></tr>
      </tbody>
    </table>
    <p class="conf-small">Statistics are over the other 71 methods computed from all 80 trials.</p>
  </div>
</div>

Note:
  At full budget, the story is reassuring: most methods land close to the accuracy-based target.
  The interesting differences appear when we reduce the trial budget.

---

<div class="conf-kicker">Result 2: low-budget ranking</div>
<h2>At \(N=1\), the best method depends on the target</h2>

<table class="conf-table compact wide-text">
  <thead>
    <tr>
      <th>Benchmark</th>
      <th>Best vs. \( \mathrm{Bayes}_{\mathcal{U}}@80 \)</th>
      <th>\(\tau_b\)</th>
      <th>Best self-consistency</th>
      <th>\(\tau_b\)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>AIME'24</td>
      <td class="conf-method-cell">\( \mathrm{Bayes}_{\mathbf{R}_0}@1 \)</td>
      <td>0.779 &plusmn; 0.034</td>
      <td class="conf-method-cell">Rasch MML LCB</td>
      <td>0.804 &plusmn; 0.051</td>
    </tr>
    <tr>
      <td>AIME'25</td>
      <td class="conf-method-cell">\( \mathrm{Bayes}_{\mathbf{R}_0}@1 \)</td>
      <td>0.798 &plusmn; 0.045</td>
      <td class="conf-method-cell">Rasch MML LCB</td>
      <td>0.834 &plusmn; 0.054</td>
    </tr>
    <tr>
      <td>HMMT'25</td>
      <td class="conf-method-cell">\( \mathrm{Bayes}_{\mathcal{U}}@1 \) and 20 tied methods</td>
      <td>0.790 &plusmn; 0.053</td>
      <td class="conf-method-cell">Rasch MML LCB</td>
      <td>0.810 &plusmn; 0.056</td>
    </tr>
    <tr>
      <td>BrUMO'25</td>
      <td class="conf-method-cell">\( \mathrm{Bayes}_{\mathbf{R}_0}@1 \)</td>
      <td>0.858 &plusmn; 0.028</td>
      <td class="conf-method-cell">\( \mathrm{Bayes}_{\mathbf{R}_0}@1 \)</td>
      <td>0.858 &plusmn; 0.028</td>
    </tr>
    <tr>
      <td>Combined</td>
      <td class="conf-method-cell">\( \mathrm{Bayes}_{\mathcal{U}}@1 \) and 20 tied methods</td>
      <td>0.865 &plusmn; 0.049</td>
      <td class="conf-method-cell">Nanson avg ties</td>
      <td>0.892 &plusmn; 0.050</td>
    </tr>
  </tbody>
</table>

<div class="conf-callout teal" style="margin-top: 22px;">
  \( \mathrm{Bayes}_{\mathbf{R}_0}@1 \) is strongest when greedy decoding is aligned with stochastic sampling. Self-consistency winners can be different.
</div>

Note:
  The practical point is not that one method always wins. It is that you have to name whether you want accuracy agreement or repeatability before you pick one.

---

<div class="conf-kicker">Result 3: empirical priors</div>
<h2>A greedy prior reduces variance, but can shift the rank</h2>

<div class="conf-figure-grid">
  <figure class="conf-figure">
    <img src="/assets/slides/2026-06-12-ranking-reasoning/figures/main_convergence_by_dataset.webp" alt="Kendall tau agreement of uniform and greedy-prior Bayes rankings as the number of trials increases" />
    <figcaption>\( \mathrm{Bayes}_{\mathcal{U}}@N \) versus \( \mathrm{Bayes}_{\mathbf{R}_0}@N \) across trial budgets.</figcaption>
  </figure>
  <div class="conf-stack">
    <table class="conf-table compact">
      <thead>
        <tr><th>Benchmark</th><th>Difficulty</th><th>\(\tau_{G-S}\)</th><th>&Delta;\(\tau\)</th><th>Std. red.</th></tr>
      </thead>
      <tbody>
        <tr><td>AIME'24</td><td>0.620</td><td>0.739</td><td class="conf-win">+0.020</td><td>42%</td></tr>
        <tr><td>AIME'25</td><td>0.533</td><td>0.660</td><td class="conf-win">+0.008</td><td>17%</td></tr>
        <tr><td>HMMT'25</td><td>0.333</td><td>0.635</td><td class="conf-loss">-0.022</td><td>16%</td></tr>
        <tr><td>BrUMO'25</td><td>0.588</td><td>0.768</td><td class="conf-win">+0.049</td><td>52%</td></tr>
      </tbody>
    </table>
    <div class="conf-callout amber">
      At \(N=1\), the greedy prior reduces the standard deviation of agreement by 16% to 52%, but HMMT shows a negative mean shift.
    </div>
  </div>
</div>

Note:
  The greedy prior behaves like shrinkage toward the greedy ordering.
  Shrinkage reduces variance, but if greedy and stochastic sampling disagree, it can move the ranking away from the high-budget stochastic target.

---

<div class="conf-kicker">Diagnostic</div>
<h2>Greedy-sampling alignment explains when the prior helps</h2>

<div class="conf-figure-grid reverse">
  <div class="conf-stack">
    <div class="conf-card">
      <h3>Prior helps when</h3>
      <p>Greedy decoding and stochastic sampling induce similar model rankings.</p>
      <div class="conf-chip-row">
        <span class="conf-chip teal">BrUMO'25</span>
        <span class="conf-chip blue">\(\tau_{G-S}=0.768\)</span>
        <span class="conf-chip blue">&Delta;\(\tau=+0.049\)</span>
      </div>
    </div>
    <div class="conf-card">
      <h3>Prior hurts when</h3>
      <p>Greedy decoding under-explores hard instances and drifts from sampled reasoning paths.</p>
      <div class="conf-chip-row">
        <span class="conf-chip amber">HMMT'25</span>
        <span class="conf-chip blue">\(\tau_{G-S}=0.635\)</span>
        <span class="conf-chip amber">&Delta;\(\tau=-0.022\)</span>
      </div>
    </div>
    <div class="conf-callout">
      Recommendation: before using \( \mathbf{R}_0 \) as a prior, check a small pilot sample for greedy-sampling rank alignment.
    </div>
  </div>
  <figure class="conf-figure">
    <img src="/assets/slides/2026-06-12-ranking-reasoning/figures/main_rank_alignment_scatter.webp" alt="Scatter plots comparing greedy ranks with stochastic sampling ranks across four benchmarks" />
    <figcaption>Model-level ranks under greedy decoding versus stochastic sampling at \(N=80\).</figcaption>
  </figure>
</div>

Note:
  The scatter plot gives an operational diagnostic.
  A prior is not free information; it changes the implied target unless the auxiliary policy is aligned with the sampled policy.

---

<div class="conf-kicker">Robustness</div>
<h2>The low-budget conclusions persist across model-pool bootstraps</h2>

<div class="conf-grid three">
  <div class="conf-card">
    <div class="conf-num">1000</div>
    <h3>bootstrap subsets</h3>
    <p>Repeated for model-pool sizes 5, 10, and 15 on each benchmark.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">5-15</div>
    <h3>pool sizes</h3>
    <p>Larger pools mostly reduce between-subset variance rather than changing the winner.</p>
  </div>
  <div class="conf-card">
    <div class="conf-num">29-30</div>
    <h3>tie-rich winners</h3>
    <p>On HMMT and Combined, many methods share the best gold-standard agreement.</p>
  </div>
</div>

<table class="conf-table compact" style="margin-top: 24px;">
  <thead>
    <tr><th>Benchmark</th><th>Best representative</th><th>Pool 5</th><th>Pool 10</th><th>Pool 15</th></tr>
  </thead>
  <tbody>
    <tr><td>AIME'24</td><td class="conf-method-cell">\( \mathrm{Bayes}_{\mathbf{R}_0}@1 \)</td><td>0.769 &plusmn; 0.209</td><td>0.776 &plusmn; 0.107</td><td>0.780 &plusmn; 0.057</td></tr>
    <tr><td>AIME'25</td><td class="conf-method-cell">\( \mathrm{Bayes}_{\mathbf{R}_0}@1 \)</td><td>0.802 &plusmn; 0.144</td><td>0.797 &plusmn; 0.071</td><td>0.798 &plusmn; 0.038</td></tr>
    <tr><td>HMMT'25</td><td class="conf-method-cell">\( \mathrm{Bayes}_{\mathcal{U}}@1 \)</td><td>0.788 &plusmn; 0.114</td><td>0.789 &plusmn; 0.059</td><td>0.790 &plusmn; 0.033</td></tr>
    <tr><td>BrUMO'25</td><td class="conf-method-cell">\( \mathrm{Bayes}_{\mathbf{R}_0}@1 \)</td><td>0.854 &plusmn; 0.136</td><td>0.856 &plusmn; 0.062</td><td>0.858 &plusmn; 0.032</td></tr>
    <tr><td>Combined</td><td class="conf-method-cell">\( \mathrm{Bayes}_{\mathcal{U}}@1 \)</td><td>0.863 &plusmn; 0.084</td><td>0.866 &plusmn; 0.042</td><td>0.864 &plusmn; 0.023</td></tr>
  </tbody>
</table>

Note:
  The recommendations are not an artifact of one fixed model pool. Increasing the pool mainly narrows the uncertainty across subsets.

---

<div class="conf-kicker">Beyond binary correctness</div>
<h2>Auxiliary signals can stabilize a ranking while changing its meaning</h2>

<div class="conf-figure-grid">
  <figure class="conf-figure short">
    <img src="/assets/slides/2026-06-12-ranking-reasoning/figures/main_categorical_tradeoff.webp" alt="Scatter plot showing the tradeoff between gold-standard agreement and self-consistency for categorical ranking schemes" />
    <figcaption>Representative categorical schemes at \(N=1\) on the Combined benchmark.</figcaption>
  </figure>
  <div class="conf-stack">
    <table class="conf-table compact">
      <thead>
        <tr><th>Scheme</th><th>\(\tau_{GS}\)</th><th>\(\tau_{Self}\)</th></tr>
      </thead>
      <tbody>
        <tr><td>Conservative</td><td class="conf-win">0.856</td><td>0.861</td></tr>
        <tr><td>Format-aware</td><td>0.849</td><td>0.881</td></tr>
        <tr><td>OOD-robust</td><td>0.840</td><td>0.892</td></tr>
        <tr><td>Verifier-only</td><td class="conf-loss">0.824</td><td class="conf-win">0.897</td></tr>
      </tbody>
    </table>
    <div class="conf-callout rose">
      Verifier-only is the most self-consistent, but it is farthest from the correctness-based gold standard among the representative schemes.
    </div>
    <ul class="conf-list conf-fit-list">
      <li>Report the category mapping.</li>
      <li>Report the utility weights.</li>
      <li>Do not interpret higher self-consistency as accuracy fidelity.</li>
    </ul>
  </div>
</div>

Note:
  This result generalizes the target-choice warning.
  Richer signals can be useful, but once we use them, the ranking is no longer pure correctness ranking unless we define that target explicitly.

---

<div class="conf-scorio-header">
  <div class="conf-scorio-logo-panel">
    <img src="/assets/slides/2026-06-12-ranking-reasoning/logo/scorio.svg" alt="Scorio" />
  </div>
  <h2>Open-source scoring and ranking for repeated-trial evaluation</h2>
</div>

<div class="conf-scorio-layout">
  <div class="conf-scorio-panel">
    <pre class="conf-code conf-scorio-code"><span class="kw">import</span> numpy <span class="kw">as</span> np
<span class="kw">from</span> scorio <span class="kw">import</span> eval, rank
R = np.<span class="fn">array</span>([[0, 1, 1, 0, 1],
              [1, 1, 0, 1, 1]])
w = np.<span class="fn">array</span>([0.0, 1.0])
mu, sigma = eval.<span class="fn">bayes</span>(R, w)
pass2 = eval.<span class="fn">pass_at_k</span>(R, <span class="arg">k</span>=2)
S = np.random.<span class="fn">randint</span>(0, 2, (3, 5, 4))
r_bayes = rank.<span class="fn">bayes</span>(S, <span class="arg">w</span>=w)
r, s = rank.<span class="fn">avg</span>(S, <span class="arg">return_scores</span>=True)</pre>
    <div class="conf-scorio-note">
      Score an outcome matrix with uncertainty, or rank models from a response tensor \(R \in \{0,1\}^{L \times M \times N}\).
    </div>
    <div class="conf-scorio-use" aria-label="Use cases">
      <span class="conf-scorio-app">Evaluation</span>
      <span class="conf-scorio-app">Ranking</span>
      <span class="conf-scorio-app">RL feedback loops</span>
    </div>
  </div>
  <div class="conf-api-board">
    <h3>Public API map <span>scorio.eval / scorio.rank / scorio.utils</span></h3>
    <div class="conf-api-group">
      <div class="conf-api-head"><code>scorio.eval</code><span>metrics and uncertainty</span></div>
      <div class="conf-api-columns">
        <p class="conf-api-line"><strong>Bayes/avg</strong> <code>bayes</code><code>bayes_ci</code><code>avg</code><code>avg_ci</code></p>
        <p class="conf-api-line"><strong>Pass/AUC/majority/max</strong> <code>pass_at_k</code><code>pass_hat_k</code><code>pass_at_k_ci</code><code>pass_hat_k_ci</code><code>auc_at_k</code><code>auc_at_k_ci</code><code>maj_at_k</code><code>maj_at_k_ci</code><code>max_at_k</code><code>max_at_k_ci</code></p>
        <p class="conf-api-line"><strong>Generalized pass</strong> <code>g_pass_at_k</code><code>g_pass_at_k_tau</code><code>mg_pass_at_k</code><code>g_pass_at_k_ci</code><code>g_pass_at_k_tau_ci</code><code>mg_pass_at_k_ci</code></p>
        <p class="conf-api-line"><strong>GeoSpectrum</strong> <code>geom_at_k</code><code>geom_at_k_ci</code><code>geom_ds_at_k</code><code>geom_ds_at_k_ci</code><code>geo_spectrum_at_k</code><code>geo_spectrum_at_k_ci</code><code>geo_spectrum_star_at_k</code><code>geo_spectrum_star_at_k_ci</code><code>threshold_spectrum_at_k</code><code>threshold_spectrum_at_k_ci</code></p>
      </div>
    </div>
    <div class="conf-api-group">
      <div class="conf-api-head"><code>scorio.rank</code><span>model ranking families</span></div>
      <div class="conf-api-columns">
        <p class="conf-api-line"><strong>Priors</strong> <code>Prior</code><code>EmpiricalPrior</code><code>GaussianPrior</code><code>LaplacePrior</code><code>CauchyPrior</code><code>UniformPrior</code><code>CustomPrior</code></p>
        <p class="conf-api-line"><strong>Eval rankers</strong> <code>avg</code><code>bayes</code><code>pass_at_k</code><code>pass_hat_k</code><code>g_pass_at_k_tau</code><code>mg_pass_at_k</code></p>
        <p class="conf-api-line"><strong>Point/graph/structure</strong> <code>inverse_difficulty</code><code>rank_centrality</code><code>hodge_rank</code><code>serial_rank</code><code>pagerank</code><code>spectral</code><code>alpharank</code><code>nash</code></p>
        <p class="conf-api-line"><strong>Pairwise/probabilistic</strong> <code>elo</code><code>trueskill</code><code>glicko</code><code>bradley_terry</code><code>bradley_terry_map</code><code>bradley_terry_davidson</code><code>bradley_terry_davidson_map</code><code>rao_kupper</code><code>rao_kupper_map</code></p>
        <p class="conf-api-line"><strong>Bayesian/IRT</strong> <code>thompson</code><code>bayesian_mcmc</code><code>rasch</code><code>rasch_map</code><code>rasch_mml</code><code>rasch_mml_credible</code><code>rasch_2pl</code><code>rasch_2pl_map</code><code>rasch_3pl</code><code>rasch_3pl_map</code><code>dynamic_irt</code></p>
        <p class="conf-api-line"><strong>Voting/Luce</strong> <code>borda</code><code>copeland</code><code>win_rate</code><code>minimax</code><code>schulze</code><code>ranked_pairs</code><code>kemeny_young</code><code>nanson</code><code>baldwin</code><code>majority_judgment</code><code>plackett_luce</code><code>plackett_luce_map</code><code>davidson_luce</code><code>davidson_luce_map</code><code>bradley_terry_luce</code><code>bradley_terry_luce_map</code></p>
      </div>
    </div>
    <div class="conf-api-group">
      <div class="conf-api-head"><code>scorio.utils</code><span>ranking helpers</span></div>
      <p class="conf-api-line"><code>rank_scores</code><code>compare_rankings</code><code>ranking_hash</code><code>unhash_ranking</code><code>lehmer_hash</code><code>lehmer_unhash</code></p>
    </div>
  </div>
</div>

Note:
  Scorio is the reusable artifact.
  It makes the paper's comparison protocol available as code rather than just a set of static tables.

---

<div class="conf-takeaway-header">
  <img class="conf-takeaway-icon" src="/assets/slides/2026-06-12-ranking-reasoning/takeaway.webp" alt="" aria-hidden="true" />
  <div>
    <h2 class="conf-takeaway-title">Takeaways</h2>
    <div class="conf-takeaway-subtitle">Ranking needs a reporting discipline</div>
  </div>
</div>

<div class="conf-takeaway-grid">
  <div class="conf-card conf-takeaway-card">
    <div class="conf-takeaway-num">1</div>
    <h3>Name the target</h3>
    <p>Accuracy agreement, self-consistency, and task utility are different rankings.</p>
  </div>
  <div class="conf-card conf-takeaway-card">
    <div class="conf-takeaway-num">2</div>
    <h3>Report stability</h3>
    <p>Low-budget ranks need \( \tau_b \), uncertainty, and convergence with \(N\).</p>
  </div>
  <div class="conf-card conf-takeaway-card">
    <div class="conf-takeaway-num">3</div>
    <h3>Use BayesU first</h3>
    <p>\( \mathrm{Bayes}_{\mathcal{U}}@N \) is transparent; priors and auxiliary signals need audits.</p>
  </div>
</div>

Note:
  End the technical story with an actionable reporting rule.
  The recommendation is Bayes_U as a strong default, not because every method is wrong, but because the target is transparent and uncertainty-aware.

---

<div class="conf-closing-layout">
<div class="conf-closing-main">
<h1 class="conf-closing-title">Thank you! Question?</h1>
<div class="conf-subtitle">Ranking Reasoning LLMs under Test-Time Scaling</div>
<div class="conf-qr-grid" aria-label="Project links">
<div class="conf-qr-item">
<img src="/assets/slides/2026-06-12-ranking-reasoning/qr_codes/00_poster.svg" alt="QR code for the interactive poster" />
<strong>Poster</strong>
</div>
<div class="conf-qr-item">
<img src="/assets/slides/2026-06-12-ranking-reasoning/qr_codes/01_github.svg" alt="QR code for the GitHub repository" />
<strong>GitHub</strong>
</div>
<div class="conf-qr-item">
<img src="/assets/slides/2026-06-12-ranking-reasoning/qr_codes/02_arxiv.svg" alt="QR code for the arXiv paper" />
<strong>arXiv</strong>
</div>
<div class="conf-qr-item">
<img src="/assets/slides/2026-06-12-ranking-reasoning/qr_codes/03_doc.svg" alt="QR code for the Scorio documentation" />
<strong>Docs</strong>
</div>
</div>
<div class="conf-contact-line">Please feel free to contact <span class="conf-contact-email">mohsen.hariri@case.edu</span> with any questions or feedback, especially if you'd like to collaborate</div>
</div>
<div class="conf-ack-stack" aria-label="Acknowledgments">
<div class="conf-ack-item conf-ack-nsf">
<img src="/assets/slides/2026-06-12-ranking-reasoning/nsf.svg" alt="National Science Foundation" />
<div>
<h3>NSF Awards</h3>
<p>2117439, 2112606, 2320952</p>
</div>
</div>
<div class="conf-ack-item conf-ack-cwru">
<img src="/assets/slides/2026-06-12-ranking-reasoning/cwru_hpc_logo.webp" alt="Case Western Reserve University" />
<div>
<h3>High Performance Computing (HPC)</h3>
<p>Center at CWRU</p>
</div>
</div>
</div>
</div>

Note:
  Close with links to the poster, code, paper, and package.
  Acknowledge NSF support and the High Performance Computing Center at CWRU.

Navigation