Scorio for Computer Science (Slides)

/* ---------- base ---------- */

.reveal .slides { text-align: left; }
  .reveal .slides section { box-sizing: border-box; }

.reveal h1, .reveal h2, .reveal h3 {
    text-align: left;
    letter-spacing: 0;
    color: var(--pk-ink);
    text-transform: none;
  }

.reveal h1 { font-size: 66px; line-height: 1.06; }
  .reveal h2 { font-size: 48px; line-height: 1.1; margin: 0 0 12px 0; }
  .reveal h3 { font-size: 25px; line-height: 1.16; margin: 0 0 9px 0; }
  .reveal p, .reveal li { line-height: 1.34; }
  .reveal strong { font-weight: 750; }
  .reveal code { font-size: 0.9em; }

.reveal .pk-kicker {
    font-size: 18px;
    font-weight: 800;
    letter-spacing: 0.09em;
    text-transform: uppercase;
    color: var(--pk-faint);
    margin-bottom: 7px;
  }

.reveal .pk-note {
    max-width: 1330px;
    margin: 0 0 16px 0;
    color: var(--pk-muted);
    font-size: 24px;
    line-height: 1.34;
  }

.reveal .pk-muted { color: var(--pk-muted); }
  .reveal .pk-center { text-align: center; }

/* ---------- grids and cards ---------- */

.reveal .pk-grid { display: grid; gap: 20px; margin-top: 16px; }
  .reveal .pk-grid.two { grid-template-columns: repeat(2, minmax(0, 1fr)); }
  .reveal .pk-grid.three { grid-template-columns: repeat(3, minmax(0, 1fr)); }
  .reveal .pk-grid.four { grid-template-columns: repeat(4, minmax(0, 1fr)); }
  .reveal .pk-grid.stretch { align-items: stretch; }

.reveal .pk-card {
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: var(--pk-card);
    padding: 18px 20px;
    box-shadow: 0 10px 26px rgba(15, 23, 42, 0.06);
  }
  .reveal .pk-card.blue { border-top: 4px solid var(--pk-blue); }
  .reveal .pk-card.green { border-top: 4px solid var(--pk-green); }
  .reveal .pk-card.orange { border-top: 4px solid var(--pk-orange); }
  .reveal .pk-card.purple { border-top: 4px solid var(--pk-purple); }
  .reveal .pk-card.rose { border-top: 4px solid var(--pk-rose); }

.reveal .pk-card p { margin: 0; color: var(--pk-muted); font-size: 22px; }
  .reveal .pk-card p + p { margin-top: 9px; }
  .reveal .pk-card ul {
    margin: 9px 0 0 0;
    padding-left: 22px;
    color: var(--pk-muted);
    font-size: 22px;
  }
  .reveal .pk-card li { margin: 5px 0; }

/* ---------- callouts and chips ---------- */

.reveal .pk-callout {
    border-left: 5px solid var(--pk-blue);
    border-radius: 8px;
    background: rgba(var(--pk-blue-rgb), 0.075);
    padding: 15px 19px;
    margin-top: 16px;
    color: var(--pk-ink);
    font-size: 24px;
    line-height: 1.32;
  }
  .reveal .pk-callout.green { border-left-color: var(--pk-green); background: rgba(var(--pk-green-rgb), 0.08); }
  .reveal .pk-callout.orange { border-left-color: var(--pk-orange); background: rgba(var(--pk-orange-rgb), 0.09); }
  .reveal .pk-callout.purple { border-left-color: var(--pk-purple); background: rgba(var(--pk-purple-rgb), 0.08); }
  .reveal .pk-callout.rose { border-left-color: var(--pk-rose); background: rgba(var(--pk-rose-rgb), 0.08); }

.reveal .pk-chip-row { display: flex; flex-wrap: wrap; gap: 9px; }
  .reveal .pk-chip {
    display: inline-flex;
    align-items: center;
    border-radius: 999px;
    padding: 6px 14px;
    border: 1px solid var(--pk-border);
    background: rgba(255, 255, 255, 0.74);
    color: var(--pk-ink);
    font-size: 18px;
    font-weight: 750;
    letter-spacing: 0.03em;
  }
  .reveal .pk-chip.blue { color: var(--pk-blue); border-color: rgba(var(--pk-blue-rgb), 0.3); background: rgba(var(--pk-blue-rgb), 0.09); }
  .reveal .pk-chip.green { color: var(--pk-green); border-color: rgba(var(--pk-green-rgb), 0.3); background: rgba(var(--pk-green-rgb), 0.09); }
  .reveal .pk-chip.orange { color: var(--pk-orange); border-color: rgba(var(--pk-orange-rgb), 0.3); background: rgba(var(--pk-orange-rgb), 0.09); }
  .reveal .pk-chip.purple { color: var(--pk-purple); border-color: rgba(var(--pk-purple-rgb), 0.3); background: rgba(var(--pk-purple-rgb), 0.09); }

/* ---------- equations ---------- */

.reveal .pk-equation {
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.68);
    padding: 14px 18px;
    margin-top: 14px;
    font-size: 31px;
    text-align: center;
  }
  .reveal .pk-equation.sm { font-size: 25px; }
  .reveal .pk-equation.xs { font-size: 21px; }
  .reveal .pk-equation.plain { border: none; background: none; padding: 4px 0; }
  .reveal .pk-equation .katex { max-width: 100%; }
  .reveal .pk-eq-label {
    margin-top: 8px;
    color: var(--pk-faint);
    font-size: 18px;
    text-align: center;
  }

/* ---------- figures ---------- */

/* The slide template injects `.reveal .slides section img { border-radius; box-shadow;
     max-width: 100%; height: auto }` at specificity (0,2,2). Every image rule below has to
     outrank that, hence the `.reveal .slides section` prefix; a bare `.pk-figure img`
     (0,2,1) silently loses and the figure renders at its natural height with a stray frame. */
  .reveal .pk-figure {
    display: flex;
    flex-direction: column;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.9);
    padding: 16px;
  }
  .reveal .slides section .pk-figure img {
    display: block;
    flex: 1 1 auto;
    min-width: 0;
    min-height: 0;
    width: 100%;
    height: 100%;
    max-width: 100%;
    margin: 0;
    object-fit: contain;
    border: none;
    border-radius: 0;
    box-shadow: none;
    background: none;
  }
  .reveal .pk-figure figcaption {
    flex: 0 0 auto;
    margin-top: 10px;
    color: var(--pk-muted);
    font-size: 18px;
    line-height: 1.26;
  }
  .reveal .pk-fig-xs { height: 370px; }
  .reveal .pk-fig-short { height: 470px; }
  .reveal .pk-fig-mid { height: 560px; }
  .reveal .pk-fig-tall { height: 650px; }

.reveal .pk-split {
    display: grid;
    grid-template-columns: minmax(0, 1fr) minmax(470px, 0.62fr);
    gap: 22px;
    align-items: stretch;
    margin-top: 14px;
  }
  .reveal .pk-split.reverse { grid-template-columns: minmax(470px, 0.62fr) minmax(0, 1fr); }
  .reveal .pk-split.even { grid-template-columns: repeat(2, minmax(0, 1fr)); }

.reveal .pk-stack { display: grid; gap: 16px; align-content: start; }

/* Grid/flex children are min-width:auto by default, so a wide table refuses to shrink
     and overflows its column; inside .pk-board that overflow is silently clipped. */
  .reveal .pk-split > *,
  .reveal .pk-grid > *,
  .reveal .pk-board-main > *,
  .reveal .pk-stack > * { min-width: 0; }

/* ---------- tables ---------- */

.reveal .pk-table {
    width: 100%;
    border-collapse: collapse;
    margin-top: 14px;
    font-size: 23px;
  }
  .reveal .pk-table.compact { font-size: 20px; }
  .reveal .pk-table th,
  .reveal .pk-table td {
    border-bottom: 1px solid rgba(15, 23, 42, 0.12);
    padding: 9px 11px;
    vertical-align: top;
    text-align: left;
  }
  .reveal .pk-table th {
    color: var(--pk-ink);
    font-weight: 800;
    font-size: 0.86em;
    letter-spacing: 0.03em;
    text-transform: uppercase;
  }
  .reveal .pk-table td { color: var(--pk-muted); }
  .reveal .pk-table td.num, .reveal .pk-table th.num { text-align: right; font-variant-numeric: tabular-nums; }
  .reveal .pk-table .pk-win { color: var(--pk-green); font-weight: 800; }
  .reveal .pk-table .pk-bad { color: var(--pk-orange); font-weight: 800; }

/* ---------- numbered roadmap ---------- */

.reveal .pk-roadmap { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 18px; margin-top: 18px; }
  .reveal .pk-roadmap.seven { grid-template-columns: repeat(4, minmax(0, 1fr)); }

.reveal .pk-step {
    position: relative;
    padding: 17px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.8);
    box-shadow: 0 12px 26px rgba(15, 23, 42, 0.07);
  }
  .reveal .pk-step::before {
    content: attr(data-step);
    display: grid;
    place-items: center;
    width: 40px;
    height: 40px;
    margin-bottom: 12px;
    border-radius: 8px;
    background: rgba(15, 23, 42, 0.9);
    color: #fff;
    font-size: 19px;
    font-weight: 900;
  }
  .reveal .pk-step.blue::before { background: var(--pk-blue); }
  .reveal .pk-step.green::before { background: var(--pk-green); }
  .reveal .pk-step.orange::before { background: var(--pk-orange); }
  .reveal .pk-step.purple::before { background: var(--pk-purple); }
  .reveal .pk-step.rose::before { background: var(--pk-rose); }
  .reveal .pk-step h3 { font-size: 23px; margin: 0 0 8px 0; }
  .reveal .pk-step p { margin: 0; color: var(--pk-muted); font-size: 19px; line-height: 1.28; }

/* ---------- rail + main board (grid paper) ---------- */

.reveal .pk-board {
    display: grid;
    grid-template-columns: 300px minmax(0, 1fr);
    overflow: hidden;
    margin-top: 14px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background:
      linear-gradient(90deg, rgba(15, 23, 42, 0.04) 1px, transparent 1px),
      linear-gradient(0deg, rgba(15, 23, 42, 0.035) 1px, transparent 1px),
      linear-gradient(135deg, rgba(var(--tone-rgb), 0.09), rgba(255, 255, 255, 0.8) 44%, rgba(15, 23, 42, 0.035));
    background-size: 30px 30px, 30px 30px, auto;
    box-shadow: 0 16px 34px rgba(15, 23, 42, 0.09);
    --tone-rgb: var(--pk-blue-rgb);
    --tone: var(--pk-blue);
  }
  .reveal .pk-board.green { --tone-rgb: var(--pk-green-rgb); --tone: var(--pk-green); }
  .reveal .pk-board.orange { --tone-rgb: var(--pk-orange-rgb); --tone: var(--pk-orange); }
  .reveal .pk-board.purple { --tone-rgb: var(--pk-purple-rgb); --tone: var(--pk-purple); }
  .reveal .pk-board.rose { --tone-rgb: var(--pk-rose-rgb); --tone: var(--pk-rose); }

.reveal .pk-rail {
    display: flex;
    flex-direction: column;
    gap: 13px;
    padding: 20px;
    border-right: 1px solid rgba(15, 23, 42, 0.12);
    background: linear-gradient(180deg, rgba(var(--tone-rgb), 0.15), rgba(255, 255, 255, 0.6));
  }
  .reveal .pk-rail-badge {
    display: grid;
    place-items: center;
    width: 58px;
    height: 58px;
    border: 1px solid rgba(var(--tone-rgb), 0.28);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.86);
    color: var(--tone);
    font-size: 26px;
    font-weight: 900;
  }
  .reveal .pk-rail h3 { margin: 0; color: var(--tone); font-size: 26px; line-height: 1.1; }
  .reveal .pk-rail p { margin: 0; color: var(--pk-muted); font-size: 19px; line-height: 1.3; }
  .reveal .pk-rail-foot { margin-top: auto; display: grid; gap: 7px; }
  .reveal .pk-rail-fact {
    padding: 10px 12px;
    border: 1px solid rgba(15, 23, 42, 0.1);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.72);
    font-size: 18px;
    line-height: 1.26;
    color: var(--pk-muted);
  }
  .reveal .pk-rail-fact b { display: block; color: var(--tone); font-size: 25px; font-variant-numeric: tabular-nums; }

.reveal .pk-board-main { padding: 18px; display: grid; gap: 14px; align-content: start; }

/* ---------- good / bad compare ---------- */

.reveal .pk-compare { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 20px; margin-top: 16px; }
  .reveal .pk-world {
    padding: 18px 20px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.8);
    box-shadow: 0 14px 30px rgba(15, 23, 42, 0.07);
  }
  .reveal .pk-world.bad { border-top: 5px solid var(--pk-orange); }
  .reveal .pk-world.good { border-top: 5px solid var(--pk-green); }
  .reveal .pk-world ul { margin: 12px 0 0 0; padding-left: 22px; color: var(--pk-muted); font-size: 22px; }
  .reveal .pk-world li { margin: 5px 0; }

/* ---------- response-bank matrix ---------- */

.reveal .pk-mat-row { display: flex; gap: 28px; align-items: flex-start; justify-content: center; }
  .reveal .pk-mat-unit { display: grid; gap: 9px; justify-items: center; }
  .reveal .pk-mat-title { color: var(--pk-ink); font-size: 20px; font-weight: 800; }
  .reveal .pk-mat { display: grid; gap: 5px; }
  .reveal .pk-mat span {
    width: 38px;
    height: 38px;
    border-radius: 4px;
    border: 1px solid rgba(15, 23, 42, 0.14);
    background: rgba(15, 23, 42, 0.05);
  }
  .reveal .pk-mat span.on { background: rgba(var(--pk-green-rgb), 0.78); border-color: rgba(var(--pk-green-rgb), 0.8); }
  .reveal .pk-mat.alt span.on { background: rgba(var(--pk-blue-rgb), 0.78); border-color: rgba(var(--pk-blue-rgb), 0.8); }
  .reveal .pk-mat-cap { color: var(--pk-faint); font-size: 18px; font-variant-numeric: tabular-nums; }

/* ---------- stat row ---------- */

.reveal .pk-stat-row { display: grid; grid-auto-flow: column; gap: 18px; margin-top: 16px; }
  .reveal .pk-stat {
    padding: 15px 17px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.8);
    text-align: center;
  }
  .reveal .pk-stat b {
    display: block;
    color: var(--pk-ink);
    font-size: 42px;
    font-weight: 850;
    line-height: 1.05;
    font-variant-numeric: tabular-nums;
  }
  .reveal .pk-stat.blue b { color: var(--pk-blue); }
  .reveal .pk-stat.green b { color: var(--pk-green); }
  .reveal .pk-stat.orange b { color: var(--pk-orange); }
  .reveal .pk-stat.rose b { color: var(--pk-rose); }
  .reveal .pk-stat span { display: block; margin-top: 6px; color: var(--pk-muted); font-size: 18px; line-height: 1.26; }

/* ---------- sign path ---------- */

.reveal .pk-signs { display: flex; gap: 11px; justify-content: center; margin-top: 20px; }
  .reveal .pk-sign {
    display: grid;
    place-items: center;
    width: 92px;
    padding: 13px 0 9px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.8);
  }
  .reveal .pk-sign b { font-size: 40px; font-weight: 900; line-height: 1; }
  .reveal .pk-sign span { margin-top: 6px; color: var(--pk-faint); font-size: 18px; }
  .reveal .pk-sign.a { border-color: rgba(var(--pk-green-rgb), 0.4); background: rgba(var(--pk-green-rgb), 0.1); }
  .reveal .pk-sign.a b { color: var(--pk-green); }
  .reveal .pk-sign.b { border-color: rgba(var(--pk-orange-rgb), 0.4); background: rgba(var(--pk-orange-rgb), 0.1); }
  .reveal .pk-sign.b b { color: var(--pk-orange); }
  .reveal .pk-sign.flip { box-shadow: 0 0 0 3px rgba(15, 23, 42, 0.16); }

/* ---------- title slide ---------- */

.reveal .pk-cover { margin-top: 230px; text-align: center; }
  .reveal .pk-cover h1 { margin: 0; text-align: center; }
  .reveal .pk-cover-meta { margin-top: 34px; color: var(--pk-muted); font-size: 24px; line-height: 1.5; }
  .reveal .pk-cover-meta b { color: var(--pk-ink); font-weight: 800; }

/* ---------- opening story ---------- */

.reveal .pk-ask {
    margin: 30px 0 0 0;
    padding-left: 24px;
    border-left: 4px solid rgba(15, 23, 42, 0.18);
    font-size: 34px;
    line-height: 1.3;
    color: var(--pk-muted);
  }
  .reveal .pk-ask b { color: var(--pk-ink); font-weight: 800; }
  .reveal .pk-said { margin: 52px 0 0 0; font-size: 58px; font-weight: 800; color: var(--pk-ink); }
  .reveal .pk-aside { margin: 16px 0 0 42px; font-size: 29px; font-style: italic; line-height: 1.5; color: var(--pk-faint); }
  .reveal .pk-aside span { display: block; }
  .reveal .pk-then { margin: 52px 0 0 0; font-size: 34px; line-height: 1.3; color: var(--pk-ink); }
  .reveal .pk-then b { font-weight: 800; }
  .reveal .pk-line { margin: 30px 0 0 0; font-size: 34px; line-height: 1.3; color: var(--pk-ink); }
  .reveal .pk-line b { font-weight: 800; }
  .reveal .pk-table.lg { font-size: 30px; }
  .reveal .pk-table.lg th, .reveal .pk-table.lg td { padding: 16px 14px; }

/* ---------- metric progression ---------- */

.reveal .pk-chain {
    display: grid;
    grid-template-columns: repeat(5, minmax(0, 1fr));
    gap: 26px;
    margin-top: 38px;
    align-items: start;
  }
  .reveal .pk-chain-step {
    position: relative;
    padding: 22px 20px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.72);
  }
  .reveal .pk-chain-step::after {
    content: "\203A";
    position: absolute;
    top: 24px;
    right: -20px;
    width: 14px;
    color: var(--pk-faint);
    font-size: 30px;
    line-height: 1;
    text-align: center;
  }
  .reveal .pk-chain-step:last-child::after { content: none; }
  .reveal .pk-chain-step h3 { font-size: 29px; margin: 0 0 16px 0; }
  .reveal .pk-chain-step ul { margin: 0; padding: 0; list-style: none; }
  .reveal .pk-chain-step li {
    position: relative;
    margin: 0 0 14px 0;
    padding-left: 22px;
    font-size: 23px;
    line-height: 1.28;
    color: var(--pk-muted);
  }
  .reveal .pk-chain-step li:last-child { margin-bottom: 0; }
  .reveal .pk-chain-step li::before {
    content: "";
    position: absolute;
    left: 0;
    top: 10px;
    width: 11px;
    height: 11px;
    border-radius: 50%;
    background: var(--dot, rgba(15, 23, 42, 0.28));
  }
  .reveal .pk-chain-step li.unc { --dot: var(--pk-blue); }
  .reveal .pk-chain-step li.cov { --dot: var(--pk-green); }
  .reveal .pk-chain-step li.cat { --dot: var(--pk-purple); }
  .reveal .pk-chain-step li.shape { --dot: var(--pk-orange); }
  .reveal .pk-chain-step li.prior { --dot: var(--pk-rose); }
  .reveal .pk-chain-step.scalar { background: rgba(15, 23, 42, 0.035); }
  .reveal .pk-chain-step.scalar h3 { color: var(--pk-faint); }
  .reveal .pk-chain-step.here {
    border-color: rgba(var(--pk-orange-rgb), 0.45);
    background: rgba(var(--pk-orange-rgb), 0.06);
    box-shadow: 0 14px 30px rgba(15, 23, 42, 0.09);
  }
  .reveal .pk-chain-step.here h3 { color: var(--pk-orange); }
  .reveal .pk-chain-step.open { border-style: dashed; background: none; }
  .reveal .pk-chain-step.open h3 { color: var(--pk-faint); }
  .reveal .pk-chain-step.open p { margin: 0; font-size: 23px; font-style: italic; color: var(--pk-faint); }

/* ---------- api slides ---------- */

.reveal .pk-code {
    display: inline-block;
    margin-top: 26px;
    padding: 15px 21px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(15, 23, 42, 0.045);
    font-family: var(--r-code-font);
    font-size: 24px;
    line-height: 1.55;
    color: var(--pk-ink);
    text-align: left;
  }
  .reveal .pk-code .kw { color: var(--pk-purple); }
  .reveal .pk-code .cm { color: var(--pk-faint); }
  .reveal .pk-table code { color: var(--pk-ink); font-weight: 700; }
  .reveal .pk-table a { color: var(--pk-blue); text-decoration: none; border-bottom: 1px solid rgba(var(--pk-blue-rgb), 0.35); }
  .reveal .pk-table a code { color: inherit; }
  .reveal .pk-foot { margin-top: 20px; font-size: 20px; line-height: 1.4; color: var(--pk-faint); }
  .reveal .pk-foot code { color: var(--pk-muted); font-weight: 700; }
  .reveal .pk-meth {
    margin: 0;
    font-family: var(--r-code-font);
    font-size: 17px;
    line-height: 1.55;
    color: var(--pk-muted);
    word-break: break-word;
  }

/* ---------- test-time scaling ---------- */

.reveal .pk-dim { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 22px; margin-top: 28px; }
  .reveal .pk-dim .pk-card { padding: 22px 22px 24px; }
  .reveal .pk-dim h3 { font-size: 27px; margin: 0 0 10px 0; }
  .reveal .pk-dim p { font-size: 22px; line-height: 1.3; }
  .reveal .pk-dim .who { margin-top: 12px; font-family: var(--r-code-font); font-size: 19px; color: var(--pk-faint); }
  .reveal .pk-tree {
    margin-top: 34px;
    font-family: var(--r-code-font);
    font-size: 27px;
    line-height: 1.62;
    white-space: pre;
    color: var(--pk-faint);
    overflow-x: auto;
  }
  .reveal .pk-tree .mod { color: var(--pk-ink); font-weight: 700; }
  .reveal .pk-tree .fn { color: var(--pk-blue); }
  .reveal .pk-tree .desc { color: var(--pk-muted); }
  .reveal .pk-tree .soon { color: var(--pk-orange); }
  .reveal .pk-areas { margin: 14px 0 0 0; font-size: 23px; line-height: 1.45; color: var(--pk-faint); }
  .reveal .pk-list {
    display: grid;
    grid-template-columns: max-content minmax(0, 1fr);
    gap: 22px 38px;
    margin-top: 36px;
    align-items: baseline;
  }
  .reveal .pk-list .k { font-family: var(--r-code-font); font-size: 25px; }
  .reveal .pk-list .k a { color: var(--pk-blue); text-decoration: none; border-bottom: 1px solid rgba(var(--pk-blue-rgb), 0.35); }
  .reveal .pk-list .v { font-size: 25px; line-height: 1.36; color: var(--pk-muted); }
  .reveal .pk-list .v b { color: var(--pk-ink); font-weight: 750; }

.reveal .pk-paper { margin-top: 40px; text-align: center; }
  .reveal .slides section .pk-paper img {
    width: 1080px;
    max-width: 82%;
    height: auto;
    margin: 0;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    box-shadow: none;
    background: #fff;
  }

.reveal .pk-logo-strip {
    display: flex;
    align-items: center;
    justify-content: center;
    gap: 52px;
    flex-wrap: wrap;
    margin-top: 46px;
  }
  .reveal .slides section .pk-logo-strip img {
    height: 80px;
    width: auto;
    max-width: 290px;
    margin: 0;
    object-fit: contain;
    border: none;
    border-radius: 0;
    box-shadow: none;
    background: none;
  }

/* ---------- takeaway ---------- */

.reveal .pk-takeaway { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 22px; margin-top: 22px; }
  .reveal .pk-takeaway .pk-card { padding: 26px; }
  .reveal .pk-take-num {
    display: grid;
    place-items: center;
    width: 48px;
    height: 48px;
    margin-bottom: 15px;
    border-radius: 8px;
    background: rgba(15, 23, 42, 0.9);
    color: #fff;
    font-size: 23px;
    font-weight: 900;
  }
  .reveal .pk-takeaway .pk-card p { font-size: 23px; }
  .reveal .pk-takeaway h3 { font-size: 28px; }

.reveal .pk-thanks { margin-top: 26px; color: var(--pk-muted); font-size: 22px; line-height: 1.45; }
  .reveal .pk-thanks b { color: var(--pk-ink); }
  /* ---------- part divider ---------- */

.reveal .pk-divider {
    display: grid;
    grid-template-columns: 300px minmax(0, 1fr);
    gap: 48px;
    align-items: center;
  }
  .reveal .pk-divider { min-height: 760px; }

.reveal .pk-part-mark {
    display: grid;
    justify-items: start;
    padding: 28px 32px;
    border-radius: 8px;
    border: 1px solid rgba(15, 23, 42, 0.12);
    background: rgba(255, 255, 255, 0.74);
    box-shadow: 0 20px 44px rgba(15, 23, 42, 0.1);
  }
  .reveal .pk-part-label {
    font-size: 21px;
    font-weight: 850;
    letter-spacing: 0.18em;
    text-transform: uppercase;
    color: var(--pk-faint);
  }
  .reveal .pk-part-num {
    font-size: 78px;
    font-weight: 900;
    line-height: 0.94;
    letter-spacing: -0.015em;
    color: var(--pk-ink);
  }
  .reveal .pk-part-rule {
    width: 76px;
    height: 6px;
    margin-top: 14px;
    border-radius: 999px;
    background: var(--pk-blue);
  }
  .reveal .pk-part-mark.two .pk-part-rule { background: var(--pk-purple); }

.reveal .pk-divider-body h1 { font-size: 74px; margin: 0; line-height: 1.03; }
  .reveal .pk-divider-sub {
    margin: 20px 0 0 0;
    max-width: 940px;
    color: var(--pk-muted);
    font-size: 27px;
    line-height: 1.34;
  }

.reveal .pk-divider-map {
    display: grid;
    grid-template-columns: repeat(4, minmax(0, 1fr));
    gap: 11px;
    margin-top: 26px;
  }
  .reveal .pk-divider-map span {
    display: grid;
    place-items: center;
    min-height: 64px;
    padding: 8px 10px;
    border: 1px solid rgba(15, 23, 42, 0.12);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.82);
    color: var(--pk-ink);
    font-size: 16px;
    font-weight: 800;
    letter-spacing: 0.03em;
    text-align: center;
    text-transform: uppercase;
  }
  .reveal .pk-divider-map span.proj {
    border-color: rgba(var(--pk-blue-rgb), 0.28);
    background: rgba(var(--pk-blue-rgb), 0.08);
    color: rgb(3, 105, 161);
  }
  .reveal .pk-divider-map span.aux {
    border-color: rgba(var(--pk-purple-rgb), 0.26);
    background: rgba(var(--pk-purple-rgb), 0.07);
    color: var(--pk-purple);
  }

.reveal .pk-notation-strip {
    display: grid;
    grid-template-columns: repeat(3, minmax(0, 1fr));
    gap: 16px;
    margin-top: 22px;
  }
  .reveal .pk-notation-item {
    padding: 13px 16px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.72);
    text-align: center;
    font-size: 23px;
  }
  .reveal .pk-notation-item small {
    display: block;
    margin-top: 7px;
    color: var(--pk-faint);
    font-size: 16px;
  }

/* ---------- one-run flow (bank to scalar) ---------- */

.reveal .pk-flow {
    display: flex;
    align-items: center;
    justify-content: center;
    gap: 34px;
    margin-top: 16px;
  }
  .reveal .pk-flow-arrow {
    color: rgba(15, 23, 42, 0.34);
    font-size: 54px;
    line-height: 1;
  }
  .reveal .pk-scalar-pair { display: grid; gap: 12px; }
  .reveal .pk-scalar {
    min-width: 232px;
    padding: 14px 22px;
    border: 1px solid var(--pk-border);
    border-radius: 8px;
    background: rgba(255, 255, 255, 0.84);
    box-shadow: 0 12px 28px rgba(15, 23, 42, 0.07);
    text-align: center;
  }
  .reveal .pk-scalar b {
    display: block;
    font-size: 46px;
    font-weight: 850;
    line-height: 1;
    font-variant-numeric: tabular-nums;
    color: var(--pk-blue);
  }
  .reveal .pk-scalar.alt b { color: var(--pk-green); }
  .reveal .pk-scalar span {
    display: block;
    margin-top: 7px;
    color: var(--pk-muted);
    font-size: 17px;
  }

</style>

# How to Deal with a Stochastic Model?

<div class="pk-cover-meta">
    <b>Mohsen Hariri</b><br />
    Case Western Reserve University<br />
    August 24, 2026
  </div>
</div>

Note:
Everything in this talk is about one line in a results table. You sample k times, you check whether any attempt was right, you average over questions, and you print a number. Seven separate things go wrong with that number. The first half shows you each one happening; the second half states each one exactly.

---

## My first experience with a language model

<p class="pk-ask"><b>My mentor:</b> Run Llama 3 on GSM8K in two cases: (1) full-precision KV cache, (2) 4-bit quantized KV cache. Compare the results.</p>

<div class="pk-aside">
  Thinking: 
  <span>what is Llama?</span>
  <span>what is a KV cache?</span>
  <span>what is 4-bit quantization?</span>
</div>

<p class="pk-then">I ran it. <b>The accuracy changed a lot</b>, and the same prompt gave me <b>a different output every run</b>.</p>

Note:
The task was a clean comparison: two cache settings, one benchmark, report the difference. I could not do it, because the number moved between runs. That is where the rest of this talk comes from.

---

## What I learned from it

<p class="pk-line">An LLM is not a model of language. It models a <b>probability distribution over the next token</b>, given the context.</p>

<div class="pk-equation">
  \[ x_{t+1} \sim p_\theta(\cdot \mid x_{1:t}) \]
</div>

<p class="pk-then">Generating language is <b>the model plus a sampling method</b>. So there is not one probabilistic model here. There are <b>two</b>, running in sequence, at every token.</p>

Note:
This is the correction that fixed my confusion. I had been treating the LLM as the whole system. It is only the first half. The weights give you a distribution; the sampler turns that distribution into text. Both are probabilistic, and only one of them is usually written down in the methods section.

---

## Two ways to think about LLMs

<table class="pk-table lg">
  <thead>
    <tr>
      <th style="width: 30%;"></th>
      <th style="width: 35%;">Deterministic?</th>
      <th style="width: 35%;">Identifiable?</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><b>The model</b><br />\( p_\theta(\cdot \mid x) \)</td>
      <td><b>Yes.</b> Same context, same distribution.</td>
      <td><b>No.</b> Permutation, scaling, and rotation symmetries.</td>
    </tr>
    <tr>
      <td><b>The model in action</b><br />generation</td>
      <td><b>No.</b> The sampler injects randomness.</td>
      <td><b>No.</b> It inherits the above, and adds a \( \theta \) / decoding confound.</td>
    </tr>
  </tbody>
</table>

<p class="pk-then">Run an LLM twice on the same prompt and you get two different outputs.</p>

<p class="pk-line">Whatever my research turned out to be about, I had to deal with that first.</p>

Note:
The row that matters is the second one. Almost everything people say about an LLM is a statement about the first row, but everything we measure comes from the second. The gap between those two rows is where the rest of this talk lives.

---

## Evaluation of LLMs

<p class="pk-line">Whatever the research is:</p>

<p class="pk-areas">improving a model, training a new model, fine-tuning, reinforcement learning, KV cache compression, model compression, creating a dataset or benchmark, reasoning, &hellip;</p>

<p class="pk-line">the last link is the same: <b>how to evaluate the model</b>, and <b>how to rank the models</b>.</p>

<div class="pk-chain">
  <div class="pk-chain-step scalar">
    <h3>Avg@N</h3>
    <ul>
      <li>one number</li>
    </ul>
  </div>
  <div class="pk-chain-step scalar">
    <h3>Pass@k</h3>
    <ul>
      <li>one number</li>
      <li>one budget</li>
    </ul>
  </div>
  <div class="pk-chain-step">
    <h3>Bayes@N</h3>
    <ul>
      <li class="unc">a distribution, not a number (&mu;, &sigma;)</li>
      <li class="cat">categorical outcomes</li>
    </ul>
  </div>
  <div class="pk-chain-step">
    <h3>Geom@k</h3>
    <ul>
      <li class="unc">uncertainty</li>
      <li class="cov">coverage</li>
      <li class="cat">categorical outcomes</li>
    </ul>
  </div>
  <div class="pk-chain-step here">
    <h3>TailPass@k</h3>
    <ul>
      <li class="shape">the shape of success</li>
      <li class="unc">uncertainty</li>
      <li class="cov">coverage</li>
      <li class="prior">prior knowledge</li>
      <li class="cat">categorical outcomes</li>
    </ul>
  </div>
</div>

Note:
Read it left to right and watch the dots pile up. Avg@N and Pass@k give you a single number. Bayes@N puts a distribution around it. Geom@k adds coverage. TailPass@k is the one this talk is about: it carries all of it, plus the shape. Same colour means the same property, so you can see what each step actually bought.

---

## Report a distribution, not a number: Bayes@N

<p class="pk-note">Two models, 64 attempts each. One got 40 right, the other 46. On a leaderboard that is 0.63 against 0.72, and 0.72 wins.</p>

<div class="pk-eq-label">The same two results, drawn twice. Top: the scores as a table prints them. Bottom: the posterior over each model's success rate, under a flat prior.</div>

---

## Separate finding it from repeating it: Geom@k

<div class="pk-mat-row" style="gap: 64px;">
  <div class="pk-mat-unit">
    <div class="pk-mat-title" style="color: rgb(5,150,105);">Bank A &mdash; five of eight, every question</div>
    <div class="pk-mat" style="grid-template-columns: repeat(8, 38px);">
      <span class="on"></span><span class="on"></span><span></span><span class="on"></span><span></span><span class="on"></span><span class="on"></span><span></span>
      <span></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span class="on"></span><span></span><span class="on"></span>
      <span class="on"></span><span></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span class="on"></span><span></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span></span><span class="on"></span><span></span><span></span><span class="on"></span>
    </div>
    <div class="pk-mat-cap">20 of 32 correct</div>
  </div>
  <div class="pk-mat-unit">
    <div class="pk-mat-title" style="color: rgb(2,132,199);">Bank B &mdash; two solid questions, two shaky</div>
    <div class="pk-mat alt" style="grid-template-columns: repeat(8, 38px);">
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span>
      <span class="on"></span><span class="on"></span><span></span><span class="on"></span><span></span><span></span><span class="on"></span><span></span>
      <span></span><span class="on"></span><span class="on"></span><span></span><span class="on"></span><span></span><span></span><span class="on"></span>
    </div>
    <div class="pk-mat-cap">24 of 32 correct</div>
  </div>
</div>

<div class="pk-split even">
  <div class="pk-callout orange">Pass@4 calls it a tie. Bank B is the better system by every other reading, and the metric cannot see it.</div>
  <div class="pk-callout green">Geom@k is the discovery score scaled down by how often a hit repeats, so it stays low unless a model does both. Geom@2 recovered the full-budget ranking faster than any other fixed-budget metric we tried.</div>
</div>

Note:
Bank A is spread thin: five of eight on every question, nothing it can be trusted on. Bank B has two questions nailed and two it is guessing at. Draw four attempts and both find something essentially always, so Pass@4 reports a tie. Ask instead for all four to be right and they are seven times apart. Geom@k is what you get when you refuse to pick one of those two questions and multiply them together instead.

---

## The whole curve: TailPass@k

<p class="pk-note">The same two banks. Instead of one threshold, walk all of them: at least one of the four right, at least two, at least three, all four.</p>

<div class="pk-eq-label">Exact values for the two banks at a budget of four attempts. The average height of a curve is plain accuracy: 0.63 for A, 0.75 for B.</div>

---

## Every question has its own curve

<p class="pk-note">Bank B, question by question. Two were solved 8 of 8, two were solved 4 of 8.</p>

<table class="pk-table compact">
  <thead>
    <tr><th style="width: 24%; text-transform: none;">profile</th><th class="num" style="text-transform: none;">t = 1</th><th class="num" style="text-transform: none;">t = 2</th><th class="num" style="text-transform: none;">t = 3</th><th class="num" style="text-transform: none;">t = 4</th><th class="num" style="text-transform: none;">t = 5</th><th class="num" style="text-transform: none;">t = 6</th><th class="num" style="text-transform: none;">t = 7</th><th class="num" style="text-transform: none;">t = 8</th></tr>
  </thead>
  <tbody>
    <tr><td>Q1, Q2 &nbsp; \( S_{i,k} \)</td><td class="num">1.000</td><td class="num">1.000</td><td class="num">0.998</td><td class="num">0.991</td><td class="num">0.971</td><td class="num">0.918</td><td class="num">0.794</td><td class="num">0.529</td></tr>
    <tr><td>Q3, Q4 &nbsp; \( S_{i,k} \)</td><td class="num">0.980</td><td class="num">0.912</td><td class="num">0.782</td><td class="num">0.601</td><td class="num">0.399</td><td class="num">0.218</td><td class="num">0.088</td><td class="num">0.020</td></tr>
    <tr style="background: rgba(15,23,42,0.04);"><td><b>the report</b> &nbsp; \( S_k \)</td><td class="num">0.990</td><td class="num">0.956</td><td class="num">0.890</td><td class="num">0.796</td><td class="num">0.685</td><td class="num">0.568</td><td class="num">0.441</td><td class="num">0.275</td></tr>
  </tbody>
</table>

<div class="pk-eq-label">Each question has its own profile \( S_{i,k} \). What gets printed is their average, \( S_{k,t} = \frac{1}{M}\sum_{i} S_{i,k,t} \).</div>

---

## TailPass@k: Momentum

<p class="pk-note">A scalar out of the curve is a choice of weights over the thresholds.</p>

<div class="pk-equation xs plain">\[ U_\lambda(k) = \sum_{t=1}^{k} \omega^{(\lambda)}_{t,k} S_{k,t}, \qquad \omega^{(\lambda)}_{t,k} = \left(\tfrac{t}{k}\right)^{\lambda} - \left(\tfrac{t-1}{k}\right)^{\lambda} \]</div>

<div class="pk-split even">
<svg viewBox="0 0 700 450" preserveAspectRatio="xMidYMid meet" role="img" aria-label="Three weight profiles over the eight thresholds. At lambda 0.3 the first threshold takes 0.54 of the weight and the rest share the remainder. At lambda 1 all eight weights are equal at 0.125. At lambda 4 the weight climbs steadily to 0.41 on the eighth threshold." style="display:block;width:100%;height:372px;"><text x="0" y="30" font-size="21" font-weight="800" fill="rgb(15,23,42)">&#955; = 0.3</text><text x="82" y="30" font-size="19" fill="rgba(15,23,42,0.6)">almost everything rides on the first success</text><rect x="46.0" y="32.3" width="58.0" height="85.7" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="128.0" y="98.2" width="58.0" height="19.8" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="210.0" y="104.3" width="58.0" height="13.7" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="292.0" y="107.3" width="58.0" height="10.7" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="374.0" y="109.0" width="58.0" height="9.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="456.0" y="110.2" width="58.0" height="7.8" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="538.0" y="111.1" width="58.0" height="6.9" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="620.0" y="111.7" width="58.0" height="6.3" rx="3" fill="rgba(15,23,42,0.62)" /><line x1="38.0" y1="118" x2="686.0" y2="118" stroke="rgba(15,23,42,0.28)" stroke-width="2" /><text x="0" y="164" font-size="21" font-weight="800" fill="rgb(15,23,42)">&#955; = 1</text><text x="82" y="164" font-size="19" fill="rgba(15,23,42,0.6)">equal weight &mdash; this is plain accuracy</text><rect x="46.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="128.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="210.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="292.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="374.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="456.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="538.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="620.0" y="232.0" width="58.0" height="20.0" rx="3" fill="rgba(15,23,42,0.62)" /><line x1="38.0" y1="252" x2="686.0" y2="252" stroke="rgba(15,23,42,0.28)" stroke-width="2" /><text x="0" y="298" font-size="21" font-weight="800" fill="rgb(15,23,42)">&#955; = 4</text><text x="82" y="298" font-size="19" fill="rgba(15,23,42,0.6)">the weight piles onto the last thresholds</text><rect x="46.0" y="386.0" width="58.0" height="0.6" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="128.0" y="385.4" width="58.0" height="0.6" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="210.0" y="383.5" width="58.0" height="2.5" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="292.0" y="379.2" width="58.0" height="6.8" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="374.0" y="371.6" width="58.0" height="14.4" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="456.0" y="359.8" width="58.0" height="26.2" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="538.0" y="342.8" width="58.0" height="43.2" rx="3" fill="rgba(15,23,42,0.62)" /><rect x="620.0" y="319.8" width="58.0" height="66.2" rx="3" fill="rgba(15,23,42,0.62)" /><line x1="38.0" y1="386" x2="686.0" y2="386" stroke="rgba(15,23,42,0.28)" stroke-width="2" /><text x="75.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">1</text><text x="157.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">2</text><text x="239.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">3</text><text x="321.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">4</text><text x="403.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">5</text><text x="485.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">6</text><text x="567.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">7</text><text x="649.0" y="412" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">8</text><text x="350" y="438" text-anchor="middle" font-size="19" fill="rgba(15,23,42,0.52)">threshold t</text></svg>
<svg viewBox="0 0 700 450" preserveAspectRatio="xMidYMid meet" role="img" aria-label="Utility against lambda on a log scale for the two banks. Both start near 0.99 at the discovery end, where bank A is a hair ahead. Bank B pulls away as lambda grows, reaching 0.33 against bank A's 0.10 at lambda equals ten." style="display:block;width:100%;height:372px;">
  <line x1="150.0" y1="268.0" x2="600.0" y2="268.0" stroke="rgba(15,23,42,0.09)" stroke-width="1.5" /><text x="136.0" y="275.0" text-anchor="end" font-size="19" fill="rgba(15,23,42,0.52)">0.25</text><line x1="150.0" y1="196.0" x2="600.0" y2="196.0" stroke="rgba(15,23,42,0.09)" stroke-width="1.5" /><text x="136.0" y="203.0" text-anchor="end" font-size="19" fill="rgba(15,23,42,0.52)">0.50</text><line x1="150.0" y1="124.0" x2="600.0" y2="124.0" stroke="rgba(15,23,42,0.09)" stroke-width="1.5" /><text x="136.0" y="131.0" text-anchor="end" font-size="19" fill="rgba(15,23,42,0.52)">0.75</text><line x1="150.0" y1="52.0" x2="600.0" y2="52.0" stroke="rgba(15,23,42,0.09)" stroke-width="1.5" /><text x="136.0" y="59.0" text-anchor="end" font-size="19" fill="rgba(15,23,42,0.52)">1.00</text>
  <text x="136.0" y="347" text-anchor="end" font-size="19" fill="rgba(15,23,42,0.52)">0</text>
  <line x1="450.0" y1="52" x2="450.0" y2="340" stroke="rgba(15,23,42,0.3)" stroke-width="2" stroke-dasharray="5 6" />
  <text x="450.0" y="320" text-anchor="middle" font-size="20" font-weight="700" fill="rgba(15,23,42,0.7)">accuracy</text>
  <polyline points="150.0,55.6 155.0,55.8 160.0,55.9 165.0,56.1 170.0,56.2 175.0,56.4 180.0,56.6 185.0,56.8 190.0,57.0 195.0,57.3 200.0,57.5 205.0,57.8 210.0,58.1 215.0,58.5 220.0,58.8 225.0,59.2 230.0,59.6 235.0,60.1 240.0,60.5 245.0,61.0 250.0,61.6 255.0,62.2 260.0,62.8 265.0,63.5 270.0,64.3 275.0,65.1 280.0,65.9 285.0,66.8 290.0,67.8 295.0,68.9 300.0,70.0 305.0,71.2 310.0,72.5 315.0,74.0 320.0,75.4 325.0,77.1 330.0,78.8 335.0,80.6 340.0,82.6 345.0,84.6 350.0,86.9 355.0,89.2 360.0,91.7 365.0,94.4 370.0,97.2 375.0,100.2 380.0,103.4 385.0,106.8 390.0,110.3 395.0,114.0 400.0,117.9 405.0,122.0 410.0,126.3 415.0,130.8 420.0,135.5 425.0,140.3 430.0,145.4 435.0,150.6 440.0,156.0 445.0,161.5 450.0,167.2 455.0,173.0 460.0,178.9 465.0,185.0 470.0,191.1 475.0,197.2 480.0,203.4 485.0,209.6 490.0,215.8 495.0,222.0 500.0,228.1 505.0,234.1 510.0,240.0 515.0,245.8 520.0,251.4 525.0,256.8 530.0,262.1 535.0,267.1 540.0,271.9 545.0,276.5 550.0,280.9 555.0,285.0 560.0,288.9 565.0,292.5 570.0,295.9 575.0,299.0 580.0,301.9 585.0,304.5 590.0,307.0 595.0,309.2 600.0,311.2" fill="none" stroke="rgb(5,150,105)" stroke-width="4" />
  <polyline points="150.0,56.2 155.0,56.3 160.0,56.4 165.0,56.5 170.0,56.7 175.0,56.8 180.0,57.0 185.0,57.1 190.0,57.3 195.0,57.5 200.0,57.7 205.0,57.9 210.0,58.1 215.0,58.4 220.0,58.6 225.0,58.9 230.0,59.2 235.0,59.6 240.0,59.9 245.0,60.3 250.0,60.7 255.0,61.2 260.0,61.7 265.0,62.2 270.0,62.8 275.0,63.4 280.0,64.0 285.0,64.7 290.0,65.5 295.0,66.3 300.0,67.1 305.0,68.0 310.0,69.0 315.0,70.1 320.0,71.2 325.0,72.4 330.0,73.7 335.0,75.1 340.0,76.5 345.0,78.1 350.0,79.8 355.0,81.5 360.0,83.4 365.0,85.4 370.0,87.5 375.0,89.7 380.0,92.1 385.0,94.5 390.0,97.1 395.0,99.9 400.0,102.8 405.0,105.8 410.0,108.9 415.0,112.2 420.0,115.6 425.0,119.1 430.0,122.7 435.0,126.5 440.0,130.4 445.0,134.3 450.0,138.4 455.0,142.5 460.0,146.8 465.0,151.0 470.0,155.4 475.0,159.7 480.0,164.1 485.0,168.5 490.0,172.9 495.0,177.2 500.0,181.5 505.0,185.8 510.0,190.0 515.0,194.1 520.0,198.1 525.0,202.1 530.0,205.9 535.0,209.6 540.0,213.3 545.0,216.7 550.0,220.1 555.0,223.3 560.0,226.4 565.0,229.3 570.0,232.1 575.0,234.7 580.0,237.2 585.0,239.6 590.0,241.8 595.0,243.9 600.0,245.8" fill="none" stroke="rgb(2,132,199)" stroke-width="4" />
  <line x1="150.0" y1="340" x2="600.0" y2="340" stroke="rgba(15,23,42,0.28)" stroke-width="2" />
  <line x1="150.0" y1="340" x2="150.0" y2="348" stroke="rgba(15,23,42,0.28)" stroke-width="1.5" /><text x="150.0" y="370" text-anchor="middle" font-size="20" fill="rgba(15,23,42,0.7)">0.01</text><line x1="300.0" y1="340" x2="300.0" y2="348" stroke="rgba(15,23,42,0.28)" stroke-width="1.5" /><text x="300.0" y="370" text-anchor="middle" font-size="20" fill="rgba(15,23,42,0.7)">0.1</text><line x1="450.0" y1="340" x2="450.0" y2="348" stroke="rgba(15,23,42,0.28)" stroke-width="1.5" /><text x="450.0" y="370" text-anchor="middle" font-size="20" fill="rgba(15,23,42,0.7)">1</text><line x1="600.0" y1="340" x2="600.0" y2="348" stroke="rgba(15,23,42,0.28)" stroke-width="1.5" /><text x="600.0" y="370" text-anchor="middle" font-size="20" fill="rgba(15,23,42,0.7)">10</text>
  <text x="156.0" y="40" font-size="20" font-weight="700" fill="rgba(15,23,42,0.7)">Pass@8</text>
  <text x="600.0" y="40" text-anchor="end" font-size="20" font-weight="700" fill="rgba(15,23,42,0.7)">all 8 of 8</text>
  <text x="612.0" y="252.8" font-size="21" font-weight="700" fill="rgb(2,132,199)">Bank B</text>
  <text x="612.0" y="318.2" font-size="21" font-weight="700" fill="rgb(5,150,105)">Bank A</text>
  <text x="375" y="404" text-anchor="middle" font-size="20" fill="rgba(15,23,42,0.52)">&#955; &mdash; what you are paying for</text>
</svg>
</div>

---

## Evaluation in action

<div class="pk-code">pip install scorio<br /><span class="kw">from</span> scorio.eval <span class="kw">import</span> *</div>

<table class="pk-table">
  <thead>
    <tr>
      <th style="width: 30%;">Function</th>
      <th style="width: 44%;">What it reports</th>
      <th style="width: 26%;">With uncertainty</th>
    </tr>
  </thead>
  <tbody>
    <tr><td><code>avg(R)</code></td><td>the plain average over all trials</td><td><code>avg_ci</code></td></tr>
    <tr><td><code>bayes(R, w)</code></td><td>Bayes@N, a posterior over the score</td><td><code>bayes_ci</code></td></tr>
    <tr><td><code>pass_at_k(R, k)</code></td><td>at least one of k attempts correct</td><td><code>pass_at_k_ci</code></td></tr>
    <tr><td><code>pass_hat_k(R, k)</code></td><td>all k attempts correct</td><td><code>pass_hat_k_ci</code></td></tr>
    <tr><td><code>maj_at_k(R, k)</code></td><td>the majority vote over k attempts</td><td><code>maj_at_k_ci</code></td></tr>
    <tr><td><code>geom_at_k(R, k)</code></td><td>Geom@k, discovery and stability blended</td><td><code>geom_at_k_ci</code></td></tr>
    <tr><td><code>max_at_k(R, k, w)</code></td><td>the best outcome in k, under a utility w</td><td><code>max_at_k_ci</code></td></tr>
    <tr><td><code>g_pass_at_k_tau(R, k, tau)</code></td><td>at least a &tau; fraction of k correct</td><td><code>g_pass_at_k_tau_ci</code></td></tr>
  </tbody>
</table>

<p class="pk-foot"><code>R</code> is the outcome matrix, M questions by N trials. Every <code>_ci</code> returns (&mu;, &sigma;, lo, hi).</p>

---

## Ranking

<div class="pk-code"><span class="kw">from</span> scorio.rank <span class="kw">import</span> *<br />ranks = bradley_terry(R)&nbsp;&nbsp;<span class="cm"># R: L models &times; M questions &times; N trials</span></div>

<div class="pk-grid four">
  <div class="pk-card blue">
    <h3>From the metrics</h3>
    <p class="pk-meth">avg, bayes, pass_at_k, pass_hat_k, g_pass_at_k_tau, mg_pass_at_k, inverse_difficulty</p>
  </div>
  <div class="pk-card purple">
    <h3>Paired comparison</h3>
    <p class="pk-meth">bradley_terry, bradley_terry_davidson, rao_kupper, each with a _map variant</p>
  </div>
  <div class="pk-card orange">
    <h3>Rating systems</h3>
    <p class="pk-meth">elo, glicko, trueskill</p>
  </div>
  <div class="pk-card green">
    <h3>Bayesian</h3>
    <p class="pk-meth">thompson, bayesian_mcmc</p>
  </div>
  <div class="pk-card rose">
    <h3>Voting</h3>
    <p class="pk-meth">borda, copeland, win_rate, minimax, schulze, ranked_pairs, kemeny_young, nanson, baldwin, majority_judgment</p>
  </div>
  <div class="pk-card blue">
    <h3>Item response theory</h3>
    <p class="pk-meth">rasch, rasch_2pl, rasch_3pl, rasch_mml, rasch_mml_credible, dynamic_irt, mirt</p>
  </div>
  <div class="pk-card orange">
    <h3>Graph and geometry</h3>
    <p class="pk-meth">pagerank, spectral, rank_centrality, alpharank, nash, serial_rank, hodge_rank</p>
  </div>
  <div class="pk-card purple">
    <h3>Luce family</h3>
    <p class="pk-meth">plackett_luce, davidson_luce, bradley_terry_luce, each with a _map variant</p>
  </div>
</div>

---

## Big Picture: Test-time scaling

<p class="pk-line">Spending more compute at inference...</p>

<div class="pk-dim">
  <div class="pk-card blue">
    <h3>1. Sampling</h3>
    <p>How the candidates are produced: temperature, top-p, typical sampling, and how many you draw.</p>
    <p class="who">scorio.sampling</p>
  </div>
  <div class="pk-card green">
    <h3>2. Aggregation</h3>
    <p>How a pool of candidates becomes one answer: vote, best-of-N, verifier score, early stopping.</p>
    <p class="who">scorio.aggregate</p>
  </div>
  <div class="pk-card orange">
    <h3>3. Evaluation</h3>
    <p>How that answer is scored, and what uncertainty is reported alongside it.</p>
    <p class="who">scorio.eval</p>
  </div>
</div>

Note:
The reason to separate the three is that papers routinely change two of them at once and attribute the difference to the model. Sampling decides what is in the bank. Aggregation decides which item comes out. Evaluation decides what number you print. Report one without the other two and nobody can reproduce you.

---

## Reducers in action

<div class="pk-code"><span class="kw">from</span> scorio.aggregate <span class="kw">import</span> *<br />answer = weighted_majority_vote(Z, deepconf_confidence(logprobs))</div>

<div class="pk-split even">
  <div class="pk-card blue">
    <h3>Signals: how good is this candidate?</h3>
    <p class="pk-meth">mean_logprob, sequence_logprob, perplexity, self_certainty, token_confidence, deepconf_confidence, token_entropy, varentropy, max_softmax_probability, logprob_margin, picsar, prm_aggregate</p>
    <p>Scored from the trace's own log-probabilities, so most of these need no external verifier.</p>
  </div>
  <div class="pk-card green">
    <h3>Rules: which candidate wins?</h3>
    <p class="pk-meth">majority_vote, weighted_majority_vote, softmax_weighted_vote, rank_weighted_vote, logit_weighted_vote, filtered_vote, best_of_n, best_of_majority, majority_of_the_bests, kde_weighted_vote, cges_vote</p>
    <p>Each takes the answers <code>Z</code> and, where it needs them, the scores <code>S</code>.</p>
  </div>
</div>

<div class="pk-callout green">
  <p class="pk-foot">Six more decide when to stop sampling: <code>esc_stop</code>, <code>cges_stop</code>, <code>deepconf_online_stop</code>, and the three <code>adaptive_consistency_*</code> rules.</p>
</div>

---

## Summary of scorio APIs

<div class="pk-code">pip install scorio</div>

<div class="pk-tree"><span class="mod">scorio</span>
├── <span class="mod">eval</span>
│   ├── <span class="fn">bayes</span>          <span class="desc">Bayes@N</span>
│   ├── <span class="fn">geom_at_k</span>      <span class="desc">Geom@k</span>
│   ├── <span class="fn">tail_pass_at_k</span> <span class="desc">TailPass@k</span>   <span class="soon">(waiting for NeurIPS decision)</span>
│   └── <span class="fn">risk_pass_at_k</span> <span class="desc">RiskPass@k</span>   <span class="soon">(waiting for NeurIPS decision)</span>
├── <span class="mod">rank</span>
│   ├── <span class="fn">bayes</span>          <span class="desc">Bayesian ranking</span>
│   └── <span class="fn">bradley_terry</span>  <span class="desc">Bradley&ndash;Terry ranking</span>
├── <span class="mod">aggregate</span>
│   └── <span class="fn">best_of_n</span>      <span class="desc">Best of N</span>
└── <span class="mod">sampling</span>
    └── <span class="fn">typical</span>        <span class="desc">sampling methods</span></div>

Note:
If you take one thing away, take the install line. Everything named in this talk is one function call away. The last two are not out yet.

---

## Datasets

<p class="pk-line">Each of these is a bank of repeated runs.</p>

<table class="pk-table">
  <thead>
    <tr>
      <th style="width: 26%;">Dataset</th>
      <th style="width: 74%;">What is in it</th>
    </tr>
  </thead>
  <tbody>
    <tr><td><a href="https://huggingface.co/datasets/harimo/scorio-trace"><code>harimo/scorio-trace</code></a></td><td>20 reasoning LLMs and 2 reward models on AIME&rsquo;24, AIME&rsquo;25, HMMT&rsquo;25 and BrUMO&rsquo;25, at <b>80 runs</b> each</td></tr>
    <tr><td><a href="https://huggingface.co/datasets/harimo/scorio-lite"><code>harimo/scorio-lite</code></a></td><td>AIME&rsquo;26, CMIMC&rsquo;25, HMMT&rsquo;26, HMMT&rsquo;25 (Nov), SMT&rsquo;25 and superGPQA, with a reward model and an LLM-as-verifier, at <b>80 runs</b> each</td></tr>
    <tr><td><a href="https://huggingface.co/datasets/harimo/scorio-math"><code>harimo/scorio-math</code></a></td><td>scorio-lite, plus the full token history for the math sets</td></tr>
    <tr><td><a href="https://huggingface.co/datasets/harimo/scorio-gpqa"><code>harimo/scorio-gpqa</code></a></td><td>scorio-lite, plus the full token history for GPQA</td></tr>
  </tbody>
</table>

<p class="pk-line" style="margin-top: 42px;">1,403,520 traces, 9.61 B tokens, 17,544 questions, 9.9 TiB.</p>

---

## Collaboration: How Scorio is built

<div class="pk-tree"><span class="mod">scorio</span>
├── <span class="mod">theory</span>
│   └── <span class="mod">statistics</span>
│       ├── <span class="fn">Bayesian statistics</span>          <span class="desc">Bayes@N</span>
│       ├── <span class="fn">survival analysis</span>            <span class="desc">TailPass@k</span>
│       ├── <span class="fn">copulas, Fr&eacute;chet&ndash;Hoeffding</span>   <span class="desc">RiskPass@k</span>
│       └── <span class="fn">sampling methods</span>             <span class="desc">&hellip;</span>
└── <span class="mod">experimental design</span>
    └── <span class="mod">run LLMs on</span>
        ├── <span class="fn">healthcare</span>
        ├── <span class="fn">agentic tasks</span>                <span class="desc">episodic reasoning</span>
        ├── <span class="fn">hard reasoning</span>               <span class="desc">math, science, coding</span>
        └── <span class="fn">&hellip;</span></div>

---

<div class="pk-divider">
  <div class="pk-part-mark">
    <div class="pk-part-label">Part</div>
    <div class="pk-part-num">One</div>
    <div class="pk-part-rule"></div>
  </div>
  <div class="pk-divider-body">
    <h1>What goes wrong</h1>
    <p class="pk-divider-sub">Seven ways one repeated-sampling number misdescribes the run behind it.</p>
    <div class="pk-divider-map" aria-label="The seven failure modes">
      <span class="proj">01 &middot; Rankings<br />that flip</span>
      <span class="proj">02 &middot; A moving<br />target</span>
      <span class="proj">03 &middot; Runs that<br />collide</span>
      <span class="aux">04 &middot; Verifiers<br />that lie</span>
      <span class="aux">05 &middot; Attempts<br />that repeat</span>
      <span class="aux">06 &middot; Attempts<br />that cost</span>
      <span class="aux">07 &middot; Numbers<br />that wobble</span>
      <span>What to report<br />instead</span>
    </div>
  </div>
</div>

Note:
One failure at a time, each with the smallest example that shows it. The exact statements come in Part Two.

---

## One run, two different answers

<p class="pk-note">A run leaves a grid: one row per question, one column per attempt, each cell right or wrong.</p>

<div class="pk-flow">
  <div class="pk-mat-unit">
    <div class="pk-mat-title">One run</div>
    <div class="pk-mat" style="grid-template-columns: repeat(10, 34px);">
      <span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span>
    </div>
    <div class="pk-mat-cap">5 questions, 10 attempts each, 18 of the 50 correct</div>
  </div>
  <div class="pk-flow-arrow" aria-hidden="true">&rarr;</div>
  <div class="pk-scalar-pair">
    <div class="pk-scalar alt">
      <b>0.36</b>
      <span>Pass@1: one attempt is right</span>
    </div>
    <div class="pk-scalar">
      <b>0.80</b>
      <span>Pass@10: ten attempts find something</span>
    </div>
  </div>
</div>

<div class="pk-callout orange">
  Same grid, same metric. Only the budget changed, and the reader takes whichever you print as the score.
</div>

Note:
Nobody thinks 0.36 and 0.80 are the same claim about this model, and both are Pass@k on this exact grid. Everything else in this half follows from publishing one of them.

---

## Same accuracy, opposite shape

<p class="pk-note">20 of 50 cells correct in both. Accuracy is 0.40 either way.</p>

<div class="pk-mat-row">
  <div class="pk-mat-unit">
    <div class="pk-mat-title" style="color: rgb(5,150,105);">Packed into two rows</div>
    <div class="pk-mat" style="grid-template-columns: repeat(10, 38px);">
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span>
      <span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span>
    </div>
    <div class="pk-mat-cap">2 questions always solved, 3 never touched</div>
  </div>
  <div class="pk-mat-unit">
    <div class="pk-mat-title" style="color: rgb(2,132,199);">Spread across all five</div>
    <div class="pk-mat alt" style="grid-template-columns: repeat(10, 38px);">
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span>
      <span class="on"></span><span class="on"></span><span class="on"></span><span class="on"></span><span></span><span></span><span></span><span></span><span></span><span></span>
    </div>
    <div class="pk-mat-cap">every question reachable, none reliable</div>
  </div>
</div>

<div class="pk-split even">
  <div class="pk-callout green">Two questions in five are reachable, and on those every attempt works. Call it once.</div>
  <div class="pk-callout">All five are reachable, none of them reliably. Needs a verifier and a retry budget.</div>
</div>

Note:
If you are choosing between these two for deployment, the answer is never "they are equivalent." Every failure in this half is a version of that.

---

## Seven failures, two kinds

<p class="pk-note">Three are lossy reporting on good data. Four are the assumptions breaking.</p>

<div class="pk-roadmap">
  <div class="pk-step blue" data-step="01">
    <h3>Rank instability</h3>
    <p>The winner changes with the budget.</p>
  </div>
  <div class="pk-step blue" data-step="02">
    <h3>Coverage, not accuracy</h3>
    <p>Big budgets stop measuring accuracy.</p>
  </div>
  <div class="pk-step blue" data-step="03">
    <h3>Non-identifiability</h3>
    <p>One scalar, a huge family of runs.</p>
  </div>
  <div class="pk-step purple" data-step="04">
    <h3>Verifier &amp; selector</h3>
    <p>The label is acceptance, not correctness.</p>
  </div>
</div>

<div class="pk-roadmap" style="margin-top: 18px;">
  <div class="pk-step purple" data-step="05">
    <h3>Dependence</h3>
    <p>Echoing attempts shrink the budget.</p>
  </div>
  <div class="pk-step purple" data-step="06">
    <h3>Harm blindness</h3>
    <p>Failures are scored at zero cost.</p>
  </div>
  <div class="pk-step purple" data-step="07">
    <h3>Finite-bank noise</h3>
    <p>The number wobbles where it looks safest.</p>
  </div>
  <div class="pk-card" style="background: rgba(15,23,42,0.045);">
    <h3>Common cause</h3>
    <p>The same shape goes missing every time.</p>
  </div>
</div>

Note:
The distinction matters for what you do about it. For the first three the data is fine and the report is lossy. For the last four, the thing you measured is not the thing you meant, and reporting more of it will not save you.

---

## The winner depends on the budget you picked

<div class="pk-split">
  <figure class="pk-figure" style="height: 430px;">
    <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/rank_bump.webp" alt="Pass@k rank path for two response banks; rank 1 denotes the larger Pass@k value, and the ordering changes at three separate budgets" />
    <figcaption>Rank path for two banks. The ordering changes three times.</figcaption>
  </figure>
  <div class="pk-stack">
    <div class="pk-card blue">
      <h3>Two banks, five questions, ten attempts</h3>
      <p>One spreads its successes thin, the other concentrates them. Pass@k sees only the per-question counts.</p>
    </div>
    <div class="pk-callout orange">
      The first wins at 1 attempt and at 3 to 5. The second wins at 2 and at 6 to 10.
    </div>
  </div>
</div>

<div class="pk-signs">
  <div class="pk-sign a"><b>+</b><span>1 try</span></div>
  <div class="pk-sign b flip"><b>&minus;</b><span>2</span></div>
  <div class="pk-sign a flip"><b>+</b><span>3</span></div>
  <div class="pk-sign a"><b>+</b><span>4</span></div>
  <div class="pk-sign a"><b>+</b><span>5</span></div>
  <div class="pk-sign b flip"><b>&minus;</b><span>6</span></div>
  <div class="pk-sign b"><b>&minus;</b><span>7</span></div>
  <div class="pk-sign b"><b>&minus;</b><span>8</span></div>
  <div class="pk-sign b"><b>&minus;</b><span>9</span></div>
  <div class="pk-sign b"><b>&minus;</b><span>10</span></div>
</div>

<div class="pk-eq-label">Who is ahead at each budget. Green is the first bank, orange the second.</div>

Note:
Nobody reports a rank path. You report one budget, whatever your compute allowed, and the reader takes it as the ordering. These are exact values on fixed grids, so it is not noise.

---

## How often that happens

<div class="pk-split even">
  <div class="pk-stack">
    <div class="pk-card rose">
      <h3>Any pattern is constructible</h3>
      <p>Name any sequence of winners across budgets. Some pair of runs produces exactly it.</p>
    </div>
    <div class="pk-card green">
      <h3>And it survives a nudge</h3>
      <p>Perturb either run and the flips stay. Reversals occupy regions, not points.</p>
    </div>
  </div>
  <div class="pk-stack">
    <div class="pk-stat-row" style="grid-auto-flow: row;">
      <div class="pk-stat orange">
        <b>1 in 5</b>
        <span>random pairs reverse twice or more, at 128 attempts per question</span>
      </div>
      <div class="pk-stat blue">
        <b>3,077</b>
        <span>of 827,541 small runs, exhaustively enumerated</span>
      </div>
    </div>
    <div class="pk-callout orange">
      128 attempts per question is ordinary now. One comparison in five would read differently at another budget.
    </div>
  </div>
</div>

Note:
The construction result is worst case and you could dismiss it as adversarial. The simulation is why you should not: twenty thousand random pairs per setting, and about a fifth of them flip more than once.

---

## Enough attempts and you stop measuring accuracy

<div class="pk-split">
  <figure class="pk-figure" style="height: 430px;">
    <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/inf_nonconvergence.webp" alt="Left: a weak solver whose average accuracy stays at one percent while its repeated-sampling score climbs toward one. Right: a ranking inversion between two models as the budget grows" />
    <figcaption>Left: every question solved one time in a hundred. Right: strong-on-half against weak-on-all.</figcaption>
  </figure>
  <div class="pk-stack">
    <div class="pk-card orange">
      <h3>Three different targets</h3>
      <ul>
        <li><b>One attempt.</b> Accuracy.</li>
        <li><b>A few.</b> Part accuracy, part reach.</li>
        <li><b>Many.</b> How many questions the model can <em>ever</em> solve.</li>
      </ul>
    </div>
    <div class="pk-callout orange">
      A bigger budget does not measure accuracy better. It measures something else.
    </div>
  </div>
</div>

<div class="pk-split even">
  <table class="pk-table compact">
    <thead>
      <tr><th>Attempts</th><th class="num">Model A</th><th class="num">Model B</th><th>Winner</th></tr>
    </thead>
    <tbody>
      <tr><td>1</td><td class="num">0.400</td><td class="num">0.100</td><td class="pk-win">A</td></tr>
      <tr><td>5</td><td class="num">0.500</td><td class="num">0.410</td><td class="pk-win">A</td></tr>
      <tr><td>7</td><td class="num">0.500</td><td class="num">0.522</td><td class="pk-bad">B</td></tr>
      <tr><td>100</td><td class="num">0.500</td><td class="num">1.000</td><td class="pk-bad">B</td></tr>
    </tbody>
  </table>
  <div class="pk-stack">
    <div class="pk-eq-label" style="text-align: left;">A: 8 in 10 on half the questions, never on the rest. B: 1 in 10 on everything.</div>
    <div class="pk-stat-row">
      <div class="pk-stat rose"><b>28.5%</b><span>of random pairs ranked differently by accuracy and by large-budget Pass@k</span></div>
    </div>
  </div>
</div>

Note:
Model A is four times more accurate and it loses from seven attempts onward. The 28.5 percent is exact, with no sampling noise in it at all.

---

## One number, a huge family of runs

<p class="pk-note">Two questions, four attempts. Both runs report 0.50 at two attempts.</p>

<div class="pk-split reverse">
  <div class="pk-stack">
    <div class="pk-card" style="padding: 18px 22px;">
    <div class="pk-mat-row" style="justify-content: flex-start; gap: 44px;">
      <div class="pk-mat-unit">
        <div class="pk-mat-title">Run A: one success each</div>
        <div class="pk-mat" style="grid-template-columns: repeat(4, 38px);">
          <span class="on"></span><span></span><span></span><span></span>
          <span></span><span class="on"></span><span></span><span></span>
        </div>
      </div>
      <div class="pk-mat-unit">
        <div class="pk-mat-title">Run B: none, then three</div>
        <div class="pk-mat alt" style="grid-template-columns: repeat(4, 38px);">
          <span></span><span></span><span></span><span></span>
          <span class="on"></span><span class="on"></span><span class="on"></span><span></span>
        </div>
      </div>
    </div>
    </div>
    <div class="pk-card purple">
      <h3>Two things get thrown away</h3>
      <ul>
        <li><b>Which attempts.</b> Only the count per question survives.</li>
        <li><b>How many.</b> Past a point, extra successes stop moving the score.</li>
      </ul>
    </div>
  </div>
  <div class="pk-stack">
    <table class="pk-table compact">
      <thead>
        <tr><th>Attempts</th><th class="num">Distinct scores</th><th class="num">Largest group</th><th class="num">Collision chance</th></tr>
      </thead>
      <tbody>
        <tr><td>1</td><td class="num">101</td><td class="num">5,448</td><td class="num">5.6%</td></tr>
        <tr><td>3</td><td class="num">1,012</td><td class="num">601</td><td class="num">0.6%</td></tr>
        <tr><td>5</td><td class="num">1,436</td><td class="num">1,385</td><td class="num">0.7%</td></tr>
        <tr><td>10</td><td class="num">11</td><td class="num pk-bad">92,378</td><td class="num pk-bad">98.1%</td></tr>
      </tbody>
    </table>
    <div class="pk-eq-label">Exact over all 184,756 runs of 10 questions by 10 attempts.</div>
    <div class="pk-callout rose">
      Inside one mid-budget score, one-attempt accuracy ranges from 0.36 to 0.90.
    </div>
  </div>
</div>

Note:
The 98 percent is the full-budget case and the easiest to dismiss, so use the middle row instead. Five attempts is the best-resolved single budget here and it still puts 1,385 genuinely different runs behind one number.

---

## The other four break the assumptions

<p class="pk-note">These four are not lossy reporting. The grid you measured is not the grid you meant.</p>

<table class="pk-table">
  <thead>
    <tr><th style="width: 24%;">Mode</th><th style="width: 36%;">Hidden assumption</th><th style="width: 40%;">What it looks like when it fails</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><b>Verifier &amp; selector</b></td>
      <td>The check is exact.</td>
      <td>The score goes to one; what you hand back stays wrong.</td>
    </tr>
    <tr>
      <td><b>Dependence</b></td>
      <td>Attempts are independent tries.</td>
      <td>A large budget is worth a much smaller one.</td>
    </tr>
    <tr>
      <td><b>Harm blindness</b></td>
      <td>Wrong attempts cost nothing.</td>
      <td>Equal scores, arbitrarily different harm rates.</td>
    </tr>
    <tr>
      <td><b>Finite-bank noise</b></td>
      <td>The estimate is precise enough to rank on.</td>
      <td>Sampling error still flips rankings.</td>
    </tr>
  </tbody>
</table>

Note:
If you are in the first group you can fix the report. If you are in the second group the number is answering a question about a different grid than the one you meant.

---

## Repeated sampling amplifies verifier errors

<div class="pk-board">
  <div class="pk-rail">
    <div class="pk-rail-badge">04</div>
    <h3>Verifier &amp; selector</h3>
    <p>The recorded label is acceptance, not correctness. Unit tests are incomplete and judges are models.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>0.095</b>Reported after 100 attempts on a task nobody can solve</div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split even">
      <figure class="pk-figure pk-fig-short">
        <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/verifier_gap.webp" alt="Left: reported scores climb on impossible tasks under several false-alarm rates. Right: the reported score rises toward one while the chance of returning a truly correct sample flattens out" />
        <figcaption>Right: the reported score goes to one, returned correctness flattens well below it.</figcaption>
      </figure>
      <div class="pk-stack">
        <table class="pk-table compact">
          <thead><tr><th>Task</th><th class="num">Alarms</th><th class="num">Reported</th><th class="num">Correct</th><th class="num">False</th></tr></thead>
          <tbody>
            <tr><td>impossible</td><td class="num">1 in 1,000</td><td class="num">0.095</td><td class="num">0.000</td><td class="num pk-bad">0.095</td></tr>
            <tr><td>1 in 100</td><td class="num">1 in 100</td><td class="num">0.866</td><td class="num">0.435</td><td class="num pk-bad">0.431</td></tr>
            <tr><td>1 in 20</td><td class="num">1 in 100</td><td class="num">0.998</td><td class="num">0.839</td><td class="num">0.159</td></tr>
          </tbody>
        </table>
        <div class="pk-eq-label">Alarms: how often the checker accepts something wrong. All at 100 attempts.</div>
        <div class="pk-callout orange">
          Sample enough and something always gets accepted. What survives is a fact about the checker.
        </div>
      </div>
    </div>
  </div>
</div>

Note:
Reported 0.866 sounds like the problem is mostly solved, and half of what comes back is a false positive. Remember the first row when a benchmark suddenly looks saturated: a task with zero chance of being solved reports nearly ten percent. Reported score and returned correctness rank two systems differently 41 percent of the time at 250 attempts.

---

## A budget of 64 attempts can be worth 19

<div class="pk-board green">
  <div class="pk-rail">
    <div class="pk-rail-badge">05</div>
    <h3>Dependence</h3>
    <p>Duplicated traces, shared decoding state, and prompt-level randomness shrink the effective budget.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>30%</b>The fraction of a nominal 64 that survives, at mild correlation</div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split">
      <figure class="pk-figure pk-fig-short">
        <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/dependence_keff.webp" alt="Left: discovery curves under increasing correlation between attempts, all sharing the same single-attempt success rate. Right: the fraction of a 64-attempt budget that survives as correlation grows" />
        <figcaption>Every curve on the left has the same one-attempt success rate.</figcaption>
      </figure>
      <div class="pk-stack">
        <table class="pk-table compact">
          <thead><tr><th>Attempts echo</th><th class="num">Finds one</th><th class="num">Independent-equivalent</th><th class="num">Budget kept</th></tr></thead>
          <tbody>
            <tr><td>not at all</td><td class="num">0.962</td><td class="num">64.0</td><td class="num pk-win">100%</td></tr>
            <tr><td>mildly</td><td class="num">0.623</td><td class="num">19.0</td><td class="num">30%</td></tr>
            <tr><td>moderately</td><td class="num">0.391</td><td class="num">9.7</td><td class="num pk-bad">15%</td></tr>
          </tbody>
        </table>
        <div class="pk-eq-label">64 nominal attempts, each right about 1 time in 20 on its own.</div>
        <div class="pk-callout orange">
          The one-attempt rate says little about the sixty-four-attempt rate unless the attempts are really independent.
        </div>
      </div>
    </div>
  </div>
</div>

Note:
Two things. The one-attempt rate pins the many-attempt rate only under independence, and the range it leaves open is enormous. And any variation across a batch pushes the real curve below the independent one, so extrapolating always overstates the gain.

---

## The metric never charges you for a bad attempt

<div class="pk-board orange">
  <div class="pk-rail">
    <div class="pk-rail-badge">06</div>
    <h3>Harm blindness</h3>
    <p>Pass@k is one-sided: credit for a success, nothing for a wrong or dangerous attempt.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>0.634</b>Chance of a harmful attempt when 1 in 100 is harmful</div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split even">
      <figure class="pk-figure pk-fig-short">
        <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/harm_utility.webp" alt="Left: the reported score is flat as the harmful-attempt rate varies while harm risk grows with the budget. Right: a score that subtracts harm risk peaks at a much smaller budget" />
        <figcaption>Left: the score ignores the harm rate. Right: charge for harm and the best budget shrinks.</figcaption>
      </figure>
      <div class="pk-stack">
        <table class="pk-table compact">
          <thead><tr><th>Harm rate</th><th class="num">Attempts</th><th class="num">Success</th><th class="num">Harm</th><th class="num">Net</th></tr></thead>
          <tbody>
            <tr><td>none</td><td class="num">100</td><td class="num">0.994</td><td class="num">0.000</td><td class="num pk-win">0.994</td></tr>
            <tr><td>1 in 100</td><td class="num">50</td><td class="num">0.923</td><td class="num">0.395</td><td class="num">0.528</td></tr>
            <tr><td>1 in 100</td><td class="num">100</td><td class="num">0.994</td><td class="num">0.634</td><td class="num pk-bad">0.360</td></tr>
            <tr><td>1 in 10</td><td class="num">10</td><td class="num">0.401</td><td class="num">0.651</td><td class="num pk-bad">&minus;0.250</td></tr>
          </tbody>
        </table>
        <div class="pk-eq-label">Chance of at least one correct attempt, at least one harmful one, and the difference.</div>
        <div class="pk-callout orange">
          At a one-percent harm rate the best budget is 50, not 100. Pass@k says 100 and keeps going.
        </div>
      </div>
    </div>
  </div>
</div>

Note:
For an offline code benchmark where you throw the failures away, this mode is irrelevant and you should say so. For anything that touches a tool, spends money, or reaches a person, it is the whole story: both curves rise with every extra attempt, and only one of them is on your slide.

---

## The number is noisiest where it looks safest

<div class="pk-board purple">
  <div class="pk-rail">
    <div class="pk-rail-badge">07</div>
    <h3>Finite-bank noise</h3>
    <p>The formula averages over enormously many ways to draw attempts, all reusing the same few generated answers.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>31%</b>Chance the weaker model is ranked first at the full budget, with 200 questions</div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split">
      <figure class="pk-figure pk-fig-short">
        <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/uncertainty_finite_bank.webp" alt="Left: exact standard errors across budgets. Right: two close models where the finite-sample estimate can rank the weaker one first" />
        <figcaption>Right: a model better at every budget, and how often the measurement says otherwise.</figcaption>
      </figure>
      <div class="pk-stack">
        <table class="pk-table compact">
          <thead><tr><th>Questions</th><th class="num">1 attempt</th><th class="num">10</th><th class="num">40</th><th class="num">80</th></tr></thead>
          <tbody>
            <tr><td>50</td><td class="num">0.158</td><td class="num">0.161</td><td class="num">0.226</td><td class="num pk-bad">0.403</td></tr>
            <tr><td>100</td><td class="num">0.078</td><td class="num">0.081</td><td class="num">0.144</td><td class="num pk-bad">0.365</td></tr>
            <tr><td>200</td><td class="num pk-win">0.022</td><td class="num">0.024</td><td class="num">0.066</td><td class="num pk-bad">0.312</td></tr>
          </tbody>
        </table>
        <div class="pk-eq-label">How often the weaker of two close models is ranked first. 80 attempts banked.</div>
      </div>
    </div>
  </div>
</div>

Note:
Quadrupling the dataset takes the one-attempt error from 24 percent down to 2 and barely touches the full budget: 43 down to 31. At the full budget each question contributes a single yes-or-no, so the extra combinations buy no extra evidence. This is separate from mode one, where the true ranking itself moves.

---

## Seven symptoms, one missing picture

<p class="pk-note">A run has a shape. One thresholded number reports a single point on it.</p>

<div class="pk-grid four">
  <div class="pk-card blue">
    <h3>Finds anything</h3>
    <p>At least one attempt worked. This is Pass@k.</p>
  </div>
  <div class="pk-card green">
    <h3>Works every time</h3>
    <p>Every attempt worked. The edge you need when you cannot retry.</p>
  </div>
  <div class="pk-card orange">
    <h3>The area under it</h3>
    <p>Average accuracy, exactly.</p>
  </div>
  <div class="pk-card purple">
    <h3>How fast it falls</h3>
    <p>How quickly reliability decays as you ask for more.</p>
  </div>
</div>

<div class="pk-callout green">
  TailPass@k reports the whole picture with uncertainty. Pass@k stays available, as the discovery edge, where it belongs.
</div>

Note:
Which edge you report decides what you can see. Discovery is sensitive to questions the model almost never solves, stability to questions it almost always solves. And you do not have to adopt anything from our paper to stop making these mistakes: stop reporting a single thresholded scalar as if it summarised the run.

---

<div class="pk-divider">
  <div class="pk-part-mark two">
    <div class="pk-part-label">Part</div>
    <div class="pk-part-num">Two</div>
    <div class="pk-part-rule"></div>
  </div>
  <div class="pk-divider-body">
    <h1>Why, exactly</h1>
    <p class="pk-divider-sub">The same seven failures as closed-form statements, with what bounds each one.</p>
    <div class="pk-notation-strip">
      <div class="pk-notation-item">
        \( R \in \{0,1\}^{M\times N} \)
        <small>the response bank: M questions, N attempts</small>
      </div>
      <div class="pk-notation-item">
        \( \gamma_i = \sum_{j} R_{ij} \)
        <small>correct attempts on question i</small>
      </div>
      <div class="pk-notation-item">
        \( h_c = \#\{i : \gamma_i = c\} \)
        <small>questions with exactly c correct</small>
      </div>
    </div>
  </div>
</div>

Note:
Three symbols carry this half: the bank, the per-question count, and the histogram of those counts. Everything from here is a functional of the histogram, and most of the failures are statements about what that functional forgets.

---

## The object is a bank, the report is one coordinate

<p class="pk-note">Fixing a threshold and a budget picks one coordinate of a two-dimensional object.</p>

<div class="pk-split">
  <div class="pk-stack">
    <div class="pk-equation sm">
      \[ \widehat S_{k,t}(R)=\frac{1}{M}\sum_{i=1}^{M}\Pr(\mathrm{Hyp}(N,\gamma_i,k)\ge t) \]
      <div class="pk-eq-label">draw <em>k</em> of the <em>N</em> banked attempts, ask for at least <em>t</em> correct</div>
    </div>
    <div class="pk-equation sm">
      \[ \widehat S_{k,1}(R)=\frac{1}{M}\sum_{i=1}^{M}\left(1-\frac{\binom{N-\gamma_i}{k}}{\binom{N}{k}}\right) \]
      <div class="pk-eq-label">the usual Pass@k estimator: the profile at <em>t</em> = 1</div>
    </div>
    <div class="pk-callout">
      Pass@k is the <b>t = 1</b> coordinate. The threshold axis is never reported.
    </div>
  </div>
  <div class="pk-stack">
    <div class="pk-card blue">
      <h3>What the axes mean</h3>
      <ul>
        <li><b>t = 1.</b> Discovery.</li>
        <li><b>t = k.</b> Stability, every attempt correct.</li>
        <li><b>Area over t.</b> Average single-sample accuracy.</li>
      </ul>
    </div>
    <div class="pk-card">
      <h3>Latent view</h3>
      <p>With <em>p</em><sub><em>i</em></sub> the chance one attempt on question <em>i</em> is right:</p>
      <div class="pk-equation xs plain">
        \[ S_{k,t}=\frac{1}{M}\sum_{i=1}^{M}\Pr(\mathrm{Bin}(k,p_i)\ge t) \]
      </div>
    </div>
  </div>
</div>

Note:
Two notations, one object. The finite-bank version draws without replacement from what you actually generated; the latent version is about a fresh budget. Every asymptotic claim later is about the second, every enumeration about the first.

---

## Different budgets read different coefficients

<p class="pk-note">The gap at budget k is one coefficient of a polynomial. Its sign moves freely along the path.</p>

<div class="pk-split reverse">
  <div class="pk-stack">
    <div class="pk-equation sm">
      \[ \Delta_k=\frac{1}{\binom{N}{k}}\sum_{f=0}^{N}\sigma_f\binom{f}{k}=\frac{[z^k]B_\sigma(z)}{\binom{N}{k}} \]
      <div class="pk-eq-label">with \( B_\sigma(z)=\sum_f \sigma_f (1+z)^f \) over the failure-count difference</div>
    </div>
    <div class="pk-card blue">
      <h3>Read the two ends</h3>
      <ul>
        <li><b>k = 1:</b> mean success mass.</li>
        <li><b>k = 2:</b> subtracts a second factorial moment, penalizing concentration.</li>
        <li><b>k = N:</b> coverage, the fraction of rows with any success.</li>
      </ul>
    </div>
    <div class="pk-callout">
      Early budgets compare mean and concentration, late budgets compare unsolved mass. Nothing forces them to agree.
    </div>
  </div>
  <figure class="pk-figure pk-fig-tall">
    <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/rank_budget_functionals.webp" alt="Row-level Pass@k functional versus row success count for several budgets at N=64, with marginal gains in the second panel" />
    <figcaption>N = 64. Linear in the success count at k = 1, saturating like coverage at large k. Panel B is the marginal gain per correct sample.</figcaption>
  </figure>
</div>

Note:
A rank flips not because the measurement got noisier but because at k=1 you are asking about average correctness and at k=N you are asking how many questions the model can touch at all. A model that concentrates its successes wins the first and loses the second.

---

## Any sign pattern is achievable

<div class="pk-split even">
  <div class="pk-stack">
    <div class="pk-card rose">
      <h3>Universal construction</h3>
      <p>Fix N. Every sign vector in {&minus;1, +1}<sup>N</sup> is realized exactly by two integer histograms. Alternating signs give N &minus; 1 strict reversals on one path.</p>
    </div>
    <div class="pk-card green">
      <h3>What bounds it</h3>
      <p>The Pascal kernel is totally positive, so the transform is variation-diminishing:</p>
      <div class="pk-equation xs plain">
        \[ V(\Delta_1,\ldots,\Delta_N)\le V(\sigma_0,\ldots,\sigma_N) \]
      </div>
      <p>One crossing permits one reversal. Two or more need three separated correctness regimes.</p>
    </div>
  </div>
  <div class="pk-stack">
    <figure class="pk-figure pk-fig-short">
      <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/rank_dirichlet_landscape.webp" alt="Probability of at least two strict Pass@k ranking reversals and mean number of sign changes, over Dirichlet-sampled profile pairs" />
      <figcaption>20,000 Dirichlet-sampled profile pairs per cell.</figcaption>
    </figure>
    <table class="pk-table compact">
      <thead>
        <tr><th>N</th><th class="num">&alpha; = 0.1</th><th class="num">&alpha; = 0.3</th><th class="num">&alpha; = 1</th><th class="num">&alpha; = 3</th></tr>
      </thead>
      <tbody>
        <tr><td>8</td><td class="num">0.079</td><td class="num">0.055</td><td class="num">0.040</td><td class="num">0.040</td></tr>
        <tr><td>32</td><td class="num">0.172</td><td class="num">0.126</td><td class="num">0.110</td><td class="num">0.101</td></tr>
        <tr><td>128</td><td class="num pk-bad">0.269</td><td class="num pk-bad">0.221</td><td class="num pk-bad">0.193</td><td class="num pk-bad">0.186</td></tr>
      </tbody>
    </table>
    <div class="pk-eq-label">Pr[two or more strict reversals]. Monte Carlo standard error at most 0.0035.</div>
  </div>
</div>

Note:
The theorem is worst case and you could dismiss it as adversarial. The simulation is why you should not, and the exact small-case enumeration agrees: 3,077 of 827,541 unordered pairs at M=5, N=8. Strict patterns also survive perturbation, so reversals occupy open regions rather than measure-zero curves.

---

## Large budgets measure coverage, not accuracy

<div class="pk-split reverse">
  <div class="pk-stack">
    <div class="pk-equation sm">
      \[ \big(1-(1-p)^k\big)-p=(1-p)\big\{1-(1-p)^{k-1}\big\}\ \ge\ 0 \]
      <div class="pk-eq-label">the row-level gap, strictly positive for every k &gt; 1 whenever 0 &lt; p &lt; 1</div>
    </div>
    <div class="pk-equation sm">
      \[ \lim_{k\to\infty}\big(1-(1-p)^k\big)=\mathbf{1}\{p>0\} \]
    </div>
    <div class="pk-card orange">
      <h3>Three targets, not one</h3>
      <ul>
        <li><b>Pass@1</b> is exactly average accuracy.</li>
        <li><b>Pass@k</b> applies a concave saturation map to each question.</li>
        <li><b>Pass@&infin;</b> is Pr(<em>p</em> &gt; 0), the fraction ever solvable.</li>
      </ul>
    </div>
  </div>
  <figure class="pk-figure pk-fig-tall">
    <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/inf_population_functionals.webp" alt="Population question-level functionals: identity map for average accuracy, at-least-one-success transform for Pass@k, and a step function in the infinite-budget limit" />
    <figcaption>Average accuracy is the identity map on <em>p</em>, which is Pass@1. Every larger budget bends toward the step function at <em>p</em> = 0.</figcaption>
  </figure>
</div>

Note:
This gets waved away as an asymptotic curiosity. Read it as: any question with any nonzero chance of being solved eventually receives full credit. A model that is slightly capable everywhere beats one that is genuinely good on half the benchmark, once the budget is large enough. Over 200,000 zero-inflated-Beta pairs with exact closed-form targets, they disagree 28.5 percent of the time.

---

## Two independent losses, before any budget is chosen

<div class="pk-split reverse">
  <div class="pk-stack">
    <div class="pk-card purple">
      <h3>Row-sum factorization</h3>
      <p>Pass@k sees R only through the row counts. Banks sharing them are indistinguishable by the whole curve, and the number sharing one count vector is</p>
      <div class="pk-equation xs plain">
        \[ \prod_{i=1}^{M}\binom{N}{c_i} \]
      </div>
    </div>
    <div class="pk-card purple">
      <h3>Row-level saturation</h3>
      <p>The row functional increments by</p>
      <div class="pk-equation xs plain">
        \[ \psi_{k,N}(c+1)-\psi_{k,N}(c)=\frac{\binom{N-c-1}{k-1}}{\binom{N}{k}} \]
      </div>
      <p>which is zero once c exceeds N &minus; k, leaving <b>N &minus; k + 2</b> distinct row values. At k = N there are two: solved once and solved N times are merged.</p>
    </div>
  </div>
  <figure class="pk-figure pk-fig-tall">
    <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/nonid_row_resolution.webp" alt="Row-level Pass@k resolution at N=32: the map from success count to row value saturates earlier as k grows, and the resolution count is N minus k plus 2" />
    <figcaption>N = 32. Larger budgets saturate earlier. Panel B is the exact resolution count.</figcaption>
  </figure>
</div>

Note:
The two losses stack, and only the second gets any attention. People do worry that a high budget is coarse. Almost nobody notices that even the full curve cannot see whether the successes were the same three samples every time or three different ones: the row sum destroyed that before budgets entered the picture.

---

## The equivalence classes, counted exactly

<div class="pk-split even">
  <div class="pk-stack">
    <div class="pk-card blue">
      <h3>Identifiability rank of a budget set</h3>
      <p>Reporting L budgets leaves a local fiber of affine dimension</p>
      <div class="pk-equation xs plain">
        \[ N-L \]
      </div>
      <p>The full Pass@1:N path pins the histogram and still leaves the matrix unidentified.</p>
    </div>
    <table class="pk-table compact">
      <thead>
        <tr><th>Budget</th><th class="num">distinct values</th><th class="num">largest fiber</th><th class="num">matrix collision</th></tr>
      </thead>
      <tbody>
        <tr><td>k = 1</td><td class="num">101</td><td class="num">5,448</td><td class="num">5.6%</td></tr>
        <tr><td>k = 5</td><td class="num">1,436</td><td class="num">1,385</td><td class="num">0.7%</td></tr>
        <tr><td>k = 10</td><td class="num">11</td><td class="num pk-bad">92,378</td><td class="num pk-bad">98.1%</td></tr>
      </tbody>
    </table>
    <div class="pk-eq-label">Exact. M = N = 10, over all 184,756 row-success histograms.</div>
  </div>
  <div class="pk-stack">
    <figure class="pk-figure pk-fig-short">
      <img src="/assets/slides/2026-08-24-scorio-for-computer-science/figures/nonid_auxiliary_scatter.webp" alt="Auxiliary variation inside a single Pass@5 equivalence class: Pass@1, coverage, and row-count variance all range widely" />
      <figcaption>Inside one Pass@5 = 0.90 class: 1,385 histograms.</figcaption>
    </figure>
    <div class="pk-callout rose">
      One mid-budget score fits a model right 36% of the time and one right 90% of the time.
    </div>
  </div>
</div>

Note:
Use the middle row rather than the coverage endpoint, which is too easy to dismiss. Pass@5 is the best-resolved single budget here and still admits a class of 1,385 histograms whose single-sample accuracy spans more than half the unit interval.

---

## False positives compound, and selection does not fix them

<div class="pk-board">
  <div class="pk-rail">
    <div class="pk-rail-badge">04</div>
    <h3>Verifier &amp; selector</h3>
    <p>Acceptance replaces correctness. Write r for the true-positive rate and f for the false-positive rate.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>0.409</b>Rank disagreement at k = 250 between reported and returned correctness</div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split even">
      <div class="pk-stack">
        <div class="pk-equation xs">
          \[ a=rp+f(1-p),\qquad D^{\mathrm{obs}}_k=1-(1-a)^k \]
          <div class="pk-eq-label">acceptance rate, and the reported Pass@k built on it</div>
        </div>
        <div class="pk-card orange">
          <h3>False-positive amplification</h3>
          <p>At p = 0 the reported value tends to one while true discovery is zero. To first order the distortion is</p>
          <div class="pk-equation xs plain">
            \[ k(1-p)^{k-1}\delta+O(\delta^2),\qquad \delta=a-p \]
          </div>
        </div>
      </div>
      <div class="pk-stack">
        <div class="pk-equation xs">
          \[ S_k=\big\{1-(1-a)^k\big\}\frac{rp}{a}\ \longrightarrow\ \frac{rp}{rp+f(1-p)} \]
          <div class="pk-eq-label">returned correctness under uniform selection among accepted candidates</div>
        </div>
        <table class="pk-table compact">
          <thead><tr><th>p / f, k=100</th><th class="num">reported</th><th class="num">true</th><th class="num">false</th></tr></thead>
          <tbody>
            <tr><td>0 / 0.001</td><td class="num">0.095</td><td class="num">0.000</td><td class="num pk-bad">0.095</td></tr>
            <tr><td>0.01 / 0.010</td><td class="num">0.866</td><td class="num">0.435</td><td class="num pk-bad">0.431</td></tr>
            <tr><td>0.05 / 0.010</td><td class="num">0.998</td><td class="num">0.839</td><td class="num">0.159</td></tr>
          </tbody>
        </table>
        <div class="pk-callout orange">
          Reported Pass@k tends to one whenever acceptance is possible. Returned correctness tends only to verifier precision.
        </div>
      </div>
    </div>
  </div>
</div>

Note:
The two gaps are separate. Amplification is about the label; the selector gap is about what a downstream system actually hands back. As k grows acceptance becomes near-certain, so the only thing still distinguishing systems is the precision of the accepted set.

---

## The marginal does not identify the discovery curve

<div class="pk-board green">
  <div class="pk-rail">
    <div class="pk-rail-badge">05</div>
    <h3>Dependence</h3>
    <p>The budget axis is a test-time-scaling axis only if the attempts are independent draws.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>19.0</b>Independent-equivalent budget at nominal k = 64, correlation 0.10</div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split even">
      <div class="pk-stack">
        <div class="pk-equation xs">
          \[ p\ \le\ D_k\ \le\ \min\{1,\ kp\} \]
          <div class="pk-eq-label">sharp Fr&eacute;chet bounds: equal marginals, wildly different discovery</div>
        </div>
        <div class="pk-card green">
          <h3>Positive dependence depresses discovery</h3>
          <p>Under an exchangeable de Finetti mixture with latent success probability P,</p>
          <div class="pk-equation xs plain">
            \[ D_k=1-\mathbb{E}\big[(1-P)^k\big]\ \le\ 1-(1-p)^k \]
          </div>
          <p>by convexity, with equality only when P is degenerate.</p>
        </div>
      </div>
      <div class="pk-stack">
        <div class="pk-equation xs">
          \[ k_{\mathrm{eff}}=\frac{\log(1-D_k)}{\log(1-p)} \]
          <div class="pk-eq-label">the independent budget giving the same discovery</div>
        </div>
        <table class="pk-table compact">
          <thead><tr><th>&rho;</th><th class="num">discovery</th><th class="num">k<sub>eff</sub></th><th class="num">fraction</th></tr></thead>
          <tbody>
            <tr><td>0.00</td><td class="num">0.962</td><td class="num">64.0</td><td class="num pk-win">1.000</td></tr>
            <tr><td>0.10</td><td class="num">0.623</td><td class="num">19.0</td><td class="num">0.297</td></tr>
            <tr><td>0.25</td><td class="num">0.391</td><td class="num">9.7</td><td class="num pk-bad">0.151</td></tr>
          </tbody>
        </table>
        <div class="pk-eq-label">Beta-binomial mixture, marginal Pass@1 fixed at 0.05, nominal k = 64.</div>
      </div>
    </div>
  </div>
</div>

Note:
The Fréchet bounds say the marginal alone leaves discovery anywhere between p and k times p. Jensen says any latent variation across a batch puts the real curve strictly below the i.i.d. curve, so extrapolating from Pass@1 always overstates the gain. This is the one mode intervals cannot rescue: a mislabelled budget axis is a protocol problem.

---

## A utility with zero cost for every failure

<div class="pk-board orange">
  <div class="pk-rail">
    <div class="pk-rail-badge">06</div>
    <h3>Harm blindness</h3>
    <p>Outcomes are correct, harmful, or benign-incorrect. Pass@k reads only the first.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>0.994</b>Pass@100 at p<sub>C</sub> = 0.05, invariant in p<sub>H</sub></div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split even">
      <div class="pk-stack">
        <div class="pk-equation xs">
          \[ C_k=1-(1-p_C)^k,\qquad H_k=1-(1-p_H)^k \]
          <div class="pk-eq-label">discovery (this is Pass@k) and harm exposure, driven by different rates</div>
        </div>
        <div class="pk-card orange">
          <h3>Exact blindness</h3>
          <p>For fixed p<sub>C</sub> and k, every p<sub>H</sub> in [0, 1 &minus; p<sub>C</sub>] gives the same Pass@k, while harm exposure ranges up to</p>
          <div class="pk-equation xs plain">
            \[ 1-p_C^{\,k} \]
          </div>
        </div>
      </div>
      <div class="pk-stack">
        <div class="pk-equation xs">
          \[ U_\lambda(k)=1-(1-p_C)^k-\lambda\big\{1-(1-p_H)^k\big\} \]
          <div class="pk-eq-label">Pass@k is the special case &lambda; = 0</div>
        </div>
        <table class="pk-table compact">
          <thead><tr><th class="num">p<sub>H</sub></th><th class="num">k</th><th class="num">Pass@k</th><th class="num">Pr(harm)</th><th class="num">U</th></tr></thead>
          <tbody>
            <tr><td class="num">0.00</td><td class="num">100</td><td class="num">0.994</td><td class="num">0.000</td><td class="num pk-win">0.994</td></tr>
            <tr><td class="num">0.01</td><td class="num">50</td><td class="num">0.923</td><td class="num">0.395</td><td class="num">0.528</td></tr>
            <tr><td class="num">0.01</td><td class="num">100</td><td class="num">0.994</td><td class="num">0.634</td><td class="num pk-bad">0.360</td></tr>
            <tr><td class="num">0.10</td><td class="num">10</td><td class="num">0.401</td><td class="num">0.651</td><td class="num pk-bad">&minus;0.250</td></tr>
          </tbody>
        </table>
        <div class="pk-callout orange">
          At a 1% harm rate the utility-optimal budget is 50, not 100.
        </div>
      </div>
    </div>
  </div>
</div>

Note:
When failed attempts really are discarded at zero cost, say so and skip this mode. Otherwise the honest object is a utility with a declared cost, and Pass@k is the corner of it where that cost is set to zero.

---

## The variance survives the averaging

<div class="pk-board purple">
  <div class="pk-rail">
    <div class="pk-rail-badge">07</div>
    <h3>Finite-bank noise</h3>
    <p>It averages over all C(N, k) subsets, which reuse the same N generated candidates.</p>
    <div class="pk-rail-foot">
      <div class="pk-rail-fact"><b>0.312</b>False-ranking probability at k = N = 80 with M = 200</div>
    </div>
  </div>
  <div class="pk-board-main">
    <div class="pk-split even">
      <div class="pk-stack">
        <div class="pk-equation xs">
          \[ \operatorname{Var}(\widehat Q_{N,k}\mid p)=\mathbb{E}\big[(1-p)^{2k-S}\big]-(1-p)^{2k} \]
          <div class="pk-eq-label">S is the hypergeometric overlap of two random k-subsets</div>
        </div>
        <div class="pk-card purple">
          <h3>The two endpoints</h3>
          <p>At k = 1 this is the sample-mean variance p(1 &minus; p)/N. At k = N the estimator collapses to an indicator:</p>
          <div class="pk-equation xs plain">
            \[ \operatorname{Var}=Q_N(1-Q_N) \]
          </div>
          <p>One Bernoulli draw per row, whatever the formula averages over.</p>
        </div>
      </div>
      <div class="pk-stack">
        <table class="pk-table compact">
          <thead><tr><th>M</th><th class="num">k = 1</th><th class="num">k = 10</th><th class="num">k = 40</th><th class="num">k = 80</th></tr></thead>
          <tbody>
            <tr><td>50</td><td class="num">0.158</td><td class="num">0.161</td><td class="num">0.226</td><td class="num pk-bad">0.403</td></tr>
            <tr><td>100</td><td class="num">0.078</td><td class="num">0.081</td><td class="num">0.144</td><td class="num pk-bad">0.365</td></tr>
            <tr><td>200</td><td class="num pk-win">0.022</td><td class="num">0.024</td><td class="num">0.066</td><td class="num pk-bad">0.312</td></tr>
          </tbody>
        </table>
        <div class="pk-eq-label">Pr(the weaker model is ranked first), N = 80, p<sub>A</sub> = 0.050 against p<sub>B</sub> = 0.055.</div>
        <div class="pk-callout purple">
          Separate from mode 01: the population ranking is stable and the noise still reverses it.
        </div>
      </div>
    </div>
  </div>
</div>

Note:
Quadrupling M takes the k=1 false-ranking probability from 24 percent to 2, and barely touches k=80: 43 to 31. At the full budget each row contributes one Bernoulli draw, so the extra subsets buy no extra evidence.

---

## Seven symptoms, one missing object

<p class="pk-note">Every failure is a coordinate of the discovery&ndash;stability tail surface going unreported, or measured on the wrong bank.</p>

<div class="pk-split">
  <div class="pk-stack">
    <div class="pk-equation sm">
      \[ S_{k,t}=\frac{1}{M}\sum_{i=1}^{M}\Pr(\mathrm{Bin}(k,p_i)\ge t) \]
    </div>
    <div class="pk-grid two">
      <div class="pk-card blue">
        <h3>Left edge, t = 1</h3>
        <p>Discovery. The population Pass@k.</p>
      </div>
      <div class="pk-card green">
        <h3>Right edge, t = k</h3>
        <p>Stability. Every attempt works.</p>
      </div>
      <div class="pk-card orange">
        <h3>Area</h3>
        <p>Average single-sample accuracy.</p>
      </div>
      <div class="pk-card purple">
        <h3>Slope</h3>
        <p>How fast repeatability decays.</p>
      </div>
    </div>
    <div class="pk-equation xs">
      \[ \frac{1}{k}\sum_{t=1}^{k}S_{k,t}=\frac{1}{M}\sum_{i=1}^{M}p_i \]
      <div class="pk-eq-label">accuracy is the area, so same-accuracy comparisons isolate allocation</div>
    </div>
  </div>
  <div class="pk-stack">
    <div class="pk-card rose">
      <h3>Why the sensitivity differs</h3>
      <p>The derivative of the tail kernel in <em>p</em> is a Beta(<em>t</em>, <em>k</em> &minus; <em>t</em> + 1) density. Discovery localizes near <em>p</em> = 0, stability near <em>p</em> = 1.</p>
    </div>
    <div class="pk-callout green">
      TailPass@k reports the profile with uncertainty: closed-form posterior mean and covariance, Monte Carlo credible intervals.
    </div>
  </div>
</div>

Note:
The tail surface is not a new measurement, it is the same bank read on both axes. The Beta-density fact is why the choice of threshold is not a matter of taste: it decides which slice of the difficulty distribution the metric can resolve at all.

---

## What each mode needs from the report

<table class="pk-table compact">
  <thead>
    <tr><th style="width: 26%;">Failure</th><th style="width: 34%;">What is missing</th><th style="width: 40%;">What TailPass@k reports</th></tr>
  </thead>
  <tbody>
    <tr><td><b>01 Rank instability</b></td><td>The budget path.</td><td>Profiles across budgets, so a reversal is visible.</td></tr>
    <tr><td><b>02 Coverage, not accuracy</b></td><td>Discovery and accuracy, merged.</td><td>The left edge and the area, separately.</td></tr>
    <tr><td><b>03 Non-identifiability</b></td><td>The threshold axis.</td><td>The full profile, whose interior thresholds separate what a scalar merges.</td></tr>
    <tr><td><b>04 Verifier &amp; selector</b></td><td>Acceptance read as correctness.</td><td>Declared outcome categories.</td></tr>
    <tr><td><b>05 Dependence</b></td><td>Independence, assumed.</td><td>Uncertainty on the profile, plus a stated generation protocol.</td></tr>
    <tr><td><b>06 Harm blindness</b></td><td>Any cost for a failure.</td><td>Categorical outcomes with an explicit utility.</td></tr>
    <tr><td><b>07 Finite-bank noise</b></td><td>The uncertainty of the estimate.</td><td>Posterior means, covariances, and credible intervals.</td></tr>
  </tbody>
</table>

Note:
Be honest about the split. Modes 01 to 03 are solved by reporting more coordinates of the same object. Four through seven need a richer outcome space and honest uncertainty, and mode five only halfway: intervals tell you the estimate is uncertain, not that the budget axis was mislabelled.

---

## What to carry out of this

<div class="pk-takeaway">
  <div class="pk-card blue">
    <div class="pk-take-num">1</div>
    <h3>The failures are exact</h3>
    <p>Every mode has a closed-form statement and a counterexample, down to an exact enumeration over 184,756 histograms.</p>
  </div>
  <div class="pk-card orange">
    <div class="pk-take-num">2</div>
    <h3>And they are common</h3>
    <p>Two or more reversals in 20% of random pairs at N = 128. Accuracy and large-budget discovery disagree on 28.5% of pairs.</p>
  </div>
  <div class="pk-card green">
    <div class="pk-take-num">3</div>
    <h3>One object fixes most of it</h3>
    <p>Report the profile with uncertainty instead of one thresholded scalar. Pass@k stays, as the left edge.</p>
  </div>
</div>

<div class="pk-callout">
  Minimum honest report: Pass@1 apart from coverage, the budget path when you rank, verifier precision, harm rates, and an interval on all of it.
</div>

Note:
If someone remembers one thing, make it the callout. You do not have to adopt anything from this paper to stop making these mistakes; you have to stop reporting a single thresholded scalar as if it summarised the run.

---

## Thank you

<div class="pk-split">
  <div class="pk-stack">
    <div class="pk-card">
      <h3>Success Has a Shape: TailPass@k for Repeated-Sampling Evaluation</h3>
      <p>Mohsen Hariri and Vipin Chaudhary, Case Western Reserve University.</p>
      <p>The failure-mode analysis in this talk is Appendices F and G.</p>
    </div>
    <div class="pk-card green">
      <h3>Contact</h3>
      <p><b>mohsen.hariri@case.edu</b><br />mohsenhariri.github.io</p>
    </div>
  </div>
  <div class="pk-stack">
    <div class="pk-card">
      <h3>Acknowledgments</h3>
      <p>Michael Hinczewski, for helpful discussions.</p>
      <p>Supported in part by NSF awards 2117439, 2112606, and 2320952.</p>
    </div>
  </div>
</div>

Note:
Questions I expect: whether the reversals matter at realistic N, whether the coverage limit is just an asymptotic artifact, and whether the posterior is worth the cost. Answers: about one in five random pairs at N=128; no, the ranking inversion happens at k=7; and the closed forms make the posterior mean and covariance cheap.

Navigation