LLM Research Directions (Slides)

# Day 3: Attention Under the Hood & LLM Research Directions

<div style="margin-top: 1.8em; text-align: center; opacity: 0.9;">
SCIPE Workshop on Large Language Models
</div>

<div style="margin-top: 2.0em; opacity: 0.75; font-size: 0.85em; text-align: center;">
Case Western Reserve University • January 18, 2026
</div>

---

.reveal .c-grid { display: grid; gap: 1.2rem; }
  .reveal .c-grid-2 { grid-template-columns: 1fr 1fr; }
  .reveal .c-grid-3 { grid-template-columns: 1fr 1fr 1fr; }
  .reveal .c-card {
    background: var(--card-bg);
    border: 1px solid var(--card-border);
    border-radius: 18px;
    padding: 1.15rem 1.25rem;
    box-shadow: 0 16px 34px rgba(15,23,42,0.07);
  }
  .reveal .c-kicker {
    font-size: 0.55em;
    letter-spacing: 0.14em;
    text-transform: uppercase;
    opacity: 0.7;
    margin-bottom: 0.4rem;
  }
  .reveal .c-muted { color: var(--muted); }
  .reveal .c-titleRow { display: flex; align-items: center; gap: 0.6rem; margin-bottom: 0.65rem; }
  .reveal .c-titleRow strong { font-weight: 700; }

.reveal .c-pill {
    display: inline-flex;
    align-items: center;
    gap: 0.4rem;
    padding: 0.25rem 0.55rem;
    border-radius: 999px;
    font-size: 0.55em;
    font-weight: 700;
    border: 1px solid rgba(15,23,42,0.14);
    background: rgba(255,255,255,0.75);
    letter-spacing: 0.04em;
    text-transform: uppercase;
  }
  .reveal .c-pill.tool { border-color: rgba(2,132,199,0.24); background: var(--tool-soft); color: rgba(2,132,199,1); }
  .reveal .c-pill.rag  { border-color: rgba(249,115,22,0.26); background: var(--rag-soft); color: rgba(194,65,12,1); }
  .reveal .c-pill.mix  { border-color: rgba(16,185,129,0.26); background: var(--mix-soft); color: rgba(5,150,105,1); }

.reveal .c-list { margin: 0.65rem 0 0 0; padding-left: 1.15rem; font-size: 0.82em; }
  .reveal .c-list li { margin: 0.28rem 0; }

.reveal .c-callout {
    border-left: 6px solid rgba(2,132,199,0.55);
    background: rgba(2,132,199,0.06);
    padding: 0.9rem 1.0rem;
    border-radius: 16px;
    font-size: 0.82em;
  }
  .reveal .c-callout.warn { border-left-color: rgba(249,115,22,0.70); background: rgba(249,115,22,0.08); }
  .reveal .c-callout.ok   { border-left-color: rgba(16,185,129,0.70); background: rgba(16,185,129,0.08); }

.reveal .c-flow { display: flex; align-items: stretch; justify-content: center; gap: 0.65rem; flex-wrap: wrap; margin-top: 1.0rem; }
  .reveal .c-node {
    min-width: 240px;
    padding: 0.85rem 0.95rem;
    border-radius: 16px;
    border: 1px solid rgba(15,23,42,0.12);
    background: rgba(255,255,255,0.72);
    text-align: center;
    font-size: 0.78em;
  }
  .reveal .c-node .small { display: block; margin-top: 0.35rem; font-size: 0.78em; opacity: 0.75; }
  .reveal .c-arrow { display: flex; align-items: center; font-size: 1.2em; opacity: 0.4; padding: 0 0.15rem; }

.reveal .c-pre {
    margin: 0.5rem 0 0 0;
    padding: 0.85rem 0.9rem;
    border-radius: 16px;
    background: rgba(15,23,42,0.05);
    border: 1px solid rgba(15,23,42,0.10);
    font-size: 0.68em;
    overflow: hidden;
  }
  .reveal .c-pre code { white-space: pre; }

.reveal .c-svg {
    width: 100%;
    height: auto;
    display: block;
    margin: 0 auto;
  }

.reveal .c-foot { font-size: 0.55em; opacity: 0.75; margin-top: 0.85rem; }

@media (max-width: 1100px) {
    .reveal .c-grid-3 { grid-template-columns: 1fr; }
    .reveal .c-grid-2 { grid-template-columns: 1fr; }
  }
</style>

## Attention Under the Hood

<div style="display: flex; align-items: center; justify-content: space-between; gap: 1.2rem; margin-top: 0.8rem;">
  <div style="text-align: left;">
    
    
  </div>
  
</div>

<div class="c-card" style="margin-top: 1.0rem; padding: 1.05rem 1.15rem;">
  <svg class="c-svg" viewBox="0 0 760 320" role="img" aria-label="Stylized engine bay diagram showing Q, K, V flowing into softmax and producing context">
    <defs>
      <linearGradient id="hoodGrad" x1="0" y1="0" x2="1" y2="1">
        <stop offset="0%" stop-color="rgba(2,132,199,0.10)"/>
        <stop offset="55%" stop-color="rgba(249,115,22,0.08)"/>
        <stop offset="100%" stop-color="rgba(16,185,129,0.10)"/>
      </linearGradient>
      <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
        <path d="M 0 0 L 10 5 L 0 10 z" fill="rgba(15,23,42,0.42)"/>
      </marker>
    </defs>
    
    <path d="M170 118 L380 34 L590 118" fill="none" stroke="rgba(15,23,42,0.22)" stroke-width="4" stroke-linecap="round"/>
    <path d="M170 118 L130 152" fill="none" stroke="rgba(15,23,42,0.18)" stroke-width="3" stroke-linecap="round"/>
    <path d="M590 118 L630 152" fill="none" stroke="rgba(15,23,42,0.18)" stroke-width="3" stroke-linecap="round"/>
    
    <rect x="120" y="120" width="520" height="170" rx="26" fill="url(#hoodGrad)" stroke="rgba(15,23,42,0.14)" stroke-width="2"/>
    
    <g font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Helvetica, Arial" font-size="18" font-weight="700">
      <g opacity="0.9">
        <rect x="250" y="78" width="54" height="30" rx="10" fill="rgba(255,255,255,0.78)" stroke="rgba(15,23,42,0.14)"/>
        <text x="277" y="99" text-anchor="middle" fill="rgba(15,23,42,0.70)">x₁</text>
        <rect x="316" y="78" width="54" height="30" rx="10" fill="rgba(255,255,255,0.78)" stroke="rgba(15,23,42,0.14)"/>
        <text x="343" y="99" text-anchor="middle" fill="rgba(15,23,42,0.70)">x₂</text>
        <rect x="382" y="78" width="54" height="30" rx="10" fill="rgba(255,255,255,0.78)" stroke="rgba(15,23,42,0.14)"/>
        <text x="409" y="99" text-anchor="middle" fill="rgba(15,23,42,0.70)">x₃</text>
      </g>
    </g>
    
    <g font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Helvetica, Arial" font-size="22" font-weight="800">
      <circle cx="250" cy="198" r="34" fill="rgba(2,132,199,0.14)" stroke="rgba(2,132,199,0.45)" stroke-width="3"/>
      <text x="250" y="206" text-anchor="middle" fill="rgba(2,132,199,0.95)">Q</text>
      <circle cx="380" cy="198" r="34" fill="rgba(249,115,22,0.14)" stroke="rgba(194,65,12,0.50)" stroke-width="3"/>
      <text x="380" y="206" text-anchor="middle" fill="rgba(194,65,12,0.95)">K</text>
      <circle cx="510" cy="198" r="34" fill="rgba(16,185,129,0.14)" stroke="rgba(5,150,105,0.55)" stroke-width="3"/>
      <text x="510" y="206" text-anchor="middle" fill="rgba(5,150,105,0.95)">V</text>
    </g>
    
    <path d="M277 108 L255 160" stroke="rgba(15,23,42,0.35)" stroke-width="2.5" fill="none" marker-end="url(#arrow)"/>
    <path d="M343 108 L380 160" stroke="rgba(15,23,42,0.35)" stroke-width="2.5" fill="none" marker-end="url(#arrow)"/>
    <path d="M409 108 L505 160" stroke="rgba(15,23,42,0.35)" stroke-width="2.5" fill="none" marker-end="url(#arrow)"/>
    
    <rect x="334" y="236" width="92" height="30" rx="12" fill="rgba(255,255,255,0.78)" stroke="rgba(15,23,42,0.16)"/>
    <text x="380" y="257" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Helvetica, Arial" font-size="16" font-weight="750" fill="rgba(15,23,42,0.70)">softmax</text>
    
    <path d="M250 232 L340 246" stroke="rgba(15,23,42,0.32)" stroke-width="2.5" fill="none" marker-end="url(#arrow)"/>
    <path d="M380 232 L380 236" stroke="rgba(15,23,42,0.32)" stroke-width="2.5" fill="none" marker-end="url(#arrow)"/>
    <path d="M544 198 L610 194" stroke="rgba(15,23,42,0.32)" stroke-width="2.5" fill="none" marker-end="url(#arrow)"/>
    
    <rect x="610" y="182" width="106" height="46" rx="16" fill="rgba(255,255,255,0.80)" stroke="rgba(15,23,42,0.16)"/>
    <text x="663" y="205" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Helvetica, Arial" font-size="16" font-weight="800" fill="rgba(15,23,42,0.72)">context</text>
    <text x="663" y="224" text-anchor="middle" font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Helvetica, Arial" font-size="13" font-weight="650" fill="rgba(15,23,42,0.55)">Σ αᵢ·vᵢ</text>
    <path d="M426 251 L610 206" stroke="rgba(15,23,42,0.35)" stroke-width="3" fill="none" marker-end="url(#arrow)"/>
  </svg>
  <div class="c-muted" style="margin-top: 0.75rem; font-size: 0.76em; text-align: center;">
    <strong>Q / K / V</strong> projections • masking • multi-head • <span style="white-space: nowrap;">KV-cache</span>
  </div>
</div>

<div class="c-muted" style="margin-top: 0.75rem; text-align: right; font-size: 0.65em;">
  Next: <strong>LLMs &lt;-&gt; Your Field</strong>
</div>

---

## LLMs &lt;-&gt; Your Field

<div class="c-muted" style="margin-top: 0.35rem; font-size: 0.82em;">
Two complementary ways to do research with LLMs: <strong>apply</strong> them to your domain, and <strong>import</strong> domain ideas back into models.
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-titleRow">
      <span class="c-pill tool">LLM -&gt; Field</span>
      <strong>Apply LLMs in your workflow</strong>
    </div>
    <ul class="c-list" style="font-size: 0.78em;">
      <li><strong>Automate</strong> extraction -&gt; labeling -&gt; analysis -&gt; writing</li>
      <li><strong>Augment</strong> experts: search, summarization, explanation, code</li>
      <li><strong>Integrate tools</strong>: RAG + APIs + simulators + domain software</li>
      <li><strong>Make it research</strong>: measure quality, time, cost, and failure modes</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-titleRow">
      <span class="c-pill mix">Field -&gt; LLM</span>
      <strong>Use domain knowledge to improve LLMs</strong>
    </div>
    <ul class="c-list" style="font-size: 0.78em;">
      <li><strong>Inductive bias</strong>: constraints, invariances, structure (graphs, equations)</li>
      <li><strong>New objectives</strong>: denoising/diffusion (physics), control, Bayesian updates</li>
      <li><strong>Better evaluation</strong>: benchmarks that mirror real-world domain errors</li>
      <li><strong>New models</strong>: reinterpret a field process as a learning/sampling algorithm</li>
    </ul>
    <div class="c-foot">
      Example: diffusion processes in physics -&gt; diffusion models; similar "process -&gt; model" transfers can inspire new LLMs.
    </div>
  </div>
</div>

<div class="c-callout" style="margin-top: 1.0rem;">
  <strong>Research flywheel:</strong> build a domain tool -&gt; collect failures -&gt; turn them into a benchmark -&gt; improve the model -&gt; repeat.
</div>

<div class="c-muted" style="margin-top: 0.75rem; text-align: right; font-size: 0.65em;">
  Next: <strong>LLM Research Directions</strong>
</div>

---

## LLM Research Directions

<div class="c-grid c-grid-2" style="margin-top: 1.2rem;">
  <div class="c-card">
    <div class="c-kicker">Statistical Methods • Evaluation</div>
    <div class="c-titleRow"><strong>Model evaluation (w/ Department of Physics)</strong></div>
    <ul class="c-list" style="font-size: 0.72em;">
      <li><strong>Ranking Reasoning LLMs under Test-Time Scaling</strong> <span class="c-muted">(under review; PDF upon request)</span></li>
      <li><a href="https://arxiv.org/abs/2510.04265" target="_blank" rel="noopener noreferrer">Don’t Pass@k: A Bayesian Framework for Large Language Model Evaluation</a></li>
      <li><a href="https://mohsenhariri.github.io/posts/2025-10-22-simulating-llm-answers-to-eval-datasets.html" target="_blank" rel="noopener noreferrer">Simulating LLM Answers to Evaluation Datasets</a></li>
      <li><a href="https://mohsenhariri.github.io/posts/2025-10-22-item-response-theory-llms.html" target="_blank" rel="noopener noreferrer">Simulating LLM Evaluation Datasets Using Psychometric Models</a></li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Systems • Efficiency</div>
    <div class="c-titleRow"><strong>Compression &amp; quantization</strong></div>
    <ul class="c-list" style="font-size: 0.72em;">
      <li><a href="https://arxiv.org/abs/2504.11651" target="_blank" rel="noopener noreferrer">70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float (DFloat11)</a></li>
      <li><a href="https://arxiv.org/abs/2502.15075" target="_blank" rel="noopener noreferrer">Quantize What Counts: More for Keys, Less for Values</a></li>
      <li><a href="https://mohsenhariri.github.io/posts/2025-10-28-data-type-entropy.html" target="_blank" rel="noopener noreferrer">Entropy of bfloat16: 8 Bits Are Doing 2.6 Bits of Work</a></li>
      <li><a href="https://mohsenhariri.github.io/posts/2025-11-17-bf16-evolution.html" target="_blank" rel="noopener noreferrer">Entropy of bfloat16 During Training: How Optimizers Shape Weight Distributions</a></li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Medical • Imaging</div>
    <div class="c-titleRow"><strong>Medical (w/ Cleveland Clinic &amp; University Hospitals)</strong></div>
    <ul class="c-list" style="font-size: 0.72em;">
      <li><a href="https://arxiv.org/abs/2408.04762" target="_blank" rel="noopener noreferrer">Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2</a></li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Voice • Speech</div>
    <div class="c-titleRow"><strong>Voice &amp; speech (w/ Department of Psychological Sciences)</strong></div>
    <ul class="c-list" style="font-size: 0.72em;">
      <li><a href="https://journals.sagepub.com/doi/10.1177/23312165251403080" target="_blank" rel="noopener noreferrer">Masked-speech Recognition Using Human and Synthetic Cloned Speech</a></li>
    </ul>
  </div>
</div>

---

## Don’t Pass@k
### A Bayesian framework for LLM evaluation

<div class="c-callout ok" style="margin-top: 1.0rem;">
  <strong>Key idea:</strong> replace <code>Pass@k</code> with posterior estimates (<code>μ</code>) and credible intervals (<code>σ</code>) — stable rankings under small compute budgets.
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Why Pass@k breaks</div>
    <ul class="c-list">
      <li><strong>Unstable</strong> when trials are limited (<span class="c-muted">small <code>N</code>, <code>k≈N</code></span>)</li>
      <li><strong>Misleading rankings</strong> from seed / decoding variance</li>
      <li><strong>No uncertainty</strong> in closed form (<span class="c-muted">often relies on bootstrap</span>)</li>
      <li><strong>Binary-only</strong> — hard to express rubric signals</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">What Bayes@N provides</div>
    <ul class="c-list">
      <li><strong>Posterior mean</strong> <code>μ</code> for performance (any weighted rubric)</li>
      <li><strong>Credible intervals</strong> <code>μ ± CI</code> for “is the gap real?”</li>
      <li><strong>Decision rule</strong>: don’t declare a winner when CIs overlap</li>
      <li><strong>Optional priors</strong> to reuse evidence from earlier runs</li>
    </ul>
    <div class="c-foot">
      Paper: <a href="https://arxiv.org/abs/2510.04265" target="_blank" rel="noopener noreferrer">arXiv:2510.04265</a>
    </div>
  </div>
</div>

---

## Bayes@N: from outcomes to a posterior

<div class="c-flow" style="margin-top: 0.9rem;">
  <div class="c-node"><strong>Run model</strong><span class="small"><code>N</code> trials × <code>M</code> questions</span></div>
  <div class="c-arrow">→</div>
  <div class="c-node"><strong>Categorize</strong><span class="small">correct / partial / format / …</span></div>
  <div class="c-arrow">→</div>
  <div class="c-node"><strong>Dirichlet posterior</strong><span class="small">closed-form moments</span></div>
  <div class="c-arrow">→</div>
  <div class="c-node"><strong>Report</strong><span class="small"><code>μ</code>, <code>σ</code>, CI, confidence</span></div>
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Data model</div>
    <ul class="c-list">
      <li>Results matrix <code>R ∈ {0..C}^{M×N}</code> (categorical outcomes)</li>
      <li>Per-question probabilities <code>π<sub>α</sub> ∈ Δ<sup>C</sup></code> with a Dirichlet prior</li>
      <li>Any metric is a <strong>weighted rubric</strong> <code>w</code> over categories</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Closed-form estimator</div>
    $\nu_{\alpha k} = 1 + \mathrm{count}_k(R_\alpha) [+ \mathrm{count}_k(R^0_\alpha)]$<br>
    $T = 1 + C + D + N$<br>
    $\mu = w_0 + \frac{1}{M T} \sum_\alpha \sum_k \nu_{\alpha k}(w_k - w_0)$<br>
    $\sigma = \mathrm{closedform\ from\ Dirichlet\ moments}$
  </div>
</div>

---

## Rankings with uncertainty

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Decision rule</div>
    <ul class="c-list">
      <li>Compute CIs from <code>μ</code> and <code>σ</code></li>
      <li><strong>Only claim a win</strong> when intervals don’t overlap</li>
      <li>Pairwise ranking confidence (normal approx. for large <code>M</code>):</li>
    </ul>
    $z = \frac{|\mu - \mu'|}{\sqrt{\sigma^2 + \sigma'^2}}$<br>
    $\rho = 0.5(1 + \erf\left(\frac{z}{\sqrt{2}}\right))$
  </div>

<div class="c-card">
    <div class="c-kicker">Useful facts</div>
    <ul class="c-list">
      <li><strong>Uniform prior</strong>: Bayes ordering ≡ <code>avg@N</code> (aka <code>Pass@1</code>)</li>
      <li><strong>No bootstrapping</strong> needed for uncertainty estimates</li>
      <li><strong>Online-friendly</strong>: widen/narrow CIs as new trials arrive</li>
      <li><strong>Prior evidence</strong>: reuse older runs (<code>R^0</code>) when justified</li>
    </ul>
  </div>
</div>

---

## Empirical: faster convergence, steadier leaderboards

<div class="c-grid c-grid-2" style="margin-top: 0.9rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Convergence to “gold standard” ranking</div>
    <img src="/assets/slides/2025-11-19-bayesian-inference-llms/four_panel_kendall_boot_10000_top_p.webp" alt="Kendall’s tau vs number of trials for Bayes@N and pass@k variants" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Average Kendall’s τ vs trial budget <code>N</code> across AIME’24/’25, HMMT’25, BrUMO’25.</div>
  </div>

<div class="c-card">
    <div class="c-kicker">Takeaways</div>
    <ul class="c-list">
      <li><strong>Bayes@N</strong> reaches high τ with fewer trials than Pass@k variants</li>
      <li>On hard sets (e.g., AIME’25), <strong>uncertainty matters</strong>: don’t over-interpret tiny gaps</li>
      <li>Categorical rubrics make trade-offs explicit (format, partial credit, verifier, …)</li>
      <li>Practical protocol: <strong>report <code>μ</code> + CI</strong>, then spend compute only where CIs are wide</li>
    </ul>
  </div>
</div>

---

## Ranking Reasoning LLMs under Test-Time Scaling
### Robust rankings when you sample many answers

<div class="c-callout ok" style="margin-top: 1.0rem;">
  <strong>Key idea:</strong> under test-time scaling, evaluation is a repeated-sampling problem — compare ranking rules by <strong>stability</strong> (low budget) and <strong>convergence</strong> (high budget).
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Setting</div>
    <ul class="c-list">
      <li><strong>Test-time scaling</strong>: sample <code>N</code> outputs per prompt</li>
      <li>Dense benchmark: <code>L</code> models × <code>M</code> questions × <code>N</code> trials</li>
      <li>Binary outcome tensor <code>R ∈ {0,1}^{L×M×N}</code> (solved / not)</li>
      <li>Question: <strong>which ranking method</strong> is reliable at small <code>N</code>?</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">What the paper does</div>
    <ul class="c-list">
      <li>Unifies ranking families via pointwise / pairwise / setwise transforms of <code>R</code></li>
      <li>Evaluates <strong>72 ranking methods</strong> on <strong>20 reasoning LLMs</strong> × 4 olympiad math benchmarks (<code>N≤80</code>)</li>
      <li>Introduces a protocol: <strong>consistency</strong> (trial-to-trial) + <strong>convergence</strong> (as <code>N</code> grows)</li>
      <li>Releases <strong>scorio</strong>: a library of ranking methods for LLM evaluation</li>
    </ul>
  </div>
</div>

---

## From trials to rankings: three representations

<div class="c-flow" style="margin-top: 0.9rem;">
  <div class="c-node"><strong>Raw data</strong><span class="small"><code>R ∈ {0,1}^{L×M×N}</code></span></div>
  <div class="c-arrow">→</div>
  <div class="c-node"><strong>Pointwise</strong><span class="small">per-(model,question) rates</span></div>
  <div class="c-arrow">→</div>
  <div class="c-node"><strong>Pairwise</strong><span class="small">win/tie counts</span></div>
  <div class="c-arrow">→</div>
  <div class="c-node"><strong>Setwise / Graph</strong><span class="small">votes, spectral ranking</span></div>
</div>

<div class="c-grid c-grid-3" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Pointwise</div>
    <ul class="c-list">
      <li>Solve rate <code>p̂<sub>lm</sub> = (1/N)∑<sub>n</sub> R<sub>lmn</sub></code></li>
      <li>Rank by means / Bayesian estimates</li>
      <li>IRT / Rasch models (latent ability + difficulty)</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Pairwise</div>
    <ul class="c-list">
      <li>Win/tie counts between models across <code>M×N</code> comparisons</li>
      <li>Bradley–Terry / Elo-like models</li>
      <li>HodgeRank / graph-based rankings</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Setwise / Voting</div>
    <ul class="c-list">
      <li>Each question induces a preference ordering over models</li>
      <li>Borda, Copeland, Nanson, minimax variants</li>
      <li>Can be stable yet biased vs the gold standard</li>
    </ul>
  </div>
</div>

---

## Ranking with High-budget

<div class="c-grid c-grid-2" style="margin-top: 0.9rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Agreement with the Bayesian gold standard</div>
    <img src="/assets/slides/2026-01-18-llm-research/acl26_rank/bump_chart_rankings_show_case.webp" alt="Ranking method agreement with gold standard on easy vs hard benchmarks" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Kendall’s <code>τ</code> between each method (at <code>N=80</code>) and <code>Bayes<sub>𝒰</sub>@80</code>, on BrUMO (easy) vs HMMT (hard).</div>
  </div>

<div class="c-card">
    <div class="c-kicker">Headline results</div>
    <ul class="c-list">
      <li>Gold standard: <code>Bayes<sub>𝒰</sub>@80</code> (order-equivalent to <code>avg@80</code>)</li>
      <li>Across benchmarks: mean <code>τ</code> ≈ <strong>0.93–0.95</strong>; <strong>19–34</strong> methods recover the exact ordering</li>
      <li>Divergence concentrates on harder benchmarks and a small set of voting / difficulty-weighted rules</li>
    </ul>
  </div>
</div>

---

## Low-budget: priors help

<div class="c-grid c-grid-2" style="margin-top: 0.9rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">N=1…32: agreement vs gold standard</div>
    <img src="/assets/slides/2026-01-18-llm-research/acl26_rank/low_n_detailed.webp" alt="Kendall’s tau vs number of trials for uniform prior, greedy prior, and conservative CI ranking" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Uniform prior vs greedy empirical prior (<code>R<sub>0</sub></code>) and conservative CI ranking.</div>
  </div>

<div class="c-card">
    <div class="c-kicker">Takeaways</div>
    <ul class="c-list">
      <li>At <code>N=1</code>, best methods reach <code>τ</code> ≈ <strong>0.86</strong> (Combined); uncertainty dominates</li>
      <li><strong>Greedy prior</strong> reduces variance at <code>N=1</code> by <strong>16–52%</strong>, but can <strong>bias</strong> rankings when greedy and sampling disagree</li>
      <li>Practical rule: use empirical priors only in tiny-<code>N</code> regimes and after checking greedy–sampling alignment on a pilot</li>
    </ul>
    <div class="c-foot">
      Toolkit: <a href="https://github.com/mohsenhariri/scorio" target="_blank" rel="noopener noreferrer">scorio</a>
    </div>
  </div>
</div>

---

## Simulating LLM Answers to Evaluation Datasets
### Why “one run” is not an evaluation

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Core intuition</div>
    <ul class="c-list">
      <li>LLMs are <strong>stochastic</strong> under sampling → outputs vary run-to-run</li>
      <li>Each attempt is a <strong>Bernoulli trial</strong>: success / failure (biased coin)</li>
      <li>Different questions act like different coins (<span class="c-muted">difficulty + ambiguity</span>)</li>
      <li>Simulation lets you quantify <strong>variance</strong>, CIs, and sample budgets</li>
    </ul>
    <div class="c-foot">
      Post: <a href="https://mohsenhariri.github.io/posts/2025-10-22-simulating-llm-answers-to-eval-datasets.html" target="_blank" rel="noopener noreferrer">Simulating LLM Answers to Evaluation Datasets</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">Coin flip view</div>
    <img src="/assets/slides/2026-01-18-llm-research/sim_answers/coin_single.webp" alt="Coin-flip analogy for repeated LLM sampling" style="width: 70%; height: auto; border-radius: 14px;" />
    <div class="c-foot">One question, many stochastic runs.</div>
  </div>
</div>

---

## Simulate a whole benchmark (L×M×N)

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Generative recipe</div>
    <pre class="c-pre"><code>for each model ℓ:
  choose capability θ_ℓ
for each question m:
  sample difficulty → p_{ℓm}
for each trial n:
  R_{ℓmn} ~ Bernoulli(p_{ℓm})</code></pre>
    <ul class="c-list">
      <li><strong>Beta–Bernoulli</strong>: question-to-question variation</li>
      <li><strong>IRT</strong>: latent ability <code>θ</code>, difficulty <code>b</code>, discrimination <code>a</code>, guessing <code>c</code></li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">What you get</div>
    <ul class="c-list">
      <li>Stress-test metrics and rankings (Pass@k, avg@N, Bayes@N, …)</li>
      <li>Estimate the trial budget needed for stable conclusions</li>
      <li>Study sensitivity to decoding, seeds, and benchmark size (<code>M</code>)</li>
    </ul>
    <img src="/assets/slides/2026-01-18-llm-research/sim_answers/coin_many.webp" alt="Many coins analogy for many benchmark questions" style="width: 70%; height: auto; border-radius: 14px; margin-top: 0.6rem;" />
  </div>
</div>

---

## Simulating LLM Evaluation Datasets Using Psychometric Models
### Item Response Theory (IRT) for LLM benchmarking

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">The idea</div>
    <ul class="c-list">
      <li>Scoreboards wobble because benchmarks mix easy + hard items</li>
      <li>IRT fits a <strong>latent ability</strong> per model and <strong>difficulty</strong> per question</li>
      <li>Gives an interpretable “ruler” to compare models and design better datasets</li>
    </ul>
    Rasch / 1PL:<br>
    $P(\mathrm{correct} \mid \theta, b) = \sigma(\theta - b)$
    <div class="c-foot">
      Post: <a href="https://mohsenhariri.github.io/posts/2025-10-22-item-response-theory-llms.html" target="_blank" rel="noopener noreferrer">Simulating LLM Evaluation Datasets Using Psychometric Models</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">From “score” to probability</div>
    <img src="/assets/slides/2026-01-18-llm-research/irt/squashing_functions.webp" alt="Sigmoid/probit/tanh as squashing functions in IRT" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Smooth squashing functions map ability–difficulty gaps to success probabilities.</div>
  </div>
</div>

---

## Beyond binary: richer psychometric models

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Model families</div>
    <pre class="c-pre"><code>2PL: P = σ(a(θ - b))          (discrimination a)
3PL: P = c + (1-c)σ(a(θ-b))  (guessing floor c)
GRM: ordered categories for rubric scores</code></pre>
    <ul class="c-list">
      <li><strong>Discrimination</strong>: which items separate close models?</li>
      <li><strong>Guessing</strong>: handles “lucky” successes</li>
      <li><strong>Rubrics</strong>: extends naturally to graded evaluation</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">What you can do with IRT</div>
    <ul class="c-list">
      <li>Calibrate benchmarks (difficulty ladders + information curves)</li>
      <li>Generate synthetic datasets with controlled properties</li>
      <li>Detect “bad” items (too easy, too hard, low discrimination)</li>
    </ul>
    
  </div>
</div>

---

## DFloat11 (Lossless LLM Compression)
### 70% size, 100% accuracy — bit-for-bit identical outputs

<div class="c-callout ok" style="margin-top: 1.0rem;">
  <strong>Key idea:</strong> BF16 weights are statistically redundant → apply entropy coding for <strong>lossless</strong> compression, then decode efficiently on GPU.
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Why it matters</div>
    <ul class="c-list">
      <li>Model weights dominate GPU memory and limit deployment</li>
      <li>Quantization is lossy; DFloat11 is <strong>exact</strong> (bit-identical outputs)</li>
      <li>Enables bigger models and longer generations under fixed HBM</li>
    </ul>
    <div class="c-foot">
      Paper: <a href="https://arxiv.org/abs/2504.11651" target="_blank" rel="noopener noreferrer">arXiv:2504.11651</a> • Code: <a href="https://github.com/LeanModels/DFloat11" target="_blank" rel="noopener noreferrer">LeanModels/DFloat11</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">Headline numbers</div>
    <ul class="c-list">
      <li><strong>≈30%</strong> size reduction (BF16 → ~11 bits avg)</li>
      <li><strong>2.3–46.2×</strong> higher throughput vs CPU-offload baseline</li>
      <li><strong>5.7–14.9×</strong> longer generation under fixed GPU memory</li>
      <li>Lossless inference of <strong>Llama 3.1 405B</strong> on <strong>8×80GB</strong> GPUs</li>
    </ul>
  </div>
</div>

---

## Where the redundancy comes from (BF16 exponent entropy)

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">BF16 layout</div>
    <img src="/assets/slides/2026-01-18-llm-research/df11/bfloat16_layout.webp" alt="BF16 bit layout" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">BF16 has an 8-bit exponent — but trained weights use a small subset.</div>
  </div>

<div class="c-card">
    <div class="c-kicker">Measured entropy</div>
    <img src="/assets/slides/2026-01-18-llm-research/df11/exponent_entropy.webp" alt="Entropy of BF16 exponent bits" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Exponent carries only a few bits of information → compressible.</div>
  </div>
</div>

---

## DFloat11: dynamic-length encoding + fast GPU decode

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Format</div>
    <img src="/assets/slides/2026-01-18-llm-research/df11/dfloat11_layout.webp" alt="DFloat11 encoding overview" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Entropy-coded fields yield ~11-bit average storage with exact reconstruction.</div>
  </div>

<div class="c-card">
    <div class="c-kicker">Kernel ideas</div>
    <ul class="c-list">
      <li>Compact <strong>hierarchical LUTs</strong> fit in GPU SRAM</li>
      <li><strong>Two-phase</strong> kernel coordinates read/write offsets</li>
      <li><strong>Transformer-block</strong> decompression amortizes overhead</li>
    </ul>
    <img src="/assets/slides/2026-01-18-llm-research/df11/hierarchical_luts.webp" alt="Hierarchical lookup tables for decoding" style="width: 100%; height: auto; border-radius: 14px; margin-top: 0.6rem;" />
  </div>
</div>

---

## Practical impact: fit bigger models, free memory for KV cache

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Memory savings</div>
    <img src="/assets/slides/2026-01-18-llm-research/df11/memory_usage.webp" alt="Memory usage comparison for DF11 vs BF16" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">≈68% of original BF16 size across many LLMs and diffusion models.</div>
  </div>

<div class="c-card">
    <div class="c-kicker">When to use it</div>
    <ul class="c-list">
      <li>Need <strong>bit-exact</strong> outputs (deployment parity, regression safety)</li>
      <li>Memory is the bottleneck; quantization risk is unacceptable</li>
      <li>Want longer context: weights shrink → more room for KV cache</li>
    </ul>
  </div>
</div>

---

## Quantize What Counts (KV Cache)
### More for keys, less for values

<div class="c-callout ok" style="margin-top: 1.0rem;">
  <strong>Key idea:</strong> KV-cache quantization should allocate bits based on Transformer geometry — keys are more information-dense.
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Problem</div>
    <ul class="c-list">
      <li>KV cache dominates inference memory and scales with context length</li>
      <li>Uniform KV quantization wastes bits on the wrong component</li>
      <li>Goal: for a fixed budget, <strong>minimize quantization error</strong> and preserve accuracy</li>
    </ul>
    <div class="c-foot">
      Paper: <a href="https://arxiv.org/abs/2502.15075" target="_blank" rel="noopener noreferrer">arXiv:2502.15075</a> • Code: <a href="https://github.com/mohsenhariri/spectral-kv" target="_blank" rel="noopener noreferrer">mohsenhariri/spectral-kv</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">Headline result</div>
    <ul class="c-list">
      <li>Keys typically have larger spectral/Frobenius norms than values</li>
      <li>Key-favored splits like <strong>K4V2</strong> keep near-baseline quality</li>
      <li>Transforms bit allocation from tuning → theorem-backed design</li>
    </ul>
  </div>
</div>

---

## The geometry: keys dominate in norm

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Observation</div>
    <ul class="c-list">
      <li>$ \|W^K\| $ tends to exceed $ \|W^V\| $ across layers</li>
      <li>Higher norm → higher dynamic range → more sensitive to quantization</li>
    </ul>
    <div class="c-callout" style="margin-top: 0.9rem;">
      <strong>Implication:</strong> if you must “spend bits”, spend them on <strong>keys</strong>.
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">Evidence</div>
    <img src="/assets/slides/2026-01-18-llm-research/kvq/key_vs_value_norms.webp" alt="Key vs value norms across layers" style="width: 100%; height: auto; border-radius: 14px;" />
  </div>
</div>

---

## Why “more for keys” is optimal under a fixed budget

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Quantization error scaling</div>
    <pre class="c-pre"><code>Uniform scalar quantization:
E[||M - M̃||_F²] = Θ(||M||_F² · 2^{-2b})</code></pre>
    <ul class="c-list">
      <li>If $||K||$ has more energy than $||V||$, equal bits are sub-optimal</li>
      <li>Swapping bits to make <code>b_K &gt; b_V</code> strictly reduces expected error</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Practical rule</div>
    <ul class="c-list">
      <li>Choose mixed precision with <strong>keys ≥ values</strong></li>
      <li>Simple default: <strong>K4V2</strong> for strong memory reduction</li>
      <li>Works across quantization backends (token-wise / channel-wise)</li>
    </ul>
    <div class="c-callout ok" style="margin-top: 0.9rem;">
      <strong>Takeaway:</strong> bit allocation becomes a geometry-driven design choice.
    </div>
  </div>
</div>

---

## Empirical: K4V2 keeps accuracy + integrates with rotations

<div class="c-grid c-grid-2" style="margin-top: 0.9rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Accuracy snapshot</div>
    <img src="/assets/slides/2026-01-18-llm-research/kvq/gsm_accuracy.webp" alt="GSM8K accuracy for key/value bit allocations" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Key-favored allocations consistently outperform value-favored allocations.</div>
  </div>

<div class="c-card">
    <div class="c-kicker">Synergy with outlier-handling</div>
    <ul class="c-list">
      <li>Mixed precision is <strong>orthogonal</strong> to inference-time KV methods</li>
      <li>Rotation-based methods: <strong>key-only rotation</strong> often helps most</li>
      <li>Empirically, key-favored + key-only rotation can outperform K4V4</li>
    </ul>
    <img src="/assets/slides/2026-01-18-llm-research/kvq/rotation_heatmap.webp" alt="Rotation impact heatmap for KV quantization" style="width: 50%; height: auto; border-radius: 14px; margin-top: 0.6rem;" />
  </div>
</div>

---

## Entropy of bfloat16 (Exponent Bits)
### “8 bits are doing ~2.6 bits of work”

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Observation</div>
    <ul class="c-list">
      <li>BF16 exponent has 256 states, but trained weights occupy a small subset</li>
      <li>Measured exponent entropy is consistently <strong>≈2.5–2.7 bits</strong> across LLMs</li>
      <li>Sign and mantissa are near-fully utilized; exponent is the outlier</li>
    </ul>
    <div class="c-foot">
      Post: <a href="https://mohsenhariri.github.io/posts/2025-10-28-data-type-entropy.html" target="_blank" rel="noopener noreferrer">Entropy of bfloat16: 8 Bits Are Doing 2.6 Bits of Work</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">Visuals</div>
    <img src="/assets/slides/2026-01-18-llm-research/bf16_entropy/bfloat16_layout.webp" alt="BF16 layout" style="width: 100%; height: auto; border-radius: 14px;" />
    <img src="/assets/slides/2026-01-18-llm-research/bf16_entropy/exponent_entropy.webp" alt="Exponent entropy plot" style="width: 70%; height: auto; border-radius: 14px; margin-top: 0.6rem;" />
  </div>
</div>

---

## Why this matters: lossless compression becomes possible

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Compression takeaway</div>
    <ul class="c-list">
      <li>Entropy coding can shrink exponent storage toward its entropy (~2.6 bits)</li>
      <li>That’s ~5 “free” bits per weight → big wins at billions of params</li>
      <li>Lossless compression composes cleanly with other inference tricks</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Connection</div>
    <div class="c-callout ok" style="margin-top: 0.9rem;">
      This observation directly motivates <strong>DFloat11</strong>: a GPU-friendly, lossless compressed format for BF16 weights.
    </div>
    <ul class="c-list">
      <li>Enables larger models and longer contexts under fixed memory</li>
      <li>Preserves bit-exact outputs for deployment parity</li>
    </ul>
  </div>
</div>

---

## Entropy During Training (Optimizer Effects)
### How optimizers shape BF16 exponent entropy

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Question</div>
    <ul class="c-list">
      <li>Why do trained LLMs consistently land near <strong>~2.6 bits</strong> of exponent entropy?</li>
      <li>Is it architecture? initialization? data?</li>
      <li>Experiment: track sign/exponent/mantissa entropy throughout training</li>
    </ul>
    <div class="c-foot">
      Post: <a href="https://mohsenhariri.github.io/posts/2025-11-17-bf16-evolution.html" target="_blank" rel="noopener noreferrer">Entropy of bfloat16 During Training</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">Bit components</div>
    <img src="/assets/slides/2026-01-18-llm-research/bf16_evolution/three_components.webp" alt="Distributions of sign/exponent/mantissa components" style="width: 140%; height: auto; border-radius: 14px;" />
  </div>
</div>

---

## Finding: optimizer dominates the “2.6 bits” outcome

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">What happens</div>
    <ul class="c-list">
      <li>Entropy trajectories cluster by <strong>optimizer</strong> more than initialization</li>
      <li><strong>AdamW</strong> consistently lands in the LLM-like entropy band</li>
      <li>Explains why many LLMs share similar compressibility (AdamW is standard)</li>
    </ul>
    <img src="/assets/slides/2026-01-18-llm-research/bf16_evolution/optimizer_effect.webp" alt="Optimizer effect on exponent entropy trajectories" style="width: 80%; height: auto; border-radius: 14px; margin-top: 0.6rem;" />
  </div>

<div class="c-card">
    <div class="c-kicker">AdamW consistency</div>
    <img src="/assets/slides/2026-01-18-llm-research/bf16_evolution/adamw_consistency.webp" alt="AdamW drives exponent entropy near 2.6 regardless initialization" style="width: 100%; height: auto; border-radius: 14px;" />
    <div class="c-foot">Key implication: compressibility depends on training dynamics, not just datatype.</div>
  </div>
</div>

---

## Zero-shot 3D Knee MRI Segmentation with SAM2
### Treat MRI slices as video frames

<div class="c-callout ok" style="margin-top: 1.0rem;">
  <strong>Key idea:</strong> adapt a video segmentation foundation model (SAM2) to 3D MRI by processing slices as frames, enabling single-prompt, zero-shot segmentation.
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Problem</div>
    <ul class="c-list">
      <li>Medical segmentation models struggle with <strong>domain transfer</strong> and labeled data scarcity</li>
      <li>3D MRI segmentation is expensive to annotate and retrain per domain</li>
    </ul>
    <div class="c-kicker" style="margin-top: 0.9rem;">Result highlight</div>
    <ul class="c-list">
      <li>OAI-ZIB dataset: tibia Dice ≈ <strong>0.964</strong> (zero-shot)</li>
      <li>No additional training or fine-tuning</li>
    </ul>
    <div class="c-foot">
      Paper: <a href="https://arxiv.org/abs/2408.04762" target="_blank" rel="noopener noreferrer">arXiv:2408.04762</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">How it works</div>
    <div class="c-flow" style="margin-top: 0.2rem;">
      <div class="c-node"><strong>3D MRI volume</strong><span class="small">stack of slices</span></div>
      <div class="c-arrow">→</div>
      <div class="c-node"><strong>“Video” view</strong><span class="small">slices as frames</span></div>
      <div class="c-arrow">→</div>
      <div class="c-node"><strong>SAM2</strong><span class="small">memory across frames</span></div>
      <div class="c-arrow">→</div>
      <div class="c-node"><strong>3D mask</strong><span class="small">propagated segmentation</span></div>
    </div>
  </div>
</div>

---

## What’s new / why it’s useful

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Practical benefits</div>
    <ul class="c-list">
      <li><strong>Single-prompt</strong> interaction (point/box) for an entire 3D volume</li>
      <li><strong>Scalable</strong> labeling aid for new domains without retraining</li>
      <li>Compares SAM2 model sizes and prompt schemes; includes SAM1 baseline</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Where this fits</div>
    <ul class="c-list">
      <li>Rapid prototyping for clinical workflows (knee MRI)</li>
      <li>Bootstrapping annotations for specialized 3D segmenters</li>
      <li>General recipe: reuse “video memory” models for 3D medical stacks</li>
    </ul>
  </div>
</div>

---

## Masked-speech Recognition Using Human and Synthetic Cloned Speech
### Voice clones as evaluation tools

<div class="c-callout ok" style="margin-top: 1.0rem;">
  <strong>Key idea:</strong> test whether voice-cloned speech matches human speech for intelligibility and perceptual qualities in masked-sentence recognition.
</div>

<div class="c-grid c-grid-2" style="margin-top: 1.1rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Experimental setup</div>
    <ul class="c-list">
      <li>5 human talkers + their <strong>synthetic voice clones</strong></li>
      <li>Masked-sentence recognition in speech-shaped noise at <strong>−6 dB SNR</strong></li>
      <li>Two sentence types: meaningful vs nonsense</li>
      <li>Scoring: human listeners + automatic speech recognition (ASR)</li>
    </ul>
    <div class="c-foot">
      Paper: <a href="https://journals.sagepub.com/doi/10.1177/23312165251403080" target="_blank" rel="noopener noreferrer">Trends in Hearing (2025)</a>
    </div>
  </div>

<div class="c-card">
    <div class="c-kicker">Perceptual measures</div>
    <ul class="c-list">
      <li><strong>Intelligibility</strong>: keyword accuracy under masking</li>
      <li><strong>Human-likeness</strong>: listener ratings</li>
      <li><strong>Voice similarity</strong>: same vs different speaker judgments</li>
    </ul>
  </div>
</div>

---

## Findings and implications

<div class="c-grid c-grid-2" style="margin-top: 1.0rem; align-items: start;">
  <div class="c-card">
    <div class="c-kicker">Main findings</div>
    <ul class="c-list">
      <li>Recognition worse for nonsense vs meaningful sentences (<span class="c-muted">expected</span>)</li>
      <li>Recognition and human-likeness were <strong>similar</strong> for human vs cloned speech</li>
      <li>Listeners detect different voices well, but often judge human/clone pairs as the same</li>
      <li>ASR reliability scoring matched human scoring for <strong>98%</strong> of keywords</li>
    </ul>
  </div>

<div class="c-card">
    <div class="c-kicker">Why it matters</div>
    <ul class="c-list">
      <li>Voice cloning can help scale controlled intelligibility experiments</li>
      <li>ASR can act as a fast proxy for human scoring in many cases</li>
      <li>Tools for evaluating both <strong>human</strong> and <strong>synthetic</strong> speech systems</li>
    </ul>
  </div>
</div>

---

## Getting Started with Research on LLMs

<div style="text-align: center; margin-top: 2em;">
  <blockquote style="font-size: 1.2em; font-style: italic; margin: 1em 0; width: 100%;">
    Bojack Horseman: "Okay, okay. Here’s the secret to doing research in LLMs. Just pretend you are doing research, and eventually, you’ll forget you’re pretending."
  </blockquote>
  
  <div style="margin-top: 2em;">
    <img src="/assets/slides/2026-01-18-llm-research/image.webp" alt="Bojack Horseman meme" style="max-width: 90%; height: auto; border-radius: 10px; box-shadow: 0 4px 8px rgba(0,0,0,0.1);" />
  </div>
</div>

---

## Thank you

<div class="c-callout" style="margin-top: 2.0rem; font-size: 0.9em;">
Please feel free to contact <a href="mailto:mohsen.hariri@case.edu">mohsen.hariri@case.edu</a> with any questions or feedback, especially if you’d like to collaborate on any of the research directions mentioned!
</div>

Navigation