Quick keys
— Navigate with ← → or spacebar.
F
fullscreen ·
S
speaker view ·
O
overview ·
Alt+Click
zoom
Navigation
×
<!-- The qslide typography and palette follow the first 16 slides of the qualifying-examination deck. --> <style> :root { --q-ink:#142d48; --q-muted:#516477; --q-line:#ccd6dd; --q-card:#fff; --q-blue:#17639a; --q-green:#16715f; --q-orange:#a74919; --q-tint:#edf3f6; } html.dark { --q-ink:#edf3f8; --q-muted:#b8c7d5; --q-line:#496074; --q-card:#1c2e3e; --q-blue:#8bcef5; --q-green:#89dbbf; --q-orange:#ffc197; --q-tint:#20384b; } .reveal .slides section .qslide { box-sizing:border-box; width:100%; height:850px; padding:24px 44px 22px; color:var(--q-ink); font-family:Inter,Arial,sans-serif; display:flex; flex-direction:column; text-align:left; font-size:29px; line-height:1.34; } .reveal .slides section .qslide h1 { color:var(--q-ink); font-size:88px; font-weight:700; line-height:1.09; letter-spacing:-1.3px; text-transform:none; margin:0 0 30px; } .reveal .slides section .qslide h2 { color:var(--q-ink); font-size:51px; font-weight:650; line-height:1.14; letter-spacing:-.7px; text-transform:none; margin:10px 0 24px; } .reveal .slides section .qslide h3 { color:var(--q-ink); font-size:30px; font-weight:650; line-height:1.25; text-transform:none; margin:0 0 16px; } .reveal .slides section .qslide p { line-height:1.38; margin:0 0 20px; font-size:inherit; } .reveal .slides section .qslide .lead { font-size:38px; line-height:1.3; } .reveal .slides section .qslide .muted { color:var(--q-muted); } .reveal .slides section .qslide .small { font-size:24px; } .reveal .slides section .qslide .body { flex:1; min-height:0; } .reveal .slides section .qslide .split { display:grid; grid-template-columns:minmax(0,.85fr) minmax(0,1.65fr); gap:44px; align-items:center; } .reveal .slides section .qslide footer { font-size:19px; line-height:1.3; color:var(--q-muted); border-top:1px solid var(--q-line); padding-top:11px; margin-top:18px; } .reveal .slides section .qslide footer.reference { display:grid; grid-template-columns:minmax(0,1fr) auto; gap:3px 28px; } .reveal .slides section .qslide .reference-title { grid-column:1; } .reveal .slides section .qslide .reference-authors { grid-column:1; } .reveal .slides section .qslide .reference-venue { grid-column:2; grid-row:1 / 3; align-self:center; } .reveal .slides section .qslide .reference-venue img { display:block; width:92px; height:58px; max-width:none; max-height:none; margin:0; padding:5px; border:0; border-radius:4px; box-shadow:none; object-fit:contain; background:#fff; } .reveal .slides section .qslide .reference-venue img.website-logo { width:64px; } .reveal .slides section .qslide a { color:var(--q-blue); text-decoration:underline; text-underline-offset:3px; } .reveal .slides section .qslide a:focus-visible { outline:3px solid var(--q-orange); outline-offset:5px; } .reveal .slides section .qslide > h1, .reveal .slides section .qslide > h2, .reveal .slides section .qslide > p, .reveal .slides section .qslide > footer { flex-shrink:0; } .reveal .slides section .qslide .figure { display:block; box-sizing:border-box; width:100%; height:100%; min-height:0; max-height:none; margin:0; padding:10px; border:0; border-radius:8px; box-shadow:none; object-fit:contain; background:#fff; } .reveal .slides section .qslide .figure-link { display:block; min-height:0; height:100%; } .reveal .slides section .qslide.cover { justify-content:center; padding-left:82px; } .reveal .slides section .qslide.cover h1 { max-width:1200px; font-size:96px; margin-bottom:60px; } .reveal .slides section .qslide.cover .author { font-size:32px; margin-bottom:10px; } .reveal .slides section .qslide.cover .institution { font-size:25px; color:var(--q-muted); } .reveal .slides section .qslide.cover .advisors { font-size:25px; color:var(--q-muted); margin-top:28px; } .reveal .slides section .qslide.cover .date { font-size:25px; color:var(--q-blue); margin-top:28px; } .reveal .slides section .qslide .qr-layout { display:grid; grid-template-columns:minmax(0,1fr) 320px; gap:64px; align-items:center; height:100%; min-height:0; } .reveal .slides section .qslide .qr-card { box-sizing:border-box; display:block; width:308px; padding:18px; border:1px solid var(--q-line); border-radius:8px; background:var(--q-card); text-align:center; text-decoration:none; } .reveal .slides section .qslide .qr-card img { display:block; width:270px; height:270px; max-width:none; max-height:none; margin:0 auto 14px; border:0; border-radius:0; box-shadow:none; background:#fff; object-fit:contain; } .reveal .slides section .qslide .qr-card span { display:block; color:var(--q-ink); font-size:24px; line-height:1.25; } .reveal .slides section .qslide .metric { color:var(--q-blue); font-size:84px; line-height:1.1; font-weight:700; margin:0 0 14px; } .reveal .slides section .qslide .entropy-copy { align-self:center; } .reveal .slides section .qslide .bit-fields { display:grid; grid-template-columns:1fr 8fr 7fr; gap:4px; margin:34px 0 12px; height:50px; } .reveal .slides section .qslide .bit-fields span { display:flex; align-items:center; justify-content:center; border:1px solid var(--q-line); border-radius:3px; font-size:24px; background:var(--q-tint); } .reveal .slides section .qslide .bit-fields .exponent { color:var(--q-blue); border-color:var(--q-blue); } .reveal .slides section .qslide .bit-labels { display:flex; justify-content:space-between; gap:12px; color:var(--q-muted); font-size:22px; } .reveal .slides section .qslide.training .experiment { margin-bottom:12px; } .reveal .slides section .qslide .optimizer-legend { display:flex; gap:48px; align-items:center; margin:0 0 10px; font-size:25px; } .reveal .slides section .qslide .optimizer-legend span { display:flex; align-items:center; gap:10px; } .reveal .slides section .qslide .optimizer-legend i { display:inline-block; width:52px; border-top:3px solid var(--q-muted); } .reveal .slides section .qslide .optimizer-legend .sgd { height:3px; border:0; background:repeating-linear-gradient(to right,var(--q-muted) 0 18px,transparent 18px 24px,var(--q-muted) 24px 27px,transparent 27px 33px); } .reveal .slides section .qslide .optimizer-legend .adamw { border-top-style:dashed; } .reveal .slides section .qslide.dfloat .body { display:flex; flex-direction:column; gap:16px; } .reveal .slides section .qslide .dfloat-format { flex:1; min-height:0; } .reveal .slides section .qslide.dfloat .method-line { margin:0; font-size:28px; text-align:center; } .reveal .slides section .qslide.kv-spectra .body { display:flex; flex-direction:column; justify-content:center; } .reveal .slides section .qslide.kv-spectra .spectrum-figure { height:420px; } .reveal .slides section .qslide.kvq .body { display:grid; grid-template-columns:minmax(0,1.35fr) minmax(0,1fr); gap:72px; align-items:center; } .reveal .slides section .qslide.kvq table { width:100%; margin:0; font-size:30px; line-height:1.3; border-collapse:collapse; } .reveal .slides section .qslide.kvq th { font-weight:600; color:var(--q-muted); font-size:23px; border-bottom:2px solid var(--q-line); padding:14px 15px; text-align:left; } .reveal .slides section .qslide.kvq td { padding:26px 15px; border-bottom:1px solid var(--q-line); vertical-align:middle; } .reveal .slides section .qslide.kvq tbody th { font-size:30px; color:var(--q-ink); border-bottom:1px solid var(--q-line); } .reveal .slides section .qslide.kvq .allocation { color:var(--q-blue); font-size:44px; margin-bottom:34px; } .reveal .slides section .qslide.kvq .retention .metric { font-size:76px; } .reveal .slides section .qslide.kvq .retention .saving { font-size:29px; margin-top:32px; } .reveal .slides section .qslide.timeline { display:grid; grid-template-columns:270px minmax(0,1fr); gap:26px; align-items:center; } .reveal .slides section .qslide.timeline .timeline-copy h2 { font-size:48px; margin-bottom:36px; } .reveal .slides section .qslide.timeline .figure-link { height:790px; } .reveal .slides section .qslide.paper-figure h2 { margin-bottom:16px; } .reveal .slides section .qslide.thanks { display:grid; grid-template-rows:minmax(0,1fr) auto; gap:36px; padding:54px 80px 76px; } .reveal .slides section .qslide.thanks .closing-main { display:grid; justify-items:center; align-content:start; text-align:center; } .reveal .slides section .qslide.thanks h1 { font-size:84px; font-weight:400; line-height:1.1; margin:0 0 66px; } .reveal .slides section .qslide.thanks .closing-qrs { display:grid; grid-template-columns:repeat(3,270px); justify-content:center; gap:26px; } .reveal .slides section .qslide.thanks .qr-card { width:270px; padding:22px; border-radius:9px; box-shadow:0 12px 28px rgba(15,23,42,.055); } .reveal .slides section .qslide.thanks .qr-card img { width:224px; height:224px; margin-bottom:16px; } .reveal .slides section .qslide.thanks .qr-card span { font-size:24px; font-weight:600; line-height:1.2; } .reveal .slides section .qslide.thanks .acknowledgments { display:grid; grid-template-columns:460px 740px; justify-content:center; align-items:center; gap:40px; width:100%; } .reveal .slides section .qslide.thanks .ack-item { display:grid; grid-template-columns:auto minmax(0,1fr); align-items:center; gap:22px; } .reveal .slides section .qslide.thanks .ack-item img { display:block; max-width:none; max-height:none; margin:0; border:0; border-radius:0; box-shadow:none; object-fit:contain; } .reveal .slides section .qslide.thanks .ack-nsf img { width:84px; height:84px; } .reveal .slides section .qslide.thanks .ack-cwru img { box-sizing:border-box; width:270px; height:auto; padding:6px; background:#fff; } .reveal .slides section .qslide.thanks .ack-item h3 { font-size:26px; font-weight:400; line-height:1.1; white-space:nowrap; margin:0 0 12px; } .reveal .slides section .qslide.thanks .ack-item p { color:var(--q-muted); font-size:22px; line-height:1.2; white-space:nowrap; margin:0; } @media screen and (max-width:1024px) { body { padding-top:0; overflow:hidden; } .layout-layer { position:fixed; } .reveal { height:100dvh; min-height:0; } } @media print { .reveal .slides section .qslide { --q-ink:#142d48; --q-muted:#516477; --q-line:#ccd6dd; --q-card:#fff; --q-blue:#17639a; --q-green:#16715f; --q-orange:#a74919; --q-tint:#edf3f6; } } </style> <div class="qslide cover"> <div class="qr-layout"> <div> <h1>Efficient LLM inference</h1> <p class="author">Mohsen Hariri</p> <p class="institution">Case Western Reserve University</p> <p class="advisors">Advisors: Prof. Vipin Chaudhary and Prof. Sanmukh Kuppannagari</p> <p class="date"><time datetime="2026-11-03">November 3, 2026</time></p> </div> <a class="qr-card" href="https://mohsenhariri.github.io/slides/2026-11-03-from-bits-to-tokens.html"> <img src="/assets/slides/2026-11-03-from-bits-to-tokens/slide_qr.svg" alt="QR code for this slide deck"> <span>Slides</span> </a> </div> </div> Note: The first part concerns how weights and attention caches occupy memory. The second concerns how the next token is selected from the model's distribution. DFloat11 changes the stored representation while preserving the original weights. KV-cache quantization changes precision. Semantic Sampling changes the decoding rule. --- <div class="qslide entropy"> <h2>Eight exponent bits carry about 2.6 bits of information</h2> <div class="body split"> <div class="entropy-copy"> <p class="metric">≈2.6 bits</p> <p>of exponent entropy<br>in the measured LLM weights.</p> <div class="bit-fields" role="img" aria-label="Bfloat16 stores one sign bit, eight exponent bits, and seven mantissa bits."><span>1</span><span class="exponent">8</span><span>7</span></div> <div class="bit-labels"><span>sign</span><span>exponent</span><span>mantissa</span></div> <p class="small muted" style="margin-top:28px">16 stored bits.<br>About 10.6 bits of component entropy.</p> </div> <img class="figure" src="/assets/slides/2026-11-03-from-bits-to-tokens/figures/weight-entropy.webp" alt="Measured sign, exponent, and mantissa entropy for five LLMs. Sign and mantissa approach their allocated widths; exponent entropy is about 2.6 bits out of eight."> </div> <footer class="reference"> <a class="reference-title" href="https://mohsenhariri.github.io/posts/2025-10-28-data-type-entropy.html">Entropy of bfloat16: 8 Bits Are Doing 2.6 Bits of Work</a> <span class="reference-authors">Mohsen Hariri</span> <span class="reference-venue"><img class="website-logo" src="/assets/slides/2026-11-03-from-bits-to-tokens/logo/website.webp" alt="Desiderata Kashkul website"></span> </footer> </div> Note: Bfloat16 assigns one bit to the sign, eight to the exponent, and seven to the mantissa. The sign and mantissa distributions use nearly their full capacity. The exponent distribution is concentrated on a small subset of its possible values, giving roughly 2.6 bits of Shannon entropy in these measured models. The sum of the three component entropies is about 10.6 bits per weight. It motivates compression but does not specify an exact code length or include coding overhead. Source: _posts/2025-10-28-data-type-entropy.md; figure copied from _posts/_assets/2025-10-19-df11/entropy.png. --- <!-- The training plot removes only the title, legend, and empty upper range of the source image. All 15 trajectories remain. --> <div class="qslide training"> <h2>The optimizer changes exponent entropy</h2> <p class="small muted experiment">Five initializations, three optimizers, 50 epochs</p> <div class="optimizer-legend" aria-label="Line styles identify optimizers; colors identify initialization and optimizer combinations."> <span><i class="sgd" aria-hidden="true"></i>SGD: lower</span> <span><i class="adamw" aria-hidden="true"></i>AdamW: stays low</span> <span><i aria-hidden="true"></i>Adam: higher</span> </div> <div class="body"> <img class="figure" src="/assets/slides/2026-11-03-from-bits-to-tokens/figures/entropy-training.webp" alt="All 15 measured exponent-entropy trajectories over 50 epochs. Adam increases entropy, SGD generally lowers it, and AdamW remains low. The red dashed reference is 2.6 bits; the shaded band is 2.4 to 2.8 bits."> </div> <footer class="reference"> <a class="reference-title" href="https://mohsenhariri.github.io/posts/2025-11-17-bf16-evolution.html">Entropy of bfloat16 During Training: How Optimizers Shape Weight Distributions</a> <span class="reference-authors">Mohsen Hariri</span> <span class="reference-venue"><img class="website-logo" src="/assets/slides/2026-11-03-from-bits-to-tokens/logo/website.webp" alt="Desiderata Kashkul website"></span> </footer> </div> Note: The experiment crosses Xavier, Kaiming, orthogonal, normal, and uniform initialization with SGD, Adam, and AdamW. Entropy is recorded every five epochs. Adam's solid traces move upward, SGD's dash-dot traces mostly move downward or remain low, and AdamW's dashed traces stay low. The red line marks the 2.6-bit observation in trained LLMs. These runs do not all converge to that line. The experiment shows that the optimizer affects the weight distribution; it does not establish a universal 2.6-bit law. Sign and mantissa entropy change little in the accompanying measurements. Source: _posts/2025-11-17-bf16-evolution.md. Figure: all_experiments.png, cropped below the legend without removing any trajectory. --- <div class="qslide dfloat"> <h2>DFloat11: 70% size, 100% accuracy</h2> <div class="body"> <div class="dfloat-format"> <a class="figure-link" href="../assets/slides/2026-11-03-from-bits-to-tokens/figures/dfloat11-format.webp" target="_blank" rel="noopener" aria-label="Open the DFloat11 encoding figure at full size"> <img class="figure" src="/assets/slides/2026-11-03-from-bits-to-tokens/figures/dfloat11-format.webp" alt="DFloat11 stores Huffman-coded exponents separately from the unchanged sign and mantissa bits. The tree assigns short codes to common exponents and longer codes to rare exponents."> </a> </div> <p class="method-line">A custom GPU kernel decodes weights during inference.</p> </div> <footer class="reference"> <a class="reference-title" href="https://mohsenhariri.github.io/research/2025-10-19-dfloat11.html">70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float</a> <span class="reference-authors">Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, Anshumali Shrivastava</span> <span class="reference-venue"><img src="/assets/slides/2026-11-03-from-bits-to-tokens/logo/neurips.svg" alt="NeurIPS"></span> </footer> </div> Note: DFloat11 Huffman-codes the exponents and keeps sign and mantissa bits intact. Common exponents receive shorter codes, so eleven bits is an average storage cost rather than a fixed-precision arithmetic format. Compressed weights remain in GPU memory. A custom GPU kernel reconstructs BF16 weights for computation, and the temporary decoded block can then be discarded. Compact hierarchical lookup tables and transformer-block-level batching make online decoding practical. The paper reports roughly 30 percent smaller model storage and lossless inference of Llama 3.1 405B on eight 80 GB GPUs. Decompression adds work; the memory saving does not imply a speedup over BF16 when both fit. Sources: quillpod/2025/neurips/df11/abstract.tex, method.tex, method_luts.tex, two_phase_kernel.tex, and experiments.tex; _research/2025-10-19-dfloat11.md. Encoding figure: _posters/_assets/2025-12-02-dfloat11/assets/figures/DFloat11.png. --- <div class="qslide kv-spectra"> <h2>Singular value spectra of keys and values</h2> <p class="small muted">Llama 3.3-70B-Instruct on C4</p> <div class="body"> <a class="figure-link spectrum-figure" href="../assets/slides/2026-11-03-from-bits-to-tokens/figures/kv-singular-value-spectra.webp" target="_blank" rel="noopener" aria-label="Open the key and value singular value spectra at full size"> <img class="figure" src="/assets/slides/2026-11-03-from-bits-to-tokens/figures/kv-singular-value-spectra.webp" alt="Singular value spectra of key and value caches in Llama 3.3-70B-Instruct on C4, shown for eight layers. Blue key spectra have larger magnitudes than green value spectra. Dashed lines show means across heads and shaded regions show their ranges."> </a> </div> <footer class="reference"> <a class="reference-title" href="https://mohsenhariri.github.io/research/2025-10-20-more-for-keys-less-for-values.html">Quantize What Counts: More for Keys, Less for Values</a> <span class="reference-authors">Mohsen Hariri, Alan Luo, Weicong Chen, Tianyi Zhang, Qifan Wang, Xiaotian Han, Vipin Chaudhary</span> <span class="reference-venue"><img src="/assets/slides/2026-11-03-from-bits-to-tokens/logo/acl.svg" alt="ACL"></span> </footer> </div> Note: The top row shows key activations and the bottom row shows value activations at the same eight layers. Compare the axis scales: key singular values remain larger across the spectrum. The shaded areas give the minimum and maximum across attention heads, and dashed lines give the mean. The displayed spectra omit the first four singular values for clarity. This is the empirical geometry that motivates assigning more precision to keys. Sources: _posters/_assets/2026-06-04-more-for-keys-less-for-values/assets/figures/Llama3.3-70B-it_5_to_end_L.png; quillpod/2026/acl/kvq/Texts/methodology_new.tex, Figure 3; https://mohsenhariri.github.io/posters/2026-06-04-more-for-keys-less-for-values.html. --- <div class="qslide kvq"> <h2>More for keys, less for values</h2> <div class="body"> <div> <p class="allocation">4-bit keys, 2-bit values</p> <table aria-label="KV-cache quantization guidance from the ACL poster"> <thead><tr><th scope="col">Cache</th><th scope="col">Bits</th><th scope="col">Group size</th><th scope="col">Rotation</th></tr></thead> <tbody> <tr><th scope="row">Keys</th><td>Higher</td><td>Smaller</td><td>Apply</td></tr> <tr><th scope="row">Values</th><td>Lower</td><td>Larger</td><td>Skip</td></tr> </tbody> </table> </div> <div class="retention"> <p class="metric">98.3%</p> <p>of K<sub>4</sub>V<sub>4</sub> accuracy retained by K<sub>4</sub>V<sub>2</sub><br>with HQQ, on average</p> <p class="saving">25% fewer KV payload bits</p> </div> </div> <footer class="reference"> <a class="reference-title" href="https://mohsenhariri.github.io/research/2025-10-20-more-for-keys-less-for-values.html">Quantize What Counts: More for Keys, Less for Values</a> <span class="reference-authors">Mohsen Hariri, Alan Luo, Weicong Chen, Tianyi Zhang, Qifan Wang, Xiaotian Han, Vipin Chaudhary</span> <span class="reference-venue"><img src="/assets/slides/2026-11-03-from-bits-to-tokens/logo/acl.svg" alt="ACL"></span> </footer> </div> Note: The larger key-path norms motivate higher key precision. The poster's deployment guidance combines that bit allocation with smaller key groups and key-only rotation. The HQQ study reports that four-bit keys and two-bit values retain 98.3 percent of the four-plus-four accuracy on average. Rotation and group-size results come from the separate integration study. Six versus eight payload bits per key-value pair gives a 25 percent reduction for equal-sized tensors, excluding scales, other quantization metadata, and any full-precision residual buffer. The results depend on the model, task, and quantizer. Sources: _posters/_assets/2026-06-04-more-for-keys-less-for-values/index.html, headline results and deployment table; quillpod/2026/acl/kvq/Texts/results_new.tex, downstream accuracy and integration study. --- <!-- Semantic Sampling: Appendix Figure A1, then main-text Figures 1 and 2. Figures are rendered directly from the manuscript's TikZ sources. --> <div class="qslide timeline"> <div class="timeline-copy"> <h2>Choosing<br>the next<br>token</h2> </div> <a class="figure-link" href="../assets/slides/2026-11-03-from-bits-to-tokens/figures/decoding-timeline.webp" target="_blank" rel="noopener" aria-label="Open Figure A1 at full size"> <img class="figure" src="/assets/slides/2026-11-03-from-bits-to-tokens/figures/decoding-timeline.webp" alt="Figure A1: timeline of full and temperature sampling, beam search, greedy decoding, top-k, and top-p. The lower row shows RadiK, FlashInfer, Qrita, and FlashSampling GPU implementations."> </a> </div> Note: Decoding turns a next-token distribution into a sequence. Full sampling draws from that distribution. Temperature changes its concentration. Top-k and top-p restrict the eligible set using individual token probabilities. The lower row concerns how sampling is executed on the GPU. Sorting-free kernels already exist for truncation sampling, so avoiding sorting alone is not the contribution of Semantic Sampling. The next figure adds a different source of evidence: probability mass on embedding neighbors. Source: quillpod/2027/mlsys/semsam/appendix/history.tex, Figure A1, and figures/decoding_timeline.tex. Historical citations are given in the manuscript caption and appendix. Click the figure to open the full-size asset. --- <div class="qslide paper-figure"> <h2>Semantic Sampling</h2> <div class="body"> <a class="figure-link" href="../assets/slides/2026-11-03-from-bits-to-tokens/figures/semantic-overview.webp" target="_blank" rel="noopener" aria-label="Open Semantic Sampling Figure 1 at full size"> <img class="figure" src="/assets/slides/2026-11-03-from-bits-to-tokens/figures/semantic-overview.webp" alt="Figure 1: precompute embedding neighborhoods, draw candidate tokens, sum their neighborhood probability masses, select a candidate slot, and emit its representative. Dog has probability 0.35 and neighborhood mass 0.42; cat has probability 0.28 and neighborhood mass 0.56."> </a> </div> </div> Note: The neighborhoods are constructed once from the model's input embeddings. At each token position, all proposals and scores use one model forward pass. The example draws dog and cat independently with replacement. Dog is individually more probable, but cat's neighborhood has more mass: 0.56 versus 0.42. With beta equal to one, the two proposed slots are selected with probabilities four-sevenths and three-sevenths. This is a stochastic selection, not a deterministic maximum. The figure shows representative output, which emits the selected proposal. The alternative neighborhood output draws within the selected neighborhood. Duplicate proposals remain separate slots. Source: quillpod/2027/mlsys/semsam/main/introduction.tex, Figure 1; main/semsam.tex; figures/semsam_overview.tex and figures/semsam_precompute.tex. --- <div class="qslide paper-figure"> <h2>Semantic Sampling in vLLM</h2> <div class="body"> <a class="figure-link" href="../assets/slides/2026-11-03-from-bits-to-tokens/figures/semantic-implementation.webp" target="_blank" rel="noopener" aria-label="Open Semantic Sampling Figure 2 at full size"> <img class="figure" src="/assets/slides/2026-11-03-from-bits-to-tokens/figures/semantic-implementation.webp" alt="Figure 2: initialize a shared GPU neighborhood graph while loading the model. Each generation step proposes tokens from processed logits, gathers and reduces neighborhood logits, selects and emits a token, and merges semantic and native batch rows in vLLM."> </a> </div> </div> Note: Initialization loads a cached graph or constructs neighborhoods while the remaining model weights load. The graph stores integer neighbor IDs, row offsets, and eligibility on the GPU and is shared across requests. At each decoding step, hierarchical proposals share a vocabulary pass. Sparse gathers then score the proposed neighborhoods using the same processed logits. A final kernel selects a slot and emits the representative or a neighborhood sample. The integration returns token IDs to vLLM with the original batch-row mapping. It does not search the embedding space online or add another model forward pass. Source: quillpod/2027/mlsys/semsam/main/semsam.tex, Figure 2, and figures/semsam_implementation.tex. --- <div class="qslide thanks"> <div class="closing-main"> <h1>Thank you! Questions?</h1> <div class="closing-qrs" aria-label="Slides and paper QR codes"> <a class="qr-card" href="https://mohsenhariri.github.io/slides/2026-11-03-from-bits-to-tokens.html"> <img src="/assets/slides/2026-11-03-from-bits-to-tokens/closing-slide_qr.svg" alt="QR code for this slide deck"> <span>Slides</span> </a> <a class="qr-card" href="https://mohsenhariri.github.io/research/2025-10-19-dfloat11.html"> <img src="/assets/slides/2026-11-03-from-bits-to-tokens/dfloat11_qr.svg" alt="QR code for the DFloat11 paper"> <span>DFloat11</span> </a> <a class="qr-card" href="https://mohsenhariri.github.io/research/2025-10-20-more-for-keys-less-for-values.html"> <img src="/assets/slides/2026-11-03-from-bits-to-tokens/kvq_qr.svg" alt="QR code for More for Keys, Less for Values"> <span>KV cache</span> </a> </div> </div> <div class="acknowledgments" aria-label="Acknowledgments"> <div class="ack-item ack-nsf"> <img src="/assets/slides/2026-11-03-from-bits-to-tokens/logo/nsf.svg" alt="National Science Foundation"> <div><h3>NSF Awards</h3><p>2117439, 2112606, 2320952</p></div> </div> <div class="ack-item ack-cwru"> <img src="/assets/slides/2026-11-03-from-bits-to-tokens/logo/cwru-hpc.webp" alt="Case Western Reserve University"> <div><h3>High Performance Computing (HPC)</h3><p>Center at CWRU</p></div> </div> </div> </div> Note: Thank you. QR codes link to this slide deck, DFloat11, and More for Keys, Less for Values. Acknowledge NSF awards 2117439, 2112606, and 2320952 and the High Performance Computing Center at CWRU.