Ayushman Singh ✈️ NeurIPS’26 🇦🇺

LLM Inference @Sesame. Views are my own.

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

I’ve published a follow-up to my earlier post on speculative decoding.

This one traces how the drafter has evolved, from Medusa’s parallel heads and EAGLE’s feature-level autoregression to Gemma 4 MTP, DFlash, and the recently released DSpark.

What changed is not just improved draft accuracy. Newer designs also try to remove sequential draft passes, reuse target state, and avoid verifying suffix tokens that are unlikely to survive.

https://t.co/soANxyEivD

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

There's still a lot of juice left in existing draft model architectures, just from training them better. DStill keeps the DFlash drafter architecture unchanged and gets up to 28% higher acceptance length and 25% more speedup at the same inference cost. Great work by @shreyask269 and @madhav_kanda_ .

A few training insights from the paper I liked:

1. Running vanilla DFlash for 8 denoising passes lowers acceptance length (4.45 → 3.70 on GPT-OSS-120B), because it only saw fully masked blocks in training. A drafter trained on partially revealed blocks improves with 8 passes (4.50 → 5.31).

2. They distill that 8-pass teacher into a one-pass student with the same architecture. The student is trained to predict the teacher's final block from any state on its refinement trajectory, and recovers 29-36% of the multi-step gain in one pass.

3. Distillation uses a curriculum: the student trains on both intermediate and fully masked states, and the share of fully masked states rises from 0.5 to 0.85 over training. This beats training on fully masked states only.

4. Longer training doesn't explain the gain. DFlash trained for twice as many epochs gets slightly lower acceptance length. The DStill gains over DFlash hold at concurrency up to 32, and the method also improves DSpark (+4.5% acceptance length).

[Shreyas Kulkarni @shreyask269]

Can diffusion drafters benefit from multi-step denoising while still drafting in a single pass?

Excited to share DStill, our new work on accelerating block diffusion speculative decoding! 🚀

We learn from the drafter’s denoising trajectories during training to improve draft acceptance, while keeping drafting to a single forward pass at inference.

⚡ Up to 6.45× speedup over autoregressive decoding.
📄 https://t.co/XTtlzqsv9I

A thread 🧵👇

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

@tugot17 @sgl_project Plz live stream or share a recording for those of who can't make it in person

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

I finally got the time to read the JetSpec++ paper. Some thoughts.

It reports impressive results on Qwen3-8B: up to 7.45× speedup over autoregressive decoding, 1.54× over DSpark, and 1,361 tok/s on GB200 GPUs. But imo the headline numbers are misleading. The AR baseline runs on a single GB200, while JetSpec++ uses two.

Table 3 compares JetSpec++ on 2 GPUs against DSpark and DFlash on 1. Simply running two replicas (DP=2) would, by linear extrapolation, give DSpark 1,804 tok/s and DFlash 1,591 tok/s, both exceeding JetSpec++'s 1,361 tok/s on MATH-500.

I'd argue JetSpec++'s speedup over DSpark relies heavily on the additional compute: acceptance lengths are nearly identical (9.41 vs 8.98 on MATH-500), so almost all of the 1.51× comes from hiding drafting with an additional GPU.

In fairness, JetSpec++ optimizes single-request latency, where DP=2 wouldn't help. But even latency-sensitive serving runs at low concurrency, not batch size 1, and the paper reports nothing beyond batch 1.
And in edge scenarios, where batch-1 latency does matter, a second accelerator dedicated to drafting is a big ask.

[Hao AI Lab @haoailab]

Introducing JetSpec++: we find causal parallel drafting paired with hidden states predictor and anchor token decoder can effectively draft the next future block before the current one is verified, and spending the freed time on self refinement.

This allows drafting to be pipelined with verification, hiding almost all of its drafting cost.

JetSpec++ reaches up to 1.54x end-to-end speedup over DSpark and 7.45x over autoregressive decoding while keeping lossless, at 1,361 TPS on Qwen3-8B and 367 TPS on Qwen3.8-27B in BF16.

With NVFP4 and kernel optimizations, JetSpec++ further reaches over 4,000 TPS on Qwen3-8B and over 900 TPS on Qwen3.8-27B, on two GB200 GPUs. ⚡️

Project page: https://t.co/tuhf2KPfsW

[NeurIPS Conference @NeurIPSConf]

📰 The NeurIPS October 2026 Newsletter is out!

Catch up on #NeurIPS2026 updates: Grants, Affinity events, Tutorials, Accessible AI research, Journal Track, Invited talks, and Regional AI gatherings.

🔗 https://t.co/jroQuHLlDQ

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

@tugot17 NY >>> SF

[Manya Ghobadi @systalyze]

@systalyze is #1 in output speed on @ArtificialAnlys for GLM 5.2, in both reasoning and non-reasoning modes.

→ 538 tok/s: 1.7× faster than @baseten and @CoreWeave , 5.5× @Zai_org own endpoint
→ Cheapest provider on non-reasoning
Try it: https://t.co/2T88u08Tpx

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

@nkohari @sriniously I'd be interested in getting early access!

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

@sharongoldman Would love for you to try out the @sesame app, we shipped a big update this week

[Raven Jiang @DarkMirage]

I've been exploring using voice for steering coding sessions on my commutes at @sesame and I really like it.

Working with Opus/Sonnet 5.5 is at the point where I often find it useful to multitask over voice even while I'm at my desk.

Try it at https://t.co/OiJRJi6f1h

[Sesame @sesame]

Frames you love to wear, with an assistant you enjoy talking to.

Voice assistants available to everyone today. Glasses coming 2027.

This release is a step towards our vision—a voice OS on the frames that can connect to your tools, act on your behalf, and help you throughout your day.

iOS: https://t.co/OEQs8yJJyf
Android: https://t.co/9spgYH9Ubl

[Sesame @sesame]

Our collection will be Made in Japan, available in 2027.

To celebrate the craftsmanship behind it, we commissioned a photograph by Osamu Yokonami. Today on the blog, we’re sharing more about our road to eyewear and a look behind the scenes of the commission.

https://t.co/4tyqp49Jtl

[Sesame @sesame]

Open... 10/6

Image by Osamu Yokonami

[Manya Ghobadi @systalyze]

@systalyze is now #1 in output speed on @ArtificialAnlys, on every model we entered.

→ Gemma 4 26B: ~10× faster than Google
→ DeepSeek V4 Pro: 4× faster than DeepSeek
→ Muse Glimmer: 2.5× faster than Fireworks

Try it via our OpenAI-compatible API:
https://t.co/2T88u09rf5

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

@sarahookr @adaption_ai Well you can now see where the blue bins go when they disappear. Still a partial victory lol

[Shekhar @indianspeedster]

We’ve been cooking on low-precision pretraining at AMD, and I’m excited to give a talk about it with Liz Li at PyTorch Conference!

We’ll share our work on scaling MXFP8 training with TorchAO and TorchTitan on October 21, 2:15–2:40 PM PDT, in LL20AB. Hope to see you at the talk!

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

@tugot17 And here’s the view from @sesame HQ

[Sesame @sesame]

To craft for conversation, timing is everything. Natural conversation is a social dance, and timing matters just as much as tone and delivery.

Most voice AI doesn't do this well yet.

So we built TurnBench: an open benchmark that measures whether voice AI knows when your turn is really over, and when an "mm-hmm" is agreement, not an interruption.

Learn more about how we built this on our blog.

https://t.co/idk8fzJH0W

[Ayushman Singh ✈️ NeurIPS’26 🇦🇺 @ayush1399]

@jessiedong_ @tugot17 Utilyze from @systalyze shows how much work is your GPU really doing. Worth checking it out imo

https://t.co/sxABzzqgj7

[Sesame @sesame]

Hey Charlie,

I hear the Sesame preview is now on Android.