Tal Schuster

Working on Gemini @GoogleDeepMind | formerly: PhD @MIT_CSAIL @MITNLP. Opinions my own

[Tal Schuster @TalSchuster]

Introducing our work @GoogleAI CALM: Confident Adaptive Language Modeling 🧘

Large Language Models don't need their full size for every generated token. We develop an Early Exit framework to significantly #accelerate decoding from #Transformers!

🔗: https://t.co/KOlXTKKoqd
🧵1/

[Behnam @OrganicGPT]

Local AI at 400 tok/s, zero loss, Gemma 4 31B 🤯 Google's MTP approach is way better than Qwen's because it SCALES with MTP parameter but Qwen peaks at MTP=3. I can now have FULL BF16 Gemma 4 31B at +400 tok/s on RTX 6000 Pro using MTP=100, video coming soon!

[Tal Schuster @TalSchuster]

@liranringel @urieli17 For example multilingual output and temp sampling and serving setup without too much wasted compute (e.g. CPU/ macos or batch size > 1 per GPU).

But if you have compute to give and want to maximize speed but not throughout, then DDTree looks like a very nice solution!

[Google Gemma @googlegemma]

Gemma 4 up to 3x faster, directly in your phone! 🚀

Check out the difference Speculative Decoding makes! Multi-Token Prediction (MTP) is supercharging inference speeds for Gemma 4.

[Tal Schuster @TalSchuster]

@zahihod @urieli17 https://t.co/sKP5pCrHsI

[Omar Sanseviero @osanseviero]

Gemma 4 Drafters landing across the OS ecosystem

✅transformers
✅VLLM
✅MLX
✅SGLang
✅Ollama
✅AI Edge Gallery

And more coming!

[Tal Schuster @TalSchuster]

@ZanyMan_e @urieli17 אורך קונטקסט וגודל מודל זה שני דברים שונים

[Tal Schuster @TalSchuster]

@liranringel @urieli17 בסטאפים מאוד ספציפיים אולי לפי מה שאני מכיר כרגע. לא בכללי. אבל מעולה שיש התקדמות

[Tal Schuster @TalSchuster]

@urieli17 @LiorParente Was released with day 0 support thanks to great partnership :)
https://t.co/pAkjgxqqOD

[Prince Canuma @Prince_Canuma]

Congratulations to @GoogleDeepMind on the launch of Gemma 4 Multi-Token-Prediction Drafters 🎉🚀

Happy to have partnered with them for Day-0 support on MLX

The new drafters accelerate both single and batch requests by upto 3x.

Here is a graph showing how different block sizes affect performance.

MLX-VLM release coming soon!

PR and model collection 👇🏽

[Benjamin Marie @bnjmn_marie]

I benchmarked Google’s new MTP for Gemma 4 31B using vLLM with 4 speculative tokens, a fairly conservative setup.

Results:
- Much higher throughput than Qwen3.6’s MTP

- Lower latency too, helped by Gemma 4 generating fewer tokens

- For coding tasks with reasoning enabled, Gemma 4 is now at least 6x faster than Qwen3.6. So you can generate 5 outputs, run your tests to select the best one, and it would still be cheaper than a single output by Qwen3.6.

I’ve updated my full comparison with the new numbers:
https://t.co/WU9VpQVU2Q

I also confirmed what others have reported: Gemma 4’s MTP handles a high number of speculative tokens very well.
On simple text generation, I’m now testing values above 10 and reached 129 tok/s on an RTX Pro 6000, compared with 20 tok/s without MTP.

Next step: confirming how this translates to real tasks.

[Tal Schuster @TalSchuster]

@bnjmn_marie Thanks for benchmarking! Nice to see

[stevibe @stevibe]

Google dropped MTP versions of Gemma4. Ran them on my DGX Spark.

The 31B dense model went from 3.94 → 8.91 tok/s. That's +126%.

Full results:

[26B A4B]
> 25.24 → 31.69 tok/s (+25.6%)
> TTFT 755 → 332ms (-56%)

[31B]
> 3.94 → 8.91 tok/s (+126%)
> TTFT 599 → 378ms (-37%)

If you're not running MTP, you're leaving free perf on the table.

[Tal Schuster @TalSchuster]

@zhijianliu_ Really nice to see the fast progress

[Tal Schuster @TalSchuster]

Nice work from @zhijianliu_'s lab!

Native Gemma drafter gives high speedups across the board. For certain cases like low entropy outputs (greedy decoding, structured etc.) and memory bound stup (small bsz+strong device), specialized techniques like this could further boost gen!

[Zhijian Liu @zhijianliu_]

DFlash for Gemma 4: Up to 6x Faster. ⚡⚡

Great to see MTP land natively in Gemma 4 today. If you want to push it further, try DFlash — open source, same quality, more speed!!

https://t.co/wKcRoibuOB

[Tal Schuster @TalSchuster]

@urieli17 נשמח לשמוע חוות דעת :)

[AshutoshShrivastava @ai_for_success]

🚨 Google just made Gemma 4 up to 3x faster with MTP ⚡

Same quality, way more speed. It predicts multiple tokens at once and verifies them in parallel, removing latency bottlenecks.

You can also run powerful models locally on mobile like me using Google AI Edge Gallery.

[vLLM @vllm_project]

🚀 Day-0 MTP support for Gemma4 now available at vLLM with ready-to-use docker image!

⚡️Enjoy up to 3x faster decoding performance to supercharge your development with zero quality degradation!

Check out the full vLLM recipes for Gemma 4 model series👇
https://t.co/IrCaaa6SIo

[Tal Schuster @TalSchuster]

@ttunguz Actually haven't seen it on M5 yet so that's very helpful:)

[Tal Schuster @TalSchuster]

We've just released open source MTP style drafters for Gemma 4 models ⚡

Now Gemma 4 models are even faster on your choice of hardware, without losing quality!

Grateful for the fruitful collaboration between my team, Gemma team, and many collaborators to enable this release!

[Omar Sanseviero @osanseviero]

Excited to introduce Gemma 4 Multi-Token Prediction Drafters⚡️Accelerated inference right in your pockets

- Up to a 3x speedup
- Same quality guarantees
- Available in your favorite open-source tools

[Tal Schuster @TalSchuster]

@ttunguz Very nice! Thanks for sharing

[Tomasz Tunguz @ttunguz]

@TalSchuster 2x speedup on Mac M5 is real.

[Google Gemma @googlegemma]

Gemma 4 just got even faster!

We're releasing Multi-Token Prediction (MTP) drafters that deliver up to a 3x speedup, without any degradation in output quality or reasoning logic.

[Tal Schuster @TalSchuster]

And beautiful benchmarks from @Prince_Canuma with MLX on Apple silicon

https://t.co/pAkjgxqqOD

[Prince Canuma @Prince_Canuma]

Congratulations to @GoogleDeepMind on the launch of Gemma 4 Multi-Token-Prediction Drafters 🎉🚀

Happy to have partnered with them for Day-0 support on MLX

The new drafters accelerate both single and batch requests by upto 3x.

Here is a graph showing how different block sizes affect performance.

MLX-VLM release coming soon!

PR and model collection 👇🏽