Ze-Wei (Johnny) Liou

First year PhD @Princeton working on ML Systems

[Liane Galanti @lianegalanti]

Robot policies can move but can't think.
LLMs can think but can't move.
So we connected them.
Real robot: 16.7% → 97.3%
Sim (LIBERO-PRO): 12.8% → 53.3%

[Ze-Wei (Johnny) Liou @zwljohnny]

@KaichaoYou Thanks so much! Yeah, I'd love to. Just DM'ed

[Ze-Wei (Johnny) Liou @zwljohnny]

Why do we store the SSM state at all?
More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters.

We only write it back every step so the next step can read it.
ReplaySSM caches the recent inputs instead and rebuilds the state on the fly.

Same outputs, half the memory traffic
→ ~2x on spec decode at large batch sizes, which barely even helped SSMs before
→ up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B)

Work with @tri_dao

Blog + Code👇

[Ze-Wei (Johnny) Liou @zwljohnny]

@woosuk_k Thanks! I'd love to

[Ze-Wei (Johnny) Liou @zwljohnny]

The key idea is that storing the updated state is the natural design choice, but SSM actually has the flexibility to store recent input (k and v) or store the states.

The output-only route is a benefit from the concept. This part is more similar to chunkwise parallel training (outputs within a chunk are computed from a shared initial state)

[Ze-Wei (Johnny) Liou @zwljohnny]

@VipulS_1 The lack of parallelism and the state materialization cost become more pronounced in GDN and at larger batch sizes, where the benefit of amortizing weight loads becomes smaller.

[Ze-Wei (Johnny) Liou @zwljohnny]

Hi, previous rollback approaches still materialize and write at least one recurrent state back to HBM at every step. Committing that state each step makes them still suffer from sequential state dependency, since the state for the last committed tokens has to be rebuilt before the current drafts are verified.

[Ze-Wei (Johnny) Liou @zwljohnny]

@adityastomar_ Yes, in flush route, both state and output are needed, so we multiply VK^T for the state, and read the output by multiplying with q_t.

In most of the step, we directly calculate the output (we don't need the state)

[Ze-Wei (Johnny) Liou @zwljohnny]

[6/N]
ReplaySSM works for both standard and speculative decoding, and it generalizes to GDN. We built it on vLLM and tested at serving batch sizes, 4B to 550B. Standard decoding gets up to 1.48x, and speculative decoding hits 1.96x. The speculative decoding result is particularly important because vLLM's current SSM speculative decode is actually slower than just decoding token-by-token.

[Ze-Wei (Johnny) Liou @zwljohnny]

[5/N]
ReplaySSM also changes what must be produced. Before, states and outputs were both needed. Now, most steps only need the outputs.

So we compute the output directly from the buffer and never materialize the state at all, which is what the figure below shows for a single decode step. Speculative decoding is where this really pays off, since the same output-only form removes the serial state dependence that made SSMs hard to parallelize and turns the whole draft verification into GEMMs.