Connor Li

Research Engineer @togethercompute | Prev @CMU | | First author of FlashStates

[Connor Li @connorliml]

FlashState is out ⚡️

Wild to see so much love for this! It started with a long car ride to SFO with @_weilix, chatting about ideas that grew out of Weili’s summer at @togethercompute, and here we are.

So much fun co-leading this with Weili!

Next up at @togethercompute : taking it from research to distributed production serving, and bringing it to more diverse models, agentic workloads, and newer hardware.

Long road ahead, stay tuned! 🚀

[Weili Xu @_weilix]

Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads.

On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.

[Connor Li @connorliml]

@zainhas right on! then what might also be needed is an algorithmically verifiable system.

[Connor Li @connorliml]

@ywangfirstlean @_weilix @togethercompute thanks Yubo! Hope we will learn more as we promote it into production.

[Connor Li @connorliml]

FlashState is out ⚡️

Wild to see so much love for this! It started with a long car ride to SFO with @_weilix, chatting about ideas that grew out of Weili’s summer at @togethercompute, and here we are.

So much fun co-leading this with Weili!

Next up at @togethercompute : taking it from research to distributed production serving, and bringing it to more diverse models, agentic workloads, and newer hardware.

Long road ahead, stay tuned! 🚀

[Weili Xu @_weilix]

Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads.

On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.

[Connor Li @connorliml]

@lm_zheng Amazing!

[Connor Li @connorliml]

@jisenli_ai @_weilix Thanks Jisen!

[Connor Li @connorliml]

We report p90 TTFT in the paper: up to ~88% lower vs vLLM. And yeah, I agree the tail is where it shows up first. p90 is just the number I personally trust most, lol.

[Connor Li @connorliml]

@Alex7647276771 @_weilix Good catch, the 60s isn't a knob, it's just how we sliced the data. Long tool waits are exactly what hurts naive checkpointing: with plain LRU those requests lose their state. Cost-aware eviction gets most of it back, since expensive long-context states hold priority longer.

[Connor Li @connorliml]

Hey Sebastian, that's actually the whole setup: dozens of independent agent sessions fighting over one cache. vLLM's hit rate falls off a cliff while ours stays high (Fig 6 in the paper). We also defined it as "capacity misses" in Paper motivation part.

[Connor Li @connorliml]

@jack_feifei @_weilix Yeah we swept that! Short version: when everything fits in the cache, gains are modest. Once the working set outgrows memory, vLLM's hit rate falls off a cliff and that's where FlashStates pulls away. Fig 6 + appendix C have the curves to answer the detailed numbers!

[Connor Li @connorliml]

@SonglinYang4 Thanks for sharing, Songlin!

[Connor Li @connorliml]

@simran_s_arora @togethercompute @_weilix Thank you Simran! Hearing this from you means a lot 💙

[Connor Li @connorliml]

@tejasybhakta @_weilix it should work on disk offload, we just haven't covered this scenario yet.

[Connor Li @connorliml]

@_weilix It's been great to work with you! 🥳

[Connor Li @connorliml]

Hybrid Attention models are becoming the default for frontier serving, but prefix caching was built for pure attention. Under agentic workloads it breaks: a KV hit is useless without the matching recurrent state, so every turn re-prefills.

FlashState checkpoints that state exactly where the next turn resumes, and evicts by how much prefill each state saves. No tuning.

On 48 Claude Code-style agents (2×B200 per engine):
→ up to 2.9× output TPM vs vLLM
→ 97% fewer uncached tokens
→ 87% lower p90 TTFT

Tested on GLM-5.3-Flash (@Zai_org), Nemotron 3 Super (@NVIDIAAI) and Qwen3.8-Next (@Alibaba_Qwen), plus a Kimi-K3.1-Pro (@Kimi_Moonshot) build on GB300.

Benchmarked on the open AgentX traces and harness from @SemiAnalysis_. Huge thanks for making agentic serving measurable. Would love to see hybrid-model caching in InferenceX!

#LLM #inference #AgentX #infernceX

[Weili Xu @_weilix]

Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads.

On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.

[Connor Li @connorliml]

Hi, There!