Yubo Wang

Inference at Together AI, Member of LightSeek Foundation. Author of Torchspec:

[Yubo Wang @ywangfirstlean]

@connorliml @_weilix @togethercompute nice work, Connor!

[Yubo Wang @ywangfirstlean]

@ekzhang1 DeepInfer is all you need

[Yubo Wang @ywangfirstlean]

@zijingliu__ bro you need to follow me

[Yubo Wang @ywangfirstlean]

@LijieyYang I think some ablations would be really helpful! I spent some time investigating numerical issues, and I attributed them to batch variance. Have you been able to isolate the source of the noise?

[Yubo Wang @ywangfirstlean]

@woosuk_k In the study itself, looks like its explained as numeric issues due to positions of token in the block but I don't see actual ablation. The numerics could be caused from the batch variant behavior. However, the adversarial spec idea looks solid.

[Yubo Wang @ywangfirstlean]

happened to have a similar observation and had a discussion on this with @woosuk_k, If you remember. happy to see it is well studied now!

[Lijie(Derrick) Yang @LijieyYang]

Spec decoding is lossless in distribution but fails exact match in precision 🫠, because in real-world serving, later draft slots get noisier.

We taught the drafter to fumble key decisions, where the target rejects it and writes them from the cleanest slot. +7.0 on DeepSWE for GLM-5.3, same speed 🚀.

Check our work led by @lhyTHU!

[Yubo Wang @ywangfirstlean]

@woosuk_k so basically I was saying you can actually alternate the model behavior if you tune draft model differently, but the argument you made is it is lossless so model behavior shouldn't change.

[Yubo Wang @ywangfirstlean]

@woosuk_k hint: some dinner event at Village Pub

[Simran Arora @simran_s_arora]

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨

Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.

Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.

From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!

[Yubo Wang @ywangfirstlean]

@_marwanad @aaronbatilo @ArfurGrok @ekzhang1 no longer a machine anymore

[Yubo Wang @ywangfirstlean]

@ekzhang1 @zhyncs42 You can also pin weights onto GPU memory and hot reload the engine while agent developing

[Yubo Wang @ywangfirstlean]

@cHHillee TP is buried in yesterday already.

[Yubo Wang @ywangfirstlean]

@cHHillee Used to be a troll on benchmaxxing inference performance that does not work for production traffic.
Example:
"8k in 1k out with 1000 Tokens/s"

[Yubo Wang @ywangfirstlean]

@nrehiew_ tp would lead to waste compute on the MLA kernel as it requires padding I think

[Yubo Wang @ywangfirstlean]

@RaghavMehtaM1 @vllm_project its open source, man

[Yubo Wang @ywangfirstlean]

speculator acceptance rate = 💵, iykyk

[LightSeek Foundation @lightseekorg]

EAGLE-4 is here—a game changer for agentic coding, with massive decoding speedups.

Honored to be a day-0 engine partner with @EagleCorp

Train with TorchSpec. Serve with TokenSpeed. Training and inference in one diagram 👇

Technical deep dive: https://t.co/0UuG3JLkFN

[Yubo Wang @ywangfirstlean]

@SuJinYan123 为什么host KV不做池化呢，是为了统一有无ib的集群么

[LightSeek Foundation @lightseekorg]

Day-0 support for Muse Spark Open on TokenSpeed.

190 output tokens/s per user on NVIDIA B300 — and that’s without speculative decoding.

Also supported on AMD Instinct MI355X.

👉 https://t.co/MJzhCEg7m8