I’m beyond excited about this. When I first started working on on-device personal agents, this was the dream—and now it’s real.
We built a real-time visual indexing method and brought it to life in a working product. The idea is wild: you shouldn’t have to remember every little thing you’ve seen or done on your computer. Your agent should remember it for you. 🤯
It’s already helped me dig up an old Golden State Warriors ticket and find the GEN vs. BLG recap video I was watching over lunch yesterday. Those “wait, where did I see that?” moments? Just ask your agent.
Multimodal. Multilingual. Real-time. All running quietly on your computer, private, lightweight, and ready when you need it.
Incredible work led by @dayoon12161, and an amazing collaboration as always with @sewon__min 🚀
💻 Code https://t.co/oQYVJ7XG8v
📄 Paper https://t.co/HsLeKuxMHF
The age of bespoke serving engines is upon us! To build them, having a principled understanding of the performance characteristics before you build is super important. Check out @KanZhu854772’s work on building a bespoke prefill engine for GLM 5.3 using ServingStudio!
Pipeline parallelism, custom prefix caches, and thoughtful schedulers really do boost peformance of LLM serving for coding agents! With ServingStudio to predict how a design performs, we were able to build a bespoke prefill engine for GLM 5.3-Flash that kills the competition!
Work led by our own @KanZhu854772!
Kan is on a tear 🔥 We’re already building bespoke systems for specific models and chips. Go one level deeper: prefill and decode each deserve their own serving stack.
Kan’s GLM-5.3-Flash prefill engine hits 3.2× vLLM’s peak throughput replaying coding-agent traces on four B200s. 👇
Solid work from @KanZhu854772 on SSEngine 🔥🚀 Open-source prefill engine for GLM-5.3-Flash on four B200s, hitting 3.1–3.2× higher request throughput than vLLM/SGLang on coding-agent traces 📈
Pipeline parallelism + tiered prefix cache + shortest-remaining-prefill-first scheduling really moved the needle here. Great example of how much specialized prefill stacks can outperform general serving frameworks.
🔥 Serving coding agents well needs a high-performance prefill engine, but existing systems fall short.
🚀 Introducing SSEngine-GLM5.3-Flash: an open-source prefill engine for GLM-5.3-Flash on four B200 GPUs. It combines pipeline parallelism, a tiered prefix cache across GPU memory, DRAM and SSD, and a shortest-remaining-prefill-first scheduler.
📈 On real hardware, it sustains 3.1× (AgentX) and 3.2× (TraceLab) the request rate of any vLLM or SGLang setup we tested.
🧵 More in the thread below (1/8)
Coding agents now write a growing share of ML compiler code. Which language should they write it in: Rust, Python, or C++?
Each language has different strengths, so why pick just one?
Meet TIRx Rust 🚀: a Rust-native, cross-language ML compiler foundation for agents, with:
🔗 Fine-grained access to shared IR objects: Rust natively uses objects from any language, no bindings.
🦀 Ergonomic Rust APIs: write passes natively in Rust, as easily as in C++ or Python.
🧩 Dynamic extension: add new objects and functions as standalone modules, with zero changes to the core.
How does it work under the hood? Full post 👇
blog link: https://t.co/UtXaL4RQoj
Reasoning models can now drive robots. But the robot sits still while the model thinks.
SpecPilot now speeds up agentic robotics. On PEAR (Qwen3.8-27B as a robot policy), the same LIBERO task finishes in 49 s instead of 189 s.
3.84× faster task completion
71 → 241 tok/s decode
No training 🤖
Beyond Score Centering: what else can we fix in async RL? 🧵
Score centering (SC) is the hot new trick for off-policy LLM RL. We stress-tested it under real async staleness (every batch up to 127 steps old) and found what it fixes, what it doesn't, and two cheap wins on top
🔍 What SC actually fixes
• Without SC, a trajectory's gradient norm grows ~linearly with its age: at age 64 it's 3.2× an on-policy one. With SC: 1.5×
• The culprit is a hidden drift term, an uncontrolled −A·KL(μ‖π) baked into the policy gradient. SC removes it
⚠️ What SC + IS gets wrong
• SC then IS → adds a new drift
• IS then SC → a no-op without truncation; with truncation it centers on a distribution with no guarantee of being closer to π
• So "IS and SC are orthogonal" (from the SC paper) doesn't hold
Two wins on top of SC
1️⃣ State-level KL mask: drop states where KL(μ‖π) > τ, instead of masking by token-level ratios. It depends only on the state, so SC's baseline property survives. At age 127: 3/3 seeds climb steadily, while TIS+SC and MIS+SC spike and crash
2️⃣ Random buffer sampling, a free lunch from the system: same buffer, sample at random instead of FIFO. Mean age 127 → 52, better for every method at equal wall-clock
Recipe = SC + KL penalty + KL mask. At lr 4e-2 (w/ SGD), MIS+SC and DPPO collapse and TIS+SC swings, while every recipe seed trains and learns faster early
🤔 So far, we’ve only tested on Countdown with Qwen3-0.6B (wants to scale it up!). When staleness is low, all the fixes perform similarly, but the differences become clearer as staleness increases. We also haven’t compared against sequence-level masking (which might already be enough!) or isolated kernel mismatch from async lag. Both are worth looking into next.
#LLM #RL #AsyncRL #ReinforcementLearning
We breathed life into LLM system.
Introducing Evolving Infra⚙️: an inference framework that adapts diverse and dynamic agent workload purely online and retires fixed PD disaggregation.
vs. NVIDIA Dynamo on up to 256 GB300 GPUs:
➡️ Up to 4.1× lower latency
➡️ Up to 14× lower P95 TTFT
➡️Up to 2.36× more revenue
on real agent traces(Computer Use, Coding and math agents) and opensource models.
🌐 Website: https://t.co/7I3iERjE6b
📄 Tech report: https://t.co/qDGcwl0svx
💻 Code: https://t.co/0VOBCptKcb
Which agent(e.g. Pi/Openhands) do you want us to accelerate next? Tell us below 👇
Everyone's been showing Fable and GPT-6 Astra driving robot arms.
We trained an open 27B Qwen model to do it, purely from its own attempts in simulation, with zero demonstrations.
And it can run locally at 220+ tok/s on an RTX 5090.
On a real robot it had never touched, it now succeeds in 18 of 30 tests, up from 0 before training. It also copies a painting and designs its own chemistry experiment.
It generalizes and follows instructions better than π0.5, and on LIBERO it comes close to Astra max.
All without losing its general capabilities, like math and coding.
Built on Qwen3.8-27B. We've open-sourced the model weights, code, and full training infra! Check out our blog at https://t.co/CYHRKigPQP.
So glad to have the opportunity to share our team's recent work of rebuilding a training framework at #SGLangSummit. We'll share blog posts and open source the project very soon. Stay tuned 😜
🚀 Qwen, Kimi, GLM are all betting on linear attention. But it's still slow: the recurrent state is read & rewritten every token.
Meet LeapQuant: the first training-free method to make 8-bit or 4-bit recurrent states work.
📈 Across Qwen, Kimi, and GLM models:
• Matches FP32 accuracy
• 4.25× faster linear attention kernels
• Up to 2× higher inference throughput
• Cuts memory by up to 56%