Harry He

Founding member @radixark
Trust the process

[LMSYS Org @lmsysorg]

Inkling, @thinkymachines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio.

Serving and RL support are already live: you can run and shape it on an open stack, starting now.

Day 0 support on SGLang @sgl_project and Miles @radixark👇

- Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache

- Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization

- DFlash speculative decoding from @modal for low-latency serving

Launching now, blog and cookbook in the comments ⬇️

[DigitalOcean @digitalocean]

.@radixark brought @deepseek_ai V4 to production with DigitalOcean and @AMD.

AMD Instinct™ MI350X GPU Droplets, plus joint engineering that drove a ~10x throughput gain.

When the team that builds the inference engine picks your stack, that's the signal.

[Harry He @harryheisme]

@digitalocean @radixark @deepseek_ai @AMD Thx for collab!

[Harry He @harryheisme]

🧡🧡🧡

[SGLang @sgl_project]

⭐ SGLang just crossed 30,000 GitHub stars!

Thank you to our contributors, and to all the partners, model labs, and hardware teams who build with us. Every star, issue, and PR got us here.

MAX LOAD. MAX OUTPUT! 🚀

[LMSYS Org @lmsysorg]

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉

DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales.

We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec.

At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads.

With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300.

DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks @deepseek_ai for open-sourcing!

Blog with full technical details and commands to run below 👇

[Harry He @harryheisme]

Our open-source RL training framework, Miles, just hit the @PyTorch blog! Built to solve the distributed engineering pain points of frontier LLM RL (MoE, multi-precision, scheduling).
Perfect if you're building on DeepSeek V4, Qwen3.6, and beyond.

[RadixArk @radixark]

Miles is now featured on the PyTorch Foundation blog.

As models grow, shift from dense to MoE, and span more specialized hardware, RL post-training is no longer just about the algorithm. It is a distributed systems problem. Miles is our open-source RL training framework, built for exactly that.

It comprises four systems behind a small, pluggable trainer: SGLang (@sgl_project) for rollout, Megatron-LM (@NVIDIAAI) for training, Ray (@raydistributed) for orchestration, and PyTorch (@PyTorch) as the common layer for models and numerics.

Out of the box, you also get MoE-aware rollout/training alignment, a unified BF16/FP8/MXFP8/NVFP4/INT4-QAT pipeline, fast NCCL/RDMA weight sync, fault tolerance, and ready-to-run recipes for frontier models like DeepSeek V4, GLM 5.2, Qwen3.6, Kimi K2.6, and Nemotron 3 Ultra.

Our goal is simple: make frontier-scale LLM RL easier to reproduce, extend, and operate.

Thank you, PyTorch Foundation, and everyone who got Miles here, especially the legendary @slime_framework team!

[Harry He @harryheisme]

Best BBQ

[RadixArk @radixark]

🔥 Summer done right.

A whole roast lamb BBQ to celebrate the good times with friends, family, builders, and our community, World Cup on the big screen (good game of Spain vs. Uruguay ⚽), smoke rolling off the grill, cold drinks in the sun, and a backyard full of great people just soaking up the vibes, that's how we enjoy the summertime! 🍢☀️🍺

Great food, great people, great times. Huge thanks to everyone who came out, and shoutout to @FishAudio for co-hosting this BBQ with us. Here's to the rest of this summer together! 🫶

[SGLang @sgl_project]

🎉 SGLang v0.5.14 is out!

First, we welcome 55 new contributors to this release 🙌

And the supported new models: GLM-5.2, Kimi-K2.7-Code, LiquidAI LFM2.5, Poolside Laguna-M.1, DiffusionGemma, Zyphra ZAYA1, and MiMo-V2-ASR.

Here are the highlights for this release:
- New LPLB load balancer that evens out MoE expert traffic across GPUs (DeepEP)
- Kimi-Linear (KDA) runs faster on NVIDIA Blackwell with a new CuteDSL prefill kernel
- Lower memory use for linear-attention (GDN/KDA) models
- Faster multi-GPU communication with MSCCL++ and MNNVL allreduce fusion
- Nemotron now supports DP attention and MTP
- Breakable CUDA Graphs now run on AMD ROCm/HIP
- Multiple DeepSeek-V4 performance updates: NVFP4 MoE, FlashMLA head64 decode, and faster FP8 quantization

Thanks to our amazing partners and model makers: @NVIDIAAI @AMD @intel @Kimi_Moonshot @Zai_org @liquidai @poolsideai @GoogleDeepMind @ZyphraAI @XiaomiMiMo

Now. MAX LOAD! MAX OUTPUT!

[RadixArk @radixark]

Great to be at the @hud_evals hackathon @ycombinator! We met old and new friends and were really impressed by everyone working on the hard problems in “RL” (reinforcement learning and real life)!

We’re always hiring ambitious, amazing people who’d love to bring frontier RL infra to everyone. Come build with us!

https://t.co/HjhcQeF5fv

[SGLang @sgl_project]

SGLang 🤝 Cloudflare

[LMSYS Org @lmsysorg]

🚀 New blog: The next generation of speculative decoding: DFlash and Spec V2

DFlash + Spec V2 hit >4.3X baseline throughput for LLM inference, now the default speculative decoding engine in SGLang! Together with @modal and https://t.co/ZXetBKIRym, our jointly-released DFlash drafter for Qwen 3.5 397B-A17B beats both baseline and native MTP in every setting we benchmarked:
1️⃣ >4.3X baseline & 1.5X native MTP throughput (concurrency 1, HumanEval, 8xB200)
2️⃣ Block diffusion drafter: a full token block in one forward pass
3️⃣ KV injection: target-model features fed into every draft layer’s KV cache for higher acceptance
4️⃣ Spec V2 overlap scheduler: +33% end-to-end

Read the code, deploy a DFlash server, and start experimenting!

[SGLang @sgl_project]

🎉 SGLang v0.5.13 is out!

First, new model support!

Nemotron 3 Ultra, Step-3.7-Flash, Command A+, plus new diffusion models: Cosmos3, FLUX.2-Klein, Ideogram 4, LingBot-World, SANA-WM, and Ernie-Image.

Here are the highlights for this release:
- Speculative Decoding V2 is now the default! Tree drafting (topk>1) for faster generation
- Breakable CUDA Graphs now make prefill faster
- Qwen 3.5 runs faster on NVIDIA Blackwell with new GDN kernels
- HiCache with UnifiedTree on by default for hybrid SWA/Mamba models
- SGLang-Diffusion now supports realtime generation! Plus progressive resolution
- Multiple performance and feature updates for DeepSeek V4

Thanks to our amazing partners and model makers:
@NVIDIAAI @AMD @intel @awscloud @boson_ai @cohere @bfl_ml @ideogram_ai @deepseek_ai @Kimi_Moonshot @Alibaba_Qwen @StepFun_ai @Baidu_Inc @robbyant_brain

[Cheng Wan @ChengWan17]

The standout feature in SGLang v0.5.13 is BCG (Breakable CUDA Graph). It delivers prefill efficiency comparable to PCG (Piecewise CUDA Graph), while being significantly more flexible and compatible with advanced optimizations.

BCG is also a powerful debugging tool — it enables eager execution inside CUDA graph replay, so you can easily print debug info or inspect intermediate states.

Prototype by @csy_789. @Oasis_a19 turned it into a production-ready prefill optimization. I was fortunate to pick the name for this awesome technique. 😊

[Harry He @harryheisme]

pls stop those bias but that's really funny lol.

[Robert Sterling @RobertMSterling]

AI startup CFO, CEO, CMO, and CTO

[Harry He @harryheisme]

@michellechen no doubt Knicks in 5

[Harry He @harryheisme]

Happy to see y'all next time

[SGLang @sgl_project]

🏙️ SGLang NY Tech Week Happy Hour Recap

Last Wednesday, SGLang hosted a NY Tech Week Happy Hour in NYC, co-hosted with @HOFCapital, @Cloudflare, @CrusoeAI, and @ArklexAI. 380+ registered, 200+ in the room, and one unforgettable night. 🧡

The room was packed with engineers, researchers, and enthusiasts from quant funds, banks, and trading firms, all there to talk about one thing: where inference is headed as LLMs move into latency-sensitive production across trading, research, compliance, and risk.

NYC, you showed up and brought the energy. We loved every minute. Until next time! ☀️

#NYTechWeek @Techweek_

[Harry He @harryheisme]

@michellechen @sgl_project @lmsysorg Shining girl!

[Harry He @harryheisme]

oh yeahhhhh, SGLang community account is reactivated! What you want to see? Drop your idea here!

[SGLang @sgl_project]

👋 @sgl_project is back!

Welcome to the home of the SGLang community! While @lmsysorg keeps you posted on technical drops and partner news, this space is for you!

Here's what we've got lined up:
🚀 Version releases: every new SGLang drop, unpacked
🎙️ Office Hours: deep dives, live deployments, and team Q&A
📺 Tutorials: short how-tos and the best Office Hour moments
🌟 Community spotlights: the cool stuff you're building with SGLang
📅 Event updates: meetups, workshops, and where to catch us next

And we'd love to hear from you! What do you want to see? Benchmarks? Model deep dives? A topic for the next Office Hour?

Drop it below 👇 Every idea gets read!

[Intel @intel]

@lmsysorg Great collaboration with @lmsysorg - this is a strong example of how Intel Xeon + AMX unlocks better latency and throughput by offloading vision encoding. Heterogeneous CPU + GPU scaling is delivering real-world gains for VLM serving.

[LMSYS Org @lmsysorg]

Huge milestone for the Microsoft AI team: seven frontier MAI models, led by MAI-Thinking-1. Proud that SGLang powered the RL inference stack behind it. Their Rocket framework runs SGLang and the SGLang router for load balancing, traffic control, prefix caching, and graceful failure recovery across thousands of inference chips.

Congrats to the team @MicrosoftAI 👏

Read more on how SGLang powers the stack: https://t.co/60fxfv6DWb