Ratish P

improving multimodal inference @sgl_project @lmsysorg | Graduate @HeriotWattUni

[Ratish P @ratishp22]

This article explains why mathematically equivalent interpolation changed Qwen3-Omni, why EOS was not a safe cleanup point, and how a Higgs regression found on the upgrade branch turned out to predate the bump.

https://t.co/19iir0hooP

[Ratish P @ratishp22]

Upgrading SGLang Omni’s pinned SGLang runtime from 0.5.12.post1 to 0.5.16 looked like dependency maintenance. It became a 162-file integration project spanning decode-step token handoff, Qwen3-Omni floating-point behavior, request cleanup, and GPU memory accounting

[Ratish P @ratishp22]

Upgrading SGLang Omni’s pinned SGLang runtime from 0.5.12.post1 to 0.5.16 looked like dependency maintenance. It became a 162-file integration project spanning decode-step token handoff, Qwen3-Omni floating-point behavior, request cleanup, and GPU memory accounting

[Ratish P @ratishp22]

@AIatMeta co-residency is the systems result: 4-bit weights share 24 gb with the kv cache, vision encoder and dflash’s 16-token drafter. the 3.1× rtx 5090 result is batch-1 greedy; curious how acceptance and speedup hold under temp=1 sampling across reasoning and tool-call phases.

[Ratish P @ratishp22]

the non-obvious boundary is host-wide, not tensor-wide: retaining pinned buffers is optimal for a fixed training loop, but the same cache can squeeze every colocated workload’s page cache. @ezyang makes a good case that pinned-memory cache policy is really workload scheduling.

[Edward Z. Yang @ezyang]

Claude wants to teach you about pinned memory! New blog post: https://t.co/nEHJBmy01c Edited by yours truly.

[Ratish P @ratishp22]

@Meta_Engineers the real scaling result is across the stage boundary: the async user model gets 45× the online ranker’s sequence compute and compresses thousands of events into one 2048-d vector, yet ~50% of its gain survives downstream with no measurable p99 latency change.

[Ratish P @ratishp22]

@MistralAI the key is iso-content contrastive training: the same document must score yes for a target policy and no for its sibling, so the model cannot hide behind generic unsafety. curious whether a fixed 0.5 threshold stays calibrated across unseen policy phrasing and strictness.

[Ratish P @ratishp22]

@GenAI_is_real imagine a VC seeing this pitch deck, I think their names are enough to say everything

https://t.co/mCKwcOkASV

[Jeff Dean @JeffDean]

We created a pitch deck to tell a handful of VC firms about us and what we were up to (a fun experience!). Here’s a few slides about our background and some of the things we’ve worked on from the pitch deck (it was fun putting together the list of people in our teams who have gone on to found a whole range of exciting companies). We are delighted to have selected @radicalvcfund
and @khoslaventures to lead our initial funding round, along with participation from @lightspeedvp, @kleinerperkins, Doerr Capital (@johndoerr), and Alphabet (@Google). We’ll be working with them to close our seed round over the next few weeks.

[Ratish P @ratishp22]

2/2 resolve rebuilt reporting ids on gpu; .tolist() copied them back. keeping autoregressive ids on gpu and reporting ids on cpu removed the bounce.

benchmarked on seed-tts en at c8, 1,088 requests: -7.1% mean latency, +14.8% request throughput.

[Ratish P @ratishp22]

1/2 a tensor can be on the right device for one consumer and wrong for another.

while profiling higgs tts async decode in sglang omni, i found a reporting tensor bouncing cpu→gpu→cpu even though the next-step token ids were already available on gpu.

[Ratish P @ratishp22]

@PyTorch kernel count is not fusion quality. inductor keeps two reduction stages here but removes the full-size intermediate. for serving, what matters is whether that memory-traffic win survives variable batch/sequence shapes without recompilation or a silent fusion change.

[Ratish P @ratishp22]

@cursor_ai moe routing makes communication direction a scheduling decision. pull moves more bytes yet wins under expert skew by using both nvlink directions and removing cross-rank signals. curious if minibatch size and the comm-sm split can be tuned online as router balance drifts.

[Ratish P @ratishp22]

@OpenAI what is the cutover boundary? with both instances running, do they switch on an audio-frame boundary and suppress one output, or synchronize before audio rendering? that detail decides whether the handoff is truly inaudible.

[Ratish P @ratishp22]

@radixark @wafer_ai 📈📈📈

[Ratish P @ratishp22]

1/3 api pricing is already huge. the real unlock: H3 ranks above Seedance 2.0 in text-to-video with audio, and @sgl_project runs its open 33B base model on 2×32GB 5090s with lossless layerwise offload. now the serving is yours to optimize and deploy. @MiniMax_AI

[LMSYS Org @lmsysorg]

@MiniMax_AI H3 is live in SGLang Diffusion, with day-0 serving support 🎬

This open model matches Seedance 2.0 at 1/3 the cost, or $0 if you run it locally on 2x 5090 or 1 RTX 6000.

With SGLang Diffusion, you can build visual concepts, motion design, e-commerce creatives, precise video edits, animation and stylized visuals, all locally on your own machine. No API bill, no waitlist.

H3 takes text, image, video and audio in a single context, and generates 5-15s clips at native 2K, 24fps with native stereo audio.

SGLang Diffusion also runs on @NVIDIAAI Blackwell and Hopper, and @AMD MI355X / MI300X.

Let's create something with MiniMax H3 👇

[Ratish P @ratishp22]

the qwen3.8 result that got me was: 500+ turns through iverilog, yosys and openroad, taking a crypto accelerator from 8,298 gates to 678 while closing timing. this is what long-horizon agents should look like: optimizing against real tool feedback, not just vibes. @Alibaba_Qwen

[Qwen @Alibaba_Qwen]

📢Meet Qwen3.8-Max — our most capable model to date.

Next week, the open weights of Qwen3.8-Max will be released, and Qwen3.8-27B is also going open-weights to meet you all!🎉

Qwen3.8-Max, a new bar for coding and cowork at 2.4T parameters:

- Autonomous coding: 10+ days of self-evolving development, from empty folder to production without hand-holding, complete project trace in the GitHub：https://t.co/iVHZWQoeSo
- Real work, real results: Production-quality deliverables across hundreds of professions.
- Long-horizon mastery: System-level autonomous planning with closed-loop adaptive learning, driving 500+ turns of chip design optimization and 365 days of e-commerce strategy.
- Native multimodal intelligence: Vision isn't just input — it's a continuous feedback loop for planning, execution, and self-correction.

💰Pricing:
Input: $2.0 / M tokens
Output: $6.0 / M tokens
Implicit Caching: $0.25 / M tokens

Start building with Qwen3.8-Max! 🚀

📖 Blog: https://t.co/iwjmQxLBof
✅ Qwen Studio: https://t.co/4V2pFvDovG
⚡ API: https://t.co/gAGqaLQGbN

[Ratish P @ratishp22]

one of the least glamorous and most important changes we made this cycle. 2,840 lines gone, bit-for-bit identical outputs, flat performance, and 3 models validated. no model-name branches, no premature abstractions, just clear lifecycle ownership. proud of this team. @YichiZ03

[Yichi Zhang @YichiZ03]

The hardest part of maintaining an open-source project is serving them at scale while keep high code quality.

In SGLang-Omni, we spent a month refactoring, removing 2000+ non-test duplication. Here is what we changed, and what we learned. https://t.co/kaxYsYokmo

[Ratish P @ratishp22]

the key safety eval for open weights is not refusal rate. strip safeguards through adversarial fine-tuning, then test whether the underlying model adds dangerous capability beyond what is already downloadable. pair that with staged access so defenders can catch up @thinkymachines

[Thinking Machines @thinkymachines]

Releasing weights indiscriminately isn't safe. Neither is keeping capable models inside a few labs.

We think there's a path between them. We haven't mapped all of it. Our new post covers the part we can see: how we assessed Inkling, and why access should widen in stages.

https://t.co/zGFbuXNr0Y

[Ratish P @ratishp22]

quantization gets messy when each format owns the path from parsing to kernel launch. config → method → scheme → kernel is cleaner: checkpoint semantics stay separate from backend execution, and formats can reuse kernels. great work from the @Huawei team and @sgl_project

[LMSYS Org @lmsysorg]

🚀 New blog: Toward a Cleaner Quantization Stack in SGLang

We're dividing the quantization path into four focused layers: Config, Method, Scheme, and Kernel. The cleaner stack already delivers 2.83X smaller weights on Qwen3-30B-A3B with almost no accuracy loss:
☑️ 61.08GB → 21.59GB with W4A4_W8A8, GSM8K 91.1% → 89.4%
☑️ 26.23ms → 23.84ms ITL
☑️ One kernel, many formats: AWQ, GPTQ, Compressed-Tensors, ModelSlim & Quark share backend kernels

Next up in the roadmap we will expand NVFP4, MXFP4, and MXFP8 support across CUDA, ROCm, Ascend NPU, and CPU.

[Ratish P @ratishp22]

big models get headlines; trainable models move ecosystems. Inkling-Small matches Inkling at 276B/12B active, making full-parameter RL reachable to more teams. Miles trains it; SGLang serves it at 648 tok/s with DSpark at simulated acceptance 4. @thinkymachines @lmsysorg

[LMSYS Org @lmsysorg]

Inkling-small is out today! With SGLang, you can get 648 tok/s decode with DSpark (simulated acc len=4) and 288 tok/s w/o DSpark, under the same setup (8x @NVIDIAAI B200, TP 8, NVFP4, bs=1).

What makes this model different is the size. 276B total with 12B active is a sweet spot for RL, and both LoRA and full-parameter training become well within reach. Miles is ready and verified for multimodal RL on Inkling-small, so you can turn your multimodal data into real capability gains.

At ~1/4 the size, Inkling-small matches the bigger version in capability and even wins on some benchmarks.

Run Inkling-small with SGLang, and customize it with Miles.

[Ratish P @ratishp22]

the interesting bit in gpt-5.6 isn’t only about their model getting better. their serving stack is getting sharper too: better routing, kv-cache tuning, kernels, and speculative decoding adding up to 20% lower serving cost and 15%+ better token-generation efficiency. @OpenAI

[OpenAI @OpenAI]

After deployment, we applied GPT-5.6 Sol to advance the frontier of efficiency by making itself more efficient to run.

The results:
- 20% lower serving costs from production GPU kernel improvements.
- 15%+ better token-generation efficiency from improved speculative decoding.