Sheng Zha

λ☕: (λ 🤖: ⚙️)

prev: DL Frameworks / LLM pretraining / algo-sys co-design.

[Sheng Zha @szha_ai]

@Jianlin_S This has indeed been our empirical finding for some time now. Awesome to see the analysis.

[Sheng Zha @szha_ai]

@haozhangml Kids these days

[Yuxin Tang 🦉 @yuxin_tang_]

We release our tool to help with parallelism layout search to obtain the optimal MFU for your LLM/MoE pre-training/post-training/inferece on Blackwell chips: https://t.co/hP57GAmZQI Feel free to have a try and give us any comment!
@soum_sarkar
@szha_ai
#amazon #moe #blackwell #mfu #pretraining #posttraining

[Sheng Zha @szha_ai]

@VictoriaLinML Congrats, the multimodal capabilities look great!

[Bing Xu @bingxu_]

I started INT21 two months ago, and I’m proud to announce that we’re coming out of stealth today with our first product: PTX Kernel Factory. We’ve created the first self-improving agent swarm for producing expert-level PTX kernels autonomously and at massive scale. https://t.co/NgoX0MHsIx

[Beidi Chen @BeidiChen]

Rollout generation is becoming a central bottleneck in long-context RL says everyone — and recent efforts from Cursor @cursor_ai and Fireworks @FireworksAI_HQ
show how just important disaggregating rollout and training is today.

Excited to share Sparrow🐤, which makes sparse rollout practical and accelerate stable long-context RL.

Check it out!

[Sheng Zha @szha_ai]

Neoclouds and neolabs are both capex-heavy but have very different return profiles.

Neoclouds are priceable from day one: good short-term business while token demand outpaces supply. But unless they own the adjacent layer (accelerator chips, orchestration), they eventually commoditize when the market equalizes.

Neolabs are asking the market to price breakthroughs that may never commercialize. High-variance upside, but obviously not all bets stick.

xAI pivoting to sell Colossus capacity is a case study in derisking when neolab bets don't pan out. This could become a pattern: neolabs exiting into neocloud mode as investors demand returns.

[Beidi Chen @BeidiChen]

📢Vortex feels like an early glimpse of recursive self-improvement for ML systems: agents discovering better architectures, compiling them into real serving stacks, and shifting the bottleneck from "can it code?" to "can we still steer it?"

AI building AI just went from theory to benchmark chart.

[Sheng Zha @szha_ai]

@PandaAshwinee @itsandrewgao I'm not as active on x so definitely could have missed it. If you have a reference to share, it would definitely help balance my perspectives.

[Sheng Zha @szha_ai]

@PandaAshwinee @itsandrewgao A couple of works mentioned this, https://t.co/6wtUa7OvGp, https://t.co/2VXI4ax6OO, https://t.co/fvmjrdkmvX. Labs probably all have their internal versions now.

[DeepSeek @deepseek_ai]

🚀 DeepSeek-V4 Preview is officially live & open-sourced! Welcome to the era of cost-effective 1M context length.

🔹 DeepSeek-V4-Pro: 1.6T total / 49B active params. Performance rivaling the world's top closed-source models.
🔹 DeepSeek-V4-Flash: 284B total / 13B active params. Your fast, efficient, and economical choice.

Try it now at https://t.co/GCdiMzk1Dl via Expert Mode / Instant Mode. API is updated & available today!

📄 Tech Report: https://t.co/drlDrxkYtp
🤗 Open Weights: https://t.co/T13Y8i7SDM

1/n

[Sheng Zha @szha_ai]

@DZhang50 Barely enough to keep things warm on cold days.

But to be fair, the more concerning part is probably their president discussing about MFU without discerning compute platform or workload.

[Sheng Zha @szha_ai]

@GaryMarcus Unequivocal evidence that tech bros are indeed summoning the god.

[Epoch AI @EpochAIResearch]

AI has solved one of the problems in FrontierMath: Open Problems, our benchmark of real research problems that mathematicians have tried and failed to solve.

See thread for more.

[Bing Xu @bingxu_]

Just open-sourced VibeTensor — the first deep learning system fully generated by an AI agent, with 0 lines of human-written code: https://t.co/8wwe1LB7xK

It’s a working DL system with RCU style dispatcher, a cache allocator and reverse-mode autograd. The agent also invented a Fabric Tensor system — something that doesn’t exist in any current framework.

The Vibe Kernel includes 13 kinds and 47k LOC of generated Triton and CuteDSL kernels with strong performance.

VibeTensor was generated by our 4th-generation agent. It shows a “Frankenstein Effect”: the system is correct, but some critical paths are designed in inefficient ways. As a result, performance isn’t comparable to PyTorch.

I haven’t written a single line of code since summer 2025. I started this effort after @karpathy 's podcast — I didn’t agree with his arguments, so Terry Chen and I began using it as a stress test for our agents. The “Frankenstein Effect” ended up exposing some of our agent’s limitations — but the direction is clear.

[Sheng Zha @szha_ai]

@cHHillee @alhyunsoo @jacobmenick Nice blog. How much would reduction order still be an issue if we use stable sum where it matters?

[Yulin Chen @YulinChen99]

Reasoning models overthink, generating multiple answers during reasoning. Is it because they can’t tell which ones are right?

No! We find while reasoning models encode strong correctness signals during chain-of-thought, they may not use them optimally.

🧵 below

[(((ل()(ل() 'yoav))))👾 @yoavgo]

NOTE: Most MIT Profs who I know are honest and not racist.

(well thats probably not even true, but they surely are smarter about not letting their dishonesty or racism slip in their public talks)

[He He @hhexiy]

Unbelievable. This quote is blatantly false and unnecessary for the argument. And she surely had expected the backlash with the patronizing NOTE. This is racism, not "cultural generalization". @NeurIPSConf