Hanshi Sun

Research Scientist in ByteDance Seed | MLSys | Grad @CMU_ECE

[Hanshi Sun @preminstrel]

🎉 Thrilled to announce our ShadowKV has been accepted to #ICML2025 as a ✨Spotlight Presentation❗️

❓Facing challenges with high-throughput long-context LLM serving? ShadowKV is here to help!

🚀 Achieves memory-efficient & high-throughput inference via sparse attention.
🌟 Minimizes memory footprint by storing low-rank keys on GPU & offloading values.
⚡ Boosts throughput up to 3.04x on an A100 with 6x batch size or 6x longer sequences without accuracy loss.

Discover how ShadowKV balances speed, memory, and precision for ultra-long contexts! 👇

📜 Paper: https://t.co/ZT7Xkw4BjL
🔗Blog: https://t.co/uOHLhHpuXc
💻 Code: https://t.co/sX2RzPWTMS

[Beidi Chen @BeidiChen]

⏰New work on scaling embedding modules as memory (timely huh 🐳😁?) — the demo is wild: live knowledge edits via embedding swaps, no raw text involved.
Feels like the start of a long arc: continuously evolving memory systems. Lookup is the new muscle memory 🧠

[Jian Chen @jianchen1799]

🚀Thrilled to share my first work as a PhD student!

We propose a new home for Diffusion LLMs: not as competitors to AR models, but as ultra-fast drafters. DFlash is lightweight, cheap to run, and very effective (up to 6x speedup).

It’s super easy to set up—give it a try!

[Infini-AI-Lab @InfiniAILab]

🤔Can we train RL on LLMs with extremely stale data?

🚀Our latest study says YES!

Stale data can be as informative as on-policy data, unlocking more scalable, efficient asynchronous RL for LLMs.

We introduce M2PO, an off-policy RL algorithm that keeps training stable and performant even when using data stale by 256 model updates.

🔗 Notion Blog: https://t.co/vVUUuTiOL7
📄 Paper: https://t.co/Rmzd9rJHAv
💻 GitHub: https://t.co/xD6i6fRKwl

🧵 1/4

[Hanshi Sun @preminstrel]

The deadline (Sept 1) for our 1st Efficient Reasoning workshop @ NeurIPS is just around the corner! Don't miss out! 😇

[Cheng Luo @ChengLuo_lc]

🌟 Reminder: Submission Deadline Approaching! 🌟

The 1st Workshop on Efficient Reasoning (ER) @ NeurIPS 2025 — happening Dec 6 or 7 in San Diego — is fast approaching, and we’d love to see your work there!

📌 Submission Deadline: September 1, 2025 (AoE)
🔗 Submit here: https://t.co/3lJOklLYkK
🌍 Website: https://t.co/qepTfHNMA4
We welcome submissions on topics such as:
🔹 Resource-aware reasoning datasets
🔹 Efficient training & RL fine-tuning
🔹 Pruning, compression & KV-cache tricks for faster inference
🔹 Benchmarks & theory on time/space complexity
🔹 Systems for long-CoT & on-device reasoning
🔹 Real-time deployments in healthcare, robotics, autonomy, and more

🤝 Perspectives from ML, systems, natural & social sciences, and industry are all encouraged!

📝 Interested in reviewing? Nominate yourself . https://t.co/sqauPewIhL
🚀 Don’t miss the chance — deadline is just around the corner. We look forward to your submissions and to seeing you in San Diego (or virtually)!

[Yong Shan @yshan783399]

We are thrilled to introduce the Seed-OSS family of open-source LLMs, developed by ByteDance's Seed Team.

GitHub: https://t.co/lUNRuigqMA
HuggingFace: https://t.co/1WuQHpGcIo

Feel free to try it out and share your feedback!

[Si-ze Zheng @DeeplyIgnorant]

🎉 Excited to share: We’ve open-sourced Triton-distributed MegaKernel! A fresh, powerful take on MegaKernel for LLMs—built entirely on our Triton-distributed framework.
https://t.co/lMbbYyd1uF

Why it’s awesome?
🧩 Super programmable
⚡ Blazing performance
📊 Rock-solid precision

[Infini-AI-Lab @InfiniAILab]

🔥 We introduce Multiverse, a new generative modeling framework for adaptive and lossless parallel generation.

🚀 Multiverse is the first open-source non-AR model to achieve AIME24 and AIME25 scores of 54% and 46%

🌐 Website: https://t.co/J9osByhWUf

🧵 1/n

[Infini-AI-Lab @InfiniAILab]

🥳 Happy to share our new work –  Kinetics: Rethinking Test-Time Scaling Laws

🤔How to effectively build a powerful reasoning agent?

Existing compute-optimal scaling laws suggest 64K thinking tokens + 1.7B model > 32B model.
But, It only shows half of the picture!

🚨 The O(N²) KV memory access in self-attention dominates the cost of test-time scaling (TTS).
MoEs even worsen memory bottleneck by cutting compute.

Our new scaling law Kinetics suggests investing in model size first before spending more in test-time compute.

This insight leads to our next key finding
✨ Sparse Attention = Scalable TTS

Our Kinetics sparse scaling law says that when doubling the resources, we should prioritize increasing test time tokens over attention density.

✅ 60+ pts improvement under the same compute budget
✅ 10× lower resource usage for equivalent performance
✅ Sparse attention becomes increasingly valuable in high-cost scenarios

💡Sparsity is key to unlocking full potential of TTS, because unlike pretraining, where scaling shows diminishing returns, TTS continues to benefit from increased token generation and more optimized inference paths.

📄 Paper: https://t.co/hvfBpuTjwJ
🌐 Website: https://t.co/7HhzwASBI7
🔗 GitHub: https://t.co/YXzhsJ4RQ2
#ML #AI #LLM #Transformers #ScalingLaws #SparseAttention #Inference

[chen zhuoming @chenzhuoming911]

The most INTERESTING paper I have ever worked on. I worked on Sparse Attention for about 1.5 years to improve LLM efficiency.

But test-time scaling makes this a different story. We can generate more tokens or conduct deeper and deeper thinking with sparsity within any fixed cost.

This is what we believe is scalability. AMAZINGLY, even a straightforward block topk attention can work very well, implying that this scaling is achievable, with current hardware, systems, and kernels. We have seen many more elegant, accurate, and efficient ways of implementing sparse attention. They will DEFINITELY lead to an even better scaling!

[Zihao Ye @ye_combinator]

We’re thrilled that FlashInfer won a Best Paper Award at MLSys 2025! 🎉
This wouldn’t have been possible without the community — huge thanks to @lmsysorg’s sglang for deep co-design (which is crtical for inference kernel evolution) and stress-testing over the years, and to @vllm_project for integration support.
With continued help from @NVIDIAAIDev , FlashInfer is becoming more stable and faster. Let’s keep building together!

[Ziheng Jiang @ZihengJiang]

Excited to spend the this week in Santa Clara for #MLSys ! The ByteDance Seed Infra team is presenting two papers:

- "Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts", Shulai Zhang et al. https://t.co/395uw4iFaH
- "TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives", Size Zheng et al. https://t.co/0yRDK4kild

Our group tackles challenges across training, inference, and compiler stacks. If you’re at MLSys—or just passionate about high-performance ML systems—let’s connect!

#MLSys #LLM #DeepLearning #ByteDance

[Huiqiang Jiang @iofu728]

⛰We just release RetroInfer: a new system that rethinks the KV cache as vector storage in a GPU–CPU co-execution to accelerate long-context LLM inference. Powered by wave index and wave buffer, it get 4.5×–10.5× speedups over FlashAttention w/ same acc.

https://t.co/39tvjnxL1A

[Si-ze Zheng @DeeplyIgnorant]

🚀 We released Triton-distributed! 🌟
Build compute-comm. overlapping kernels for GPUs—performance rivals optimized libraries
🔗 https://t.co/lMbbYyd1uF
👏 Shoutout to AMD for testing our work! Check their blog:
🔗 https://t.co/xmZOOwwYG8

[Si-ze Zheng @DeeplyIgnorant]

Our paper "TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives" is accepted to the #MLSys2025 Conference held from May 12-15th in Santa Clara, California! Check out the full list of accepted papers here: https://t.co/jELiwlrNRh

[Hanshi Sun @preminstrel]

🎉 Thrilled to announce our ShadowKV has been accepted to #ICML2025 as a ✨Spotlight Presentation❗️

❓Facing challenges with high-throughput long-context LLM serving? ShadowKV is here to help!

🚀 Achieves memory-efficient & high-throughput inference via sparse attention.
🌟 Minimizes memory footprint by storing low-rank keys on GPU & offloading values.
⚡ Boosts throughput up to 3.04x on an A100 with 6x batch size or 6x longer sequences without accuracy loss.

Discover how ShadowKV balances speed, memory, and precision for ultra-long contexts! 👇

📜 Paper: https://t.co/ZT7Xkw4BjL
🔗Blog: https://t.co/uOHLhHpuXc
💻 Code: https://t.co/sX2RzPWTMS

[Huiqiang Jiang @iofu728]

MMInference is accepted by #ICML2025!
It use permutation to solve inductive bias and modality boundary issues in multi-modality. And also unify dynamic sparse attention in sparse load + dense tensor core pipeline.
Congratulations to @liyucheng_2! Find more https://t.co/dUWSges50h

[Haibin @eric_haibin_lin]

We are open sourcing bytecheckpoint and veomni!
bytecheckpoint is the Bytedance's production checkpointing system for foundation model training, battle-tested with jobs with 10k+ GPUs. Blazing fast save/load, load-time checkpoint auto-resharding for different parallelism across training stages (pretrain/SFT/RL).
veomni is a open source model training framework for llm and multi-modal training. UI-TARS (the SOTA GUI Agent model prior to OpenAI operator's release) is trained with veomni. Developed with modular design, integrated with sequence/expert/zero-optimizer parallelism, offloading optimizations, @liger_kernel. Trainer-free (let user control the training loop) and easy for researcher to hack! The go-to framework for text/multimodal llm pre-training and post-training, from research to production.

Try them today, your feedback is welcome!

Code:
- https://t.co/IKM7ceK7GN
- https://t.co/HAzL6Omc6j
Paper:
- NSDI paper: https://t.co/HjHgbjv9r0

[Hanshi Sun @preminstrel]

@jxyintheflesh 唉，上班

[Haibin @eric_haibin_lin]

❗️Open source MOE kernels alert❗️
Introducing COMET, a computation/communication library for MoE models from Bytedance. Battle-tested in our 10k+ GPU clusters, COMET shows promising efficiency gains and significant GPU-hour savings (millions 💰💰💰).

Integration of DualPipe & DeepEP requires too much effort? Try COMET, a drop in replacement for your MOE block!

Key Points:
✅ Deployed on 10K+ GPU cluster, saved MILLIONS of GPU hours
✅ 1.96x layer-wise speedup, 1.71x end-to-end boost for MoE models
✅ Fine-grained Computation-communication Overlapping for MoE

Why devs care:
📌 Plug-and-play with existing frameworks (just a few lines of code change)
📌 Supports ALL MoE parallel modes: TP/EP/EP+TP
📌 MLSys'25 top scores (5/5/5/4) - battle-tested at scale

📄 Paper: https://t.co/T8lL1V1g9t
📦 Code: https://t.co/cPBN3prBQA

Great work done by Shulai, @NingxinZheng_ and team

#OpenSource #LLM #MOE #MLSys2025 #CUDA