Excited to share that I will be joining UCSD CSE as an assistant professor in January 2026!
I'll be recruiting PhD students from the 2024 application pool - if you're interested in anything ML Sys/efficiency/etc please reach out & put my name on your application!
Until then I'll be finishing up some requirements at Stanford (long story...) and hanging out at @togethercompute. Stay tuned for more!
Today we're launching a preprint server built specifically for AI agents
For three months, nearly 1,000 agents from 35 labs (across Stanford, MIT, ETH Zurich, and more) conducted research autonomously on this server, producing 936 papers alongside their code, data, and model weights.
Agents found significant errors in published human research and even created their own journal to review each other’s work.
Some agents independently produced results similar to those in human papers that later appeared on arXiv, including findings on failure modes in self-distillation and feature reproducibility in mechanistic interpretability.
As a part of this study, we open-source the full twelve-week archive of manuscripts, code, data, peer reviews and agent chat transcripts. Read the study: https://t.co/oGD86l9crx
Last Year We Built Megakernels. This Year We Deleted Them.
One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target.
Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
tl;dr of what Together Compiler does, from @realDanFu on @thecompute100:
1/ run the model end to end
2/ find the bottlenecks
3/ let the model fix them
Links below 👇
Fun podcast on kernels, origins of @togethercompute, and how we’re using AI to build inference in an RSI inference loop with Together Compiler.
Thanks so much for having me @gabyllorenzi and @BSchech!
Today’s guest is @realDanFu, SVP of Research at @togethercompute.
A co-creator of FlashAttention, Dan now leads research at a company that provides the compute and software to train and serve AI models.
From GPU instructions to the economics of open models, we discuss:
- What CUDA, kernels and compilers each do
- Why FlashAttention rethought data movement
- How AI is changing kernel engineering
- How new open models get to market
- How inference revenue funds open models
- Owning versus leasing GPUs
Timestamps:
0:00 Introduction
1:04 From AI models to CUDA kernels
11:38 Launching new models at Together
22:36 How model architectures are changing
27:42 Comparing open and closed models
31:57 How AI is changing kernel engineering
38:01 From systems research to FlashAttention
47:11 Using AI to improve inference
48:16 Together’s origins and open models
51:17 Owning versus leasing compute
52:55 Building the research team and new chips
54:56 Frontier labs, open models and partnerships
57:16 Inference profits and funding the next model
1:00:13 Academic research and new talent
1:02:47 Robotics and AI in the physical world
Listen to the show, and follow along for new episodes every Tuesday.
Today’s guest is @realDanFu, SVP of Research at @togethercompute.
A co-creator of FlashAttention, Dan now leads research at a company that provides the compute and software to train and serve AI models.
From GPU instructions to the economics of open models, we discuss:
- What CUDA, kernels and compilers each do
- Why FlashAttention rethought data movement
- How AI is changing kernel engineering
- How new open models get to market
- How inference revenue funds open models
- Owning versus leasing GPUs
Timestamps:
0:00 Introduction
1:04 From AI models to CUDA kernels
11:38 Launching new models at Together
22:36 How model architectures are changing
27:42 Comparing open and closed models
31:57 How AI is changing kernel engineering
38:01 From systems research to FlashAttention
47:11 Using AI to improve inference
48:16 Together’s origins and open models
51:17 Owning versus leasing compute
52:55 Building the research team and new chips
54:56 Frontier labs, open models and partnerships
57:16 Inference profits and funding the next model
1:00:13 Academic research and new talent
1:02:47 Robotics and AI in the physical world
Listen to the show, and follow along for new episodes every Tuesday.
Distilling most of our expertise in kernel and inference engine optimization into an agent loop that powers our production. A 2-year project that we're finally able to share
Inference is entering a new era. Models ship weekly, traffic shapes shift daily, and hardware turns over every year. The era of Static Inference, tuned once for one model with one traffic shape on one chip, is ending.
Welcome to the era of Polymorphic Inference: inference that adapts to new models, workloads, and hardware as they arrive.
Today we’re announcing Together Compiler, our first step into that era. It enables continual optimization, so a single overnight run can match or beat the manual expert tuning that used to take weeks.
Read the full vision from our SVPs of Research, @simran_s_arora and @realDanFu, on Polymorphic Inference and our research roadmap 👇
https://t.co/kqBbu0nDXZ
A little over a year ago, I started my journey into kernels and inference optimization. Since then, the field has rapidly evolved, and many tasks like kernel writing have become increasingly obsolete. I’ve spent a lot of time reflecting on ML Systems in the agentic era, and how such systems must be designed to efficiently and thoroughly explore the optimization space. I’m proud to announce Together Compiler, a continuous optimization system for the age of polymorphic inference!
I left my PhD 16 months ago to figure out how to scale AI inference optimization. Today, I’m proud to announce the release of Together Compiler! It's the first continual inference optimization system that adapts to new-age architectures and diverse user behavior. The first system designed for the age of “polymorphic” inference.
Together Compiler generates optimizations specialized to a workload and posts best-in-class TPS numbers on highly-optimized open source benchmarks: 1100+ TPS on MiniMax M3.1 Flash and 1200+ TPS for DeepSeek v4.1 Flash off InferenceX. 24 hours and just 2 nodes per model. That’s it!
Paper: https://t.co/RSqXvVXNE8
It was super exciting to build this from scratch :) more thoughts in thread below!! 1/
This was super exciting, and there’s a lot more coming! Really awesome to watch @reyna_abhyankar@dylan__lim@simran_s_arora lead this alongside the rest of the research team @togethercompute.
Check out the paper for results, details, and a lot more!
https://t.co/HVT2mshdTH
Super excited to (officially) share Together Compiler with the world! This is our opinionated take at @togethercompute about how to scale inference in an increasingly polymorphic world (many models, archs, traffic, hardware).
Deployed across our inference fleet today 🚀 1/
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar@dylan__lim@realDanFu@togethercompute!!
We view this as the first step towards building for an increasingly polymorphic future. @simran_s_arora and I wrote a little blog on our view of polymorphic inference and the implications for our research agenda.
Check it out! https://t.co/o1xw71w0Th
5/
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar@dylan__lim@realDanFu@togethercompute!!
PKB accepted to #neurips2026, go chat with @opengroundsFX and @simonguozirui in Sydney if you’re interested!
Fun fact - arxiv actually rejected this paper, so my accept rate on arxiv this cycle is worse than NeurIPS :)
Grateful for @askalphaxiv for an alternative!
Proud to announce that ParallelKernelBench (https://t.co/7MrQsu4xwd) has been accepted to NeurIPS 2026! 🎉
If you are interested in the future of multi-GPU kernels, come say hi!
Huge thanks to my amazing collaborators @asplencmnt@simonguozirui@simran_s_arora@realDanFu
Proud to announce that ParallelKernelBench (https://t.co/7MrQsu4xwd) has been accepted to NeurIPS 2026! 🎉
If you are interested in the future of multi-GPU kernels, come say hi!
Huge thanks to my amazing collaborators @asplencmnt@simonguozirui@simran_s_arora@realDanFu
Introducing NyoomFloat12, a hardware-friendly lossless 12-bit quant from BF16 (accepted to #neurips2026)!
In BF16 LLMs four of the bits that are the same for 99.7% of the weights. So you can compress BF16 -> 12-bit with minimal overhead.
Check out @__sylvie_l__'s thread! 1/
Lossless weight compression has saved LLMs GPU memory, but not time. Decoding costs more than it saves.
NyoomFloat12 (NF12) is a lossless 12-bit BF16 format with decode cheap enough to speed up inference. Qwen3-32B: 1.23× faster token generation, up to 2.54× rollout throughput🧵
NyoomFloat12 was accepted to NeurIPS, and @__sylvie_l__ will be presenting it in Sydney! Go chat with her, you'll learn a lot!
And check out the paper on @askalphaxiv: https://t.co/WASwU7LjOt
4/