Dylan Lim

research @togethercompute | prev. @stanford @jumptrading @hazyresearch

[Dylan Lim @dylan__lim]

A little over a year ago, I started my journey into kernels and inference optimization. Since then, the field has rapidly evolved, and many tasks like kernel writing have become increasingly obsolete. I’ve spent a lot of time reflecting on ML Systems in the agentic era, and how such systems must be designed to efficiently and thoroughly explore the optimization space. I’m proud to announce Together Compiler, a continuous optimization system for the age of polymorphic inference!

[Simran Arora @simran_s_arora]

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨

Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.

Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.

From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!

[alphaXiv @askalphaxiv]

"Together Compiler: Continual Compilation for AI Inference"

AI agents can optimize LLM inference, but testing every change requires restarting the entire inference engine, making experimentation extremely slow.

This paper by Together AI introduces Together Compiler, which lets coding agents continuously modify and test inference code without restarting the engine, using real workloads to measure performance.

This enables 252x more agent experiments and achieves up to 2.6x higher inference throughput than top open-source engines across frontier models.

https://t.co/vIydtRphEX

[Aleksa Gordić (水平问题) @gordic_aleksa]

@simran_s_arora amazing work!

[Dylan Lim @dylan__lim]

@ShubhamMal72313 @tri_dao agents can directly write nki!

[Hasith Vattikuti @hasith_v]

Congratulations to @reyna_abhyankar and @dylan__lim for their >1yr effort on Together Compiler!

Their work was so incredible that I found it generalized well beyond LLMs to SciML models 🔬.

Here it is outperforming Claude on nearly every model I tried

1/n

[Tri Dao @tri_dao]

Distilling most of our expertise in kernel and inference engine optimization into an agent loop that powers our production. A 2-year project that we're finally able to share

[Dylan Lim @dylan__lim]

@MLaurusevicius @simran_s_arora The compiler can run for as long or as little as you wish! We’ve found that a single night is enough to observe significant speed ups, ready to be deployed in production the following morning.

[Together AI @togethercompute]

“AI is now improving the systems that run AI.” Thanks to @Forbes for covering Together Compiler and the shift to Polymorphic Inference.

New open models can now get fully optimized for production performance in hours, not weeks.

https://t.co/HTGqSwiLdr

[Dylan Patel @dylan522p]

Great work that pushes our understanding of how agents and experts can collaborate on optimizing system performance
Beyond the strong results, the team also found some surprising cases where our human inductive biases end up limiting what agents can achieve

[Vipul Ved Prakash @vipulved]

Together Compiler discovers the most efficient inference runtimes with continuous compilation for any combination of workload, model and silicon.

Excited to share the details about the research and engineering that went into it. Together Compiler (or TKCC) is a production service that is deployed and operational across portions of our inference fleet, and optimizing our inference stack continuously.

We think of this as a shift in inference technology from Static Inference (inference engines, hand-tuned configs, megakernels etc), to Polymorphic Inference, which are inference systems that continuously and autonomously adapt to workload, hardware and models, and are essential for achieving vast scale alongside the best possible utilization and economics.

A thread from @simran_s_arora, who along with the research team at
@togethercompute spearheaded the work on Together Compiler.

[Dylan Lim @dylan__lim]

@ethanboneh Congrats Ethan!

[Dylan Lim @dylan__lim]

the big win of our "tiered" system is that we are able to efficiently validate increasingly rigorous correctness checks! from individual layers to whole model logprobs to full GPQA / GS8MK / InferenceX eval quality checks, we're able to ensure that the optimized forward is accurate!

[Dan Fu @realDanFu]

Super excited to (officially) share Together Compiler with the world! This is our opinionated take at @togethercompute about how to scale inference in an increasingly polymorphic world (many models, archs, traffic, hardware).

Deployed across our inference fleet today 🚀 1/

[Kesavan Ramakrishnan @keramakr]

Had a great time working on this! The future of systems is in agents that can continuously optimize and target the workloads that they serve.

[Dylan Lim @dylan__lim]

A little over a year ago, I started my journey into kernels and inference optimization. Since then, the field has rapidly evolved, and many tasks like kernel writing have become increasingly obsolete. I’ve spent a lot of time reflecting on ML Systems in the agentic era, and how such systems must be designed to efficiently and thoroughly explore the optimization space. I’m proud to announce Together Compiler, a continuous optimization system for the age of polymorphic inference!

[SemiAnalysis @SemiAnalysis_]

The Together Compiler (TKCC) from @simran_s_arora and her team is honestly one of the coolest things ever, and we have validated one of the InferenceX/AgentX results! The ability to continuously improve performance automatically while ensuring accuracy is great enough to quickly deploy to production!

Simran and her team are using TKCC and working on applying it to even more use cases. They will share more soon!
https://t.co/DHDsC3R3yZ

[Together AI @togethercompute]

Introducing Together Compiler: agents that continually optimize inference for production traffic.

Serving trillions of tokens/day. Up to 2.6x faster than leading InferenceX configs, up to 50% faster on customer workloads.

Congrats @reyna_abhyankar @dylan__lim @simran_s_arora and the team!

[Reyna Abhyankar @reyna_abhyankar]

I left my PhD 16 months ago to figure out how to scale AI inference optimization. Today, I’m proud to announce the release of Together Compiler! It's the first continual inference optimization system that adapts to new-age architectures and diverse user behavior. The first system designed for the age of “polymorphic” inference.

Together Compiler generates optimizations specialized to a workload and posts best-in-class TPS numbers on highly-optimized open source benchmarks: 1100+ TPS on MiniMax M3.1 Flash and 1200+ TPS for DeepSeek v4.1 Flash off InferenceX. 24 hours and just 2 nodes per model. That’s it!

Paper: https://t.co/RSqXvVXNE8

It was super exciting to build this from scratch :) more thoughts in thread below!! 1/

[Dylan Lim @dylan__lim]

Together Compiler is still early and will continue to evolve. From formal verification to open model optimizers, I’m excited about many things! This work would not have been possible without my collaborators @reyna_abhyankar @simran_s_arora and the rest of the amazing @togethercompute team. The future of optimization is continual.

Paper: https://t.co/NNDeL43qNR

[Dylan Lim @dylan__lim]

A little over a year ago, I started my journey into kernels and inference optimization. Since then, the field has rapidly evolved, and many tasks like kernel writing have become increasingly obsolete. I’ve spent a lot of time reflecting on ML Systems in the agentic era, and how such systems must be designed to efficiently and thoroughly explore the optimization space. I’m proud to announce Together Compiler, a continuous optimization system for the age of polymorphic inference!

[Simran Arora @simran_s_arora]

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨

Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.

Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.

From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!

[Dylan Lim @dylan__lim]

We believe to have identified fundamental “inference primitives of the agentic era”, enabling state of the art inference optimization in under 24 hours, on no more than two nodes. From existing Megakernel baselines to open source models and arbitrary models, we’ve found our system holds up.

[Simran Arora @simran_s_arora]

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨

Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.

Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.

From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!