Very insightful post from @simran_s_arora@dylan__lim@reyna_abhyankar at @togethercompute that investigated the tradeoffs between Torch Compile and Megakernels which leads to the magical Together Compile! Always investigate in both sides!
Checkout the video in @dylan__lim's post to see the hidden bubbles: Look Ma, Some Bubbles
Building on a long and wonderful tradition in Hazy lab of always investigating both sides (tb to S4 no attention vs FlashAttention)
Last Year We Built Megakernels. This Year We Deleted Them.
One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target.
Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
We breathed life into LLM system.
Introducing Evolving Infra⚙️: an inference framework that adapts diverse and dynamic agent workload purely online and retires fixed PD disaggregation.
vs. NVIDIA Dynamo on up to 256 GB300 GPUs:
➡️ Up to 4.1× lower latency
➡️ Up to 14× lower P95 TTFT
➡️Up to 2.36× more revenue
on real agent traces(Computer Use, Coding and math agents) and opensource models.
🌐 Website: https://t.co/7I3iERjE6b
📄 Tech report: https://t.co/qDGcwl0svx
💻 Code: https://t.co/0VOBCptKcb
Which agent(e.g. Pi/Openhands) do you want us to accelerate next? Tell us below 👇
We breathed life into LLM system.
Introducing Evolving Infra⚙️: an inference framework that adapts diverse and dynamic agent workload purely online and retires fixed PD disaggregation.
vs. NVIDIA Dynamo on up to 256 GB300 GPUs:
➡️ Up to 4.1× lower latency
➡️ Up to 14× lower P95 TTFT
➡️Up to 2.36× more revenue
on real agent traces(Computer Use, Coding and math agents) and opensource models.
🌐 Website: https://t.co/7I3iERjE6b
📄 Tech report: https://t.co/qDGcwl0svx
💻 Code: https://t.co/0VOBCptKcb
Which agent(e.g. Pi/Openhands) do you want us to accelerate next? Tell us below 👇
@sebuzdugan about the cache churn, we actually did a more careful study in appendix C of the paper, and you are very welcomed to take a closer look at that! also wdym by unrelated agent sessions?
Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads.
On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.
For a single 5090 on the edge, I'd highly recommend FreeToken https://t.co/CuZTdzDLyR, led by the amazing @Andy_ShuoYang and @Chenfeng_X! FlashState targets efficient serving of hybrid models in the cloud (B200/GB300), while FreeToken targets efficient inference on the edge. The two share a lot of core insights though, so definitely check out both papers!
Great question! We jointly manage state and KV with their dependency in mind, trading off access recency (as a proxy of cache reuse potential) and recompute cost. More details are in Section 4 of the paper and you are more than welcomed to check it out! I promise its been hand written and should be easy to follow
I'm not an expert in mac but i feel like the FreeToken paper might help you out for that! i roughly remember chenfeng told me they encountered similar problem (ram being used) on windows https://t.co/ZHNPV1YHyZ
cc'ing @Andy_ShuoYang@Chenfeng_X who are the cutting edge experts leading the amazing FreeToken work! https://t.co/CuZTdzDLyR
For a single 5090 on the edge, I'd highly recommend FreeToken https://t.co/CuZTdzDLyR, led by the amazing @Andy_ShuoYang and @Chenfeng_X! FlashState targets efficient serving of hybrid models in the cloud (B200/GB300), while FreeToken targets efficient inference on the edge. The two share a lot of core insights though, so definitely check out both papers!
For a single 5090 on the edge, I'd highly recommend FreeToken https://t.co/CuZTdzDLyR, led by the amazing @Andy_ShuoYang and @Chenfeng_X! FlashState targets efficient serving of hybrid models in the cloud (B200/GB300), while FreeToken targets efficient inference on the edge. The two share a lot of core insights though, so definitely check out both papers!