Introducing Together Compiler: agents that continually optimize kernels for your model, your hardware, and your traffic.
We built this to scale frontier research from the teams behind FlashAttention, ThunderKittens, ThunderAgent, and more to every customer deployment.
So far we're seeing:
π +12% to +20% median TPS on serverless endpoints our researchers had already heavily optimized by hand
π Customers in production cutting response and generation times by about a third to nearly half
Read the full story: https://t.co/wnRX2rWDnx
What does coaching basketball have to do with RL? π
Zach Wentz answers it at Open by Design with @nvidia and @reflection_ai.
Watch the full panel: https://t.co/2uRFCC42C4
For languages off-the-shelf speech models underserve, @wonderful_ai, the AI OS for the enterprise, builds its own voice models.
Running on @NVIDIA accelerated computing, through Together AI:
β 274ms time to first sentence
β 100% uptime
β Zero human interaction to scale, ship new weights, or run blue/green tests
https://t.co/IKTqR1r3q7
@kawaidgaf we're planning to open source our eval framework next week! the ~10 tasks are very specific to tasks you can do on Together, we recommend folks come up with their own tasks
Here's how we do agent evals at Together AI:
β Define ~10 representative tasks
β Run agents (Claude Code) in sandboxes
β Test 3 configs (baseline, MCP, skills) for each task
β Analyze trajectories and improve docs + APIs
Agent task completion improved from 67% to 96%!
@petrusenko_max The beauty of Together Compiler is it can re-optimize overnight in most cases. And even optimizes deployments with diverse traffic profiles like our serverless endpoints, where we've started seeing meaningful improvements over already heavily optimized configs.
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput.
Today, weβre sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization.
Shout out to @dylan__lim@reyna_abhyankar@simran_s_arora and the team!
Read more: https://t.co/9Yb1toI5jq
coding agents resend the whole conversation on every turn. prefix caching keeps long sessions fast, but only if the harness and the inference engine both do their part.
@zainhas from Together and @parthsareen from @ollama wrote down what each side needs to get right.
https://t.co/o3Ve8Nm9XB
Last Year We Built Megakernels. This Year We Deleted Them.
One year ago, we helped introduce some of the first AI megakernels to the world. Since then, weβve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, weβve found that they overcomplicate and are not always an ideal optimization target.
Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization β Torch Compile and Megakernels
in a world where agent experience (AX) >> developer experience (DX) you need to make sure agents can use your platform
this is how we use evals to make sure agents can use the Together platformπ₯
How we do agent evals at Together:
β Define ~10 representative tasks
β Run agents (Claude Code) in sandboxes
β Test 3 configs (baseline, MCP, skills) for each task
β Analyze trajectories and improve docs + APIs
Agent task completion improved from 67% to 96%! More below:
closed β open migration works best as an iterative rollout
discover β evaluate β adapt β decide β production
evaluate on real traffic, improve what falls short, then canary the strongest candidate before moving more traffic
How do you build a small coding model from scratch?
@RicursiveAI 's Dan Zhang breaks down the architecture and decisions behind Ripple-65B Preview with @zainhas: dense over MoE, the ablations that held up (and didnβt), and what happened inside the RL runs.
Tuesday, Nov 17 . 11 A.M. PT / 2 P.M. ET