What UX becomes possible at 10,000 tok/s?
What about 1M tok/s?
Fascinating talk on heterogeneous inference to kick off AIE…very happy I now have six GPUs and four cpus at home for “research”
sat in @nserrino 's talk yesterday at infer/con.
most of us think of inference in tokens/sec, but she showed it as a tradeoff.
1/ more tokens for same speed
2/ more & faster tokens
3/ faster tokens for the same budget
This opens a lot more ideas as inference speeds up.
And tomorrow: @gimletlabs co-founder & CEO @zainasgar joins a panel on disaggregated infrastructure for agentic AI, moderated by Gimlet’s Tom St. John, with UT Austin, NVIDIA, Argonne and d-Matrix. Thu 1:30pm CST, W185abc 👇
https://t.co/1M7pZS81JE
Make inference 10x faster. Now the tool calls are the bottleneck.
@nserrino of @gimletlabs walks through why faster models make the rest of your agent stack matter more.
From AI Engineer Code Summit.
Disaggregating the disaggregated model.
Speculative decoding lets us generate fast tokens with a draft model. What if we could accelerate the drafter itself?
At Gimlet, we disaggregated the drafter: prefill on GPUs, decode on SRAM-centric accelerators.
Why it matters for first-token latency and energy efficiency:
https://t.co/QiSMCZcL4q
1/ We built a board game. Settlers of Catan, but for Compute. Come play with us, and win a custom board (comment or apply at link below to be included)👇
Going to SuperComputing 26? Join our panel, “Orchestrating the Future,” on scaling agentic AI across heterogeneous hardware.
Let’s talk performance, efficiency, and cost.
Nov 19 · 1:30–3 pm CST · W185abc
Session details: https://t.co/PM4oDQ1KK8
What will you build when AI is 10x faster?
Join Gimlet co-founder @nserrino at @aiDotEngineer Code Summit to explore more responsive voice apps and agents that can try, test, and update in seconds.
Nov 11 , 9:10–9:30 AM PT
Continental 4–6 (Mainstage)
Fred Kjolstad joins Gimlet as Chief Scientist.
In our new post, Fred and @zainasgar outline Gimlet’s research vision for recursive self-optimization: agents that help improve the inference stack for increasingly specialized hardware and models.
https://t.co/TAptK7eBvi
@nserrino in the latest episode of @tech_fit_talks discussing the need for inference efficiency. And how we can get more from our infrastructure through smart orchestration across hardware systems.
Getting more useful work from available AI hardware is a substantial engineering opportunity.
@nserrino, Co-Founder of @gimletlabs, on why efficiency matters.
Where do you see the biggest opportunity to use AI hardware more efficiently?
Follow @tech_fit_talks for more AI infrastructure conversations.
#AIInfrastructure
In the '90s, general-purpose chips wiped out the specialists. The original Thinking Machines, Cray Computer, and a wave of others shut down, and their best ideas got absorbed into other chips.
Zain Asgar of @gimletlabs argues the conditions that made generalists win have changed. Dennard scaling (shrinking transistors made chips faster at the same power) ended around 2005, transistor costs stopped falling around 2012, and memory bandwidth never kept up with compute.
Meanwhile, token demand is projected to grow 24x by 2030 while available power grows about 2x. When power is the bottleneck, performance per watt wins, and that's where specialized chips shine.
Full piece from Zain on why "the time of specialization is upon us": https://t.co/dFA8McPid5
Why hardware specialization is back.
AI inference needs more tokens/sec per user and more tokens per watt.
Power limits. Slowing transistor scaling. The memory wall. Different chips are optimized for different constraints.
Our co-founder @zainasgar on what comes next.
Faster AI agents need more than compute.
Michelle Nguyen (Gimlet) and Daniel Kim (@cerebras) unpack how matching each phase of inference to the right silicon delivers faster tokens.
Watch their AI Engineer talk: https://t.co/rE5H1XlCqr
When we designed our wafer-scale systems, we focused on keeping compute close to memory so we could generate tokens quickly. Bringing that speed to more developers also takes work across the software and infrastructure around it.
That’s why I’m excited about our partnership with @gimletlabs. Their software maps different phases of inference to the hardware best suited to each, bringing Cerebras into Gimlet Cloud behind a single API.
Together, we plan to deploy Cerebras-powered inference capacity, with the first datacenter expected online later this year.
As agents take on longer tasks, the time spent waiting on each model call adds up. I’m looking forward to seeing what developers build when they can work through those steps faster.
Congrats to @zainasgar and the Gimlet team. We're just getting started.
Gimlets are one of my favorite drinks. Gin + juice. 🍸
@gimletlabs is one of my favorite clouds. GPUs + @cerebras wafer-scale accelerators. ☁️
Some combinations are just delicious together. 🚀
https://t.co/ARAaLdVYu3