Positron's Titan is joining the Parasail fleet. 5MW. Trillions of tokens a day of cost-optimized compute by late 2027. @parasail_io@positron_ai@parasail_io is the first multi-silicon inference cloud that routes your prompts to the best chip for the job, serving trillions of tokens a day to hundreds of customers. Positron fills a vital part of compute spectrum on the cost optimized side: 4X better cost-per token than next gen NVIDIA/AMD systems which is perfect for large-scale agent swarms, background agents, and physical AI.
Positron has done incredible work building a chip and system that orchestrates across DRAM and SRAM allowing for massively scalable memory pools with commodity memory giving the best of both worlds for memory capacity and bandwidth.
We've been excited to partner with them in the early days through the launch of their Atlas chip and now big capacity plans with Titan.
@positron_ai
Congrats to @MeghanaMadhyas1 and Ben for the great talk on the many layers of agents we use to drive top tier performance with a small and mighty team.
Our agents tune LLM inference on production GPUs. Here’s what had to be built for that to work:
- GPU leases + a queue so experiments don’t steal customer capacity
- Cloned deployments on mirrored live traffic
- Canaries with auto rollback and a human sign-off
Checkout @MeghanaMadhyas1 and Ben Segal's talk at @QCon later today to learn more about our autonomous LLM performance optimization system.
Announcing the first of many GPU clusters that we own and operate ourselves. Given the volatility of the market, a hybrid strategy of owning compute and renting from the utility-scale GPU providers (i.e. not Neoclouds trying to claim software margins) is the way to deliver the best economics and flexible capacity to our customers.
Our newest owned cluster is live: NVIDIA B300s in Denver, in the @parasail_io fleet the day after burn-in.
We have a mixed fleet of renting & owned capacity. Rented lets us scale fast and pick up new silicon as it lands. Owned clusters give us a base we control and tune for the largest customer workloads. An inference cloud locked into one way of getting compute gets stuck when the winds change.
Implication -
If the moat on your HW is actually SW, you no longer have a moat.
If, otoh, the moat on your HW is actually HW, not only is your moat intact, but customers' ability and willingness to try your HW is dramatically increased.
Btw, actual HW moats exist in two distinct flavors:
1. Roofline HW capability. Everyone can get much closer to roofline, much faster now.
2. Supply chain. Do you have the wafers, parts, manufacturing capacity, ... to deliver at scale?
Both must be considered.
It feels like the 3-4X is achieved where the KV cache is very full and starts missing a lot. Is this just another way of saying it increases effective KV cache space by ~10%?
Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads.
On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.
Solving utilization for heterogenous compute is the defining problem and a lot going on at @parasail_io behind the scenes on this. Stay tuned for some big launches later this year.
A chip that’s 3× faster but idle half the day isn’t 3× cheaper:
A prefill or decode device only pays off when your traffic matches the ratio you bought, and that ratio keeps changing. Your GPUs can switch roles when that happens, but specialized silicon just sits idle while you keep paying for it. Software keeps moving the bar too, since megakernels alone have pushed low-concurrency decode on ordinary GPUs up by 40–75% this year, and every one of those gains is another knock on specialized chips.
Consider: “how fast?” but “how fast, times how often it’s actually busy, after moving the KV cache over the network?” If your GPUs run at 70% and their chip averages 40% across a real year of traffic, it needs to be ~1.75× better just to tie. For a single model with very stable traffic (like a premium fast tier), that can work out, but in many case, general-purpose computing can be more useful.
Was talking with the founder of a large inference cloud yesterday.
His thoughts:
1/ lot's of rumours of some large chunks of 2027 capacity has disappeared - now no longer being built or not financed
2/ demand for tokens isn't going away it's exploding and pushing spot prices up
3/ whoever secures real compute and already has real compute in an even more supply constrained environment has the cards right now
Interesting times
Connecting a bunch of SRAM + acceleration together to achieve extreme decode speed is incredibly hard no matter what you try. At Groq, the synchronization and pipelining was insanely hard with hundreds of discreet accelerators. Wafer scale is also hard. There isnt anything on anyone’s roadmap for the next two years that can sustain high throughput and high decode speed for > 100B parameter models and is shipping in volume other than CBRS and there is a sizable market for it. Source: I was CPO at Groq when LLM product was conceived and launched.