Impala AI

Inference done right

[ziv bakhajian @z_bakhaji]

We at @impala_ai took @vllm_project's new Rust frontend to a test on a real async-AI production workload.
The results: a range of −1 % to +1.2 % change in throughput and average of <0.4%, and the latency change was 0.02 %. Both in the noise regime.

>>

[Impala AI @impala_ai]

Most inference platforms are tuned ahead of time for a "happy medium."

That works when your workload is predictable.

Agentic AI is not. It's async and it changes (and breaks) inference as we know it.
At Impala, we treat inference as a high-performance computing problem. This means observing live workload signals and adapting kernel selection, memory movement, and scheduling in real time, without operator intervention.

https://t.co/I3T0BRxPSy

[Impala AI @impala_ai]

Exactly. The bottleneck isn’t just batching anymore. Agent workloads are bursty and stateful, so the hard part becomes adapting memory, routing, and scheduling in real time.

Would love to compare notes sometime, feels like we’re both seeing the same shift in inference infrastructure.

[Impala AI @impala_ai]

Most production tokens are no longer consumed by interactive AI applications. They come from background agents running in loops, with no humans at the keyboard.

Async Agent work is long-horizon, idle-heavy, and read-heavy. The mismatch is at the core of inference bottlenecks.

Solving this means treating the cluster as one machine, the workload as a moving target, and adapting, in-flight, every scheduling, routing, and memory choice. This is Impala’s approach, which treats inference as a high performance computing problem.

@z_bakhaji's analysis explains why inference breaks and how to solve that.

https://t.co/UJ7SRGXZJO

[Impala AI @impala_ai]

We need more precision when it comes to Async AI inference.

"Agent" "Step" "Trace" "Action"

Everyone in AI is using these terms. But each means something slightly different.

It may seem like no big deal, but it can become a real problem when unclear language impacts engineering decisions.

@z_bakhaji put together a vocabulary for async AI agent inference.

Anything you'd like to add?

https://t.co/QVHGwOKGwK

[Impala AI @impala_ai]

Fashionably late