We can speculate all day on how much we use AI in our meager daily lives, but on the supply side, demand is THERE. Many likely psyop themselves into thinking AI isn’t useful to THEM, but the tokens continue to go brrrr
AGENTIC TRAFFIC NOW MAKES UP MORE THAN 70% OF ALL INFERENCE TRAFFIC 🚀
Agentic workloads are characterized by four elements:
🟠 Multi-turn: a session includes tens or hundreds of turns, leading to high potential KV-cache reuse.
🟠 Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly.
🟠 High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on the amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached input typically tends towards 1.
🟠 Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KV-cache patterns.