The big theme at @KubeCon_ this week was inference.
And "Inference" is such a loaded term, so let's think about each piece and where it matters for production teams.
1. GPUs and memory: Making sure you have enough, but at the same time, how it's allocated and where/how it can be shared. Dynamic Resource Allocation (DRA) was talked about a lot here. It handles the GPU slicing, and KEDA/Karpenter handle the scaling.
2. vLLM to serve LLMs and help with KV caching via PagedAttention. The goal is to cache memory efficiently, which should reduce memory waste and enhance throughput.
3. Inference-aware Gateways: AI Gateways with the Gateway API Inference extension capabilities, like @agentgateway, for routing to LLMs.
4. Distributed inference/scheduling: llm-d, which sits between vLLM for cache-aware request scheduling.
5. Isolation: Agents running on this infrastructure need a way to be truly sandboxed, with controls around ingress and egress. The sandbox layer needs to come from a lower-level engine that runs efficiently and helps implement not only isolation, but scale. @kagent_dev + Agent Substrate.
6. Where it all runs: @kubernetesio
This six-layer stack is inference infrastructure and agent infrastructure to serve LLMs and Agents efficiently.
#agenticai #ai #kubernetes
New lab out in Agentic AI The Hard Way for multi-cluster
✅ Spills over to a second cluster when the first is full. It fills both workers in cluster A, gets ResourceExhausted when it tries to resume a third actor, then deletes that record and creates and resumes it in cluster B instead.
✅ Keeps session state outside the actor. It stores a counter and an epoch number in Cloud SQL for PostgreSQL.
✅ Simulates an outage where the old actor is still running. It kills the tunnel to cluster A while A's actor keeps writing in the background.
✅ Proves failover. The shared count carries on in Cluster B, but Cluster B's local file starts at local turns: 1. Cluster A's accepted writes stay frozen at the fence, and Cluster A's PostgREST log shows 409 rejections.
https://t.co/vO64Tq9Ksj
#agenticai #ai #kubernetes
@IamYounesKh This is the way I've been defining it: The Agent is the puppet master, simply put. It's a process that gives the ability for a human to provide input and get an output.
But that's the LLM, not the Agent... so what's the Agent, you know what I mean? 😀. That's why I'm asking these questions in the way I'm asking them BTW.
The majority of the answers when you ask someone "What's an Agent" are typically something like "a loop" or something that has to do with an aspect that's outside of the Agent (the LLM, Skills, MCP, etc.).
@IamYounesKh So if we're thinking CompSci 101, a loop is essentially if/elif/else statements that go until the output reaches what it's supposed to be. Are you saying that's all an Agent is?
I've never one-shotted an Agent and got the exact result I wanted the first time.
It typically takes consulting the output with another Agent, sending the output back and forth between the two, and then getting a logical output.
#agenticai#ai#kubernetes
“one Agent; one Pod… but that doesn’t scale”
Ironically enough, scale isn’t the biggest problem.
The biggest problem with one Agent per Pod is density.
If you have 100 Agents, you have 100 Pods. That means multiple Network Policies, Security Contexts, deployments (100 gitops pipelines?), and a team that has to manage all of those extra workloads.
That’s why the way to think about Agents now should be “multiple Actors; one Pod”.
#agenticai #ai #kubernetes
Kubernetes, by itself, wasn’t designed for AI.
What it was designed for, however, is scheduling and orchestrating workloads.
That’s why running agentic/inference infrastructure and agents on k8s is perfect as long as you have the right engine running within it.
Otherwise, you fall into the trap of:
1. Memory/CPU lock that can’t be allocated to other objects
2. A Control Plane that isn’t designed for tracking millions of persistent objects
3. A major identity problem (identity tied to pods, not agents)
4. Agents hogging resources while staying idle most of the time waiting for human response
And that’s why agent-substrate exists.
It complements all of the beautiful parts of Kubernetes with what’s necessary for the agentic era.
#agenticai #ai #kubernetes
My question whenever I see “all of the sandboxes” is “are they sandboxes?”
Conceptually, what’s a sandbox?
It’s an isolation layer between your machine/server and a workload.
However, by default, that workload still can call out to whatever it wants.
So, is that really a sandbox?
#agenticai #ai #platformengineering
The sandbox market is CROWDED AF
But I still think we're early in figuring out what compute (and even infrastructure) should look like for agents.
Compute is finite and agent demand isn’t slowing down.
The first step is figuring out how to orchestrate containers and microVMs more intelligently. Better resource sharing, higher utilization, faster startup, and tighter control over the infrastructure underneath our sandboxes.
But that’s only the tip of the iceberg.
I want to go deeper: Kernels, networking, storage, operating systems. Every layer of the stack deserves to be rethought for agents.
Competing mainly on price feels like a trap to me. A slightly lower price can just mean a smaller margin, without changing what the infrastructure can do.
I’m much more interested in unlocking things that aren’t practical today.
Imagine one agent forking its live sandbox into a million isolated environments in seconds, each inheriting its memory and execution state.
Now imagine those forks spawning more forks. Millions becoming billions. Each branch exploring a different path, preserving its own state, and sharing discoveries through agent-to-agent (A2A) communication.
I don't think anyone has figured out the winning approach yet...
@codentient@Saboo_Shubham_@rakyll Hey Ale, could you DM me (your inbox is closed which is why I’m asking 😂). Wanted to ask you a question about the kagent integration.
My 5th book is officially published! 🎉
AI is moving so fast in every aspect of our lives from work to personal to healthcare to daily tasks...
And especially for everyone in the technical space. That's why where you run agentic workloads and infrastructure is the make or break between a successful implementation of AI in production and just testing.
That's what this book covers 🤓
https://t.co/C1DVKQTKDr
#agenticai #ai #kubernetes