Sharon Sahadevan

AI infrastructure engineer. GPU scheduling, LLM serving, agent platforms. Courses at

[Sharon Sahadevan @sharonsahadevan]

There is no supported way to downgrade a Kubernetes cluster.

kubeadm does not implement it. etcd does not reverse its storage format.

Your rollback plan is the snapshot you took before you started. Without one, your options are roll forward or rebuild.

[Sharon Sahadevan @sharonsahadevan]

kubeadm certificates expire after one year. No warning, no alert, no graceful degradation.

The cluster runs perfectly until the second they expire.

Workloads survive. Kubelets keep pods alive without the apiserver. You lose scheduling and kubectl.

Urgent, not an emergency.

[Sharon Sahadevan @sharonsahadevan]

Mounted the CA bundle. curl works. The application still fails.

Trust stores are per library, not per container. Python's ssl reads the system bundle. requests reads certifi's own copy and ignores it. Java reads a binary cacerts file.

A successful curl proves that curl works.

[Sharon Sahadevan @sharonsahadevan]

Prefill is compute bound. Decode is memory bandwidth bound.

They are different bottlenecks on the same hardware, which is why running both on one replica underuses each, and why disaggregated serving exists.

[Sharon Sahadevan @sharonsahadevan]

Two GPUs on the same node talk over NVLink.

Two GPUs on different nodes talk over the network.

That gap is why your distributed training job is slower than the GPU count suggests, and why topology-aware scheduling exists.

[Sharon Sahadevan @sharonsahadevan]

Your dashboard says 40,000 requests a day.

Your invoice does not care.

Two requests are not comparable units. One is a three word question. The other carries a 2,000 token system prompt and 6,000 tokens of context. Same row in your logs, roughly a hundred times the cost.

[Sharon Sahadevan @sharonsahadevan]

Put a standard Kubernetes Service in front of your vLLM replicas and you quietly destroy your KV cache.

Round robin assumes every replica serves every request equally well. False the moment prefix caching exists.