Sumukha Pk

What am I doing?

[Sumukha Pk @impk46]

@coolcoder56 Even seniors don't make that much chill it's misleading

[Raghu Kishore Vempati @raghuvempati]

I have taken one of the problems I used to ask in interviews a few years ago and applied it to the world of AI, since it seems like AI is everywhere these days.

Someone who is Senior/Principal Engineer should be able to easily answer this (I assume...)

Now - please be fair! I’m hoping you won’t rely too much on LLMs (though I won’t stop you if you do). The goal is to help understanding the subject how it can manifest in the real world...

Here is the problem:

Context:
- Kubernetes Cluster
- A single GPU node runs ~200 inference pods (can it?)
- Everything healthy. High QPS. GPU fully utilized.

Now:
- containerd suddenly gets SIGKILL’ed.
- Nothing crashes.
- Traffic continues.
- Pods remain Running.
- metrics still scraped.
- GPU utilization unchanged.
- Inference latency looks normal.

And then: You restart containerd.

Things start getting weird:

- k8s logs work
- However, container not found
- Deleting pods hangs in Terminating, never gets deleted.
- New pods schedule successfully — but fail GPU allocation intermittently.
- Old pods still serve traffic and still hold GPU memory.
- Only a node reboot clears the state.

Questions:
1. Why are workloads:
- alive at the kernel level
- visible to k8s cluster
- still consuming GPU
- and yet partially invisible to the runtime?

2. And why does GPU scheduling degrade only after containerd restarts?

Bonne Chance! 🙂

@RealAnkush @sunnykgupta @arpit_bhayani - please share among your acquaintances

[Sumukha Pk @impk46]

@arpit_bhayani Good things to draw from but I feel folks should have originality and ask things what they feel like as well. Except for office food or amenities ofc.

Also @arpit_bhayani have you gotten any good answers from the first q? Surprised if well oiled machines also have issues like G

[Sumukha Pk @impk46]

@ShashiTharoor I think this is called deflection

[Sumukha Pk @impk46]

@btppubliceye app isn't letting any sign ups.
The city is full of no parking parks that are blocking the road. Please help

[Sumukha Pk @impk46]

@ankurmittal Genuine question how do you get a burgundy and epm since they both need an account in the respective banks?

[Sumukha Pk @impk46]

@iSatishAgarwal Now they don't allow buying from gift voucher balance on most of these websites because of siphoning like this

[Sumukha Pk @impk46]

@ankurmittal This literally has 3 airports. Not worth it

[Sumukha Pk @impk46]

@DoBaniye This is exactly why it's being devalued across platforms where you can buy vouchers :))

[Sumukha Pk @impk46]

@sharwars_ Hi Micheal

[Sumukha Pk @impk46]

@anand_luhar Looks like this can easily file our taxes instead of us scrambling for documents

[Sumukha Pk @impk46]

@Niuu_d Aura

[Sumukha Pk @impk46]

Proud @JeremyClarkson fan after honking at some cyclists today

[Sumukha Pk @impk46]

@sharwars_ Microing in staying sane

[Sumukha Pk @impk46]

@saaannvii Bro why do I get notifications from these toxic relationships😭

[Sumukha Pk @impk46]

@anaanya__ Why do I get notified of these random people??

[Sumukha Pk @impk46]

@sharwars_ Man I'm regretting googling mercury gatorade

[Sumukha Pk @impk46]

@sharwars_ You can't stop there without explaining...

[Sumukha Pk @impk46]

@sharwars_ Disclaimer: might cause choking

[Sumukha Pk @impk46]

@sharwars_ Evolution has occurred