Average latency was calm. A few users were stuck waiting.
Baseline: p50 20 ms, p99 34 ms. Fine, right?
p99.9 was already 1532 ms.
Same sampler, five-branch fan-out: p99 jumps to 597 ms. Nothing downstream got slower. Only the request shape changed.
That is what average latency hides.
Full experiment and runnable code here:
https://t.co/49pd16YUOx
I learnt a long time ago that there is a big difference between making a living and making a life. In the times to come, AI will get increasingly better at the skills that we've used to make a living. Our imperative will be to instead make lives. Not artificial lives. Or artificial lives. But our own lives and of those we love. As machines get better at answering, and solving what they are asked, our work is to get better at asking, at making, creating, and deciding which questions are worth a life, and refusing to outsource that.
Another reminder. Tools we depend on can be switched off overnight, by a government we have no vote in.
We have seen this before. In manufacturing, we depend on imported machines, strait of Hormuz, imported technology licenses. In AI, we depend on imported models. Different industry, same pattern. We build on someone else’s foundation, and someone else decides when we can stand on it. Stop building services on top of these models. Start building the models. We have the engineers. We have the capital. Why are we so interested in 10 min deliveries.?
Where are our deep tech founders. Where is the capital willing to wait for long gestation?
The guy confidently claims India’s EVMs are “programmed in Python” and can be hacked with just 3 lines of code. 😂
Bro,EVMs run on basic microcontrollers with firmware burned in C/assembly into OTP chips not some Raspberry Pi Python script.
Hey Rishabh Pant,
Don't listen to Gautam Gambhir.
You're doing fine. You win games for us. You bring joy to us.
He only brings grumpiness & bitterness which he masks as grit and toughness.
So, don't change.
I spent lot of time suspecting our hardware for a serach service.
Surprisingly the problem was an OpenSearch call with no timeout. p50 looked fine the whole time. p99 was the only one telling the truth.
Missing timeouts are not a minor config issue. They are an SLO violation you have not found yet.
IT'S WORKING! 🥳
Program counts to 100 and halts. Uses ALU, registers, cmp, branch, unconditional jump, GPIO and immediates. And to get this far it also means the bootloader, SRAM and program counter are functioning too.
I'm honestly shocked the thing works!
They removed Kubernetes from production, and their AWS bill dropped 68%.
Kubernetes was solving a problem they did not really have.
small team, few services, low scaling needs.
But they were managing:
- Helm charts
- YAML complexity
- custom operators
- platform maintenance
- debugging pods instead of product issues
At some point, the infrastructure became more complex than the application.
That is the lesson
✅Kubernetes is powerful when you need scale, orchestration, and platform control.
⚠️if your system is simple, Kubernetes become expensive