Karthikeyan

Professional “why is prod slow” investigator. Off-duty: arguing about cricket and pretending to understand markets.

[Karthikeyan @NarainTak]

Our dashboard said checkout takes 0.08s. p95 was 2s. Meanwhile 1% sampling kept 0 of our 18 errors and a card number was sitting in our traces 😅

Free 1.5 h OpenTelemetry course, every bug in a lab you can run. #OpenTelemetry #SRE

https://t.co/EAfXyeNE0p

[Karthikeyan @NarainTak]

#KISSHT went upwards after this 📈

[Mаղʝմռα𝗍𝗁 Տ @ChikouTrader]

#KISSHT : CMP - 304

Bearish, should test 225- 240 levels shortly !

SEBI Unregistered. DYOR. Refer to bio for disclaimer.

#Investing
#OnEMITechnology

[Karthikeyan @NarainTak]

A status label next to a Claude Code background agent read 52,264 tokens to write 4 words. Across my four sessions those labels were 13-21% of all tokens, and in 2.1.316 there's no setting to turn them off. The lab is public.
https://t.co/lcAM1A07za
#ClaudeCode #OpenTelemetry

[Karthikeyan @NarainTak]

This number has been sitting with me.

Nifty 50 total return did 6.38% a year over 5 years. NSE factsheet, as of 30 Sep.

I keep nodding at stories that need a much better year than that, and I never write the gap down.

What number are you using? What would make you scrap it?

[Karthikeyan @NarainTak]

Prod is slow. Everyone blames the database.

The timeline is doing that to datacenters.

Friction and "momentum never stops" can both be true this week.

The trace is orders, utilization, or the multiple.

Which one are you reading, and what print proves you wrong?

[Karthikeyan @NarainTak]

Our AI support bot was being helpful: every "where's my OTP?" got a resend. By minute 7 the SMS vendor had turned away 50 messages, incl. login codes for people who never spoke to the bot 🙃

Nobody owned it, so I mapped it with eBPF. #OpenTelemetry #SRE
https://t.co/5YokMVxbCO

[Karthikeyan @NarainTak]

@DotsonZion Your question turned into a Short 😄 Gemini CLI's tool_call event only logs the file path and size. But api_request ships the whole conversation, so one read of a .env put the key in my telemetry. Twice.
https://t.co/O1jTQ2txce

[Karthikeyan @NarainTak]

Found this while testing coding agents: Claude Code and Codex keep prompts out of telemetry unless you opt in. Gemini CLI? logPrompts is on by default.

So if you switched on OTel for all three, one of them has been sending your prompts along. Oops.

https://t.co/jE9IeQRvDl

[Karthikeyan @NarainTak]

Traced one of my coding agent's commits back to the exact run that wrote it. 15 spans: 4 model calls, the file write, the git commit, and 2 commands it tried that got blocked 👀

git log will never show you that part. #OpenTelemetry #AIagents

https://t.co/34HjDqhqjA

[Karthikeyan @NarainTak]

@DotsonZion Honestly, I’ve only checked the prompts so far. There’s also a separate tool_call event and an api_response log, so that’s what I’m planning to dig into next. Fun fact: https://t.co/HVI7J8oAyq is included in every event when you’re logged in.

[Karthikeyan @NarainTak]

@seals007vx Yep, learned that the hard way! I'd also strip the prompt content at the Collector, just in case someone's laptop config overrides yours.

[Karthikeyan @NarainTak]

@yuanshandao Just a heads-up: from what I can tell, enabling telemetry in settings.json also enables logPrompts by default, unless you explicitly turn it off. I didn’t notice any warning about this, which is kind of the whole problem.

[Karthikeyan @NarainTak]

Wabag: ₹19,400cr order book, >4x annual revenue.

Q1 revenue ₹887cr (+21%)
PAT ₹90cr (+37%)
Net cash ₹965cr

But orders aren't earnings.

The test: convert projects into revenue while protecting 13–15% margins and controlling working capital.

Track execution, not wins.

[Karthikeyan @NarainTak]

Cardinality is usually treated as a cost problem. It can also make an SLO look healthier than it is.

In OpenTelemetry, overflowed values remain while their attributes are removed.

The total stays right. The filtered query undercounts.

#OpenTelemetry
https://t.co/DztPSpEWvx

[Karthikeyan @NarainTak]

@danizhu @langfuse A shared schema would be valuable. It would let teams query memory behavior across agents and platforms without rebuilding the same conventions for every stack. Thanks for the detailed explanation, Dani. I’ll follow how this develops.

[Karthikeyan @NarainTak]

@danizhu @langfuse That separation makes sense. Shared trace context connects memory to agent action. The next challenge is consistency: are you defining common attributes for memory ID, retrieval rank and confidence so these traces remain queryable across implementations?

[Karthikeyan @NarainTak]

@livingdevops Tool questions show familiarity. Incident scenarios reveal judgment. Give the candidate incomplete telemetry, a failing service and limited time. Then ask what they investigate first, when they roll back and which risks they accept. That is much closer to testing real SRE owners

[Karthikeyan @NarainTak]

The broader lesson:

Noisy alerts may not signal weak ownership. Improving them may simply be slow, risky and hard to validate.

Make alert behavior testable before merge, and alert quality becomes part of the engineering workflow.

Source: https://t.co/aVxDLG2nQx

[Karthikeyan @NarainTak]

Airbnb cut alert noise 90% while managing 300,000 alerts.

The breakthrough was pre-merge feedback: engineers backtested proposed rules against historical data before production.

Alert fatigue was a feedback-loop problem, not a culture problem.

#Observability #SRE

[Karthikeyan @NarainTak]

Why ordinary code review was not enough:

It could validate syntax and logic, but not answer:

• Would this alert fire on real production data?
• How often?
• Would it create noise at 3 AM?

Airbnb added alert diffs and historical firing timelines to the PR workflow.