Rishab

[Akhilesh Mishra @livingdevops]

In an interview, I was asked to design an AI agent for Kubernetes incident response. I started my answer with three goals for my agent

- Reduce noise. Correlate alerts and filter known non-actionable ones before they wake someone up.
- Reduce MTTR. Let the agent do the repeat work (events, logs, metrics, runbooks) and hand over a diagnosis when a human is needed.
- Auto-resolve simple issues. Handle low-risk problems automatically. Everything else needs human approval.

>> Step 1: The alert
- Prometheus Alertmanager fires the alert.
- Email and PagerDuty stay as they are.
- The agent is an addition, not a replacement. If the agent goes down, engineers still get their alerts.

>> Step 2: Getting the alert to the agent
- SNS sends the alert to an SQS queue.
- This gives me buffering, retries and a dead letter queue.
- I also add deduplication so one incident does not start 50 agent runs, and idempotency because SQS can deliver the same message twice.

>> Step 3: Triggering the agent
- A Lambda picks up the message and checks whether the alert comes from an approved source and environment. If not, it stops.

- Lambda has execution time limits, so I keep it as a thin trigger and run the longer agent workflow on AgentCore, with Step Functions if workflow orchestration is needed.

>> Step 4: Giving the agent tools

Three read-only MCP servers:
- EKS
- Grafana
- CloudWatch

Every tool call goes through a gateway with OIDC authentication.

The agent never holds cluster credentials directly. Kubernetes access uses least-privilege RBAC.

>> Step 5: Investigation
The agent first understands the environment, cluster and service from the alert. Then it:
- Pulls events and log tails from EKS, including previous container logs for crash loops.
- Checks Grafana metrics for 10 minutes before and after the alert.
- Checks CloudWatch if it needs more AWS-side signals.

No need to dump everything into the model. Give it the information it needs.

>> Step 6: Knowledge base

A Bedrock knowledge base contains troubleshooting playbooks and a rulebook mapping events to recommendations, with examples.

An ingestion API keeps it updated. A chat component on ECS lets engineers query the same knowledge.

If no playbook matches, the agent says so instead of guessing.

>> Step 7: Deciding the action
This is where I separate investigation from execution.
- Known non-actionable alerts are suppressed using predefined rules.
- Alerts needing human attention go to Slack and PagerDuty with the diagnosis attached.
- A small list of known-safe actions can run automatically.
- Everything else waits for human approval.

>> Step 8: Guardrails
A few things I would never leave to the agent:

- No deleting deployments, StatefulSets or secrets.
- Pod deletion only when sufficient healthy replicas exist and a controller can replace the pod.
- Approved environments and triggers only.
- Rate limits per service. A restart will not fix a bad deployment or dependency outage.
- Logs are untrusted input. They are data, never instructions.

>> Step 9: Cost and execution limits

An agent can become expensive very quickly if you don't put limits around it. A few things I'd implement:

- Use rules and a small model for initial triage. Send only new or unclear incidents to a larger model.
- Set a token budget per run, along with hourly and daily limits.
- Trim tool responses. Error lines instead of full logs, Grafana summaries instead of raw metrics, and only the top 3 knowledge base results.
- Maximum 3 retries per tool call, with backoff, only for timeouts and server errors. Never retry permission errors.
- 10 seconds per tool call and 5 minutes for the entire investigation.
- Maximum 10 tool calls and 6 reasoning turns. Repeating the same call is treated as a loop.

The SQS visibility timeout must also account for the maximum processing time to prevent duplicate processing while an investigation is running.

When any limit is reached, the agent stops, sends whatever it has found marked as incomplete, and notifies a human.

>> Step 10: Proving it works
- Every tool call and decision is traced using OpenTelemetry.
- I replay past incidents and compare the agent's diagnosis with the actual human root cause.
- I track diagnosis accuracy, pages avoided, time to diagnosis and cost per successful investigation.
- If accuracy is low, the agent stays advisory. It doesn't get more permissions just because we want more automation.

Building the agent is the easy part.
Deciding what it must never do is the real engineering challenge.

[Akhilesh Mishra @livingdevops]

Kubernetes is beautiful.

Every Concept Has a Story, you just don't know it yet.

In k8s, you run your app as a pod. It runs your container. Then it crashes, and nobody restarts it. It is just gone.

So you use a Deployment. One pod dies and another comes back. You want 3 running, it keeps 3 running.

Every pod gets a new IP when it restarts. Another service needs to talk to your app but the IPs keep changing. You cannot hardcode them at scale.

So you use a Service. One stable IP that always finds your pods using labels, not IPs. Pods die and come back. The Service does not care.

But now you have 10 services and 10 load balancers. Your cloud bill does not care that 6 of them handle almost no traffic.

So you use Ingress. One load balancer, all services behind it, smart routing. But Ingress is just rules and nobody executes them.

So you add an Ingress Controller. Nginx, Traefik, AWS Load Balancer Controller. Now the rules actually work.

Your app needs config so you hardcode it inside the container. Wrong database in staging. Wrong API key in production. You rebuild the image every time config changes.

So you use a ConfigMap. Config lives outside the container and gets injected at runtime. Same image runs in dev, staging and production with different configs.

But your database password is now sitting in a ConfigMap unencrypted. Anyone with basic kubectl access can read it. That is not a mistake. That is a security incident.

So you use a Secret. Sensitive data stored separately with its own access controls. Your image never sees it.

Some days 100 users, some days 10,000. You manually scale to 8 pods during the spike and watch them sit idle all night. You cannot babysit your cluster forever.

So you use HPA. CPU crosses 70 percent and pods are added automatically. Traffic drops and they scale back down. You are not woken up at 2am anymore.

But now your nodes are full and new pods sit in Pending state. HPA did its job. Your cluster had nowhere to put the pods.

So you use Karpenter. Pods stuck in Pending and a new node appears automatically. Load drops and the node is removed. You only pay for what you actually use.

One pod starts consuming 4GB of memory and nobody told Kubernetes it was not supposed to. It starves every other pod on that node and a cascade begins. One rogue pod with no limits takes down everything around it.

So you use Resource Requests and Limits. Requests tell Kubernetes the minimum your pod needs to be scheduled. Limits make sure no pod can steal from everything around it. Your cluster runs predictably.

[Rishab @rishab_485]

@akkiex007 Hi Akash
I can’t send DM
Please do find me profile here
I have 8.5 years of experience in supply chain inventory, affiliate fraud systems, physical stores checkout and developer productivity tooling across my career
Actively looking for opportunities

https://t.co/Pr5GrtnX4j

[Viora Tech @Viora_Tech_Ai]

A woman bought a MacBook Pro for $2,000 three years ago.

She used it for Safari, Mail, and Word. She assumed that was what a Mac was for. She assumed the trackpad was just a trackpad. She assumed Spotlight was just a search bar. She assumed the green maximize button was just a maximize button.

Her coworker a designer who's been using Macs for 12 years borrowed her laptop for 5 minutes to look something up. She watched her use it for 30 seconds, then said:

"You're using a $2,000 machine like a $400 Chromebook. Your MacBook has a clipboard that syncs from your iPhone. A screen recorder built into the keyboard. A file renamer that handles 100 files at once. A background noise machine for focus. A window manager hiding behind the green button. A text expander that writes your emails for you. Most of it is one shortcut away. You've never pressed any of them."

She changed 10 settings and learned 10 shortcuts in about 20 minutes.

Her MacBook felt like a different computer. She stopped downloading third-party apps. She stopped doing things manually. She stopped reaching for her mouse.

She said: "macOS is designed to be learned gradually. Apple assumes you'll discover things over years. Most people never do. The features are there. They're just not on the surface."

Here are the 10 MacBook settings and shortcuts most owners have never used:🧵

[Neha Sharma @hellonehha]

I have shared 2 blogs, they should help everyone to start with GTV.

what it is,
who can apply,
what is the eligibility etc.

read this:

https://t.co/W279ulACLq

https://t.co/m5pisBcHzp

https://t.co/7YWygl1Egv

[Swati Gupta @hrswatigupta]

My younger brother applied to 180 jobs on
LinkedIn over 3 months.

Result: 0 interviews. 142 automated rejection
emails. 38 complete ghosts.

He has a solid degree and 4 years of experience.
He thought he was unemployable.

Then we changed how his resume talked to the
screening software.

In the next 10 days, he got 9 interview
requests from the exact same tier of companies.

Here's the exact 5-minute system:

[Akash Singh @akkiex007]

thank me later https://t.co/wyIfWlmtD8

[Pushkar @86pushkar24]

TopMate Experience Series:

Had a call recently with someone who spent 6 years at Amazon (intern → SDE-3) and got laid off in the Feb cycle.

She’s still getting 6 months of severance, so financially she’s okay for now. The problem is interviews. She’s been giving them non-stop — Postman, Stripe, Palantir London, Meta London, Razorpay, and a couple of well-paying startups — but keeps failing the DSA rounds.

When I asked what she did in the last 7 months, the answer was mostly “gave more interviews.” She joined a startup briefly (they were matching her Amazon package for her prev role) but left because the work felt too hectic after 6 years at Amazon. She even got the final offer at Postman before it got revoked as they ended up hiring someone else.

The pattern was clear: LLD and HLD she was confident about as she did that only last 6 years at Amazon. DSA was the consistent bottleneck esp. Trees, Graphs, and DP. She never really prepared DSA properly after joining Amazon as an intern with basic knowledge of it, and it never came back to bite her… until now.

Current situation : she keeps starting a topic, loses confidence while solving problems, restarts, and the loop continues. When I shared a focused set of questions and told her to just do those properly, she said she already knows that set and just doesn’t feel like doing them. When I asked how she plans to clear interviews then, she had no answer.

Two clear takeaways :

1. DSA is still very much alive & being asked at good companies (even for people with 6+ YoE).
2. If you want to stay relevant for product companies / Big Tech, finish solid DSA once properly instead of treating it as something you can “manage later.”

Now,she’s got a Google interview coming up. I told her to buy time from the recruiter and actually prepare DSA this time instead of jumping into another interview in the same state. I also think it's not her mistake : when you’re unemployed, the urge to keep applying is strong, but without reflecting on why previous rounds failed, it just becomes a loop. (also her parents also don't know that she had no job from last 7 months 😶)

But clearly, sometimes the gap isn’t talent. it’s just that DSA that was never properly closed.

[Puneet Patwari @system_monarch]

System design round at BookMyShow for SDE-3 (₹70L+ CTC):

12:00 PM. Coldplay tickets go live. 8 lakh people are in the queue for 50,000 seats.

Two users tap seat F-14 in the same millisecond. Both see "Seat selected."

You can't lock the whole seat map on every click. If you lock nothing, one of them pays ₹6,500 for a seat someone else already owns.

How do you make sure every seat goes to exactly one person, without freezing the seat map for everyone else?

[Suraj Sharma @suraj_sharma14]

For people who keep asking what to build as a Backend Engineer in the AI Era.

➣ Build your own AI Gateway
(semantic caching + token budgeting + provider fallback chains)

➣ Build your own Durable Workflow Engine
(state checkpointing + saga patterns + human-in-the-loop pauses)

➣ Build your own PII Redaction Proxy
(UUID masking + deterministic encryption + post-generation unmasking)

➣ Build your own Hybrid Search Engine
(pgvector + BM25 + cross-encoder reranking + row-level security)

➣ Build your own Streaming SSE Proxy
(token chunking + backpressure handling + TTFT/ITL metric emission)

➣ Build your own Token-Aware Rate Limiter
(Redis sliding window + prompt complexity scoring + per-tenant quotas)

➣ Build your own Webhook Reconciliation Engine
(idempotency keys + dead-letter queues + automated replay tooling)

➣ Build your own Event-Sourced Trajectory Store
(immutable agent logs + time-travel debugging + state replay)

➣ Build your own Inter-Agent Auth Broker
(delegated OAuth flows + scoped JWTs + least-privilege tool access)

➣ Build your own Cost Kill-Switch Middleware
(spend projection + circuit breakers + graceful degradation chains)

➣ Build your own Shadow Traffic Router
(5% async mirroring + trajectory diffing + silent cost comparison)

➣ Build your own Queue-Based GPU Autoscaler
(queue depth tracking + spot instance provisioning + drain logic)

➣ Build your own LLM Tracing Pipeline
(OpenTelemetry spans + token counting + correlated DB query logs)

➣ Build your own Edge Fallback API
(cloud 503 detection + local model routing + offline request queuing)

➣ Build your own Ephemeral AI Provisioner
(Terraform scripts + isolated vector DBs + per-PR test environments)

Pick 3. Build them from scratch. Document every decision.

Most people write CRUD APIs.

Builders architect the nervous system for autonomous agents.

Bookmark this. You'll need it.

[Puneet Patwari @system_monarch]

As a Backend Engineer, you must build these projects.

1.) URL shortener that survives 10x traffic
Build: Postgres + Redis cache + rate limiting, load tested to 5,000 requests a second.
Why: caching, ID generation and capacity math in one project.

2.) Distributed rate limiter
Build: token bucket and sliding window in Redis, shared by 3 app servers.
Why: it shows up in almost every Senior loop.

3.) Job queue with retries
Build: workers with backoff, a dead-letter queue and idempotent jobs.
Why: you learn that "at least once" means "sometimes twice."

4.) Chat server
Build: WebSockets, presence, delivery receipts, reconnects.
Why: real-time is where most candidates start guessing.

5.) Payments ledger
Build: double-entry ledger with idempotency keys and nightly reconciliation.
Why: money forces you to care about correctness.

6.) Zero-downtime migration
Build: rename a column on a 10M-row table while a load test runs.
Why: the Senior skill nobody practises.

Put the load test numbers in your README. That's your system design round, already rehearsed.

[Suraj Sharma @suraj_sharma14]

If you learn AI Agent Engineering, you will never be unemployed again.

I’d do this.

Stage 1: Agent Architecture & State Machines
- Learn: Directed Acyclic Graphs (DAGs), cyclic workflows, LangGraph, Temporal dot io, state serialization.

- Practice: Build a multi-step agent using a state machine that can loop, branch and pause without using a fragile `while True` loop.

- Why: Chat loops crash in production. Real-world workflows require deterministic state management and crash recovery.

Stage 2: Tool Use & MCP (Model Context Protocol)
- Learn: JSON-RPC, strict Pydantic schemas, MCP server/client architecture, dynamic tool discovery.

- Practice: Write a raw MCP server from scratch exposing a local Postgres DB, handling auth and scoped permissions without using a wrapper SDK.

- Why: Agents need standardized, secure hands to do real work. Bespoke API wrappers do not scale across an enterprise.

Stage 3: Computer Use & GUI Automation
- Learn: DOM parsing, Set-of-Mark prompting, coordinate mapping, screenshot diffing, Playwright/Puppeteer.

- Practice: Build a vision-based agent that logs into a legacy web portal, navigates menus via screenshots and downloads a CSV.

- Why: 80% of enterprise software has no API. GUI agents unlock the legacy world that text-based agents cannot touch.

Stage 4: Advanced Memory & Context Engineering
- Learn: 3-tier memory (working, episodic, semantic), vector eviction policies, context compression, token budgeting.

- Practice: Build a memory engine that automatically compresses a 50-turn conversation into an episodic summary to prevent context overflow.

- Why: Infinite context causes hallucinations and OOM errors. Memory must be actively managed, scored, and evicted.

Stage 5: Multi-Agent Swarms & A2A Handoffs
- Learn: Agent-to-Agent (A2A) protocols, typed state passing, supervisor vs. peer-to-peer topologies, consensus voting.

- Practice: Build a 3-agent swarm (Researcher, Coder, Reviewer) that passes strictly typed JSON state, never raw text.

- Why: Single agents fail on complex tasks. Swarms divide and conquer, but raw text handoffs cause context drift and hallucinations.

Stage 6: Sandboxed Execution Environments
- Learn: E2B, WebContainers, gVisor, Docker-in-Docker, resource limits, network egress blocking.

- Practice: Build a code-interpreter agent that writes and executes Python in an isolated microVM with zero internet access.

- Why: Letting an LLM run `os.system()` on your host machine or production server is a catastrophic security risk.

Stage 7: Inter-Agent Security & Guardrails
- Learn: Indirect prompt injection, tool poisoning, sanitization proxies, least-privilege scoping.

- Practice: Build a middleware that strips malicious instructions from an email before the "Email Reader" agent passes it to the "Database" agent.

- Why: Multi-agent systems create internal attack surfaces. Agent A can be tricked into hijacking Agent B's tools.

Stage 8: Human-in-the-Loop (HITL) & Breakpoints
- Learn: Asynchronous approvals, webhook routing, Slack/Teams integrations, durable execution pauses.

- Practice: Build an agent that drafts a $10k invoice, serializes its state, and waits 3 days for a human to click "Approve" in Slack before resuming.

- Why: High-stakes automation requires human accountability. Agents must know how to pause, wait and resume cleanly.

Stage 9: Self-Correction & Reflection Loops
- Learn: Critic agents, traceback analysis, retry with context, compiler error parsing.

- Practice: Build a coding agent that reads its own stack traces, rewrites the failing code and retries up to 3 times before escalating to a human.

- Why: Agents that fail on the first try are useless. Agents that debug themselves are digital employees.

Stage 10: Trajectory Evals & CI Gates
- Learn: Step-level grading, golden DAGs, trajectory distance metrics, LLM-as-a-judge for tool calls.

- Practice: Build a CI pipeline that replays 100 agent trajectories and blocks the PR if the tool-call sequence deviates from the golden path.

- Why: Grading the final text output hides the fact that the agent hallucinated an API call or skipped a safety check to get there.

Stage 11: Graceful Degradation & Fallbacks
- Learn: Circuit breakers, API-to-browser fallbacks, model cascading, cost-kill switches.

- Practice: Build a routing layer where if the primary REST API 404s, the agent seamlessly spawns a headless browser to finish the job.

- Why: The real world is messy. Agents must adapt to broken environments and degraded APIs without crashing the workflow.

Stage 12: Public Agent Teardowns & Portfolios
- Learn: Architecture decision records (ADRs), cost-per-task analysis, latency profiling, public benchmarking.

- Practice: Publish a teardown of your multi-agent swarm, detailing the exact cost, failure rate, state-machine design and eval methodology.

- Why: Top AI labs hire Agent Engineers based on their system design judgment and production reality not their ability to import LangChain.

Bookmark and Repost!

[Puneet Patwari @system_monarch]

During my system design round at Amazon, the interviewer took down my cache.

During my system design round at Walmart, the interviewer killed my primary database.

During my system design round at Atlassian, the interviewer made one downstream service take 10 seconds to respond.

In every round, someone broke my design on purpose.

Because any design works when nothing fails.
Interviewers want to see what yours does when something does.

After sitting through 100s of interviews in the last 13 years, one thing is clear to me:

If you're starting system design prep, focus hard on Failure & Reliability Fundamentals.

Here's a list to help you get started:

[1] Timeouts & Deadlines

- Connection vs request timeouts
- Deadlines that travel across service calls
- What happens when a call has no timeout at all

[2] Retries

- Exponential backoff with jitter
- Retry budgets
- Retry storms and how they stretch outages

[3] Idempotency

- Idempotency keys for write APIs
- At-least-once delivery and duplicate handling
- Safe vs unsafe retries

[4] Isolation

- Circuit breakers
- Bulkheads and separate thread pools
- Blast radius per tenant, region and feature

[5] Overload Protection

- Rate limiting
- Backpressure
- Load shedding and priority queues

[6] Replication & Failover

- Leader election and split brain
- Sync vs async replication
- Failover vs failback

[7] Graceful Degradation

- Serving stale data on purpose
- Fallbacks and feature switches
- Read-only mode

[8] Data Safety

- Write-ahead logs
- Backups and point-in-time recovery
- Restores you've actually tested

[9] Observability

- p50 vs p99 latency
- SLIs, SLOs and error budgets
- Alerts that map to a runbook step

[10] Recovery & Operations

- Rollbacks and feature flags
- Zero-downtime migrations
- Postmortems that change the system

[Mansi 👩‍💻 @MansiCodez]

Recently I interviewed a Senior Engineer.

Strong candidate.

Kafka? ✅
Redis? ✅
Microservices? ✅
Distributed systems? ✅

Then I asked:
"You're building the inventory service for a flash sale.
Only 1 item left in stock."

Two users hit “Buy Now” at the exact same millisecond.
Both requests pass the “is available?” check.
Both create an order.
Both deduct stock.

Now you’ve sold 2 items when only 1 existed.

How do you guarantee you never oversell — even with concurrent requests, client retries, and partial failures?

"Just use a database transaction."

Doesn’t fully solve it if both requests read `stock = 1` before either writes, or if the response is lost and the client retries.

"Check stock before deducting."

What happens when both checks happen at the same time?

"Use Redis DECR and reject if it goes negative."

What happens if Redis is down, or the decrement succeeds but the order is never persisted?

Your turn:
How would you design this inventory reservation API so concurrent buyers and retries never oversell the last item?

There’s a specific distributed-systems concept behind this.

And you’ll encounter it everywhere inventory, tickets, seats, coupons, and limited resources are involved.

[Akash Singh @akkiex007]

Resume & Screening Hack 101

Your resume isn’t just a list of responsibilities. It needs to show what you did, why it mattered and the impact you made. Something I strongly believe in as a recruiter and try to reinforce during my screening calls as well.

An ex-Google, Uber and TikTok recruiter explains why strong candidates get overlooked and how to fix it.

You have ~six seconds to prove you can do the job.

[Puneet Patwari @system_monarch]

Took a mock recently. Backend candidate, strong on microservices.

Then I asked:

"Two services both store the user's email. A user changes it. One service updates, the other doesn't. Six months later they've silently drifted across millions of users. What did the design get wrong?"

Silence.

[Puneet Patwari @system_monarch]

System design round at Zepto for SDE-3:

It's currently 9 PM. 2 lakh people add Maggi to their cart at the same time. The app tells every one of them "In stock."

The dark store has 340 packets.

You can't lock stock for 2 lakh carts. If you don't lock it, 1.6 lakh people find out at checkout.

How do you show stock you can actually promise, without making the cart slow?

[Mansi 👩‍💻 @MansiCodez]

Recently I interviewed a Senior Engineer.

Strong candidate.

Kafka? ✅
Redis? ✅
Microservices? ✅
Distributed systems? ✅

Then I asked:
"Your API gateway enforces a hard limit of 100 requests per minute per user. You have 15 gateway instances behind a load balancer."

A user fires 25 concurrent requests at the start of a new minute (or one times out and the client retries).

Each instance checks its local (or stale) counter:

Count: 8 ✅

All 25 pass.

The user just exceeded the limit.

How do you guarantee a user never exceeds the rate limit — even with concurrent requests across instances, retries, and network failures?

"Just use Redis INCR + EXPIRE."

Doesn’t fully solve the check-then-act race if you read the value first, or if the increment succeeds but the request still needs to be rejected after the fact.

"Sliding window counter in each instance’s memory."

What happens when requests land on different instances at the same time?

"Token bucket in Redis."

What happens if Redis is slow/unavailable — do you fail open and risk abuse, or fail closed and take down the API? And how do you make the token calculation itself safe under concurrency?

Your turn:
How would you design this distributed rate limiter so concurrent requests and retries never allow a user to exceed the configured limit?

There’s a specific distributed-systems concept behind this.

And you’ll encounter it everywhere quotas, throttling, and fair-usage limits are involved.

[Puneet Patwari @system_monarch]

If you learn distributed systems, you will never be unemployed again.

Stage 1: How requests travel
- Learn: DNS, TCP, TLS, load balancers, latency numbers, threads vs processes.
- Practice: build an HTTP server and load test it until it breaks.

Stage 2: Data
- Learn: indexes, transactions, isolation levels, replication, sharding.
- Practice: shard a Postgres table by user_id and write the rebalancing script.

Stage 3: Async
- Learn: queues, Kafka partitions, consumer groups, at-least-once delivery, idempotency.
- Practice: an order pipeline that survives a consumer crash mid-message.

Stage 4: Failure
- Learn: timeouts, retries with jitter, circuit breakers, backpressure.
- Practice: kill random services in your project and make it recover on its own.

Stage 5: Consistency
- Learn: CAP, quorums, Raft, clocks.
- Practice: build a 3-node key-value store with leader election.

Frameworks change every 3 years. These haven't changed in 30.

[Mansi 👩‍💻 @MansiCodez]

CAP theorem gets all the attention.

But CAP only tells you what happens when the network breaks.

What about the other 99.9% of the time?

That’s where PACELC comes in. 🧵