Aayush Mittal

Building · Backend eng @Agoda · Helping mid → senior engineers level up

[Aayush Mittal @_aayuuush_]

Most interview prep stops at LeetCode, but senior engineering interviews don't.

When companies evaluate senior developers, they probe how you think at scale:
- How would you debug a cascading failure in production at 2 AM?
- What trade-offs would you make designing a system that needs to handle 10 million requests per day? 
- How do you review a pull request that's technically correct but architecturally wrong?

https://t.co/7nPLYfg5d0 is built for exactly this gap. It's a curated collection of real-world interview questions that senior developers actually face, spanning high-level design, low-level design, production debugging scenarios, code review challenges, and the deep technical discussions that separate engineers who build high-performance systems from those who only talk about them. Stop grinding toy problems. Start thinking like a senior.

[Aayush Mittal @_aayuuush_]

@i_mika_el That all comes down to the reviewer.

[Aayush Mittal @_aayuuush_]

@neerajjj6785 Wannabes

[Aayush Mittal @_aayuuush_]

Idempotency is the one that actually trips people up live, not in theory. Everyone can define it, few catch that the "check cache, then call PSP" step usually isn't atomic. Saw this exact bug in a payment middleware review: two concurrent retries both missed the cache and both charged. The fix is a lock around the check-and-write, not just a key.
Built a whole set of these production-shaped bugs to practice on : https://t.co/Vsvsrj1u5X

[Aayush Mittal @_aayuuush_]

Idempotency is the one that actually trips people up live, not in theory. Everyone can define it , few catch that the "check cache, then call PSP" step usually isn't atomic. Saw this exact bug in a payment middleware review: two concurrent retries both missed the cache and both charged. The fix is a lock around the check-and-write, not just a key.
Built a whole set of these production-shaped bugs to practice on : https://t.co/Vsvsrj1u5X

[Aayush Mittal @_aayuuush_]

@simonw The harder test is whether it catches the failure modes that only show up under concurrent load.

[Aayush Mittal @_aayuuush_]

One gap worth adding: if the location beacon retries after a dropped ack, you need idempotent writes downstream, or the same ping double-counts as two location updates. Same failure mode as any at-least-once delivery system, just harder to notice because nobody's balance goes negative when it happens.

[Aayush Mittal @_aayuuush_]

Saw this happen with an LLM on a caching bug last month. It suggested raising a TTL to fix a stale cache. Technically correct. It would've served customers a wrong price for hours. The model had the right instinct and the wrong constraint, because nobody told it what "correct" meant for that field.

[Aayush Mittal @_aayuuush_]

A payout server dies at 2 AM. The on-call fix: spin up a second server so it never happens again.

Two weeks later, customers get paid twice.

That's the trap with distributed scheduling, the naive fix for "job didn't run" almost always creates "job ran twice." I wrote up how systems like Airflow, Quartz, and Temporal actually solve this, because it comes down to three things happening at once:

- Every job fires : nothing silently skipped
- Each job fires exactly once : no duplicate runs
- Each job finishes : even if the machine executing it dies mid-way

The core techniques:
- Claim-based dispatch with Postgres' FOR UPDATE SKIP LOCKED, so multiple machines can grab jobs without colliding
- Fencing tokens, so a machine that freezes and wakes up late can't overwrite a job another machine already picked up
- Durable replay, crash mid-workflow, and a new machine resumes from a logged history instead of re-running everything from scratch, including a 24-hour sleep

You can't get "exactly once" for free over a flaky network. You build "effectively once" one safely-repeatable step at a time.

Full writeup: https://t.co/m8QoxAJc04

#DistributedSystems #SoftwareEngineering #SystemDesign #BackendEngineering #Payments

[Aayush Mittal @_aayuuush_]

You set a 2-second timeout on the dependency. You wrapped it in a circuit breaker. On paper, a slow dependency costs you 2 seconds, then the breaker takes over.
Then it doesn't fail. It hangs. And your 2 seconds turns out to be fiction.

The textbook kit is right: per-call timeout, circuit breaker, retry budget. The words are right. The wiring is wrong, and not just in your service.

A timeout is really three: connect, read, and a total deadline. Your 2 seconds was the connect timeout, and connecting takes milliseconds, so it never trips. The hang lives in the read phase, waiting on bytes that never arrive. That's the timeout nobody set, so it falls through to the client default.

The breaker won't catch it either. A hung call hasn't failed yet. It's in flight, holding a thread and a connection, reporting nothing, and a breaker only trips on calls that come back to fail.

Fix all that and you've solved it for one service. That's usually where the write-up stops. But the same bug sits in every service you own. The generated clients ship with the read timeout unset, and nobody owns the deadline once it crosses the call graph. That's not a bug in a service. It's a contract nobody wrote.

The contract is deadline propagation. A request arrives with a budget. Every hop spends some and passes down what's left, as an absolute deadline, not a fresh timeout that resets at each call. gRPC bakes this in. Most REST stacks don't, so a 3-second budget at the edge becomes 30 seconds, four hops deep.

And there's a trap in the fix. A deadline frees the caller, not the socket. You cancel the future, and the thread stays blocked in a native read until the real timeout fires. It counts only if the transport enforces it: a client that aborts the request and reclaims the connection.

A timeout you set is a wish. A timeout your platform enforces is a contract. Most of what you've configured is the first kind.

[Aayush Mittal @_aayuuush_]

@americosmjr @Sanika_0305 Can you explain this? How?

[Aayush Mittal @_aayuuush_]

@arpit_bhayani Real progress comes from first principles, not clever workarounds that skip understanding.

[Aayush Mittal @_aayuuush_]

Failure is not an edge case.

[Aayush Mittal @_aayuuush_]

@mchulet @X https://t.co/bmllwQNdO1

[Aayush Mittal @_aayuuush_]

Your wallet service has a bug. Two concurrent refunds both read balance = ₹500, both pass the check, both debit. Balance goes negative.

The UPDATE was atomic. The check-then-act wasn't.

Fix everyone reaches for: SELECT FOR UPDATE. Works. But every wallet operation now serializes. Under load, your payments service becomes a queue.

The fix that actually holds: stop storing balance as a value you mutate.

A ledger stores events. Every credit, debit, refund is an INSERT. Balance is SELECT SUM(amount). No row to lock, no race. Concurrent debits become concurrent inserts.

The catch: SUM across years of transactions gets slow. So you add snapshots, sum only the delta. Now you have two things to keep in sync. That's the new problem you bought.

Ledger doesn't eliminate complexity. It moves it somewhere honest.

What does your wallet model look like?

Free graded exercise on exactly this:
https://t.co/GTQCxYx4VU

[Aayush Mittal @_aayuuush_]

@erornotfoundme @X Lets connect, currently building.

https://t.co/QdXEL19UU9

[Aayush Mittal @_aayuuush_]

@dk119819 @X Lets connect, I am building the real backend engineer interview loop.
https://t.co/QdXEL19UU9

[Aayush Mittal @_aayuuush_]

@faizionweb3 working on https://t.co/QdXEL19UU9

[Aayush Mittal @_aayuuush_]

Origin traffic jumped ~10x overnight. Nobody had touched the CDN.

Our product page had been sitting at a 90% cache hit rate. It fell to almost nothing. The last CDN change was weeks old.

The page is stitched from three subgraphs. Catalog owns the title and images, stable for hours. Pricing owns the price, stable for minutes. Inventory had just shipped a "3 left" badge. It's a live count, so they set TTL 0. That's the right call; you never cache live stock.

Here's the part that bit us. The gateway doesn't cache fields individually. It computes one policy for the whole response and takes the minimum TTL across every field. One field at zero drops the entire response to zero. The badge made the whole page uncacheable. Title, images, and all.

Worth knowing: this isn't a federation quirk. A single Apollo server does the same thing. One resolver at TTL 0 zeroes the response. Federation just meant the field that did it shipped in another team's diff, one nobody on the page side ever saw.

Raising the badge's TTL isn't the fix. That serves a stale count, which is worse than an uncached page. If the volatile field can't share a lifetime with the stable shell, stop putting them in the same cached response: serve the shell as its own cacheable response and fetch the live count as a separate, uncached call. Shell caches for hours, badge stays live.

The check I added: CI fails the build if a field pulls an operation's composed TTL below its budget. A field zeroing the page's TTL should break a build, not show up on next month's CDN bill.

Caching felt like something each service tuned on its own. For a composed response it isn't. It's one number, set by the field that cares about it least.