Syed Basim

Building software that scales. AI, cloud, automation, and product engineering.

[Syed Basim @syedbasim]

A customer got charged twice for one order. The database showed exactly one row. The message queue showed exactly two events.

Nobody made a mistake. No bug, no outage, no crash.

This is the dual-write problem, and most systems with a database and a queue have it. 🧵

[Syed Basim @syedbasim]

A team's p99 latency hit two and a half minutes during a traffic spike.

CPU: 71%. Database: fine. No bug, no outage, no error anywhere in the stack.

The queue was doing exactly what it was designed to do. That's exactly the problem. 🧵

[Syed Basim @syedbasim]

If the transaction fails, neither row exists. Nothing gets published. If it succeeds, both rows exist together, and the relay can retry the queue side as many times as it needs to, since duplicate delivery is now a much easier problem than duplicate state.

Full breakdown, the failure diagram, and when this pattern isn't worth the complexity: https://t.co/KMaycl8AQN

If this kind of stuff is useful, I write about it often, distributed systems failure modes and the non-obvious tradeoffs nobody puts in the docs. Come hang out @syedbasim, more of this every week.

[Syed Basim @syedbasim]

A customer got charged twice for one order. The database showed exactly one row. The message queue showed exactly two events.

Nobody made a mistake. No bug, no outage, no crash.

This is the dual-write problem, and most systems with a database and a queue have it. 🧵

[Syed Basim @syedbasim]

The fix: stop writing to the queue directly. Write a row to an "outbox" table in the same database, in the same transaction as your actual data. A separate process reads that table and relays events to the queue.

Now there's only one transaction that needs to be atomic, and databases are already good at that.

[Syed Basim @syedbasim]

If this kind of stuff is useful, I write about it often, distributed systems failures, cloud infra traps, AI/RAG production issues.

Come hang out @syedbasim, more of this every week.

[Syed Basim @syedbasim]

One malformed message can take down your entire queue.

Not by crashing it once. By crashing it forever. 🧵

[Syed Basim @syedbasim]

Before you ship a queue consumer:

- Set a max redelivery count, not infinite retries
- Route exhausted messages to a dead-letter queue instead of dropping or looping them
- Alert on dead-letter queue depth, not just consumer health
- Decide who actually looks at dead-lettered messages, a DLQ nobody monitors is the same failure with extra steps

[Syed Basim @syedbasim]

More of these coming: distributed systems failure modes, caching and queueing pitfalls, AI and LLM engineering breakdowns.

Follow @syedbasim for the rest of the series.

[Syed Basim @syedbasim]

A team's p99 latency hit two and a half minutes during a traffic spike.

CPU: 71%. Database: fine. No bug, no outage, no error anywhere in the stack.

The queue was doing exactly what it was designed to do. That's exactly the problem. 🧵

[Syed Basim @syedbasim]

Queue depth and CPU graphs can both look fine while this happens, because they measure how much work is waiting, not how long it's been waiting.
Full math, the chart, and the fix: https://t.co/frnZfLexGI

[Syed Basim @syedbasim]

Checklist if you’re running Redis at any real scale:

→ Jitter every TTL set in a batch or at deploy time
→ Coalesce requests on your hottest individual keys
→ Watch for connection-pool spikes lining up with cache-miss spikes. That’s the signature
→ Never assume a cache miss is the cheap path

Full breakdown + the math behind every number: https://t.co/gsCOjSDAyZ

[Syed Basim @syedbasim]

2:14am. PagerDuty: database connections at 98%.

No deploy. No migration. No bot attack.

Just a normal Tuesday, and a database that looked like it got hit by a truck.

Nobody had made a mistake. That’s what made it hard to find. 🧵

[Syed Basim @syedbasim]

Two fixes, different jobs:

TTL jitter: instead of TTL = 300, use TTL = 300 + random(0, 60). Spreads expirations so keys don’t all die in the same second. Costs nothing.

Request coalescing (singleflight): only the first request on a miss runs the query. Everyone else waits on that same answer instead of running their own.