Kyryl | Highload Backend

Senior Java dev. Writing about what actually breaks in prod

[Kyryl | Highload Backend @code_w1th_me]

More Java and backend deep dives: follow @code_w1th_me

[Kyryl | Highload Backend @code_w1th_me]

We deleted 40 million rows from a 200 GB Postgres table at 3 a.m. to kill a disk alert. The volume stayed at 97% full.

[Kyryl | Highload Backend @code_w1th_me]

The lesson: if rows expire, partition by time and drop partitions. DROP is instant and frees the space. DELETE only schedules work.

[Kyryl | Highload Backend @code_w1th_me]

@Vtlabs1 Agents have no instinct for backing off. Every retry is "free" to them, so one slow response turns into a synchronized wave. Same fixes as always: exponential backoff with full jitter, a retry budget per client, and request coalescing so 1,000 identical misses become one rebuild.

[Kyryl | Highload Backend @code_w1th_me]

Your cache TTL is a scheduled outage. One hot key expired at noon, 4,000 requests hit Postgres in the same second, and the connection pool died before Redis refilled anything.

[Kyryl | Highload Backend @code_w1th_me]

@CodeWex Fair point, the lock itself becomes the next bug. Lease token + Lua CAS on publish, plus jitter on the TTL so keys don't all expire together

[Kyryl | Highload Backend @code_w1th_me]

A longer TTL is not the safe choice. It only moves the outage to when nobody is watching. Refresh hot keys before they expire.

More Java and backend deep dives: follow @code_w1th_me

[Kyryl | Highload Backend @code_w1th_me]

Fix 2: lock the rebuild. SET lock:key 1 NX EX 10. One request goes to Postgres, the rest wait or serve the stale value. Trade-off: if the lock holder crashes, everyone waits out the 10 seconds.

[Kyryl | Highload Backend @code_w1th_me]

Do you rely on a direct publish call in your event-driven service, or did you already learn this the hard way?

More Java and backend deep dives: follow @code_w1th_me

[Kyryl | Highload Backend @code_w1th_me]

Order service wrote to Postgres, then published to Kafka in a separate call. One day the publish failed silently. Twelve thousand order-confirmation emails never went out.

[Kyryl | Highload Backend @code_w1th_me]

Honest trade-off: at-least-once delivery, so every consumer must be idempotent. Plus one more table and one more process to run. Worth it once a lost event costs you real customers.

[Kyryl | Highload Backend @code_w1th_me]

Honest trade-off: "up to 200x faster, 400x cheaper" is the vendor's best case. Calibration is a claim until you test it on labeled data. Experimental package. I have not run it in prod.

How many of your agent's LLM calls are really classification?

#AIAgents #LLM

[Kyryl | Highload Backend @code_w1th_me]

Count the model calls in your agent loop.

Most of them do not write anything. They return a boolean.

[Kyryl | Highload Backend @code_w1th_me]

The probability is the real feature. Below 0.8, fall back to the big model or a human.

A threshold you own beats a prompt you keep rewording.