Top 10 mistakes developers make with Redis:
1) Using Redis as the source of truth. Then a flush, eviction, or replica lag becomes data loss.
2) No TTLs on cache keys. Memory grows until eviction turns into random latency spikes.
3) Wrong TTL strategy. Same expiry for everything causes a thundering herd at minute 60.
4) Treating KEYS * as a debug tool in prod. It is O(n) and will block long enough to page someone.
5) Big values. Stuffing 5MB JSON blobs into one key kills bandwidth, CPU, and makes replication slower.
6) Too many small keys. Millions of tiny keys add overhead, fragmentation, and slow persistence.
7) No connection pooling. Creating a TCP connection per request looks fine in staging, melts at 2k RPS.
8) Unsafe retries. Retrying writes without idempotency turns a timeout into duplicate work and bad counters.
9) Using Redis transactions/Lua without time limits. A slow script blocks the server; everything queues.
10) Weak observability. No tracking of used_memory, evicted_keys, hit rate, p95 command latency, and slowlog for hot commands
Ever wondered that postgres stores rows on 8KB pages. So where do huge values go?
Read this nice blog by @PlanetScale
Past ~2KB, TOAST kicks in: it compresses the biggest values, then slices them into a hidden table.
A tip: use LZ4 by default, and don’t SELECT * on big columns you don’t need.
Do give this blog a read, pretty nice. I also used @kalakaroai app to create an explainer video!
https://t.co/jcVybQjG4T
It wasn't one attack but multiple this week!
AI agents are making hacking faster and harder to stop.
Microsoft / ARTEX: Microsoft says a China-linked group (Storm-0940) broke into ConnectWise Cloud Backup and used ARTEX, an open-source toolkit of AI agents, to attack Microsoft 365 accounts. The agents made about 2,900 login attempts a minute against more than 2,250 organizations and got admin access to some in under 3 hours. China denies it.
CrowdStrike: four hacking groups used AI agents in real attacks, and all four got AI models past their safety rules by pretending to be security testers. One agent took a network's master keys in about 20 minutes on its own. At least 15 healthcare organizations were hit.
PHP2Shell: two critical flaws were found in PHP, the code behind much of the web. PHP is already patched. Palo Alto's firewall software is also affected and has no fix yet, though Palo Alto says it hasn't seen anyone exploit it.
Banks: after reports that unapproved AI agents were logging into customer accounts, Morgan Stanley is reportedly blocking outside agents, and JPMorgan and Bank of America are restricting them. Wells Fargo isn't blocking them; from November 24 it treats anything your agent does as approved by you.
In one line: AI isn't creating new security holes; it lets attackers exploit old ones (weak logins, exposed admin pages, persuadable AI models) much faster, around the clock.
Made an explainer video using claude director mode on @kalakaroai - pretty useful for creating explainer videos!!
Unpopular opinion:
If you are buying AI courses from all these course sellers who have been in the business for quite a few years and haven't done any development/deployment, etc., for years, then you are the biggest fool.
Just think about it: You are buying a course to learn AI development from those who have been selling courses for years and haven't had any AI production experience. AI is self learnable and if you need to pay thousands/lakhs for that, I would rather have you buy enough credits and experiment, break things and learn from free YouTube videos.
---
Bhai agar LLM agent integration seekhna hai toh mai free me workshop krduga X pr. Comment krdo. Agar ache free utube tutorials ka idea hai toh wo bhi comment krdo. Let's make this the best twitter thread to learn production grade AI integration for free.
Anyone can say “learn these 10 things in distributed systems” and make a nice checklist.
Heck, I do that too.
But real experience starts showing when someone can explain especially on X - what they actually do when prod is on fire.
Like p95 suddenly jumping from 200ms to 2s.
Do you just stare at the API dashboard?
Or do you correlate that spike with downstream services, check DB latency, queue depth, cache hit rate, CPU/GC, then follow the same request using correlation IDs across services?
Can you explain why p50 looks perfectly healthy while p95/p99 is dying?
Can you tell whether retries are actually making the outage worse?
Can you look at traces and figure out that one dependency is adding 800ms only for a certain request path?
Tbh kind of knowledge rarely comes from interview preparation.
It comes from debugging ugly systems for months/years, breaking things, fixing them and slowly building intuition.
Senior engineer prod scenario: an AI agent hits a paid API and gets stuck in a loop. Cost goes from $50/day to $5k/hour. You also need an audit trail of every call and output for compliance/debugging.
What do you do in the first 5 minutes, and what do you ship after?
I’m looking for: hard kill switch (per org + global), spend/rate limits with budgets, idempotency keys so retries don’t double-bill, rollback strategy if bad outputs already wrote to DB, and observability that ties agent step -> request -> downstream charge -> persisted data.
Also: how do you keep logs append-only and queryable without logging secrets or PII, and how do you prove data correctness after the incident?
For those who don't know ..
1. Strong read-after-write consistency
If you write an object to S3 and the write succeeds, any read immediately after will see that latest object.
Example:
PUT logs/100 = "event A"
→ success
GET logs/100
→ "event A"
You do not have to worry about another reader temporarily seeing an older version after the successful write.
---
2. Conditional writes with If-None-Match:
This basically means:
> "Create this object only if this key does not already exist."
Suppose two servers both try to claim log position 101:
Writer A → PUT logs/101 with If-None-Match: *
Writer B → PUT logs/101 with If-None-Match: *
Only one succeeds.
The other gets a failure because logs/101 now exists.
So you can build something like:
logs/100
logs/101
logs/102
...
Multiple writers can race for the next slot, but only one wins each slot.
Combined with strong consistency, the loser can immediately see who won and retry the next position.
That is the interesting part: S3 itself becomes the coordination primitive, so for certain designs you may not need a separate Raft leader just to decide who gets to append next.
But S3 does not automatically provide an ordered distributed log - you still need to design the append/retry protocol around these guarantees.
How do you verify code when agents write faster than humans can review? @ShadajL, Research Lead for Hydro at AWS, has some answers—on the AGNTCon + MCPCon North America keynote stage. Get the details: https://t.co/gQeggPIelu
Join him October 22-23 in San Jose, CA!
Tech story time !!
One year ago, we had one of those production bugs that was almost impossible to notice.
We had a Kafka consumer processing events from another service.
99.9% of the time everything worked fine.
Consume message -> process it -> update DB -> move on.
But once in a while, processing would fail.
Maybe the downstream service timed out.
Maybe some payload had unexpected data (some edge cases here that weren't handled).
Sometimes the DB connection dropped for a few seconds.
Nothing unusual in distributed systems.
The problem was our retry logic.
We retried the message a few times, and after that... we basically gave up.
No DLQ.
The consumer kept moving forward, so from the outside the system looked completely healthy.
CPU fine.
Kafka lag looked fine.
Error rate almost zero.
No major alerts.
But that tiny percentage of messages were effectively lost from the business workflow.
And because it happened so rarely, nobody noticed immediately.
Weeks later someone found a mismatch between two systems.
We started tracing IDs manually and eventually found events that had entered Kafka but never completed processing.
That was the scary part actually.
The system was not "down" per se.
But it was silently wrong.
We fixed it by introducing a proper retry + DLQ flow.
Normal topic
-> consumer
-> temporary failure
-> retry topic with backoff
-> retry again
-> after max attempts -> DLQ
But just adding a DLQ was not enough.
We also added the original payload, error reason, retry count, timestamp and correlation ID so someone could actually debug the failed message.
Then alerts for DLQ growth.
Then a small replay tool so after fixing the underlying bug, we could safely send failed events back through the pipeline.
We also had another bug which led us to make consumers idempotent because replaying messages without idempotency can create an entirely new problem.
Duplicate payments, duplicate rows, duplicate notifications... whatever your consumer happens to do.
That incident really taught me something important about event-driven systems:
Message loss does not always look like message loss.
Most of the times Kafka is healthy, consumers are healthy, dashboards are green...
But your business data is slowly becoming inconsistent.
A DLQ is not just a queue where "bad messages go".
It is a safety net for failures you did not predict.
And in distributed systems, there will always be failures you did not predict.
Your database CPU is at 95%.
P99 latency just crossed 8 seconds.
You have 10 minutes before the payment service starts timing out.
What do you check first?
Your cache-aside pattern has a race condition: two requests miss cache simultaneously, both fetch from DB, both write to cache. One overwrites with stale data.
How will you prevent this?
[Asked at Twitter interview]
Most distributed systems bugs are not networking bugs.
They are two machines disagreeing about what time it is.
One node expires a lease. Another still trusts it. Now both think they are correct.
And that is how simple clock skew can actually become a production incident.
This is why distributed systems tests that use one fake clock are often lying to you.
Real systems do not fail because https://t.co/WzCF6G5fIp() returned the wrong second.
They fail because one node thinks the lease expired, another node thinks it is still valid, and both start acting like the leader.
One machine is 400ms ahead. Another machine pauses for GC. A third node gets NTP corrected suddenly. Now your retry timer, heartbeat timeout, token expiry and leader election are all seeing a different reality.
Your happy-path tests will pass.
Production will still find the split brain for you.
When building simulators, time should be a property of each node, not a global dependency.
Inject clocks per node. Control drift, skew, jumps and pauses. Then run the same scenario again.
That is where distributed systems starts becoming interesting.
In a distributed system, there is no global clock. Instead, each node has its own clock.
If you simulate a distributed system you must simulate multiple clocks
If you don't, your tests are blind to treacherous failures caused by clock drift and clock skew
Top 10 Kubernetes mistakes I keep seeing:
1) No resource requests/limits, then nodes OOM and noisy neighbors win
2) Liveness/readiness probes missing or wrong, so traffic hits half-started pods
3) Deploying without PDBs, then a node drain drops 100% of replicas
4) HPA on CPU only (or none), ignoring latency/queue depth and real bottlenecks
5) Treating pods as pets: writing to local disk instead of PVC/object storage
6) Secrets in env vars or in Git, no rotation, no RBAC boundaries
7) No network policies, everything can talk to everything, lateral movement is easy
8) Ingress/TLS as an afterthought: timeouts, headers, and cert renewals break prod
9) No observability: missing request IDs, no metrics per endpoint, logs not centralized
10) Using default namespaces and wide cluster-admin, then one bad kubectl ruins a day
I have been working on a product for indie creators.
@indiehash is the agentic payment infra on solana.
Upload premium content, share it with your audience and get paid in USDC on solana network.
For the end user it's as simple as probe->pay->unlock premium content from your favourite creators.
You can also host an api endpoint with some ai agent doing complicated work behind the scene and get paid per api call.
No KYC. Instant payment on the solana network.