Your database CPU is at 95%.
P99 latency just crossed 8 seconds.
You have 10 minutes before the payment service starts timing out.
What do you check first?
Your cache-aside pattern has a race condition: two requests miss cache simultaneously, both fetch from DB, both write to cache. One overwrites with stale data.
How will you prevent this?
[Asked at Twitter interview]
Most distributed systems bugs are not networking bugs.
They are two machines disagreeing about what time it is.
One node expires a lease. Another still trusts it. Now both think they are correct.
And that is how simple clock skew can actually become a production incident.
This is why distributed systems tests that use one fake clock are often lying to you.
Real systems do not fail because https://t.co/WzCF6G5fIp() returned the wrong second.
They fail because one node thinks the lease expired, another node thinks it is still valid, and both start acting like the leader.
One machine is 400ms ahead. Another machine pauses for GC. A third node gets NTP corrected suddenly. Now your retry timer, heartbeat timeout, token expiry and leader election are all seeing a different reality.
Your happy-path tests will pass.
Production will still find the split brain for you.
When building simulators, time should be a property of each node, not a global dependency.
Inject clocks per node. Control drift, skew, jumps and pauses. Then run the same scenario again.
That is where distributed systems starts becoming interesting.
In a distributed system, there is no global clock. Instead, each node has its own clock.
If you simulate a distributed system you must simulate multiple clocks
If you don't, your tests are blind to treacherous failures caused by clock drift and clock skew
Top 10 Kubernetes mistakes I keep seeing:
1) No resource requests/limits, then nodes OOM and noisy neighbors win
2) Liveness/readiness probes missing or wrong, so traffic hits half-started pods
3) Deploying without PDBs, then a node drain drops 100% of replicas
4) HPA on CPU only (or none), ignoring latency/queue depth and real bottlenecks
5) Treating pods as pets: writing to local disk instead of PVC/object storage
6) Secrets in env vars or in Git, no rotation, no RBAC boundaries
7) No network policies, everything can talk to everything, lateral movement is easy
8) Ingress/TLS as an afterthought: timeouts, headers, and cert renewals break prod
9) No observability: missing request IDs, no metrics per endpoint, logs not centralized
10) Using default namespaces and wide cluster-admin, then one bad kubectl ruins a day
I have been working on a product for indie creators.
@indiehash is the agentic payment infra on solana.
Upload premium content, share it with your audience and get paid in USDC on solana network.
For the end user it's as simple as probe->pay->unlock premium content from your favourite creators.
You can also host an api endpoint with some ai agent doing complicated work behind the scene and get paid per api call.
No KYC. Instant payment on the solana network.
High-scale database modeling is mostly deciding what you refuse to do at runtime.
1. Start from access patterns: top 5 queries, write paths, latency SLOs. Model for those, not for elegance.
2. Pick your consistency boundaries early: one aggregate per transaction, clear ownership, avoid cross-service joins.
3. Denormalize on purpose: duplicate the 3-5 fields you need for reads; keep a source of truth + async repair.
4. Plan keys like you plan outages: avoid hot partitions, use time-bucketing, add random suffixes when needed.
5. Index budget is real: every index is extra write cost and storage. Track p95 write latency after adding one.
6. Version everything: schema versions, event versions, backfill jobs. Assume online migration with dual writes.
7. Model deletion upfront: TTL vs hard delete, GDPR, tombstones, and how it affects secondary indexes
Your Postgres database is at 95% disk usage. You delete 1 million rows but disk space doesn't decrease.
Why not and how do you reclaim space?
[We faced this last quarter]
You added caching and your memory usage grows until server crashes (OOM). Cache hit rate is 95%.
What's wrong with your cache implementation?
[Topic: Memory Management]
Just talked to a person via my 1:1 call and he said that after 9yrs of experience he is still not staff.
He works at a reputed company but writing APIs in python.
I suggested him to use AI to get better at system design etc and also get better at understanding the fundamentals.
And tbh this is more common than people think.
9 years of experience does not automatically become Staff. A lot of people spend years doing scoped API work, closing tickets, staying reliable, but never intentionally building the skills that higher levels actually demand.
Staff is not just writing more code.
It is system design.
It is driving ambiguity into clarity.
It is taking ownership beyond your Jira board.
It is influencing roadmap, mentoring people, seeing tradeoffs early, and communicating well with managers and teams.
He said he is earning 27 LPA fixed which is not some disaster tbh.
It is a decent base.
One solid switch with better prep can move that to 40 to 45+ LPA very realistically.
But the bigger point is this:
- Stop thinking in "how do I escape next month"
- Start thinking in "what can I become in the next 6 months"
- Use AI daily.
- Study fundamentals properly.
- Practice system design.
- Do Leetcode consistently.
- Talk more with your manager.
- Ask for ownership.
- Get closer to architecture and decision making.
Most careers do not change in one interview.
They change when someone spends 6 months very seriously becoming a different engineer.
Consistency looks boring in the moment.
But in software, boring consistency is what usually creates the big jump.
---
You can book 1:1 call if you want: https://t.co/AiU99AysNC
Practice System Design for free: https://t.co/ZO6RAvFxIc
Read interview Experiences: https://t.co/zsnSuIDd9p
Follow pathways: https://t.co/lna5GL3FIL
And invest and buy the community token on Solana: https://t.co/j9W77GgFmv
---
Thanks for reading. Godspeed.
I finally completed building @0xffdevs over the past few days - a learning platform for devs who want to code their way to financial independence.
pathways for Go, Rust, Solana, system design. interview stories. a free system design canvas. jobs board. all free.
We also launched a token on solana via market on @orynth so the crypto community can be a part of this ambitious, community driven project. Just a way for the community to have skin in what we're building together.
Official CA(can be found on the wesbite too): 7bGT4zV1XfaDrjCGNDA5Tghsy83cJSaiqQhwmBBGory
checkout https://t.co/L0Evx186hf and start learning some niche skills, do checkout the roadmap for what's coming next!
some exciting things in the pipeline!!
Seeing some good growth for a platform launched yesterday