Shubham Arora

Software Developer | React.js Python Django Typescript

[Ritesh Roushan @devXritesh]

System Design Series - Day 13/30

Database Failover & High Availability – How to Survive When Your Primary DB Dies

Your primary database will crash one day.
It’s not a question of if, but when.

The real question is: how fast can you recover without losing revenue or trust?

Here’s exactly how production systems handle failover, including the real 2 AM incident we faced.

The Nightmare Scenario

2:47 AM.
Primary database goes down.

All writes fail instantly.
Your app becomes read-only.
Users can’t sign up, post, buy, or update anything.

Without a proper plan, you’re looking at hours of downtime and angry customers.

What is Failover?

Failover is the process of automatically or manually promoting a healthy replica to become the new primary when the old one dies.

Before crash:
Primary (handles all writes) → Replicas (handle reads)

After failover:
Replica 1 becomes the new primary (now handles writes) → remaining replicas follow it.

Manual Failover (Most startups do this at 2 AM)

bash

# On the chosen replica
psql -c "SELECT pg_promote();

# Update your application config to point to the new primary

# Restart your API servers
systemctl restart api-server

Downtime: 10–15 minutes.
It works, but it’s stressful and slow.

Automatic Failover with Patroni (Production Standard)

Patroni + etcd monitors the primary every 10 seconds. If it doesn’t respond for 30 seconds, it automatically promotes the best replica.

Downtime drops to 30–60 seconds.

The Dangerous Split-Brain Problem

What happens during a network partition? Both the old primary and a replica think they are the leader and start accepting writes.

Result: Data divergence and corruption.

Solution: Use distributed consensus (etcd or Consul). Only one node can hold the leader lock at any time.

Our Production Setup

- HAProxy with virtual IP (db. prod. com)
- Patroni cluster running on Primary + Replicas
- etcd cluster (3 nodes) for leader election

Tested scenarios:
- Primary crash → new primary promoted in ~45 seconds
- Network partition → etcd prevents split-brain
- Replica crash → no impact

Result: 99.95% uptime.

Key Takeaways:

- Manual failover = 10–15 min downtime (acceptable for early startups)
- Automatic failover with Patroni = 30–60 sec downtime (required for real SaaS)
- Always use etcd/Consul to prevent split-brain
- Test your failover process every quarter
- Monitor replication lag constantly

Database high availability isn’t glamorous, but it saves your company at 2 AM.

Questions about failover, Patroni, or etcd?

[Shubham Arora @samsyntx]

@vidhhii9 Build your own repido 🚀

[Shubham Arora @samsyntx]

The clearest visual breakdowns of Naive RAG vs Agentic RAG I’ve seen. Naive RAG often fails on complex or ambiguous questions — it retrieves once and generates once, Agentic RAG fixes this with intelligent loops
#RAG #AgenticRAG #LLM

[Avi Chawla @_avichawla]

Naive RAG vs. Agentic RAG, explained visually:

Naive RAG has well-known failure modes:

- It retrieves once and generates once. If the context isn't relevant, it can't search again.
- It treats every query the same. A simple lookup and a complex multi-hop reasoning task go through the identical retrieve-then-generate path.
- There's no verification. The system blindly trusts whatever the retriever returns.

Agentic RAG introduces decision-making loops at each stage to fix this.

Steps 1-2) A query rewriting agent reformulates the raw query. This goes beyond fixing typos, like optimizing it for retrieval by making vague terms precise, decomposing complex queries into sub-queries, and expanding abbreviations.

Steps 3-5) A routing agent decides if the query even needs external context. If not, retrieval is skipped. If yes, a source selector picks the best backend for this specific query type.

Steps 6-7) The source selector routes to the most appropriate source: vector DB for semantic search, web search for real-time info, or structured APIs for tabular data. The retrieved context and rewritten query are combined into the prompt.

Steps 8-9) The LLM generates an initial response.

Steps 10-12) A validation agent (known as Corrective RAG) checks whether the response is relevant, grounded, and complete. If it passes, it's returned. If not, the system loops back to Step 1 with a reformulated query.

This continues for some iterations until we get a satisfactory response or the system admits it cannot answer.

The reason this works is that each agent acts as a quality gate. The rewriter ensures retrieval precision. The router ensures the right source is queried. The validator ensures the output is grounded. Individual failures get caught and corrected rather than silently propagated.

That said, the diagram below shows one of many blueprints of an Agentic RAG system. Production systems increasingly combine Corrective RAG, Adaptive RAG, Self-RAG, and hybrid search (vector + lexical with reranking) based on latency budgets and accuracy requirements.

👉 Over to you: What does your Agentic RAG setup look like?
____
Find me → @_avichawla
Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

[Shubham Arora @samsyntx]

Reached 50 active coding days in 2026 on LeetCode.

Every problem solved is a step toward better problem-solving and stronger fundamentals.

#LeetCode #DSA #Coding #Python

[Vaibhav Agarwal @va_a14]

Settling for 5–6 LPA when you’re good at coding is wasted effort.

With the same skillset,
20–25 LPA is possible in product-based companies.

Don’t hide behind “no shortlists.”
Apply more, push harder.

Respect your potential —
your 20s won’t come back.

[Shubham Arora @samsyntx]

Grateful for the vibrant Diwali celebration with my amazing colleagues at the office!

Wishing everyone a bright, prosperous, and joyful Diwali! 🪔✨

#OfficeVibes #DiwaliCelebration #TeamSpirit #FestiveSeason #OfficeCulture #Diwali2025

[Shubham Arora @samsyntx]

Helpful Python roadmap!

This visual guide is a perfect step-by-step plan for anyone looking to learn Python, from fundamentals to real-world applications like web frameworks, automation, and data science.

Great resource for structured learning and upskilling in programming!

[Zoice AI @zoiceai]

Checking claim status shouldn’t feel like a waiting game.

Zoice AI Voice Agent gives real-time claim updates, approval timelines, and instant notifications — all in a natural, human-like tone.

🎧 Hear the demo 👇

#VoiceAI #InsuranceTech #HealthTech #CustomerExperience

[Zoice AI @zoiceai]

From voice-to-voice agents to AI that sees and understands your screen, the possibilities are endless.

Read how it’s transforming real-time interactions: https://t.co/zDl1JQo93x

#VoiceAI #AIInnovation #Multimodality #Zoice

[Anuprash Gupta @GuptaAnuprash]

Why do top call centers choose Zoice AI?
From no-code call flows to LLM-powered agents, we’re redefining AI calling.
✅ Supports Hindi, English, and regional languages like Marathi and Telugu for authentic conversations.
✅ Easy integration and effortless scaling.
✅ Cost-effective with human-like accuracy.

Learn more → https://t.co/OSsERc2OJR
#ZoiceAI #VoiceAI #AIChallenge #CallCenter #AIBusinessTransformation

[Zoice AI @zoiceai]

Interruptions are human.

Your voice bot should know how to handle them too.
🧠 Cancellable pipelines
🔊 Echo/noise detection
📍 Word-level timestamps

New on the Zoice blog:
👉 https://t.co/kuiq2KEgbC

#VoiceAI #AI #ConversationalAI #Zoice

[Zoice AI @zoiceai]

Our latest blog explores strategies to keep AI chats coherent, fast & cost-efficient.

🔗 https://t.co/s5sMbTdqpG

#ConversationalAI #VoiceAI #LLM #Zoice

[Anuprash Gupta @GuptaAnuprash]

Hey devs! 👋 I'm a Lead Software Developer looking for a super talented Frontend Developer (React/Next.js) to join my team in Pune!

📷What you need:
2.5+ years of experience
React.js, Next.js, JavaScript, TypeScript
GSAP, Framer Motion for cool animations
TailwindCSS, Responsive Design
API integration with tricky logic
Nginx, PM2, AWS
Location: Pune
#ReactJS #NextJS #FrontendDevs #PuneTech #Coding
Let’s build awesome stuff together at The Design Trip! DM me to chat or visit https://t.co/V49mkPq2Jw to learn more about us.

[Shubham Arora @samsyntx]

🎶 Fun Friday Vibes with the Team! 🎉

We ended the week on a musical note at Zucol – full of rhythm, laughter, and good energy.

Grateful for a team that vibes together and bonds stronger every Friday! 💪🎤🎵

Here’s to creating more memories beyond the desks.

#FunFriday

[Shubham Arora @samsyntx]

@cryptofergani Done

[Shubham Arora @samsyntx]

🚀 Excited to Share My Latest Project! 🌟

I'm thrilled to unveil my recently completed SaaS Landing Page, which was built using Next.js, Framer Motion, and Tailwind CSS.

🔗 Live Demo: https://t.co/9pmQa7lylY
💻 Give a Star here: https://t.co/kzHkFX9qR4

[Shubham Arora @samsyntx]

NEXT LEVEL SaaS Landing Page with Next.js, React, TailwindCSS and Framer Motion

https://t.co/jn7k07Ur2Y

#nextjs #reactjs #typescript #tailwindcss #tailwind #framermotion