Kuldeep Singh

ManWith2manyThoughts. Interests: Politics, History, Religion, Tech, Human Behaviour. Liberal, Argumentative, Witty(self proclaimed)

[Vishwanath Patil @patilvishi]

System Design - Databases and Data System Desing - Day 83 - Database Scaling: Putting It All Together
We have spent the last 20 days going deep into databases:
Internals → Storage Engines → Pages → Indexes → Query Plans → Transactions → MVCC → Locks → Connections → Replication → Partitioning → Sharding → OLTP/OLAP → NoSQL → Polyglot Persistence

Today we bring it all together.

Database scaling isn't one technique. It's a sequence of decisions based on the actual bottleneck.

1. Start With the Workload
Before changing the database, understand:
Reads
Writes
Query patterns
Data volume
Traffic
Latency requirements
Consistency requirements

Don't start with:
Let's add Redis.

Start with:
What exactly is slow or overloaded?

2. First: Fix the Query
A slow application doesn't always mean the database needs scaling.
Check:
Slow Request
↓
Slow SQL?
↓
EXPLAIN
↓
Bad Query?
↓
Missing / Wrong Index?
↓
Fix Query / Index

Sometimes the biggest improvement comes from fixing one inefficient query.

3. Indexing
If the query is searching for specific rows:
SELECT *
FROM orders
WHERE customer_id = ?;

an appropriate index can dramatically reduce the amount of data the database needs to examine.

Think:
No Index
→ Scan lots of data

Good Index
→ Locate relevant rows

But indexes also have costs:
INSERT / UPDATE
↓
Index Maintenance
↓
Extra Storage

4. Connection Pooling
If the application is constantly creating database connections:
Request
↓
Create Connection
↓
Query
↓
Close

use a connection pool:
Request
↓
Borrow Connection
↓
Query
↓
Return Connection

But remember:
More connections do not automatically mean more throughput.

The database has finite CPU, memory, I/O and concurrency capacity.

5. Caching
If the same data is repeatedly requested:
Application
↓
Cache
↙ ↘
Hit Miss
↓ ↓
Return Database

A cache can reduce database reads significantly.
But now you have another problem:
When does cached data become stale?

That brings us back to:
Cache Invalidation
TTL
Eviction
Consistency

6. Read Replicas
If the database is primarily overloaded by reads:
WRITE
↓
Primary
/ \
↓ ↓
Replica 1 Replica 2
↑ ↑
READ READ
Read replicas can distribute read workload.
But remember:
Primary updated
↓
Replica catches up
↓
Replication lag

So not every read can necessarily go to a replica.

7. Partitioning
When a table becomes extremely large:
Orders
↓
Huge Table
partition it logically:

Orders
├── 2024
├── 2025
└── 2026
This can help with:
Partition pruning
Data management
Retention
Maintenance
Large-table operations

Partitioning is particularly useful when queries naturally filter by the partition key.

8. Sharding
If one database node itself becomes the bottleneck:
One Database
↓
CPU / Storage / I/O limits

sharding can distribute data across nodes:
Application
↓
Shard Router
/ | \
↓ ↓ ↓
DB1 DB2 DB3
But sharding introduces complexity:
Shard Keys
Routing
Rebalancing
Hot Shards
Cross-Shard Queries
Distributed Transactions

So it shouldn't be the first solution you reach for.

9. OLTP vs OLAP
Don't make transactional workloads compete with analytical workloads.

Application
↓
OLTP
↓
Transactions
Then:
OLTP
↓
Data Pipeline
↓
OLAP / Warehouse
↓
Reports / Analytics

The transactional database handles the business operations.
The analytical system handles large-scale analysis.

10. NoSQL
Sometimes the relational model isn't the best fit for a particular workload.
You might choose:
Key-Value
→ Fast key lookups

Document
→ Flexible documents

Wide-Column
→ Massive distributed workloads

Graph
→ Relationship-heavy workloads

The key question remains:
Does the data model match the access pattern?

11. Polyglot Persistence
A production system may use several storage technologies:
Application
│ ┌──────────┼──────────┐
↓ ↓ ↓
PostgreSQL Redis Search
↓ ↓ ↓
Transactions Cache Full-text Search
│
↓
Events
│
├───→ Analytics
│
└���──→ Other Services
Each system has a specific responsibility.

But every additional system increases operational complexity.

12. The Database Scaling Ladder
A useful mental model:
Database Bottleneck
↓
Measure First
↓
Fix Slow Queries
↓
Add Indexes
↓
Connection Pooling
↓
Cache
↓
Read Replicas
↓
Partitioning
↓
Sharding

Not every system needs every step.

13. Different Bottlenecks Need Different Solutions
Slow Query
EXPLAIN
Query Optimization
Indexes

Too Many Reads
Cache
Read Replicas

Huge Table
Partitioning

Too Much Data for One Node
Sharding

Heavy Analytics
OLAP / Data Warehouse

Flexible / Specialized Workload
NoSQL / Specialized Store

14. A Real Production Example
Imagine an e-commerce platform:
100M Customers
1B Orders
10K Requests/sec

A possible evolution:
PostgreSQL
↓
Query Optimization
↓
Indexes
↓
Connection Pool
↓
Redis Cache
↓
Read Replicas
↓
Partition Orders
↓
Shard by Customer / Tenant
↓
Analytics Warehouse

The architecture evolves as the workload evolves.

15. The Most Important Rule
Database scaling should follow the bottleneck.
Measure
↓
Identify Bottleneck
↓
Choose Solution
↓
Implement
↓
Measure Again

Not:
Problem
↓
Add Technology
↓
Problem?
↓
Add Another Technology

That approach can turn a simple database into a very complicated distributed system.

Final Takeaway
After 20 days of database and data-system concepts, the big picture looks like this:

DATABASE
│ ┌────────────┼──────────┐
↓ ↓ ↓
PERFORMANCE SCALE WORKLOAD
│ │ │
Indexes Replicas OLTP
Queries Partitioning OLAP
Connections Sharding NoSQL
Caching Search
│
↓
Polyglot Persistence

Remember the progression:
Optimize → Index → Pool → Cache → Replicate → Partition → Shard

And always:
Measure first. Scale the actual bottleneck.

[Swapna Kumar Panda @swapnakpanda]

The BEST one there for Agentic AI.

Absolutely beginner friendly. Covers every topic.

https://t.co/03jDhYP07L

- Langchain
- Langgraph
- RAG
- Vectorless RAG
- Deep Agents
- Guardrails
- LLM Evals
- LLM Gateways

Give 30 mins to 1 hour everyday. You won't regret at all.

[Residual @res1dualedge]

Andrew Ng just dropped the best 2-hour course on Graph Engineering: from single agent to full automation

9:14 - your first agent
33:11 - loop engineering
1:02:46 - graph engineering
1:30:15 - agents that rewrite themselves
1:49:05 - full graph system

two hours, and it replaces every agent tutorial you bookmarked this year

Prompts → Agents → Loops → Graphs

most people will stop after the first agent and call it automation

he saves the last forty minutes for the graph that runs it without him

same model, same tokens, completely different week

watch it today

the step-by-step guide is below, save it while it is still early ↓

[Amisha Yadav @amisha_explains]

STOP WASTING HOURS TRYING TO FIGURE OUT WHAT TO LEARN IN AI.

I put together one practical roadmap with videos, GitHub repos, guides, books, research papers, and courses.

VIDEOS:
1. LLM Introduction — https://t.co/6UfUcjMnLz
2. LLMs from Scratch — https://t.co/uFE6V6qfDc
3. Agentic AI Overview (Stanford) — https://t.co/zCYeoZr80b
4. Building & Evaluating Agents — https://t.co/90Tsz6VXfs
5. Building Effective Agents — https://t.co/VNjQVwNbTF
6. Building Agents with MCP — https://t.co/EWQBZDaerU
7. Building an Agent from Scratch — https://t.co/IhsxsG4PLF
8. Philo Agents — https://t.co/Ni3EqflKXX

GITHUB REPOS:
1. GenAI Agents — https://t.co/8Rf5V00nc3
2. Microsoft AI Agents for Beginners — https://t.co/wykYg3uID1
3. Prompt Engineering Guide — https://t.co/LJ5I8kvjmH
4. Hands-On Large Language Models — https://t.co/cKYH5bYNpo
5. GenAI Agents — https://t.co/i8UvZEluvU
6. Made with ML — https://t.co/ItPjjmeX7a
7. Hands-On AI Engineering — https://t.co/rb4LyE2yiu
8. Awesome Generative AI Guide — https://t.co/sOvPOy4I4p
9. Designing Machine Learning Systems — https://t.co/NT2i32i1QT
10. Machine Learning for Beginners — https://t.co/1WsbMfbTFq
11. LLM Course — https://t.co/wfyLz8WClB

GUIDES:
1. Google's Agent Whitepaper — https://t.co/YvvSby2Fz1
2. Google's Agent Companion — https://t.co/7Re1Vj2DZ5
3. Building Effective Agents by Anthropic — https://t.co/hbklxrplI9
4. Claude Code Agentic Coding Practices — https://t.co/0JgQUEqJuh
5. OpenAI's Practical Guide to Building Agents — https://t.co/Z3skx6pez3

BOOKS:
1. Understanding Deep Learning — https://t.co/sxxFC5wH5H
2. Building an LLM from Scratch — https://t.co/UZZngciFnh
3. The LLM Engineering Handbook — https://t.co/mAUwbj3570
4. AI Agents: The Definitive Guide — https://t.co/tuLxujMNYt
5. Building Applications with AI Agents — https://t.co/BNpMacbGET
6. AI Agents with MCP — https://t.co/cdsng3fryS
7. AI Engineering — https://t.co/gOh0Xb0Y5X

RESEARCH PAPERS:
1. ReAct — https://t.co/XWig6i6KM9
2. Generative Agents — https://t.co/QjDYMDdTf5
3. Toolformer — https://t.co/A4x9Dt147X
4. Chain-of-Thought Prompting — https://t.co/AJqbHcRTXA

COURSES:
1. Hugging Face Agent Course — https://t.co/0mdR5axTEk
2. MCP with Anthropic — https://t.co/6kUWyZzDbl
3. Building Vector Databases with Pinecone — https://t.co/2yHOFCtdUF
4. Vector Databases: Embeddings to Apps — https://t.co/hTNOtJh0tl
5. Agent Memory — https://t.co/WyiOLyqKi7

No endless searching.
No information overload.
Just resources you can actually use. 🔖

Your future AI skillset will come from building, experimenting, and learning consistently.

Repost so someone else can find this roadmap, and pls consider following @amisha_explains for more content around AI, Beauty, and businesses.

[Rahul @sairahul1]

This is the best site on the internet to learn how LLMs actually work.

Free. Completely.

https://t.co/NBlP0Q9YV3

Bookmark this site.

Then read this ↓

[Julian Goldie SEO @JulianGoldieSEO]

There's a free AI model that beats Claude Opus right now.

It is called GLM 5.3.

You can use it for zero dollars inside ZCode.

New users get 5 million tokens a day to test it.

But most business owners just type random things and get stuck.

You need a real system to make it work.

I use a setup called Agent OS to run my whole business.

It takes one video transcript and turns it into emails, posts, and updates.

It does this all by itself in one single window.

You can get the exact zip file to install my setup today.

Join the AI Profit Boardroom to get the file and start building.

[Umashankar Singh उमाशंकर सिंह @umashankarsingh]

कैपुटो: अगर मैं आपसे पूछूँ कि दुनिया के दो सबसे महान नेता कौन हैं जिनको आप पसंद करते हैं?

ट्रंप : लीडरशिप के मामले में शी

कैपुटो : और दूसरे कौन हैं?

ट्रंप : मोदी… मोदी बहुत अच्छे हैं। उन्होंने बड़े नंबरों का ऐलान किया है। वह लड़ाइयों से दूर रहते हैं, जो सम��दारी है। वह 1.5 बिलियन लोग हैं। भारत असल में सबसे बड़ा है। मोदी एक महान लीडर हैं, और हम उनके साथ बहुत सारा बिज़नेस करते हैं, लेकिन अब हम फेयर बिज़नेस करते हैं।

वे सच में हमें लूटते थे। मैं इसके लिए उन्हें दोष नहीं देता। हमारे पास बेवकूफ पॉलिटिशियन थे जिन्होंने ऐसा होने दिया।

लेकिन अब हम बहुत सारा बिज़नेस करते हैं। वे इससे उतने ख़ुश नहीं हैं। वे पहले बहुत बेहतर करते थे। लेकिन मोदी महान हैं।

Courtesy: @axios

[Anthropic @AnthropicAI]

The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.

The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.

Access to all other Claude models is not affected.

We apologize for this disruption to our customers. We believe this is a misunderstanding and are working to restore access as soon as possible.

Read our full statement: https://t.co/bwn0sximKZ

[Kuldeep Singh @KuldipKumar]

Forward Deployed Engineer - be ready to keep hearing this everywhere in sales pitch in days to come.

[Maharashtra Progress Tracking @abhirammodak]

#Pune
22 lac sqft Commercial Project coming up in Baner
With opening of Missing Link, Western Pune will experience unprecedented growth in coming years

[Neo Kim @systemdesignone]

If you're serious about AI engineering (in 2026), then learn these 13 concepts:

1 How Vector Database Works
→ https://t.co/FVxan8xHH3

2 How RAG Works
→ https://t.co/cGmunPTUlb

3 Design Personal Chat Assistant
→ https://t.co/nNWq3onTnW

4 LLM Concepts - A Deep Dive
→ https://t.co/5lCKxq2g4N

5 How to Design an AI Agent
→ https://t.co/JvnPd9773A

6 What is Reinforcement Learning
→ https://t.co/AVpl9j1oit

7 LLM Evals 101
→ https://t.co/nv3Ol8W53p

8 Context Engineering 101
→ https://t.co/OMkiZhkODL

9 AI Coding Workflow 101
→ https://t.co/paIf9ksIU9

10 Agentic Patterns, Simply Explained
→ https://t.co/8YdBBWvTj1

11 How AI Agents Work
→ https://t.co/tk3zkCjRvg

12 Multi-Agent Architectures, Clearly Explained
→ https://t.co/rS5QQS7Jln

13 How MCP Works
→ https://t.co/wgf8gHnnkn

What else should make this list?

===

👋 PS - Want my System Design Playbook (for Free)?

Join my newsletter with 200K+ software engineers now:
→ https://t.co/ByOFTtOihX

===

💾 Save now & repost to help others learn AI engineering.
👤 Follow @systemdesignone + turn on notifications.

[Santiago @svpino]

30 agents every AI Engineer must build.

This is the most comprehensive and practical book on AI Engineering that I've ever seen.

I can't think of a single use case that they didn't cover here:

1. The autonomous decision-making agent
2. The planning agent
3. The memory-augmented agent
4. The knowledge retrieval agent
5. The document intelligence agent
6. The scientific research agent
7. The tool-using agent
8. The agentic workflow system
9. The data analysis agent
10. The verification and validation agent
11. The general problem solver agent
12. The code generation agent
13. The security-hardened agent
14. The self-improving agent
15. The conversational agent
16. The content creation agent
17. The recommendation agent
18. The vision language agent
19. The audio processing agent
20. The physical world sensing agent
21. The ethical reasoning agent
22. The explainable agent
23. The healthcare intelligence agent
24. The scientific discovery agent
25. The financial advisory agent
26. The legal intelligence agent
27. The education intelligence agent
28. The collective intelligence agent
29. The embodied intelligence agent
30. The domain-transforming integration agent

I also read 50 Algorithms Every Programmer Should Know by Imran. Same vibe.

Here is the Amazon link: https://t.co/buLPqjToiu

[Andrew Ng @AndrewYNg]

How we prompt AI is very different in 2026 than 2022 when ChatGPT came out.

I'm teaching a new course, AI Prompting for Everyone, to help you become an AI power user — whatever your current skill level.

It covers skills that apply across ChatGPT, Gemini, Claude, and other AI tools. How to use deep research mode for well-researched reports on complex questions. How to give AI the right context, including more documents and images than most people realize you can provide. When to ask AI to think hard for several minutes on important decisions like what car to buy, what to study, or what job to take. And how to use AI to generate images, analyze data, and build simple games and websites.

I also cover intuitions about how these models work under the hood, so you know when to trust an answer and when not to.

Along the way, you'll see flying squirrels, a creativity test, some of my old family photos, and fireworks.

Join me at https://t.co/tcQc4iJAJG

[Ayaan 🐧 @twtayaan]

🚨 S3 is no longer just Object Storage.

Yesterday (April 7, 2026), AWS officially launched Amazon S3 Files.

This is the biggest update to S3 in 20 years.

It can:
→ Mount S3 buckets as native file systems
→ Provide sub-millisecond file access
→ Handle POSIX permissions (UID/GID) natively
→ Connect to Lambda, EC2, and EKS directly
→ Eliminate the need for s3fs or data staging

Your AI agents can read/write to S3 like a local disk, while your data team access the same objects via API.

DevOps just got a massive upgrade.

Source: https://t.co/gwGIhDlInU

[Umashankar Singh उमाशंकर सिंह @umashankarsingh]

यह एस्ट्रोफोटोग्राफर दरिया कावा मिर्ज़ा की खींची हुई चांद की असली तस्वीर है। यह 708 GB की है जिसमें चार दिनों में ली गई 81,000 से ज़्यादा तस्वीरों को एक साथ रखा गया है, और इसका रिज़ॉल���यूशन 159.7-मेगापिक्सल है। मिनरल में होने वाले बदलावों (जैसे, टाइटेनियम के लिए नीला, लोहे के लिए लाल) को हाईलाइट करने के लिए रंगों को बेहतर बनाया गया है, जो नंगी आंखों से दिखाई नहीं देते, लेकिन मौजूदा डेटा पर आधारित हैं। इसमें कोई AI इस्तेमाल नहीं किया गया।

[Avinash Singh @AvinashSingh_20]

You are a genius if you know all of this. If not, tutorials are included!

AI Agents – https://t.co/R333r2AuML

Context – https://t.co/eP75VN4kTP

MCP – https://t.co/Mdh345XWZH

Claude Code – https://t.co/N9jrKSf2Ql

APIs – https://t.co/4TqFNE05M7

Cursor – https://t.co/ZLYenj3hqQ

Prompts – https://t.co/uuHkpiAA46

OpenClaw – https://t.co/YB2xawclXR

Free AI agents resources - https://t.co/XlbOBUV4HZ .

[Kuldeep Singh @KuldipKumar]

@anishmoonka the app is very good. Out of curiosity which TTS have you used? Seems doing the job quite well.

[Anish Moonka @anishmoonka]

Started a week ago, not knowing how to write a single line of code

I wanted to read the Bhagavad Gita daily, but couldn't find an app that felt right. So I built one.

Ended with a full iOS app live @10minutegita on the App Store:
→ 239 daily readings of the Bhagavad Gita
→ Original Sanskrit shlokas + transliteration
→ Verse-by-verse translations & commentary
→ Personal daily reflections
→ Streak tracking with calendar heatmap
→ Shareable verse cards with 8 gradient themes
→ Hindi & English bilingual support
→ Light/dark mode, adjustable fonts
→ Completely offline after download

Total cost: $200 Claude Max Subscription + $20 ChatGPT Pro Subscription + $99 Apple Developer fee
Lines of code I wrote: 0

Claude Code wrote everything. I just described what I wanted in plain English (non-technical background). Codex reviewed it. Now it's live on the App Store.

The barrier to building isn't coding anymore. It's just knowing what problem you want solved.

Links & Full Process in 🧵↓

Built with @AnthropicAI's Claude Code

[Maharashtra Progress Tracking @abhirammodak]

#Pune
There are 120+ Software and related open Job openings in NiCE Pune. Pl share with those interested
https://t.co/3ZUyQUBZrX

[Yash @Ai_YashAgrawal]

10 GitHub repositories that will teach you more practical AI engineering than most paid courses:

1. AI Agents for Beginners (Microsoft)
https://t.co/xzfpzAcKXj

2. Awesome Generative AI Guide
https://t.co/K6wPAReKfr

3. Designing Machine Learning Systems (Resources)
https://t.co/TMNhZKf1Lw

4. GenAI Agents
https://t.co/E2qko18eMW

5. Hands-On AI Engineering
https://t.co/x7s8foTkbM

6. Hands-On Large Language Models
https://t.co/AOVmopcNaL

7. LLM Course
https://t.co/nMccKpz8Lr

8. Machine Learning for Beginners (Microsoft)
https://t.co/VXFdlypvq0

9. Made With ML
https://t.co/1T9iUmNvO9

10. Prompt Engineering Guide
https://t.co/6O5MaxIJxa

[Maharashtra Progress Tracking @abhirammodak]

#Pune
Wolter Kluwers hiring for multiple positions in Pune