Top 10 Database Scaling Techniques You Should Know:
1. 𝐈𝐧𝐝𝐞𝐱𝐢𝐧𝐠: Create indexes on frequently queried columns to speed up data retrieval.
2. 𝐕𝐞𝐫𝐭𝐢𝐜𝐚𝐥 𝐒𝐜𝐚𝐥𝐢𝐧𝐠: Upgrade your database server by adding more CPU, RAM, or storage to handle increased load.
3. 𝐂𝐚𝐜𝐡𝐢𝐧𝐠: Store frequently accessed data in-memory (e.g., Redis) to reduce database load and improve response time.
4. 𝐒𝐡𝐚𝐫𝐝𝐢𝐧𝐠: Distribute data across multiple servers by splitting the database into smaller, independent shards, allowing for horizontal scaling and improved performance.
5. 𝐑𝐞𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧: Create multiple copies (replicas) of the database across different servers, enabling read queries to be distributed across replicas and improving availability.
6. 𝐐𝐮𝐞𝐫𝐲 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧: Fine-tune SQL queries, eliminate expensive operations, and leverage indexes effectively to improve execution speed and reduce database load.
7. 𝐂𝐨𝐧𝐧𝐞𝐜𝐭𝐢𝐨𝐧 𝐏𝐨𝐨𝐥𝐢𝐧𝐠: Reduce the overhead of opening/closing database connections by reusing existing ones, improving performance under heavy traffic.
8. 𝐕𝐞𝐫𝐭𝐢𝐜𝐚𝐥 𝐏𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧𝐢𝐧𝐠: Split large tables into smaller, more manageable parts (partitions), each containing a subset of the columns from the original table.
9. 𝐃𝐞𝐧𝐨𝐫𝐦𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧: Store data in a redundant but structured format to minimize complex joins and speed up read-heavy workloads.
10. 𝐌𝐚𝐭𝐞𝐫𝐢𝐚𝐥𝐢𝐳𝐞𝐝 𝐕𝐢𝐞𝐰𝐬: Pre-compute and store results of complex queries as separate tables to avoid expensive recalculation, reducing database load and improving response times.
♻️ Repost to help others in your network
Main Docker Concepts on a Single Diagram 🔽
If you're trying to get started with containers, I've got a cheat sheet for you that covers the essential Docker concepts.
Share, like, and follow me for more Linux, networking, containers, and Kubernetes materials 😎
9 Software Design Principles Every Developer Should Know:
1. 𝐃𝐑𝐘 (𝐃𝐨𝐧'𝐭 𝐑𝐞𝐩𝐞𝐚𝐭 𝐘𝐨𝐮𝐫𝐬𝐞𝐥𝐟): Avoid duplicating code. Keep logic centralized to make your codebase easier to maintain.
2. 𝐊𝐈𝐒𝐒 (𝐊𝐞𝐞𝐩 𝐈𝐭 𝐒𝐢𝐦𝐩𝐥𝐞, 𝐒𝐭𝐮𝐩𝐢𝐝): Aim for simplicity in your solutions. Avoid overengineering or adding unnecessary layers.
3. 𝐘𝐀𝐆𝐍𝐈 (𝐘𝐨𝐮 𝐀𝐫𝐞𝐧’𝐭 𝐆𝐨𝐧𝐧𝐚 𝐍𝐞𝐞𝐝 𝐈𝐭): Only build what you need today. Don't waste time on hypothetical features that may never be used.
4. LOD (𝐋𝐚𝐰 𝐨𝐟 𝐃𝐞𝐦𝐞𝐭𝐞𝐫): Talk only to your immediate neighbors. Don’t chain too many calls.
𝐒𝐎𝐋𝐈𝐃 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞𝐬:
5. 𝐒𝐑𝐏 (𝐒𝐢𝐧𝐠𝐥𝐞 𝐑𝐞𝐬𝐩𝐨𝐧𝐬𝐢𝐛𝐢𝐥𝐢𝐭𝐲 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞): A class should have one responsibility only. Keep each component focused and cohesive.
6. 𝐎𝐂𝐏 (𝐎𝐩𝐞𝐧/𝐂𝐥𝐨𝐬𝐞𝐝 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞): Code should be open for extension but closed for modification. Add new features without altering existing logic.
7. 𝐋𝐒𝐏 (𝐋𝐢𝐬𝐤𝐨𝐯 𝐒𝐮𝐛𝐬𝐭𝐢𝐭𝐮𝐭𝐢𝐨𝐧 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞): Subclasses must be usable in place of their parent classes without breaking functionality.
8. 𝐈𝐒𝐏 (𝐈𝐧𝐭𝐞𝐫𝐟𝐚𝐜𝐞 𝐒𝐞𝐠𝐫𝐞𝐠𝐚𝐭𝐢𝐨𝐧 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞): Design small, focused interfaces instead of large, general ones.
9. 𝐃𝐈𝐏 (𝐃𝐞𝐩𝐞𝐧𝐝𝐞𝐧𝐜𝐲 𝐈𝐧𝐯𝐞𝐫𝐬𝐢𝐨𝐧 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞): High-level modules should not depend on low-level modules. Both should depend on abstractions.
♻️ Repost to help others in your network.
5. Two Minute Papers
Two Minute Papers is the right choice if you don't have much time to learn about AI.
They discuss the latest AI and machine learning research projects based on papers - and explain in 2 minutes.
🔗 https://t.co/YnX4R0ih9b
𝗛𝗼𝘄 𝘁𝗼 𝗟𝗲𝗮𝗿𝗻 𝗦𝘆𝘀𝘁𝗲𝗺 𝗗𝗲𝘀𝗶𝗴𝗻?
If you're working as a software developer and you want to move towards a software architect role, or you're preparing for coding interviews for more senior roles, systems design is an important skill you need to have. Without knowing it, you will have a hard time designing new software systems and understanding existing ones.
System design refers to 𝘁𝗵𝗲 𝗽𝗿𝗼𝗰𝗲𝘀𝘀 𝗼𝗳 𝗱𝗲𝗳𝗶𝗻𝗶𝗻𝗴 𝗮 𝘀𝘆𝘀𝘁𝗲𝗺'𝘀 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀. Architecture, modules, components, interfaces, and data are a few examples of these aspects. The process of defining, creating, and designing systems that suit the particular objectives and requirements of an organization is what you need to understand when it comes to system design.
To understand system design, you will need to know the following:
𝟭. 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝘃𝘀 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆
𝟮. 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 𝘃𝘀 𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗮𝗻𝗱 𝗔𝘃𝗮𝗶𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝘃𝘀 𝗖𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆 (𝘄𝗶𝘁𝗵 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀)
𝟯. 𝗔𝘃𝗮𝗶𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀
𝟰. 𝗕𝗮𝗰𝗸𝗴𝗿𝗼𝘂𝗻𝗱 𝗷𝗼𝗯𝘀
𝟱. 𝗗𝗼𝗺𝗮𝗶𝗻 𝗡𝗮𝗺𝗲 𝗦𝘆𝘀𝘁𝗲𝗺𝘀
𝟲. 𝗖𝗗𝗡𝘀
𝟳. 𝗟𝗼𝗮𝗱 𝗕𝗮𝗹𝗮𝗻𝗰𝗲𝗿𝘀
𝟴. 𝗖𝗮𝗰𝗵𝗶𝗻𝗴
𝟵. 𝗔𝘀𝘆𝗻𝗰𝗵𝗿𝗼𝗻𝗶𝘀𝗺
𝟭𝟬. 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀
𝟭𝟭. 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗮𝗻𝘁𝗶𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀
𝟭𝟮. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴
𝟭𝟯. 𝗖𝗹𝗼𝘂𝗱 𝗱𝗲𝘀𝗶𝗴𝗻 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀
To learn this, here are some 𝗯𝗼𝗼𝗸𝘀 where you can start learning:
𝟭. 𝗦𝘆𝘀𝘁𝗲𝗺 𝗗𝗲𝘀𝗶𝗴𝗻 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄
One of the best books in this field, where you will learn how to design systems such as web crawlers or even YouTube.
Link: https://t.co/FS1xScwM1a
𝟮. 𝗗𝗲𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝗗𝗮𝘁𝗮-𝗜𝗻𝘀𝗲𝗻𝘀𝗶𝘁𝗶𝘃𝗲 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀
In this book, the author writes about different technologies used to store and process data. By reading it, you will gain insight into different algorithms used in the database world.
Link: https://t.co/FeB1UiM2qp
𝟯. 𝗛𝗲𝗮𝗱 𝗙𝗶𝗿𝘀𝘁 𝗗𝗲𝘀𝗶𝗴𝗻 𝗣𝗮𝘁𝘁𝗲𝗿𝗻𝘀
A great book on OO design patterns, written in a simple style with examples in Java.
Link: https://t.co/SiqEzg1Fug
𝟰. 𝗖𝗿𝗮𝗰𝗸𝗶𝗻𝗴 𝘁𝗵𝗲 𝗖𝗼𝗱𝗶𝗻𝗴 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄
It is a general-purpose coding interview book where the author shares her insights into programming interviews at big tech companies like Microsoft and Google. It covers all basic topics like algorithms, data structures, SQL, etc.
Link: https://t.co/j295Zvs8KW
𝟱. 𝗙𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹𝘀 𝗼𝗳 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲
As System Design is related to Software Architecture, this book will give you an introduction to how to architect software systems. This book examines architecture patterns, components, soft skills, modernity, architecture as engineering, and many more.
Link: https://t.co/nhNA8xXvDr
Check the System Design Roadmap in the image (by roadmap .sh).
#softwareengineering
I made a new video on: 15 𝐏𝐚𝐭𝐭𝐞𝐫𝐧𝐬 𝐭𝐡𝐚𝐭 𝐦𝐚𝐝𝐞 𝐋𝐞𝐞𝐭𝐂𝐨𝐝𝐞 𝐞𝐚𝐬𝐢𝐞𝐫 𝐟𝐨𝐫 𝐦𝐞.
For each pattern, I talk about when to use it, walk through an example and share Leetcode problems you can practice to learn it better.
Link to the video: https://t.co/Kfz4FlSB5Q
Subscribe for more such videos.
The Big Archive for System Design - 2023 Edition (PDF) is available now. And it's completely FREE.
The PDF contains 𝐚𝐥𝐥 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐩𝐨𝐬𝐭𝐬 published in 2023.
What’s included in the PDF?
🔹 Netflix's Tech Stack
🔹 Top 5 common ways to improve API performance
🔹 Linux boot Process Explained
🔹 CAP, BASE, SOLID, KISS, What do these acronyms mean?
🔹 Explaining JSON Web Token (JWT) to a 10 year old Kid
🔹 Explaining 8 Popular Network Protocols in 1 Diagram
🔹 Top 5 Software Architectural Patterns
🔹 OAuth 2.0 Flows
🔹 What does API gateway do?
🔹 Linux file system explained
🔹 18 Key Design Patterns Every Developer Should Know
🔹 Best ways to test system functionality
🔹 Top 6 Load Balancing Algorithms
🔹 Top 12 Tips for API Security
🔹 𝐀𝐧𝐝 100+ 𝐦𝐨𝐫𝐞
–
Like, follow and subscribe to our newsletter to receive the PDF download link: https://t.co/6j06DUIbVn
Improving API Performance with Database Connection Pooling
The diagram below shows 5 common API optimization techniques. Today, I’ll focus on number 5, connection pooling. It is not as trivial to implement as it sounds for some languages.
When fulfilling API requests, we often need to query the database. Opening a new connection for every API call adds overhead. Connection pooling helps avoid this penalty by reusing connections.
How Connection Pooling Works
1. For each API server, establish a pool of database connections at startup.
2. Workers share these connections, requesting one when needed and returning it after.
Challenges for Some Languages
However, setting up connection pooling can be more complex for languages like PHP, Python and Node.js. These languages handle scale by having multiple processes, each serving a subset of requests.
- In these languages, database connections get tied to each process.
- Connections can't be efficiently shared across processes. Each process needs its own pool, wasting resources.
In contrast, languages like Java and Go use threads within a single process to handle requests. Connections are bound at the application level, allowing easy sharing of a centralized pool.
Connection Pooling Solution
Tools like PgBouncer work around these challenges by proxying connections at the application level.
PgBouncer creates a centralized pool that all processes can access. No matter which process makes the request, PgBouncer efficiently handles the pooling.
At high scale, all languages can benefit from running PgBouncer on a dedicated server. Now the connection pool is shared over the network for all API servers. This conserves finite database connections.
Connection pooling improves efficiency, but its implementation complexity varies across languages.
Have you run into database connection limit issues as your API traffic grew? How did you troubleshoot and fix that?
–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/kNfv0DVDdf
Here are 300 hours of curated courses focused on Machine Learning Engineering.
There are 15 courses. From beginner to advanced. From Google. For free.
Some of the topics they cover:
• Fundamentals of Machine Learning
• Feature Engineering
• Production Machine Learning Systems
• Computer Vision and Natural Language
• Recommendation Systems
• MLOps
• TensorFlow, Google Cloud, VertexAI
The courses are well structured. They aren't just links to YouTube videos. You have to join the course, and they have an interface that takes you through every module.
This is good content. And it's free.
https://t.co/eqTjRT6BZF
There are over 1,000 engineering blogs. Here are my top 9 favorites:
- Netflix TeachBlog
- Uber Blog
- Cloudflare Blog
- Engineering at Meta
- LinkedIn Engineering
- Discord Blog
- AWS Architecture
- Slack Engineering
- Stripe Blog
Over to you - What are some of your favorite engineering blogs?
--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/uc5M7CdXXC
Before I met my husband, I was:
Stressed, had little fun, worked all the time, & thought that was the only way to win.
After him: We've been around the world, AND had multiple 8-figure businesses.
13 lessons from him on his birthday (so he can change your life a little too):