Gabriel Capparelli

Computer scientist with artificial intelligence, machine learning and data science interests trying to understand the universe.

[Alex Xu @alexxubyte]

How To Release A Mobile App

[Nikki Siapno @NikkiSiapno]

6 API Architecture Styles You Should Know.

API architectural styles determine how applications communicate.

The choice of an API architecture can have significant implications on the efficiency, flexibility, and robustness of an application.

So it is very important to choose based on your application's requirements, not just what is often used.

Let’s examine some prominent styles:

REST

A cornerstone in web services, REST leverages HTTP methods for streamlined operations and a consistent interface.

Its stateless nature ensures scalability, while URI-based resource identification provides structure.

REST's strength lies in its simplicity, enabling scalable and maintainable systems.

Learn more about REST here: https://t.co/Ncu4BQRjZn

GraphQL

Whilst REST uses multiple endpoints for each resource and necessitates multiple requests to obtain interconnected data;

GraphQL uses a single endpoint, allowing users to specify exact data needs, and delivers the requested data in a single query.

This approach reduces over-fetching, improving both performance and user experience.

Learn more about GraphQL here: https://t.co/r3GxVYzSo2

SOAP

Once dominant, SOAP remains vital in enterprises for its security and transactional robustness.

It’s XML-based, versatile across various transport protocols, and includes WS-Security for comprehensive message security.

Learn more about SOAP here: https://t.co/pkNfVs6GVg

gRPC

gRPC is efficient in distributed systems, offering bidirectional streaming and multiplexing.

Its use of Protocol Buffers ensures efficient serialization and is suitable for a variety of programming languages and use cases across different domains.

Learn more about gRPC here: https://t.co/nWyX7oo0gk

WebSockets

For applications demanding real-time communication, WebSockets provide a full-duplex communication channel over a single, long-lived connection.

It's popular for applications requiring low latency and continuous data exchange.

Learn more about WebSockets here: https://t.co/S6ODJTbgLE

MQTT

MQTT is a lightweight messaging protocol optimized for high-latency or unreliable networks.

Its pub/sub model ensures efficient data dissemination among a vast array of devices, making it a go-to choice for IoT applications.

Learn more about MQTT here: https://t.co/EnAb3Mlee8

API architectural styles are more than just communication protocols; they are strategic choices that influence the very fabric of application interactions.

There is no best architectural style.

Each offers unique benefits, shaping the functionality and interaction of applications. It's about making the right choice(s) based on your application's requirements.

If you want to learn more about API development, the 2024 State of the API Report has just been released.

Check it out for key trends and insights: https://t.co/MUku9j1wjs

~~
📩 If you like this post, then you'll love our newsletter where we help you level up your engineering and system design skills. Subscribe here: https://t.co/wfCnjeldMh

[Alex Xu @alexxubyte]

Explaining 8 Popular Network Protocols in 1 Diagram.

Network protocols are standard methods of transferring data between two computers in a network.

1. HTTP (HyperText Transfer Protocol)
HTTP is a protocol for fetching resources such as HTML documents. It is the foundation of any data exchange on the Web and it is a client-server protocol.

2. HTTP/3
HTTP/3 is the next major revision of the HTTP. It runs on QUIC, a new transport protocol designed for mobile-heavy internet usage. It relies on UDP instead of TCP, which enables faster web page responsiveness. VR applications demand more bandwidth to render intricate details of a virtual scene and will likely benefit from migrating to HTTP/3 powered by QUIC.

3. HTTPS (HyperText Transfer Protocol Secure)
HTTPS extends HTTP and uses encryption for secure communications.

4. WebSocket
WebSocket is a protocol that provides full-duplex communications over TCP. Clients establish WebSockets to receive real-time updates from the back-end services. Unlike REST, which always “pulls” data, WebSocket enables data to be “pushed”. Applications, like online gaming, stock trading, and messaging apps leverage WebSocket for real-time communication.

5. TCP (Transmission Control Protocol)
TCP is is designed to send packets across the internet and ensure the successful delivery of data and messages over networks. Many application-layer protocols build on top of TCP.

6. UDP (User Datagram Protocol)
UDP sends packets directly to a target computer, without establishing a connection first. UDP is commonly used in time-sensitive communications where occasionally dropping packets is better than waiting. Voice and video traffic are often sent using this protocol.

7. SMTP (Simple Mail Transfer Protocol)
SMTP is a standard protocol to transfer electronic mail from one user to another.

8. FTP (File Transfer Protocol)
FTP is used to transfer computer files between client and server. It has separate connections for the control channel and data channel.

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/uc5M7CdXXC

[Sahn Lam @sahnlam]

A Visual Guide to CI/CD

𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 (𝗖𝗜) is a foundational practice in DevOps where developers frequently merge code changes into the main branch, often multiple times a day. This process is complemented by automated testing to ensure new changes integrate seamlessly with existing code. The primary goal of CI is to find and address bugs quicker, improve software quality, and reduce the time required to validate and release updates

𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 (𝗖𝗗) automates deploying code changes to a production without human intervention. It ensures every change passing all automated tests gets deployed. It accelerates customer feedback by releasing updates more frequently. CD also reduces pressure on developers by eliminating manual release processes.

Some companies rely on 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝗹𝗶𝘃𝗲𝗿𝘆 instead. Continuous Delivery extends CI by automatically preparing code changes for release to production. However, unlike Continuous Deployment, it requires manual approval prior to production deployment. This practice ensures that all changes are automatically built, tested, and ready for release. It allows teams to deploy new changes anytime at the push of a button.

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/kNfv0DV5nH

[Alex Xu @alexxubyte]

The Ultimate Software Architect Knowledge Map

[Alex Xu @alexxubyte]

A Crash Course on Scaling the Data Layer

[Alex Xu @alexxubyte]

Data Pipelines Overview.

[Bytebytego @bytebytego]

Who are the Fantastic Four of System Design?

Scalability, Availability, Reliability, and Performance.

They are the most critical components to crafting successful software systems.

Let’s look at each of them with implementation techniques:

1 - Scalability
Scalability ensures that your application can handle more load without compromising performance.

2 - Availability
Availability makes sure that your application is always ready to serve the users and downtime is minimal.

3 - Reliability
Reliability is about building software that consistently delivers correct results.

4 - Performance
Performance is the ability of a system to carry out its tasks at an expected rate under peak load using available resources.

Over to you: What are the other pillars of system design and strategies you’ve come across?

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/FIzCeaWsZV

[Alex Xu @alexxubyte]

What does ACID mean?

[Kirk Borne @KirkDBorne]

Data Scientist Roadmap

#Infographic by @DataScienceDojo

[Nikki Siapno @NikkiSiapno]

What does an API gateway do?

An API gateway acts as a single entry point for clients, handling request routing, response composition, and protocol translation. It simplifies client interactions with microservices and offers additional features like rate limiting, authentication, monitoring, and more.

To better understand how an API gateway works, let's look at how it processes a request:

1) Initial request handling
Client requests are sent to the API gateway, which acts as the entry point for all incoming API traffic, rather than directly accessing the backend services.

2) Request validation
The API gateway processes and validates the request’s attributes to ensure it’s correctly formatted.

3) Security checks
It then performs checks against allow-lists and deny-lists to filter out unauthorized or harmful requests.

4) Authentication and authorization
The API gateway validates the request, checking for proper authentication (e.g., verifying tokens or credentials) and ensuring the client has the necessary permissions (authorization) to access the requested resources.

5) Rate limiting
Rate limiting rules are enforced; if the request exceeds the allowed limit, it’s rejected.

6) Service discovery and routing
Once passing basic checks, the API gateway then finds the relevant service to route the request to by matching the path.

7) Protocol translation
The API gateway transforms the request into the appropriate protocol and sends it to the service.

8) Response aggregation
If the request requires data from multiple services, the API gateway aggregates the responses. It sends requests to each relevant service, collects the results, and composes them into a single, cohesive response.

9) Response delivery
The gateway sends the processed response back to the client, ensuring it’s delivered in the expected format and within an optimal timeframe.

10) Logging, monitoring, fault handling, and caching
Throughout this process, the API gateway logs each request and response and monitors key metrics such as latency, error rates, and throughput. These logs and metrics help in troubleshooting, scaling, and optimizing the system. It also deals with faults (circuit break), and provides response caching.

An API gateway is a powerful tool that not only simplifies client interactions with microservices but also enhances security, performance, and reliability through comprehensive request processing and monitoring.

~~
Thank you to our partner Postman who keeps our content free to the community.

How can we automate the health and performance of our APIs? Postman Monitors is a great way to do this.

Learn more about it here: https://t.co/Bg20Qcnpqj

[Nikki Siapno @NikkiSiapno]

API Gateway vs Load Balancer — What's the Difference?

An API Gateway acts as a single entry point for clients, handling request routing, composition, and protocol translation. It simplifies client interactions with microservices and offers features like rate limiting, authentication, and monitoring.

Load balancers are concerned with routing client requests across multiple servers to distribute load and prevent bottlenecks. This helps maximize throughput, reduce response time, and optimize resource use.

So what are the key differences?

API gateways focus on request management and microservice communication, while Load Balancers focus on traffic distribution and server load management.

API gateways operate at the application layer (L7), while Load Balancers can operate at both transport (L4) or application (L7) layers.

API gateways offer features like routing, rate limiting, authentication, service discovery, parameter validation, circuit breakers, and more. Load Balancers handle traffic distribution and failover.

An API gateway is Ideal for microservice architectures needing centralized API request management.

A load balancer is essential for applications requiring high availability, distributing traffic across multiple servers.

In summary, API Gateways manage and secure API calls, while Load Balancers ensure efficient traffic distribution.

💭 I’d love to hear your thoughts. What would you like to add? What topics would you like me to cover? 💬

~~
Thank you to our partner Postman who keeps our content free to the community.

Do you have team members who are blocked because they're waiting for you to ship a new API? Unblock them using mock servers.

Learn more here: https://t.co/ejCawV31lP

[Alex Xu @alexxubyte]

10 Key Data Structures We Use Every Day

[Nikki Siapno @NikkiSiapno]

How do we design secure and safe APIs?

The rise in API-related security breaches highlights the necessity for robust API security.

Let’s look at 12 essential tips for improving API security:

🔹HTTPS

Enforcing HTTPS for all API connections is a critical step in securing sensitive data since it ensures data encryption in transit, preventing attacks and interceptions.

🔹 Rate limiting and throttling

Throttling and rate limiting are vital for reducing API abuse and protecting against DDoS attacks as they manage request rates, which keeps our API available for legitimate users.

🔹 Authentication

Authentication is another must-have. Leverage strong authentication mechanisms, such as OAuth, to verify user or system identities.

Learn more about API authentication here: https://t.co/2fCWP0y933

🔹 Authorization

After authentication comes authorization. Follow the least privilege principle to ensure users access only role-relevant data and actions, reducing unauthorized access risks.

🔹 Input validation

Validating API inputs is crucial to safeguard against vulnerabilities like SQL injection and XSS. Whitelisting can also be useful here to ensure only valid data is processed.

🔹 API gateway

Deploy an API Gateway as a security layer, managing authentication, monitoring traffic, and enforcing policies like rate limits.

Learn more about API gateways here: https://t.co/WaErXbCjPS

🔹 Regular security audits

Regular security audits and penetration testing are advisable to identify and fix vulnerabilities, preventing exploitation and maintaining API security.

🔹Dependency management

Regularly updating software dependencies is important to mitigate risks from vulnerabilities in external libraries.

🔹 Logging and monitoring

Investing in comprehensive logging and real-time monitoring is vital for early detection of suspicious activities, enabling swift incident response to mitigate security breaches.

🔹API versioning

To manage changes and updates securely, utilize proper API versioning, which prevents compatibility and security issues.

Learn more about API versioning here: https://t.co/YOyImAdQKR

🔹Data encryption at rest

Encrypting sensitive data at rest is crucial to prevent unauthorized access and comply with data protection regulations.

API security isn’t a nice to have; it’s a must. Following the techniques and best practices I’ve shared above will take you a long way, they are the foundations of building safe and secure APIs.

A great way to automate checking the health, performance, and security of your APIs is Postman Monitors.

Learn more about it here: https://t.co/845BowkB6G

[Alex Xu @alexxubyte]

100X Postgres Scaling at Figma.

With 3 million monthly users, Figma’s user base has increased by 200% since 2018.

As a result, its Postgres database witnessed a whopping 100X growth.

1 - Vertical Scaling and Replication
Figma used a single, large Amazon RDS database.

As a first step, they upgraded to the largest instance available (from r5.12xlarge to r5.24xlarge).

They also created multiple read replicas to scale read traffic and added PgBouncer as a connection pooler to limit the impact of a growing number of connections.

2 - Vertical Partitioning
The next step was vertical partitioning.

They migrated high-traffic tables like “Figma Files” and “Organizations” into their separate databases.

Multiple PgBouncer instances were used to manage the connections for these separate databases.

3 - Horizontal Partitioning
Over time, some tables crossed several terabytes of data and billions of rows.

Postgres Vacuum became an issue and max IOPS exceeded the limits of Amazon RDS at the time.

To solve this, Figma implemented horizontal partitioning by splitting large tables across multiple physical databases.

A new DBProxy service was built to handle routing and query execution.

Over to you - Would you have done something differently?

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/uc5M7CdXXC

[Alex Xu @alexxubyte]

What do Amazon, Netflix, and Uber have in common?

They are extremely good at scaling their system whenever needed.

Here are 8 must-know strategies to scale your system.

1 - Stateless Services
Design stateless services because they don’t rely on server-specific data and are easier to scale.

2 - Horizontal Scaling
Add more servers so that the workload can be shared.

3 - Load Balancing
Use a load balancer to distribute incoming requests evenly across multiple servers.

4 - Auto Scaling
Implement auto-scaling policies to adjust resources based on real-time traffic.

5 - Caching
Use caching to reduce the load on the database and handle repetitive requests at scale.

6 - Database Replication
Replicate data across multiple nodes to scale the read operations while improving redundancy.

7 - Database Sharding
Distribute data across multiple instances to scale the writes as well as reads.

8 - Async Processing
Move time-consuming and resource-intensive tasks to background workers using async processing to scale out new requests.

Over to you: Which other strategies have you used?

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/uc5M7CdXXC

[Alex Xu @alexxubyte]

Best ways to test system functionality.

Testing system functionality is a crucial step in software development and engineering processes.

It ensures that a system or software application performs as expected, meets user requirements, and operates reliably.

Here we delve into the best ways:

1. Unit Testing: Ensures individual code components work correctly in isolation.

2. Integration Testing: Verifies that different system parts function seamlessly together.

3. System Testing: Assesses the entire system's compliance with user requirements and performance.

4. Load Testing: Tests a system's ability to handle high workloads and identifies performance issues.

5. Error Testing: Evaluates how the software handles invalid inputs and error conditions.

6. Test Automation: Automates test case execution for efficiency, repeatability, and error reduction.

Over to you: How do you approach testing system functionality in your software development or engineering projects?

Over to you: what's your company's release process look like?

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/uc5M7CdXXC

[Nikki Siapno @NikkiSiapno]

Load balancing algorithms developers should know.

Effective load balancing is crucial in system design, providing high availability and optimizing resource utilization.

Let's look at how some of the most popular load balancing algorithms work.

🔹 Static Algorithms

1) Round robin

It distributes requests sequentially between servers, ensuring equitable distribution.

Despite its simplicity, it does not account for server load, which might be a drawback when demand changes significantly.

2) Random

Implements a simple way of distributing requests regardless of server load or capability.

This form of load distribution is basic, less precise, and suitable for less complicated applications.

3) IP hash

Uses a consistent hashing method depending on the client's IP address to route requests.

This technique is one way to ensure session persistence by consistently directing requests from the same client to the same server.

4) Weighted round robin

Improves round robin by assigning requests based on server capacity, aiming to better utilize resources by allocating more requests to higher-capacity servers.

This approach seeks to optimize resource use, though actual results can vary with request complexity and system conditions.

🔹 Dynamic Algorithms

5) Least connections

Intelligently sends requests to the server with the fewest active connections, adapting to changing loads.

This technique aims to better reflect current server utilization, potentially leading to more efficient resource consumption.

6) Least response time

Targets performance by routing requests to the server with the quickest response time.

By considering both current server load and performance, this technique supports faster processing, potentially reducing response times for users.

While these are some of the most popular load-balancing strategies, there are other algorithms that also address specific needs and challenges. Choosing the right algorithm is very important to ensuring your application remains scalable, reliable, and efficient.

~~

A big thank you to our partner Postman who keeps our content free to the community.

Did you know there is a VS Code extension for Postman? Check it out: https://t.co/KhKB5OIXGA

[LetsDefend @LetsDefendIO]

TCP/IP Model

[Nikki Siapno @NikkiSiapno]

What is MQTT, how does it work, and why is it becoming increasingly more important?

MQTT was developed in the late 90s by IBM.

It was used in specific industries that required lightweight communication.

But it’s widespread adoption occurred with the rise of the Internet of Things (IoT) landscape.

MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol optimized for high-latency or unreliable networks.

Its minimal bandwidth and resource consumption, along with support for real-time data transmission, among other qualities, have made it ideal for IoT environments.

From simple sensors to complex industrial machines, MQTT facilitates efficient data exchange, even in unreliable network conditions.

MQTT operates on top of TCP/IP.

It's focused on simple, efficient, and reliable message exchange.

Below are the core concepts of MQTT.

🔹 Broker — this central hub ensures that messages reach their intended destinations.

🔹 Client — any MQTT-enabled device capable of publishing and receiving messages.

🔹 Topic — a defined category for publishing and subscribing to messages that makes communication organized.

🔹 Quality of service — categorizes the guarantee level for message delivery based on its urgency.

🔹 Message — the actual data being transmitted between clients and the broker.

🔹 Session — the state maintained between a client and a broker, including subscriptions and undelivered messages.

The MQTT communication flow is quite straightforward:

1) Clients connect to the MQTT broker and subscribe to topics of interest.

2) When a client publishes a message to a topic, the broker receives and processes it.

3) The broker then forwards the message to every client who has subscribed to that topic.

This streamlined process, together with the protocol's lightweight nature, makes MQTT ideal for IoT scenarios with limited connectivity or stringent power constraints.

MQTT ensures efficient data transmission and offers reliable message delivery across various network conditions.

This reliability is complemented by support for multiple QoS levels of diverse requirements.

These features enable real-time monitoring and control and improve the scalability and responsiveness of IoT systems.

With its reliable, scalable, and resource-conscious messaging solution, MQTT forms the backbone of IoT communication.

~~

A big thank you to our partner Postman who keeps our content free to the community.

Wish building APIs was simpler and more organized? Postman Collections does just that. Check it out: https://t.co/gPYExXoZSx