Wathon Aung

Thrilled to make #data and #software_engineering

[Wathon Aung @Wathon_Aung]

Just signed up for connpass. https://t.co/TbGzpDhnfs #connpass

[GitHub Projects Community @GithubProjects]

Run a full virtual desktop inside a Docker container, accessible via WebRTC, right from your browser.

[Wathon Aung @Wathon_Aung]

@kyiphyukhant There are a lot vacant in MM 😅

[˗ˏˋ Jesse Hanley ˎˊ˗ @jessethanley]

coding on a friday with my last 2 brain cells

[Wathon Aung @Wathon_Aung]

@TechByMarkandey Notes

[Towards Data Science @TDataScience]

Using the combined power of CLI, Streamlit, and Ollama, Stefan Pietrusky presents an end-to-end workflow that allows us to "talk" to PDF files without using proprietary models. https://t.co/BMWoYBovaL

[sysxplore @sysxplore]

Linux grep command cheatsheet

[Google Cloud Tech @GoogleCloudTech]

AWS Glue federated datasets are available in BigQuery, which connects #BigQuery and an external data source at the dataset level. Federated dataset tables are populated from the tables in AWS Glue, and updates are reflected in BigQuery.

Check it out → https://t.co/CxxiR8TB8V

[Wathon Aung @Wathon_Aung]

Do you remember when you joined X? I do! #MyXAnniversary

[Sahn Lam @sahnlam]

Scaling Database: When and How to Shard

Database sharding refers to splitting data across multiple database servers and is commonly used for scaling. However, sharding introduces major operational and infrastructure complexity that should be 𝗮𝘃𝗼𝗶𝗱𝗲𝗱 𝘂𝗻𝗹𝗲𝘀𝘀 𝗮𝗯𝘀𝗼𝗹𝘂𝘁𝗲𝗹𝘆 𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆.

𝗔𝗹𝘁𝗲𝗿𝗻𝗮𝘁𝗶𝘃𝗲 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵𝗲𝘀

Vertical Scaling: Use more powerful single database servers with more CPUs, memory, storage and I/O bandwidth. Much simpler to manage than sharding.

SQL Optimization: Tune SQL queries and database schema to maximize performance on a single server using proper indexes, efficient SQL, etc.

Caching: Use in-memory caches like Redis to reduce database load by serving common queries from the cache instead of hitting the database every time.

Read Replicas + Load Balancer: Add horizontal read scaleability without full complexity of sharding. Directs reads across replicas.

These optimization approaches should be exhausted before considering sharding.

𝗦𝗵𝗮𝗿𝗱𝗶𝗻𝗴 𝗠𝗲𝘁𝗵𝗼𝗱𝘀

There are two high-level approaches:

Vertical Sharding: Split database into columnar tables or sections vs rows. For example, having one table for names and another table for emails.

Horizontal Sharding: Split database into row partitions distributed evenly across multiple servers. Methods include range based, directory based, and hash based sharding.

𝗪𝗵𝗲𝗻 𝘀𝗵𝗮𝗿𝗱𝗶𝗻𝗴, 𝘂𝘀𝗲 𝘁𝗵𝗲 𝘀𝗶𝗺𝗽𝗹𝗲𝘀𝘁 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 that meets requirements to minimize complexity. Seek to avoid sharding until necessary despite the scaling benefits. The infrastructure and operational overheads often outweigh gains.

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/kNfv0DVDdf

[Wathon Aung @Wathon_Aung]

@TechByMarkandey X

[Elon Musk @elonmusk]

If you’re reading this post, it’s because our servers are working

[X @X]

we know why you’re all here rn

[Bytebytego @bytebytego]

Why is Kafka fast?

There are many design decisions that contributed to Kafka’s performance. In this post, we’ll focus on two. We think these two carried the most weight.

1️. The first one is Kafka’s reliance on Sequential I/O.

2️. The second design choice that gives Kafka its performance advantage is its focus on efficiency: zero copy principle.

The diagram below illustrates how the data is transmitted between producer and consumer, and what zero-copy means.

🔹Step 1.1 - 1.3: Producer writes data to the disk

🔹Step 2: Consumer reads data without zero-copy
2.1: The data is loaded from disk to OS cache
2.2 The data is copied from OS cache to Kafka application
2.3 Kafka application copies the data into the socket buffer
2.4 The data is copied from socket buffer to network card
2.5 The network card sends data out to the consumer

🔹Step 3: Consumer reads data with zero-copy
3.1: The data is loaded from disk to OS cache
3.2 OS cache directly copies the data to the network card via sendfile() command
3.3 The network card sends data out to the consumer

Zero copy is a shortcut to save multiple data copies between the application context and kernel context.

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/FIzCeaWsZV

[Sahn Lam @sahnlam]

Why is Redis Fast?

Redis is fast for in-memory data storage. Its speed has made it popular for caching, session storage, and real-time analytics. But what gives Redis its blazing speed? Let's explore:

RAM-Based Storage

At its core, Redis primarily uses main memory for storing data. Accessing data from RAM is orders of magnitude faster than from disk. This is a major reason for Redis's speed.

However, RAM is volatile. To persist data, Redis supports disk snapshots and append-only file logging. This combines RAM's performance with disk's permanence.

There is a tradeoff though - recovery from disk is slow. If a Redis instance fails, restarting from disk can be slow compared to failing over to a replica instance fully in memory. So while Redis offers durability via disk, it comes at the cost of slower recovery.

A better solution is Redis replication. With a synchronized replica kept in memory, failover is instant with no rehydration. This maintains speed and near-instant recovery.

IO Multiplexing & Single-threaded Read/Write

Redis uses an event-driven, single-threaded model for its core operations. A main event loop handles all client requests and data operations sequentially. This single-threaded execution avoids context switching and synchronization overhead typical of multi-threaded systems.

Redis uses non-blocking I/O to handle multiple connections asynchronously. This allows it to support many client connections with very low overhead,

Redis does leverage threading in certain areas:

- Background tasks like taking snapshots.
- I/O threads are used for certain operations.
- Modules can use threads.
- Since Redis 6.0, it supports multi-threaded I/O for network communication, improving performance on multi-core systems.

Redis also uses pipelining for high throughput. Clients pipeline commands without waiting for each response. This allows more efficient network round trips, boosting overall performance.

Efficient Data Structures

Redis supports various optimized data structures, from linked lists, zip lists, and skip lists to sets, hashes, and sorted sets, among others. Each is carefully designed for specific use cases for quick and efficient data access.

Over to you: With Redis now supporting some multi-threading, how should we configure it to fully utilize all the CPU cores of modern hardware when deploying in production?

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/kNfv0DVDdf

[Kirk Borne @KirkDBorne]

EDA (Exploratory Data Analysis) for Data-Driven Insights

Source for #infographic: https://t.co/aenyXdQ3WK
———
#BigData #BI #Analytics #DataScience #MachineLearning #Statistics #DataAnalytics #DataLiteracy #DataViz #VisualAnalytics #DataStorytelling #DataScientists

[Simon @simonholdorf]

✨The Technology Behind Netflix Unveiled!✨

[Python Space @python_spaces]

Harvard University is offering Statistics Course for FREE!

Statistics 110: Probability is one of the best probability courses on the internet and it's FREE.

Course link 🧵👇

[Dr Milan Milanović @milan_milanovic]

𝗪𝗵𝗶𝗰𝗵 𝗰𝗹𝗼𝘂𝗱 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝗿 𝘀𝗵𝗼𝘂𝗹𝗱 𝗯𝗲 𝘂𝘀𝗲𝗱 𝘄𝗵𝗲𝗻 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗯𝗶𝗴 𝗱𝗮𝘁𝗮 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻?

The diagram below illustrates the detailed comparison of AWS, Google Cloud, and Microsoft Azure, created by Satish Chandra Gupta.

They all use similar parts of solutions:

1. 𝗗𝗮𝘁𝗮 𝗶𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 of structured or unstructured data.

2. 𝗥𝗮𝘄 𝗱𝗮𝘁𝗮 𝘀𝘁𝗼𝗿𝗮𝗴𝗲.

3. 𝗗𝗮𝘁𝗮 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴, including filtering, transformation, normalization, etc.

4. 𝗗𝗮𝘁𝗮 𝘄𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲, including key-value storage, relational database, OLAP database, etc.

5. 𝗣𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗹𝗮𝘆𝗲𝗿, with dashboards and real-time notifications.

Hyperscalers use different names for the same service type, e.g., AWS "lambda" is Azure "function."

Which services did you use, and for what kind of solutions? What are your experiences with these three providers?

The full text is in the comments.

--
If you like my posts, please follow me @milan_milanovic and hit the 🔔 on my profile to get a notification for all my new posts.

Learn something new every day 🚀!

#aws #technology #cloudomputing #data #softwareengineering

[javinpaul @javinpaul]

10 System design topics you must prepare for interviews
1. Caching
2. Sharding
3. load-balancing
4. replication
5. fault-tolerance
6. high-availability
7. Performance
8. Scalability
9. replication
10. indexing
Learn more on Design Guru - https://t.co/VuZLWnCdMw