John Dutchover

[Dr Milan Milanović @milan_milanovic]

𝗛𝗼𝘄 𝗔𝗽𝗮𝗰𝗵𝗲 𝗞𝗮𝗳𝗸𝗮 𝘄𝗼𝗿𝗸𝘀?

If you're building distributed systems or working with real-time data, you need to understand Kafka. It's a streaming platform that handles millions of events per second without breaking.

Kafka moves data between applications using a distributed pub-sub model. Producers write messages to topics. Consumers read about those topics. Brokers store and serve the data.

Here's what happens under the hood when a message flows through Kafka:

𝟭. 𝗣𝗿𝗼𝗱𝘂𝗰𝗲𝗿 𝘀𝗲𝗻𝗱𝘀 𝗮 𝗺𝗲𝘀𝘀𝗮𝗴𝗲

An application writes a message to a specific topic. The producer assigns the message to a partition based on a key or round-robin distribution. Messages are batched together before sending to maximize throughput.

𝟮. 𝗕𝗿𝗼𝗸𝗲𝗿 𝘀𝘁𝗼𝗿𝗲𝘀 𝘁𝗵𝗲 𝗺𝗲𝘀𝘀𝗮𝗴𝗲

The broker receives the message and writes it to disk immediately. This makes it durable. Each partition is replicated across multiple brokers, so if one broker fails, another takes over. The broker assigns an offset to each message, which acts as a unique identifier within the partition.

𝟯. 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿 𝗿𝗲𝗮𝗱𝘀 𝘁𝗵𝗲 𝗺𝗲𝘀𝘀𝗮𝗴𝗲

Consumers poll the broker for new messages from their assigned partitions. In a consumer group, each consumer owns specific partitions. This prevents two consumers from processing the same message. The consumer tracks its position using offsets, so it can restart without losing its place.

𝟰. 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻 𝗺𝗼𝗱𝗲𝗹 𝗲𝗻𝗮𝗯𝗹𝗲𝘀 𝘀𝗰𝗮𝗹𝗲

Topics are split into partitions, which can be distributed across different brokers. This enables parallel processing. Multiple consumers can read from different partitions simultaneously. Need more capacity? Add more partitions and more consumers.

𝗪𝗵𝘆 𝗞𝗮𝗳𝗸𝗮 𝗶𝘀 𝗳𝗮𝘀𝘁?

Kafka writes everything to disk using sequential writes, which are faster than random memory access. It keeps messages for days or weeks instead of deleting after delivery. This lets consumers reprocess data or new consumers catch up from the beginning.

The system handles multiple producers and consumers independently. Each operates at its own pace without coordination. This is why Kafka can sustain millions of messages per second across distributed systems.

Some 𝗰𝗼𝗺𝗺𝗼𝗻 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀 are:
🔹 Event processing systems that rely on real-time data (IoT)
🔹 Stream processing (real-time analytics)
🔹 Metrics collection & logging

[Real Python @realpython]

The Best Programmers I Know #python https://t.co/8eOeHCkHaU

[John Dutchover @johndutchover]

Dear Duolingo: How does a country’s name change? https://t.co/x2XvxHAvJt

[John Dutchover @johndutchover]

Dotfiles are the underrated tool most users are missing out on https://t.co/QECXR7ejXp

[John Dutchover @johndutchover]

Take a look my Digital Collectibles https://t.co/71PRWJBmLS #nycfc

[John Dutchover @johndutchover]

Serene Skies Over Montenegro: Captivating Drone Voyage https://t.co/gZroVab322

[John Dutchover @johndutchover]

Saquon Barkley surprises students at Milton Hershey School, making the dream of an Eagles Autism Community member come true  https://t.co/NBDxfm4J7C

[The Athletic @TheAthletic]

That championship feeling 🏆

@UConnWBB | #MarchMadness

[George Mason Baseball @MasonBaseball]

⬆️3️⃣ The Patriots set a new NCAA record of runs scored in an inning (23).

The previous record was 21 runs scored by Wichita State (1984) and Penn State (1983)

[John Dutchover @johndutchover]

#Cheer4USA Sent good wishes to atletes competing at the Special Olympics World Winter Games Turin 2025!
https://t.co/dBkgNsJlRL

[John Dutchover @johndutchover]

#Cheer4USA https://t.co/dBkgNsJlRL

[John Dutchover @johndutchover]

100 week streak with https://t.co/oFyCB0QU3N and I'm still feeling the difference. Curious? Discover your music prescription here: https://t.co/PE0dNmJGPQ

[John Dutchover @johndutchover]

Elon Beyond the Bricks – Episode #1 – @sea76n https://t.co/oqEJkSlRKJ via @YouTube Looking forward to visiting @elonuniversity

[John Dutchover @johndutchover]

@BlueAngels Shock NFL Crowd with Unexpected Flyover: What Really Happene... https://t.co/4J5ZOZ6YqG via @YouTube 1997 rare #nfl in-game flyover

[John Dutchover @johndutchover]

@steren @RomanStanek https://t.co/yfVRrga715 by @prefix_dev is a must-try.

[John Dutchover @johndutchover]

Gameday Roster Moves: Eagles activate DE Brandon Graham from Injured Reserve https://t.co/OzDIGG3Ap2 #FlyEaglesFly

[John Dutchover @johndutchover]

Eagles 2024 Super Bowl HYPE VIDEO https://t.co/9sJct2zoUE via @YouTube #FlyEaglesFly

[John Dutchover @johndutchover]

⁦@Eagles⁩ #FlyEaglesFly https://t.co/qPWIoSAthK

[John Dutchover @johndutchover]

Quote of the day https://t.co/wRpiM3GOoq #GoBirds #FlyEaglesFly