I've been using 𝐀𝐩𝐚𝐜𝐡𝐞 𝐊𝐚𝐟𝐤𝐚 for years now and I absolutely love it. Let me explain the message/event flow in simple terms. Give it a read. 👇
𝐃𝐨 𝐲𝐨𝐮 𝐤𝐧𝐨𝐰 𝐰𝐡𝐚𝐭? 𝐀𝐩𝐚𝐜𝐡𝐞 𝐊𝐚𝐟𝐤𝐚 𝐰𝐚𝐬 𝐛𝐨𝐫𝐧 𝐨𝐮𝐭 𝐨𝐟 𝐚 𝐩𝐫𝐨𝐛𝐥𝐞𝐦. 😉
LinkedIn engineers => faced difficulties in tracking website metrics, activity streams and other operational data.
A team of engineers => led by @jaykreps , @nehanarkhede and @junrao started developing a distributed publish-subscribe messaging system that could handle high-throughput, low-latency data streams.
This system eventually became Apache Kafka.
It was open sourced in early 2011.
The name 'Kafka' was chosen by Jay Kreps.
He named the system after the famous author 'Franz Kafka'. 😊
Kreps was an admirer of Franz Kafka's writing and found the name fitting for a system that dealt with the flow of information.
It's written in Java and Scala.
Later they founded => 'Confluent (@confluentinc)' (a company) in 2014 to provide commercial support and additional tools for Kafka users.
📌 𝐋𝐞𝐭'𝐬 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝 𝐭𝐡𝐞 𝐛𝐚𝐬𝐢𝐜 𝐟𝐥𝐨𝐰.
[1.] Producer sends a message
◾ An application acts as a producer, creating a message with data (payload) and optional key.
◾ The producer connects to a broker in the Kafka cluster and identifies the target topic.
◾ Kafka uses a partitioner to determine which partition within the topic should receive the message. This ensures load balancing and parallel processing.
◾ The message is delivered to the leader replica of the chosen partition.
[2.] Message storage and replication
◾ The leader replica appends the message to its log segment.
◾ The message receives a unique offset, serving as its position within the log.
◾ The leader replicates the message to follower replicas for fault tolerance.
[3.] Consumer fetches messages
◾ An application acts as a consumer, joining a consumer group.
◾ Consumers within the same group share offsets and coordinate consumption.
◾ Each consumer fetches messages from its assigned partitions based on its committed offset.
◾ The consumer receives batches of messages and processes them.
[4.] Acknowledging consumption
◾ Once processing is complete, the consumer commits its new offset.
◾ This tells Kafka which messages have been successfully consumed.
◾ Kafka tracks committed offsets for each consumer in the group.
[*.] Flow continues
◾ Producers continue sending messages and consumers keep fetching and processing them based on their latest offsets.
◾ This cycle ensures ordered delivery and reliable consumption even with failures or restarts.
Remember,
👉 Message flow is asynchronous. Producers don't wait for consumers to process messages.
👉 Consumers can lag behind producers if processing is slow.
👉 Kafka offers mechanisms for handling failures and ensuring at-least-once or exactly-once delivery semantics.
Topics => Partitions =>Log Segments
(Data is actually stored in log segments)
Follow @techNmak
🎉Celebramos esa generación con un #SORTEO de #NintendoSwitch2🎉
Para participar:
➡️Sigue a @xtralife_es
🔁Haz RT a esta publicación
💬Comenta con #NS2xtralife en esta publicación
📅Hasta 14/04/2025
@VivianVvj No te olvides de que los platelmintos hacen esgrima de penes y el ganador le derrama el semen al otro en la piel el cual será absorbido y automáticamente fecundado