Day 1 : I've been using Kafka for almost a year, but, to be honest, I think I still don't have a deep knowledge about Kafka. Hopefully by starting this challenge, I can build something useful with it. @apachekafka https://t.co/wMtRnDtdUw #100DaysOfCode#ApacheKafka
I should have made a GitHub repo to documented all the confluent-cli command that I came across on this 100 days of code challenge.. just for my personal library of commands and future use..
may be doing it tomorrow, idk..
Day 1 : I've been using Kafka for almost a year, but, to be honest, I think I still don't have a deep knowledge about Kafka. Hopefully by starting this challenge, I can build something useful with it. @apachekafka https://t.co/wMtRnDtdUw #100DaysOfCode#ApacheKafka
Processing the ksql query using only 1 partition solves the problem that I had, but, what if I want the throughput not bottlenecked by just using a single partition? I think this one will be answered later on this journey.
I'm curious about the keying and the repartition, because of my project that required to use repartitioning/rekeying using ksql to get different kind of processed topic. Somehow, because I processed the topic with grouping, the result become messed up.
Quoted from https://t.co/osEJktS7lU
This blog was also mentioned about the disadvantages of big number of partitions, like more open file handles, increase unavailability, increase end-to-end latency, and more memory used in the client (prod/cons).
The good way to calculate the right num of partitions is "You measure the throughout that you can achieve on a single partition for production (call it p) and consumption (call it c). Letโs say your target throughput is t. Then you need to have at least max(t/p, t/c) partitions"