Levani Kokhreidze

Principal Engineer at @Wise. My heart belongs to distributed streams. JVM and data engineering enthusiast. Focusing on stream processing platforms.

[Levani Kokhreidze @l4ik8e]

📰 @apachekafka 3.2 comes with rack awareness support for @kafkastreams. In this post, I'm trying to summarise the current implementation, how it works and why you should consider adopting it if you run stateful @kafkastream workloads in the #cloud. #kafka https://t.co/ajHcxERvEr

[Levani Kokhreidze @l4ik8e]

@gunnarmorling Example:
On client side -> bootstrap.servers: kafka:9093
On broker side -> advertised.listeners: PLAINTEXT://localhost:57491, BROKER://kafka:9093
https://t.co/ZrfawH4mK9 for the BROKER listener should also be set as PLAINTEXT.

[Levani Kokhreidze @l4ik8e]

@gunnarmorling If you give a broker a hostname, your client can use that hostname + port in the bootstrap.servers. That way it should connect based on BROKER listener, I think.

[Levani Kokhreidze @l4ik8e]

@gunnarmorling @criccomini @ApacheFlink Even with the application mode, you still have to manage separate JobManager and TaskManager processes. Kafka Streams, imho, has an edge in this case. Looking forward to Flink 2.0 and improvements in this area https://t.co/Ahajiyywur

[Levani Kokhreidze @l4ik8e]

🎉 Made my first small contribution to the @ApacheIceberg project. It would be possible to set the table-level description in the AWS Glue catalog in the new versions.
https://t.co/ky2AiKgzpM

[Levani Kokhreidze @l4ik8e]

@antonmry @DolanRyanne @Mr_mitchellh I will definitely share more after some initial experiments.

[Levani Kokhreidze @l4ik8e]

@antonmry @DolanRyanne @Mr_mitchellh Technically compaction is just an upsert and Iceberg does support it. Ordering is also solvable as it depends how you construct the replay from Iceberg table. Thanks for the Apache Paimon pointer.

[Levani Kokhreidze @l4ik8e]

@antonmry @DolanRyanne @Mr_mitchellh We have the same use case. I'm playing with an idea of implementing tiered storage API backed by Apache Iceberg. Having the best of two worlds.

[Levani Kokhreidze @l4ik8e]

@BleeGoldman @kafkastreams Great post, thanks! One other big problem for large stateful Kafka Streams applications with EOS is wiping out the local state when encountering the exception. I have big hopes for https://t.co/XrQXqN797z

[Levani Kokhreidze @l4ik8e]

@DolanRyanne @gunnarmorling @AlexNoonan6 100%! Operational gaps are one of the main reasons why we avoided CDC on some critical flows and trying to move away from it where we can.

[Levani Kokhreidze @l4ik8e]

@kozlovski @AdityaSharma_a We did still get some performance benefits, especially on client side. Client side performance was much better once we delegated the TLS handling to envoy+spire.

[Levani Kokhreidze @l4ik8e]

@SerCeMan @forked_franz @SanneGrinovero Amazing talk, thanks! Out of interest, what's the take on sealed classes and pattern matching with project Amber?

[Levani Kokhreidze @l4ik8e]

Tomorrow I'll be presenting at #kafkasummit London! Join the session about how we built a stream processing platform at @Wise to power our customers' instant money movement experience. We will discuss pretty interesting details about Kafka Streams and a bit about Flink ;)

[Levani Kokhreidze @l4ik8e]

@DolanRyanne @antonmry

[Levani Kokhreidze @l4ik8e]

My talk "Streaming Infrastructure at Wise" has been accepted for @apachekafka Summit London 2023! I'm excited to share @Wise's story with the whole community!

[Levani Kokhreidze @l4ik8e]

@Carnage4Life Computers had similar arguments against them in the early days.

[Levani Kokhreidze @l4ik8e]

@kafkastreams @MatthiasJSax 1. Some sort of savepointing. Rolling back the topology state to a consistent snapshot.
2. Not specifically Kafka Streams related, but tiered storage support for compacted topics would be really useful for offloading changelog topic data to remote storage.

[Levani Kokhreidze @l4ik8e]

@jessetanderson Same here. But possibility of that is smaller compared to ordinary pod restart which is considered as a normal operation.

[Levani Kokhreidze @l4ik8e]

@jessetanderson Not necessary small. It depends how you run it. With dynamic environment Kafka Streams is better. Flinks JobManager restart triggering a full recovery is a major drawback. So I think both Flink and Kafka Streams have their pros and cons and it depends what's the use case

[Levani Kokhreidze @l4ik8e]

@jessetanderson In addition, the bigger issue of Flink is that whenever JobManager is restarted, even with HA mode, it triggers full recovery from S3. Which is pretty disruptive if you run JobManager in dynamic environment.
https://t.co/D1HW4MgOY0

[Levani Kokhreidze @l4ik8e]

@jessetanderson That is true. But the same is true for Flink. But I've seen recovering from S3 for instance is faster. But nevertheless, a need for that can be minimised with Kafka Streams with standby tasks. There's no such option in Flink.