Barak Merimovich

Software developer, Cloud guy, Gamer.

[A. Jesse Jiryu Davis @jessejiryudavis]

I've meant to write about this for 7 years: how to actually use MongoDB causal consistency https://t.co/VZJLSQ945r

[Barak Merimovich @barakmer]

@tomerg Peopleware. Relatively old, but I found it insightful. Highly recommended if you are just starting out as a manager.

[Programming Wisdom @CodeWisdom]

"The most amazing achievement of the computer software industry is its continuing cancellation of the steady and staggering gains made by the computer hardware industry." — Henry Petroski (1942-2023)

[Barak Merimovich @barakmer]

Interesting network latency numbers from the big three cloud vendors. Some of these surprised me a little, especially around cross-AZ traffic. Definitely worth reading.

[Aleksey Charapko @AlekseyCharapko]

How predictable is the latency of communication in the cloud? If we have two processes on some VMs talking to each other, what round-trip latency can you expect? I may have some hints for you, thanks to the work my class and I did last semester: https://t.co/mbc5ZQJMnY🧵👇

[pikuma.com @pikuma]

"Here's a nickel, Kid. Go buy yourself a real computer."

[Gergely Orosz @GergelyOrosz]

Started reading The Art of Science and Engineering by Richard W Hamming and it is fascinating from the get go.

“In science, if you know what you are doing, you should not be doing it.

In engineering, if you do not know what you are doing, you should not be doing it.”

[Mitchell Hashimoto @mitchellh]

I'm convinced everyone who actually likes JS/TS and the whole ecosystem is just suffering from Stockholm syndrome paired with being forced to use it. We're all just stuck with this reality. 😵‍💫 Layers and layers of madness, pure madness.

[Barak Merimovich @barakmer]

Finally! Official Queue Support for #Kafka. Can we get this yesterday? The KIP is quite informative: https://t.co/VWNxpHmDAs

[Stanislav Kozlovski @kozlovski]

Queues for Kafka is the hottest new feature being discussed right now!

"KIP-932: Queues for Kafka" was announced just 7 days ago.

But what is it?

First - let’s define a queue.

A traditional queue system is one where either:
🔹- many consumers read from the same queue (pub-sub)
🔹- one specific consumer reads from one specific producer (point to point)

The messages are typically stored until they’re consumed once - the queues have a maximum depth.

Kafka has never supported traditional queuing like this.

One of its strengths has precisely been the decoupling between producer and consumer.

A bad consumer has a close-to-zero effect on a producer. (unless it causes Kafka to read from disk and exhaust IOs)

One pain point with this approach is that consumer groups are coupled with the number of partitions in a topic.

If you have a topic with 10 partitions, you cannot scale beyond 10 consumers.

So, people usually over-partition.
But that's very unintuitive for a uniform workload. 🙅‍♂️

If all your messages are independent work items with no logical grouping, a single queue consumed by a pool of applications is the intuitive solution.

So, KIP-932 proposes a solution with the following benefits:
✅ - the ability for many consumers to read from the same partition
✅ - individual records acknowledgments
✅ - still keep producers and consumers decoupled
✅ - no maximum queue depth
✅ - messages are still retained - so you have the ability to replay

And the following limitations:
🔴 - Ordering is NOT guaranteed. Out-of-order delivery is possible within a partition.
🔴 - No exactly once - it is at least once.
🔴 - Maximum processing delta. A consumer cannot read more than N messages ahead of the slowest one.

How does it do it?

Shared Consumer Groups. ✨

Each broker will be a shared group coordinator for the data partitions it is leading and manage the sharing of reads.

It will keep a sliding window of start <-> end offset for each pair of partition and group.

The records available for consumption will only be those within that offset range, essentially adding a maximum lag between the slowest and fastest consumer. 🛑✋

Consumers from the same shared group can read from the same partition by exclusively reserving a few records (offset range) in the partition via a time-limited acquisition lock.

A consumer can then ack/release/reject the message(s):

🥇 ack - acknowledges successful processing and moves the shared group’s offset progress

🥈 release - unsuccessful processing - retry. Release the record for another delivery.

🥉 reject - unsuccessful processing - abort. Blacklists the record, making it unavailable for another delivery.

☠️ To avoid poison messages - a delivery count is kept per message. When it goes over a maximum retry limit, the message is rejected.

That’s it! Quite the out of the box proposal. 💡

In one sentence, this is a usability feature for un-ordered consumption by an arbitrary number of consumers.

Note that nothing here is set in stone. This proposal is still pending discussion.

I was the first person to reply to the discussion on the mailing list (humblebrag).

The proposal may look quite different by the time it gets in.

What do you think about it?

[Dave Farley @davefarley77]

Microservices are maybe not what you think they are, so here's a #Thread to describe them...

1/14

[🌩️ NVIDIA GeForce NOW @NVIDIAGFN]

Moments ago, we announced a partnership with Microsoft to bring Xbox PC games to GeForce NOW. 🤝

Read more about it here, and tune into GFN Thursday updates for more details ➡️ https://t.co/725z5u3r6v

[Tim Ottinger @tottinge]

Seriously, you have no idea how must waste, blockage, delay, communication failure, risk, and interpersonal conflict is CAUSED by working solo on separate partial, dependent tasks.

If you start to see it, it will break your heart. It is so much unnecessary work and trouble.

[Tomer levy @TomerLevy]

US scientists boost clean power hopes with fusion energy breakthrough | Financial Times https://t.co/87RTEJjZ5R

[Barak Merimovich @barakmer]

#Squaredle
https://t.co/lKHtIlkrZl:
890/890 words (+504 bonus words)
📖 In the top 37% by bonus words

VICTORY!!!

[Bryson 🦄 @brysonbort]

No requirement in the history of humankind was fully understood on Day 1.

[Barak Merimovich @barakmer]

10,000 e-mails and slacks on the wall,
10,000 e-mails and slack.
Take one down, archive and frown...
&lt;Sigh&gt;

[Dave Farley @davefarley77]

Here are a few thoughts on things I wish I knew when I was first a tech lead...

a thread

1/7

[Dave Farley @davefarley77]

If you can improve just one thing in your software development, make it getting faster feedback.

[Asaf Mesika @asafmesika]

@JavaSummitIL 2022 is the first community-content-driven Java conference in Israel. The CFP is closing in 13 days!
https://t.co/BZtihoGfLI

[𝙳𝚘𝚝𝚊𝚗 𝙷𝚘𝚛𝚘𝚟𝚒𝚝𝚜 ✈️WeAreDevelopers🇪🇺 @horovits]

what's Synthetic Monitoring? How does it work? When should you use it? How is it different from RUM (real user monitoring)?
An excellent post by @BargoDoron with everything you need to know #syntheticmonitoring
https://t.co/6CSOm3NLbh
@logzio #monitoring #observability #APM

[Micah Learns @micahlerner]

This week's paper is "Druid: A Real-time Analytical Data Store": https://t.co/2spBD8xOj3

Druid (@druidio) is an open-source database designed for near-realtime + historical data analysis with low-latency. One use case is alerting (@NetflixEng uses it for alerting on events).