Aadithya Chandra

Software Engineer

[Stanislav Kozlovski @kozlovski]

This is the most impactful Apache Kafka mentor you’ve never heard of:

Chia-Ping Tsai.

In just 18 months, he bootstrapped a large Taiwanese open source community boasting:
• 5000 participants
• 15,000 Slack messages/month
• 10 meetings/week
• 20+ Apache committers

[Stanislav Kozlovski @kozlovski]

The Blue July Crisis.

When one bad production deployment took down MILLIONS of machines in EVERY country across the globe...

Today we experienced the largest global software-inflicted outage.

A world-wide deployment of the CrowdStrike Falcon agent caused massive widespread outages amidst… basically everything that runs Windows. 💀

What did they break?
• ☎️ Emergency Departments - 911 lines
• ✈️ Airports - 1000s of flights grounded
• 🏥 Hospitals
• 📺 TV broadcasters (ABC, SkyNews, etc.)
• 💻 Any other business running Windows with Falcon

But what even is CrowdStrike? 🤔

CrowdStrike is a very successful security company - it’s one of the top 200 richest companies in the world (by market cap).

Its customers account for 298 out of the Fortune 500.

What does it do?

It basically sells advanced anti-virus software. 🛡️
The new buzzwords for this are NGAV and EDR:
• NGAV: new-generation antivirus
• EDR: endpoint detection and response

CrowdStrike Falcon is their flagship product - a very widely used cloud-based endpoint security system. 🦅

An endpoint security system is basically a kernel-level (ring 0) agent that monitors anything that’s happening on your machine:

• processes that are executed
• detailed network activity - DNS requests, established connections, open ports
• removable hardware media usage
• user activity
• etc.

And gives you a firewall, virus/intrusion detection, encryption control and more.

It monitors everything happening in real time and uses its kernel-privileges to block any threats. 👌

It even sends data back to CrowdStrike’s servers (the cloud) for their heavyweight ML algorithms to help detect problems.

Since it’s cloud-based, CrowdStrike can auto-update the software at will. ☁️

And they did.

On the 18th, CrowdStrike deployed an update to its Windows agent that bricked many machines, causing them to enter a loop where they hit a Windows error screen - Blue Screen of Death (BSOD).

Windows automatically restarts from it and then… hits the BSOD again. 😥

🔵 Some say the most viewed color in the world on 19th July 2024 was Blue.

In what seems like a monumentally embarrassing moment, CrowdStrike went against the very basic software engineering best practices of performing a staged roll out (and perhaps any kind of testing?) - and seemingly deployed everywhere, globally, at once. 🤦🏻‍♂️

The worst thing?

They can’t even roll out a fix! 😬

Since the agent at the kernel level crashes Windows and prevents it from starting up - no more updates can be deployed!

A complete massacre.

There’s a reason why competitors like Apple have deprecated kernel extensions (kexts) in MacOS and opted for system extensions. 💡

The only mitigation is a manual one, and it has to be done on EVERY machine:

1. Reboot the machine in Windows’ safe mode
2. Delete a CrowdStrike file
3. Reboot the machine

My sincere apologies go out to every IT admin working today. 💀

It'll likely take a few days to fix all worldwide instances of this problem.

Think again next time you deploy software. 🙂

[Confluent @confluentinc]

Confluent Cloud is now 10x faster than #ApacheKafka, thanks to the Kora Engine!
This means:
⭐10x faster data delivery
🌟Consistently low latency across various workloads
💫Auto monitoring & mitigation for any infra hiccups impacting latency
Learn how -> https://t.co/KQqOfJpBOQ

[Aadithya Chandra @aadithya]

@FrancoisMassot @fulmicoton I've mostly seen gzip being used with compression ratio of more than 10.

[Aadithya Chandra @aadithya]

@FrancoisMassot @fulmicoton On AWS, I have seen folks use S3 for storage and Amazon Athena for ad hoc analysis of compressed logs.

[Aadithya Chandra @aadithya]

@fulmicoton @FrancoisMassot I am trying to understand what the cost comparison looks like for storing raw data vs indexed data for some of the most common use-cases (logs for ex)

[Aadithya Chandra @aadithya]

@fulmicoton @FrancoisMassot Very interesting that storing logs for 7 days is equivalent to the indexing costs. Do you know how this compares to compressed logs (since raw logs are always compressed).

[Anna Povzner🌻 @AnnaPovzner]

I’m excited to share that our paper @ConfluentInc, "Kora: A Cloud-Native Event Streaming Platform For Kafka" was awarded the Best Industry Paper at @VLDBconf out of 29 accepted papers in the Industry track. 🎉 Let's take a look at the details of the paper (a thread 🧵)

[Confluent @confluentinc]

🍾 🏆 Break out the virtual champagne, we brought home Best Industry Paper at the Very Large Data Bases conference (@VLDB2023) with our Kora engine!

Get the details in the blog & check out the winning paper about how Kora meets its cloud-native goals → https://t.co/CWFZOhimIv

[kepano @kepano]

The plan for Obsidian is to never grow beyond 10-12 people, never take VC funding, never collect personal data or analytics.

Continue building with the assumption that software is ephemeral, files matter more than apps. Use formats that are open and durable.

See our manifesto:

[Addison Huddy @addisonhuddy]

Nothing kills tail latency like degraded infrastructure 💾 📈 but Confluent’s Kora Engine continues to improve. Last week, the Kora team added enhancements to Kora’s automatic degraded infrastructure and remediation capabilities. The above graph shows Kora detecting an EBS issue and automatically remediating it in under 5 minutes. Building Kora is never done but this is one big step forward. Kudos Niket Goel, @tikachu99, Adithya Chandra, @dhruvilshah, Ning Shan and the entire Kora team.

[Aadithya Chandra @aadithya]

@PaperBagInvest @wholemars I pay around $25 per month with GEICO for third party insurance. I don't think Tesla or lemonade is going to beat that anytime soon. Low rates and no telemetry but GEICO will continue to make money underwriting my cars.

[Stanislav Kozlovski @kozlovski]

Our paper won the VLDB award for Best Industry Paper!

Kora: A Cloud-Native Event Streaming Platform For Kafka

Not out yet, but trust me when I say it is 🔥🔥🔥

Kudos and thanks to everyone who worked on both the paper and the platform that lead to it!

https://t.co/O3yK4B7TnX

[Aadithya Chandra @aadithya]

@Kiwi_Investor Why would that make any difference to a person comparing it to a hotel?

[Aadithya Chandra @aadithya]

@loobah_l Strategy is telling people to go to hell. Ops is actually making the trip.

[Aadithya Chandra @aadithya]

@deadly_onion I think I used readwise for 3 days. I still use obsidian after 3 years. They are clearly helping solve real problems.

[Stanislav Kozlovski @kozlovski]

The way Datadog calculates percentiles at scale is very innovative 🔥

Usually, calculating the percentiles of large datasets is very expensive.

To know the 99th percentile of a stream of values, you need to:
- keep all the values
- sort them
- return the value whose rank matches the percentile (e.g 99th item)

Datadog cannot afford to do this with the many millions of data points that come in every second - the space and CPU requirements are not practical for a company with thousands of customers. 🐾

Naturally, they opted for sketch algorithms - those should provide them with a good-enough probabilistic result while being vastly more efficient to compute.

Unfortunately - they couldn’t get satisfactory results.
The algorithms would produce results that were too inaccurate. ❌

Why?

Many percentile sketches had guarantees in terms of *rank error*.

A rank-error guarantee of 2% means that the p95 value returned by the sketch is somewhere between the p93-p97 value.

But system latencies exhibit very fat tails - the difference between the p97 and p99 values can be 2-10x!

So what did the dogs do? 🐶

They invented a new sketch algorithm - DDSketch.

Instead of rank error guarantees, they designed it for *relative error* guarantees.

If the p99 is 60s, a 2% error means the sketch would return 58.8-61.2s.

The algorithm is surprisingly pretty simple:

• They create buckets covering ranges of the desired error rate. (+- 2% in this case) 🪣

• Each bucket keeps a counter of the amount of data points within that range. 💯

• When processing an item (latency metric data point), increment the counter of the appropriate bucket. ➕

• To count the desired percentile, you sum up the bucket’s values until you get to the desired percentile. Whatever bucket that percentile is in - that’s your value. 🏆

In this example, the 50th percentile is 1033ms. (4th value out of our total of 8)

Going by count, the 4th value is in the second bucket (b-1) and the algorithm would produce a result of 1021-1061ms.

To cover the range from 1 millisecond to 1 minute, you only need 275 buckets.
With 64-bit counters, that's just ~2kB of memory, regardless of the amount of input data.

This is why we call sketch algorithms sublinear in space growth - memory requirements do NOT grow linearly with input.

The exponential nature of the bucket distribution makes it cheap to cover an even wider range: 1 nanosecond to 1 day takes just 3x more buckets:
• 802 buckets at ~6kB.

As you can probably tell, this is pretty easy to parallelize.

You can divide this bucket-building exercise into many parallel lightweight substreams, and then merge the results freely. 🕊

The merge operation is a simple sum of the buckets & their counters, which ensures that the accuracy is kept in the same range.

It is a very scalable and performant sketch algorithm.

Kudos to Datadog for inventing it.
Good boy! 🫳🐕‍🦺

[Aadithya Chandra @aadithya]

@investor_daddy 16% IRR translates to gross return of something like 60% over 3 years. Cost of capital is still high. Those models better be right!

[Aadithya Chandra @aadithya]

@josephazam @Airbnb In my experience and from what I've read on Twitter their customer service is terrible. Why does someone have to post this on Twitter for them to take action?

[Aadithya Chandra @aadithya]

@bj23air 2020 $LMND, 2023 $TSLA