Uber has one of the largest Kafka deployments in the WORLD…
• 138 million messages a second
• 89 GB/s
• 38 clusters
So how do they secure them?
mTLS and strong authorization rules.
Uber models its production environment as a zero-trust network. Since any host can be compromised, they rely on strong cryptographic primitives to establish trust between services. 🥷
uPKI is Uber’s identity platform, based on top of Spire.
It’s responsible for issuing short-lived, auto-rotated cryptographic key pairs.
Those pairs consist of an X.509-SVID, a private key and a trust bundle. 🔐
With this setup, the system automatically refreshes (rotates) its certificates once they’re near expiry. ♻️
The Spire agent keeps a long-lived connection open to the Spire server, and proactively generates a new cert when the time is right.
💡 This is key, because it allows Uber to issue very short-lived certificates which then bound the time a bad actor has to make use of any leaked credentials.
mTLS gives Uber authentication, confidentiality via encryption, and data integrity. 👍
What’s missing is authorization.
Similar to checking in to a flight after they’ve verified your ID (you are who you say you are)...
The next check is - are you actually allowed to board this flight? ✋
(do you have the ticket with the right permissions)
For this, Uber has a service named Charter.
Similar to AWS IAM, it’s a framework allowing you to specify who and what can access specific services and resources. 🔍
Kafka has a pluggable authorization framework, where you can configure a custom Authorizer class to do the authz for you (the authorizer_class_name Kafka config).
This authorizer gets called as part of any request flow. In Uber’s config, it’s called with a pair of actor, resource and operation.
Definitions:
🎬 actor - the entity that is the subject of the authorization decision (e.g you at an airport). Also called a KafkaPrincipal in Kafka.
🪨 resource - the resource upon which the authorization decision is made (e.g the flight)
🔧 operation - the operation performed on the resource (e.g boarding). Also called an ACLOperation in Kafka
The Authorizer makes a remote RPC call to Charter to figure out the decision for this pair, and from then-on it caches the result.
Simple, right?
I’m giving away 5 memberships to @striver_79’s new platform TUF+:
1. Repost this X post!
2. Add a comment below on why you should be selected!
Make sure you’re following me so I can DM you. Winners will be announced on Monday.