tushar

making compilers grin, building distributed systems, previous @ihc__official, research @iiit_hyderabad

[tushar @gpu004]

@ptr_to_joel @TigerBeetleDB lately i have doing small implementation of casandra paxos lwts, tigerbeetle viewstamped replication, cockroachdb raft per replica group

playing around with different consensus ideas

[tushar @gpu004]

@ekzhang1 @bcrussett I would love to become part of nysrg, dropped a mail, hope I get accepted

[tushar @gpu004]

@GoshawkTrades they have stopped doing it, it's not an efficient way to allocate compute

[David @dzhng]

https://t.co/VA6b6kdAl6

[Nathan Flurry 🔩 @NathanFlurry]

Everyone else is talking about S3 databases right now...

So let's talk FoundationDB.

It's the database that lets Rivet have:

- 0.8 ms reads / 3.6 ms writes (triple AZ, p95)
- Scale writes horizontally
- Orchestrate millions of actors
- Bottomless, high-performance SQLite storage
- Backs our globally-distributed edge network for actors
- Supports our S3 tiered storage system

Similar to Tigris, we made a decision early on to invest heavily in FoundationDB instead of building our own database or sharding Postgres.

It's not a crazy proposition to build your own database for use cases like this. Many durable execution & orchestration projects went this route (Nomad on HashiCorp Raft, Durable Objects on SRS, Restate).

Here's why we went with FoundationDB instead.

~~~

When building Rivet, our requirements looked like:

High performance → we're dealing with orders of magnitude more Actors than Nomad / Kubernetes containers, so our write throughput needed to scale linearly

Scale writes horizontally → we wanted a "just add nodes" solution to scaling throughput, instead of maintaining a single large primary VM (e.g. Postgres) or weakening consistency guarantees (e.g. Cassandra)

Strong consistency → eventual consistency creates an incredibly complicated class of bugs

Enable user-facing database → we provide a KV API (now SQLite) to Actors, our database needed to be able to support that

Self-hostable & permissive licensing → no proprietary components or vendor lock

Don't shard Postgres → you're basically rebuilding a slower, more complicated database when you do this

~~~

FoundationDB fit our use case very well:

- Scale writes by adding nodes
- ACID compliant
- Flexible enough to power all of Rivet & actor use cases
- Apache 2.0 permissive license
- Powers Apple (iCloud), OpenAI, Palantir, Adobe, Datadog, Snowflake, many more

We don't have (nor want) to join the Twitter database wars because ours is delightfully boring. It just works, it's already fast, and it already runs the world's largest companies.

And most importantly, it's correct. In the words of Kyle Kingsbury (Jepsen author):

"haven't tested foundation[db], in part because their testing appears to be waaaay more rigorous than mine"

~~~

Additionally, we put a lot of effort in to making sure Rivet can run on your existing infra.

Rivet supports multiple storage backends:

- FoundationDB → large scale + multi-tenant deployments
- Postgres → smaller self-hosted deployments
- RocksDB → local dev + single-node deployments

In addition to S3 tiered storage

~~~

I've also discussed how we implement our zero-disk, S3 tiered storage architecture here: https://t.co/vcM0Zd4Hoo

[tushar @gpu004]

@jhleath Yep, clay might a real steal name

[tushar @gpu004]

@aaronmring 10 billion dollar acqutition, if it happens this year, maybe be more if it happens next year

[tushar @gpu004]

@eatonphil @ssougou feeling so much fomo, will definetely attend the next nyc systems meetup

[John Carmack @ID_AA_Carmack]

https://t.co/896OIixzGh

[Aaron Villalpando @aaronvi]

Everything runs off s3 now

- clickhouse
- durable objects
- turbopuffer
- cursor Origin

It might be one of the greatest products ever made

[Eric Zhang @ekzhang1]

I thought about object storage a lot when I was working on serverless stuff and want to nerd out

In some ways you can build every piece of software in the world on key-value stores. A key-value map is the most fundamental data structure in all computing (see: Redis, DNS, LevelDB, etcd, git, ext4, BGP… are all dicts if you squint hard enough)

So in addition to S3 just being… the first cloud product ever made, there are a few systemic reasons why S3 won over the past 20 years:

- Moore’s law stopped but NICs continued doubling in bandwidth every 2 years, eventually network becomes faster than SSDs (even for consumers! https://t.co/p02gO8CQRh ), so disaggregated storage became the default (e.g., Snowflake paper)
- There was the “big data” craze and as datasets grew too big for one machine, the sensible place to put them is not in an RDBMS, but a bunch of blobs with “infinite” read bandwidth
- AWS charges massive egress costs. As everyone moved to the cloud, you needed to move storage there as well, otherwise it became too expensive to process your data. Storage became the moat for a lot of cloud companies (and still is!)
- AWS has really solid full stack reliability, better than any competitors. S3 is critical infrastructure for the whole world.

[Joran Dirk Greef @jorandirkgreef]

Live for the kickoff of SD26 online with @axboe right now in the chat…

https://t.co/BwcF9W7Bdl

[tushar @gpu004]

@shivamhwp Nice new dp

[tushar @gpu004]

@boristane Is this polylane going through the observability logs, alerts and everything and finding out the root causes and then spinning up claude for fixing the bug?

[tushar @gpu004]

@ditsuke awesome read

[tushar @gpu004]

@poonai_ super cool name man

[Ben Dicken @BenjDicken]

This is one of the best engineering blogs you'll ever read.

Not *just* because you'll learn a lot, but because @vmg sprinkles in a bit of everything.

A great story. History of git infrastructure. Distributed systems. Benchmarks. All written by someone who can speak to every point from deep, personal experience.

It's also 5,300 words, so grab a drink and enjoy.

[tushar @gpu004]

@ditsuke Thank you so much Tushar

[tushar @gpu004]

Just landed in NYC with big dreams.

Chapter 1 starts now. 🗽

[tushar @gpu004]

@pankajsuthar95 Thank you pankaj

[tushar @gpu004]

@SerratedLiminal Thank you 🙏