Ben Aubin

systems at @notionhq

[Ben Aubin @benisahuman]

yeah the c9g instances are crazy good. at Notion we are rolling this out today. early results show p95 api latency is ~30% faster.

(also: the c8a/amd cpu instance types are also very fast and worth trying!)

[Nate Berkopec @nateberkopec]

The new c9g instances on AWS absolutely rip

Because your fleet is cattle and not snowflakes (right?) anyone with a webapp should look at upgrading. Seeing 20-25% less latency across multiple apps

[Ben Aubin @benisahuman]

@garrettfidalgo maybe. but what are the odds only 10% of your api servers are down, and the right level of containment is per-server? if you loose servers to the point the service is fully saturated, you need to shed some traffic or you'll have a complete outage.

[Ben Aubin @benisahuman]

liveliness and readiness checks need to be stupid simple. just return "ok"

never ever check the health of a cache or database, etc. all this does is turn small outages into large ones

database fail? you can still serve other shards or cache. you shouldn't kill your api servers!

[Ben Aubin @benisahuman]

my friend gave me $1 in 1 second, call that $31,556,952.00 in ARR

[Ben Aubin @benisahuman]

This was my guess at what scott aaronson was hinting at in a blog post last month (https://t.co/FHe0d9oR6v)

[Ben Aubin @benisahuman]

wow.

[Ben Aubin @benisahuman]

@bnkrft Ooooo both look good

[Ben Aubin @benisahuman]

after living here for a while, i have managed to find 4 good late night dinner spots:

ABV till 12am (all their food is excellent!)
The unmapped taco stand at SW corner of 19th and mission (till 3am or later)
Nepa Indian (around Nopa) till 12am
In-n-out (near Stonestown) till 1am

[conway @ConwayAnderson]

Wild night out in SF

Managed to find a place to eat after 9pm

[Ben Aubin @benisahuman]

@hrlomax @bnkrft oooo ++ to el farolito

[Ben Aubin @benisahuman]

@sfautist lol this is the planetscale business model more or less: buy ec2 instances, run postgres on them, sell to customers for less than rds or better performance

[Ben Aubin @benisahuman]

very cool and powerful! though: the slightly better version of this is that duckdb can query your database. you should consider that, because duckdb is very compute intensive, database compute is scarce and lambda compute is easily accessible on demand and plentiful.

[Andy Jassy @ajassy]

A painful part of working with your data has always been that your live data and historical data are stuck in separate systems: the order a customer just placed lives in your database, while their last five years of orders sit in a data lake in S3.

And answering a real question usually needs both at once (is this a normal purchase for them, or should we flag it?), and to do that you had to move the data together first, copying history out of the data lake into your database (or the other way around), because the database couldn’t read it where it lived.

That meant guessing ahead of time which data you’d want, keeping a second copy of it all, building pipelines to move it, and constantly syncing so the two didn’t drift apart. A lot of plumbing, and slow going, all before you could answer one question. And even then, the answers were only as fresh as your last sync.

That now changes with Aurora PostgreSQL, which can call your live data and historical data in S3 together, in a single query. No copying, no pipelines to keep in sync.

And it’s fast, because we’ve built in DuckDB, a popular open source engine that’s really good at reading and analyzing data right where it’s stored. DuckDB reads the open formats like Parquet and Iceberg already sitting in your data lake, so there’s nothing to convert or move.

As folks build AI agents into their apps, the data their agent needs will depend on the task in front of it. Being able to query that specific data live, instead of copying it over just in case, is gonna be a big help for builders. https://t.co/R8h2Hcp1K7

[Ben Aubin @benisahuman]

And also - at least as of a year ago - DuckDB crashes a lot! we keep evaluating it for various things. we want it to work. but then it crashes. not good! don't do that to your Postgres process.

[Ben Aubin @benisahuman]

Your database cpu is the most expensive kind!!! amazon marks up RDS instances by ~93% vs ec2. lambda is ~43% marked up. but the most significant cost difference is that w/ lambda you only pay when your function is running. with a database, you pay all the time for peak usage

[Ben Aubin @benisahuman]

one thing llms still struggle with: deciding whether to fail open or fail closed in system design

caught so many bugs from an llm over eagerly validating something "for safety"

it feels like they are rl'd hard to try to be safe, but make choices performatively

[Ben Aubin @benisahuman]

@theombl was there ever much institutional interest in microstrategy? i have always assumed a lot of the momentum came from retail

[Ben Aubin @benisahuman]

@sliminality this thread is opposite of conflict free

[Ben Aubin @benisahuman]

@tab_delete even though white hat bounties are less than criminal proceeds could be, bug bounty programs have historically been more attractive than other avenues because they pay quickly in clean legal cash

[Ben Aubin @benisahuman]

notion builds our durable logs on dynamodb, not s3. it's 10x cheaper and 10x faster for writes, and you don't need fancy techniques or software!

Just use: sort key = sequence number

no fancy caches needed! you can read by sequence number.

[Ben Aubin @benisahuman]

such cool work!!

[emma @emguoz]

Notion runs one of the largest CRDT deployments in production, processing millions of operations every minute!

We wrote about the data structures we adopted from CRDT research to make our text editor more collaborative, and how we built on those concepts to fit our block-based data model.

https://t.co/rNZtmmylu9

[emma @emguoz]

Notion runs one of the largest CRDT deployments in production, processing millions of operations every minute!

We wrote about the data structures we adopted from CRDT research to make our text editor more collaborative, and how we built on those concepts to fit our block-based data model.

https://t.co/rNZtmmylu9