Elliot Levin

cursing cursed code @recallai

[David Gu @davidgu]

we had an incident because we migrated traffic to a brand new s3 bucket.

our millions of servers instantly crushed the new bucket’s partitions and started getting slammed with 5xx errors

[David Gu @davidgu]

Did you know AWS has a quota for quota increase requests???

[David Gu @davidgu]

thanks pg&e for reducing my screen time

[David Gu @davidgu]

we run 18 million EC2 instances per month. At our scale, we see very rare bugs very frequently.

Last week, we received *half* an HTTP request. Not a HTTP 206, literally half a request.

Content-Length was 2350 bytes. Body was actually 1200 bytes, and was truncated mid json doc.

[David Gu @davidgu]

By default, the most popular Rust crate for Postgresql (tokio_postgres) waits for *2 hours!!* before timing out a dead connection.

All because of a bad decision from 1989 🧵👇️

[David Gu @davidgu]

A famous bit of 90s hacker lore, after a server update Trey Harris at MIT realized he couldn't send email to any server over 500 miles away.

The root cause: a timeout erroneously set to "0", which rounded up to 3ms. At the speed of light, 3ms ~= 500 miles.

Last week, something very similar happened to us. Except that instead of a misconfigured timeout, in our case the issue involved speeding up time.

[David Gu @davidgu]

We forked Chromium and bolted on a lock-free, zero-copy, low-latency shared memory ringbuffer written in Rust

We needed to IPC 100+ MB/s of raw video, and Chromium's WebSocket implementation is dreadfully inefficient

[Elliot Levin @Time_Toogo]

the pr's i'm the most proud of have this common trait: they delete significantly more code than they add

[Elliot Levin @Time_Toogo]

what % of your codebase could be deleted and the business will be just fine?

[Elliot Levin @Time_Toogo]

great engineering is not about building marvellous abstractions, it's about solving the problem in front of you

[Elliot Levin @Time_Toogo]

production code has a surprisingly high maintenance cost attached to it

[David Gu @davidgu]

At 7:00am PT every day, load on our systems spikes 300% in 60 seconds

Recall is the API for meeting recording.

Everyone starts their meetings at the top of the hour, meaning we handle Black Friday level traffic spikes 20x every day.

At our scale, this involves launching hundreds of thousands of EC2 instances within seconds of each other.

We process over 3 TiB/sec of raw video at our peak load

We're hiring systems engineers.
If this sounds interesting to you, DM me your GitHub!

[Elliot Levin @Time_Toogo]

and we ran head-first into the speed of light from the east <-> west coast (~13 ms -> ~26 ms rrt)

[Elliot Levin @Time_Toogo]

a fun trick to test real-time code quickly is to speed up the clock https://t.co/QtxZYthqPi

[Elliot Levin @Time_Toogo]

turns out this also speed up all the network timeouts for external services to ridiculously low values

[Elliot Levin @Time_Toogo]

we were mildly puzzled when our tests succeeded against our us-east-1 region but failed against our us-west-2 region

[Elliot Levin @Time_Toogo]

we actually do this daily at @recallai, it 10-30x's the speed which we can run out test suite

[David Gu @davidgu]

@_anmonteiro @recallai Any cracked engineers that make it to the last stage of our interview process will also receive a framed photo of Linus Torvalds.

[Elliot Levin @Time_Toogo]

kind of insane python's gc is so bad that it's standard practice to restart workers every N ops

[Antonio Monteiro @_anmonteiro]

At @recallai, we ran 18M EC2 instances last month.

We use Rust to process 3TB/second of raw video in real-time.

If you’re a cracked engineer in SF, DM me your GitHub – if you interview with us, we’ll gift you a free bundle of K&R’s C, TLPI and TCP/IP Illustrated Vol 1 🔥