S3 feels like magic. Bottomless storage, instant retrieval and infinitely scalable.
Turns out that it’s just a massive fleet of servers with real capacity limits and those servers 5xx when overloaded.
We run millions of ec2 instances a day and regularly see S3 errors in our logs.
we had an incident because we migrated traffic to a brand new s3 bucket.
our millions of servers instantly crushed the new bucket’s partitions and started getting slammed with 5xx errors
On any network, you can see who is on a video call, their movements or speech patterns by sniffing the RTP headers. Even if your browser shows as “secure”, RTP headers are completely unencrypted over the wire. Here's what's leaking 🧵
We asked a dozen DevTool founders from companies like @RevenueCat, @greptile, @firecrawl, @infisical, @ollama, @resend, @mintlify, @UnslothAI, @porterdotrun, and @recallai, about the state of AI agents and the future of software engineering.
In this episode of Founder FAQ, we covered everything from agents as customers and the end of coding, to advice for founders starting out and what they're most excited about going forward. Their answers might surprise you.
00:00 – Meet the Founders
03:00 – Building for Agents First
04:22 – Biggest Early Mistakes
07:15 – Do Founders Still Write Code?
09:22 – Most Unexpected AI Discoveries
12:09 – What's Underrated Right Now
14:38 – Predictions & What's Next
We are hiring!
If you're interested in launching millions of instances, operating large-scale video processing clusters, and writing high-performance Rust:
DM me your GitHub :)
We launched 18 million EC2 instances last month.
10 things you didn’t know about the EC2 API, part 2.
Fact: AWS's code path for booting new EC2 instances is buggy. When you boot a lot (e.g. 10,000s) of EC2 instances you often see failures with Server.InternalError
10 things you didn’t know about the EC2 API, part 1.
Fact: RunInstances returning a 201 with N instances does not mean you have actually been allocated N instances. Any number of those instances can fail to boot with Server.InsufficientInstanceCapacity.
We launch over 18 million EC2 instances per month.
One thing we've learned is that cold-booting an EC2 instance is very slow, but this happens for a surprising reason.
When you boot an EC2 instance from an AMI, creating a new root EBS volume is near-instantaneous.
However, the reason AWS can create a 20GiB - 100GiB volume instantly is because no data is actually copied on volume creation.
All blocks in the file system are lazily loaded from an S3 backing store on first read. This S3 backing mechanism is invisible to EC2 users, but results in the first read being very slow.
As a VM starts up with a fresh EBS volume, every block it reads needs to be lazy loaded, which can results in 10x - 100x higher read latency!