Most don't realize that nearly *every* database is built with a WAL.
Postgres. MySQL. SQLite. Mongo. Kafka. ClickHouse.
WALs are fundamental for fast, reliable, transactional data processing. Everyone should know how they work.
At 250 billion DNS cache entries, one wasted byte costs 250 GB of RAM.
Five Rust optimizations later: 100 TB freed, inserts 43% faster, lookups 19% faster.
We didn't trade speed for space. https://t.co/mMyOYnj0mQ
At 250 billion DNS cache entries, one wasted byte costs 250 GB of RAM.
Five Rust optimizations later: 100 TB freed, inserts 43% faster, lookups 19% faster.
We didn't trade speed for space. https://t.co/mMyOYnj0mQ
BIG ANNOUNCEMENT FROM HUGGING FACE TODAY:
We're unveiling Microduck ๐ฅ๐ค
It's a tiny $399 open-source robot you can teach new tricks with reinforcement learning. It can walk, pick things up, get back up when it falls, and even roller-skate.
Welcome to the era of open-source affordable robots to democratize physical AI and world models!
๐ค๐ค๐ค
If you want to see what running a system at real scale looks like, read this deep dive on how S3 works.
The numbers alone are hard to think about: 280 trillion objects, 100 million requests per second.
Three aspects particularly struck me:
โข S3 is not a single system. It consists of hundreds of microservices managed by different teams, communicating via API-based contracts.
โข The enemy is โheatโ (i.e., load concentration), not storage capacity. Too many requests hit the same disk, the disk becomes overloaded, and everything slows down. Data is protected using erasure coding and replicated across numerous disks, and objects within the same bucket are deliberately distributed separately. A traffic spike from a single customer is spread across many disks rather than burdening just one.
โข The storage node code was rewritten in Rust and formally verified. Itโs rare to see formal verification outside academia and safety-critical niches.
Capacity planning is easy. Preventing the load from building up in one spot is not.
Link in the reply:
A compiler can inject a backdoor into itself and into every future compilation of itself which is invisible in the source, i.e. the trusting trust problem (Ken Thompson, 1984).
the engineer (@axboe) who built io_uring in linux really cooked
I read his writeup: https://t.co/992fpb7pi0
the chief idea is simple: Pre-allocate a pair of shared rings for IO submissions and completions. Use mmap on the rings' memory to share with the kernel and avoid system call overhead. Add entries to the submission ring and poke the kernel (one system call for up to N io operations). Then the userspace app reads from the completions ring when it can.
The hard part was getting the API right so that this is a generalizable and performant interface, 0 copies, etc, and not super bespoke like aio. Given the level of adoption, I think Jens Axboe nailed it!
Ten years ago, I stopped using P99 and switched to P100, always.
- itโs easier to calculate (just take the max)
- itโs easier to understand (this is as bad as it gets, thereโs nowhere to hide)
- optimizing for P100 tends to optimize for all percentiles (especially over time)
like celld, walgit depends only on s3 for storage and coordination. This has all become possible because of S3's support for Compare And Swap since 2024.
It's just a practical pattern for reliability and costs.
Introducing Dactyl
Dactyl is a vibe coding platform for native iOS and native Android apps. What's interesting about it is the device simulator: we've built a SwiftUI renderer in WASM.
This means as you're developing, you get low latency in browser canvas simulation of the app to play with. Also things like oauth flows happen in the browser.
If the app uses the camera - the browser camera API allows us to display it in the simulator.
We also let you connect your ChatGPT subscription so you don't have to buy tokens through us (+ markup)
Through [magic] - we can run the same SwiftUI code the agent writes for iOS in android, with native components.
This is the work of @undefined_void (primarily), @piscisaureus, myself, and others at Deno.
It's built on Durable Objects perhaps unsurprisingly
https://t.co/IeBKwweusN
Git at Scale (by cursor) has been one of the most interesting blog posts i've read in a while. It came right when I was frustrated with Shopify's internal git system.
As an exercise, I've implemented it over the weekend as open source. It's a single rust binary that you can point at any S3 type object store. It uses WAL and CAS primitives and requires no other data store.
It also implements bundle-uri so large git repos (like our mono) are very fast to download as a chain of static bundles. Also comes with basic familiar UX.
https://t.co/oEnWK3eIL4
@DabsMalone Price's Law states that 50% of the work in a company or group is done by the square root of the total number of employees. For example, in a company of 100 people, just 10 employees do half the work, while the other 90 people do the remaining half.