Notes from the paper. The big takeaway is that io_uring isn't useful without architecting the application to make use of it.
It's remarkable how similar Turso's architecture is to the paper's recommendations.
- batch I/O operations
- batch I/O operations better (with adaptive batching). Use a counter to determine how many SQE's to submit at once
- without I/O batching, io_uring will tie you to device latency
- benefit only comes through at high I/O concurrency. Otherwise you're better off with pread/pwrite
- async execution model is a pre-requisite - coroutines/fibers that yield on I/O allow you to generate high queue depth.
- use ring-per-thread with computation and I/O colocated on same thread. pin worker threads and NIC's to the same chiplet (chiplets consist of cores). this is effectively thread-per-core with a ring-per-core.
- register buffers up front to avoid cost of validation and pinning physical pages for DMA access
- use NVMe passthrough + IOPoll under high load. Bypasses a lot of layers but IOPoll burns a CPU core.
- for network, zero-copy send + recv via io-uring will give benefit over epoll but only for messages above ~1KiB.
- use SQPoll so kernel thread continuously polls SQ for entries. burns a core but useful in high throughput I/O scenarios
XTDB has a SQL adapter powered by @ApacheCalcite. I always love to hear about databases that offer SQL as well as another language (in this case Datalog and full-text search). https://t.co/ggTTkUzbBU
Because the pandemic won't end, @Lin_Ma_ and I are hosting another @CMUDB Vaccination DB Seminar Series - Booster Edition! https://t.co/0x53QGGFPw
All talks open to public. Youtube afterwards.
Thnx again @OtterTuneAI and Steven Moy Foundation for Keeping it Real for sponsoring.
Born on this day in 1944, Jim Gray received the 1998 #ACMTuringAward for seminal contributions to database and transaction processing research and technical leadership in system implementation.
In the video, Gray shares his thoughts on database transactions and the ACID test: