Harald Ng

Ph.D. student at @KTHUniversity working on distributed systems and databases, Creator & Maintainer of OmniPaxos

[Harald Ng @haraldng]

Say hello to OmniPaxos! 👋 A cutting-edge #Rust library making your applications highly available, resilient, and flexible. More robust than Raft and VR, it thrives even in partial network partitions. Stay tuned for more insights into its internals! 🚀

🌐https://t.co/Lua2k54ZbX

[Audrey Cheng @audreyccheng]

What is AI not good at? Database transactions!

In our latest blog post, we dive into how AI could not find better scheduling algorithm than our VLDB '24 paper (in collaboration with @pbailis, @siobhcroo, @istoica05, and many others).

[Harald Ng @haraldng]

@eatonphil This is an excellent classification of how consensus is used in practice. Ultimately, consensus boils down to providing a distributed log. CfM uses the log to build a consistent service holding metadata, while CfD uses the log directly as a changelog/WAL.

[Harald Ng @haraldng]

🚀Check out the latest tutorial on the OmniPaxos playground and dashboard! It runs on docker with an interactive dashboard in the terminal where you can input different network failures and KV-operations to see how OmniPaxos behaves.

More info + video 🧑‍💻: https://t.co/7c6K28GD9l

[Harald Ng @haraldng]

@kryoxnist In the post, we only read locally from a server. To ensure a read returns the latest committed value (i.e., linearizable read) can be implemented in several ways, such as using leader leases or treating the read as a write and committing it in the OmniPaxos log.

[Harald Ng @haraldng]

New blog post! We show how to use OmniPaxos to turn a single-server #RocksDB service in #Rust into a distributed version. OmniPaxos hides all complexities related to consensus and provides consistent replication through a simple log API 🚀🧑‍💻

Post + video: https://t.co/ds8ThoUuxE

[Harald Ng @haraldng]

Hyped to be at #SOSP2023. Drop by the poster session tonight for a chat on how we make consensus proactive and efficient with workload-driven techniques using MetaCoordination 🚀💫

[Harald Ng @haraldng]

@AlekseyCharapko I agree! Great to see changes to MultiPaxos that overcome one of the scenarios. Further modifications can probably be made to handle all scenarios in the paper, including the chained scenario. But that should not be left to engineers, especially if protocols are underspecified!

[Gunnar Morling 🌍 @gunnarmorling]

"Building distributed #RocksDB with OmniPaxos in 8 minutes"

TIL about #OmniPaxos, acting here as a distributed database changelog. Super-interesting write-up, by @haraldng 👍.

https://t.co/2PE4N5tKBq

[Paris Carbone @SenorCarbone]

A great ACM recognition both for the groundbreaking data streaming research and innovations behind @ApacheFlink but also for the magnitude of what this system became with nothing less than the help of all the amazing minds and hard work of the whole open source community!

[Harald Ng @haraldng]

@eatonphil Yes! The decoupling of leader election and replication is crucial but not unique, most paxos-based/inspired protocols do that. The second and equally crucial part in OmniPaxos' leader election is to omit all unnecessary candidate restrictions and focus on quorum-connectivity.

[Harald Ng @haraldng]

@jorandirkgreef But for sure, borrowing some ideas from OmniPaxos to VR would probably both solve most issues and optimize for less view changes!

[Harald Ng @haraldng]

🚀 Excited to share our first blog post! Discover how OmniPaxos offers unmatched resilience to network failures by eliminating unnecessary restrictions in leader election and focusing on what is essential: quorum-connectivity.

Details💡: https://t.co/gPJkRlCj4k

[Harald Ng @haraldng]

@jorandirkgreef intuitively, I'm a bit worried if servers directly send DoViewChange as soon as they think the leader has failed. Consider a scenario A-B-C-D-E where regardless of who is the leader, there is at least one server disconnected from it which triggers higher views causing livelock.

[Harald Ng @haraldng]

@jorandirkgreef Thanks for a great catch! We will try updating the table and annotate that particular scenario for VR.

[Paris Carbone @SenorCarbone]

This distills the essence of partial connectivity, a greatly overlooked issue in today's consensus systems originally observed by @heidiann360 @ittaia a few years back. OmniPaxos shows how to boost liveness by decoupling leader election from Sequence Paxos-based log replication.

[Harald Ng @haraldng]

@eatonphil @jim_dowling Our blog post is a good place to start! https://t.co/gPJkRlCj4k

[Paris Carbone @SenorCarbone]

⚡The future of high availability is here! Our #Rust library offers profound resilience, flexibility, & more.
☁️ Try OmniPaxos to replicate your apps today: https://t.co/yZMaITZvh0 .
More juicy details to be revealed in the following week! @EuroSys_conf

[Harald Ng @haraldng]

Thrilled to have presented my paper on UniCache at #EDBT2023 in Greece! Discussed how RSMs such as Raft and Paxos can learn from the workload to reduce the network I/O and improve performance.

Full paper: https://t.co/RS4m1aTz6h

[Paris Carbone @SenorCarbone]

Data compression meets consensus! UniCache can greatly reduce primary data transmission overheads in any replicated service. Presented today by @haraldng at @edbticdt2023 . Full paper: https://t.co/IchK1QkRCQ