Benjamin Ostrov

Local AI enthusiast

[Benjamin Ostrov @b_ostrov]

צרוב ובוהק

[Benjamin Ostrov @b_ostrov]

@digitalix @digitalix I use adt-link for pci to thunderbolt

[Benjamin Ostrov @b_ostrov]

2 weeks of driver and client optimizations for working with RDMA seems to be 1 victory. Already, RDMA mode on my M2 Ultra with a unified memory bandwidth of 800 GB/s and DGX Spark with a unified memory bandwidth of 273 GB/s gives this result
https://t.co/Pff6vgNARp

[Benjamin Ostrov @b_ostrov]

@digitalix Tb4 Now works on Mac Studio M2 Ultra + dgx spark

[Benjamin Ostrov @b_ostrov]

@MiaAI_lab @volatilemarkts Maybe you could also use connecting rdma between machines?

https://t.co/Pff6vgNARp

[Benjamin Ostrov @b_ostrov]

@ivanfioravanti Maybe you’ll be interested here

[Benjamin Ostrov @b_ostrov]

If you have a Spark and a Mac Studio M3 or newer, let's test the driver with your Mellanox cards

[Benjamin Ostrov @b_ostrov]

@washed_boogie @ashxhart @washed_boogie

[Benjamin Ostrov @b_ostrov]

@ashxhart In the end, were you able to reduce all overhead costs in rdma with your project?

[Benjamin Ostrov @b_ostrov]

And I'm not on PCI gen4 yet, but on PCI gen3.

[Benjamin Ostrov @b_ostrov]

@u1tra_instinct @ivanfioravanti @audreyt Well, it’s like I’m on the finish line with the Roce v2 driver to work rdma between Spark and the studio

https://t.co/Pff6vgNARp

[Benjamin Ostrov @b_ostrov]

Eyeing ConnectX-5/6/7, more PCIe lanes, and feedback from anyone with cards. Repo + release: https://t.co/Pff6vgNARp

[Benjamin Ostrov @b_ostrov]

Shipped MelonDMA v0.2.0 — a native RDMA (RoCEv2) driver for macOS that doesn't officially exist. Mellanox ConnectX runs on an M2 Ultra Mac Studio via DriverKit. 40G. No Apple driver, no NVIDIA stack.
https://t.co/yjfbNF7Rgd

[Benjamin Ostrov @b_ostrov]

The goal: make RDMA normal on macOS — llama.cpp / MLX / distributed inference moving KV cache and weights over RoCE, no TCP. Disaggregated inference on Apple Silicon with better TTFT. Eventually a Metal GPUDirect-style CPU bypass.

[Benjamin Ostrov @b_ostrov]

Where it stands: data path + isolation are done and proven. Open: MSI-X delivery (timer polling today), observability, a true kernel-bypass fast path. Pre-1.0, honestly — but it's a driver holding real traffic, not a demo.

[Benjamin Ostrov @b_ostrov]

Live-verified on real hardware: full firmware lifecycle, P0/P1/P3 gates pass on a loaded DEXT — 1M msgs with zero errors, isolation, quotas, ABI fuzzing, inline sends, atomics. ~73µs RTT, 20–28 Gbit/s (PCIe Gen3 x4 cap, not the driver).

[Benjamin Ostrov @b_ostrov]

MelonDMA is a DriverKit DEXT that takes over the Mellanox ConnectX-4 Lx PCI card and brings up real RoCEv2: QP/MR/CQ, SEND/RECV, RDMA read/write, atomics, DCQCN. Plus a libibverbs-compatible layer, so existing RDMA apps just work.

[Benjamin Ostrov @b_ostrov]

If it is implemented, then in the near future I will be able to make a benchmark apple silicon M2 Ultra rdma vs tcp40Gbe vs tcp10gbe

[Benjamin Ostrov @b_ostrov]

One of the most important in the work of the driver is the implementation of the GPUDirect analogue for M series chips (and not to cause panic reboot)

[Benjamin Ostrov @b_ostrov]

Well

I thought it would be easy to assemble the rdma driver.
Now the driver’s work is at the stage of surpassing TCP and finding all optimization points.