David Chen

GM Robotics @LiveKit. Building the platform for realtime Physical AI. Previously @Agtonomy, @skycatch, @twitter.

[LiveKit @livekit]

Introducing the LiveKit C++ SDK.

Realtime audio, video, and data tracks for C++ apps, with the same low-latency transport our other clients use. Built for the C++ stacks behind robotics, autonomous vehicles, and high-performance media pipelines.

https://t.co/T6NuISyRqb

[David Chen @chenosaurus]

@avizurlo We should chat. Low latency. SDKs for Rust, C++, Python, web, Unity and more. HW encoder support for every platform. Frame level timestamping.

[David Chen @chenosaurus]

@Rajath_DB @livekit @southpkcommons The whole frame. WebRTC on a decent internet can def do <200ms glass to glass so there’s another 100ms budget for inference.

[David Chen @chenosaurus]

Best way to kick off a week with the @LiveKit robotics team: win the @southpkcommons Embodied AI Hackathon the weekend before.

Over 48 hours, we really cooked something special. Distributed low latency inference (VLMs, ACT, MolmoAct 2), teleop, voice agent orchestration, all powered by @LiveKit.

[Binh Pham @pham_blnh]

we won the embodied ai hackathon at @southpkcommons last week

this is how we did it 🧵

[David Chen @chenosaurus]

@Rajath_DB @livekit @southpkcommons In some ways it’s more tolerant than voice. If voice takes > 1s to respond, it starts to feel off. If the robot waits an extra second to move, nobody notices.

[David Chen @chenosaurus]

@Rajath_DB @livekit @southpkcommons Ideally under 300ms rt for full roundtrip of camera to inference and actions back over webRTC.

[South Park Commons @southpkcommons]

@pham_blnh, @chenosaurus and Jacob’s Desktop Robot Assistant looks simple, but the architecture is wild. To move a candy bar, it orchestrates: Voice Agents, VLMs for spatial awareness, ACT policies, and Momo Act 2.

The models are distributed across laptops in the room and an H200 server in Finland, executing physical tasks over the internet with real-time, ultra-low latency.

[Binh Pham @pham_blnh]

check out my claude knob @trq212

https://t.co/xZKCfGarLm

[Binh Pham @pham_blnh]

what can i say, my models always get the best data there is

so they always work first try

built this to test remote hg-dagger style data collection, can’t wait to share the tutorial

[David Chen @chenosaurus]

@kamathsblog https://t.co/02gYaaChxN

[David Chen @chenosaurus]

@kamathsblog You can use LiveKit to stream from robot to VR.

[David Chen @chenosaurus]

@zoox just expanded their coverage to most of the eastern side of SF!

[David Chen @chenosaurus]

@jaiselsingh What are some good models that are available today?

[Binh Pham @pham_blnh]

collect data remotely using @livekit, infer remotely using @livekit

excited for what’s to come, handling transport layer for robots should be as easy as setting up a call

[Binh Pham @pham_blnh]

You can now play Doom as a world model inside your browser.

I’m happy to finally release learning world model learning part 3.

https://t.co/KDGnu1ipuF

This time we finally go through the entire pipeline and train a “playable” world model.

Unlike previous articles, in this one, we’ll go through some more practical notes like:
> How to train a model cheaply using spot instances
> Identifying when low loss is bad
> Loading data from large datasets

Hope you like this one. Thanks to my readers for pressuring me into releasing this in time.

[David Chen @chenosaurus]

@DeoArlo Hell yea. That thing rips!!

[David Chen @chenosaurus]

@tnkrdotai yes

[David Chen @chenosaurus]

@pham_blnh @livekit @LeRobotHF @rerundotio This is the best example of dogfooding your own product. 🤖🦾🐝