Introducing the LiveKit C++ SDK.
Realtime audio, video, and data tracks for C++ apps, with the same low-latency transport our other clients use. Built for the C++ stacks behind robotics, autonomous vehicles, and high-performance media pipelines.
https://t.co/T6NuISyRqb
@avizurlo We should chat. Low latency. SDKs for Rust, C++, Python, web, Unity and more. HW encoder support for every platform. Frame level timestamping.
@Rajath_DB@livekit@southpkcommons The whole frame. WebRTC on a decent internet can def do <200ms glass to glass so there’s another 100ms budget for inference.
Best way to kick off a week with the @LiveKit robotics team: win the @southpkcommons Embodied AI Hackathon the weekend before.
Over 48 hours, we really cooked something special. Distributed low latency inference (VLMs, ACT, MolmoAct 2), teleop, voice agent orchestration, all powered by @LiveKit.
@Rajath_DB@livekit@southpkcommons In some ways it’s more tolerant than voice. If voice takes > 1s to respond, it starts to feel off. If the robot waits an extra second to move, nobody notices.
@pham_blnh, @chenosaurus and Jacob’s Desktop Robot Assistant looks simple, but the architecture is wild. To move a candy bar, it orchestrates: Voice Agents, VLMs for spatial awareness, ACT policies, and Momo Act 2.
The models are distributed across laptops in the room and an H200 server in Finland, executing physical tasks over the internet with real-time, ultra-low latency.
what can i say, my models always get the best data there is
so they always work first try
built this to test remote hg-dagger style data collection, can’t wait to share the tutorial
collect data remotely using @livekit, infer remotely using @livekit
excited for what’s to come, handling transport layer for robots should be as easy as setting up a call
You can now play Doom as a world model inside your browser.
I’m happy to finally release learning world model learning part 3.
https://t.co/KDGnu1ipuF
This time we finally go through the entire pipeline and train a “playable” world model.
Unlike previous articles, in this one, we’ll go through some more practical notes like:
> How to train a model cheaply using spot instances
> Identifying when low loss is bad
> Loading data from large datasets
Hope you like this one. Thanks to my readers for pressuring me into releasing this in time.