Sameer Dharur

AI Researcher | Quiz Master in a previous life | 🏏/🎾 nerd

[Sameer Dharur @SameerDharur]

Trying something out after a 5+ year hiatus this weekend!

If you’re interested to consume compelling stories from around the world and its history, especially if you’ve never quizzed before, drop by on Sunday afternoon!

Entry is free and open to all.

[Bay Area Quiz Club @sfbayquiz]

Quiz for the month of July : July 14th, 2024.
Time: 2 PM
QM: Sameer Dharur
Venue: Midtown Room,
Mitchell Park Palo Alto Library
3700 Middlefield Rd
Palo Alto, CA 94303

[Ming-Yu Liu @liu_mingyu]

Introducing NVIDIA Cosmos 3

We released NVIDIA Cosmos 3 last night.

And today, seeing it take the top spots across 8+ open model leaderboards feels surreal. We spent months working towards this moment.

Here’s the breakdown:

The Leaderboard Wins

World Reasoning
🏆 #1 open model on VANTAGE-Bench for vision AI
🏆 #1 overall on Traffic Anomaly Reasoning (TAR)

World Generation
🏆 #1 open model on Artificial Analysis Image-to-Video leaderboard
🏆 #1 open model on Artificial Analysis Text-to-Image leaderboard
🏆 #1 open model on PAI-Bench for physical AI synthetic data generation
🏆 #1 open model on Physics-IQ, which measures accuracy on physical laws
🏆 #1 open model on R-Bench for world generation quality

World Action
🏆 #1 on RoboArena for specialized policy
🏆 #1 on RoboLab for action generation

But the leaderboards are only part of the story. The real story is why we built Cosmos 3 in the first place.

The Problem

Training robots and autonomous systems in the real world is painfully hard.

Robots need to try the same thing numerous times before they succeed reliably. Self-driving cars need rare edge cases that may never happen naturally. Smart machines need to understand physics, motion, contact, failure, and surprise.

And real-world data is slow, expensive, and sometimes dangerous to collect. At some point, the answer cannot just be “collect more data.”

You can’t collect your way out of an infinite physical world. You have to generate it.

That… was the question behind Cosmos: Can one model understand the physical world deeply enough to reason about it, simulate it, and generate actions inside it?

What We Built

Cosmos 3 is the first omni-model for physical AI. It can understand and generate across: language · images · video · audio · action sequences

It is not just a VLM.

Not just a video generator.

Not just a robot policy model.

It is all of them, in one single model.

That matters because physical AI has been fragmented for a long time. Cosmos 3 is our attempt to collapse that fragmentation.

Depending on how you configure the inputs and outputs, the same model can act as a vision-language model, a video/world generator, a world simulator, or a world-action model.

No separate architecture required.

The Architecture

Under the hood, Cosmos 3 uses a dual-tower Mixture-of-Transformers architecture.

One tower is autoregressive for reasoning. It handles next-token prediction for language and discrete understanding.

The other tower is diffusion-based- for generation. It denoises images, video, audio, and action trajectories.
Two towers. Dual-stream joint attention. One shared world representation.

Each modality gets its own tools: visual encoders, video VAEs, audio VAEs, and action projectors that can map different embodiments into a unified action space.

Action is a first-class modality in Cosmos 3.

That’s what makes it more than a video model. It doesn’t just predict and generate what the world might look like. It can connect reasoning and world modeling to physically grounded action.

Why This Matters

One of the most interesting findings from the ablation work is that training action domains together creates positive transfer.

That means adding more embodiments does not just add more use cases. It can actually make the model better.

This is the heart of why omnimodal training matters.

A shared world representation is not just convenient. It can make each individual task stronger. That’s the part that feels like the beginning of something much bigger.

The part I’m most excited about is that Cosmos 3 is fully open.

Developers get the models, scripts, optimization, inference endpoints, post-training recipes, datasets, and benchmarks.

Everything is available under the Linux Foundation’s OpenMDW 1.1 License.

You can use Cosmos 3 out of the box. You can use the VLM, world model, or world-action pieces separately.

You can post-train it for your own domain, embodiment, or accuracy target.

That’s what makes this feel different.

Cosmos 3 is not just a model release. It is the foundation for building intelligence for autonomous machines.

For me, Cosmos 3 feels like a step toward a world where physical AI development becomes much more scalable and accessible - to a new age of developers and agents.

That’s what we built Cosmos 3 for. I cannot wait to see what you build with it.

Download Models on Hugging Face
https://t.co/LAZoVygeim

Customize Models on GitHub
https://t.co/ZVQBNdqXDD

Read the Tech Blog to Learn More
https://t.co/Hn6Op9YeG1

[Sameer Dharur @SameerDharur]

@DhruvBatra_ @yutori_ai Woohoo congratulations 🎉

[Dhruv Batra @DhruvBatra_]

Introducing Yutori Delegate.

Why cowork when you can delegate?

[Sameer Dharur @SameerDharur]

@sarahookr They’re actually so ubiquitous in India that many of us grew up with the misconception that it was an Indian brand!

[Nathan @OIuwatosin]

One thing I love about traveling, is that it breaks the illusion that your current life is the only possible one.

[Sameer Dharur @SameerDharur]

@shantanugoel Yessir, looking forward 🫡

[Sameer Dharur @SameerDharur]

@deviparikh Haven’t watched a Bollywood film in 5+ years but glad I made an exception for this one. Very entertaining from start to finish!

[Ben Rothenberg @BenRothenberg]

Honestly this was one of the most important tennis moments of the last decade.

[Sameer Dharur @SameerDharur]

@DaniilMedwed Try a famous local crepe called the Benne Masala Dosa while you are in Bangalore. You’ll quickly find yourself searching for it in cities around the world as you travel on the ATP tour.

[Sameer Dharur @SameerDharur]

@abhshkdz Congratulations! And what a wonderfully done ad to go with it!

[Sameer Dharur @SameerDharur]

@gauravsabnis Yeah, I fully relate to this too. Feels like we’re a minority on this. Serendipitously created bags of chemicals floating on a rock through space etc.

[Sameer Dharur @SameerDharur]

@DhruvBatra_ Big congratulations! And thank you for the detailed write-up!

[rain @redactedrain]

They don’t teach you this in school but you can just have a delusional sense of optimism towards the world and believe everything is going to work out and you’ll succeed in every facet of life.

[TG @Tennisgeeking]

Carlos plays this sport like it’s an art not a science. In every match it feels like he’s searching for his muse and when he finds it, it’s glorious. He plays like a human not a machine and this means there’s uncertainty, jeopardy but also unbridled creativity.

[Sameer Dharur @SameerDharur]

Much awaited release from one of the best non-fiction authors I’ve read. Congrats and can’t wait to dive in, @samanth_s!

[Sameer Dharur @SameerDharur]

@himganj153 Wow nice! Where?

[Sameer Dharur @SameerDharur]

@SyedMustafaHash Congratulations, officer!

[Astronomer @astronomerio]

Thank you for your interest in Astronomer.

[Isabel🌻 @isabelunraveled]

the reason doing elite sports puts you ahead of the Game early on is because developing a positive, gravitational relationship to discomfort—understanding that doing the uncomfortable thing gets you the desirable thing—is one of the most valuable beliefs you can hold as an adult.