Can an agent learn a pool tableโs physics by playing on it? ๐ฑ
While agents can close the visual gap, the difference between real world and sim dynamics remains significant. For CoRL late breaking๐ค , we let an agent take real shots to try and close this real2sim dynamics gap: https://t.co/g12jkuuB4n
Very fun short project with @XilunZhangXZ@MatthewFoutter@Luis_A_Pabon@drmapavone
If you think arranging the balls into letters is too easy, I challenge you to come by our poster on Tuesday, Nov 10 at CoRL during lunch and see how many tries you need. We're bringing the mini pool table and it's actually trickier than you'd think.
Also, to hear why we think real2sim is not yet solved, read our full blog post here: https://t.co/1Q8AGs4yRw
Opus5.5 wrote its own controller to hit pool balls, built its own vision tools to track ball trajectories and chose actions that would better inform the sim dynamics reconstruction. In the end it was able to match the real pool table dynamics much better than visual reconstruction only.
If you are serious about robotics (aka Physical AI), hire students from the Robotics, Systems and Control masterโs at @ETH . In my opinion, it is THE strongest robotics program in the world:
โขcontrol,
โขoptimization,
โขperception,
โขlocomotion,
โขaerial robotics,
โขlearning,
โขsystems engineering, robot dynamics
If the student has GPA > 5.75, you can probably skip the technical interview.
๐ ๐ง๐ต๐ฒ ๐๐ฟ๐ฎ๐ป๐ฑ ๐๐ถ๐ป๐ฎ๐น๐ฒ: ๐ง๐ผ๐บ๐ผ๐ฟ๐ฟ๐ผ๐ ๐ถ๐ ๐๐ง๐ ๐ฅ๐ผ๐ฏ๐ผ๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐๐ฒ๐บ๐ผ ๐๐ฎ๐ '๐ฎ๐ฒ! ๐คโก
Tomorrow, the final showcase of our "Robot Learning: From Fundamentals to Foundation Models" course at @ETH is open to everyone, and you are invited to come see the results live! ๐
After mastering the fundamentals in the first half of the course in simulation, 300 students assembled 55 real-world SO-101 robots ๐ ๏ธ and took the leap to real hardwareโdeploying ๐ฉ๐๐๐, ๐ช๐ผ๐ฟ๐น๐ฑ ๐ ๐ผ๐ฑ๐ฒ๐น๐, ๐ฉ๐ถ๐ฑ๐ฒ๐ผ-๐๐ฐ๐๐ถ๐ผ๐ป ๐ ๐ผ๐ฑ๐ฒ๐น๐, ๐๐ถ๐ณ๐ณ๐๐๐ถ๐ผ๐ป and ๐ฅ๐ ๐ฝ๐ผ๐น๐ถ๐ฐ๐ถ๐ฒ๐ directly onto the physical robots they built with their own hands.
If you are in Zurich and want to see cutting-edge AI running on real hardware, drop by to see 40+ student teams tackle an incredible variety of live tasks. For example, teams will face semantic generalization with VLAs using the classic RT-2 benchmark task: instructing robots to "move the coke can to Taylor Swift." ๐ฅคโญ To truly evaluate zero-shot generalization, we have printed pictures of famous people that the teams won't even see until competition day! Other challenges the student's policies will tackle include cloth folding ๐, keyboard typing โจ๏ธ, singulating cluttered objects, poking, pushing and grasping.
The stakes are high: beyond counting towards their final course grades, the top teams will win the incredible opportunity to present their work to an evaluation committee featuring @ylecun , @SongShuran and @JitendraMalikCV on May 29th! ๐
๐ Event Details (Open to the Public)
๐ Date: Tomorrow, May 21, 2026
๐ Time: 08:00 โ 16:15
๐ข Location: ETH Main Building (Rรคmistrasse 101, 8006 Zรผrich | Foyer HG E Sรผd, HG EO Nord/Sรผd)
A massive thank you to our 14 dedicated TAs for their incredible support during this journey ๐ to @nvidia for the compute.
If you are around Zurich or the ETH campus tomorrow, come by the foyers to watch the live demos, chat with the students and see the future of robot learning in action!
๐ฃ ๐ง๐ต๐ฟ๐ถ๐น๐น๐ฒ๐ฑ ๐๐ผ ๐ต๐ผ๐๐ ๐๐ถ๐ฒ๐๐ฒ๐ฟ ๐๐ผ๐ (@fox_dieter17849) ๐ณ๐ผ๐ฟ ๐ฎ ๐ด๐๐ฒ๐๐ ๐น๐ฒ๐ฐ๐๐๐ฟ๐ฒ ๐ฎ๐ ๐๐ผ๐ฑ๐ฎ๐'๐ ๐๐ฒ๐๐๐ถ๐ผ๐ป ๐ผ๐ณ ๐ฅ๐ผ๐ฏ๐ผ๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด: ๐๐ฟ๐ผ๐บ ๐๐๐ป๐ฑ๐ฎ๐บ๐ฒ๐ป๐๐ฎ๐น๐ ๐๐ผ ๐๐ผ๐๐ป๐ฑ๐ฎ๐๐ถ๐ผ๐ป ๐ ๐ผ๐ฑ๐ฒ๐น๐ ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๐ @ETH!
Today's speaker almost needs no introduction. Dieter is Professor at the @UW and Senior Research Director at @allen_ai, co-author of the landmark textbook ๐ฃ๐ฟ๐ผ๐ฏ๐ฎ๐ฏ๐ถ๐น๐ถ๐๐๐ถ๐ฐ ๐ฅ๐ผ๐ฏ๐ผ๐๐ถ๐ฐ๐, Fellow of @IEEEorg, @RealAAAI, and @TheOfficialACM, and recipient of the ๐ฎ๐ฌ๐ฎ๐ฌ ๐๐๐๐ ๐ฃ๐ถ๐ผ๐ป๐ฒ๐ฒ๐ฟ ๐ถ๐ป ๐ฅ๐ผ๐ฏ๐ผ๐๐ถ๐ฐ๐ ๐ฎ๐ป๐ฑ ๐๐๐๐ผ๐บ๐ฎ๐๐ถ๐ผ๐ป ๐๐๐ฎ๐ฟ๐ฑ and the ๐ฎ๐ฌ๐ฎ๐ฏ ๐๐๐๐๐ ๐๐ผ๐ต๐ป ๐ ๐ฐ๐๐ฎ๐ฟ๐๐ต๐ ๐๐๐ฎ๐ฟ๐ฑ.
Today he will be talking about: "๐ง๐ผ๐๐ฎ๐ฟ๐ฑ ๐๐ผ๐๐ป๐ฑ๐ฎ๐๐ถ๐ผ๐ป ๐ ๐ผ๐ฑ๐ฒ๐น๐ ๐ณ๐ผ๐ฟ ๐๐ผ๐บ๐ฝ๐ฒ๐๐ฒ๐ป๐ ๐ฅ๐ผ๐ฏ๐ผ๐๐" ๐ค
His talk will cover large-scale data generation for robot manipulation, the role of simulation and sim-to-real transfer, and ongoing work at Ai2 on training models that are genuinely competent across a broad set of tasks โ moving beyond narrowly trained, disconnected skills ๐
If you are in Zรผrich and want to join us:
๐ Where: Room NO C 60, Sonneggstrasse 5
๐ When: Today, May 18, 17:00 โ 18:00
See you there!
๐ฃ ๐ง๐ต๐ฟ๐ถ๐น๐น๐ฒ๐ฑ ๐๐ผ ๐ต๐ผ๐๐ ๐๐๐ฐ๐ฎ๐ ๐๐ฒ๐๐ฒ๐ฟ (@giffmana) ๐ณ๐ผ๐ฟ ๐ฎ๐ป ๐ถ๐ป-๐ฝ๏ฟฝ๏ฟฝ๐ฟ๐๐ผ๐ป ๐ด๐๐ฒ๐๐ ๐น๐ฒ๐ฐ๐๐๐ฟ๐ฒ ๐ฎ๐ ๐๐ผ๐ฑ๐ฎ๐'๐ ๐๐ฒ๐๐๐ถ๐ผ๐ป ๐ผ๐ณ ๐ฅ๐ผ๐ฏ๐ผ๐ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด: ๐๐ฟ๐ผ๐บ ๐๐๐ป๐ฑ๐ฎ๐บ๐ฒ๐ป๐๐ฎ๐น๐ ๐๐ผ ๐๐ผ๐๐ป๐ฑ๐ฎ๐๐ถ๐ผ๐ป ๐ ๐ผ๐ฑ๐ฒ๐น๐ ๐ฎ๐ @ETH !
I will be starting the afternoon at 16:15 with a lecture on Frontier & Open Problems in robot learning and how to do research in the field, before handing over to Lucas at 17:15 to close out the session.
Lucas has been behind some of the most impactful work in computer vision and multimodal AI in recent years: co-author of the ๐ฉ๐ถ๐๐ถ๐ผ๐ป ๐ง๐ฟ๐ฎ๐ป๐๐ณ๐ผ๐ฟ๐บ๐ฒ๐ฟ (๐ฉ๐ถ๐ง), the landmark paper that helped establish transformers as a core architecture for image understanding, ๐ฆ๐ถ๐ด๐๐๐ฃ, the contrastive vision-language model that underpins much of today's multimodal AI, and ๐ฃ๐ฎ๐น๐ถ๐๐ฒ๐บ๐บ๐ฎ, @GoogleDeepMind's open vision-language model that has powered many VLAs in robotics. After @GoogleDeepMind and @OpenAI, he is now at @Meta @AIatMeta Superintelligence Labs. Today he will be talking about: "๐ฉ๐ถ๐๐ถ๐ผ๐ป ๐ถ๐ป ๐๐ต๐ฒ ๐๐ด๐ฒ ๐ผ๐ณ ๐๐๐ ๐ย " ๐
If you are in Zurich and want to join us:
๐ Where: Room NO C 60, Sonneggstrasse 5
๐ When: Today, May 11, 16:15 โ 18:00
See you there!
๐ ๐บ๐ถ๐บ๐ถ๐ฐ-๐๐ถ๐ฑ๐ฒ๐ผ ๐ต๐ฎ๐ ๐ฏ๐ฒ๐ฒ๐ป ๐ฎ๐ฐ๐ฐ๐ฒ๐ฝ๐๐ฒ๐ฑ ๐๐ผ #๐ฅ๐ฆ๐ฆ๐ฎ๐ฌ๐ฎ๐ฒ ๐ถ๐ป ๐ฆ๐๏ฟฝ๏ฟฝ๐ป๐ฒ๐!
Sharing this one with a lot of pride. This work was led by my two students, @jjonpai and @Liam862373. Both are Master's students at @ETH who I have the privilege of advising at @Microsoft, and TAs for my new robot learning course at ETHZ. Watching them take an ambitious idea and carry it all the way to RSS has been one of my highlights of the past year. They are going to do extraordinary things.
For those new to the work: mimic-video is a Video-Action Model that grounds robot policies in pretrained video models (NVIDIA Cosmos) instead of static image-text VLM backbones. The result is ๐ญ๐ฌ๐ ๐ฏ๐ฒ๐๏ฟฝ๏ฟฝ๐ฒ๐ฟ ๐๐ฎ๐บ๐ฝ๐น๐ฒ ๐ฒ๐ณ๐ณ๐ถ๐ฐ๐ถ๐ฒ๐ป๐ฐ๐ ๐and ๐ฎ๐ ๐ณ๐ฎ๐๐๐ฒ๐ฟ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด โฉthan standard VLAs, with performance that scales directly with video model quality, and no expensive video reconstruction at inference time.
And we have recently ๐ผ๐ฝ๐ฒ๐ป-๐๐ผ๐๐ฟ๐ฐ๐ฒ๐ฑ ๐ฒ๐๐ฒ๐ฟ๐๐๐ต๐ถ๐ป๐ด: code, models, the works. If you are building in robot learning, please take it, break it, build on it. We genuinely cannot wait to see where the community takes this.
Last but not least: huge shoutout to the team at @mimicrobotics, they brought world-class robot systems, deep dexterity expertise and a genuine research partnership that shaped this work from day one. This is exactly the kind of partnerships between @Microsoft and cutting-edge robotics startups that I believe moves the field fastest.
See you in Sydney! ๐ฆ๐บ
Code: https://t.co/mxqrIFa4Z1
Project: https://t.co/s1QWGVNzeV
Paper: https://t.co/ZLakdfOwyg
I am thrilled to host @QuanVng (Co-Founder of @physical_int ) as our Guest Spotlight at @ETH!
I will be starting the afternoon with a lecture on Generalist Robot Policies and how to train VLAs, before handing over to Quan to close out the session. Quan has been a central figure in the push toward large-scale pre-training for generalist robot policies. From co-leading the RT-X and Open X-Embodiment projects at Google DeepMind to his current work at Physical Intelligence, his efforts in scaling physical AI are defining the next frontier of robotics.
๐ง๐ต๐ฒ ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๐ถ๐๐๐ถ๐ผ๐ป: ๐๐๐๐ฒ๐บ๐ฏ๐น๐ถ๐ป๐ด ๐ฑ๐ฑ ๐ฟ๐ผ๐ฏ๐ผ๐๐ ๐๐ถ๐๐ต ๐ฏ๐ฌ๐ฌ ๐๐๐๐ฑ๐ฒ๐ป๐๐ ๐ถ๐ป ๐ท๐๐๐ ๐ฎ.๐ฑ ๐ต๐ผ๐๐ฟ๐ ๐คโก
Today, our @ETH Robot Learning course successfully assembled ๐ฑ๐ฑ ๐ฆ๐ข-๐ญ๐ฌ๐ญ ๐ฟ๐ผ๐ฏ๐ผ๐๐ robots to kick off the second half of the semester!
In parallel to my weekly lectures, we have spent the first weeks mastering the fundamentals of robot learning in simulation with homework assignments. Now we finally move to real robots. I am excited to see what the students will build as they spend the coming weeks implementing ๐ฉ๐๐๐, ๐ฉ๐ถ๐ฑ๐ฒ๐ผ-๐๐ฐ๐๐ถ๐ผ๐ป ๐ ๐ผ๐ฑ๐ฒ๐น๐, ๐ช๐ผ๐ฟ๐น๐ฑ ๐ ๐ผ๐ฑ๐ฒ๐น๐, ๐ฎ๐ป๐ฑ ๐ฅ๐ ๐ฝ๐ผ๐น๐ถ๐ฐ๐ถ๐ฒ๐ directly on real hardware!
Huge thanks to my TA team for making this hardware marathon possible. This is very much a collective effort: together, we are striving to build a scalable, hands-on robot learning course that brings cutting-edge research into the classroom ๐
If you missed @chichengcc's guest lecture on "Robotics: Beyond Algorithms" from my @ETH robot learning course, check it out on YouTube! He shares insights that are rarely taught & hard to learn in academia.
๐ฝ๏ธ YouTube: https://t.co/coRqOHHB3n
๐ Course: https://t.co/QJcfXJRfX8
Excited to welcome @ajwagenmaker for a Guest Spotlight to close today's session of my robot learning course at @ETH today!
He'll dive into "Robots That Learn From Experience" and how physical interaction scales robotics beyond imitation learning.
Excited to share the recording of @aviral_kumar2's guest lecture on "How to Replicate the LLM Recipe in Robot Learning" from my @ETH_en robot learning course on YouTube!
๐ฅ https://t.co/u0yCEe5mRv
๐ https://t.co/dL0gKA0StO
Excited to have @aviralku join us today for a Guest Spotlight to close today's session of my robot learning course at @ETH_en! He'll be tackling one of the biggest open questions in the field: "What Would It Take to Replicate the LLM Recipe in Robot Learning?"
Excited to share that by popular demand, the recording of @danfei_xu's guest lecture on "Human Data as a Foundation for Robot Learning" from my @ETH_en Robot Learning course is now public on YouTube!
๐ฅ https://t.co/TLycJ06UJd
๐ https://t.co/QJcfXJRfX8
Excited to have @abhishekunique7 join us today for a special Guest Spotlight to close today's lecture in my robot learning course!
If you are a robot learner in Zรผrich, please join us:
๐ Where: Room NO C 60, Sonneggstrasse 5, 8092 Zรผrich
๐ When: Today, Feb 23, 16:15 - 18:00