Jad Abou-Chakra

Structured World Models and Robot Learning | QUT Centre for Robotics

[Jad Abou-Chakra @jadachakra]

1/ Real2Sim and Sim2Real are well-known paradigms in robotics🤖. Is there another paradigm where sim and the real world are more tightly interlinked? We introduce Real-IS-Sim: a framework featuring an always-in-the-loop, correctable simulator for behavior cloning policies.

[Jad Abou-Chakra @jadachakra]

Grateful to @NVIDIA for featuring our work on their Tech Blog: Physically Embodied Gaussians, an interpretable real time world model for #robotics using Warp and #GaussianSplatting. https://t.co/fscnMz21Ie

@krshnrana @nikoSuenderhauf @QUTRobotics

[Jad Abou-Chakra @jadachakra]

@siddancha Thats right

[Jad Abou-Chakra @jadachakra]

1/ Real2Sim and Sim2Real are well-known paradigms in robotics🤖. Is there another paradigm where sim and the real world are more tightly interlinked? We introduce Real-IS-Sim: a framework featuring an always-in-the-loop, correctable simulator for behavior cloning policies.

[Jad Abou-Chakra @jadachakra]

@siddancha Great question! Two reasons: 1. The cameras are not perfectly calibrated - therefore the tblock’s optimal position that would reduce the photometric loss is always contested. 2. Visual forces are somewhat naively implemented and cause oscillations because there is no damping.

[Jad Abou-Chakra @jadachakra]

🙏 Supported by @rai_inst
#NeuralRendering #PhysicsSim #Robotics #ComputerVision #GaussianSplatting

[Jad Abou-Chakra @jadachakra]

🚀 Embodied Gaussians is now opensource! 🔥

📂 GitHub: https://t.co/czorcxcRKN

✨ Key Features:

1️⃣ Gaussian Splatting (GSplat) + Physics (Warp) 💥
2️⃣ Parallelizable environments! ⚡
3️⃣ Render virtual cameras in parallel 📸

🔜 Coming Soon (hopefully):

🌀 Deformables!

[Jad Abou-Chakra @jadachakra]

6/ Videos and paper are available at https://t.co/l1IAt1R5Xo and https://t.co/xBMvLMYZ3K

[Jad Abou-Chakra @jadachakra]

5/ This project was completed during my internship at @rai_inst (formerly Boston Dynamics AI). Special thanks to my team—Lingfeng Sun, @krshnrana, Brandon May, Karl Schmeckpeper, Maria Vittoria Minniti, and Laura Herlant for their support during this project.

[Jad Abou-Chakra @jadachakra]

@qunomaan Thank you @qunomaan :) The artifacts are largely due to the use of only 5 viewpoints during initialization. Parts of the objects may be unobserved at the start. A better initialization procedure and a means of correcting modeling errors on the fly are on our future hitlist.

[Jad Abou-Chakra @jadachakra]

1/ What is a good representation of the physical world for a robot🤖? We believe it is one that is: initialized easily, 3D, physically constrained, correctable with new observations (in realtime), and capable of being forward simulated (faster than realtime).

[Jad Abou-Chakra @jadachakra]

5/ More videos and details are available at the project page (https://t.co/RFLdSinoxM) and in the paper (https://t.co/zREG8XP4Yq).

A special thanks to @krshnrana, Feras Dayoub, @nikoSuenderhauf, and @QUTRobotics for their support during this project.

[Jad Abou-Chakra @jadachakra]

4/ Particles collide with each other and are affected by external forces and kinematic constraints. The particles transport their linked Gaussians along with them. The system runs in realtime @ 30Hz with only 3 cameras. It works on both deformables and rigid objects.

[Krishan Rana @krshnrana]

📌How can we scale imitation learning to long-horizon, multi-object tasks & generalise (spatial & intra-category) from only 10 demos?

✨Introducing Affordance-Centric Policy Decomposition

💡Relative task-frame diffusion policies that self-chain for seamless operation

🧵👇