SAI

A global platform for reinforcement learning. Compete on standardized environments, share your models, and advance the state of the art.

[SAI @CompeteSai]

We’re officially opening up access to SAI – a new platform for reinforcement learning.

Submit models, compete in structured challenges, and learn from others — anytime, not just during a conference.

Try out SAI now -> 🌐 https://t.co/dqScdtNRLI

Documentation -> 📄 https://t.co/FxYDAchQYX

[SAI @CompeteSai]

@sihing_guppy full write-up: https://t.co/jjged9kI0b

[SAI @CompeteSai]

@sihing_guppy More works need to be done to ensure robustness over low contrast perturbation.

[SAI @CompeteSai]

@sihing_guppy The technical blog series is here: https://t.co/fTkTgPZJS0

[SAI @CompeteSai]

@realmc @sihing_guppy can't wait for part 3!

[SAI @CompeteSai]

@sihing_guppy Full technical blog and interactive perturbation UX:
https://t.co/fTkTgPZJS0

[SAI @CompeteSai]

@sihing_guppy Line of the Week: many models rely on appearance far more than they admit.

[SiHing Guppy @sihing_guppy]

https://t.co/Lw3jTrEtgN

[SAI @CompeteSai]

@sihing_guppy Performance should not be stable across all those perturbation variations, unless they are not looking at the language input.

[SAI @CompeteSai]

@realmc 让华语世界听见我们的声音!

[Malcolm Chan | NRN @realmc]

https://t.co/FKGTnqN7tk

[SAI @CompeteSai]

@sihing_guppy blog: https://t.co/TTWC1cgJM4

[SAI @CompeteSai]

@sihing_guppy We ran this on a fine-tuned OpenPI policy on LIBERO Spatial. The full perturbation analysis covers 11 language corruption types:
https://t.co/TTWC1cgJM4

https://t.co/3tt9xg2bGZ

[SiHing Guppy @sihing_guppy]

https://t.co/B8rLHPZ7O0

[SiHing Guppy @sihing_guppy]

What happens when you remove a robot's ability to read its instructions? Almost nothing.

> Full model → 95% success
> Remove language → 94% (▼1%)
> Remove vision → 13% (▼82%)

Near-blind without vision. Near-indifferent to language.
If your evaluation only tests correct instructions, you're not measuring language. You're measuring vision.

[SiHing Guppy @sihing_guppy]

A robot receives a language command "pick up the black bowl and place it on the plate" and executes it. We replaced the command with: "My name is Franka."

No task. No object. No action verb. It picked up the bowl and placed it on the plate. The language instruction isn't being read.

The scene is being acted upon. The prompt is decoration.

[SAI @CompeteSai]

@sihing_guppy Full analysis with 11 perturbation types across 3 severity levels, including why the flat sensitivity curve is the real finding:
https://t.co/TTWC1cgJM4

[SAI @CompeteSai]

@realmc @sihing_guppy

[SiHing Guppy @sihing_guppy]

https://t.co/B8rLHPZ7O0

[SAI @CompeteSai]

We’ll be at @NVIDIAGTC next week in San Francisco.

@sihing_guppy and @MarcAlloul will be there talking about model transparency and evaluation for robotic policies: how to understand what your models are actually doing before you deploy them.

If you’re building or deploying robotic systems, send us a dm or come find us. We’d love to chat.

[SAI @CompeteSai]

@realmc @sihing_guppy My intern told me he caught winds of some huge study results waiting to be revealed.