Joy Hsu

MTS @OpenAI. Incoming Assistant Professor @ImperialCollege. PhD @Stanford.

[Joy Hsu @joycjhsu]

What’s left w/ foundation models? We found that they still can't ground modular concepts across domains.

We present Logic-Enhanced FMs:🤝FMs & neuro-symbolic concept learners. We learn abstractions of concepts like “left” across domains & do domain-independent reasoning w/ LLMs.

[Emily Jin @emilyzjin]

During long-horizon task planning, a robot must decide what to do, while ensuring each action is geometrically feasible.

To this end, we propose 𝗽𝗹𝗮𝗻𝗻𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗶𝗻𝘁𝗲𝗿𝗹𝗲𝗮𝘃𝗲𝗱 𝘃𝗶𝘀𝗶𝗼𝗻-𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀.

[Kenny Jones @RKennyJones]

The Visual Concepts Workshop is coming soon to @CVPR 2026!

Join us on June 4 for an exciting discussion on all aspects of visual concepts: from discovery and reasoning, to controllable generation, robotics, and more!

Hope to see you in Denver! #CVPR

[Chuhan Li @_Chuhan_Li]

Human perception is inherently situated – we understand the world relative to our own body, viewpoint, and motion.

To deploy multimodal foundation models in embodied settings, we ask:

“Can these models reason in the same observer-centric way?”

We study this through SAW-Bench: a novel benchmark for observer-centric situated awareness:
- 786 real world egocentric videos
- 2,071 human-annotated QA pairs

Across all tasks, we evaluate 24 state-of-the-art MFMs:
�� Best model: 53.9%
🧑 Humans: 91.6%

Models systematically:
❌ Confuse head rotation with physical movement
❌ Collapse under multi-turn trajectories
❌ Fail to maintain persistent world-state memory

👉 We see that maintaining a stable observer-centric representation remains challenging.
As MFMs are increasingly integrated into embodied agents, situated awareness becomes essential for reliable real-world interaction.

We release SAW-Bench and encourage further research toward improving observer-centric reasoning in multimodal foundation models.

[Joy Hsu @joycjhsu]

Happy to bring back the 3rd Workshop on Visual Concepts at @CVPR 2026! Call for papers is now open.

We welcome submissions on the following topics. See our website for more info: https://t.co/LKJxxO5ZGQ

Join us in Denver!

[Joy Hsu @joycjhsu]

Thanks, Lucas! Big fan of your work and very excited about the next generation of large-scale neuro-symbolic systems…

[Lucas Beyer (bl16) @giffmana]

Works by @joycjhsu are nice modern neurosymbolic incarnations which make sense:

https://t.co/By2Qe1x0yK

[Joy Hsu @joycjhsu]

@GabrielPoesia Congratulations!

[Joy Hsu @joycjhsu]

We'll be presenting Deep Schema Grounding at @iclr_conf 🇸🇬 on Thursday (session 1 #98). Come chat about abstract visual concepts, structured decomposition, & what makes a maze a maze!

& test your models on our challenging Visual Abstractions Benchmark: https://t.co/UnMIoJHN8a

[Joy Hsu @joycjhsu]

What makes a maze look like a maze?

Humans can reason about infinitely many instantiations of mazes—made of candy canes, sticks, icing, yarn, etc.

But VLMs often struggle to make sense of such visual abstractions.

We improve VLMs' ability to interpret these abstract concepts.

[Joy Hsu @joycjhsu]

Come chat with us about few-shot state classification & predicate hierarchies at #ICLR2025 poster session 1!

[Emily Jin @emilyzjin]

State classification of objects and their relations (e.g. the cup is next to the plate) is core to many tasks like robot planning and manipulation.

But dynamic real-world environments often require models to generalize to novel predicates from few examples.

We present PHIER, a method that leverages predicate hierarchies to effectively generalize in few-shot scenarios.

[Joy Hsu @joycjhsu]

Excited to bring back the 2nd Workshop on Visual Concepts at @CVPR 2025, this time with a call for papers!

We welcome submissions on the following topics. See our website for more info:
https://t.co/gk0NgYAcEx

Join us & a fantastic lineup of speakers in Tennessee!

[Joy Hsu @joycjhsu]

We're organizing the first Workshop on Visual Concepts at @eccvconf with an incredible lineup of speakers!

Join us on Sep 30 afternoon at Suite 3, MiCo Milano 🇮🇹

https://t.co/lnyaci4orM

[Joy Hsu @joycjhsu]

Work done with the wonderful @maojiayuan, Josh Tenenbaum, @noahdgoodman, and @jiajunwu_cs!

Paper: https://t.co/bvLFJEldnF
Website: https://t.co/zP3dpzkufL

[Joy Hsu @joycjhsu]

What makes a maze look like a maze?

Humans can reason about infinitely many instantiations of mazes—made of candy canes, sticks, icing, yarn, etc.

But VLMs often struggle to make sense of such visual abstractions.

We improve VLMs' ability to interpret these abstract concepts.

[Joy Hsu @joycjhsu]

DSG improves GPT-4V by 5.4 percent point overall on VAD (↑ 8.3% relative improvement), and by 6.7 percent point (↑ 11.0% relative improvement) on questions that involve counting.

We believe DSG is a promising step towards human-aligned understanding of visual abstractions!

[Joy Hsu @joycjhsu]

Will be presenting LARC at @CVPR next Thursday morning (session #3) with our fantastic intern @chunfeng3364, as well as @Weiyu_Liu_ and @jiajunwu_cs!

Paper: https://t.co/n8WgZ0mmwu
Website: https://t.co/L3HhfGwZrh
Code: https://t.co/dHIN7fCpxv

[Joy Hsu @joycjhsu]

How can we learn 3D visual grounding with natural supervision—only looking at QA pairs, without ground truth bounding boxes or object classification labels?

We inject explicit language priors, e.g., the symmetric property that A near B ⇒ B near A, in structured vision models.

[Joy Hsu @joycjhsu]

LARC shows strong data efficiency and generalization to new concepts & dataset, and is a promising step towards injecting structured visual reasoning frameworks with explicit language-based priors, for learning in settings without dense visual supervision.