Darina Koishigarina

PhD student at @uni_tue and IMPRS-IS

[Darina Koishigarina @kdariina]

Why do embedding models struggle with binding, a core requirement for multi-object understanding?

We find that objects don't decompose into simple components. Binding is possible, but requires going beyond the linear representation hypothesis.

Co-led with @a_uselis
🧵👇

[Darina Koishigarina @kdariina]

@a_rubique @a_uselis @coallaoh Could be interesting to @yihaoli_0302, @marthaflinders, @feng_jiahai, @JacobSteinhardt, @RaphiKang, @rim_assl, @EricElmoznino, @jacobandreas, @KAIST_AI.

[Darina Koishigarina @kdariina]

We then ask what kind of structure best approximates the binding function underlying the embeddings. Turns out it's a multiplicative one.

[Darina Koishigarina @kdariina]

@a_rubique @a_uselis @coallaoh If you want to know more and discuss, come to our poster at #ICML2026 🇰🇷 next week!

🗓️ Thu, Jul 9 • 2:30 PM – 4:15 PM KST
📍 Hall A #2903

[Darina Koishigarina @kdariina]

Interestingly, these models that have generalization also have low-complexity binding functions!

So there is a pattern:
CLIP doesn't generalize — has a high-complexity binding function
Models that generalize — have a low-complexity binding function

[Darina Koishigarina @kdariina]

Despite a large number of possible concept combinations, we observe binding generalization from a fraction of the observed objects to new objects.

[Darina Koishigarina @kdariina]

The object components are responsible for the uni-modal binding observed in prior work.

But if both image embeddings and text embeddings contain the binding information, why is that information not aligned? 🤔

[Darina Koishigarina @kdariina]

This led us to ask: is it even possible to have embedding models that generalize to unseen objects?

The answer is yes.

[Darina Koishigarina @kdariina]

🧩 If CLIP’s vision and text embeddings each rely on complex binding mechanisms that do not generalize to new objects, this could explain why their binding fails to align cross-modally.

[Darina Koishigarina @kdariina]

We use MLPs to approximate input embeddings (CLIP or DINO) from scene descriptions (inspired by @EricElmoznino's work on complexity).

Turns out those approximations encode concepts well.

But they do not generalize to unseen objects. Even after observing 90% of other objects.

[Darina Koishigarina @kdariina]

You can even edit the scene embeddings just from the single-object embeddings.

[Darina Koishigarina @kdariina]

Why do embedding models struggle with binding, a core requirement for multi-object understanding?

We find that objects don't decompose into simple components. Binding is possible, but requires going beyond the linear representation hypothesis.

Co-led with @a_uselis
🧵👇

[Darina Koishigarina @kdariina]

We find that CLIP embeddings have an approximately additive structure: scenes decompose into objects, and objects decompose into concepts.

But the object components are not fully explained by their concepts alone. They contain extra information that may be crucial for binding.

[Darina Koishigarina @kdariina]

First, we formalize binding.

Scenes contain objects. Objects are combinations of concepts.

A model exhibits binding if it can not only recognize the concepts present in a scene, but also identify which concepts belong together as the same object.