Nils Höhing

Vision Language Model evaluation / benchmarking | PhD student @ml_labs_irl @UCDCompSci

[Nils Höhing @nilshoehing]

@ariccio @thsottiaux yes very confusing that using a reset pushes back the next normal reset you get and now tibos reset isn't bankable...

[Nils Höhing @nilshoehing]

Small evaluation datasets are a benefit, not a downside. Running evals is expensive.

[Nils Höhing @nilshoehing]

Just noticed that using one of the new codex self-resets also resets the time until you get the next self-reset. I thought the idea was to make usage more plannable, but this just incentivises burning through all resets in one day. Is that intended @thsottiaux?

[Nils Höhing @nilshoehing]

In my experience not supervising agents yourself wastes a ton of tokens and time. Often you can see when a model goes off into the wrong direction, starts obsessing about minor details or ignores instructions. And human perception is still much better at catching visual bugs.

[Nils Höhing @nilshoehing]

Just noticed that using one of the new codex self-resets also resets the time until you get the next self-reset. I thought the idea was to make usage more plannable, but this just incentivises burning through all resets in one day. Is that intended @thsottiaux?

[Mayug Maniparambil @ ICML 26 @mayugmm]

Excited to share that our paper, TopoBench: Benchmarking LLMs on Hard Topological Reasoning, has been accepted at the #ICLR2026 Workshop on Logical Reasoning of Large Language Models. Nils Hoehing will be presenting it at the workshop.

Many real-world tasks require maintaining global spatial constraints through a sequence of local updates: for example in circuit layout, route planning, and molecular structure analysis. We introduce TopoBench, a benchmark for topological reasoning across 6 puzzle families and 3 difficulty levels, designed to study how well LLMs handle global spatial constraints such as connectivity, loop closure, symmetry, and visibility.
Our main finding is that even frontier reasoning models solve fewer than 25% of hard instances, and two puzzle families are nearly unsolved. So hard topological reasoning still appears to be a real challenge, even for very strong models.
To understand why, we annotated 750 chain-of-thought traces and built an error taxonomy. The main breakdowns looked less like a failure of abstract reasoning alone, and more like failures to extract, maintain, and update the right spatial constraints while solving the puzzle.
We also tested mitigation strategies, including cell-aligned representations and tool-based structured constraint checking. These help, but they point to the same conclusion: the main bottleneck seems to lie less in reasoning over constraints once available, and more in reliably deriving them from spatial representations in the first place.
Paper: https://t.co/RYGdiRJkGw
Website: https://t.co/OeINymKegR

Happy to hear others’ thoughts and discuss :)

Work with @nilshoehing , @JanakKapuriya , @ArjunK018 , Ellen Rushe, Anthony Ventresque, @oconnorn , and @fergal_reid .

We also want to thank Intercom Research @intercom for supporting this work.

#ICLR2026

[Nils Höhing @nilshoehing]

We just released RocketScience, our new Spatial Understanding Benchmark for VLMs. It is contrastive, made of entirely new real-world data and quite diverse in the spatial layouts. It is designed to be super easy for humans but still difficult for many models. 1/3

[Nils Höhing @nilshoehing]

This was joint work with: @mayugmm, Ellen Rushe, @oconnorn and @AnthoVentresque. 3/3

[Nils Höhing @nilshoehing]

We just released RocketScience, our new Spatial Understanding Benchmark for VLMs. It is contrastive, made of entirely new real-world data and quite diverse in the spatial layouts. It is designed to be super easy for humans but still difficult for many models. 1/3

[Nils Höhing @nilshoehing]

We found that most basic VLMs struggle, chain-of-thought prompting helps improve performance and top reasoning models do well.
Code: https://t.co/Qs26uk2c5j
Data: https://t.co/WydIy51nD9
Paper: https://t.co/0cxrHv0uHH 2/3

[Nils Höhing @nilshoehing]

@LiJunnan0409 Would love to talk to you!

[Nils Höhing @nilshoehing]

@giffmana And Cap only has a slightly higher performance than the blind model on the VG-Relations (Rel.) part.

[Nils Höhing @nilshoehing]

@giffmana Cool work! I would be interested in the “A left of B” results for Cap. (Since you already mentioned that ;)) Because the blind decoder should only be able to learn relations with a common direction (“person on bike” not “bike on person”) but not the left vs right distinction.

[Nils Höhing @nilshoehing]

@srchvrs @giffmana @SashaMTL Also the idea of using the current sota model to assess the capabilities of future models just makes no sense

[Nils Höhing @nilshoehing]

@jackclarkSF

[Nils Höhing @nilshoehing]

I am at 77 followers now, which is exactly CLIP's context length.

[Nils Höhing @nilshoehing]

"Thanks Bro"-Prompting for Vision and Language Learning (BLIP-2 on @huggingface)

[Nils Höhing @nilshoehing]

@askerlee @srchvrs But then the nlu of the small encoder bottlenecks the performance of the whole thing. So you won‘t get to make much use of the power of the big model.

[Nils Höhing @nilshoehing]

@askerlee @srchvrs How do you find the semantic duplicates without using an LM?

[PyTorch @PyTorch]

Today we’re releasing TorchMultimodal, a new PyTorch library containing models, building blocks and end to end examples for training and evaluating SoTA multimodal models.

📖 Learn more about scaling multimodal foundation models in the PyTorch Ecosystem: https://t.co/CPR1MA9hSx