Jona Ruthardt

AI Research | PhD student @ Fundamental AI Lab | Vision-Language Modelling and Multi-Modal Learning

[Jona Ruthardt @jonaruthardt]

What if you could 𝘵𝘦𝘭𝘭 vision encoders 𝘸𝘩𝘢𝘵 to encode?

Check out our latest work where we introduce 𝗦𝘁𝗲𝗲𝗿𝗮𝗯𝗹𝗲 𝗩𝗶𝘀𝘂𝗮𝗹 𝗥𝗲𝗽𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀.

[Manu Gaur @gaur_manu]

Pretrained ViTs like DINOv2 or CLIP are great, but they produce fixed, generic representations that encode the most salient visual concepts (e.g., "cat").
In human vision, prior priming with language changes how people parse an image. We believe visual encoders should do the same
🚨 Introducing Steerable Visual Representations, a new family of visual features you can steer with text towards specific visual concepts.

[Yuki @y_m_asano]

[🔁🚨PhD opening]

We have a new opening for a PhD student at our @FunAILab at the University of Technology Nuremberg on combining Vision-Language-Action Models with 3D Geometry.

This PhD position is in close collaboration with @HelsingAI with the amazing researcher @TomUnterthiner and will be a unique opportunity to conduct research and have impact.

Please see below for more information:
https://t.co/9FU8ryPTPN

We particularly encourage applications from people of diverse and underrepresented backgrounds. If you have any doubt if you should apply or not, send me a message.

[Massimiliano Viola @massiviola01]

Pretrained ViTs are powerful but completely static.

Run a forward pass a thousand times, and you always get the same result.

But what if you could tell the model what to look for? What if you could adapt the representation to encode attributes relevant to your task?

[Jona Ruthardt @jonaruthardt]

One thing we ran into is that representational similarity metrics can be hard to interpret.

With ShareLock, we explored the PRH from a complementary direction by instead training small alignment probes between frozen vision and language backbones. We strictly control which classes are seen during training and then test alignment via generalization to entirely novel concepts.

Not the full answer to the Platonic/Aristotelian debate, but a useful complementary test.

https://t.co/1gEsejX6LP

[Hirokatsu Kataoka | 片岡裕雄 @HirokatuKataoka]

[LIMIT Workshop has been accepted at #ECCV2026 ]

LIMIT: Representation Learning with Very Limited Resources explores how we can build powerful AI systems when data, labels, modalities, compute, or supervision are limited.

Join us at LIMIT 2026!

Website: https://t.co/Ayx5bMvv6j⁠

[Jona Ruthardt @jonaruthardt]

Today, I had the chance to present our latest work on 𝗦𝘁𝗲𝗲𝗿𝗮𝗯𝗹𝗲 𝗩𝗶𝘀𝘂𝗮𝗹 𝗥𝗲𝗽𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀 at @SonyAI_global in their Tokyo office.

This wraps up our @FunAILab visit to Japan. Grateful for the engaging discussions and new connections along the way.

[Jona Ruthardt @jonaruthardt]

@nikparth1 @y_m_asano Indeed, using SteerViT as the vision encoder of a MLLM is a promising direction (esp. from an efficiency angle).

To your 2nd point: steerability improves when scaling MLLMs (e.g. 88.2 on CORE for InternVL3.5-8B). But representation quality remains below vision-centric encoders.

[Jona Ruthardt @jonaruthardt]

@_rabiulawal @gaur_manu Thanks for the pointer! We ran your slot-attention model on our benchmarks too. On CORE, it definitely shows steerability (59.0 vs 40.6 for DINOv2-S) but doesn't quite reach SteerViT at 93.6. Also linear probing accuracy drops by 17.3 vs. 9.1 (SteerViT) compared to vanilla DINOv2

[Jona Ruthardt @jonaruthardt]

@qu3tzalify @giffmana @gaur_manu But it goes further: SteerViT also steers the dense representations. Consider this example where specifying a certain person leads to clear separation from other people in the PCA feature visualization. This helps semantic discrimination in downstream tasks (e.g., segmentation).

[Jona Ruthardt @jonaruthardt]

@qu3tzalify @giffmana @gaur_manu True, DINO patch features also encode non-salient parts. But if your task requires one object-level embedding, you'd have to build a pooling pipeline (e.g., with SAM) to determine which local embeddings to consider.
SteerViT does it implicitly within the vision encoder itself.

[Jona Ruthardt @jonaruthardt]

@MHR7DYN @gaur_manu The SteerViT variant used for most experiments in the paper builds directly on top of a frozen DINOv2. Therefore, comparing these two models best shows the capabilities of our method.
But as SteerViT is backbone-agnostic, it is easily possible to train a variant for DINOv3 too.

[Jona Ruthardt @jonaruthardt]

@lsyuan0322 @gaur_manu Have a look at our interactive demo: https://t.co/JSbyeJPzOw
There's an example with a cat.

[Yuki @y_m_asano]

Steerable Visual Representations. 👇

From @jonaruthardt, @gaur_manu, @RamananDeva, @MakarandTapaswi and me :). More Infos soon.

[DailyPapers @HuggingPapers]

Steerable Visual Representations

SteerViT lets you control Vision Transformers with natural language.

By injecting text directly into the encoder via lightweight cross-attention, you can steer attention toward any object while preserving representation quality.

[Jona Ruthardt @jonaruthardt]

Curious to learn more?

📝 Paper: https://t.co/UwC7yuAUUc

🌐 Project Page: https://t.co/Sjva0hj5VV

💻 Code: https://t.co/n3f3dBQP58

[Jona Ruthardt @jonaruthardt]

Ever asked yourself: 𝙃𝙤𝙬 𝙙𝙤 𝙩𝙚𝙭𝙩-𝙤𝙣𝙡𝙮 𝙇𝙇𝙈𝙨 "𝙨𝙚𝙚" 𝙩𝙝𝙚 𝙫𝙞𝙨𝙪𝙖𝙡 𝙬𝙤𝙧𝙡𝙙?

Our new TMLR paper finds that language capability predicts visual alignment of LLM features (r=0.768).

This means: LLM progress boosts vision(-language) models as well.

🧵⬇️

[Jona Ruthardt @jonaruthardt]

Big thanks to @gjburghouts , @SergeBelongie, and @y_m_asano for the collaboration and guidance throughout this project.