Ashutosh Adhikari

PhD student at ILCC, University of Edinburgh.
Prev @Microsoft and @mila and @UWaterlooCS.
Views my own!

[Ashutosh Adhikari @AshutoshAd63687]

VLMs benefit from tool calls as they allow for "thinking with images".

But can a VLM simply internalise the effects of calling correct tools in the latent space and not need to explicitly call them during inference? 🧵

Presenting tomorrow at 11 AM @COLM_conf !

[Ashutosh Adhikari @AshutoshAd63687]

@jomulr @COLM_conf Hi Jonas, I’m still here at COLM! Would be keen to chat more as I have been hearing about the work’s implications for VLAs

[Ashutosh Adhikari @AshutoshAd63687]

VLMs benefit from tool calls as they allow for "thinking with images".

But can a VLM simply internalise the effects of calling correct tools in the latent space and not need to explicitly call them during inference? 🧵

Presenting tomorrow at 11 AM @COLM_conf !

[Ashutosh Adhikari @AshutoshAd63687]

Paper: https://t.co/ItDgotHYtU
Code: https://t.co/7xUsi2O29B

Joint work with my amazing advisor @mlapata at @EdinburghNLP !

[Ashutosh Adhikari @AshutoshAd63687]

With zero tool calls, PEARL improves over SFT and matches complex alternatives (avg. acc., Qwen2.5-VL-7B):
- vs SFT on the same data: 59.7 → 64.1 (Latent Visual Reasoning), 51.0 → 64.7 (ThinkMorphData-SFT)
- vs LVR: 63.1 → 64.1
- vs PixelReasoner, which calls tools: 61.8 → 63.1

[Ashutosh Adhikari @AshutoshAd63687]

Unlike prior latent methods, it handles multiple tool calls, and the gains carry over to Qwen2.5-VL-3B and Qwen3-VL-4B.

And its predictive embeddings are descriptive: closer to the correct trajectory than to a visually or textually corrupted one.

[Ashutosh Adhikari @AshutoshAd63687]

Inspired by LLM-JEPA, PEARL treats image + question (view 1) and the expert's tool-use trajectory (view 2) as two views to learn predictive embeddings to internalise tool calls.

In order to get descriptive representations of the views we introduce learning to generate hidden representations that are belief state that can aid in avoiding collapse when using jepa-loss.

We also maintain cross-entropy loss on the answer tokens to allow the model to maintain the text-generation capabilities.

[Ashutosh Adhikari @AshutoshAd63687]

Prior approaches to multimodal latent reasoning use reconstruction loss to generate latent tokens that mimic the outputs of the tool calls.

However, we find identify several drawbacks to them:
1. Auto-regressively generating latent tokens doesn't actually generate images owing to training-inference mismatch of number of latent tokens used (most work use only a small fixed number of latent tokens during inference but were trained with longer latent reasoning traces).

2. They are limited to internalising only single-step trajectories, but what about trajectories that contain multiple sequential tool calls?

3. More latents during inference don't help, and they barely tell the right trajectory from a corrupted one! ⚠️

As a result, they aren't reconstructing but simply embedding! So why not just embed directly?

[Ashutosh Adhikari @AshutoshAd63687]

hello world! Had my previous account compromised. This is my new account!

[Alex Gurung @ EMNLP 🇭🇺 @AlexAag1234]

Did you know training on your own incorrect samples can beat distilling from a much larger teacher?

Traditional RFT only reinforces a narrow set of approaches. Strategically sampling for approach-level diversity gives better pass@k, test-time scaling and RL initialization!
https://t.co/pONJCHEO58
🧵

[Machine Learning (ML) Papers @Memoirs]

Multimodal Latent Reasoning via Predictive Embeddings

Ashutosh Adhikari, Mirella Lapata
https://t.co/bZt5puUQU3 [𝚌𝚜.𝙻𝙶]