Yu Wang

CS PhD student @UW-Madison | Previous SonyAI Research Intern
Homepage:

[Yu Wang @rain305f]

Excited to share our #ACL2026 paper: “Why Multimodal In-Context Learning Lags Behind?” 🧵
w/ Sharon Li @SharonYixuanLi

Paper: https://t.co/VdH2B0mmcM
Code: https://t.co/YrZBtHCpfN

Multimodal in-context learning sounds like a natural extension of text-only ICL: give a model a few image-text demonstrations, and it should infer the task and apply it to a new query.

But in practice, I’m often puzzled by a simple question: why does few-shot multimodal ICL sometimes perform much worse than text-only ICL, even when the task structure is exactly the same?

To answer this, we built a controlled framework to compare text-only and multimodal ICL under identical task formulations. The finding is striking: while zero-shot performance can be comparable across modalities, multimodal few-shot demonstrations often lead to clear degradation.

So what actually goes wrong inside the model?

We decompose multimodal ICL into two stages:

1️⃣ Task Mapping Construction
The model needs to infer the task rule from demonstrations.

2️⃣ Task Mapping Transfer
The model needs to apply the inferred rule to the query.

🤔 The counterintuitive finding: current MLLMs can often construct the right task mapping from multimodal demonstrations, but fail to reliably transfer it to query-time reasoning.

In the middle layers, demonstration labels become grounded to the correct visual evidence. This suggests that the model does understand the demonstrations and can form the task mapping internally.

However, this mapping does not reliably propagate to later-layer reasoning. The final prediction is often driven more by query-side perceptual cues than by the task rule inferred from demonstrations.

In other words, multimodal ICL is not failing simply because the model “doesn’t see” the demonstrations. The deeper issue is a perception-reasoning misalignment: visual task mappings are formed in intermediate layers, but are not stably reused during final reasoning.

Other key insights:

→ Multimodal ICL errors come from both false task recognition and correct task recognition but incorrect answer prediction.

→ Demonstration-level visual grounding is necessary, but not sufficient, for successful multimodal ICL.

→ Later layers increasingly rely on unstable or query-dominant perceptual signals, weakening the influence of demonstration-induced task mappings.

→ We propose Mapping-Guided Inference (MGI), a lightweight inference-time intervention that reinforces task mapping transfer from demonstrations to the query.

The takeaway: understanding multimodal ICL cannot stop at output accuracy. The internal dynamics reveal a richer story about when MLLMs are actually using multimodal demonstrations, when they merely construct the right mapping, and when that mapping fails to guide final reasoning.

Huge thanks to my wonderful advisor @SharonYixuanLi and all collaborators for their support and contributions to this work!

I’ll be attending ACL in San Diego from July 1–7. I’m currently working on UMM interpretability and post-training, and would love to connect with anyone interested in MLLMs/UMMs, multimodal reasoning, and generation.

Feel free to reach out — happy to chat!

[MiniMax (official) @MiniMax_AI]

MiniMax H3: Omni-Reference, Commercial-Grade Generation, Unbeatable Cost Efficiency, Open Weights https://t.co/DLB1xsFfSC

[Samuel (Min-Hsuan) Yeh @Samuel861025]

Introducing our new work!

"Tracing Agentic Failure from the Flow of Success"
with Yiwen Zhu, @ShaleenDeep, @SharonYixuanLi
🔗https://t.co/TF8ipIlAdO

Which step broke your agent? Existing methods are slow, expensive, or need annotations you don't have. There's a better way. 🧵

[Hyeong-Kyu Froilan Choi @HyeonggyuC]

🔥 Introducing our newest work!

“𝐌𝐮𝐥𝐭𝐢-𝐀𝐠𝐞𝐧𝐭 𝐋𝐋𝐌𝐬 𝐅𝐚𝐢𝐥 𝐭𝐨 𝐄𝐱𝐩𝐥𝐨𝐫𝐞 𝐄𝐚𝐜𝐡 𝐎𝐭𝐡𝐞𝐫”
with wonderful collaborators, @JiatongLi0418 , @Wendi_Li_ , @xwang_lk , and @SharonYixuanLi
🔗 https://t.co/0mvydGxSAq

If you find our work interesting, please support us on huggingface!
🤗 https://t.co/Bopf4YRYn6

[Changdae Oh @Changdae_Oh]

Outcome reward models: cheap, but vulnerable to spurious shortcuts 😣
Process reward models (PRMs): robust, but too expensive to build from scratch 😫

What if you could get a ready-to-use PRM right after any RL post-training?
Introducing 'Progress Advantage' 🧵

[Yu Wang @rain305f]

Great question — I think both matter, but our evidence points more to the architecture/interface side.

MLLMs can often construct the right visual task mapping in middle layers, but this mapping is not reliably used in later query-time reasoning. So it is not just a data-density issue; the model may “see” the visual evidence but still fail to believe/use it ( https://t.co/36TfcrisKR also discuss this phenomenon).

More multimodal ICL-style data would likely help, but I think we also need better mechanisms for visual information to participate in reasoning, e.g., moving beyond LLM-centric text-token reasoning toward “thinking with images”-style approaches.

[Yu Wang @rain305f]

Excited to share our #ACL2026 paper: “Why Multimodal In-Context Learning Lags Behind?” 🧵
w/ Sharon Li @SharonYixuanLi

Paper: https://t.co/VdH2B0mmcM
Code: https://t.co/YrZBtHCpfN

Multimodal in-context learning sounds like a natural extension of text-only ICL: give a model a few image-text demonstrations, and it should infer the task and apply it to a new query.

But in practice, I’m often puzzled by a simple question: why does few-shot multimodal ICL sometimes perform much worse than text-only ICL, even when the task structure is exactly the same?

To answer this, we built a controlled framework to compare text-only and multimodal ICL under identical task formulations. The finding is striking: while zero-shot performance can be comparable across modalities, multimodal few-shot demonstrations often lead to clear degradation.

So what actually goes wrong inside the model?

We decompose multimodal ICL into two stages:

1️⃣ Task Mapping Construction
The model needs to infer the task rule from demonstrations.

2️⃣ Task Mapping Transfer
The model needs to apply the inferred rule to the query.

🤔 The counterintuitive finding: current MLLMs can often construct the right task mapping from multimodal demonstrations, but fail to reliably transfer it to query-time reasoning.

In the middle layers, demonstration labels become grounded to the correct visual evidence. This suggests that the model does understand the demonstrations and can form the task mapping internally.

However, this mapping does not reliably propagate to later-layer reasoning. The final prediction is often driven more by query-side perceptual cues than by the task rule inferred from demonstrations.

In other words, multimodal ICL is not failing simply because the model “doesn’t see” the demonstrations. The deeper issue is a perception-reasoning misalignment: visual task mappings are formed in intermediate layers, but are not stably reused during final reasoning.

Other key insights:

→ Multimodal ICL errors come from both false task recognition and correct task recognition but incorrect answer prediction.

→ Demonstration-level visual grounding is necessary, but not sufficient, for successful multimodal ICL.

→ Later layers increasingly rely on unstable or query-dominant perceptual signals, weakening the influence of demonstration-induced task mappings.

→ We propose Mapping-Guided Inference (MGI), a lightweight inference-time intervention that reinforces task mapping transfer from demonstrations to the query.

The takeaway: understanding multimodal ICL cannot stop at output accuracy. The internal dynamics reveal a richer story about when MLLMs are actually using multimodal demonstrations, when they merely construct the right mapping, and when that mapping fails to guide final reasoning.

Huge thanks to my wonderful advisor @SharonYixuanLi and all collaborators for their support and contributions to this work!

I’ll be attending ACL in San Diego from July 1–7. I’m currently working on UMM interpretability and post-training, and would love to connect with anyone interested in MLLMs/UMMs, multimodal reasoning, and generation.

Feel free to reach out — happy to chat!

[Seongheon Park @seongheon_96]

🚀 Excited to share our ACL 2026 paper:

VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation

LVLMs can be confidently wrong when relying on language priors instead of the image. VAUQ checks whether model confidence is truly grounded in visual evidence 👀

[Sharon Li @SharonYixuanLi]

When evaluating LVLMs, should we really be asking:
“Did the model get the right answer?”
or rather
“Did the model truly integrate the visual input?”

LVLMs can rely on shortcuts learned from the underlying language model, aka language prior.
In our #ICLR2026 paper, we attempt to understand this phenomenon at a deeper, representation-level.
📄 “Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding”.
https://t.co/J8NgQ7AyGc

-------
1/ Problem: LVLMs often ignore visual evidence

While LVLMs perform well on many benchmarks, they sometimes rely on language patterns rather than actual images.
A simple example: show a model a green banana, and it may confidently describe it as “ripe and yellow” ---because that’s the most common linguistic pattern it has learned. 🍌

This raises a central question:
Where inside the model does visual information begin to influence its reasoning?

2/ Motivation: Output-level probes fall short

Most analyses inspect outputs, e.g., by removing the image or comparing predictions.
But these methods cannot reveal when the model starts integrating vision and how strongly visual signals affect internal states.

To address this, we need a representation-driven perspective. 🔍

3/ Approach: Contrasting Chain-of-Embedding (CoE)
We trace hidden representations across the model’s depth for the same prompt:

•once with the image
•once without the image

By comparing these trajectories layer by layer, we identify the exact point where visual input begins shaping the model’s internal computation.

This leads to the discovery of the Visual Integration Point (VIP) ✨--- the layer at which the model “starts seeing.”

We then define Total Visual Integration (TVI), a metric that quantifies how much visual influence accumulates after the VIP.

4/ Findings across 10 LVLMs and 6 benchmarks

Across 60 evaluation settings, we observe:
• VIP consistently appears across diverse architectures
• Pre-VIP → representations behave like a language-only model
• Post-VIP → visual signals increasingly reshape the embedding pathway
• TVI correlates strongly with actual visual reasoning performance
• TVI outperforms attention- and output-based proxies at identifying language prior

TVI thus offers a more principled indicator of whether a model actually uses the image.

5/ Impact: A new lens on multimodal behavior

Our framework has a few practical benefits. It enables (1) diagnosing over-reliance on language prior, (2) comparing LVLM architectures more rigorously, (3) informing better training and alignment strategies, and (4) improving robustness and grounding in real-world tasks.

Shout out to my students for this insightful work: Lin Long, @Changdae_Oh, @seongheon_96 🌻

Please check out our paper for more details!

[Sharon Li @SharonYixuanLi]

Check out our #ICLR2026 oral paper (top ~1-1.5%). It's a slow-cooked research that probes a fundamental question many of you have wondered about:

How do transformers actually learn semantic associations between tokens (e.g., “bird” and “flew”) during training?

Semantic associations are foundational because they enable models to go beyond memorization and instead generalize and generate coherent text.

tl;dr: This paper provides a formal theory for the emergence of semantic associations in attention-based language models, connecting training dynamics with linguistic insight and mechanistic interpretability.

📄 Read here: https://t.co/NceVXEZy93

Congratulations to my students and co-authors: @shawnim00 @Changdae_Oh @Abell_Zhen_Fang

[Samuel (Min-Hsuan) Yeh @Samuel861025]

🧵 Excited to share our new paper at #ICLR2026: LUMINA: Detecting Hallucinations in RAG System with Context–Knowledge Signals

With @tanwimallick and @SharonYixuanLi

Paper: https://t.co/wQ0xLHI1nI
Code: https://t.co/by25osWkfL

1/N

[Sharon Li @SharonYixuanLi]

I will be giving a talk at UPenn CIS Seminar next Tuesday, October 7.

More info below
https://t.co/RdnTrxfrjk

thanks @weijie444 for hosting!

[Sharon Li @SharonYixuanLi]

Excited to share our #NeurIPS2025 paper: Visual Instruction Bottleneck Tuning (Vittle)

Multimodal LLMs do great in-distribution, but often break in the wild. Scaling data or models helps, but it’s costly.

💡 Our work is inspired by the Information Bottleneck (IB) principle, which promotes representations that discard non-essential features tied to the input modality while preserving those critical for solving the task. This is ideal for robust instruction tuning because it facilitates invariance to low-level superficial features, enabling generalization.

⚠️ But here’s the catch: integrating IB into MLLMs is highly non-trivial. Why?

- Mutual information estimation is intractable at scale

- Autoregressive & multimodal architectures make standard IB formulations break down

✅ Our contributions:

(1) Derive a new variational lower bound of the IB objective tailored to MLLMs

(2) Provide a practical implementation (Vittle) as a lightweight, scalable module

(3) Show via extensive evaluations (45 datasets, 30 distribution shift scenarios) that Vittle consistently boosts robustness across open QA, closed QA, and hallucination detection.

Paper: https://t.co/JVW7vtk6c7
Huge kudos to the team: @Changdae_Oh @JiatongLi0418 @shawnim00

A personal note: This project was particularly challenging — deriving new theory, building a practical implementation, and running massive training + evaluation experiments, all within a resource-constrained academic setting. The fact that the team made all of this happen is nothing short of a miracle to me.

[Sean Du @xuefeng_du]

🚨 We’re hiring! The Radio Lab @ NTU Singapore is looking for PhD, master, undergrads, RAs, and interns to build responsible AI & LLMs. Remote/onsite from 2025. Interested? Email us: [email protected]
🔗 https://t.co/377UaRL9Ic

Please spread the word if you can!

[Chaowei Xiao @ChaoweiX]

My first time NLP conference Here is my schedule at #NAACL. We have 5 papers and 1 tutorial. I will give a tutorial on the test-time threats for #LLM , covering #jailbreak and #prompt injection. Welcome to join our tutorial if you are interested in #trustworthy LM.

[Yu Wang @rain305f]

@XiaogengLiu @muhao_chen @ChaoweiX @llm_sec
@OpenAI

[Yu Wang @rain305f]

🚨Excited to introduce our latest work, AdaShield !!!! AdaShield safeguards Multimodal Large Language Models (MLLMs) from structure-based jailbreak attacks without compromising model performance. #LLM #ML #AI #Security Poject Page: https://t.co/eNawmDgKY7

[Yu Wang @rain305f]

Thanks to all nice collaborators @XiaogengLiu @muhao_chen @ChaoweiX