Matteo Nulli

Multimodal research @ebay FoMo Team | prev @VISLab_UvA & @funailab

[Matteo Nulli @matteo_nulli]

Couldn't agree more. That's why I just wrote a blogpost about Visual Tokens, feel free to have a read 🤗

https://t.co/m3h0jLzpff

[Gowthami @gowthami_s]

You either die a vision researcher or live long enough to realize everything is a token.

[Matteo Nulli @matteo_nulli]

@Joanvelja Poetic

[Annalisa Barbara @annalisab]

Wandering around Seoul next week for ICML. Happy to talk about, in decreasing order:
🍜 Korean food
🐢 What questions are worth spending 4 years of a PhD on (or however long we have before AGI)
✨ Recent 𝐬𝐩𝐨𝐭𝐥𝐢𝐠𝐡𝐭 𝐩𝐚𝐩𝐞𝐫 on inverse game theory!

[Yuki @y_m_asano]

🎉[openings] I’m hiring postdoctoral researchers to join our @FunAILab at UTN through the Alexander von Humboldt Research Fellowship (@AvHStiftung), via the Henriette Herz Scouting Programme.

As a Henriette Herz Scout, I can nominate outstanding international researchers for this fellowship route. I’m especially keen to hear from candidates working on multimodal learning, video and image pretraining, and post-training.

Fellows would be hosted in our lab at UTN and work closely with us on these topics.

Key requirements:
* finished your doctoral studies less than 4 years ago or will finish in the next 6 months
* did not live/work in Germany in the last 10 years
* applications from female, trans* and/or non-binary candidates are highly encouraged!

Interested? Please send a short note with your CV, PhD year, current affiliation, 2–3 key publications, and a few lines on how your work connects.

Please share! 🔀

[Dwarkesh Patel @dwarkesh_sp]

Terence Tao spent a year at the Institute for Advanced Study - no teaching, no random events of committees, just unlimited time to think. But after a few months, he ran out of ideas.

Terence thinks that mathematicians and scientists need a certain level of randomness and inefficiency to come up with new ideas.

[Matteo Nulli @matteo_nulli]

⏳ Stay tuned for the next on the effect of visual token count on context windows, inference latency and VRAM.

[Matteo Nulli @matteo_nulli]

Just started a blogpost series 𝐃𝐞𝐦𝐲𝐬𝐭𝐢𝐟𝐲𝐢𝐧𝐠 𝐌𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. 🔥

I breakdown mathematical fundamentals of VLMs and how to calculate # visual tokens without any inference.

check them out ⬇️

https://t.co/oiTdHpx9DG

https://t.co/m3h0jLzpff

[Benno Krojer @icml🇰🇷 @benno_krojer]

🚨New paper

Are visual tokens going into an LLM interpretable 🤔

Existing methods (e.g. logit lens) and assumptions would lead you to think “not much”...

We propose LatentLens and show that most visual tokens are interpretable across *all* layers 💡

Details 🧵

[Matteo Nulli @matteo_nulli]

genuinely shocked by this https://t.co/CwiweRjWIE

[Timoleon (Timos) Moraitis @timos_m]

@karpathy I was told that 42 is the answer to the ultimate question of life, the universe, and everything!

[Matteo Nulli @matteo_nulli]

Big shoutout to the team at @eBay @VOrshulevi6857 @SKhadivi & @VISLab_UvA @UvA_Amsterdam @cgmsnoek 🙌

📄 Read the full paper: https://t.co/eQcjg0VXhB
🤗 Upvote on Daily Papers: https://t.co/PvBVuXVvgt

7/7 🧵

#ComputerVision #Ecommerce #VLM #MultimodalLearning

[Matteo Nulli @matteo_nulli]

How do you specialise a VLM without losing its general capabilities? 🤔

💡Our new paper "Adapting Vision-Language Models for E-commerce Understanding at Scale", accepted as an Oral at #EACL Industry Track, answers just that!
1/n 🧵

https://t.co/PvBVuXVvgt

[Matteo Nulli @matteo_nulli]

The best part? We didn't sacrifice general intelligence. ⚖️

Despite heavy domain adaptation, our models retain strong performance on broad multimodal benchmarks like MMStar, MMMU and AI2D. 🍰

6/n 🧵

[Matteo Nulli @matteo_nulli]

Results? 📈

🚀Our adapted models substantially outperform general SOTA VLMs on e-commerce tasks (Table 1). Targeted adaptation works!

⚡️We also show massive speedups (3.8x) and quality gains in "Item Intelligence" via fine-tuning.

5/n 🧵

[Matteo Nulli @matteo_nulli]

Real-world E-commerce data is NOISY. 📢

We built a "Visual Verification Pipeline" to curate 4M+ high-quality instructions. We use InternVL for captioning and Mistral to verify if listing attributes are actually visible in the image. 🧹✨

4/n 🧵

[Matteo Nulli @matteo_nulli]

We present a reproducible, backbone-agnostic recipe. 🛠️
We performed extensive ablations across components:
👁️ Vision Encoders: SigLIP2, Qwen2.5 ViT
🧠 Text Decoders: eLlama3.1, Qwen3, Lilium3, Gemma3, Llama-3.1,

The goal? Handle multi-image, attribute-centric reasoning.

3/n🧵

[Matteo Nulli @matteo_nulli]

Standard benchmarks don't cut it for online retail. We propose a novel evaluation suite covering:
1️⃣ Aspect Prediction 🖼️
2️⃣ Deep Fashion Understanding 👗
3️⃣ Dynamic Attribute Extraction 🎞️
4️⃣ Multi-image Item Intelligence 📦

Examples in the image below

2/n 🧵

[Gowthami @gowthami_s]

You either die a vision researcher or live long enough to realize everything is a token.

[Xiangming Gu @gu_xiangming]

Finally release one of my works done during my time at @GoogleDeepMind: gemma_penzai: https://t.co/2Z7a1pGmNZ. It can be used to visualize and debug (Multimodal) LLMs.

In gemma_penzai, every layer is like a building block. You can also easily insert/delete/manipulate any layer you want!!! With it, you can attempt to find interesting phenemenon in LLMs, such as attention sink, and then propose new model architectures.

[AK @_akhaliq]

BabyVision

Visual Reasoning Beyond Language
https://t.co/Vbwr0fXWFa