Manli Shu

Gemini multimodality @GoogleDeepMind | PhD @umdcs. Prev @SFResearch @Nvidia Words are my own.

[Manli Shu @ManliShu]

Thanks for sharing, @_akhaliq!

We study how an adversary can *exploit* instruction tuning via data poisoning.
For example, one can inject training data that promote their products in the example responses, and we find that the model can pick up this behavior.

[AK @_akhaliq]

On the Exploitability of Instruction Tuning

paper page: https://t.co/QUWqvVJL5b

Instruction tuning is an effective technique to align large language models (LLMs) with human intents. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose AutoPoison, an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs

[Manli Shu @ManliShu]

@ShangbangLong Congrats! Great to see it all come together and finally out in the world. It was a pleasure being part of the discussions. Impressive results!

[Manli Shu @ManliShu]

@gowthami_s Really enjoyed reading this, Gowthami. I love when a write-up walks you through the thought process and experiments and it just reads as if I did it myself!

[Google AI Developers @googleaidevs]

Gemini 3 Pro is the frontier of multimodal AI, delivering SOTA performance across document, screen, spatial, and video understanding.

Read our deep dive on how we’ve pushed our core capabilities to power hero use cases across:

+ Docs: "derender" complex docs into structured code (HTML/LaTeX)
+ Screen: build robust computer agents that automate complex tasks
+ Spatial: generate collision-free trajectories for robotics & XR
+ Video: analyze sports footage using high-FPS processing with "thinking" mode

See how these capabilities are transforming workflows in education, biomedical, and law/finance → https://t.co/01lfGyKxYQ

[JB Alayrac @jalayrac]

Really proud of what we have achieved with Gemini 3 🚀!

The Gemini MM team has worked relentlessly across image 🖼️ and video 🎥 from pre-training to post-training to simply deliver the best multimodal in the world 👏!

Looking forward to what you will build🫡!

[Phillip Lippe @phillip_lippe]

Gemini 3 Pro is out with large jumps in multimodal understanding and reasoning. Sounds useful for another application we're picturing... 🎨

[Manli Shu @ManliShu]

@jonasgeiping Haha, yeah. Looking back at these examples, I'm still surprised how well they worked. And that was from early 2023. Makes you wonder what today's models are capable of... and whether we'd even catch them.

[Manli Shu @ManliShu]

@A_v_i__S @OpenAI @unccs @uncnlp Congrats, Avi!! 🎉🥳

[Manli Shu @ManliShu]

@BoLi68567011 Thanks for the code. (You guys are moving fast!) Any instruction to try your model? The model card readme seems empty.

[Juan Carlos Niebles @jcniebles]

We just open sourced TACO 🌮 !

arxiv: https://t.co/los3b9synA
github: https://t.co/fQc6VDmvoh

See this thread to learn more! ⬇️🧵

[Jieyu Zhang @JieyuZhang20]

Excited to share my intern project at Salesforce Research! Huge thanks to everyone on the team!!

[Manli Shu @ManliShu]

I'm also representing Salesforce at the #WiML mentoring session on Tuesday. You can also catch me at the Salesforce AI Research sponsor booth Wednesday afternoon.

DM or email me - let’s chat!

[Manli Shu @ManliShu]

Just arrived in Vancouver for #NeurIPS2024 🍁 Excited to chat about all things multimodal LLMs — from data collection to efficient vision tokenizers, multimodal inference-time search, and more. Here’s where you can find me:

[Manli Shu @ManliShu]

📅 12/12 (Thurs) 11:00 AM 📍 East Exhibit Hall A-C #3604 **Poster**: *MINT-1T: Scaling Open-Source Multimodal Data by 10x with a Trillion-Token Dataset* [Read the paper](https://t.co/4gdwlaELSm)
The mm pre-training dataset you've been looking for. Led by @anas_awadalla

[Manli Shu @ManliShu]

@_akhaliq We're happy to continue the discussion here on the Huggingface paper page: https://t.co/29C4UY9qJ3

[Manli Shu @ManliShu]

Check out this recording if you missed our live session tonight. We're happy to answer more questions.
Thank you @_akhaliq for hosting us 🤗

[AK @_akhaliq]

.@ManliShu and @Le_Xue01 presented xGen-MM (BLIP-3) live on X today

if you missed the live session see the recording here: https://t.co/aDD3Jnbn8b

[Manli Shu @ManliShu]

@ruairiSpain @_akhaliq Yes, you can find the recording here: https://t.co/NRNkeATyrN

[AK @_akhaliq]

.@ManliShu and @Le_Xue01 presented xGen-MM (BLIP-3) live on X today

if you missed the live session see the recording here: https://t.co/aDD3Jnbn8b

[Manli Shu @ManliShu]

Join us this evening (8 PM PST) for a live discussion on our recent paper and model release 🤗

[AK @_akhaliq]

.@ManliShu and @Le_Xue01 will be presenting xGen-MM (BLIP-3) from Salesforce live today at 8 PM PST on X live broadcast

https://t.co/e056zqI1Oo

[Anas Awadalla @anas_awadalla]

We are excited to release🍃MINT-1T, the first one trillion token multimodal interleaved dataset with 3.4 billion images, built in collaboration with @SFResearch!

Dataset: https://t.co/qel1yFQwyq
Paper: https://t.co/jaWozwkjf7
Blog: https://t.co/ZJxcov3FhN

🧵

[Manli Shu @ManliShu]

MINT-1T is now available on 🤗 https://t.co/t6IhRIS4Cx.
A large-scale (1T tokens), open-source, interleaved image-text dataset with diverse data sources (HTML, PDFs, and ArXiv papers).

[Salesforce AI Research @SFResearch]

Breaking news! ➡️➡️➡️ We just released the MINT-1T 🍃dataset! One trillion tokens. Multimodal. Interleaved. Open-source. Perfect for training multimodal models and advancing their pre-training. Try it today!

Blog: https://t.co/e36YvEBrcP
Dataset: https://t.co/FHKhkAURdN