BerkeleyNLP

We work on natural language processing, machine learning, linguistics, and deep learning. PIs: Dan Klein, @alsuhr, @sewon__min

[Sewon Min @sewon__min]

Model competition is heating up, and distillation is 🔥🔥

Distillation detection is hard, so watermarking can help make it detectable.

But are existing watermarks robust?

We find two ways to break them: distill a model, then distill that model again; or distill only selected tokens that matter.

Then what do we do? We introduce SubWM, which outperforms prior methods and is more robust to laundering.

[Sewon Min @sewon__min]

"No OCR, read text natively from images" continues!

We showed this with deep research agents: it not only improves performance but also reduces tool calls!

Post-training to work with page images gives a big boost - likely because these open-weight models haven't been trained to work with page images yet?

Exciting work led by @Hughie_Hu43 !

[Tiansheng Hu @Hughie_Hu43]

(1/7) 🚀 Introducing DR Pixel, the first open-source deep research agent that searches and reasons directly over page images, no OCR. On 1.06M pages, our trained 27B model matches GPT-5.4.

📄 https://t.co/SByne8jRFB
Joint work from @BerkeleyNLP, @rtnldynamics!

[Kayo Yin @kayo_yin]

I'm at #COLM2026 this week! 🌉

Lately I'm thinking about what's missing in harmful AI manipulation research and governance (whitepaper soon).

Also on the job market for AI safety research roles, esp. threat modeling, strategy, evals, technical governance...

DMs open :)

[Sewon Min @sewon__min]

Long-context LMs seem great at tasks like RULER, but what about tasks like "find every contradiction in the given corpus"?

- Performance suffers, especially as the input corpus scales (even with in-domain fine-tuning, not just zero-shot)

- It changes a lot of core modeling decisions, e.g., whether to use hybrid models, whether to use block sparse attention

TL;DR: if you are a model developer, include our CTC-Bench in your evals! ✨

[Prasann Singhal @prasann_singhal]

Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵

We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)

[Sewon Min @sewon__min]

I'll be speaking at the EMM-QA Workshop tomorrow (Saturday)! #icml2026

Come say hi and hear about PixelRAG -- doing RAG but without text (?) 👀

Don't miss @andylizf's poster as well🙌

https://t.co/hNFmnXLOg9
https://t.co/6P1e5JPJZ8

[Aly Lidayan @a_lidayan]

As task horizons grow, LLM contexts can’t scale forever. Self-summarization enables concise, interpretable contexts but at a significant performance cost.
Our solution: isolate and *supervise* the information content of summaries in the form of natural-language belief states 🧵

[Sewon Min @sewon__min]

One day I tried tracing all of Olmo's dependencies manually. A few hours later, I realized I can't do it and gave up. Then @sadhikesaven and @CoderBak ModSleuth 🔥

Turns out Olmo and Nemotron have hundreds of dependencies that are super deep, recursive, and not easily visible. I'm glad I gave up early 😅

Spoiler: I thought this would be a one-week Claude Code project. It was not.

The hard part wasn't information extraction (which Claude Code is good at). The hard part was something much trickier. Check out the paper to learn more!

(And yes, if a model release says it used Claude Code, ModSleuth will trace that too... which means the model depends on Claude Code, which has its own dependencies, and ModSleuth itself depends on Claude Code 🤯)

[Yichuan Wang @YichuanM]

The web was never meant to be flattened into text.

Yet most web RAG systems start by parsing HTML --- a complex and lossy process.

🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels.

Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly.

PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%.

Why?
(1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals.
(2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever.
(3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines.

Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG.

Demo below 👇

Code: https://t.co/ssDF0nnVwZ
Paper: https://t.co/OIpQ26Vb8H
Playground: https://t.co/UdzM7GQmu3

[Sewon Min @sewon__min]

Super excited about this work, led by @YichuanM and @andylizf ! It is possible to completely remove HTML parsing by directly retrieving and reading web screenshots through VLM.

HTML parsing is a hidden bottleneck that causes significant complexity and information loss that nobody really pays attention to, and it's so exciting that VLM progress made it possible to remove it.

Please check out this demo as well which is really cool!!! https://t.co/AXZnZzofZl

[Sewon Min @sewon__min]

Really amazing results analyzing what's creative/novel vs. what's copied from Internet data, enabled by the amazing @liujc1998's Infini-gram! https://t.co/ZglfLg1dRF

This is also enabled in @allen_ai's OlmoTrace https://t.co/ayLePYwGbZ where anyone can find matching n-grams between LLM-generated text and its training data.

[Negar Arabzadeh @NegarEmpr]

1/ Thrilled to introduce T³: a corpus for RAG over reasoning tasks, built from thinking traces.
We show that surprisingly RAG can improve reasoning— with the right corpus.
Rag with Transformed Thinking Traces T³ gain by up to 43.9% on AIME 2025-2026.
🔗 https://t.co/9GPxKnszte 🧵

[Ryan Yixiang Wang @RyanYixiang]

8/ For reproducibility and to enable further study of modularity in MoEs, we’re releasing EMO, baselines, and code:

Models: https://t.co/KTN3IrZwqP
Blog: https://t.co/b9jzmjN5OU
Code: https://t.co/rTtfqNgwaf
Viz: https://t.co/SyrmN7UNou

Shoutout to @AkshitaB93 @sewon__min for making this possible!

[Sewon Min @sewon__min]

Full details + results: https://t.co/Ei2EjTY5Q0

Also check this out https://t.co/UaPXtx37XY: Our model specializes qualitatively differently (capability-level rather than lexical) -- this emerged naturally even though we didn't expose any domain prior!

[Sewon Min @sewon__min]

As MoEs grow larger and sparser, they become memory-bottlenecked.

What if experts were actually composable - so you only keep the subset relevant to your task?

We show that this doesn't emerge in standard MoEs (their training makes this hard), but you can pre-train MoEs to support this kind of modularity!

I hope everyone sees the right figure from @RyanYixiang 's original post - I was so excited when I saw this result!!

[Ryan Yixiang Wang @RyanYixiang]

MoEs are everywhere in frontier models, and they are deployed as a monolith system.

But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc.

So what if "modularity" is actually the missing opportunity for MoEs?

Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!

[Sewon Min @sewon__min]

I will give two talks at ICLR workshops!! 🇧🇷

Sunday 9:40-10:10:
"LLMs for Distributed Data Use" @ Workshop on Data Problems in Foundation Models (Room 203 A/B)

Monday 15:30–16:05 :
"Are Mixture-of-Experts Modular? Why It Matters and How to Fix It" @ ICBINB Workshop (Room 201 C)

Both happened to be related to MoEs, but tackle two completely different questions → some say hi!

[Sanjay Adhikesaven @sadhikesaven]

Imagine you fully post-trained "YourModel v1". Then, you've got better data — math, code, tool use, safety — and you want to improve it.

Today, that usually means retraining the whole model.

But what if new data could be added modularly, with a fixed cost each time?

[Sewon Min @sewon__min]

Exciting results on open-source modes for IMO-level problems - congratulations to @aviral_kumar2 and everyone involved!! Great to see @wenjie_ma's ProofGrader (https://t.co/OdRuc0csme) integrated into the development ✨