Yiran Zhang

PhD student in EECS at UC Berkeley. B.S. from Tsinghua University (IIIS).

[Fan Nie @FanNie1208]

AI is now taking on some of the hardest open problems that scientists have posed. But research is broader than solving problems. Can AI also propose research questions that experts find genuinely interesting?

We built After42 to find out. Experts judged 45 of its 135 final questions genuinely interesting, across mathematics, statistics, theoretical computer science, and physics, with wide variation across subfields.

The key idea: develop questions by trying to solve them.

We also asked whether AI can judge which questions are interesting. We find that model judges' top choices align with expert consensus but do not consistently capture individual experts' preferences.

We release the expert-assessed questions as an open resource for researchers to discuss, refine, and pursue, as a step toward integrating AI into the broader research enterprise.

https://t.co/iOrs6L7E9Y

[Yiran Zhang @zhangyir22]

A very clean study of an important question in LLM training!

[Haodong Wen @herrywen1]

(1/9)Suppose you're training an LLM on web data with limited math data. Should you place all your math data at the end of training, or mix it with web data earlier? The answer flips with the learning rate (LR) schedule!

At fixed compute and mid-training (MT) token budgets, earlier mixing gives lower MT loss when LR decays throughout training. With constant LR, placeing all MT data for the end works better.

We then vary when LR decay begins. The best tested strategy: keep LR high after MT data enters, then decay near the end.

See our new paper: "Data Schedules and Learning Rate Schedules Should Be Designed Together" !

[Yiran Zhang @zhangyir22]

These animations really help me understand how looped transformers reason!

[Hengyu Fu @HengyuF]

Frontier labs are rumored to be adopting looped Transformers for their potential to unlock implicit reasoning. But can looped Transformers really master it?

We find a bottleneck that keeps looped Transformers from perfect implicit reasoning. Using mechanistic interpretability, we show how looping helps implicit reasoning, why it still struggles to generalize, and how to fix it.

We introduce DiscoLoop: a residual design for looped Transformers that improves implicit reasoning.

🧵 Long thread ahead 👇

[Yiran Zhang @zhangyir22]

This is joint work with my awesome collaborators: Haodong Wen (@herrywen1), Yingfa Chen (@yingfachen) and Kaifeng Lyu (@vfleaking).
Paper: https://t.co/6cANI8zhn5

[Yiran Zhang @zhangyir22]

Excited to share our new paper: “Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D.” We show that frontier language models can struggle with a seemingly simple task: copying the input exactly. Viewing text in 2D can help models learn copying better.

[Haodong Wen @herrywen1]

(1/n)If you paste your data into an LLM and ask it to plot it in Python, it first has to copy every number into the code — so can you trust the result, or might it quietly miscopy a few?

Our new paper finds frontier LMs sometimes do get this wrong! Models that can solve hard reasoning problems still fail at a much simpler task: copying an input exactly, token by token.

Why? RoPE has an inductive bias that works against exact copying.
Better option? we propose 2D-RoPE, which views text as 2D instead of 1D.