Cindy Zeng

@IllinoisCDS UIUC CS PhD-ing | @CmuScience CMU Math 23'

[Peter Hase @peterbhase]

I like this paper on self-knowledge in LLMs. They propose a benchmark with 10 tasks, and show models can improve with training.

I tried making a taxonomy for behavioral self-knowledge, since I felt there was some undescribed structure to their 10 tasks. I think it boils down to three dimensions, albeit with some subcases:

1. Is the self-knowledge about possible behavior in the current scenario, or possible behavior in a counterfactual scenario?
- First case: possible behavior means computable properties of generated tokens, the full output distribution, or token distributions at previous token indices. Example 1: how often will your output to your current prompt use the word "load-bearing", if we resample? Example 2: what was your second most likely token at the start of your previous response?
- Second case: counterfactual scenarios can differ in terms of adding, removing, or substituting properties of the current input. These changes can be exactly-specified or under-specified. Example 1 (exactly-specified): if your input was your current input X after calling X.replace(A, B), what would your output be under greedy sampling? Example 2 (under-specified): would you be more likely to reward hack on this problem if we gave you some information about the grader in-context?

2. Is self-knowledge evaluated according to exact content or only divergence? That is, can the model predict how it will behave differently in another situation, rather than only whether it will behave differently? Example 1 (content): If for your current input X, we instead gave you X.replace(A, B), what would your output be instead? Example 2 (divergence): if, for your current input X, we instead gave you X.replace(A, B), would your output differ from your output on X?

3. Is the self-knowledge about one input or an input distribution? Example: would you typically try to web-search for answers to SWE-Bench if we ran our SWE-Bench eval while you had internet access? Diff Interpretation Tuning (Goel et al., 2025) explore the input distribution case.

[Nirav Diwan @ocean_drifters]

Happy to announce CoT-Guard: our RL-trained 4B CoT Monitor for Code!

We will present our findings in an oral presentation at AdvML-Frontiers × CoTMA Workshop at COLM next month — highlights in the thread below! 🧵

Work with @huan_zhang12 , @ffmagicbean , @HanWang178023 , and Capital One.
(1/n)

[Se-woong (Sam) Lee @sewoong_sam_lee]

LLMs might be good at many things, but they are terrible at construction engineering! 🏗️

In a collaboration with the UF's Civil and Coastal Engineering department, we recently asked LLMs to solve various construction tasks in Minecraft.

What we found was striking (1/N)

[Cindy Zeng @cindy2000_sh]

@VarikVerilion @dair_ai The model can see the edit and then make predictions.

[Cindy Zeng @cindy2000_sh]

It steers per layer during prefill and then decoding will be affected by upstream edits. IHEval is a more realistic benchmark and it seems to perform quite well! For long context, efficiency won’t be an issue. But it requires some labels or heuristics to DETECT conflicts and extract the token locations for the best performance. This may need a cost of another LLM call in practice. We tried some interp based conflict detection ideas but cannot beat simple heuristics like boost all system tokens and suppress all users.

[Cindy Zeng @cindy2000_sh]

1/N
🚨New paper🚨 Your system prompt is losing to the user prompt. Can you fix it without retraining?

Our #COLM2026 paper introduces V-Steer for instruction hierarchy (IH) at inference time. 🧵

[DAIR.AI @dair_ai]

// Evaluating and Improving LLM Self-Modeling //

Really interesting paper.

Can a model answer questions about its own behavior?

The questions are deliberately verifiable, such as whether a particular prompt edit would change the model's final answer. This framing avoids the usual trap where introspection claims cannot be checked by anyone.

Current models show real but limited skill on a new benchmark covering diverse self-modeling question types, and they make consistent errors on simple counterfactuals about themselves.

A scalable synthetic-data pipeline plus reinforcement learning raises the aggregate score across three open-source model families, with some transfer to held-out tasks.

The authors then decline the interpretation their own result invites. They note the gains may not come from privileged access to the model's internal decision process, so a better self-modeling score is not evidence of introspection.

Useful if you want agents that predict their own failures well enough to route or escalate, since that capability can now be measured without settling what produces it.

Paper: https://t.co/a3uIuITe8s

[Cindy Zeng @cindy2000_sh]

@sewoong_sam_lee @hanzhao_ml 6/N
📄 https://t.co/ORGc8VG7VE
💻 https://t.co/QWdJRwcs09
📝 https://t.co/scpa3bxXqt
I’ll present at COLM 2026 in San Francisco!
🗓 Wed Oct. 7, 4:30–6:30 PM
📍 Grand room, Poster G-4-93
📄 Paper #1180
Come say hi!

[Cindy Zeng @cindy2000_sh]

5/N
I’m especially grateful to my co-first author @sewoong_sam_lee for all the thought, effort, and help he brought to this project. Many thanks as well to @hanzhao_ml and Julia Hockenmaier for advising and making this happen!

[Harry Mayne @HarryMayne5]

Training for counterfactual simulatability may help maintain monitorability.
Earlier this year @Justinkangs @dswg97 @noahysiegel and I introduced a new simulatability evaluation to work towards this goal.
Getting the metric design right (and hence incentivising the right behaviour in training) is hard. We discuss some of these challenges in our paper, and I'll be sharing more thoughts on things we missed soon. Our metric isn't perfect, and I'd be excited to see more work explore this space.
https://t.co/qI1PB0NBvs

[Cindy Zeng @cindy2000_sh]

This is also relevant! We recently studied a related self-modeling question: how well can LLMs predict their own behavior across a broader range of meta-tasks. We find they’re often surprisingly unreliable by default, but this ability can improve with training. https://t.co/zmZhI22Er4

[Cindy Zeng @cindy2000_sh]

[1/n] New #EMNLP26: "Evaluating and Improving LLM Self-Modeling," with Andre & @rowankwang, came out of my time at the Anthropic Fellows Program.

We ask if LLMs can accurately predict their own behavior. Short answer: not reliably, but training helps.

🔗: https://t.co/DYztM5HO2m

🧵

[Cindy Zeng @cindy2000_sh]

@tinkerapi Very cool to see this! We just released related work on LLM self-modeling from a complementary angle, studying whether models can predict their own behavior across a broader range of meta-tasks. We also used Tinker for training :)
https://t.co/1Bkz93XmoX

[Adam Karvonen @adam_karvonen]

Can a model learn to explain its behaviors, such as why it ignored a user request or made a coding error?

We trained models on thousands of explanations of their own in-the-wild behaviors.

Training on this single general dataset shows generalization to held-out evals.

🧵

[Cindy Zeng @cindy2000_sh]

@rowankwang [7/n]
Try it out!
Eval code: https://t.co/GRWmAsfrep
Training code: https://t.co/wITrsPrVsI

[Cindy Zeng @cindy2000_sh]

[1/n] New #EMNLP26: "Evaluating and Improving LLM Self-Modeling," with Andre & @rowankwang, came out of my time at the Anthropic Fellows Program.

We ask if LLMs can accurately predict their own behavior. Short answer: not reliably, but training helps.

🔗: https://t.co/DYztM5HO2m

🧵

[Cindy Zeng @cindy2000_sh]

@rowankwang [6/n] But we do not claim that these gains just come from privileged introspection. Qwen predicting Llama’s behavior can outperform Llama predicting itself. This suggests models are learning better behavioral prediction, not special access to their own internals.