Xinyu (Rachel) Li

Ph.D. Candidate in Robotics @CarnegieMellon @AutonLab
AI Agents | Post-training | Evaluation

[Xinyu (Rachel) Li @RLixinyu]

🚀 We gave a 4B model with an 8K context window more test-time compute. Instead of improving, it got worse.

Scaling across context windows requires more than inline reasoning. It takes contextual reasoning: knowing how to delegate, preserve reasoning in compact summaries, and continue reasoning from them.
(1/10)

[Xinyu (Rachel) Li @RLixinyu]

🧬 MILO tackles automated harness discovery through meta-evolution, where better harnesses are discovered through search processes that evolve alongside them. An exciting step toward recursively self-improving agents for long-horizon tasks!

Huge shoutout to @jprithwish , @kanakaris_nikos , and the team @AmazonScience for the great work!

[Prithwish Jana @jprithwish]

[1/n]
🧬 Want a stronger agent without retraining? Give it a better harness.

MILO, our recent work from AWS AI Labs, does this automatically: +12.0% resolution rate on Terminal-Bench 2.1, and new records on 3 open math problems, beating AlphaEvolve, TTT-Discover and EvoX.

[Xinyu (Rachel) Li @RLixinyu]

🎓 SCOUT highlights an important challenge in on-policy distillation: teachers may struggle on the student-generated trajectories they need to supervise.

What I find particularly insightful is that a stronger model is not automatically a better teacher for every student trajectory. SCOUT addresses this mismatch by letting teachers learn alongside students.

Huge shoutout to @shrangoh , @hliucola , and the team @AmazonScience for this great work!

[Langlin Huang @shrangoh]

🧵 Can your teacher go off-policy in On-Policy Distillation?
Yes! An unfamiliar student prefix makes the teacher less certain, and its token supervision less reliable.
🧭 We introduce SCOUT (Student-COnditioned Updates of the Teacher), which adapts the teacher to student prefixes.
(1/10)

[Langlin Huang @shrangoh]

🧵 Can your teacher go off-policy in On-Policy Distillation?
Yes! An unfamiliar student prefix makes the teacher less certain, and its token supervision less reliable.
🧭 We introduce SCOUT (Student-COnditioned Updates of the Teacher), which adapts the teacher to student prefixes.
(1/10)

[Xinyu (Rachel) Li @RLixinyu]

🙏 Acknowledgments

This is joint work at @AmazonScience with my mentor @MononitoGoswami and my manager @PurakJain. Huge thanks to my amazing collaborators @hliucola, @kanakaris_nikos, @shrangoh, @jprithwish, and Patrick Blöbaum!

[Xinyu (Rachel) Li @RLixinyu]

🚀 We gave a 4B model with an 8K context window more test-time compute. Instead of improving, it got worse.

Scaling across context windows requires more than inline reasoning. It takes contextual reasoning: knowing how to delegate, preserve reasoning in compact summaries, and continue reasoning from them.
(1/10)

[Xinyu (Rachel) Li @RLixinyu]

💡Takeaway

More test-time compute isn't enough. Models need to learn how to use it. Contextual reasoning is a learnable capability.

📄 Paper: https://t.co/OHyaCvQhXe
🔗 Project Page: https://t.co/SDo2XlpNTz
(9/10)

[Xinyu (Rachel) Li @RLixinyu]

The learned capabilities generalize beyond training: to more subagents, deeper delegation, and extra windows where digestion carries progress forward.

Hermes-Learn also complements other TTS methods and transfers to Olmo-3-7B.
(8/10)

[Xinyu (Rachel) Li @RLixinyu]

The model doesn't just learn one delegation recipe:
🧩 It uses decomposition more when problems call for it.
🔍 It often shifts to verification as reasoning progresses.

Its decisions adapt to both the problem structure and the state of reasoning.
(7/10)

[Xinyu (Rachel) Li @RLixinyu]

On AIME/HMMT, Hermes first hurts Qwen3-4B: 37.1% inline → 16.2% with Hermes. After Hermes-Learn: 47.3%!
It even beats the base Qwen3-4B with one 56K window (41.8%) on a matched total budget. Inline reasoning accuracy barely changes.
The gain is in how it uses contexts.
(6/10)

[Xinyu (Rachel) Li @RLixinyu]

🧠 Hermes-Learn closes this gap by teaching the model how to delegate:
📚 SFT on successful teacher trajectories
🎯 Curriculum RL that gradually reduces harness steering
Synthetic problems encourage diverse exploration and verification strategies.
(5/10)

[Xinyu (Rachel) Li @RLixinyu]

Capable models use this flexibility well: Sonnet 5 and DeepSeek-V3.2 scale under Hermes. Qwen3-4B, though, does worse than reasoning in one window.

Access to more context isn't enough. This is the contextual reasoning gap.
(4/10)

[Xinyu (Rachel) Li @RLixinyu]

Introducing 🪽 Hermes: a hierarchy of harnesses that progressively shifts control from the harness to the model.
Two simple mechanisms:
🔀 Delegation: assign tasks to fresh contexts.
📝 Digestion: compact reasoning progress to continue working in a fresh context.
(3/10)

[Xinyu (Rachel) Li @RLixinyu]

Contextual reasoning involves two important decisions:
🔀 What should a fresh context do? Explore a new approach, decompose a problem, or verify prior work?
📝 What information should be preserved and carried forward?

Existing test-time scaling (TTS) harnesses typically prescribe these decisions.
(2/10)

[Xinyu (Rachel) Li @RLixinyu]

🧠 Hermes-Learn closes this gap by teaching the model how to delegate:
📚 SFT on successful teacher trajectories
🎯 Curriculum RL that gradually reduces harness steering
Synthetic problems encourage diverse exploration and verification strategies.
(5/10)

[Xinyu (Rachel) Li @RLixinyu]

Capable models use this flexibility well: Sonnet 5 and DeepSeek-V3.2 scale under Hermes. Qwen3-4B, though, does worse than reasoning in one window.
Access to more context isn't enough. This is the contextual reasoning gap.
(4/10)

[Xinyu (Rachel) Li @RLixinyu]

Introducing 🪽 Hermes: a hierarchy of harnesses that progressively shifts control from the harness to the model.
Two simple mechanisms:
🔀 Delegation: assign tasks to fresh contexts.
📝 Digestion: compact reasoning progress to continue working in a fresh context.
(3/10)

[Xinyu (Rachel) Li @RLixinyu]

Contextual reasoning involves two important decisions:
🔀 What should a fresh context do? Explore a new approach, decompose a problem, or verify prior work?
📝 What information should be preserved and carried forward?

Existing test-time scaling (TTS) harnesses typically prescribe these decisions.
(2/10)

[Mononito Goswami @MononitoGoswami]

Excited to present the first day of the @RealAAAI 2024 Spring Symposium on Clinical Foundation Models! 🎊 Today's lineup includes keynotes, a panel discussion, demos, contributed talks, and a poster session!
@clinicalfms

[Mononito Goswami @MononitoGoswami]

🤖 We have decided to extend the deadline for our @RealAAAI Spring Symposium on Clinical Foundation Models @clinicalfms to Jan 26, 2024 🥳! CfP, timelines, attendance info available at https://t.co/cr4jmfxxmI #ML4H #MLHC

[Mononito Goswami @MononitoGoswami]

🎉Exciting News! Join us for the AAAI-24 Spring Symposium on Clinical Foundation Models! 🚀Explore captivating talks, engaging panels, & cutting-edge demos. All details (CfP, timelines, ...) available at 👉https://t.co/ydKfB8KFF7. Hurry, papers due by Jan 5, 2024 @clinicalfms