Hongyi James Cai

Undergrad @DukeU @duke_nlp

[Sanxing Chen @sanxing_chen]

Most RL for LLMs today is single-step optimization on a given state (e.g., an instruction), which is essentially a bandit setup. But to learn a meta-policy that can solve various bandit problems via in-context trial and error, you need true multi-turn RL over a long horizon. So, can RL & SFT teach LLMs a meta-bandit policy to explore in-context? 🤔

The regret-based benchmarks screamed YES! But … real story is more complex. We discovered a surprising phenomenon “When Greedy Wins.”
(1/5) 🧵

[Hongyi James Cai @Hongyicai2002]

[6/n] 🔥Caution:
If we shuffle SFT trajectories (pair problems with wrong reasoning traces), RL training collapses 🚨.

Little reward signal. Very difficult to recovery.

[Hongyi James Cai @Hongyicai2002]

🚀Excited to share our new paper: How Much Backtracking is Enough?

🤔 How many backtracks should your LLM learn to reason better?
Turns out: the harder the task, the more backtracking you need!

[Hongyi James Cai @Hongyicai2002]

[5/n] Similar to many recent paper such as Spurious Rewards by Shao et al. and Cognitive Behaviors by Gandhi et al., we find that even noisy, incorrect reasoning traces are effective warmup for RL.

The structure matters more than the correctness!✅

[Junlin Wang @JunlinWang3]

Excited to share work from my @togethercompute internship—a deep dive into inference‑time scaling methods 🧠

We rigorously evaluated verifier‑free inference-time scaling methods across both reasoning and non‑reasoning LLMs. Some key findings:

🔑 Even with huge rollout budgets, non‑reasoning models + clever scaling never catch up to specialized reasoning models (DeepSeek‑R1, QwQ, etc.).
🔑Simple majority voting reigns supreme—often beating fancy Best‑of‑N, MoA, or iterative revision for both model families.
🔑For reasoning models, brevity is golden—correct answers are shorter and contain fewer hedges like “hmm”.

[Bhuwan Dhingra @bhuwandhingra]

🧵When should LLMs trust external contexts in RAG?

New paper from @YukunHuang9 and @sanxing_chen enhances LLMs’ *situated faithfulness* to external contexts -- even when they are wrong!👇