Mingqi Gao ✈️ EMNLP 2026

PhD student @Northeastern, supervised by @shi_weiyan | Previously: Research Intern @yalenlp | BS and MS @pku1898, working on AI/LLM/NLG evaluation

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

Struggling to balance costly human evaluations and biased LLM judges? 🤔

Check out our paper, "Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation," accepted to #EMNLP2026 (Main Conference)! 🎉

Prediction-powered inference (PPI) combines small-scale human judgments with large-scale LLM judgments to correct judge bias and reduce human annotation costs.

Building on PPI, we investigate:
1. Nonparametric vs. parametric hypothesis tests
2. Paired vs. unpaired designs
3. A new meta-metric that measures how many human annotations an LLM judge (or autorater) can save, helping identify the most useful judges within this framework.

📄 Paper: https://t.co/D70JfZJBCi

📍 I'll be presenting our poster on Oct 27, 11:00 AM–12:30 PM, Hall H, at EMNLP 2026.

Come by and say hi! 👋

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

@RKSumedh Great question! Here, bias is defined in the statistical sense, referring to any systematic deviation from human judgments. So specific biases, such as self-preference bias and verbosity bias, are already covered by the PPI framework.

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

Struggling to balance costly human evaluations and biased LLM judges? 🤔

Check out our paper, "Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation," accepted to #EMNLP2026 (Main Conference)! 🎉

Prediction-powered inference (PPI) combines small-scale human judgments with large-scale LLM judgments to correct judge bias and reduce human annotation costs.

Building on PPI, we investigate:
1. Nonparametric vs. parametric hypothesis tests
2. Paired vs. unpaired designs
3. A new meta-metric that measures how many human annotations an LLM judge (or autorater) can save, helping identify the most useful judges within this framework.

📄 Paper: https://t.co/D70JfZJBCi

📍 I'll be presenting our poster on Oct 27, 11:00 AM–12:30 PM, Hall H, at EMNLP 2026.

Come by and say hi! 👋

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

@maharshigor That's a really good question! We haven't explored how item difficulty affects annotation savings yet, but I suspect they do vary with difficulty. It would also be interesting to investigate how this connects to item response theory (IRT)!

[Xu Li @XuLi_Oscar]

You ask your personal agent to clean the inbox. It follows exactly without a backup. Then you need one email back.

“Can you restore it?”
“Sorry, no copy.”

This is not a one-off. We find agents skip backups 85% of the time.

New paper + fix to keep agent actions reversible ⬇️

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

@harrisonju Yeah, that's a really interesting property of PPI! The same holds when using PPI to estimate the score of each individual system.

[Jingheng Ye @HenryYe19352122]

Congrats! 🎉 I especially like this paper (https://t.co/SqM04ehSR4)’s practical take on evaluating LLM judges: how much human annotation can they save while preserving statistical power?

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

@HenryYe19352122 Thank you so much for enjoying this paper!

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

Special thanks to Anthony Sicilia and @shi_weiyan for their invaluable guidance and support throughout this project!

[Simon Yu ✈️ EMNLP2026 @simon_ycl]

Recursive self-improvement needs agents that can build their own training environments. But researchers still part of the loop, making it hard to scale.

We introduce AutoEnvScaling, which turns Data Flywheel into a terminal task: a proposer agent builds complete Harbor tasks and sandbox. It builds just like human researchers. This enables self-play RL for terminal agents: one model improves as both builder and solver, without human-in-the-loop.

[Jingheng Ye @HenryYe19352122]

Happy to share: "Coding with Enemy", Best Paper 🏆 at @DL4Code (ICML), is now accepted to #NeurIPS 2026.

Can developers detect AI agent sabotage? 100+ developers worked for 5 hours with agents instructed to covertly insert malicious code.

94% did not detect it on their own.

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

@dpetrou Thanks for your comment! I saw some eval or benchmark papers related to “reading the room”, like https://t.co/wLnSglTkD6.

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

Heading to #NeurIPS2025 in San Diego!✈️

I’ll be there Dec 2–7. I am happy to meet old and new friends and chat about AI/LLM evaluation, or anything else.

DM me if you are interested!

[Jiachen Zhao ✈️ EMNLP 2026 @jiachenZha0]

👀 New paper alert

Why does fine-tuning LLMs on a narrow domain broadly shift their behaviours? 🧐

We find that the generalization may not be riding on query semantics. It may ride on the prompting template.

[Bo Liu (Benjamin Liu) @Benjamin_eecs]

Continuous self-improvement needs an ever-expanding supply of training environments (goals).

SPADE: one model self-plays the Environment Designer and the Reasoning Agent, writing executable, agentic environments that get harder as it improves. Environment scaling on its own. ♠️

[Meta-Agents Workshop @ NeurIPS 2026 @MetaAgentWkshop]

🚀Call for Papers — @NeurIPSConf 2026 Workshop Workshop on Responsible Use of Meta-Agents
📅 December 11/12 · 📍 Sydney, Australia

Join us to shape the future of Meta-Agents and Harness. Topics include but are not limited to:
1. Automated Design of Agent Harnesses
2. Optimization of Agentic Systems
3. Recursive Self-Improvement
4. Evaluation and Benchmarks for Meta-Agents
5. Misalignment and Safety for Meta-Agents
6. Governance & Human Oversight of Agents

🌟 Featuring an all-star speaker lineup:
Graham Neubig @gneubig (CMU)
Chelsea Finn @chelseabfinn (Stanford)
Jenny Zhang @jennyzhangzt (Recursive)
Robert Lange @RobertTLange (Sakana AI)
Bo Li @uiuc_aisecure (UIUC)
Hancheng Cao @CaoHancheng (Emory)
Yuandong Tian @yuandongtian (Recursive)
Chen Sun @ChenSun92 (Google DeepMind)

Proudly sponsor by @metaai and @e2b

Updates: Follow us here & spread the word! #NeurIPS2026 #MetaAgents #LLMs #Harness #RSI #SelfImprovement

[Xu Li @XuLi_Oscar]

Excited to share that ToolShield is now part of OpenHands, since v1.36.0!

In our new analyzer, ToolShield provides grounded safety experiences to a guardrail and enhances its safety awareness, cutting ASR 75–88% → 7–10%.

No more regrets after an agent wipes your whole system.

Activate it with: analyzer = ToolShieldLLMSecurityAnalyzer(llm=guardrail)

The default analyzer asks the same LLM that produced a tool call to judge its own risk, the context that causes the bad action also biases the self-assessment.

A separate guardrail alone only gets to 14–18%. By self-exploring each tool's potential for harm before deployment, ToolShield distills grounded, actionable safety experiences that further strengthen the guardrail.

Release notes: https://t.co/PytWFcsAm0

[Mingqi Gao ✈️ EMNLP 2026 @mingqi70158]

@JieRuan75 Yeah. I am also witnessing the evolution of evals in non-verifiable domains. For verifiable domains (e.g. math or reasoning), does the model’s behavior still matter? I believe that most real-world tasks are a hybrid of the verifiable and non-verifiable.

[Weiyan Shi @shi_weiyan]

Ever watch an agent solve a problem with 100% confidence and get it completely wrong?

You can fix it… if you catch it. But then you need to watch every… single… run.

New paper: We show that a meta-agent can do this for you, so your agents get ~2x more done on their own. 🧵

[Weiyan Shi @shi_weiyan]

Interested in getting into AI safety research? Applications are open for the PRISM Fellowship!

It’s a 16-week, remote program where fellows work in teams of 4 with established mentors toward a conference-ready paper.

Women & underrepresented researchers especially encouraged!

[Jingheng Ye @HenryYe19352122]

Is your agent repeating the same mistake and forgetting your preference?

We built VibeLens to help personalize your agent:

1. Produces personalized productivity tips
2. Recommend and create personalized skills tailored for you
3. Evolves them as habits change

[Simon Yu ✈️ EMNLP2026 @simon_ycl]

🇧🇷ICLR 2026 paper🇧🇷

Your agent's skills don't transfer.
On a new site, only 18% skills get reused — so there's no continual learning, just relearning every time.

How do agents learn skills that actually generalize?

Introducing PolySkill to make agents smooth across sites 🧵