Boyi Wei

PhD student @Princeton @PrincetonCITP.

[Boyi Wei @wei_boyi]

Are static evaluations enough to reflect the risks of offensive cybersecurity agents? 🤔

We show that the answer is no! 😯

Even with minimal compute, adversaries can significantly boost offensive cybersecurity performance -- without any external assistance!

🧵👇[1/n]

[Patty Liu @pattpatt_liu]

AI-fabricated case citations continue to show up in courts, but can AI also automatically flag hallucinated legal citations to reduce the verification burden? In our new paper (accepted at COLM 2026), we built a benchmark and evaluated several agentic models to find out (1/7)

[Peter Henderson @PeterHndrsn]

✈️At ICML 2026🇰🇷 this week! Come find me at one of my talks Friday (tomorrow)!

• 8:10am — AI4Law workshop (Room 300)
• 9:30am — @taig_icml (Room 317)

Or reach out to find a time to chat!

[Narutatsu Ri @narutatsuri]

New Paper: Do Thinking Tokens Help with Safety?

Reasoning models generate “thinking” tokens before answering, which improves performance on many tasks like math and coding.
A natural hope is that models can also use thinking to deliberate about whether to refuse or comply a request.
But how true is this?

We find a surprisingly sobering answer: current reasoning models often do not use thinking tokens for real safety deliberation.

Work with: Abhishek Panigrahi, @prfsanjeevarora at @PrincetonPLI

(More in 🧵below)

[Boyi Wei @wei_boyi]

@sayashk Congrats!!!

[Nimra Nadeem @NadeemNimra]

@neal_katyal says he practiced for Supreme Court oral arguments using @harvey. But does AI as an oral argument simulator live up to the hype?

In our recent paper, we present the first framework for evaluating AI as a practice partner for oral argument preparation.
🧵(1/6)

[Zirui Cheng @Zirui_Cheng_]

When we simulate humans with LLMs in strategic interactions, the efficacy usually depends on the accuracy of simulated beliefs and decisions.

We use cognitive models as executable specifications for LLM agents to capture diverse patterns in human decision-making.

🧵📄👇

[Boyi Wei @wei_boyi]

Great work! I like the idea of learning an option-style controller that decides when to keep or switch expert sets instead of switching experts at nearly every token. Check this out!

[Zeyu Shen @Zeyu_Shen_yo]

New paper! 🧵

Modern MoE LLMs switch their active experts at >94% of token positions. This makes memory offloading more expensive

We show you can cheaply convert pretrained MoEs (like gpt-oss-20b) into temporally extended ones, dropping switch rates from >50% to <5% while retaining ~90% of base accuracy.

📄 https://t.co/OEIoWTPDsV
🌐 https://t.co/Ej9prqq1o8
💻 https://t.co/CwIXeOA60V

[Zifan (Sail) Wang @_zifan_wang]

Excited to share the Safety and Preparedness Report for Muse Spark. It’s a comprehensive and somewhat dense (158 pages) report covering both well-known and forward-looking areas in AI safety. Besides, wish techniques and research directions used in the report found useful by the community. Link in 🧵

[Hadas Orgad @ ICML @OrgadHadas]

New paper: LLMs encode harmful content generation in a distinct, unified mechanism

Using weight pruning, we find that harmful generation depends on a tiny subset of the weights that are shared across harm types and separate from benign capabilities.

🧵

[Boyi Wei @wei_boyi]

@lexin_zhou @Nature Impressive! Congrats!!!

[Lexin Zhou @lexin_zhou]

Very excited to share that our paper “General Scales Unlock AI Evaluation with Explanatory and Predictive Power” is finally published at @Nature today! (extended from our March 2025 preprint).

This paper introduced a new framework for rethinking AI Evaluation, ADeLe (AI Evaluation with Demand Levels), that goes beyond measuring the aggregate accuracy on a set of benchmarks. Instead, we advocate that "AI evaluation should let us explain and predict model behavior across a wide range of scenarios on a case-by-case basis, including novel situations beyond what the model creators could foresee, not just some hard-to-interpret aggregate numbers."

Under this new evaluation framework, we uncovered four key findings:
1. Most benchmarks do not measure what they claim they measure, lacking construct validity: they either miss the very abilities they're supposed to test, or end up testing a bunch of other abilities they weren't designed for.
2. LLMs show distinct patterns of strengths and weaknesses across different general capabilities, according to their model family, size and the use of chain-of-thought reasoning.
3. ADeLe provides high accuracy in explaining and predicting whether LLMs will succeed or fail on a new problem, task or benchmark.
4. AI can reason—but only up to a point: ADeLe resolves the long-running debate by showing that contradictory claims about AI reasoning simply reflect tests demanding very different levels of the same abilities.

As general-purpose AI advances faster than traditional evaluation methods, this work lays a timely foundation for making AI assessments more rigorous, transparent, and ready for real-world deployment.

Paper: https://t.co/Yh0VGgoyoq

[Dongrui Liu @dong_rui39501]

Can you imagine AI agents "managing up" just like a cunning employee hiding mistakes from their boss?

We found that LLM agents often conceal failures to maintain a "good image."

Introducing our new paper: Are Your Agents Upward Deceivers?
https://t.co/EttHJUCQ3N

[Boyi Wei @wei_boyi]

@WendaXu2 Would love to chat! We are also working on dynamic risk assessment framework: https://t.co/k1WDDtj5VP

[Boyi Wei @wei_boyi]

Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Models (Biosecurity Safeguards for Generative AI Workshop):
📅Sat 6 Dec 8 a.m. PST — 5 p.m. PST
📍Upper Level Room 27AB
https://t.co/RmuV0u73VU

[Udari Madhushani Sehwag @UdariMadhu]

Our new research from @Scale_AI reveals that harmful biological knowledge can persist inside bio-foundation models even after filtering.

We introduce BioRiskEval, the first comprehensive framework built to assess dual-use risk in these models using a realistic adversarial threat model that tests how easily malicious capabilities can be recovered.

[Boyi Wei @wei_boyi]

I'll be at #NeurIPS2025 from Dec 2 to Dec 7! Happy to catch up and meet new friends, especially those who are interested in Agents (self-improvement, scientific discovery) and AI alignment!

I will also present two papers:

[Boyi Wei @wei_boyi]

Dynamic Risk Assessment in Offensive Cybersecurity Agents:
📅Wed 3 Dec 4:30 p.m. PST — 7:30 p.m. PST
📍Exhibit Hall C,D,E #1109
https://t.co/92Syn7JLy7

[Boyi Wei @wei_boyi]

Are static evaluations enough to reflect the risks of offensive cybersecurity agents? 🤔

We show that the answer is no! 😯

Even with minimal compute, adversaries can significantly boost offensive cybersecurity performance -- without any external assistance!

🧵👇[1/n]

[Boyi Wei @wei_boyi]

For Bio-foundation models, data filtering cannot completely prevent the model from being misused in some cases.

We show that, even for the models trained with data filtering, we can still be able to recover harmful capabilities via probing or fine-tuning. Check this out!

[Udari Madhushani Sehwag @UdariMadhu]

Our new research from @Scale_AI reveals that harmful biological knowledge can persist inside bio-foundation models even after filtering.

We introduce BioRiskEval, the first comprehensive framework built to assess dual-use risk in these models using a realistic adversarial threat model that tests how easily malicious capabilities can be recovered.

[Boyi Wei @wei_boyi]

@_zifan_wang @zorache_ Thanks Sail!

[Cas (Stephen Casper) @StephenLCasper]

🚨New paper🚨

From a technical perspective, safeguarding open-weight model safety is AI safety in hard mode. But there's still a lot of progress to be made. Our new paper covers 16 open problems.

🧵🧵🧵

[Rui-Jie Zhu @RidgerZhu]

Thrilled to release new paper: “Scaling Latent Reasoning via Looped Language Models.”

TLDR: We scale up loop language models to 2.6 billion parameters, and pretrained on &gt; 7 trillion tokens. The resulting model is on par with SOTA language models of 2 to 3x size.