MATS Research

MATS empowers researchers to advance AI alignment, transparency, and security

[MATS Research @MATSprogram]

The most important technology in history is outpacing our ability to understand it.

We’re funding exceptional talent to work on it.

[Ryan Kidd @ryan_kidd44]

MATS Residency applications close Oct 31! If you're a brilliant computer scientist, physicist, mathematician, governance expert, neuroscientist, lawyer, economist, etc. who wants to work on AI safety & security, we'll empower you to do that work full time.
Apply: https://t.co/rRyEwrboMt

[Cas (Stephen Casper) @StephenLCasper]

🚨New paper (possibly the most important one I have written yet).

We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.

Led by @KamacheeMax.

[Max Kamachee @KamacheeMax]

It is important to re-state that the threats are not hypothetical, and that we must take action to mitigate the risks of AI-enabled terrorism.

[Lennart Finke @LennartFinke]

New paper out of @MaryPhuong10's @MATSprogram Stream, with @hrdkbhatnagar and @DoddEuodia! We made a testbed for eliciting latent knowledge in a code vulnerability scanning task. See below how we made the sandbagging model organism robust to both prompting and training! ( 1/14 )

[Jake Ward @_jake_ward]

Models love reward hacking. Maybe we should just let them!

In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!

[Neel Nanda @NeelNanda5]

> The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research.

I'm glad MATS is experimenting with new ways to support promising AI Safety research!

[Tom Kimpson @KimpsonTom]

Preprint from my work at @MATSprogram is now up: https://t.co/Fn4cvGtWo3

Looking at technical governance / verification Qs, specifically if power draw can be used to constrain covert compute.

[MATS Research @MATSprogram]

You can watch the full conversation on Youtube: https://t.co/BQcDOkzzhJ

[MATS Research @MATSprogram]

The most important technology in history is outpacing our ability to understand it.

We’re funding exceptional talent to work on it.

[MATS Research @MATSprogram]

The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research.
→ Up to $285K/yr
→ Berkeley, London, or DC
→ Research compute, relocation & visa support

Apply by Oct 31 AoE
https://t.co/RIKwGSAueh

[MATS Research @MATSprogram]

The most important problem in human history needs more brilliant people working on it.

MATS co-founder and CEO Ryan Kidd explains the talent bottleneck and how the Residency empowers exceptional people to move into AI safety and security.

Watch the full conversation.

[MATS Research @MATSprogram]

The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research.
→ Up to $285K/yr
→ Berkeley, London, or DC
→ Research compute, relocation & visa support

Apply by Oct 31 AoE ↓
https://t.co/RIKwGSAueh

[MATS Research @MATSprogram]

Apply here: https://t.co/gCCoCAf02z

[MATS Research @MATSprogram]

The best Research Managers know when to step in and where they’ll make the biggest impact.

That instinct is rare, and MATS needs more of it to support people doing some of the most consequential AI safety research today. If you want your work to move the needle, talk to us.

[Brian @bkida01]

With support from @MATSprogram and @BlueDotImpact I’m running a mixer event in London for quants who are concerned about AI risks in the evening of 15th Nov.

Come meet researchers working in the field (including @NeelNanda5 , MTSs from Anthropic, Apollo Research, and more)!

Travel from Europe covered.

Apply here by 1st Nov (takes less than 10 minutes) - https://t.co/pkpUdZ6CCS

Limited space, so earlier applications are prioritised.

[Shi Feng @ihsgnef]

New research led by @PeterNutter1 @DJROYTBURG via @MATSprogram: a better way to update model beliefs that we call grafting.

Fine-tune the base model with next-token prediction on synthetic documents (SDF), and simply apply the weight update onto the post-trained model. This works!

1/7

[Joachim Schaeffer @JSchaeff3r]

Frontier models can now detect AI control interventions.

In particular, GPT-6 Astra is incredibly good at this. When we published the initial version of our CIAware benchmark in May most models were not much above chance level, now a step change happened.

I think this is a big problem and challenges the effectiveness of control protocols. High control intervention awareness means that interventions leaks information about the monitor which helps circumventing the monitor.

Details in the thread 🧵

[Shi Feng @ihsgnef]

Building Comparative Motivation Profiles with Symmetric Belief Editing will appear at NeurIPS.

We refined the methodology and messaging of our performative misalignment work. We are excited to extend this to white-box auditing.

by @David00845018 @1700x0 @MinTaywon @JinghuaO via @MATSprogram

1/7

[Ben Hagag @BenHagag20]

New @NeurIPSConf paper: we're releasing ORBIT: A Framework for Multi-Agent Safety and Security Evaluations. Open source, with configurable attacks & defenses and dynamic topologies, all built on @AISecurityInst's Inspect.

When we started this work a year ago, we didn't imagine how urgent it would become by the time we published. Below are some of the interesting results and where we think this should go next. If you want to talk it through, let us know 🧵

[Antoine Vigouroux @teleturnip]

New paper: instead of decomposing a whole neural network, we only dissect the parts used on a task we care about. It's much cheaper, and gives mechanisms you can inspect, erase, or edit to change how the model behaves 🧵