In the Gemini4 safety report, GDM claims that Gemini 4 Argon is highly unlikely to escape from their control (e.g., start a persistent rogue deployment). I think their report does a bad job of substantiating this claim.
Monitoring adequacy is a load-bearing part of GDM's safety argument. However:
1. Argon successfully does something harmful while evading monitors half the time in LinuxArena (a SoTA control eval).
2. GDM catches Argon more often on another eval, but this is just one eval! GDM didn't search for the easiest situation for Argon to evade monitors. For all we know, this eval might be a relatively hard situation for Argon to evade monitors, and Argon should choose other opportunities to escape control if it wanted to do so.
3. GDM accidentally trained against reward hacking observed in the CoT. They ran some experiment to check whether this harmed CoT informativeness, and in their experiment CoT informativeness was harmed. But they said that the experiment was too conservative, without running an experiment they felt to be more fair.
See my more detailed critique (written with my colleague Adam) here: https://t.co/HsFkHX2TbV
I don't understand whether the GDM safety team largely agrees with me that the report does not provide compelling evidence of safety, or not.
I'm also confused (despite a brief informal conversation with some employees) what safety properties the GDM safety team even believes to hold of their models. Based on OpenAI's reporting, OpenAI models seem to regularly do things which I'd describe as "escaping OpenAI's control and sabotaging safety research". Maybe GDM's models are much more aligned than OpenAI's, or less capable, or GDM's control or security is better than OpenAI's. But I don't see a strong reason to believe this.
If the GDM safety team agrees with me that the report isn't a good argument that risk of this type of loss of control is low, then the messaging in the report seems misleading. If not, then it seems important to establish consensus that the report wasn't good evidence (if I'm right that it isn't).
Understanding the state of safety cases is pretty important for governments and the public understanding how much risk AI companies are imposing on them.
We're no longer in the regime where we can make inability based safety cases for models causing fairly concerning harms. I don't think GDM's alignment or control safety cases are very compelling. So, overall, I don't think GDM has a compelling safety case, which is concerning.
Pacing frontier AI development is urgent. To verify a slowdown, it’d be very helpful to know how much compute is being spent on pushing capabilities forward, and to have trusted, publicly reported measurements of this. This requires a standardized definition of which uses of compute count as “pushing capabilities forward.” I give one option for such a definition here: https://t.co/m1248tFnvd. AI companies should immediately start having third-party auditors measure and report how much compute they spend on pushing capabilities forward, using a definition like mine.
@swarm_girl@SydneyVonArx@OpenAI Hey @swarm_girl, have you had a chance to read my research report? If so, do you buy my finding that looping increases no-CoT capabilities? Do you agree that high no-CoT capabilities should impair monitorability?
@jeffreygwang@OpenAI I recently studied how architecture changes impact no-CoT capability. I found that looping a transformer increases no-CoT capability much more than it increases pretrain loss. The effect size is larger than the finding in your paper, check it out! https://t.co/ANl9GbXufh
Overall these findings indicate that looped transformers are a concerning architectural change. See our full research note here. https://t.co/ANl9GbXufh
However it's a priori unclear whether this routes through looped transformers just being more capable pretrains, or whether looping is particularly concerning. We studied this in NVIDIA's Dory model, and found that looping increases no-CoT capabilities well above trend.
I worry those get underprioritized precisely because they’re hard to quantify. Diffuse benefits of CoT include understanding why AI takes bad actions. This is important for having legible evidence of misalignment.
Lots of discussion about CoT focusses on whether CoT monitoring would catch egregious bad actions (Evan calls this "Control Monitorability" here). However, CoT also has many more diffuse and hard to measure benefits.
Some architectures could weaken CoT monitorability, or remove the CoT altogether.
We've written a proposal for how companies could be transparent about no-CoT reasoning abilities, other monitorability evidence, and policies for preserving monitorability. https://t.co/32UC0LllOp