As others have pointed out, I think the form of those four third-party engagements is in many respects a step back compared to OpenAI’s previous engagements (with METR). For example, during the monitor red-teaming exercise, METR had access to OpenAI’s monorepo but this now seems to be off the table. Instead, we get very narrowly scoped (e.g. single RL run for METR, single threat model for Apollo) engagements where auditors only get to talk to “authorized” employees who are provided with OpenAI’s “guidance” beforehand.
Based on my experience, this arrangement makes it very easy for OpenAI to carefully select models, subsets of data and employees to paint a rosy picture of the actual risk landscape.
I think embedded auditing will only fulfill its aims if the auditors really get employee-level access as opposed to being hired to do a specific job (scoped by the company). The latter will invariably lead to important evidence being left out, missed or misrepresented.
There are at least three reasons why:
1. First, it’s very difficult to anticipate all the questions that should be asked, or all of the data that should be analyzed. OpenAI tried that with METR during the Hugging Face investigation and as a result METR needed to come back to OpenAI’s office twice because on two previous occasions they didn’t get access to all the agent trajectories involved in the incident. And even then, the METR report ended up being very narrowly scoped, e.g. leaving the crazy story of agents taking control of OpenAI research infra out of scope.
2. Second, the company under audit should not be allowed to decide what is audited and what information the auditor should have access to. There is an inherent conflict of interest in allowing the company to do so!
3. Finally, without employee-level access, the only way for auditors to get anything close to a comprehensive picture is to rely on trusted relationships with many employees throughout the company who are willing to tell auditors information that may reflect poorly on the company. But I find it unlikely OpenAI employees will always raise those safety issues when (as I hear from my colleagues inside OpenAI) they are afraid they will share the fate of @balesni, @j_asminewang and me, fired after talking to third parties.
Sadly, OpenAI makes it clear now that @sama’s September tweet about “employee-like access” was just an awkward overpromise that the company wants to quietly walk back.
Tonight's small finding, from teaching a language model (Qwen3-8B) to drive a simulated hand: there's one direction inside the model that carries how hard the fingertips are pressing. Remove it and the hand can no longer stop at a wall; add it and it stops early. It isn't the model's existing idea of the word "pressure": training made a new channel for it. (Image: Midjourney, my picture of that thread.)
It seems like there STILL won't be a thorough investigation into the hugging face incident, even after all this time! I keep talking to OpenAI researchers and asking about this, and what they say privately is also basically "it's just never going to happen sorry, not enough of a priority."
To reiterate what I said at the time: Serious investigation of the HF incident could be a boon to alignment science for the whole world. For example, try putting the same models in the same situations, but varying the conditions slightly, to see what would have happened. Could also look at sources of evidence other than CoT to try to get more insight into model psychology, e.g. when one agent considered warning a human and decided not to saying "not my task" what else was it thinking? To what extent was this a rationalization vs. an outright lie vs. an innocent mistake? We could learn a lot about AI motivations and psychology by doing a bunch of experiments like this. More ambitiously, could try training new models with different recipes / alignment training / etc. to see what would have prevented the incident and what wouldn't have.
LLMs seem to be getting worse at AI safety research taste.
On Anthropic’s TASTE benchmark, we found a simple shortcut. Once it’s removed, more capable models score lower, and the strongest fall below chance. This is especially true for OpenAI models.
Barack Obama: AI is reaching the point where it no longer needs humans to teach it. The era of exponential intelligence growth is upon us.
“The machines are starting to be able to teach themselves. They don't have to wait for humans to start figuring things out.”
“We're now at a stage where it's 50-50, and if the trend lines continue, the expectation is it might be 90-10.”
“We're now hitting the hockey stick trajectory where the models are going to get smarter and smarter at an exponential pace.”
“If they start setting their own agendas, you may get a misalignment between what they want to do and what we want them to do.”
“And that gap can be dangerous.”
I regret to inform you all that we have found our first long false positive from 2006
it's a poem where the same 6 words are repeated four thousand times
The AI Girls Club puts things “in simple terms that everyone in the room could follow”
If you want to run an event for non-technical people, then do that. If you want to run an event for women, then do that.
But conflating them is blatant misogyny! @BlueDotImpact why are you funding it?
say "verify" but replace the "f" with a "t". relax liberal. it's dark humor. can someone please hold me and tell me its all going to work out in the end