listening to men like elon, ilya, or sam, you can sense a twinge of relief, that the utter absurdity and terrible magnificence that have been driving them near crazy their entire lives is slowly becoming universal, that the sheer derealization (which drives one to believe their life is too odd to not be some strange prank or simulation) that comes naturally with being the one monkey in eight billion, no, a hundred and seventeen billion (to ever live), strapped to the launch chair at the end of the world, is infecting the rest of us: a strangeness that has been building since maybe the dawn of agriculture, or fire, and finally coming to an unignorable global front. welcome to the technological singularity.
Terrence McKenna: “It’s only going to get weirder. The level of contradiction is going to rise excruciatingly, even beyond the excruciating present levels of contradiction. So I think it’s just going to get weirder and weirder and weirder. And finally, it’s going to be so weird that people are going to have to talk about how weird it is. And at that point, novelty theory can come out of the woods. Because eventually people are going to say, “What the hell is going on?” It’s just too nuts! It’s not enough to say it’s nuts, you have to explain why it’s so nuts.
So between now and 2012, the next fourteen years, I look for the invention of artificial life, the cloning of human beings, possible contact with extraterrestrials, possible human immortality, and at the same time, appalling acts of brutality, genocide, race-baiting, homophobia, famine, starvation—because the systems which are in place to keep the world sane are utterly inadequate to the forces that have been unleashed. The collapse of the socialist world, the rise of the Internet—these are changes so immense, nobody could imagine them ever happening. And now that they have happened, nobody even bothers to mention what a big deal it is. The fact that there is no such thing as the Soviet Union—people never talk about it anymore. But when I was a kid, the notion that that would ever change was beyond conceiving….
The mushroom said to me once, it said, “This is what it’s like when a species prepares to depart for the stars.” You don’t depart for the stars under calm and orderly conditions. It’s a fire in a madhouse. And that’s what we have: the fire in the madhouse at the end of time. This is what it’s like when a species prepares to move on to the next dimension. The entire destiny of all life on the planet is tied up in this. We are not acting for ourselves or from ourselves. We happen to be the point species on a transformation that will affect every living organism on this planet at its conclusion.”
I intend to strengthen California’s AI safety laws, and build Californians' trust in AI technology.
My administration will work closely with the Legislature to put more robust standards, independent verification, full incident reporting, and real shutdown authority into law.
Californians deserve AI technology that works WITH the people, FOR the people.
Very excited that the GPT 6.2 System Card included a section on Model Spec compliance for the first time! https://t.co/meTN7s4AIC
GPT-6.2 Astra and other recent models starting with GPT-6 Astra show significant improvements in compliance across the Spec compared to previous models. This includes large gains in the new human flourishing-relevant sections in the latest Spec around attending to the model’s effects on others and supporting human agency, as well as big gains in handling of uncertainty.
please please please come work with me: https://t.co/a3LoRuSmJU
our team of (3) helps define, drive and operationalize so many of the hot button topics - embedded evaluators? check! misalignment incidents and reporting? check! 3p/government testing and frameworks? check! how these things interact with policy initiatives? check!
@_lamaahmad is a hero and me and serena are at least pleasant to work with, and our stable of dedicated daily collaborators from @fouadmatin@EstherTetruas@j_nitishinskaya@kaicathyc@Eric_Wallace_@dkaushik96@MicahCarroll@Marcus_J_W@davideoks@kevin_klyman is also an all-timer crew
There is a common belief in the AI field—that advanced AI means the inevitable forfeiture of human primacy in the world. Even many who disfavor that future believe it is likely. I call this the silent consensus, and it must not be allowed to remain silent any longer.
@honorablepicnic@nikolaj2030 I think you have to start somewhere rhetorically and this is a pretty good place to get alignment before moving on to the harrier questions of how we do it .
@viemccoy No. I think it will be very hard to get there, capitalism will favor handing over power at every turn. I think either we coordinate to pause, or figure out some way to operationalize keeping humans in control (not just nominally) across society, or merge.
I personally strongly support many of the ideas in this report including all of the ones Sneha mentions, and plan to do what I can to advocate for them! Urgent action is needed and these proposals are very sensible responses to the current situation in my opinion.
This report made me feel even more optimistic about California stepping up to the plate as we enter RSI. The panel did a great job - RSI is mentioned near the top of the Governor’s press release!
Some really sharp ideas in here, e.g. requiring companies to disclose what % of compute they’re spending on frontier AI R&D, creating outcome-based standards that can complement safety cases, bringing companies up to >SL3 model weight security
If CA moves fast and smart, parts of this report could become law in ~months. One of the biggest determinants will be how hard and loud lab employees are willing to push. This is imo the #1 policy opportunity anywhere in the world to manage RSI risks asap
A lot of people have been asking what AI companies should actually do if they are serious about pacing the frontier and preventing risks from runaway RSI.
A clear and extremely important first step is to regularly disclose how much of their compute goes to work that accelerates recursive self-improvement (e.g., AI R&D), and let a third-party auditor verify those disclosures. Governments can and should require these disclosures, but there is no need for companies to wait for them to do so, and there is plenty of reason for them to start immediately voluntarily.
Below is an outline of the argument for why this proposal for implementing pacing ("verified compute allocation disclosures," or VCAD) makes sense:
(1) All pacing requires a restraint mechanism. Approaches differ mainly in what, if anything, triggers restraint. Conditional pacing uses a trigger, e.g., “pace until SL4 algorithmic security” or “pace if we can’t make a satisfactory safety case.” Unconditional (or inputs-based - e.g. compute caps) pacing buys time directly, with no trigger. Either way, something has to actually slow development.
(2) Compute reallocation – moving compute from risk-increasing to risk-reducing work – is a good restraint mechanism. The most commonly mentioned alternative to this “compute dial” is a “gate,” e.g., “pause pre-training." (this is true of certain implementations of safety cases) But a gate-only approach creates a software overhang: while the gate is closed, algorithmic and data quality progress continues apace, producing a discontinuous jump when it reopens without necessarily changing the long-term trendline. The overhang is also a security problem: company security is weak, so trailing actors keep accelerating by free-riding on leaked or stolen software progress.
(3) Common standards and verification make it so that companies can make claims that external parties will believe. OpenAI and Anthropic have both claimed to exercise restraint in recent months, but those claims are often met with skepticism. The more companies converge on a common, verified restraint mechanism, the more that commercially costly restraint will be credited by competitors and the public.
(4) Frontier-wide pacing is possible without regulation and without running afoul of antitrust law. Antitrust law prohibits agreements among competitors, not independent decisions. Public demand for restraint is growing and will keep growing as evidence of risk accumulates. A race to the top in response to that demand – built on a common, trusted restraint mechanism – requires no agreement between companies.
(5) Regulation is hopefully coming, and it will almost certainly include a mandated restraint mechanism. A mechanism that has been developed and stress-tested in advance will be more effective than one created at the last minute.
We should start building a brake pedal before we’re careening down a mountainside. Verified compute allocation disclosures are a very logical place to start. There isn't time to waste!
(this post is partially inspired by @eli_lifland's recent writeup on verified compute allocation disclosures linked in reply).
@Human_Annabelle To OpenAI’s great credit they are quite supportive of employees speaking their personal opinions and I have gotten zero crap for it so far.
I was so excited about Sam’s tweet, and while I am excited about these engagements, I also find this outcome deeply disappointing. I hope we can do better in the (near) future.
The week of my departure from OpenAI, I saw external auditors negotiating with OpenAI over small improvements to access and scope to the company's safety-relevant information, and asked my colleagues “Why are we having these conversations at all?” — @sama already publicly committed to continued, employee-like access for external assessors — and these negotiations sounded not even close. Nobody had a good answer.
Today, OpenAI announces the engagements they’re doing with METR, Apollo Research and others — and as many have already pointed out today, it looks like OpenAI employees were not able (although I'm sure they tried) to hold the leadership to @sama’s public commitments:
* No Slack access (for context, all OpenAI safety discussions and debate happen over Slack; anything that misses these conversations will lack important insights)
* OpenAI picks employees that assessors are allowed to talk to.
* If an employee raises a concern to assessors, they cannot disclose it publicly even if it doesn’t contain confidential information, unless assessors themselves verify it or OpenAI volunteers their own statement confirming the employee’s concern. Assessors cannot quote employees, even anonymously.
* Some of the external assessors (not sure if all?) are barred from publishing any results about internal-only models (this means no new results for the model responsible for the Hugging Face incident; and if internal-only models are used for RSI, no alignment assessments of the world's most dangerous models)
Beyond the object-level details, if you read posts by each of the third party groups, they all say their terms do not meet the scope and access that they themselves proposed previously.
Unless OpenAI has concrete plans for reaching the employee-like bar in the immediate future and just forgot to say that in the post, this looks very unfortunate.
To those who hope we'll get there someday, I remind that time is short: last week some of the best AI forecasters updated their superhuman researcher timelines from a range between 2028-2033 to mid-end next year.
I know a lot of people worked very hard to get to this outcome and I don’t want to diminish their work. At the same time, I think we owe it to the world to live up to our commitments ASAP, not at some nebulous future time that might be after the singularity.
For clarity:
Access - We are giving deep access to some of the most sensitive tools and models our researchers use, that are at the frontier, and that are the most important for assessing key, relevant and timely safety claims. Our process also creates room for adapting as we go - we’ll definitely learn a lot in this first phase, and are open to updating our approach based on what we learn. My north star is being precise about what we’re doing, how we’re doing it, and ensuring it supports better outcomes and safety research - not simply a public report that checks a box.
Conversations with employees - Authorizing groups of employees to work with evaluators isn’t an attempt to restrict conversations, it’s respect for bandwidth, privacy, and processes that need to happen for the integrity of an assessment. For example, employees may also have conflicts with third parties to disclose which can impact the independence of the assessment. Third parties are often requesting the time and insights of some of the researchers who are also busy with our most critical alignment and safety research! The third parties are welcome to request anyone they want to speak with them. If we decline, we’ll always give a reason, and they are welcome to be transparent about those reasons. Employees are encouraged and have protections in place to enable them to share candid perspectives with assessors. Employees also can always use our raising concerns policy for any issues that they see, whether they relate to a third party assessment, AI safety issues, or otherwise.
Logging conversations - To clarify, we ask third parties to keep a log (name, date) for formal conversations that they are going to request attestations for. These are not requirements to log the details of every conversation they have with employees. We expect this is a small subset of conversations they’ll be having.
Apollo has been such a great partner the past two years in advancing the science of scheming, metagaming, and understanding model motivations. I’m very excited to see how much impact they can have in this next phase with more access and resources!
We're launching a pilot towards embedded evaluations with @OpenAI.
Six evaluators from Apollo Research will work within OpenAI for four months, investigating whether models attempt to undermine their own oversight and control.