Canary Institute

DC-based non-partisan AI policy and research; available for technical briefings to policymakers, media, and others

[Canary Institute @CanaryInst]

@sgaurav yep. Pretty much everyone is going to have to go back to writing, just to prove they actually thought!
https://t.co/rtwzyvKU1L

[Canary Institute @CanaryInst]

@AraucariaArauc1 State and federal spending basically offsets local spending; almost all students in a state are having the same amount of funding.

And also, almost all variation is due to student/family fixed effects. Everything else is IIRC < 20% of the variation.

[Canary Institute @CanaryInst]

@magnushambleton Tabarrok has an excellent discussion here about why, "yes regulatory capture is A Thing, but no, this isn't it"
https://t.co/QAZsuBeFxj

[Canary Institute @CanaryInst]

@magnushambleton The Stigler theory of regulatory capture is over 50 years old: https://t.co/kfaGaNCOwL

"Bottleneckers" use the guise of public safety to exclude entrants (though it doesn't always work out like expected)
https://t.co/vTlQE5eO32

However, I don't think the AI folks are doing that

[Canary Institute @CanaryInst]

@Danmar_here Apologies, "nefarious" was a poor word choice. But I think the original paper made it sound kinda-spooky. Maybe "esoteric" would have been better?

[Canary Institute @CanaryInst]

METR works hard for their independence

[Chris Painter @ChrisPainterYup]

My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why.

Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not.

We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up).

METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website.

Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.

[Canary Institute @CanaryInst]

@hilbertspaess @eddylazzarin The key to ending the race/stampede dynamic is "Compute Verification", and it's the most important problem in the world
https://t.co/zKVR3pklJx

[Arvind Narayanan @random_walker]

🙏🙏🙏The respect is mutual. A few thoughts on our relationship to AI safety:
1) We don’t see ourselves as adversarial to the community (the sentiment is not always reciprocated but that’s okay!). We care about AI safety, but we disagree on many of the specifics, and we think there’s value in epistemic diversity and in re-examining foundational assumptions and beliefs that the safety community tends to converge on too quickly.

2) We’ve emphasized at every opportunity, including in this essay, that there’s a lot of common ground on policy despite divergence in worldviews. We hope that this can help mitigate the polarization in policy debates that leads to chronic inaction. The fact that we have very different starting assumptions from the safety community is particularly useful in this regard.

3) We’ve spent a lot of time in conversation with the AI safety folks (Sayash is a regular at The Curve, for example) to share our perspective but also to learn and update our beliefs when warranted — something we do in this essay. Much of our empirical research is also about testing the cruxes of disagreement.

4) We don’t identify as part of the safety community but I hope it’s clear that there’s a lot of value in constructive dialog. We know the “normal technology” label irritates a lot of safety people — and I will write about the name at another point, explaining in more detail why we picked it and stand by it — but I think those who engage with our ideas and not just the title will find there’s a lot of common ground on safety!

[Haven Harms @haventropy]

Her reaction gave me chills. It really does seem like the world is waking up.

“You just said that 10% doesn’t seem an unreasonable estimate that AI could kill all humans”

“Yes”

“Wow… oh my God.”

[Canary Institute @CanaryInst]

@MelancholyYuga @deanwball Found this breakdown valuable and more nuanced than others. While I wouldn't begrudge anyone being strategic about which concerns they aired publicly (the "virtue of silence" is real), *attacking* others raising those concerns is two-faced. He owes a better apology, in that case.

[Canary Institute @CanaryInst]

@ramez Always glad to see folks publicly updating their beliefs in the face of new data. Thanks!

[Canary Institute @CanaryInst]

@gnrosenberg IIRC, it was based on Paul Ekman, not Ariely?
https://t.co/LDZjN9Dz28

[Canary Institute @CanaryInst]

@TheZvi Yes, if by "read" you mean "high-level review with Claude, and dig into any sections which sound interesting"

[Canary Institute @CanaryInst]

Ahem:
https://t.co/rtwzyvKU1L

[Gergely Orosz @GergelyOrosz]

When it comes to writing w AI, there’s such a cognitive dissonance between the person writing vs those reading.

Those writing: “AI saves so much effort! And it’s much better vs what I’d have written! People will think I’m so articulated.”

Those reading: “WTH is this AI slop?!!”

[Canary Institute @CanaryInst]

@itaisher Of course, Freeman Dyson believed Dyson spheres were possible, what's your point? Folks were predicting flight impossible in 1905... 2 years after it had been *done*!

Reality is bound by physics, and little else.

[Canary Institute @CanaryInst]

@entirelyuseles @AmandaAskell Savage! Not *wrong*, per se... but still savage.

[Canary Institute @CanaryInst]

@nickcammarata Sadly, AI Safety basically won't talk to you unless you're *already* one of the cool kids. One of the best things that BlueDot is doing is having the kinda-cool kids teach the wannabes, thus giving them a chance. See eg Habryka's comment in this thread:
https://t.co/siwOZg72mK

[Bogdan Ionut Cirstea @BogdanIonutCir2]

seems very bad for AI safety grantmaking scalability; if applications need to rely on references from a very small inner circle, obviously already the number of potential suitable applications will be low; and growth will probably be slow too

[Canary Institute @CanaryInst]

@alz_zyd_ If you've never read it:
JOURNAL OF MATHEMATICAL BEHAVIOR 11, 207-219 1992
Mathematics for Little Ones
ALEXANDER ZVONKIN

[Canary Institute @CanaryInst]

@AndyMasley We're just running "Mixture of Experts" models, and you have to prompt correctly in order to get routed to the right expert...