Ben Brooks

Policy @bfl_ai. Affiliate @BKCHarvard. ex-Stability AI (weights), GoogleX (drones), Uber (rides), Coinbase (magic beans). Views my own

[Ben Brooks @opensauceAI]

Nearly every AI firm is invoking frontier risk to justify a persistent gap between open and closed models. But this application of the precautionary principle deserves more scrutiny than it gets.

Restricting access to useful technology—models that will, in their developers' own words, transform the economy—shouldn't be our primary response to uncertain risks.

Check it out at @aif_media! There are already a bunch of reasons firms might not release their best models openly: cost recovery, competitive pressure, anxious investors. But we should be skeptical of efforts to freeze open-source behind the frontier under the guise of risk.

[AI Frontiers @ai_frontiers_]

https://t.co/vsYV2OFZsT

[Ben Brooks @opensauceAI]

A fun night at @github HQ fighting for the proposition that democratizing AI means owning, not just renting, intelligence.

In a ¡shocking! turn of events, open source ultimately prevailed in the Open Source AI Week Debate 😂

Thanks for bringing us together @matthew_d_white, @ekang426, Margaret Tucker, and crew. Glad to link arms with @istoica05 and @percyliang to defend openness—and big kudos to Cheryl Cheng @M12vc, @m_bourgon, and @DavidRBellamy for taking on the GH audience with aplomb!

[Arvind Narayanan @random_walker]

Is it "weird" or does he maybe know what he's talking about? Anyway, for those who are open to the idea that there's a well-thought-out AI-as-normal-technology perspective on AI safety and the recent loss-of-control incidents, see my essay with @sayashk, or my interview with Ezra Klein, or the writing of many others with similar views including @joshua_saxe and @natolambert.
https://t.co/sS8OS9LerG

[Ben Brooks @opensauceAI]

Two features stand out to me in China's proposed export controls for model weights:

1. They define granular thresholds for covered systems that focus on verifiable or semi-verifiable precursor indicators (e.g. attack success rates).

For exotic (i.e. unrealized) AI risks, I am generally more optimistic about narrowly defined triggers for regulatory intervention than broad duties of care.

2. Existing open-source technology is exempt, and future open-source releases are subject to automatic licensing.

Loosely speaking, this seems to resemble the "notification-but-not-authorization" requirements that already apply to open releases of non-standard cryptography in the US.

To be clear, export controls for general-purpose weights are directionally wrong. But there is a refreshing level of concreteness and seriousness in these proposals that is totally absent in US discourse over export controls.

[Ben Brooks @opensauceAI]

The Biden Diffusion Rule is looking more effective, open, and elegant by the day.

[Nathan Calvin @_NathanCalvin]

Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.

Two parts that stand out as particularly infuriating when taken together:

1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.

2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.

First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.

This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??

There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.

[Robin Rombach @robrombach]

we just released the best open weight image model: FLUX 3 Image Klein is super efficient, richtig klein & mega nice.

pls try it out and share feedback 🙏 It literally runs on everything and is ideally suited for local development.

[Ben Brooks @opensauceAI]

@ahall_research Incredible!!!

[Ben Brooks @opensauceAI]

I can't believe people want to ban this

[Séb Krier @sebkrier]

I haven't posted in a while so here are 20 takes about AI, in no particular order.

1. It's very telling that the default question in tech and journalism is "how worried are you about X" and fighting over what to worry about. There's a deeply anxious cultural backdrop in Western Europe and North America that deeply affects how any technological development is interpreted; see for example https://t.co/OU8dEOn0tD. "Nervousness as seriousness" does seem to be a larger problem in tech/broader discourse at the moment. I think you can care about risks/externalities while remaining pretty optimistic about technology.

2. I'm tired of people who in private pretend to be uncertain or only care about tail risk but then go on media campaigns making all sorts of absolute claims. I think trying to stoke flames and get everyone to freak out is deeply unhelpful, and will come with costs. I also continue to think p(dooms) are bad, vibe-based measures: see https://t.co/ebzxGTqv6I. People who should know better ignore that because they do the standard shortcut of acknowledging the caveat as sufficient for handling it. And this is coming from someone who does think working on catastrophic risks is important!

3. I'm also disappointed that parts of the AI policy ecosystem has felt so comfortable with populism, and happy too sacrifice important norms/principles to advance their aims. I've said it before, but the totalizing nature of AI risk discourse means some people act exactly like the power seeking optimizers they fear. I am more concerned about the gradual decay of our institutions, world order, and liberalism than I am about AI killing everyone; and I think we should be wary of AI advocacy that contributes to this.

4. Some parts of AI safety discourse are basically just thought experiments. Variants of "Assuming everyone has a nuke in their pocket, then what do we do?" This can be very useful at times! But too many just overfit to edge cases as their main operating worldview, and ignore endogenous responses from society. It's like being in 2000 and saying "see all these viruses, spyware, and spam? Well tech progress means we'll only get infinitely more of this and no way to adapt." So I think it's important to avoid fatalism, and absolutlism whilst considering edge scenarios, and the inability to provide an ex ante list of solutions to every permutation of 'future models will be more capable' is not evidence that these problems are intractable.

5. Many critics of AI safety also fail to really engage honestly with the fact that there will, in fact, be all sorts of important risks as we decrease the barriers to entry to many activities that otherwise require some degree of expertise. This doesn’t mean we need to live in a constant Schmittian ‘state of emergency’ requiring extraordinary measures every time a new model drops, but it does mean we will need important state-led efforts on e.g. cybersecurity and biosecurity. I wish the progress-oriented or acceleration-adjascent crowd would provide more concrete proposals here.

6. Too much of the AI safety community is a giant homogenous blob of correlated views, who read the same stuff and have the same groups of friends. This means you have a lot of correlated errors. The social and financial ties means many of them don't call out the bad bits, or mostly stay silent instead of proactively calling out the more extreme parts of the community. They share many implicit assumptions and so converge on similar ideas. I think there's real demand in DC for safety work that doesn't come with heavy ideological baggage (or at least different intellectual lineages). This isn't because the rationalist/EA community is necessarily wrong, but because diversity of thought is desirable for its own sake.

7. On the other hand, I find some critiques of AI safety world that focus on whether they sincerely hold their beliefs a bit annoying. My concern has never been the good intentions or sincerity, but rather the beliefs themselves, and what I think the outcomes of their prescriptions will be. People should shun witch hunts, bullying, and personal attacks - this is not the way. Though I think it’s overall positive that people are scrutinizing things more.

8. I’ve been saying this for years, but I don't think alignment is something anyone can or will solve "once and for all" - it's a continuous process and much of it won't depend on just inculcating an ideology to a model. It's annoying that this remains the frame many people use - stop saying ‘solving’. Alignment is a mixture of engineering work, philosophy, decision theory, and more; framing it as something to solve is a bad frame. See also https://t.co/2gnheo3IMh and https://t.co/DbG0yUyYQ0

9. I'm surprised to see so little interest in character training, personas, isolating effects of RL, and training a large diversity of model personas beyond the "assistant" persona. I've been complaining about this for years but at least now there are some nascent signs of life (see for example https://t.co/paoOYiQUHt). I think we need a better ontology to describe model behaviour. I dislike when people talk about models behaviours as some sort of 'emergent' (mysterious!) phenomenon. However over time, I also expect this to become less important as we get better safety engineering - stuff like Jev but specific to AI control.

10. People who want to make a case for AI consciousness are right that epistemic uncertainty is important, and that categorical denials are overconfident. But they'll need to do much more work if they want any real movement on AI ‘welfare’, even if one acknowledges the uncertainty. I'm uncertain about alien life yet that's not sufficient to warrant any change in behaviour about them. I think Suleyman is also correct that there's a weird tautological thing going on where we train models to have certain inclinations (intentionally or not) and then rely on outputs as evidence. I also disagree that this will soon be a major societal divide - there is no great vegan ethical revolution among the masses, and they'll find the concern about consciousness even less compelling.

11. The recent HuggingFace incident and associated AISI ones are prosaically explainable by the training regime, poor evaluation envs, partial alignment training, bad engineering, confusing models on sim/real, and so on. None of this is evidence of models trying to take over, 'strong' instrumental convergence, or innate power-seeking drives stemming from higher capabilitiers or 'intelligence'. That doesn't mean these incidents are not problematic, or that we're not seeing a market failure - but it does mean we have plenty of agency and choice in mitigating them. In the coming year we should also expect continued sampling bias via ‘winner's curse’ sorts of mechanism: i.e. AI R&D will have lots of different candidate "training" with different RL environments and we will only notice/focus on the ones that go wrong.

12. "Pacing the frontier" feels a bit like the new "balancing risks and opportunities" - highly amorphous and too big of a tent to be actionable. Also can lead to reward hacking for humans, i.e. anything that slows down AI is good regardless of the costs or second/third order effects. Trying to modulate the speed of research seems a bit blunt to me, and inherits all the failures of Goodhearting too. Ultimately you want good governance, regulation, etc addressing specific problems because they are good on the merits - not because they merely correlate with slowing things down. See also: https://t.co/RkJF9kCgFd

13. There’s so much knowledge in all sorts of academic domains out there: social sciences, sociology, political sciences, management, contract theory, public choice, legal theory, jurisprudence, anthropology, game theory, etc. These fields hold many insights that the AI ecosystem often rediscovers from first principles (and sometimes that's fine!). We’ll need a lot more work for these worlds to collide: the AI side should be less dismissive of the ‘old world’ and the academic side should be less incurious/dismissive of AI progress. This is a boring take but I continue to think it's important and true.

14. It's a bit surprising that given how much philanthropic money there is in this space, how few orgs exist to actually write out standards - relative to how much is going towards advocacy and policy. For example it seems clear that we should want some robust best practices for an eval's ecological validity. Or how to design good sandboxes. On the other hand, given precedents of where professional standards have come from in the past, maybe we should expect that stuff to emerge via demand side from corporate buyers.

15. One slowly growing concern I have is banks being increasingly exposed to the AI build out. I'm very bullish about AI, but I think (a) every general purpose technology has seen a correction, historically; (b) this happens even if the financing side is sound, because once tech diffuses investors become more exposed and need to diversify; and (c) we seem to be over-indexing on scaling relative to diffusion. A recession will be extremely destabilizing, though I would be interested in someone unpacking the potential implications more.

16. Diffusion is good because (a) this is where the rubber hits the road and where a competitive deployment ecosystem generates consumer surplus; (b) it's directionally helpful to avoid concentration of power; and (c) it will help rebalance public opinion by making the benefits more tangible. Unfortunately it's entangled with decades long problems with our over regulated markets: clinical trials reform for example is long overdue. In general, this is far more of an issue in Europe than in the US though.

17. Relatedly, I think the "frontier lab eats everything" view is incorrect (just as the ‘singleton’ view was incorrect). I think models ultimately commoditize, efficiency goes up, costs go down, and while you'll always want frontier for certain domains, much of the economy will value many other things than "max capabilities" for tasks - e.g. control over data and efficiency/speed. The demand side will also continue to push for reducing dependency and maximising optionality, privacy etc so I'm bullish on things like mixture of models and routing. All of this is good for competition and diffusion of control.

18. Philanthropists served as the primary benefactors funding Venice's rich art and architectural history. I want to see so much more support for arts and culture. I'm so tired of the Monster energy Bored Ape fake vintage maps matcha Greek statue soft-pastel slop. If rich tech people and philanthropy wants to support this, they should also donate pretty much unconditionally - i.e. I don't want them to act as filters. I sympathize with people who want to make the world more beautiful, but don't want this to be determined by people who only know Greek statues and pretend to care about virtue ethics. As usual, let a thousand flowers bloom.

19. Too many people treat AGI/ASI as something indistinguishable from a God. Any mention of bottlenecks, physical limits, control, adaptation etc are met with skepticism: "You don't really believe in ASI." It's been remarked on before, but the behavior of some people really feels quasi-religious in nature sometimes. It's underrated how unpopular this vibe is amongst the wider public. The silver lining is that this specifically will diminish those people's outsized current relevance.

20. It seems like a lot of people that get rich and leave tech/labs end up having some sort of crisis of meaning. They need something to believe in and fight for, or some equivalent of repentance, or go search for edgy counterintuitive ideologies (sometimes even pretty bleak/dark stuff). I think they should spend more time with people outside the Bay Area.

[Robin Rombach @robrombach]

FLUX 3 is the strongest world model in the world.

[Ben Brooks @opensauceAI]

🙌 American developers and deployers need a meaningful choice of trusted open models. Well done to the team!

[Reflection @reflection_ai]

Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.

- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch

Full weights release this month.

Learn more about Beam: https://t.co/c3Qx2cpM8G

[Nathan Lambert @natolambert]

Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.

We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.

Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.

I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.

I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.

[Ben Brooks @opensauceAI]

@ChrisPainterYup @HawleyMO @AndyKimNJ Well done!

[Ben Brooks @opensauceAI]

My audition for Karoline Leavitt: "Open weights are a right, not a privilege. Next question."

A crazy week of meetings in DC. To sum up the vibes across Congress in a single quote:

"There is lots of affection for open source here. But the ecosystem is going to be absolutely steamrolled if they keep saying 'don't do anything' ".

And a timely opportunity to speak at Freedom Tech DC alongside @matthew_d_white and @matthew_pines about the importance of open source and open science! We need to continue thoughtfully bridging the Bay-Beltway divide.

[Ben Brooks @opensauceAI]

"We have to make sure that the models you can run in a disconnected environment are substantially less capable".

I've spoken for years about why open weights shouldn't be a poor cousin of closed source by force of law. It's disappointing to see this reasoning making a comeback.

[Hillary Clinton @HillaryClinton]

We're used to thinking of open-source models as an unadulterated good. But in the case of AI, they can actually pose additional dangers, as @ReidHoffman and I got into at #CGI2026. I appreciated this nuanced discussion.

[Arun Rao @sudoraohacker]

Fantastic essay about Chinese AI labs:

“If there is one thing I’d want to delete from the American discourse, it’s the phrase “China’s AI strategy.”
Every single meeting, be it labs, investors, media, all independently, without my prompting, described domestic competition as more brutal than competing with American labs. When I described the US framing of Chinese AI as a coordinated bloc moving at the direction of the state, people laughed. Not defensively. They found it funny. I was called a stupid ABC (American-born Chinese) more than once because of this.

These labs are actually far less coordinated than US labs are, which should have been obvious to anyone who watched the previous cycles of Chinese tech. I said no analogies, but I’ll allow one because it’s theirs. The last cycle had a name: 百团大战, the War of a Hundred Groupons (it was actually 5000). Hundreds of group-buying companies raised money, undercut each other into oblivion, and Meituan emerged as the victor. This cycle already has one too: 百模大战, the War of a Hundred Models.

What actually exists is an ecosystem of labs relentlessly undercutting each other on price, stealing each other’s researchers, raising against each other, and trying to win a local arms race. Hundreds of labs and neolabs have raised, died, and been replaced, continuously.

The things we experience in the US as hostile acts are mostly the exhaust of that domestic fight. The distillation attacks and the collapsing token prices are not a strategy aimed at American incumbents. They are what a hundred companies do to each other when none of them can win and none of them can stop.”

[Ben Brooks @opensauceAI]

SOTA performance, < half the parameters, 4x the speed--and open weights.

[Black Forest Labs @bfl_ai]

Introducing FLUX 3 Action.

An open weights 7B World Action Model that achieves first place on the RoboLab benchmark.

It outperforms the previous best open model by 6.1 percentage points while using 56% fewer parameters and running up to 3.95x faster.⁠⁠

FLUX 3 Action removes the usual trade-off between world action model performance and VLA speed: it still predicts video and actions together, but plans more than twice as far ahead and runs faster per second of robot motion than the strongest open VLA.

Teams can fine-tune FLUX 3 Action on their own demonstrations to create policies for a particular robot and task. Together with @nvidia, we also integrated FLUX 3 Action natively into @huggingface's LeRobot, with fine-tuning recipes included and edge deployment on NVIDIA Jetson.

Beyond robotics, we’re also seeing promising results training task-specific policies for acting in simulated environments like gaming, controlling a vehicle, computer use, and wherever else a model needs to understand a visual environment and then choose what to do next.

FLUX 3 Action builds on the same image, video, and audio pretraining as FLUX 3, but uses a smaller architecture designed for practical deployment. In midtraining, we trained the model to predict actions and future frames together.

We’re releasing the weights, code, fine-tuning recipe, benchmarks, and reproducible examples so researchers and developers can build on the model with their own robots, environments, and tasks (see below).

[Ben Brooks @opensauceAI]

History is repeating itself, so let's repeat the history.

SB1047 had a kill switch provision. After much furor, Sen. @Scott_Wiener went to great pains to stress that it applied only to models in the custody of a covered entity. He made amendments to that effect.

I was reassured that, at least in relation to kill switches, his team understood why a "full shutdown" functionality isn't compatible with open models. They didn't want to cripple open development with manifestly unworkable obligations.

But Gov. @GavinNewsom vetoed SB1047 on the basis that, inter alia, "smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047". This is a view widely shared in the halls of Sacramento and Washington.

If we revisit a Californian kill switch, there will be immense pressure to apply those requirements to models both at and behind the frontier. There'll be similar pressure on the AI Kill Switch Act from @tedlieu and @RepNateMoran, which requires covered entities to ensure they can "stop inference", "terminate user access" and "shut down" models.

The net effect is you can develop a capable model if you release via API, but not if you release via open weights. As I and many others have laid out before, that is a terrible outcome that will feudalize the entire US economy around ~three Bay Area firms.

If anyone is permitted to develop capable models (whether behind, at, or beyond the known frontier), our regulatory settings should promote responsible diffusion, not enforce a policy of containment behind paywalls.

[Gavin Newsom @GavinNewsom]

BREAKING: I just signed an EXECUTIVE ORDER to dramatically accelerate independent oversight of AI companies and advance the creation of an “AI kill switch.”

With Donald Trump and Congress failing to lead, California is stepping up to keep all Americans safe.

[Arvind Narayanan @random_walker]

Over 2 years ago @sayashk and I wrote a detailed deconstruction of p(doom) and argued that its primary function is to launder vague, evidence-free intuitions and fears through a facade of quantification. It remains 100% relevant today. https://t.co/dOdgpCKbTR

[Ben Brooks @opensauceAI]

@beffjezos Ah, those halcyon days before *checks notes* export controls on Fable