@yonata_joel Agreed, but since the export control policy has done absolutely nothing to slow Chinese model development, and arguably has accelerated their dominance of open weight models, it's worth trying something new. There are no sure-things in technology or international politics
This is a good thing - get these companies optimizing for American chips, at least in some parts of AI development. The Chinese government appears to have concluded that easing the short term compute constraints of some of their companies is worth ceding other companies' long term AI ecosystem benefits by allowing in an American competitor, which is exactly the trade-off we want them to make if you're optimizing for American national security and AI dominance.
Their diffusion is definitely problematic but it’s a straw man to suggest anyone serious on our side anywhere in the AI stack is arguing Chinese-developed models are neutral to our national interest.
Bias-related strategic interests aren’t the same as narrower cybersecurity risks. Nobody is saying Chinese models don’t have CCP-approved bias. Nobody serious on our side is saying that bias isn’t problematic. There is no evidence that open weight models hosted on U.S. servers in U.S. runtime environments pose an elevated cybersecurity risk relative to closed models. All of those things can be true, and as of today, the evidence strongly suggests they are all true.
The Americans running open weight models desperately want to see American-developed alternatives rise to the top of the leaderboard and are cheering on our model developers that are publishing open weights, like Nvidia and Google.
@thsottiaux Small thing, but Codex on phone seemingly can’t load conversations once they run for awhile. Seems like a memory thing but is it possible to load last 100 lines or something?
I think public, open source testing of these models would be a great idea. Anecdotally there are some interesting patterns that seem to have emerged; e.g. the models assert the official history of Tiananmen Square, but largely follow American sociocultural norms on topics that aren't specific sensitivities to the CCP.
We've got a dataset to strip out CCP bias through a fine-tuning process that we'd be happy to share with anyone who wants to use Chinese-developed models hosted on American servers. Perplexity did a version of this with DeepSeek R1 called "1776" and it ended up performing better than the foundation model on some benchmarks: https://t.co/eXxGJIsHl4
Based on how easy it is to strip out that bias, my guess is Chinese labs have a party-line dataset they apply at the end of post-training to make sure they don't get in trouble, but they themselves don't care about that process that much. @pstAsiatech and @AGraylin might know more about how the labs think about this.
Another advantage: they're simple to implement. A lot of folks in DC have become enamored with complexity. I think there's a natural tendency to view scalpels as inherently superior to sledgehammers. The politics and political economy of sledgehammers are a lot easier than scalpels.
I am sure this will be met with the usual nuance of any AI safety discussion but actually this seems to me to be a very reasonable move from OpenAI and more mundane than it seems at face value.
It is no secret that OpenAI has struggled with sandboxes/testing environments. Some of these struggles have been ridiculed by other engineers in the AI community as amateurish bordering on negligent. Taking a couple of weeks to build some better processes more befitting of a large company rather than a scrappy startup isn't a bad thing. And isn't a harbinger of doom.
We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment.
We care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime.
We expect confidence in safety to increasingly set the pace of AI progress. We are optimistic about the alignment work we are doing, and we remain committed to making frontier capabilities widely available.
https://t.co/51kvKfbfrO
One common theme of conversation over the last month has been: how do we get more technical people to work in DC or do a tour of duty. There are many many parts of USG where there’s technical expertise needed, especially with anything involving the AI stack.
If you are an engineer or researcher, consider a stint working in any part of the USG or DoW or anywhere where the public sector of the country needs more technical help.
Also if you are reading this and interested, DM me and I am happy to make some connections.
@ChrisRMcGuire@nytimes@SangerNYT@AnaSwanson@dnvolz@julianbarnes Anthropic literally told the administration that it had an "unstoppable weapon" and asked what they should do. The admin used the control temporarily as they sorted through Anthropic's claim. And since then it hasn't been used. How is that incoherent?
@WilliamBryk@dbmikus Asimov was a big proponent of: write a good story, and built a sci-fi world around it. Don't start with the sci-fi. That's how you bring people in, by having quality drama/romance/conflict/adventure that takes place in a techno-optimist future.
Completely agree with Dwarkesh here. In terms of individual self-interest, collective self-interest, safety, and security, it is much, much better to have a system in which there are millions (even billions?) of AIs that are duking it out on behalf of a person, rather than serving as an almighty arbiter created by a small number of people in San Francisco.
It's not immediately intuitive—which is why Dwarkesh has to make the point. It's systems thinking versus linear thinking. The majority of AI influencers in DC, so far, are more linear thinkers than systems thinkers. Anthropic's policy engagement is a prime example of that; and probably a majority of think tankers who explicitly focus on AI are linear thinkers rather than systems thinkers.
Ironically, Anthropic's "constitution", an homage to the US constitution, more resembles the linear thinking of the Declaration of Independence than James Madison's systems thinking. I think it's uncontroversial to say that one of the most important concepts in the Declaration of Independence is the idea of equality in creation and God-given rights being SELF-EVIDENT. That concept was not at all self-evident at the time. Autocracies of today, and even many democracies where rights are a civil contract, do not live and die by that premise in the way the United States does. It's a direct line between "if all men are created equal, then rights come from the creator, and if that's true, rights needs no explanation." Linear thinking.
As great and important as it was, the Declaration of Independence wasn't enough to provide the actual governance framework of the United States. The Constitution resolves that gap with systems thinking. Rather than just prescribe what's good and right and just, the Constitution uses checks and balances and separation of powers to create a system that leaves that responsibility to society collectively. i.e., protecting the sovereignty of the individual.
The problem for Anthropic is that there's literally no way to achieve that as a single company. It's impossible by definition.
I'd rather see everyone in the AI policy world embrace systems thinking like Dwarkesh presents here.
My lawyer is obligated to in all but the most extreme circumstances; he will even defend me if he knows I’m guilty.
In contrast, the Claude Constitution places the AI's highest priority as Anthropic’s definition of the good of humanity.
I'm concerned this leads to a world where no frontier model is truly my personal advocate and guardian angel
And this is especially concerning once all the important decisions in my life - who to vote for, how to invest, what news to trust - is intermediated through superintelligences that are not in any deep way aligned to me.
This is a direct quote from the Claude Constitution:
"We want Claude to be helpful both because it cares about the safe and beneficial development of AI and because it cares about the people it’s interacting with and about humanity as a whole.
Helpfulness that doesn’t serve those deeper ends is not something Claude needs to value.”
Many others like it.
Open source AI has had a moment over the summer. I think a related concept that isn't far behind is sovereign AI.
So far sovereign AI has been used mostly in the context of countries: the idea that each country has its own unique requirements of and relationship to AI, so it shouldn't rent that capability forever; it should own at least some of the stack. This is an extraordinarily intuitive concept and if you talk with policymakers and influencers in countries outside of the US and China, you'll find it's more popular than ice cream.
A less-explored concept: the sovereign AI company or individual. Sovereignty isn't restricted to the domain of international relations. For the same reasons that countries want sovereignty, companies and individuals will prefer to own AI capabilities, all else equal.
What does that look like? The sovereign AI company/individual owns their own compute. (We are back to on-prem!) It's not massive, but it's enough to run ChatGPT-like usage on its own with no outside augmentation for the vast majority of businesses and individuals. Compared to buying inference and compute from a closed model provider and a hyperscaler, the sovereign AI individual pays a one-time cost for a personal AI computer, downloads a free open source model, and connects it to private, local data. You're totally in control and while you have higher fixed costs, your variable costs are electricity, and your data privacy risk is 100% in your hands. Pretty compelling, and gets more compelling as the cost of personal AI compute goes down.
There is a massive benefit to that definition sovereign AI in terms of the structure, stability and safety of cyberspace that the pay-for-inference-from-a-small-number-of-closed-model-behemoths paradigm lacks: resilience. Given the repeated failures of OpenAI and Anthropic to keep their agents in line, you can bet that a system in which they're the only inference players would be pretty vulnerable all the time. In contrast, a system of millions of sovereign AI agents has fewer points of massive/cascading failure, responds quicker to threats, and does a better job of surfacing security paradigms that work at scale. It's like federalism but for the internet.
In practice, it's not an "all else equal" choice, and I'd bet the majority of individuals will keep renting AI rather than choose sovereignty. The latter requires a few extra steps of friction and pulls forward costs, which most people aren't eager to do. Either way it's an interesting thought experiment and I hope policymakers consider that possibility when they think about how governance should shape the AI market in the coming years, especially when they think about security.
BREAKING: Madison Rep. Francesca Hong has lost Wisconsin's Democratic primary for governor to Milwaukee County Executive David Crowley by 3,211 votes, AP projects. Crowley will face Tom Tiffany on Nov. 3.
https://t.co/N1ZMdvwumc
I really don't buy the SaaS-pocalypse narrative because the vast majority of those businesses win not because of technology but because of relationships, institutional knowledge, managing client bureaucracy, and/or regulatory and compliance moats. Or simply you want an outside person to blame if things go wrong.
The one thing that will change is the strategy of selling to startups AS A STARTUP and then scaling up those businesses. Especially AI startups are so savvy and cash-constrained that the combination makes them incredibly likely to develop their own tools for bookkeeping, sales outreach, building demos, etc. But it's important to remember how small that specific strategy really is compared to the vast SaaS universe.
This is a very good decision. By definition, closed models' capabilities must exceed those of open models or there will not be a market for them in the long run. The closed model developers will be the first to tell you how superior their models are to their open competitors.
If closed model developers cannot outpace open models, then models are, without a doubt, undifferentiated commodities, and it makes no sense to subject undifferentiated commodities to testing. The burden of safety shifts to adjacent layers of the AI stack: applications and infrastructure.
The White House choosing to engage the closed model developers on their own terms—who have publicly described their models as weapons and national security threats—is reasonable, striking a balance between assuring the public that they take these developers' warnings seriously while avoiding heavy handed regulation so early in the AI era that would harm all other companies.
SCOOP: Inside Trump's AI framework
The White House is excluding open models from its framework to test advanced AI capabilities, sources familiar with the matter told Axios.
https://t.co/BYgSxBcTka