My thoughts on the current calls to slow down AI development:
Nobody wants AI to become dangerous. Nobody wants AI to wipe out humanity, and nobody with common sense wants systems that can no longer be controlled.
That should be obvious.
But there is one very big question I keep coming back to:
Why is this debate becoming so loud right now?
Because the timing is hard to ignore.
Chinese labs are becoming increasingly competitive, open-weight models are closing the gap with closed frontier systems, inference costs are falling rapidly, and models like DeepSeek are showing how much capability can be delivered with dramatically greater efficiency.
And their weights are publicly available.
That changes the competitive landscape.
Frontier labs can study these models, inspect their architectures, read their technical papers, analyze their training approaches and learn from the efficiency improvements that open research makes available to everyone.
That is one of the greatest strengths of open research.
But it also creates an uncomfortable question for companies selling extremely expensive closed models:
What happens when an open-weight model gets close enough to frontier performance at a fraction of the cost?
What exactly is the customer still paying such a massive premium for?
This is why I don’t think the current AI slowdown debate should be discussed purely as a safety issue.
Safety concerns are real.
But economic incentives are real too.
Competition is real.
Geopolitics is real.
And protecting existing business models is real.
All of these things can be true at the same time.
Then there is China.
For years, US export controls have significantly restricted China’s access to the most advanced AI hardware and other critical technologies.
The goal was obvious: make it harder for China to compete at the frontier.
But did China simply give up?
Of course not.
Instead, those restrictions created an enormous incentive to build a more independent domestic AI stack: chips, software, models, infrastructure and manufacturing capacity.
And that is the part many people underestimate.
China does not stand still.
The pace of technological development there is remarkable. AI, batteries, EVs, robotics, manufacturing, semiconductors, energy infrastructure — development happens at an incredible pace.
When access to one technology becomes harder, the incentive to develop an alternative becomes even stronger.
So now look at the situation:
Chinese labs keep improving.
Open-weight models keep closing the gap.
Inference becomes cheaper.
More model weights become publicly available.
And at the same time, some of the largest Western AI companies are calling much more loudly for frontier development to slow down.
Maybe that is entirely about safety.
But are we really not allowed to question the incentives involved?
😉
To be absolutely clear: AI safety matters.
AI can absolutely become dangerous in the wrong hands.
Governments could use AI for mass surveillance, autonomous weapons, cyberattacks, political repression or identifying groups of people for persecution.
Those are real concerns.
We need strong safeguards, better evaluations, better security and international rules around genuinely dangerous applications.
But “AI safety” cannot become a magic phrase that ends every discussion.
Especially when the proposed solutions may also make it harder for smaller labs, open researchers and new competitors to catch up.
Who defines when a model becomes “too powerful”?
Who gets permission to train one?
Who gets to release weights?
Who decides which research is too dangerous to publish?
And perhaps most importantly:
Who benefits from those rules?
We also need to talk about the other side of the equation.
What do we lose by slowing AI down?
AI could dramatically accelerate drug discovery and help researchers understand diseases that are currently untreatable.
It could improve personalized medicine, protein research, materials science, battery development, energy systems, climate modeling and engineering.
It could automate parts of scientific research that currently take months or years.
It could give millions of people access to personalized, world-class education.
It could make programming and technical knowledge accessible to people who never had access to experts.
It could help scientists analyze decades of research in hours and accelerate discoveries that would otherwise take years.
So when someone says:
“We need to slow AI down.”
There should always be a second question:
At what cost?
Another argument we increasingly hear is that future AI systems could copy themselves, spread across computer systems and escape human control.
That absolutely deserves serious research.
But people should also understand what many of these safety experiments actually look like.
In some Anthropic evaluations, models were deliberately placed in adversarial environments designed to give them opportunities to exfiltrate information, access systems or behave autonomously.
That research is useful.
But there is a huge difference between a model being tested in an environment deliberately constructed to see whether it will attempt self-exfiltration and the public impression that a massive frontier model could simply “wake up” one day and secretly copy itself onto random computers.
A 100B+ parameter model does not magically teleport itself onto your PC.
Real-world autonomous replication would require access to storage, compute, memory, credentials, networks, infrastructure and suitable hardware.
Again: research these risks.
Test them aggressively.
Improve the safeguards.
But communicate the results honestly and distinguish engineered worst-case evaluations from spontaneous real-world behavior.
Otherwise we eventually reach a point where every hypothetical scenario becomes:
“See? We told you AI is too dangerous. Shut everything down.”
At this rate, the only thing missing is a conveniently timed major AI incident followed by:
“LOOK! WE WARNED YOU!”
Yes, that part is sarcasm.
Mostly. 😉
There is another contradiction I find interesting.
Some of the companies that benefited enormously from training models on huge amounts of human-created knowledge are now increasingly concerned about others extracting knowledge from their own systems.
That should at least make people think.
Knowledge flowing in one direction is called innovation.
Knowledge flowing in the other direction suddenly becomes a threat.
My position is simple:
I do not believe AI safety concerns are fake.
Some risks are real and deserve serious attention.
What I question is the idea that AI safety exists in a vacuum, completely disconnected from competition, geopolitics and enormous commercial incentives.
I am against turning “AI safety” into an unquestionable argument that conveniently protects incumbents, restricts open research and slows competitors just as the technological gap is beginning to shrink.
Open-weight AI will play a massive role in the future.
Not because open models are automatically safer.
Not because every model should be released without considering the consequences.
But because transparency, competition and access matter.
And a future where three or four corporations control the most powerful intelligence on Earth is not automatically the “safe” alternative either.
Build stronger safeguards.
Improve evaluations.
Protect critical infrastructure.
Create rules for genuinely dangerous applications.
Cooperate internationally where it makes sense.
But do not use safety as an excuse to freeze technological progress just as open models and new competitors are beginning to challenge the established players.
Safety matters.
So does openness.
So does competition.
And we should be allowed to talk honestly about all three.
What do you think?
Of course AI needs strong safety mechanisms, and many of those mechanisms already exist, including in Chinese AI systems.
What I find interesting is the timing of these debates.
Whenever open-source models or labs outside the usual frontier players make a major leap forward, suddenly the discussion shifts toward how dangerous advanced AI supposedly is and why access should be restricted.
Open-source models are already getting remarkably close to frontier performance, often at a fraction of the cost. If that gap keeps shrinking, it becomes increasingly difficult to justify extremely expensive, closed models.
That is why I think at least part of this debate is not only about safety. It is also about competition, control and business models.
Safety matters. But “safety” should not become an excuse to lock down AI progress just because open models are catching up.
President Trump prides himself on being a great deal maker. Well. He now has the opportunity to make “the deal of the century.”
With the future of humanity at stake, Trump must negotiate a comprehensive treaty with President Xi of China to establish a pause on advanced AI development and a ban on super intelligence.
In order to prevent a nuclear war, Reagan and Gorbachev negotiated a nuclear arms agreement in the 1980s. In order to prevent AI from acting independently of human control, Trump and Xi must do the same.
@dhtikna I genuinely don’t understand why anyone would still want to use DeepSeek V4 Pro.
V4.1 Flash is much cheaper and, in my experience, clearly better than V4.
I’ve been using V4.1 Flash for a while now, and it’s honestly incredible.
Thank you, Jun. These are exactly my thoughts.
The whole thing feels absurd. Anthropic itself has faced major copyright disputes over how training data was obtained, yet now they’re complaining about others “distilling” knowledge from frontier models.
To me, this increasingly looks like an attempt to frame open source and open research as dangerous, while protecting a closed and extremely expensive business model.
And with an IPO on the horizon, the timing certainly doesn’t make that narrative any less convenient.
Competition, open research and more efficient models should be welcomed, not constantly portrayed as a threat.
Funny how these kinds of “AI threat” reports always seem to appear right when DeepSeek is making major progress.
Meanwhile, Anthropic itself has faced massive copyright disputes over the data used to train its models.
DeepSeek, on the other hand, is one of the few Chinese labs doing genuinely impressive R&D while openly publishing models, technical papers and research that the entire industry can learn from.
Instead of constantly portraying open models and Chinese labs as the next great threat, maybe Anthropic should ask itself what it can learn from their efficiency and make frontier AI more affordable and accessible.
Of course, one could only speculate why keeping the narrative around closed, expensive frontier models alive might be convenient when an IPO is approaching. ��
Competition is good. Open research is good.
Constantly trying to paint everyone else as the bad guy? Not so much.
What do you think?
We're publishing our most detailed threat intelligence report to date.
It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them.
We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies.
These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve.
We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.
Read the report: https://t.co/0EJUnYEgfz
Just imagine this:
You have Hideo Kojima at PlayStation Studios, one of the most iconic game directors in the world, the creator of Metal Gear and Death Stranding, and you seriously decide to stop supporting his new espionage project, PHYSINT.
And what does Xbox do?
They obviously welcome Kojima with open arms.
Sony potentially had the chance to build a spiritual successor to Metal Gear, developed by the very man who helped define the genre, and they let him walk straight to the competition.
To me, this is one of PlayStation’s most absurd decisions in a long time.
What do you think? Can anyone actually understand this decision?
As confirmed earlier through our official channels, in mid-June of this year, we unexpectedly received notice from PlayStation Studios that they would cancel the PHYSINT project. PHYSINT is an important project both to me personally and to KOJIMA PRODUCTIONS. Determined to keep the project alive, we have spent the last three months searching tirelessly for a new partner. In the course of seeking advice and pursuing negotiations with people across many fields, we found in XBOX a partner with whom we share a common vision for the future, and we have decided to team up with them once again. Please rest assured that development of PHYSINT, our genre-defining action-espionage title, is ongoing and will continue moving forward.
Ich glaube nicht, dass KI die Menschheit aus eigenem Antrieb auslöschen wird. Wenn überhaupt, dann sind gefährliche Menschen, Staaten oder Institutionen das eigentliche Risiko, weil sie KI für Macht, Krieg, Manipulation oder Cyberangriffe einsetzen können.
Genau deshalb halte ich es für falsch, Frontier Modelle nur wenigen großen Unternehmen zugänglich zu machen. Auch kleinere Unternehmen, Entwickler und Sicherheitsforscher brauchen Zugang zu den leistungsfähigsten Modellen, damit sie ihre eigenen Systeme besser schützen, Schwachstellen finden und sich gegen immer stärkere Cyberangriffe verteidigen können.
Ich habe außerdem den Eindruck, dass ein Teil dieser extremen Warnungen auch dazu genutzt wird, Open Source einzuschränken. Offene Modelle kommen den Top Modellen immer näher und das bedroht natürlich bestehende Geschäftsmodelle. Wenn große KI Unternehmen gleichzeitig vor leistungsfähiger offener KI warnen, ihre eigenen geschlossenen Modelle aber immer weiter ausbauen und verkaufen, sollte man zumindest hinterfragen, ob es dabei ausschließlich um Sicherheit geht.
KI Sicherheit ist wichtig, aber Sicherheit darf nicht zum Vorwand werden, um leistungsfähige KI in den Händen weniger Konzerne zu konzentrieren. Breiter Zugang, offene Forschung und Wettbewerb können ebenfalls ein wichtiger Teil der Sicherheit sein.
So, so.. now the Dayton Agreement is suddenly being defended with all possible force. And everyone knows why: in 1995, after major territorial gains, Bosnian and Croatian forces were close to seriously threatening Banja Luka. Dayton came at a moment when "Republika Srpska" Forces was under enormous military pressure.
And when Zukan Helez describes Republika Srpska as a “genocidal quasi-state entity,” that wording does not come out of nowhere. To this day, convicted war criminals such as Ratko Mladić are still publicly glorified there.
Anyone invoking Dayton should therefore not only defend the existence of "Republika Srpska", but also accept that Dayton established Bosnia and Herzegovina as the sovereign state, and that "Republika Srpska" is not an independent state.
@pmwoz@jun_song This stupid behavior really gets on my nerves. DEEPSEEK itself posted that Flash now supports Vision via their API, and you, you idiot, go into the web chat and try to upload a picture.
@jun_song I've been testing Pro for a few hours now, and I have to say that it's a real blessing for my work on my actual projects! I used Flash before, and Pro feels even better and more reliable! And it uses so damn little $
@S_Fadaeimanesh@jun_song That's a problem with the harness that was used. There's a benchmark where, for example, PI uses waaaay fewer tokens than Codex or something like that.