Ulf Hammarström ⏹️

It's instrumental convergence, stupid.

[Ulf Hammarström ⏹️ @ulfhammarstrom]

Superhuman AI is by definition not aligned with humanity if it is built without humanity's strong consent. If I don't even want you to exist, you're certainly not on my side. Maybe start there if you want to talk about "alignment".

[𝕙𝕒𝕧𝕖𝕟 @haventropy]

I attended the Stop the AI Race protest in San Francisco last week and wrote about it here @AIStopWatch

https://t.co/2yoq0f4TCB

[Ulf Hammarström ⏹️ @ulfhammarstrom]

This is exactly what China is, btw. Bury your "evil CCP" caricatures deep down.

[Judd Rosenblatt @juddrosenblatt]

Xi Jinping AI speech transcript:

Distinguished colleagues and guests, ladies and gentlemen, friends,

70 years ago, a group of young scholars proposed the concept of artificial intelligence for the first time at the Dartmouth workshop in New Hampshire of the United States. In the subsequent 70 years, AI scientists and researchers from around the world ventured into this unknown territory, forged ahead through twists and turns, and made breakthroughs with persistent hard work.

Seven decades later today, amid the new wave of AI development, we are gathering by the Huangpu River to discuss how to promote AI globally for the positive, for good, and for humanity. All this makes our meeting highly important. On behalf of the Chinese government and people, I would like to extend a warm welcome to you all.

In the course of history, the invention of the steam engine heralded the industrial civilization. The widespread access to electricity brightened up modern society and the birth of the internet brought the entire world together. Each of these technological revolutions has profoundly reshaped our way of work and life and enabled a giant leap in economic and social development.

Today, major changes unseen in a century are accelerating across the world. The new round of technological revolution and industrial transformation is advancing at a faster pace. And the world has entered an unprecedented period of active innovation on AI technologies. Intelligent connectivity, human machine collaboration, cross-sector integration, joint creation and sharing and other intelligent technologies are unleashing enormous power.

All this carries within it great opportunities as well as challenges to governance. We human beings must answer the questions posed by our times. How to get along with thinking machines? How to ensure security when algorithms are part of decision making? How to tackle ethical challenges by technologies through adaptive governance. How to realize AI for all when the divide keeps widening? These questions demand serious consideration and real answers from the whole international community.

In China's view, all countries should take a people-centered approach and develop AI for the positive and for good. We should ensure that AI is an important driver for shared prosperity and common security. We should join hands to build a just and equitable system for global AI governance. To this end, I wish to share four observations.

First, we should adhere to the principle of openness and win-win and boost innovation-driven development as a new engine of world economic growth and an accelerator for the shift of growth drivers. AI is moving from the digital world into the physical world. We should seize this rare historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries, and forward-looking planning for future industries so that all sectors and businesses can benefit from AI.

Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI or placing one country's security over that of others.

Third, we should encourage inclusiveness and promote mutual learning between civilizations. AI development and its application should not erode or undermine the diversity of world civilizations or the uniqueness of cultures of different countries. We must shape the values of AI with humanity's common values and make good use of AI technologies to increase understanding, tolerance, exchanges, and sharing among all civilizations. We should tend to the garden of civilizations with great care to ensure that the beauty of each civilization is appreciated and shared.

Fourth, we should advocate solidarity and improve global governance. AI is an invaluable asset that encapsulates humanity's collective wisdom. We should practice true multilateralism and recognize the important role of the United Nations. We should enhance alignment and coordination on AI development strategies, governance rules and technical standards so as to form a consensus based global governance framework at an early date to make this frontier technology better benefit humanity. We must carry out extensive international cooperation and help global south countries with capacity building to bridge the AI and digital divides, promote sustainable development and prevent creating new historical injustice in AI.

Ladies and gentlemen, friends,

This year marks the start of China's 15th 5-year plan. It maps out China's economic and social development for the next five years and provides immense opportunities for the international community.

In recent years, China has embraced AI with open arms. We have promoted interplay between an efficient market and a well functioning government, strengthened AI innovation, actively advanced the AI plus initiative and built a healthy ecosystem for all entities to thrive in together. The core smart economy industries are worth at least 1 trillion RMB yuan. Smart devices in countless homes truly improve people's livelihood. Intelligent manufacturing in China has become another shining hallmark of Chinese modernization.

At the same time, China lays great emphasis on safety and security in AI development with a deep understanding of the trends and logic of AI development. We are continuously improving laws, regulations, policies, mechanisms, application norms as well as ethical principles to make sure that AI is safe, secure, and controllable, and that this fine steed of AI gallops with both speed and stability.

As a responsible major country, China is always committed to providing international public goods relating to AI. Since I proposed the global AI governance initiative, China has promoted the adoption of the UN General Assembly resolution on enhancing international cooperation on capacity building of artificial intelligence by consensus. Published the AI capacity building action plan for good and for all. Announced the AI plus international cooperation initiative and advocated for establishing the world artificial intelligence cooperation organization, or WAICO. China has been contributing steadily to the global AI governance.

We often say in China, a single string cannot make music and a single tree does not make a forest. AI development should not be a solo performance by a single country but a symphony of international cooperation.

Thanks to our joint efforts, WAICO has come into being in Shanghai. Our vision from one year ago is now a reality. This is a major move by China to answer the call of the global south and unite the international community together to promote vigorously AI development and governance. It will be an important milestone in the history of AI development to further support global AI development and to advance global AI capacity building.

I hereby announce that in the next five years, China will provide developing countries with 5,000 opportunities in AI training and seminar programs. China will develop international AI application cooperation centers with ASEAN, the League of Arab States, the African Union, the Community of Latin American and Caribbean States, the Shanghai Cooperation Organization and BRICS. And we will enable 30 countries to use the AI-powered meteorological warning system, Mazu, to safeguard homes around the world.

Ladies and gentlemen, friends,

As ancient Chinese observed, a man of wisdom adapts to changes. A man of knowledge acts by circumstances. With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control. We should always guide AI development with human wisdom and international consensus so that AI can truly become a mighty force that increases the well-being of humanity and advances human civilization.

China is ready to be more open, take more practical actions and assume a more visionary perspective. We are ready to work with all parties to seize the opportunities of AI development and meet the challenges and join hands to create a brighter future for humanity.

Thank you.

[Ulf Hammarström ⏹️ @ulfhammarstrom]

"What WAICO embodies is a fundamentally democratic vision of technological development: decisions about humanity’s future made not in the boardrooms of Silicon Valley or the war rooms of Washington, but through “extensive consultation and joint contribution for shared benefit” among sovereign equals. It is, equally, a socialist vision: technology as a public good, serving people before profit, with its benefits extended deliberately to those the capitalist world system has always excluded.

The United States offers the world an AI race with one permitted winner. China offers a symphony in which every country has an instrument."

[Ulf Hammarström ⏹️ @ulfhammarstrom]

"Where Washington seeks to hoard the technology, China calls for “open source, openness, collaboration and sharing”. Where Washington weaponises “national security” to justify its chip war, Xi calls on the international community to “jointly oppose overstretching the national security concept in the field of AI and placing one country’s security over that of others”. Where Washington demands that the world adopt “American values” along with the American AI stack, Xi insists that AI “should not erode or undermine the diversity of world civilisations or the uniqueness of cultures of different countries”."

[Friends of Socialist China @socialist_china]

At the World AI Conference, Xi Jinping called for AI to be treated a common asset of humanity – open source, shared, and extended to the Global South – as 29 countries founded the World Artificial Intelligence Cooperation Organisation (WAICO) in Shanghai.
https://t.co/gIfQMRemgi

[Ulf Hammarström ⏹️ @ulfhammarstrom]

"WAICO is the institutional expression of the Global AI Governance Initiative Xi proposed in 2023, and it takes its place within the broader framework of China’s four global initiatives: the Global Development Initiative (2021), the Global Security Initiative (2022), the Global Civilisation Initiative (2023) and the Global Governance Initiative (2025) – the operational programme of the community with a shared future for humanity."

[António Guterres @antonioguterres]

Technology that will shape the future of humanity must be shaped by all of humanity. 

It cannot be governed by a handful of countries or companies. Every nation needs a seat at the table. 

Technology must serve people – not the other way around.

[Rob Bensinger ⏹️ @robbensinger]

Yeah, the problem with this is that it's a product ad, not that it's acknowledging that AI has a serious chance of killing everyone. The same message is totally fine if you just state it plainly, without the flatly bizarre 'KEEP THINKING :D CLAUDE :D' aspect.

[Michaël Trazzi @MichaelTrazzi]

Last Saturday, 350 of us marched on OpenAI, Anthropic and Google asking them to commit to pausing AI if others do too

"The largest demonstration against the development of AI in American history" - Daily Cal 🧵

[Andrew Curran @AndrewCurran_]

More than 200 researchers and economists including Jack Clark, Jeff Dean, Noam Brown, Tyler Cowen, Sholto Douglas, John Schulman, Yoshua Bengio, Eric Schmidt, and Dean Ball have signed a statement urging governments and institutions to act now to prepare for AI's economic impact.

[Boden Moraski @boden_moraski]

@peterwildeford My intuition is that EA types and e/acc types are significantly better aligned with each other (even if it doesn’t seem that way sometimes!) than either is to the median American voter

[Ulf Hammarström ⏹️ @ulfhammarstrom]

Can we just be normal and have a human future with useful AI please

[Theo Jaffee @theojaffee]

Key divide in the AI safety world is between people who want the glorious aligned ASI singularity future and people who just don’t want AI at all

[Holly ⏸️ Elmore @ilex_ulmus]

Okay finally looking of AI 2040 and I'm going to repeat what many others have said: "passing the torch to AI" is not an acceptable goal. Like AI 2027, it shows sobriety about the risks but still an odd partiality to AI futures over human futures. And some magical thinking about negotiating with AI agents vs human agents. Frankly, I believe author Ryan Greenblatt and perhaps others really really really want an AI future, and that bias is criminal in a forecast bc it easily hides behind the subjective component of expert judgment.

There is a bias in the technical AI Safety world to think that technical problems are solvable and coordination problems are impossible. And I think that covers why the AI 2040 prefers plan A (hand the torch to AIs later) over plan S. (shut it all down).

Idk if they get to a principled reason why doing ANY further AI research means eventually giving up control, but I don't see one. Feels like there ought to be something between Plan S and Plan A-- that's what I imagine happening during a Pause: research to figure out how far we can safely (and with consent) go, while retaining the option to shut it all down by (among other enforcement things) not advancing the frontier.

Especially if the reason not to do plan S is thinking that cooperation between human parties will eventually break down, feels like they aren't fairly considering the risk that loss of control to AI puts us at FOREVER. There's no easy path here as far as cooperation. I wouldn't give up on other humans so quickly. In Plan A, if cooperation between humans and the machines breaks down at any point after 2040, we're dead!

Also there's a bias towards what feels like "elegance" but is actually just handwaving in the AI 2027/2040 worlds that I think easily becomes personal conflict avoidance. They fear the AI company lobbies in a way I think they wouldn't if they didn't live in San Francisco surrounded by AI industry friends and loved ones. I think personal avoidance bleeds into their judgment here. I'm just getting started, but it doesn't feel like they really considered what a liability it is not to challenge the power of the AI lobby as soon as possible. Like you can't really trust plan A to proceed according to plan if the AI lobby is not reined in.

Below you can see they make a big deal about plan S cooperation between humans eventually breaking down, basically saying this is fatal, but then elide over whole huge unsolved problems like how to fucking build aligned AI under plan A.

[Eric23332 @eric23332]

@adi_baradwaj Yes, and attack helicopters are "functionally omnipotent" against knights in armor. "Functionally omnipotent" just means greatly superior against a specific opponent.

[Eric23332 @eric23332]

@adi_baradwaj If we're talking logical fallacies, nobody thinks SI can be "definitionally omnipotent", that's a strawman.

What they worry about is AI outclassing us the way attack helicopters outclass knights on horseback. The latter is obvious though today's militaries aren't omnipotent.

[Felix⏸️De Simone @FelixDeSimone]

@ilex_ulmus "There is a bias in the technical AI Safety world to think that technical problems are solvable and coordination problems are impossible"🎯

However hard a global shutdown may be, it's not as hard as aligning ASI *and* trusting that all its future iterations will also be aligned.

[Michaël Trazzi @MichaelTrazzi]

On our way to Anthropic!

[Ulf Hammarström ⏹️ @ulfhammarstrom]

🤦