The belief that AI would eventually herald the end of humanity is not a new one. It has not arisen in response to the release of the LLMs of ChatGPT and Claude. It has not emerged as a response to recent technological developments.
The story starts in the 90s, with @allTheYud. A precocious youngster with no formal education, he joined an obscure internet mailing list [created by @perrymetzger] devoted to futuristic ideas ... There he began thinking about “superintelligence.”
At first Yudkowsky wanted to help create this so-called superintelligence... & ... helped establish an institute devoted to the project. It’s unclear when his optimism turned to fear, but at some point he came to believe that a superintelligent machine could escape human control and, unless it shared our goals, destroy all of humanity.
Yudkowsky developed these ideas online in a series of essays and attracted a community of like-minded fellow travellers ... His prolific writing became influential among people working in Silicon Valley, many of whom work in today’s AI companies.
Among the Rationalists, the imminent arrival of superintelligence – and the end of humanity that follows – is axiomatic. This conviction is shared by many people who live together in the San Francisco Bay Area practising highly unconventional lifestyles.
It is not at all surprising that a community in San Francisco would share kooky or even apocalyptic beliefs – the region has been the home of eccentric subcultures for generations. But what is surprising is how far that world view has spread.
In Australia, Yudkowsky’s recent book, If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI, co-written with Nate Soares, has been widely discussed in media circles.
Journalist @hughriminton and ABC chairman Kim Williams both have described it as “compulsory reading”.
Not everyone accepts this premise. I emailed @sapinker, to ask what he thought about superintelligence.
“‘Superintelligence’, with its comic-book prefix, is more a fantasy than a coherent concept,” he wrote. “People use it as a synonym for ‘omniscience’, imagining a magical wizard that can solve all problems with pure computation. Or they imagine that the IQ scale that differentiates humans within their natural range of variation can be extrapolated indefinitely upwards. But real problem-solving requires massive amounts of knowledge about the messy, chaotic, world which divulges its hidden workings at its own pace, only through laborious experimentation. And human intelligence is not some elixir that you simply have less or more of – it’s a gadget that evolved to solve some problems with ease and others laboriously or not at all.
“AI is a different kind of gadget with its own profile of strengths and weaknesses, not an enchanted brew that can grant any wish.”
@GaryMarcus, a cognitive psychologist and machine learning entrepreneur, likewise believes the risk of AI leading to human extinction is virtually zero. “Humans are too geographically spread out, too genetically diverse and too resourceful to simply fall apart altogether,” he writes. “The idea that AI will kill us all in five years is preposterous.”
Marcus does not argue that AI is harmless. He worries about AI being used to develop biological weapons, launch cyber attacks, spread disinformation and enable authoritarian governments – all risks that are catastrophic, if not existential. But there is an important difference between risks we can observe and risks that occur because of human negligence or malevolence, and a chain of events that exists mainly in our imagination.
Part of the disagreement stems from the language we use when we discuss AI. @MelMitchell1, a professor at the Santa Fe Institute and author of Artificial Intelligence: A Guide for Thinking Humans, has criticised our habit of describing machines as if they were people. We often say an AI “thinks”, “believes”, “lies”, “schemes” or “wants” something. These words are a convenient shorthand but they also can create the impression that software has become an independent creature with intentions of its own.
Mitchell makes this point when discussing the recent Hugging Face cyber-security incident, in which autonomous OpenAI agents escaped their “sandbox” – a computer environment isolated from the internet – and hacked a real-world server. “First, OpenAI did not have proper security measures in place,” she writes. “They turned off safeguards built into the models, instructed the models to find and exploit software vulnerabilities, and let the models run autonomously for weeks without sufficient human oversight.”
Rather than showing autonomous AI going rogue, the incident demonstrates what can happen when humans give powerful AI systems dangerous instructions without adequate safeguards.
AI is, of course, advancing rapidly. Machines can write computer code, translate languages, diagnose diseases and solve some of the hardest problems in mathematics.
But to get from the AI we have today to the extinction of humanity requires several further links in a chain, none of which are guaranteed.
Philosopher @mboudry, writing in @Quillette, offers a useful way to think about this. Humans (and other animal species) evolved to have a competitive drive, sometimes manifesting in selfishness and aggression, across a time span of millions of years. Our ancestors survived because they fought hard to secure food and mates. Out in the wild, these selection pressures led to the evolution of traits that enabled animals to hunt and capture their prey.
But artificial intelligence does not exist in the wild. It was created by us and exists in the equivalent of a petting zoo. And just as we have been able to domesticate wheat for our food and breed dogs to be our companions, we are able to select the conditions under which AI develops. We are not selecting AI models on the basis of their ability to hunt prey in the physical world. We select them on the basis of how helpful they are to us.
“We have been selecting chess computers for cognitive capacity for decades,” writes Boudry. “Their capabilities now far outstrip even the most gifted human grandmasters, yet they have not become harder to control.”
Boudry accepts that an AI could slip out of human hands one day, through accident or malice. Even then, he argues, the likely result is not extinction but something like the long battle between computer viruses and antivirus software: costly and ongoing but not the end of the world.
The problem with apocalyptic fears is that when they become mainstream, they can be hard to wind back – even in the face of contradictory evidence.
Across the past 100 years, apocalyptic anxiety has leapfrogged from nuclear annihilation, overpopulation, environmental collapse, to rogue AI. (Some of the dangers behind these warnings were very real, of course, and some of these risks remain.) But we also have to ask what happens when this anxiety becomes locked into public policy.
The ban on nuclear energy in Australia is the most obvious example of the damage this technophobia can do. Australia has 28 per cent of the world’s known uranium resources and has exported uranium for decades.
Australian engineer Bobby Gallagher has invented a nuclear reactor that can be deployed on the back of a truck, a technology that has been hailed by Trump. Yet this form of clean energy remains prohibited in our country under federal law. Public anxiety surrounding nuclear weapons, radioactive fallout, accidents and waste means that while Australians can mine uranium, put it on ships and sell it to countries that use nuclear power, we cannot build commercial reactors for ourselves, using the ingenuity of our own people. The situation is a disaster.
And in an age of superpower rivalry, technophobia does not remain purely a domestic matter. During the Cold War, the Soviets promoted a fear of a “nuclear apocalypse” in the West, and provided propaganda and funding for peace groups and antinuclear activists. This does not mean that the millions of people who opposed nuclear weapons were plotting against the West. Most were ordinary citizens sincerely frightened by the possibility of nuclear conflict. But the fear was useful to our adversaries.
To weaken democratic nations, foreign powers do not need to invent anxiety or division – all they need to do is magnify it.
In recent days Trump has said the US will not be slowing down the development of AI. In Australia, for the time being, Anthony Albanese also has resisted calls to stop AI development, instead promising national rules designed to capture its economic benefits while managing its risks. Both positions are reassuring.
While AI comes with danger, we should be sceptical of any narrative that conveys inevitability around its trajectory. AI systems do not build their own data centres. They do not manufacture their own chips or connect themselves to power grids. Humans decide which systems can access the internet, whether they can control machinery, move money or operate weapons. Humans build them, finance them, deploy them and decide what powers to give them.
We also should remember that some of the stories we are told owe more to myth than to science. As Nvidia chief executive Huang has said of the AI doomer narrative: “I appreciate that many of us grew up and enjoyed science fiction, but it’s not helpful. It’s not helpful to people. It’s not helpful to the industry. It’s not helpful to society. It’s not helpful to the governments.”
Like every technology that has come before it, humans have agency over how AI is used. Instead of adopting a posture of fatalism, we should decide what kind of AI we want to build, what problems we want it to solve, and treat it as a tool rather than a force of nature.
Read my full piece for the @australian here https://t.co/1xrntmv9SQ
I think amateur mathematicians finding easily verifiable counterexamples to old conjectures (using AI tools or not) is great. Hopefully they learn something about math too. FWIW my sense is that the primary impact on the field is "saving people time," but that's valuable!
While I'm glad many got to get out and experience the Aurora over the past few days, it was pretty disappointing to see several larger accounts posting these wild graphics of where the Auroral view line will be seen and gaining thousands of engagement likes/views and gained thousands of followers from these posts, while the individuals who attempted to share actual sciences and post legitimate information were basically looked over.
I really appreciate individuals like Matthew Cappucci, who take time to add information to their graphics and go into detail into what can be expected and the difficulties forecasting space weather.
Regardless, myself and others' will still be here for a long time to come to share legitimate information about space weather, and won't clickbait you into a forecast that doesn't verify :).
"It's good progress. In future versions, it will be fixed."
I'm no longer buying this. Hallucinations, plastification of people, broken physics, gaslighting, and sycophancy. Show me any of these fixed, in the present tense, just one, and you will earn my attention.
Otherwise, release your versions silently, with CHANGES.txt, and stop wasting everyone's time with your WILD screams.
Ever cry while cutting an onion? Well, now you don’t have to! Building off of chef J. Kenji López-Alt, we found the mathematically optimal way to dice an onion. *Cutting uniform pieces has not been proven to reduce crying, but you do get bragging rights.
https://t.co/bxNrO6M787
The #Survivor community doesn’t agree on anything.
The only thing we all seem to agree on is how much we love JLP.
I’ll miss the iconic JLP laugh, the inappropriate jokes & much more. A titan of the global franchise. A staple of #SurvivorAU. A treat to watch for almost 10 years
I get especially sad when I see researchers at my university, including very senior professors, excitedly saying "Look! Now I can use <random GenAI tool> to read papers, write papers, derive equations, write code, ..."
And it's like... why are you even here, man? (1/2)
Your risk of #LongCOVID rises with each additional infection.
A recent study has shown that by their third COVID infection, patients have a 40% chance of developing Long COVID symptoms.
Learn more about the threat of Long COVID: https://t.co/l1s2QF68GH
#LongCovid can kill. It's scientifically proven it can. #LongCovid can present with cardiovascular and thrombotic manifestations. It's scientifically proven it can. If you're glossing over this, you're not on the side of patients. You are, also, not on the side of science
« There's a lot we can do that's not panicking. We can demand ultra-pasteurization. We can pull back from dairy and beef. We can finally start cleaning the air. We can wear masks. We should've already been doing these things��» https://t.co/eZts1mHRxw
This week the UK government published the results of a survey that say that TWICE AS MANY children in England and Scotland are suffering Long Covid now than a year ago.
Thread on my routine for personal risk assessment for airborne infections. A surprise at the end, that provides a compelling additional reason to monitor CO2. 1/
A list of two dozen studies as of Sept 2023 by Proal et al showing persistence of SARS-CoV-2 RNA in tissue reservoirs with both spike and nucleocapsid protein. /end
https://t.co/8jI9MfOi7l
This is true. And conversely, there are lots of scientific studies showing that COVID can cause short and long-term heart issues, which you can protect yourself against by staying up-to-date on vaccines, and taking other precautions like wearing a mask.
Not me
Not at work
Not out shopping
Not getting a haircut
No RSV, No COVID, No Pertussis, No HMPV, No Influenza
Also applies to my wife, my daughters
Just call it ‘luck’ I guess
😷 + Enovid + CPC mouthwash
#CovidIsAirborne#CovidIsNotOver
You can abstain from eating meat, fast food, and processed sugar. You can refrain from drinking alcohol, coffee and soft drinks. You can swear off social media, television or video games, but if you refuse to get infected with Covid, there has to be something wrong with you.
Eerie-looking graphic from @WHO - they are ramping up efforts to educate about #LongCovid but what is missing entirely from their approach is that individuals shouldn't bear the burden of preventing transmission