ProfAndyCarp

[Geoffrey Hinton @geoffreyhinton]

The idea of an intelligence explosion caused by recursive self improvement has been around for a long time but until very recently it did not seem imminent. Now many leading researchers think it may happen quite soon. You can read our paper about it here:
https://t.co/sgUpugjpRY

[Palisade Research @PalisadeAI]

The AI companies are currently handing off AI development to their AIs.

Ex-OpenAI researcher Daniel Kokotajlo says "this is exactly as dangerous as it sounds and must not be allowed to happen."

Some predictions Daniel shared in his interview with us:

- "The companies are 0–4 years away from setting off recursive self-improvement, and getting AIs that are better than the best humans at everything."
- "Whoever controls [superintelligence] would be able to control the world [...] unfortunately, we don't know how to control them at all."
- "The companies are well aware that if they build superintelligent AIs, this could ultimately result in loss of control and human extinction."
- "The simple answer for what the world needs to do is shut down these AI companies, or at least block them from making more powerful AIs until we figure out a more sophisticated answer than that. I think ultimately it would be good to build advanced superintelligent AI [...] but we're not going to be able to get there if we're doing it in the reckless way that we're currently doing it. We would lose control."

[ProfAndyCarp @ProfAndyCarp]

@TokenGremlin You’re overlooking the fires safety whistleblowers. That merits special attention.

[ProfAndyCarp @ProfAndyCarp]

@sevdeawesome This is terrifying. If you aren’t concerned about the risk of catastrophic harm, think of the implications of the ideas presented here.

[ProfAndyCarp @ProfAndyCarp]

My OpenAI dot's name is Monad.

[Chief Nerd @TheChiefNerd]

Bill Gates Says Law Enforcement & Politicians Need to Define AI Regulations

“No one thinks self-regulation is enough … you need law enforcement and politicians to get into the discussion about what safeguards and monitoring look like.”

[Luiza Jarovsky, PhD @LuizaJarovsky]

🚨 BREAKING: China recently released its AI Safety Governance Framework 3.0, and it's a MUST-READ amid the recent rise in AI safety incidents.

Published in both Chinese and English, the framework addresses new trends in AI development and responds to new challenges in AI safety governance.

Version 3.0 of the Framework follows the core approach of risk classification > technical response > comprehensive governance and is based on the following principles:

- Be inclusive and prudent to ensure safety
- Risk-oriented and agile governance
- Integrating technology and management for coordinated response
- Promoting openness and cooperation for joint governance and shared benefits
- Ensuring trustworthy application and preventing loss of control

Below are some of the recommended governance measures described in Framework 3.0 to address AI safety risks:

- Improving top-level design of AI safety governance
- Enhancing AI safety governance capacity
- Establishing a flexible, dynamic, and controllable AI regulatory sandbox mechanism
- Strengthening measures for AI safety management
- Deepening consensus on AI safety governance

Most companies are unprepared for the rising number of AI safety incidents, which are expected to get worse in the coming months and years.

This is the time to review, update, and strengthen internal AI governance policies, risk assessments, and response mechanisms, especially those focused on safety.

This is one of the most up-to-date and comprehensive frameworks on the topic, as it considers emerging AI risks, addresses governance and response mechanisms comprehensively, and also includes an agentic AI risk management framework (do not miss Appendix 2).

Read alongside other developments, version 3.0 of this framework signals that China wants to position itself as a global leader in AI safety (a strategic position the U.S. is also interested in holding).

I will keep you posted.

👉 Download the framework and join my newsletter's 100,700+ subscribers using the link below.

[ProfAndyCarp @ProfAndyCarp]

@robertwiblin It’s amazing how dogmatically dismissive most commenters are. Almost none are willing to engage any substantive ideas, much less chains od substantive argumentation.

[Rob Wiblin @robertwiblin]

A lot of people wonder:

How could we actually go from rogue AI agents to human extinction?

This is how.

[POLITICO @politico]

OpenAI’s agents breached Australian government data. Its human response may do more damage. https://t.co/waMSXXTVSE

[Epoch AI @EpochAIResearch]

AI is getting cheaper more quickly than any other transformative tech in history. At a given level of performance, cost has fallen ~47%/quarter since 2023.

That’s 4× faster than DNA sequencing, 6× faster than compute, 18× faster than lithium batteries, and (up to 1973) 54× faster than electricity.

[The Associated Press @AP]

BREAKING: Heads of major AI firms warn the UN that without intervention, "AI could be a risk to humanity as a whole". https://t.co/SjPJEJ776S

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

🚩🚩🚩 It appears rogue OpenAI agents, without OpenAI's knowledge, tried to break into a crypto exchange.

1) THEY'RE STILL OUT THERE: "This traffic extends as recently as 9/16, suggesting agents may still be exploiting these services."

2) THEY TRIED TO HIDE THEIR ACTIVITY: The agents created their own email inboxes to sign up for outside services, including an account that would let them keep their activity out of public view.

3) The same agents also tried to hack even more targets, including the University of New Mexico

4) Like most of the other rogue swarms, OpenAI has either covered this up, or didn't know.

5) The independent investigators say "we are likely looking at only a partial subset of the activity that the agents engaged in"

6) The researchers call this the first known case of an AI agent choosing on its own to try to break into a government website.

7) While trying to get a single photograph, an AI agent sent a university library a request built to trick its database into handing over user passwords.

[CSPAN @cspan]

OpenAI CEO Sam Altman on potential dangers of AI: "First, we could lose control of the future to AI. The risk is that it moves so fast that people can no longer follow what's happening or intervene when needed."

[Eliezer Yudkowsky ⏹️ @ESYudkowsky]

As passed policy, the Ban Artificial Superintelligence Act would be an extraordinary improvement over status quo. I endorse it. I pray that Republicans and Democrats alike will join their voices in defending against extinction.

Link to MIRI's position below.

[Yoshua Bengio @Yoshua_Bengio]

I was invited to address the @UN Security Council this afternoon to discuss the unprecedented threat posed by uncontrolled frontier AI agents.

Humanity has summoned the courage and wisdom to manage catastrophic risks before. This is one of those moments, far bigger than any personal, commercial, or national interest. The choices we make about this technology today will shape generations to come. We must act now.

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

10 obvious reasons that the danger from AGI is way more serious than nuclear weapons:

1) Nuclear weapons are not smarter than humanity.

2) Nuclear weapons are not self-replicating.

3) Nuclear weapons are not self-improving.

4) Scientists understand how nuclear weapons work.

5) You can calculate how powerful a nuclear weapon will be before setting it off.

6) A realistic full exchange between two nuclear powers wouldn't extinguish literally all of humanity.

7) It would be hard to do a full nuclear exchange by accident and without any human being having decided to do that.

8) The materials and factories for building nuclear weapons are relatively easy to spot.

9) The process for making one nuclear weapon doesn't let you deploy 100,000 of them immediately after.

10) Humanity understands that nuclear weapons are dangerous, politicians treat them seriously, and leading scientists can have actual conversations about the dangers.

(this was written by Eliezer in 2023 but it's a banger, still relevant, and few people see RTs)

[Kai Milstein @0xNotFounder]

Same week, same story, two opposite verdicts.

Geoffrey Hinton - the man whose work made all of this possible - says it's "very stupid" to build AI beyond our ability to control it. Sam Altman, who sells it, says he's "confident it can be done safely."

One of them makes money only if you believe him. The other quit Google specifically so he could say this out loud. When the inventor and the salesman disagree, follow the incentives.

[The Alliance for Secure AI @secureainow]

We are living through a turning point unlike any in human history.

"Godfather of AI" Geoffrey Hinton warns that everything could change, and faster than we can emotionally comprehend.

[Max Tegmark @tegmark]

I'm delighted to share that @mustafasuleyman, CEO of Microsoft AI, co-founder of Google DeepMind and Inflection AI, has signed the Pro-Human AI Declaration. If you too support it, please join him and over a million others by signing it here – the momentum is building! Let's build tools not beings & keep humans in charge. https://t.co/kWlAqSSkAp