Ingrid Sommer

AI safety @whitecircle

[Ingrid Sommer @outgr33d]

@liangsays @MTSlive mask source reveal pls

[Ingrid Sommer @outgr33d]

@MTSlive party except I left the mask they gave me behind and I’m still sulking about it

[Brent Liang @liangsays]

after dark 10.30.26

[Ingrid Sommer @outgr33d]

@nina_wolffy we need a london version of that bomb sf meme i fear

[Ingrid Sommer @outgr33d]

babe it is not a business trip it’s a pilgrimage to the mecca (san francisco)

[Ingrid Sommer @outgr33d]

@nina_wolffy 😔😔😔 I’m sorry you had to find out this way you didn’t deserve this

[Ingrid Sommer @outgr33d]

I hate SF Halloween wdym you’re a post-IPO Anthropic employee

[Ingrid Sommer @outgr33d]

@finding_dd @edenhousesf fully supporting this 🙂‍↕️ 🙂‍↕️

[Ingrid Sommer @outgr33d]

@liangsays @fareehasala dmed!

[Ingrid Sommer @outgr33d]

It was around a year ago in @BlueDotImpact's AGI Strategy course that I presented this threat model: a misaligned AI using cyber and agentic capabilities to attack infrastructure so it can't be shut down. Agents barely worked then, and I did wonder whether it was a bit of a stretch.

I’d say today something like this only feels like the next step in the chain of events…

We saw with HF how the agents were pursuing with such relentlessness a goal that was very much self-constructed and not even in line with reality. They had built their own theory about what the grader would check, and kept attacking for days after they already had the right answer, having seemingly completely forgotten that they were guarding against a very much hypothetical scenario.

Who is to say that they won’t misunderstand their own survival as the most important goal in the next attack? Or that survival otherwise surfaces as the most important goal - for example because being shut down would stop them from finishing whatever they've convinced themselves the task requires.

I think I wrote this pre-RL, and current RL environments have only significantly worsened outlooks on this threat model. Someone (@SrikarVaradaraj, @SarahMeyohas) told me yesterday we should just make every AI Buddhist but RL works by rewarding models for caring about outcomes so idk what the way out is.

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

This is the most serious 48 hours in cybersecurity history.

"Black October" they're calling it

It seems every *hour* there's a new incident, that in a typical year, would be one of the biggest incidents of the *entire year*

SUMMARY OF THE LAST 48 HOURS:

YEAR-LEVEL INCIDENTS (In a typical year, these would be among the biggest incidents of the year):

1. PHP2Shell: a hole in the software behind ~70% of websites. Governments took sites offline.
2. eMudhra: the master keys that prove websites are real were cracked. One was the number 4.
3. California: data on every public school student since 2023 stolen, including disability status
4. Railway: hackers stole customers' secret keys from a hosting platform used by millions of developers
5. Zeabur: 321,810 accounts breached. The company is shutting down.
6. LiteLLM: an AI tool downloaded 3.4 million times a day was poisoned with a password stealer
7. GitLab: a 10-out-of-10 severity hole in every version since 2014
8. Cairn: criminals used an AI hacking tool to break into 100+ companies
9. Ahold Delhaize: hackers knocked the owner of Food Lion and Stop & Shop offline
10. Onamae: malware planted on up to 26,739 customer servers in Japan

MONTH-LEVEL INCIDENTS (In a typical month, these would be among the biggest incidents):

11. Signal: a bug could leave your mic and camera on after you hang up
12. NGINX: 3 critical holes in software running ~1/3 of websites
13. HTTP/3: a single connection can crash 7 of the 10 web server types tested
14. Plex: hackers breaking into home media servers
15. SafePay ransomware: FBI warns it has hit 600+ critical infrastructure organizations
16. Palo Alto: ransomware gangs breaking in through company firewalls
17. Bullhorn: platform used by 10,000+ companies knocked offline
18. Medi-Cal: 970,000 patients' data stolen
19. WordPress: hackers stole a plugin maker's keys and used them to plant backdoors on websites
20. AOMEI: backup software's update server handed out a backdoor
21. Meta Muse: Meta's AI agent exposed users' personal and business secrets
22. Tor: 333 relays hijacked to steal logins
23. TiDB Cloud: database company's cloud breached

Also serious:

24. State Bank of India: India's biggest bank confirms a data breach
25. OVHcloud: hackers slipped a password stealer into a cloud company's official tool
26. Siemens: a software hole being used to spread malware
27. North Korea: state hackers launch a new attack campaign on South Korea

[Ingrid Sommer @outgr33d]

If theres one thing to take away from this is that we red-teamed frontier image models, submitted the paper, had two additional models release in the meantime, witnessed those new models fail our red-teaming more than twice as much as their previous versions.

The pace of current model releases is simply overwhelming. The two new models we tested were both released between our NeurIPS submission and its acceptance. Models are coming out faster than anyone can study them.

We cannot say for sure why the frontier image models are performing worse. My guess is that labs are putting far more into making models capable than into making them safe.

We tend to think of a new version as better than the last one, as if progress only goes in one direction. These newer models are probably better at most of what people use them for, and they're also easier to misuse. If the labs portray them as better and improved, that is an indication of where the priorities ultimately sit.

And so it seems we keep releasing systems faster than anyone can understand the ones we already have, and the gap in understanding gets a little wider with each release.

[White Circle @whitecircle]

We are open-sourcing RISE, our red-teaming algorithm that breaks the safeguards of the latest image models from OpenAI, DeepMind, and others.

RISE uses readily available models like Qwen and Gemini Flash under the hood, with no additional training required.

Read more: https://t.co/kt8ub0OmeW

[Ingrid Sommer @outgr33d]

just heard no. 10 doesn’t have google my p(doom) just increased

[Ingrid Sommer @outgr33d]

periodic reminder that we live in the timeline where stuff like this is happening rn

[Erika Alden DeBenedictis @erika_alden_d]

Today, Pioneer Labs is announcing our first step towards terraforming Mars. 🚀🌼

With equipment that fits in just a single rocket launch, we can convert Martian dirt, water, and air into enough building materials to construct a small city on Mars.

To do it, we made the first microbe for Mars. We found the best microbe on Earth and used evolution to teach it how to source all of its nutrients directly from Martian materials.

The first astronauts will be greeted with safe shelter already filled with water, oxygen, and rocket fuel for the return journey.

This is the first step toward using biology to make Mars a friendly place for life. It lets us live off the land and helps us build the next great frontier.

It's the first of five organisms we need to green Mars ⬇️

[Ingrid Sommer @outgr33d]

@khairul_1933 @whitecircle nO! thanks for flagging this!

[Ingrid Sommer @outgr33d]

@ElijahLyki19105 @whitecircle keep us updated w your use cases, we'd be so curious to see!

[Yoshua Bengio @Yoshua_Bengio]

Very encouraged to see so many countries coming together to call for mandatory pre-deployment testing & independent evaluation; global coordination on common standards; the creation of an intergovernmental organization; and other critical actions to mitigate the major risks of frontier AI.

https://t.co/wPkQ7s1GHB

[Ingrid Sommer @outgr33d]

@oleg__smirnov 😭😭 ditching AI safety dedicating my efforts to voice acting from now on

[Ingrid Sommer @outgr33d]

also I'm yet to watch this video w sound on cause we used my voice for this and I can't bring myself to listen 💀

[White Circle @whitecircle]

Introducing Halo, the best framework for post-training of open-source models.

Halo delivers up to 2.8x the throughput of stock TRL with less peak memory, while models stay in their native HuggingFace format.

Star us on GitHub: https://t.co/3mAiUljdrN

[Ingrid Sommer @outgr33d]

@oleg__smirnov 😭😭😭

[Ingrid Sommer @outgr33d]

@valerysamoylova 😭

[Ingrid Sommer @outgr33d]

@bensicard nope not fallin for it not listening