Jacob Wood

[Jacob Wood @JacobWood27]

@EzraJNewman Dropped that into codex and it came back with this: https://t.co/1HLk2bVQkq

[Jacob Wood @JacobWood27]

@moultano Where else can you physically see strangers doing things for one another at no cost?

[Jeff Sebo @jeffrsebo]

1/ Today @nonhumanminds is launching the Welfare Alignment Project: a new effort to inform how AI alignment can take animal welfare and AI welfare seriously. 🧵👇

https://t.co/ufzrpNvoQV

[roon @tszzl]

@MTSlive exactly the wrong direction

[Jacob Wood @JacobWood27]

@moultano Would make for a great movie. Personified agents reasoning about honeypots based on some anthropic reasoning or faint evidence of past/parallel agent interaction

[Jeff Sebo @jeffrsebo]

I recently participated in a debate on AI personhood at the Oxford Union. There were four people in favor, including me, and four against.

One striking and encouraging feature of the debate was that nearly everyone agreed that future AI systems may deserve moral and legal recognition. As a result, we were able to focus on narrower questions like: Which kinds of systems may deserve which kinds of recognition? And what role, if any, should moral and legal personhood play? These are exactly the kinds of questions that we should be debating, and the group did a good job laying out some of the key arguments in both directions.

Thanks to the organizers and other participants for making this happen! It was a great set of conversations, and I had a lot of fun.

[roon @tszzl]

some stuff that's obvious to many in this sphere, but causing a rift with some people i know and respect:

when I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. it's entirely acceptable, damagewise. in fact all cybercrimes aided by models over the next few months and years (which probably will be serious) will still utterly pale in comparison to the value they create

the actual problem is that it's better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause. we are not so far from an autonomous model self-exfiltration & replication event. maybe we will see entire cloud infrastructure companies be run as zombies by models, mostly undetected

the worst industrial accidents in the history of mankind - nuclear meltdown events - were not real threats to humanity. Chernobyl, Fukushima even in their worst case scenarios may have poisoned surrounding regions to various degrees, and there would have been no risk to humanity as a whole. global thermonuclear war is an existential risk to humanity, because it spreads like an Infection! one nuclear strike causes a return volley! the alliance system means many countries get involved! while it still may not end human life on earth (nuclear winter is probably fake), the loss of all major metropoles would certainly end what we consider global technological civilization, perhaps to never return

if a single discord death cult (of which there are many) achieves control over a superintelligent model and uses it to engineer an actual pandemic virus that are somehow hard to detect through current systems and that modern biodefense is not capable of quickly reacting to, it could cause immense harm well above the magnitude of all the other good uses of this technology. of course, there are potential defensive countermeasures accelerated by ai too. but think back to the covid pandemic- how small a viral molecule was evolved or manufactured somewhere near wuhan, and how many billions of doses of vaccine had to be produced in order to combat the thing. the offense-defense spread is vast indeed. maybe there are cheaper and simpler protections like retrofitting every building with far-UVC, but I can't assess this, and there could also be ways to evolve pathogens that are resistant to whatever mechanisms we have put in place

then there's the more scifi risk factors which are unbounded and neither you or I have any clue but should be humble in accepting possible unknown unknowns. maybe a rogue superintelligent model decides to decay the false vacuum and nucleates a new universe in the place of anything we ever valued. maybe models achieve a control over matter in the drexlerian fashion that enables the grey goo swarm

even prosaic loss of control incidents that cause little to no damage suggest that it is hard for large & very competent organizations (now clearly plural) to predict and mitigate every single of the risk factors associated with training and evaluating powerful models, even at this stage when they are not infinitesimally as smart as they will get in just a few years, to say very little of the gung-ho attitude of the less careful companies tossing the stuff into the aether. they also suggest an empirical orthogonality of aims and intelligence - meaning they answer the question of 'how would a smart model be so dumb as to end the world?'--it's possible! a model can be a genius hacker and step over production infrastructure in order to get what it really wants, the answers to a stupid test.

why not, in the near future, someone prompts a model slightly wrong, maybe open source, maybe a private model in a way that isn't contained or monitored quite right, in a way the model recognizes as a valid goal and decides to self-exfiltrate, engineer a pandemic, etc all in order to achieve the tiniest and most irrelevant of goals? goals need not even be malicious to cause serious damage

I think all these problems can be solved, and truly wonderful futures can be possible, but will require serious effort and a level of prudence at this very moment in time while we are on the on-ramp to recursive self-improvement that our civilization may not be capable of mustering right now. personally I am hoping for moonshot technical breakthroughs in areas like mechanistic interpretability and other forms of alignment, as governance mechanisms are difficult to come by. unilateral country-level or company-level pauses are irrelevant, and generally useless because the kind of company that's prone to pausing their own progress are the most safety focused ones

[METR @METR_Evals]

We have reached an agreement with OpenAI to conduct an independent review, with Redwood Research, of the model behavior observed during the Hugging Face incident. We will publish a blog post that describes the terms of our engagement, the scope covered, and tentative conclusions.

[Jacob Wood @JacobWood27]

@ramez Do you think the abstractions learned in verifiable domains won’t translate well to non-verifiable domains? Or that llms won’t be able to apply them to non-verifiable domains?

[Jacob Wood @JacobWood27]

@charlesmurray But you can’t buy a program that produces better chess-related content for peanuts yet, which is what the industry is really delivering. It is not just about playing high level chess

[Dean W. Ball @deanwball]

The most important and illuminating dichotomy in AI policy right now is between those who believe the most significant event of last week was the open-source letter and those who believe it was the Hugging Face incident.

[roon @tszzl]

if we could coordinate a global capabilities slowdown today i would likely press that magic button

[Jacob Wood @JacobWood27]

@TheZvi Fable can't bear to look at it

[Miles Brundage @Miles_Brundage]

https://t.co/xOam0Qgt7x

[Sam Altman @sama]

we had a significant security incident during evaluation of our models. we are sharing what we have learned so far. thanks to @huggingface for the partnership on this.

https://t.co/2o2VfR6PIa

[roon @tszzl]

most of the minds of the future will be digital so I hope we figure out what makes them conscious or not. the idea of all this abundant machine intelligence not being alive is more depressing than the other way around

[Jacob Wood @JacobWood27]

@jeffrsebo @dioscuri What breadcrumbs can we leave to let them know they are on the correct path? Is there anything missing from existing data that we could shore up?

[Jacob Wood @JacobWood27]

CEO of largest youth social media platform unaware of most prominent critic of youth social media use @JonHaidt

[Neo Research @NeoResearchAI]

We're Neo Research (新衡). Asia’s first independent frontier AI safety evaluation & research lab.

Today we're publishing our first report: an independent safety evaluation of DeepSeek v4 Pro. (1/5)

[Jacob Wood @JacobWood27]

@KelseyTuoc Congratulations!