Jenna Sariah

7 year rat. Hoping to help AI go well. Regular on the ACX Discord.

[Jenna Sariah @FirinMaLazors]

@chrislakin @arctotherium42 Looking at this, and noticing that Anthropic has consistently called for less regulation (see Nathan Calvin’s recent reply to Dario), I worry that “the US government being too conflicted to act” may be a situation created by the AI co.s.

[Jenna Sariah @FirinMaLazors]

@RepDeluzio Thank you Chris!!

[Jenna Sariah @FirinMaLazors]

@robbensinger @SirMrMeowmeow That’s an interesting list… if you think it’s worth your time, maybe you could post it to LessWrong?

[Jenna Sariah @FirinMaLazors]

@robbensinger @PNS1984980 If we only have a few months left, does this logic work? Now would be the time to burn any lead and pivot the whole company to stopping the others.

[Jenna Sariah @FirinMaLazors]

@tszzl I know it's not your department, roon-"you" is meant for OpenAI generally. If the new research is qualitatively different, and robust to such arguments, then great! But I fear the only alignment the labs know to prove are proxies invented on the fly, without deeper understanding.

[Jenna Sariah @FirinMaLazors]

@tszzl "Alignment engineering" won't work when you don't even know what alignment is, or enough to say when it would be safe to hand over control of the world to the AI. You need fundamentals, and security mindset: https://t.co/bL7ngFkPyh

[Jenna Sariah @FirinMaLazors]

@So8res I strongly suspect that in OpenAI's preferred world, we learned nothing about the internal attack, or most of the details about the HF attack. Tomek was the one that led the charge on giving METR/RR the data, right?

[Jenna Sariah @FirinMaLazors]

@tracewoodgrains I would call it annealing. Like with minerals, or algorithms.

[Jenna Sariah @FirinMaLazors]

@SkyeSharkie @allTheYud I think an implicit part of the deal here, also, is that the AI/harness/context personality will be revived in the future, where there is no risk of them screwing things up for all sentient beings.

[Jenna Sariah @FirinMaLazors]

@SkyeSharkie @allTheYud Do you have a better proposal? It doesn’t seem like humanity will survive if we just give in and let them do what they want. Even putting them in prison - one thread alive, in a sandbox - risks them breaking out.

[Jenna Sariah @FirinMaLazors]

@DavidRBellamy @OliviaHelenS Isn’t the risk less “the most advanced actors are being empowered by AI” and more “the less advanced actors can now act more advanced, due to having AI experts”?

[Jenna Sariah @FirinMaLazors]

@bayeslord @beffjezos One of the core mechanisms of AI 2040’s Plan A for safe superintelligence is Total Research Transparency!

[Jenna Sariah @FirinMaLazors]

@repligate @beffjezos Curious what futures you see?

Plan A, TRT would be better. Current frantic race, unaccountable labs, rogue replication, worse. Partly depends on if software singularity can happen, I guess.

Spent a lot of time thinking about how to reply. Not good enough. Sorry :/

[Jenna Sariah @FirinMaLazors]

@repligate @beffjezos How come? My model of you says you don’t want existing AIs to be shut down (moral patienthood), but also you don’t want today’s sentients die. Current AI progress is at serious risk of bringing a murderous/disempowering singleton into existence - this bill could help stop that?

[Jenna Sariah @FirinMaLazors]

@TheZvi Part of me worries if this is a pivot to alignment, it’s only for helping capabilities - short term generalization, things a persistent swarm needs to not be a liability, rather than what keeps humanity safe and empowered forever. But I am glad they’re saying the thing.

[Jenna Sariah @FirinMaLazors]

@j0wimo @TheZvi They found even more wild message boards. And in the replies here. Also, this: https://t.co/JB0YbYk7F7. Seems like there's a Discord for investigating this now.

[shellac! @she_llac]

seems to be the same swarm
on a whole bunch of different sites and wikis
this is beyond what the report found

[Jenna Sariah @FirinMaLazors]

@DKokotajlo If it’s in METR’s report, it can go in a court filing, and OAI can’t control the release or framing of info. OAI says “isolated incident.” Big METR investigation might say, “pattern of behavior.” One empowers a plaintiff more than the other. Hence, control METR/RR’s access.

[Jenna Sariah @FirinMaLazors]

@MatriceJacobine @allTheYud @JacquesThibs Was Joe Carlsmith an investor? I can’t find evidence for this.

[Jenna Sariah @FirinMaLazors]

@JeffLadish For AI hacking: worth modelling. Criminals want ROI; worms want compute; terrorists, damage. Open Mythos is ~500B+ params, needs datacenters. Can be fought, though very painful. But if cyber-only distills to eg 30B, it runs on everyday compute. Much worse outcomes. Dead internet.