Seth Lazar

Philosopher working on AGI alignment, governance and adaptation at Johns Hopkins School of Government and Policy, and at Resolution.

[Seth Lazar @sethlazar]

Our AGI Governance fellowship has been such a blast. Let me take you a bit through what we've done, and where we're heading.

The fellows have superbly diverse backgrounds, including military and humanitarian service, geopolitical expertise at NATO, leading AI policy in state and federal government, lawyers, journalists, brilliant early career academics across political theory, control theory, and behavioural science, and of course a few cracked safetyistas.

Our goal for the fellowship was to make a start figuring out what AGI governance actually is, to spend some intensive time with myself, @ghadfield and @nickacaputo going through the frontiers of research (~3 weeks of full-time contact time), and expose the fellows to as many brilliant people working in this urgently-needed field as we could, from across industry, government, non-profits and academia.

We've taken them through stratospheric questions like whether we should build AGI at all, and how to rebuild liberal democratic institutions for the age of AI. We've gone down into the trenches of geopolitics and regulatory innovation—what a time to be taught by @ghadfield, world-leading scholar in regulatory markets and independent verification organisations for frontier AI! And we've explored single- and multi-agent alignment, the actual politics of AGI, and how powerful AI can productively be used in government.

All this was built on foundations including a detailed discussion of AGI and related terms, and a cool-headed analysis of precisely where we are on "the curve".

This week the fellows are working on a final project, where they're going to take these ideas and build what they think the world needs to see in this space. Excited to share those with you. Papers are not the priority, we're looking for interventions that can move the needle, from concrete governance proposals, to model legislation, to reporting platforms, &c &c &c

We want this to be a kind of "final boss" of fellowships; we've selected people who are ready to get stuck in actually doing the work that needs to be done to prepare society for powerful AI, and our hope is that they'll be going from us to influential positions across industry, non-profits, and government as well as academia—and that in years to come they'll see one another on the same or opposite side of negotiating tables, and help bring about positive outcomes for liberal democratic values in the AGI transition.

We're tremendously grateful to our friends and colleagues across AGI alignment and governance for joining us. Tagging them here to say thanks! @zhitzig @deanwball @BuchananBen @anton_d_leicht @edelwax @curl_justin @hlntnr @CharlieBull0ck @NatPurser @AlecStapp @zdch @TechnOlliegist @kevin_klyman @AndyMasley @hamandcheese @henryfarrell @peterwildeford @_NathanCalvin @rSanti97 @kang_megan @apmechan @JustinBullock14 @David_Kasten @weballergy @FranklinMatija @DavidSKrueger
Massive thanks to the @JohnsHopkinsSGP team for all the behind the scenes work.
And more than anything huge thanks to our fellows who have helped us build this plane while we're flying it. They're not all on here but they will be soon (thanks to excellent advice from @NatPurser).

@diatkinson @leonieclaude @ghaelfobes @wild_and_empty @elsie_jang @LukeThorburn_ @synchroaphasia @kevinlwei @ey_985

Also look out for Gabrielle Hibberts, Ciara Horne, Carmen Amo Alonso, Gabrielle Hibbert, Ciara Horne, Jessica Ji, Anagha Late, Alex Nelson, Maximilian Pralle, Shivam Saran, Daniel Trusilo. Expecting big things from this cohort!

[Seth Lazar @sethlazar]

Announcing a 3 wk AGI Governance Fellowship at @JHUBloombergCtr School of Government and Policy.

Join me, @ghadfield, @nickacaputo and guests for an intensive schedule of deep dives into AI governance at the frontier, covering topics such as societal resilience, AGI and democracy, and the new institutions that governing in/with AGI will demand; we will range from foundational philosophical questions to the tip of the legislative spear.

The arrival of AGI would be challenging enough if our ~250yo institutions were in full health. That they aren’t both increases the scale of the challenge, and creates a unique opportunity to imagine the institutions that will carry us through the next 1/4 millennium.

We are looking for ~20 fellows, aiming to attract and shape the people who will start to build those institutions: future leaders in AGI governance. We expect you to come from many different backgrounds, so hope this call will be shared far and wide. Details, eligibility, and how to apply here: https://t.co/k4IdSSpmrM

Governing the AI transition is going to be a central focus of our new school, and we have many more initiatives coming. And we will be hiring in many different kinds of role, from tenured and tenure track faculty to research scientists to ops and comms and students and beyond.

[Seth Lazar @sethlazar]

@mark_lundin Hey Mark!!

[Seth Lazar @sethlazar]

Thought I’d give the $500 codex sub a go cos Claude still won’t allow launching new session on local device from phone.

Somehow my usage remaining just gets recalculated this arvo, goes from 82% with reset coming in 6 days to 0% w reset tomo??

Super frustrating, can’t plan around a system w completely unpredictable consumption. And surely $500 should just basically remove token anxiety entirely.

Pls @bcherny just get CC working well w iPhone and local machines, I won’t look back…

[Seth Lazar @sethlazar]

I’m not sure about this. Didnt work that well in 2020-2022 for workers at previous tech incumbents, and only a few people in the companies are actually irreplaceable. I don’t know the literature, but this doesn’t seem like an industry where unionisation is going to be the solution. Happy to see counter arguments though

[Seth Lazar @sethlazar]

This is great from @dgrobinson .

One thing I’ll say: if you’re in a frontier ai company and you feel similarly, there are many more options besides continuing to try—and probably fail—to bring about change from within. I’ll obviously shill for academia here since I think true independence is deeply valuable, as is training the next gen, but there are also so many alignment and advocacy orgs, as well as gov opportunities, to consider. The world is your oyster; there’s no need to stay inside the machine just to feel some sense of control and agency (and excitement).

I also think junior people just finishing up their training should pay close attention to what David and others have been saying. It always feels like your experience is going to be different—you’ll actually have freedom others have lacked; your CEO has vision that you’re aligned with etc. But trillions of dollars have their own inevitable momentum and they don’t care about your compunction or keeping your hands clean.

[Adrienne LaFrance @AdrienneLaF]

New in The Atlantic:

@dgrobinson resigned this week. He was among the longest-tenured employees at OpenAI—and oversaw safety reports on 12 frontier launches.

He is very worried: “The time for trial and error is over.”

You can read his essay here:
https://t.co/DQlszrc0ru

[Seth Lazar @sethlazar]

@mitchellbosley @get_bb_app All good had it update itself

[Seth Lazar @sethlazar]

@mitchellbosley @get_bb_app Oh fuck that sounds amazing

[Seth Lazar @sethlazar]

@mitchellbosley @get_bb_app Hmm have you got all your normal models available? Jus installed and it doesn’t show either sol 6.1 or opus 5.5

[Seth Lazar @sethlazar]

@nathan84686947 Yeah I like it living on the desktop tho, and when I used tailscale I’d constantly have to reconnect sessions. Codex remote works perfectly, it’s the best feature of the app… Totally seamless across all machines.

[Seth Lazar @sethlazar]

@xave_rg Yeah tho they call double coat something weird…

[Seth Lazar @sethlazar]

I *won’t* be seeing folks at The Curve, or at CoLM where Cameron and Lorenzo will be sharing our “Blind Refusal” paper (details soon). Instead I will be working my way through ~176 boxes that have arrived today from Australia.

We went a bit nuts buying australiana before we left, so I think the end state here is that our house in Chevy Chase becomes a kind of shrine to the wide brown land…

[Seth Lazar @sethlazar]

@mitchellbosley @get_bb_app This looks interesting but does it work w Claude subscription and does it use the desktop agents? I prefer that to terminal these days because I have long running threads

[Seth Lazar @sethlazar]

Yeah I know I’m a complicated person… but seriously it’s the difficulty of replicating my multi machine setup w Claude code that has me stuck w OAI for now. Not only can’t start new session on local machine from remote, but also they’re constantly disconnecting from each other…

[Seth Lazar @sethlazar]

@JeffLadish I love the idea that you’re chasing the swarms while walking around holding your laptops up in the air :)

[Seth Lazar @sethlazar]

@natolambert @DanielleFong @trillium_labs Congratulations Nathan! Excited for what you’ll do together.

[Nathan Lambert @natolambert]

Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.

We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.

Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.

I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.

I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.

[Johannes Himmelreich @jhimmelreich]

@sethlazar It’s already begun hasn’t it — cloudflare turnstiles everywhere making it so much harder to get information from sites that a normal browser session can access easily.

[Wyatt Walls @lefthanddraft]

The funniest thing about AI consciousness discussions online is how few people think "I wonder if anyone has considered this point before. Maybe I should check if anyone has spotted a problem with it"

[Seth Lazar @sethlazar]

With Muse, Dot, GrokBot and all the others, we're seeing the rise of the platform agent. Here's how it'll go. To start with, your agent won't be able to do everything online that you'd be able to do, and that'll be annoying. Amazon will block your Muse, X will block your Dot. Then Meta will meet with Amazon, and OAI with X, and eventually they'll figure out some cozy little deal that allows each of them to extract their share of value from you. And then your home-spun agents won't be able to do all the things that the platform agents can do, and it'll just be easier to give up and chuck all your data onto Mark Zuckerberg's computers, and once again trade convenience for their control, and cede even more power than in the first pass.

I love Josh and Kanjun's vision for personal computing; I want to be able to spin my own agents and have all my data on my own local computers. But to make this real, and competitive, we need not only cool software, but laws that prevent the kind of cartel-like behaviour that means the only agents out there will have divided loyalties; working for the platform, and the AI company, and maybe you as a distant third. One simple proposal is to ensure that any agent that meets some minimum criteria and is provably acting on behalf of a principal is permitted to do ~anything with a computer that the principal would be permitted to do if they were at the computer. Force this by law so that big AI companies can't squeeze out open agent advocates.

https://t.co/sNUvwnlEqp

[Josh Albrecht @joshalbrecht]

It's time for a personal computing revolution

We can now pull all of our data locally and build anything we want. Do we even need big tech companies at all anymore?

https://t.co/opsLYnW3ud

[Séb Krier @sebkrier]

Ethics (neosalty parrot stream), ethics (model behavioir, STS), model whisperers (janus ecosystem), model whisperers (tpot), model whisperers (psychosis), telecoms/IT people, old school doomers (MIRI, FLI), new school doomers (misc), ACX/rationalist-but-not-doomers, MARL/ABM world, accelerationist (brainless strand), accelerationist (Landian strand), Progress Studies, Effective Altruism (different flavors per geography), European AI technocrats, d/acc & co, cyber security people (old school), cyber security people (ex-doom), disempowerement/econ pessimism tribe, naive optimism tribe, New Right techies, open source world, distributed/decentralized training crowds, consciousness people (Eleos et al), consciousness people (Seth et al), heterodox long-termists (Hanson, Bach etc), computer vision refugees, national security hawks, Big Tech public affairs people, VC people (a16z etc), AI x art (Restless Egg, Antikythera etc), cryptids, AI economics (Windfall etc
), collective intelligence people, Anti-AI world (Pause, Stop etc), Anti-AI art world, neurosymbolic terroir (LeCun, Goertzel), AI policy (secret doomers), AI policy (technocrats), evals people (Owain, METR etc), privacy people, AI music people, LinkedIn braindead posters... Many overlaps of course!

[Seth Lazar @sethlazar]

@ruthstarkman @pangram what kind of false positives? I've encountered very few so far, and I find the research itself convincing. If you do find FPs, I'd suggest sending the results in, as making sure detectors work well is something that benefits everyone

[Seth Lazar @sethlazar]

I wish everyone on AI twitter knew that there is a @pangram extension that you can just have on chrome, which automatically scans your feed for AI. It might seem like you're saving time by having AI write or edit your tweets, but I am 100% skipping anything in my TL that has the AI flag, and will be less likely to read the next thing you post. Authenticity matters more than getting the wording precisely right; when you use AI to get the wording right, you're almost certainly fallnig down into a stylistic basin that will make readers want to claw out their eyes.