Sarthak Agrawal

I'm a policy maker, mechanism designer, installation artist, and service dog handler. All opinions my own, I don't have a boss.

[Cassie Pritchard @hecubian_devil]

The critique of EA/the AI Safety crowd isn’t that they care about AI risk; it’s that some of them decided the way to address that risk was to build the AI they were afraid of, because they thought “surely *I* can control it.”

That’s actually the problem.

[Sarthak Agrawal @tobesarthak]

@AndyMasley I think the specific issue people are having is that EAs will publicly do & say things that implies/makes deceiving outsiders morally virtuous. It makes the cost of engaging in discussion with EAs really high. in any interfaith dialogue, folks need to trust they will be heard

[Sarthak Agrawal @tobesarthak]

@thetreygoff I agree with you that mainstream effective altruism sucks. But bro their moral bankruptcy doesn’t change the extremely real risk of human extinction from AI

[Sarthak Agrawal @tobesarthak]

I’m not funded by any EA orgs for my AI safety work. I actively encourage advisees to seek other pathways to policy impact, even though my work in the US Senate/House was funded in part by the EA community. There are people working on these issues, doing the real work, and bro I swear we are also not thrilled about who is leading this debate

[Sarthak Agrawal @tobesarthak]

@thetreygoff I agree with you that mainstream effective altruism sucks. But bro their moral bankruptcy doesn’t change the extremely real risk of human extinction from AI

[Sarthak Agrawal @tobesarthak]

Before the EA internet monoculture, there is a vibrant & philosophically diverse grassroots that is super cool. Had an amazing dinner with some Christian EA’s in NYC this week and tbh hating on all Effective Altruists because you think most of their funders are die hard utilitarians would be like hating on all Christians because you think the Catholic Church sucks. We shouldn’t visit the sins of the father on the kids, bro. The kids are alright

[Sarthak Agrawal @tobesarthak]

[5/5] If you and 49 more of their constituents call with heartfelt fears, concerns, and demands for a public statement, you have an great shot of getting them to take a stand. Get them to stand, and it will change our political reality. Change that, and they serve you far better.

[Sarthak Agrawal @tobesarthak]

[1/5] I am thrilled that Senator Hawley has sent a sternly worded letter to the CEO of OpenAI, Sam Altman. I wrote the first draft of the Hawley-Blumenthal AI Risk Evaluation Act of 2025, this letter matters more, and all American citizens can help get us more.

[Sarthak Agrawal @tobesarthak]

[4/5] If you are worried about the risk of extinction or catastrophic harm from AI, it's natural to ask for solutions. A lot of solutions are things our politicians cannot give us. But a public statement is 100% in their power, and would change the entire conversation.

[Sarthak Agrawal @tobesarthak]

@SuzanMaseki @NeelNanda5 [2/2] I can't speak for anyone else, but going to government was my last resort. I'm a builder at best, a researcher at my worst. Policy was not in my wheelhouse. The issue isn't just 'culture',the specific types of AI they are building are not compatible with a free world

[Sarthak Agrawal @tobesarthak]

@SuzanMaseki @NeelNanda5 [1/2] thank you for asking @SuzanMaseki! I tried. At OpenAI I ran our first investigation into the potential for Large Language Models (LLM's) to interfere with democratic elections, some of our first systems to prevent child abuse, and worked on detecting misuse more broadly

[Sarthak Agrawal @tobesarthak]

@thinkingheights Everyone who interviewed me at OpenAI in 2021 and every manager I had until 2022 believed that OpenAI would respect their charter by unilaterally stopping AGI development to avoid race dynamics. Most of them have left, can't speak to current employees

[Sarthak Agrawal @tobesarthak]

In December 2022, I left OpenAI for DC because I feared we were developing Artificial General Intelligence (AGI), a false machine-god that could lead to the extinction of humanity or worse. Instead, I co-organized the Senate Judiciary's "Oversight of A.I.” hearings from 2023-2024, wrote the first draft of the Hawley-Blumenthal AI Risk Evaluation Act of 2025, and advised on the Obernolte-Trahan FRONTIER Act of 2026.

I'm not afraid of the AI systems being malicious. I'm terrified of them taking the things we need to survive. Regardless of whether you think AI is using too much water and energy now, we can all agree that a world where AI consumes too much of our resources would price our basic needs out of reach. We're not there yet, but we're on track to get there. That’s how the world could end; not with a flash of fire, but the silence of starvation.

The current approach to building AI systems prioritizes growth at all costs, making systems that are more and more powerful, until there will be nothing left for anyone else. The systems built reflect the broader incentive systems that produced them. The companies that want to cut costs by firing people and make money by charging as much as they can have no incentive to do anything else; is it surprising that AI systems reflect faceless corporations, conformist schools, and heartless hospitals, which always felt like machines in the first place?

And if we survive this AI crisis, we will live in the world that is designed for those who build the technical and political solutions to the problem. If AI alignment is solved for only powerful insiders, we who are left on the outside will live in their world. As an American, I was born free, and I will not sell my rights to corporations for trivial conveniences.

If you are skeptical of this problem, I'm excited to hear from you. Tell me what you need to understand my point of view, what your doubts are, and I will do my best to explain. And if I misunderstand your perspective, I welcome clarification. You have every right to demand more from your experts, and if you will let me, I would love to serve.

[Sarthak Agrawal @tobesarthak]

and on 4 — I don't want to tell anyone what they should believe. But I think if every AI researcher refused to work for companies they thought had a 1% chance of causing human extinction in the next decade, AI progress would be slowed more than all policy work that has been done in this space in the past 5 years

[Sarthak Agrawal @tobesarthak]

@bravishghosh re 3 + 4 — the pace of development is so rapid that I don't trust anyone building AI systems whose entire workflow isn't redesigned at least once a month. However we try to control AI, we need oversight to match the pace of development. The structure is secondary in my book

[Sarthak Agrawal @tobesarthak]

re 3, AI can be aligned to both the person giving it instructions (inner alignment) and to our world at large (outer alignment). Outer alignment is not possible because humanity is not aligned to itself. The best case is enabling inner alignment to the operator, which would necessarily mean disempowering the rest of us.

[Sarthak Agrawal @tobesarthak]

@bravishghosh On 2, AI refers to a broad set of approaches. In the dominant modern paradigm of Large Language Models (LLM's) used in models like ChatGPT and Claude, there is no known technical solution to alignment or control that can be scaled to the current best available models.

[Sarthak Agrawal @tobesarthak]

@bravishghosh Great questions! On 1, I think why it can't in theory be done is less important than the fact it is empirically *not* being done right now. If we want the labs to behave differently in the future, we would need to have significantly greater transparency into their internal ops

[Sarthak Agrawal @tobesarthak]

[3/3] You might not be religious or particularly ideological, but are there things you would sacrifice your life for? Are there things your family, or your community, or your country, would literally lay down their lives to avoid? How many groups would have to suffer a fate worse than death for us to believe that the world we end up in is worse than extinction?