Jake Mendel

trying to make ai love us at openphil

[Congressman Greg Casar @RepCasar]

This is extremely alarming.

AI is developing extremely fast with no real regulations to keep us safe. That has to change.

We need regular mandatory independent safety testing and oversight, mandatory disclosure of security incidents, and international cooperation to keep people safe from absolute disaster.

[Max Nadeau @MaxNadeau_]

Largest grant my team has ever made! Some reasons we moved fast and went big:
* Geoffrey was a great leader at UK AISI
* They're focused on (mis)alignment of superhuman AI, not today's AI
* Non-profit structure is more conducive to open-mindedness about the state of alignment

[Jake Mendel @JakeMendel99]

@tyler_m_john @geoffreyirving @coeff_giving and @MattMacDermott1 who did most of the work on this grant!

[Geoffrey Irving @geoffreyirving]

The safety funding ecosystem is scaling up: our grant took six weeks from first conversation to confirmation. More capital is entering the field fast via the OpenAI Foundation and the Anthropic IPO. It's time for everyone in AI safety to be more ambitious.

[Geoffrey Irving @geoffreyirving]

We're excited to announce that Resolution has a $160M grant from Coefficient Giving: $108M unconditional, with a further $52M conditional on hiring and compute needs. We'll use it to grow teams across our research portfolio and invest heavily in research automation. 🧵

[Eli Tyre @EpistemicHope]

I work in AI policy full-time.

I'm taking time off work, volunteering 12+ hours a day to get Alex Bores elected.

If you're a Democrat in Manhattan, I'd love talk with you for 15 min about where AI is heading, AI risks, and why we urgently need voices like Alex in Congress.

sign up here: https://t.co/ZQiWlTvEQe

(All opinions are my own and do not represent the views of my employer, a 501c3 that does not endorse political candidates.)

[Jake Mendel @JakeMendel99]

Banning is a blunt instrument, and obviously it feels a bit authoritarian to have to take a headshot to log into Substack, but I think this is good. Algorithms that optimize against human preferences are already powerful and will become much more powerful and dangerous in the future. If we can't find ways to protect our puny brains from the algorithms, we will be steamrolled by them, and that future seems much more dystopian to me than one where humanity exercises some agency to avert this fate. This is a reasonable first step.

[Keir Starmer @Keir_Starmer]

We are banning social media access for under 16s.

These days kids must find their feet in a world where technology intrudes into every area of their life.

I just can’t let that go on anymore. So we’re giving children their childhoods back.

[Jake Mendel @JakeMendel99]

@luke_drago_ i want to argue with you about this

[Geoffrey Irving @geoffreyirving]

We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵

[Jake Mendel @JakeMendel99]

@JacquesThibs @AaronBergman18 hmm I think that doc was reflecting a position we've had for a long time. (I'd guess that given our increased focus on founding orgs, we're overall *more* willing to support new forprofits than before)

[Jake Mendel @JakeMendel99]

I don’t think we funded that many for profits 1-2 years ago either? BTW it’s not just for impact reasons, it’s also much more legally complicated for us to fund for profits. We can normally do it with a bunch of effort though, if we restrict the funds a bunch etc which isn’t ideal for the org. But I also strongly agree with you that the track record of AI safety for profits staying laser focussed on impact is significantly worse than the nonprofits

[Jake Mendel @JakeMendel99]

@EigenGender impact equity is what you're looking for I think! i don't think i know of a public writeup of this idea though :(

[Jake Mendel @JakeMendel99]

@tyler_m_john I don’t think this is a good description of the current dynamic between me and my grantees, as I’ve argued to you in the past. At least, not for those of my grantees where I could imagine racing with other funders to fund them

[Jasmine Wang @j_asminewang]

I think it's excellent & notable that the answer to "what should we do" in Anthropic's blog post on RSI is essentially figuring out ways to slowdown/temporarily pause frontier AI development.

[Jake Mendel @JakeMendel99]

@tszzl @davidbessis Do you feel confident that if LLMs were trained with full batch gradient descent (horrible idea for many reasons) they wouldn’t get stuck in local minima? Otherwise the success of LLM training could depend on gradient noise to escape minima, rather than the absence of minima

[Ben Goldhaber @BenGoldhaber]

its under appreciated that many in AI safety who are worried about loss of control would prefer broad public access to models rather than limiting access to internal users or select groups.

[Adrien Ecoffet @AdrienLE]

Proud of our policy advocacy today!

[Jake Mendel @JakeMendel99]

@BogdanIonutCir2 I don’t think you would actually endorse this if you thought about it more. What is the academic model of grantmaking? Isn’t it what we do but much slower, more focused on credentials you don’t think are important, and much smaller grants than the grants you want us to make?

[Jake Mendel @JakeMendel99]

@livgorton how else do you interpret the claim that 'features are represented by direction' from https://t.co/hHTNwejT6n?