Rafi Ayub

Trained AI models, now I write about how they're training us. Currently @spc. Formerly applied research @AnthropicAI, @MetaAI and @PyTorch.

[Rafi Ayub @theayubinator]

AI could’ve been a modern moon landing to rally around. Dario even had positive aspirations of seeing AI-accelerated research curing cancer. What if that was the sole mission and messaging?

Can anyone point me to a sociopositive, grounded vision for integrating AI into our lives that’s actually being worked on? Or do we need to bury this technology right now

(No, coding and reading my emails do not clear the bar)

[Evan Hubinger @EvanHub]

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

[Rafi Ayub @theayubinator]

@JasonMarder Everyone has different levers. For my goals, it required the freedom of being outside the system

[Rafi Ayub @theayubinator]

I applaud anyone who leaves a company for philosophical reasons. I did the same and left Anthropic post-training. But here’s where he’s wrong:

[Jacob Coxon @hilbertspaess]

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

[Rafi Ayub @theayubinator]

Wholeheartedly agree, this is a systems and incentives problem. We can try to speak loudly about our concerns while taking VC money and readying for IPO but if we’re not changing the structures around us then how do we actually move forward? I don’t have an answer, I am trying to figure out what that institutional invention looks like

[Rafi Ayub @theayubinator]

Not saying I’m a hero and not saying I witnessed some horrible deed that made me concerned about AI safety. There’s a wide range of impacts of AI in our day to day life that I would much rather tackle outside of existential risk that are more subtle yet consequential. I decided my highest leverage move was to do this on the outside

[Rafi Ayub @theayubinator]

@_aliuofficial It can’t be done by a venture-backed company, that’s for sure

[Rafi Ayub @theayubinator]

Right, we’re focused on the most extreme scenario of what does an AI global catastrophe look like and completely ignoring the everyday misuses of AI. No one will care if I use AI to write and read everything for me for the next decade, but I might care if I realized that I’ve learned nothing.

What does the ideal human-AI partnership look like that doesn’t lose our humanity?

[Rafi Ayub @theayubinator]

@set_trace @Turn_Trout @blueprintsmb22 we need all the ex-lab folks to ban together and create another lab to end all the labs

[Rafi Ayub @theayubinator]

We need to focus more on the institutions actively talking about current ways AI is affecting us. @Brendan_McCord of @cosmos_inst has always stressed that our first duty as a builder in this space has always been philosophical. @HumaneTech_ and @StanfordHAI are doing the work to align frontier AI with human values and incentives

[Rafi Ayub @theayubinator]

This is what’s really happening, and this is why I left, so we can actually use AI with the intention of human flourishing instead of concentrating wealth. Instead of putting the smartest minds on some hypothetical doomsday and fearmongering, lets focus on the discussion around a positive vision of AI that enhances our communities instead of eroding them.

[Rafi Ayub @theayubinator]

HuggingFace being acquired by NVIDIA marks the end of the last independent home for open AI research. What was once a thriving, collaborative community is now a profit machine, controlled by a handful of CEOs.

These collaborative communities are the few spaces where we can celebrate human achievement and connection, and they're being captured by capitalist incentives.

AI is accelerating perverse incentives in open source software and research, and it could lead to institutional collapse. More on this, in my latest substack essay: https://t.co/BQFCMNHc5X

[Philip Bontrager @FilipoGiovanni]

With models getting larger, it's gotten much more difficult to interactively work with them. I made a distributed Jupyter extension that allows instantiating and interacting with distributed models.

[Rafi Ayub @theayubinator]

@mhashemi @pycon @claudeai The best Python human expert to correct Claude 👏👏👏

[Mahmoud Hashemi @mhashemi]

Super stoked to announce @pycon videos are out! Including mine! Wherein I speedrun all the things @claudeai got wrong about Python in the browser.

[Rafi Ayub @theayubinator]

Alignment training has created mode collapsed assistants that don’t serve your interests and channel the values of the machine behind creating these frontier models.

We need to democratize post training. Generic intelligence is a bottleneck.

[Rafi Ayub @theayubinator]

User sovereign models == aligning technology with human needs

[Jonathan Ward @j_upward]

In June, I left OpenAI after five years as a posttraining researcher and, later, a research manager. I worked on ChatGPT apps, memory, early coding agents, human data, and synthetic data methods.

The past couple of months of reflection and exploration have reinforced my conviction that the field is held back by our inability to load real human goals into today’s LLMs.

So I’m now working toward sovereign, user-owned models. You should have a model that is faithful to you, completely understands you, and is fully controlled by you. Your digital life produces a continuous stream of facts, workflows, feedback, preferences, and goals. We can convert this structured flow into training data you own and make a custom model you own. This model will adapt over time based on your natural interactions with it and based on the continued stream of data from your life.

Right now I'm trying to understand who else is trying to build this, reach out if so.

https://t.co/8INRzgEQqG

[Rafi Ayub @theayubinator]

The growing traction of Instinct just proves that people want personalized intelligence that makes their life easier more than general intelligence.

If AI can make the 20 adulting things hanging over your head all the time easier and frictionless then there would be no problem with AI adoption.