Aditya Jain

Trust and Safety @ Instagram
Writing my thoughts on AI Safety -

[Aditya Jain @adityajain93]

AI Safety is getting a makeover - from being only a post training artifact - new research by folks at @CarnegieMellon @cais @GraySwanAI show how safety infused at pretraining gives natively safer model!

Highlights below!
https://t.co/bzwOGBVRos

[Aditya Jain @adityajain93]

@ChipotleTweets @Strava pls sponsor this https://t.co/CltAKk2zkc

[Aditya Jain @adityajain93]

@ChipotleTweets @Strava we out there 🫶
Checkout some stories of who’s doing the running

https://t.co/CltAKk2zkc

[Matthew Berman @MatthewBerman]

Anthropic just dropped an incredible guide on "How To Build Effective Agents"

2025 will be the year of AGENTS 🤖

Here's everything you need to know: 🧵

[Andrej Karpathy @karpathy]

# RLHF is just barely RL

Reinforcement Learning from Human Feedback (RLHF) is the third (and last) major stage of training an LLM, after pretraining and supervised finetuning (SFT). My rant on RLHF is that it is just barely RL, in a way that I think is not too widely appreciated. RL is powerful. RLHF is not. Let's take a look at the example of AlphaGo. AlphaGo was trained with actual RL. The computer played games of Go and trained on rollouts that maximized the reward function (winning the game), eventually surpassing the best human players at Go. AlphaGo was not trained with RLHF. If it were, it would not have worked nearly as well.

What would it look like to train AlphaGo with RLHF? Well first, you'd give human labelers two board states from Go, and ask them which one they like better:

Then you'd collect say 100,000 comparisons like this, and you'd train a "Reward Model" (RM) neural network to imitate this human "vibe check" of the board state. You'd train it to agree with the human judgement on average. Once we have a Reward Model vibe check, you run RL with respect to it, learning to play the moves that lead to good vibes. Clearly, this would not have led anywhere too interesting in Go. There are two fundamental, separate reasons for this:

1. The vibes could be misleading - this is not the actual reward (winning the game). This is a crappy proxy objective. But much worse,
2. You'd find that your RL optimization goes off rails as it quickly discovers board states that are adversarial examples to the Reward Model. Remember the RM is a massive neural net with billions of parameters imitating the vibe. There are board states are "out of distribution" to its training data, which are not actually good states, yet by chance they get a very high reward from the RM.

For the exact same reasons, sometimes I'm a bit surprised RLHF works for LLMs at all. The RM we train for LLMs is just a vibe check in the exact same way. It gives high scores to the kinds of assistant responses that human raters statistically seem to like. It's not the "actual" objective of correctly solving problems, it's a proxy objective of what looks good to humans. Second, you can't even run RLHF for too long because your model quickly learns to respond in ways that game the reward model. These predictions can look really weird, e.g. you'll see that your LLM Assistant starts to respond with something non-sensical like "The the the the the the" to many prompts. Which looks ridiculous to you but then you look at the RM vibe check and see that for some reason the RM thinks these look excellent. Your LLM found an adversarial example. It's out of domain w.r.t. the RM's training data, in an undefined territory. Yes you can mitigate this by repeatedly adding these specific examples into the training set, but you'll find other adversarial examples next time around. For this reason, you can't even run RLHF for too many steps of optimization. You do a few hundred/thousand steps and then you have to call it because your optimization will start to game the RM. This is not RL like AlphaGo was.

And yet, RLHF is a net helpful step of building an LLM Assistant. I think there's a few subtle reasons but my favorite one to point to is that through it, the LLM Assistant benefits from the generator-discriminator gap. That is, for many problem types, it is a significantly easier task for a human labeler to select the best of few candidate answers, instead of writing the ideal answer from scratch. A good example is a prompt like "Generate a poem about paperclips" or something like that. An average human labeler will struggle to write a good poem from scratch as an SFT example, but they could select a good looking poem given a few candidates. So RLHF is a kind of way to benefit from this gap of "easiness" of human supervision. There's a few other reasons, e.g. RLHF is also helpful in mitigating hallucinations because if the RM is a strong enough model to catch the LLM making stuff up during training, it can learn to penalize this with a low reward, teaching the model an aversion to risking factual knowledge when it's not sure. But a satisfying treatment of hallucinations and their mitigations is a whole different post so I digress. All to say that RLHF *is* net useful, but it's not RL.

No production-grade *actual* RL on an LLM has so far been convincingly achieved and demonstrated in an open domain, at scale. And intuitively, this is because getting actual rewards (i.e. the equivalent of win the game) is really difficult in the open-ended problem solving tasks. It's all fun and games in a closed, game-like environment like Go where the dynamics are constrained and the reward function is cheap to evaluate and impossible to game. But how do you give an objective reward for summarizing an article? Or answering a slightly ambiguous question about some pip install issue? Or telling a joke? Or re-writing some Java code to Python? Going towards this is not in principle impossible but it's also not trivial and it requires some creative thinking. But whoever convincingly cracks this problem will be able to run actual RL. The kind of RL that led to AlphaGo beating humans in Go. Except this LLM would have a real shot of beating humans in open-domain problem solving.

[Aditya Jain @adityajain93]

@karpathy i feel that lack of a direct reward function illustrates the dichotomies and multi objectivity which exist with human centered tasks. It's a ceiling we'll never reach but approximate using better proxy functions.

Maybe the question is what is a better proxy reward model?

[Aditya Jain @adityajain93]

lastly, a neat workflow to follow along

[Aditya Jain @adityajain93]

Adapting Large Language Models - Blog Series

sharing some best practices we've seen on adapting llms for downstream use cases within @AIatMeta @Meta and across the literature/externally

Great starting point for small teams (1-3 MLEs)
description, links - below👇

[Aditya Jain @adityajain93]

Part 3 discusses the what we think will stand the test of time - quality of data. Compile general guidance on best practices we've seen

"How to fine-tune: Focus on effective datasets"

https://t.co/1op44ujNiT

[Hamel Husain @HamelHusain]

If you remember our Applied LLMs course, you'll love this. Today, we are making all these resources available for free to everyone! 📚

We did extra work to add learning tracks, resources, and notes to each lesson to maximize your learning. Link in next tweet

[Andrej Karpathy @karpathy]

LLM model size competition is intensifying… backwards!

My bet is that we'll see models that "think" very well and reliably that are very very small. There is most likely a setting even of GPT-2 parameters for which most people will consider GPT-2 "smart". The reason current models are so large is because we're still being very wasteful during training - we're asking them to memorize the internet and, remarkably, they do and can e.g. recite SHA hashes of common numbers, or recall really esoteric facts. (Actually LLMs are really good at memorization, qualitatively a lot better than humans, sometimes needing just a single update to remember a lot of detail for a long time). But imagine if you were going to be tested, closed book, on reciting arbitrary passages of the internet given the first few words. This is the standard (pre)training objective for models today. The reason doing better is hard is because demonstrations of thinking are "entangled" with knowledge, in the training data.

Therefore, the models have to first get larger before they can get smaller, because we need their (automated) help to refactor and mold the training data into ideal, synthetic formats.

It's a staircase of improvement - of one model helping to generate the training data for next, until we're left with "perfect training set". When you train GPT-2 on it, it will be a really strong / smart model by today's standards. Maybe the MMLU will be a bit lower because it won't remember all of its chemistry perfectly. Maybe it needs to look something up once in a while to make sure.

[Jacob Hilton @JacobHHilton]

When I left @OpenAI a little over a year ago, I signed a non-disparagement agreement, with non-disclosure about the agreement itself, for no other reason than to avoid losing my vested equity. (Thread)

[Nakul Mandan @nakul]

Good post here by @sequoia - key message is that the app layer in AI needs to catch up:

"For startups, the takeaway is clear: As a community, we need to shift our thinking away from infrastructure and towards end-customer value. Happy customers are a fundamental requirement of every great business. For AI to have an impact, we need to figure out how to leverage this new technology to make people’s lives better. How can we translate these amazing innovations into products that customers use every day, love and are willing to pay for?

The AI infrastructure build out is happening. Infrastructure is not the problem anymore. Many foundation models are being developed—this is not the problem anymore, either. And the tooling in AI is pretty good today. So the $200B question is: What are you going to use all this infrastructure to do? How is it going to change people’s lives?"

[Jared Friedman @snowmaker]

(5/5) A lot of learnings here for AI startups. Many startups are working on an AI system that effectively replaces a human doing some important job.

The best way to get these systems live is to find a way to start with a near zero-risk setting.

[GeniusThinking @GeniusGTX]

Mental models are rules of thumb that simplify decisions.

12 most powerful mental models I've found:

[Aditya Jain @adityajain93]

@bindureddy yesss!

[San Francisco Chronicle @sfchronicle]

The Bay to Breakers fun continues with a dance party at the top of the Hayes Hill Challenge.

📷: Ben Fanjoy

[Michelle Fang 🌁 @michelleefang]

Let me plug you into the SF tech scene!

A list of what's happening this week (May 11 - May 18) ⬇️