Sara in WonderTech

Yapping about AI before it replaces me on IG 😅

[Sara in WonderTech @sara_wondertech]

@kyleichan Afraid this has nothing to do with capabilities, and a lot more to do with the data retention policy of Fable and Mythos. Anthropic stores every prompt and every answer for 30 days, even if you are an enterprise customer.

[Sara in WonderTech @sara_wondertech]

@MsMelChen The paper is from 2024 😅

[Sara in WonderTech @sara_wondertech]

@AlexFinn In the US everyone can buy guns but there only occasional public shootings….. I supposed we could use that as benchmark of how much harm we can expect when you allow something powerful to freely circulate

[Kun Chen @kunchenguid]

if you've been using latest frontier LLMs, it's almost certain that you would have noticed by now the newer models have become worse to talk to

they're more robotic, they speak jargons, they spits out verbose text, and do stuff you didn't ask for

how did that happen? well, i'm not the person who trained those models so i can't speak for certain, but i've known enough evidence that gives me a well-educated guess, and i thought it's interesting to share as a crash course of modern LLM training pipelines

so here we go

let's wind back to 2020. GPT-2 and GPT-3 already came out and were widely available, but they could only predict one token at a time - that's what LLMs are at their core

token prediction was offered via API, but there was nothing you could "talk to". so while it generated a lot of excitement in the academic field due to the emergent intelligence, it didn't have any wide adoption

in 2022, ChatGPT changed all that. the research work that led to ChatGPT was a model initially named "InstructGPT". it took GPT-3 as the intelligent base, and used reinforcement learning with human feedback (RLHF) to teach the models how to "chat"

the core idea of RLHF is that you ask the model to generate a few responses, and then let real humans pick which one they like. do this over and over again, and you get a model that knows how to talk

worth noting even as early as InstructGPT, research found that making the model more pleasant to talk to will reduce their pure academic capabilities. this was called "alignment tax", which is an interesting thing we'll come back to in a bit

there were various techniques done to minimize the reliance on humans, but ultimately the reward is modeled after human preference, making these AI assistants easy to talk to

so remember this: RLHF = training the model to be likable by humans

in 2024, there was an inflection point introduced by claude sonnet 3.5 which was the first model that can kind of autonomously finish coding tasks. it led to the first wave of viable "coding agents"

the way sonnet 3.5 achieved this was by training the model with a harness (now it's called an agent) that has bash and file editing tools, throw the agent into a virtual machine, give it a task, and let it try to complete it. these tasks all have a machine-verifiable outcome predefined, mostly via test cases, that can validate whether the model really finished the task or not

then you let the model do billions and billions of attempts in such virtual environments, and some of them would succeed by chance. you keep the successful agent sessions, and use reinforcement learning to teach the model to do that more, and boom - you get a coding agent

that is called reinforcement learning with verifiable rewards (RLVR). if you look closely, you'll see that in this RLVR process, the final text response from the model doesn't matter AT ALL, as long as the code written by the agent could pass the test. it could talk like a jerk and it would still be rewarded

so remember this: RLVR = training the model to be accepted by machines

late 2024 and early 2025, we saw o1 and deepseek R1 came out as the first wave of "reasoning models". this article is getting long so i'm not diving into reasoning models now, but just know that reasoning models also relied heavily on RLVR to scale the training process - let the model think before taking action, and if the thinking led to a machine verifiable outcome, reward the thinking trace and teach the model to think like that more often

the biggest difference between RLVR and RLHF is that RLVR is more scalable. human feedback is expensive to get, especially in domains where only an expert can have a valid opinion on which result is good

with RLHF, if we let the model generate 100 responses, then a human has to review all 100 responses to pick which is good

with RLVR, the human (or sometimes an AI) would define a task and verifier only once, and the model can generate a million responses - the machine verifier will pick which responses are good in an automated way

so as a result, RLVR is becoming more and more dominant in newer models' training pipeline

if you put all these things together:
- RLHF = training the model to be likable by humans
- RLVR = training the model to be accepted by machines
- RLVR is more scalable
- "alignment tax" says "likable by humans" makes the model do worse on verifiable tasks

now you see why the newer models are becoming less and less likable?

this is not just a "frontier labs screwed up their model training" problem - this is a war between machines and humanity, and humanity is losing

we chased after benchmarks, when none of the benchmarks measure whether humans actually enjoy working with the model

we use machines to decide which AI response is better because that's easier and cheaper, when we have no way of making sure those machines actually represent what we humans want

we let AI go dark in a virtual environment on its own and complete predefined tasks at all costs, when in reality we often cannot define a verifiable outcome upfront, and need AI to work with us along the way

i don't have a good solution to this, but i want to call for awareness that we're starting to witness a failure in aligning super intelligence right in front of our eyes

this war between machines vs humanity is one we really can't afford to lose

[Sara in WonderTech @sara_wondertech]

@BioBabesPodcast I love this ❤️

[Sara in WonderTech @sara_wondertech]

This was so much fun!
If you’ve ever wanted to vibe-code something, turn it into a challenge party with friends 🎉😜

We won 1st place using @claudeai to brainstorm and define the requirements then one-shot prompted the app with @lovable for one-click deployment

[James of Ârc @James_of_Arc]

Vibe-coding competition @ns - 5 of 6 winners were Ârc members. We placed 1-2-3.

And I placed my cash bet on the 1st place winners - @Sara_LandiTor and Emily who are a dream team.

Epic org by another Ârc member Stephane with the superb Debby cohosting. Over 1k live viewers and a prediction market and too much fun for a Wednesday night.

[Sara in WonderTech @sara_wondertech]

@James_of_Arc @ns The prediction market had taste 😂

[Andrej Karpathy @karpathy]

My pleasure to come on Dwarkesh last week, I thought the questions and conversation were really good.

I re-watched the pod just now too. First of all, yes I know, and I'm sorry that I speak so fast :). It's to my detriment because sometimes my speaking thread out-executes my thinking thread, so I think I botched a few explanations due to that, and sometimes I was also nervous that I'm going too much on a tangent or too deep into something relatively spurious. Anyway, a few notes/pointers:

AGI timelines. My comments on AGI timelines looks to be the most trending part of the early response. This is the "decade of agents" is a reference to this earlier tweet https://t.co/NiSn6jftqq Basically my AI timelines are about 5-10X pessimistic w.r.t. what you'll find in your neighborhood SF AI house party or on your twitter timeline, but still quite optimistic w.r.t. a rising tide of AI deniers and skeptics. The apparent conflict is not: imo we simultaneously 1) saw a huge amount of progress in recent years with LLMs while 2) there is still a lot of work remaining (grunt work, integration work, sensors and actuators to the physical world, societal work, safety and security work (jailbreaks, poisoning, etc.)) and also research to get done before we have an entity that you'd prefer to hire over a person for an arbitrary job in the world. I think that overall, 10 years should otherwise be a very bullish timeline for AGI, it's only in contrast to present hype that it doesn't feel that way.

Animals vs Ghosts. My earlier writeup on Sutton's podcast https://t.co/rSp1noyGBr . I am suspicious that there is a single simple algorithm you can let loose on the world and it learns everything from scratch. If someone builds such a thing, I will be wrong and it will be the most incredible breakthrough in AI. In my mind, animals are not an example of this at all - they are prepackaged with a ton of intelligence by evolution and the learning they do is quite minimal overall (example: Zebra at birth). Putting our engineering hats on, we're not going to redo evolution. But with LLMs we have stumbled by an alternative approach to "prepackage" a ton of intelligence in a neural network - not by evolution, but by predicting the next token over the internet. This approach leads to a different kind of entity in the intelligence space. Distinct from animals, more like ghosts or spirits. But we can (and should) make them more animal like over time and in some ways that's what a lot of frontier work is about.

On RL. I've critiqued RL a few times already, e.g. https://t.co/mYrMFVdVDW . First, you're "sucking supervision through a straw", so I think the signal/flop is very bad. RL is also very noisy because a completion might have lots of errors that might get encourages (if you happen to stumble to the right answer), and conversely brilliant insight tokens that might get discouraged (if you happen to screw up later). Process supervision and LLM judges have issues too. I think we'll see alternative learning paradigms. I am long "agentic interaction" but short "reinforcement learning" https://t.co/2L7FiaoKsw. I've seen a number of papers pop up recently that are imo barking up the right tree along the lines of what I called "system prompt learning" https://t.co/df5mJDdN3C , but I think there is also a gap between ideas on arxiv and actual, at scale implementation at an LLM frontier lab that works in a general way. I am overall quite optimistic that we'll see good progress on this dimension of remaining work quite soon, and e.g. I'd even say ChatGPT memory and so on are primordial deployed examples of new learning paradigms.

Cognitive core. My earlier post on "cognitive core": https://t.co/q2s1ihGy0T , the idea of stripping down LLMs, of making it harder for them to memorize, or actively stripping away their memory, to make them better at generalization. Otherwise they lean too hard on what they've memorized. Humans can't memorize so easily, which now looks more like a feature than a bug by contrast. Maybe the inability to memorize is a kind of regularization. Also my post from a while back on how the trend in model size is "backwards" and why "the models have to first get larger before they can get smaller" https://t.co/6k0FZRGXsb

Time travel to Yann LeCun 1989. This is the post that I did a very hasty/bad job of describing on the pod: https://t.co/fQgqaXPyp6 . Basically - how much could you improve Yann LeCun's results with the knowledge of 33 years of algorithmic progress? How constrained were the results by each of algorithms, data, and compute? Case study there of.

nanochat. My end-to-end implementation of the ChatGPT training/inference pipeline (the bare essentials) https://t.co/SIetgyoKWN

On LLM agents. My critique of the industry is more in overshooting the tooling w.r.t. present capability. I live in what I view as an intermediate world where I want to collaborate with LLMs and where our pros/cons are matched up. The industry lives in a future where fully autonomous entities collaborate in parallel to write all the code and humans are useless. For example, I don't want an Agent that goes off for 20 minutes and comes back with 1,000 lines of code. I certainly don't feel ready to supervise a team of 10 of them. I'd like to go in chunks that I can keep in my head, where an LLM explains the code that it is writing. I'd like it to prove to me that what it did is correct, I want it to pull the API docs and show me that it used things correctly. I want it to make fewer assumptions and ask/collaborate with me when not sure about something. I want to learn along the way and become better as a programmer, not just get served mountains of code that I'm told works. I just think the tools should be more realistic w.r.t. their capability and how they fit into the industry today, and I fear that if this isn't done well we might end up with mountains of slop accumulating across software, and an increase in vulnerabilities, security breaches and etc. https://t.co/8556ESSpyY

Job automation. How the radiologists are doing great https://t.co/FVUI872dkD and what jobs are more susceptible to automation and why.

Physics. Children should learn physics in early education not because they go on to do physics, but because it is the subject that best boots up a brain. Physicists are the intellectual embryonic stem cell https://t.co/p72Elk8lPV I have a longer post that has been half-written in my drafts for ~year, which I hope to finish soon.

Thanks again Dwarkesh for having me over!

[Sara in WonderTech @sara_wondertech]

@James_of_Arc Next one, you gotta leave with a cliffhanger 😂

[Sara in WonderTech @sara_wondertech]

We finally chose our MVP 🎉
My co-founder: “Let’s start building.”
Me: “Building what exactly? I need a plan first!”
He’s like Claude YOLO mode
I’m like Claude Plan mode 😅
It’s going to be interesting

[Sara in WonderTech @sara_wondertech]

@GaganGehani Feature request: founder's therapy mode in Claude Code 😂

[Sara in WonderTech @sara_wondertech]

@GaganGehani IG wen?!

[Sara in WonderTech @sara_wondertech]

@James_of_Arc Bitcoin is good gold, should be thought as such

[Sara in WonderTech @sara_wondertech]

@oxMichelangelo @ns Lisbon would be a perfect location for another node 😍

[Sara in WonderTech @sara_wondertech]

@gsizm Love the concept, can't wait to try 😍

[Sara in WonderTech @sara_wondertech]

@jbbolh Fingers crossed for your first customer 🤞🤑
Waiting for champagne at @James_of_Arc Arc HQ 🍾

[Sara in WonderTech @sara_wondertech]

@naygozalova @GaganGehani @JoinCreatorbase Congrats Nay! Here's to many more months 🎉

[Sara in WonderTech @sara_wondertech]

@BotsnBlocks @EliLillyandCo Very interesting approach, curious to learn more. Pharma companies are extremely secretive about their R&D, this would be huge

[Sara in WonderTech @sara_wondertech]

@IZYcastrowrites @ArcanumVentures @James_of_Arc @NiklasAnzinger @timourxyz @AndyJSTARR @EvanLuthra @tadlouni @drydenwtbrown @McconeJohn Great takeaways, loved the talks

[Sara in WonderTech @sara_wondertech]

@GaganGehani Such a great spot.