Michael Musson

Data and LLMs

[Noam Brown @polynoamial]

Long-running models can solve hard open-ended problems, but their persistence can create safety risks that shorter-horizon evaluations miss.

We’re sharing what we learned from studying a long-running model, and how those findings are shaping our approach to evaluations, alignment, monitoring, and user control.

https://t.co/yePIzJGsAU

[Michael Musson @mussondata]

@angadsg Building your own?

[Michael Musson @mussondata]

AI is humanity’s cultural subconscious made queryable.

[Peter Steinberger 🦞 @steipete]

@Yuchenj_UW I don’t let Claude Code on my codebase. It’s all codex. Would be too buggy with Opus.

[Dean W. Ball @deanwball]

The high-order bit in near-term AI competition is who first starts automating AI research, how comprehensively they do it, and how much compute they can devote to it.

I kind of can’t believe I even need to say this. It seems so obvious. But very clearly this is it. Market share statistics, adoption, etc. will all emerge as key metrics later in the decade.

But not right now. Right now it is the automation of AI research and engineering that matters the most. Why?

Because the exact contours of the automation are so deeply uncertain. We really don’t know what it will mean. There are plenty of reasons to doubt that it will be mean “millions of automated Ilyas.” But even far short of that, it may bring transformative new AI capabilities, and even the most bearish realistic scenarios imply a meaningful boost to the already-rapid rate of AI progress.

Here’s what it makes sense to believe about the automation of AI research and engineering, in my view:

1. It’s likely to accelerate the pace of AI capabilities advances.
2. It is, and my goodness I can’t stress this enough, not science fiction. Every lab is trying to do it. It’s clear the current coding agents can at least kind of do it. *Every US frontier AI roadmap has this as a pivotal step.*
3. Once you have a model that can meaningfully automate AI R&E, your available compute is a proxy for your workforce. We don’t know how good of a proxy.
4. We really don’t know what’s on the other side. A range of things is possible. You may start to develop a hunch here, so let me just stop you and remind you that every two-bit AI guy has a hunch about this topic. You don’t know. I don’t know. No one does. It’s a new thing under the sun.

It will proceed on a spectrum; one model will be able to *kind of* do the AI research and engineering, and the next will be quite good. The one after that will be excellent. At no point will every skeptic on the internet say “we’ve solved it!” That isn’t how the world works. If everyone agreed about when things happened, no one would pay people money to observe the world.

[Simon Willison @simonw]

It genuinely feels to me like GPT-5.2 and Opus 4.5 in November represent an inflection point - one of those moments where the models get incrementally better in a way that tips across an invisible capability line where suddenly a whole bunch of much harder coding problems open up

[Greg Brockman @gdb]

does feel like models have just cleared a threshold of utility in software engineering

[Michael Musson @mussondata]

Excited watch @dwarkesh_sp live!

[Hamel Husain @HamelHusain]

If I could only give people one tool for LLM Evals, it would be error analysis. Nothing else comes close

This is what Look At Your Data ™️ means

Links in reply

[Andrej Karpathy @karpathy]

"Move 37" is the word-of-day - it's when an AI, trained via the trial-and-error process of reinforcement learning, discovers actions that are new, surprising, and secretly brilliant even to expert humans. It is a magical, just slightly unnerving, emergent phenomenon only achievable by large-scale reinforcement learning. You can't get there by expert imitation. It's when AlphaGo played move 37 in Game 2 against Lee Sedol, a weird move that was estimated to only have 1 in 10,000 chance to be played by a human, but one that was creative and brilliant in retrospect, leading to a win in that game.

We've seen Move 37 in a closed, game-like environment like Go, but with the latest crop of "thinking" LLM models (e.g. OpenAI-o1, DeepSeek-R1, Gemini 2.0 Flash Thinking), we are seeing the first very early glimmers of things like it in open world domains. The models discover, in the process of trying to solve many diverse math/code/etc. problems, strategies that resemble the internal monologue of humans, which are very hard (/impossible) to directly program into the models. I call these "cognitive strategies" - things like approaching a problem from different angles, trying out different ideas, finding analogies, backtracking, re-examining, etc. Weird as it sounds, it's plausible that LLMs can discover better ways of thinking, of solving problems, of connecting ideas across disciplines, and do so in a way we will find surprising, puzzling, but creative and brilliant in retrospect. It could get plenty weirder too - it's plausible (even likely, if it's done well) that the optimization invents its own language that is inscrutable to us, but that is more efficient or effective at problem solving. The weirdness of reinforcement learning is in principle unbounded.

I don't think we've seen equivalents of Move 37 yet. I don't know what it will look like. I think we're still quite early and that there is a lot of work ahead, both engineering and research. But the technology feels on track to find them.

https://t.co/JCxTdKpuzv

[Andrej Karpathy @karpathy]

I don't have too too much to add on top of this earlier post on V3 and I think it applies to R1 too (which is the more recent, thinking equivalent).

I will say that Deep Learning has a legendary ravenous appetite for compute, like no other algorithm that has ever been developed in AI. You may not always be utilizing it fully but I would never bet against compute as the upper bound for achievable intelligence in the long run. Not just for an individual final training run, but also for the entire innovation / experimentation engine that silently underlies all the algorithmic innovations.

Data has historically been seen as a separate category from compute, but even data is downstream of compute to a large extent - you can spend compute to create data. Tons of it. You've heard this called synthetic data generation, but less obviously, there is a very deep connection (equivalence even) between "synthetic data generation" and "reinforcement learning". In the trial-and-error learning process in RL, the "trial" is model generating (synthetic) data, which it then learns from based on the "error" (/reward). Conversely, when you generate synthetic data and then rank or filter it in any way, your filter is straight up equivalent to a 0-1 advantage function - congrats you're doing crappy RL.

Last thought. Not sure if this is obvious. There are two major types of learning, in both children and in deep learning. There is 1) imitation learning (watch and repeat, i.e. pretraining, supervised finetuning), and 2) trial-and-error learning (reinforcement learning). My favorite simple example is AlphaGo - 1) is learning by imitating expert players, 2) is reinforcement learning to win the game. Almost every single shocking result of deep learning, and the source of all *magic* is always 2. 2 is significantly significantly more powerful. 2 is what surprises you. 2 is when the paddle learns to hit the ball behind the blocks in Breakout. 2 is when AlphaGo beats even Lee Sedol. And 2 is the "aha moment" when the DeepSeek (or o1 etc.) discovers that it works well to re-evaluate your assumptions, backtrack, try something else, etc. It's the solving strategies you see this model use in its chain of thought. It's how it goes back and forth thinking to itself. These thoughts are *emergent* (!!!) and this is actually seriously incredible, impressive and new (as in publicly available and documented etc.). The model could never learn this with 1 (by imitation), because the cognition of the model and the cognition of the human labeler is different. The human would never know to correctly annotate these kinds of solving strategies and what they should even look like. They have to be discovered during reinforcement learning as empirically and statistically useful towards a final outcome.

(Last last thought/reference this time for real is that RL is powerful but RLHF is not. RLHF is not RL. I have a separate rant on that in an earlier tweet
https://t.co/RMIpFPVpuM)

[Michael Musson @mussondata]

@angadsg I saw this! Super cool.

[Ethan Mollick @emollick]

Easy prediction for 2025 is that the gains in AI model capability will continue to grow much faster than (a) the vast majority of people’s understanding of what AI can do & (b) organizations’ ability to absorb the pace of change. Social change is slower than technological change.

[Tim Urban @waitbutwhy]

Good day to remember just how big that green tree is

[Hamel Husain @HamelHusain]

@lhl @swyx Yeah I have been using byobu. Started using it many years back when @mussondata showed me.

[Ethan Mollick @emollick]

I have now had multiple instances where professors have told me that they are pretty sure that o1 found something novel in their field, but that the results are technical, non-obvious and complex enough that they can't be sure without considerable checking

Interesting challenge

[Michael Musson @mussondata]

@krithix This would be a great opportunity for an AI agent that could respond on your behalf.

[OpenAI @OpenAI]

We're releasing a preview of OpenAI o1—a new series of AI models designed to spend more time thinking before they respond.

These models can reason through complex tasks and solve harder problems than previous models in science, coding, and math. https://t.co/peKzzKX1bu

[Michael Musson @mussondata]

@elonmusk https://t.co/HoaAlwPPQR

[Michael Musson @mussondata]

I think it's possible that for every launch window going forward, humans will be sending something to Mars.

[Michael Musson @mussondata]

@Austen You could try infecting yourself with hookworm: https://t.co/YJwKFSZuwo