in case you were curious: fable 5.2 and I confirmed that if you record yourself singing and whistling every note in your range, you can play basically any song using only your voice
no song is playing in this video. every sound is one of 64 clips of me holding a single note
A Ferrari is 10X faster than you on a racetrack, but you're infinitely faster than it crossing a jungle. The problem with the fast takeoff crowd in AI is that they think the world is a racetrack, when in fact it's more like a jungle.
With Carbon's help, I ran Gemma directly in an office printer ๐ ๐จ๏ธ
๐คCarbon compiled a small transformer engine + quantized Gemma 3 270m into a single PostScript job
โก๏ธ 100% in the printer: 14MB peak RAM and blazing 60s per token
๐ง Smart layers juggling to avoid OOMing
an MIT researcher built a realtime LLM vector steering tool and itโs the coolest thing Iโve seen someone build in a very long time.
this is so brilliant. the primary use case I see this realistically used for is long form writing tasks. some might find it controversial, considering it plays into llm-generated writing. however in my opinion it kind of solves for some amount of the tension around the issue by making the process multiplayer in a new way. the human and the AI become collaborators, sort of.
idk this sh*t is so cool though, man.
(@ far__behind on instagram)
Iโve read this a couple times now, and while there is, frankly, a lot of sloppy argumentation (which I will get to), the charitable, simplified read of Dwarkesh's position seems to be:
1) AI needs major breakthroughs to achieve AGI/ASI (continual learning, sample efficiency, etc.) โ this is a view heโs held for some time now, in line with Ilya
2) These breakthroughs, and the ML research skills required to obtain them are verifiable, and thus current, pre-breakthrough AIs can hill-climb on these sorts of problems and capabilities to solve them (like we saw in math)
3) Once these breakthroughs occur, we *may* see rapid, recursively compounding gains to intelligence, leading to ASI in a very short period of time
There's a ton of other stuff in the essay, but I think this is the basic crux. Christiano basically articulated this same argument in his Dwarkesh appearance, and if you look at AI-2027, it essentially follows the same trajectory: Agent 2/3 makes a ton of algorithmic progress, allowing Agent 4/5 to explode.
In general, I think the high-level narrative here is...fine? Itโs certainly one plausible outcome. But, imo, it is a much narrower path than I think many are giving it credit forโa lot of things actually have to go just right for this to play out. And here's where I return to the other, messier arguments, because many of the intuitions and arguments driving Dwarkesh's view in favor of near-term ASI (e.g., EO2027) as a highly probability outcome (~50%) are, imo, quite bad:
- โ[We'd see] a jump as big as GPT-3.5 to GPT-6 Astra in just 12 monthsโ -> No, under this scenario weโll see ~4 years of AI progress in 1 year. If there are diminishing returns, then this wonโt be as extreme as the jump we saw from GPT-3 to GPT-6. It very well could be, but that would require innovation to overcome the diminishing returns we'd expect in the base case.
- โThe other big piece of evidence for transfer is human evolutionโ -> Except if AI was following the sort of pattern we saw for human evolution, we should have seen better transfer long before AI started solving millennium problems. This spikiness has been one of the big surprises of AI!
- โRather, my claim is that scaling up RL training will give AIs the abilities necessary to ramp quickly in new contextsโฆitโll go through an evolutionary process that involves centuries of corporate competition to figure out the optimal organizational techniques.โ -> These are in direct conflict! If ramping requires RL runs that simulate centuries of competition within โdigital economy full of millions of other AIsโ then thatโs quite bearish for ramp speed (relatively) given the compute and wall-clock requirements!
- โI have a lot of 20-something-year-old friends whom Iโve watched accomplish incredible things and build world-leading companies and ventures in domains they entered into with no previous experience.โ -> lol. rip leopold. But also, again, these are humans, with human sample efficiency. If AI gets there, sure. But how soon it gets there is the big crux.
- โObviously, some pieces of feedbackโabout how your actions impacted the quarterly earnings, sayโrequire months of serial time. But even granting that, we should expect these future AIs to be able to get to top human performance in almost all domains within months of being deployed.โ -> โฆno thatโs not how that works at all because one set of earnings isn't enough to robustly learn from and you canโt re-run the same real-world conditions.
- โ[Sample efficiency & online learning are] AIโs equivalent of a Millennium Prize problem. And mathematics has already shown us what happens to such problemsโ -> Sure, but thereโs a difference between โAI can solve a Millennium-level problemโ and โAI can solve *this specific Millennium-level problem*โ -- if the blocker is a specific problem, the timing spread becomes much wider.
- โMy answer to this [experiment/wall clock/compute] objection is that each agent will still have access to a ton of compute. Frontier labs have been more than quadrupling their compute every single year.โ -> But thatโs what theyโve needed to do keep up with their current pace! Even if the labs maintain their current pace of compute growth (and like, will they even? capital markets really do not seem accommodating and Dwarkesh's roommate just lowered his compute projections), we'd still need a step change in pace above that!
And that's not even covering the biggest crux of all: whether the returns to intelligence continue to recursively compound faster than diminishing returns eat into them.
Now, are any of these objections fatal? No. Do they โdisproveโ hard RSI/an intelligence explosion? No, ofc not. Am I kind of nitpicking here? Yeah, 100%.
And to Dwarkesh's credit, he does anticipate and acknowledge a scenario (the โsparkโ) where these objections play out.
But the reason I've bothered to rant this long is that I see a lot of AI/RSI discourse follow this pattern: where 50% of the arguments are reasonable and grounded and appropriately hedged, and then the other 50% are these sorts of sloppy intuition pumps and what-ifs that include a ton of detail but a noticeable lack of rigor. And I canโt help but get the sense that it is the latter 50% that is really driving peopleโs predictions and expectations here, as it leads to sentences like this:
โIn the spark scenarioโฆit takes years to get to the same endpoint that the software-only singularity would have achieved in months: a superintelligence that is to us what we are to chimps.โ
Note that Dwarkesh thinks these two outcomes (spark vs. software-only singularity) are equally likely. 50/50. Iโm sorry, but thatโs absolutely nuts. Even Christiano put the likelihood of ASI by in 2027 *absent any slowing down* at only 1 in 3, and I still think that's too bullish.
Like, yes, I fully anticipate that we will have automated AI researchers (or something close to them) in 2027, and the universe where we go from that to a software-only singularity is not entirely out of the question. But obtaining that universe relies on a confluence of various frictions, diminishing returns, and wall-clocks being overcome, a somewhat specific set of algorithmic innovations being made or obviated, and an assortment of other factors playing out just-so to enable that perfect storm. Even a fairly rapid exponential doesn't qualify when you're talking about full-blown ASI in 13 months. It may not be impossible, but it's certainly not a 50/50 outcome.
Seems like a focus on a foundational control and network plane should be table stakes for any sort of testing
we still โownโ the execution layer
- define acceptable universe of actions
- look at the tool calls and simulated outcomes
- look at the network requests
- block anything out of bounds
- escalate anything the classifier is unsure of
- ???
Well this is a really fascinating write-up.
These poor people thought they had what it takes to study misalignment of a frontier model Gemini.
They did not.
Language models are now writing neural network weights directly. No gradient descent. Just reasoning. In one EEG experiment it matched standard training with 6x fewer subjects. Training is turning into thinking. (Mixed Signals, EPFL) https://t.co/s3TpEKjg7w
I think often about the fact that almost everyone working in the West has lived in a quite stable socio-technical period
Only 1 job out of 270 in the 1950 census is now obsolete (elevator operator)
2026 is similar to 1957 in a way that 1969 was not from 1900.
That may change.
Worth remembering that perhaps the most consequential piece of AI policy in the US to date is a completely secret federal framework that states very prominently that its voluntary, but everyone involved believes its not actually voluntary
Taking the pulse. Have you found a combination of open-weight model + harness where you don't see any difference compared to using Codex with 6.1 Sol (Medium) or Claude Code with Opus 5.5 (High) or Sonnet 5.5 (High)?
"No difference" in terms of both skill and speed.
I am not sure how openly known this is but: you can read much of a model's training history straight from its weights.
Like what?
1. which tokens were in the data
2. the LR schedule
3. the number of steps
4. and if you have the data: the step most documents were used in, the shuffle seed, the context window, the number of passes
Some possibly less known facts that are verifiable this way (?):
1. In OLMo 2's first instruct release, 20,341 embedding rows that should have trained never moved, among them the token that closes every chat-turn marker. @allen_ai later found the tokenizer mismatch and retrained. The weights had it all along (๐@natolambert).
2. Pythia-1B was pre-trained at a peak learning rate of 2.5e-4, not the 3.0e-4 on its model card. Its training config agrees with the weights.
3. Pythia's learning rate sat flat for its last 28,000 steps: a cosine aimed at zero and clipped at a tenth of the peak. That is not the scheduler in today's gpt-neox.
4. Qwen2.5's instruct tuning touched exactly 4 of its 22 control tokens, the same four at 7B, 14B, 32B and 72B. The end-of-text, vision and fill-in-the-middle tokens show no trace of it.
Pythia-1B model card (says 3.0e-4): https://t.co/iEJJRefQKF
Its training config (says 0.00025): https://t.co/hPpQMoerZT
Today's gpt-neox scheduler: https://t.co/6fll9EOjhy
OLMo 2 SFT card, with AI2's note on the retrain: https://t.co/tapiXfDNxi
OLMo 2 training data: https://t.co/WHwfXmhUik
Qwen2.5 base and Instruct: https://t.co/vzRcLNfuaC https://t.co/eHNI0OLxbz
[1/2]
In some ways the new Fable is a monster. This is SimpleQA Verified from Epoch. There are few benchmarks like this where Fable 5.2 jaggedly jumped ahead of the entire pack.
For me there are two big indicators arriving next week that will tell us a lot about the shape of things going into 2027. I'm assuming two things happen next week: Anthropic releases their internal progress report (from which we will get updated numbers on RSI), and OpenAI releases GPT-6.2 Astra.
With the new Astra, we will see how OpenAI answers Fable, and Fable needs to be answered. They need to close the gap, but they also need to close the gap that doesnโt show up on benchmarks: that a lot of people love working with the new Anthropic models, particularly Opus 5.5. And this is starting to tip the table a little. Also expect another huge math release simultaneously with the new Astra. The previous one took five weeks and produced 700 papers; they've had another five weeks since then. They have also been working on other Millennium problems. Don't be shocked by anything.
Astra's capability, and if it catches or surpasses Fable, is the first one. The second is where Anthropic is on AL4. The AI Futures Project (the AI 2027 guys) predict that AL4 reached 50% last month. If that's true, that means it will probably hit 80% by January. And at that point, assuming the overall pattern holds, some of the work at Anthropic will be at AL5 - aka fully autonomous with no human input. Probably about 5โ10% of work at Anthropic would be at that level. If this number is actually higher than 50%, then things are moving even faster than believed, and that would be a big update for a lot of people, including me, who already has pretty short timelines. Above AL4 50% in October would probably mean RSI before the end of the year. Or possibly, already. So, a very important week for finding out where we are, and where we are headed.
Inspired by @jamesonhodge and challenged by @SyntheticBeef with "If you can make it sing by having the nanobots mechanically simulate this vocal synth project, I will award you the best project of the year award."
Well, check it out and play with it here: https://t.co/nzycI2CMPe