OpenAI said in early Sep they'd release research showing looped transformers weren't a significant driver of Astra becoming less monitorable.
They haven't published anything yet AFAIK.
And this study suggests looped transformers do have a big effect (as expected).
@MJBiercuk Are you worried about the founders or the investors? When you made the decision to start your company, did the choice really hinge on a cell in a spreadsheet estimating your tax bill for a hypothetical future 1B exit?
Reference classes are ill-defined.
Your thinking about Fermi must have lead you to consider there's many cases in history of a new optimization process arising that runs on a different substrate and adapts so fast as to supplant the old optimizer in the "blink of an eye" in relative terms.
Took more than 2/3rds of all history for natural selection to arise on earth, then each time there's new optimization process the pace of change accelerates dramatically
E.g. sex -> brains -> culture -> industry
(Not saying this is good argument for AI FOOM, just illustrating how picking and choosing reference classes let us argue in either direction)
If we don't see any more baby rouge AI incidents next year that would be supprising to me. if we don't have self sustaining AIs existing well before any real takeover threat this would also be evidence against my view
Did maths go superexponential?
AI can do math research that would take a human expert over 10,000h.
But in April, the 80% METR time horizon was only 3h.
What happened?
I think it can be explained with 3 factors (& doesn't require a superexponential)
1) Math was ahead: July 2025 it could already do ~2h IMO problems vs. METR was only 30min.
2) Math has growing 1.5x to 2x faster than METR (e.g. 40x per year vs. 8x in 2025)
3) Progress accelerated another ~2x in 2026.
This gets you to ~10,000h tasks around now.
Switching to multiagent systems also got them another 10x boost.
Whether we call the speed ups superexponential is a bit academic. Either way, it shows how insane the current trends already are.
@fqqq4u@John_Hempton@JosephNWalker@TheStalwart@tracyalloway Fwiw if you bake in the discount you get by having an anthropic subscription and account for how better models can get the same quality output in fewer reasoning tokens, I think fable can actually be cheaper than GLM 5.2 depending on your provider
"Not everyone needs the smartest AI" is wrong.
Finance podcasts keep asking whether most people will settle for "good enough" AI. They won't.
My reply to @John_Hempton on @JosephNWalker's podcast, and to @TheStalwart and @tracyalloway on Odd Lots.
And while you're correct that I have no need for the models I use to be good at biochemistry, it turns out that as a side effect of how the technology actually works, the model that's the best at biochemistry is also the model that's the best at nicely laying out a chart on a slide.
Even for people who don't do math research, it turns out the models that have been trained on outputting correct math proofs are better at doing their online shopping for them
I agree nothing is infinite (of course!) - I don't claim people will only use the best possible models for everything no matter the price - if fable was $1 per token I wouldn't use it.
But when it's 0.004 cents per output token it's honestly rare to encounter cases where the savings of substuting to a cheaper model justify the effort of the mental acounting.
I don't think the per-token price of the frontier will end up many orders of magnitude higher than it is today
I think your diminishing returns argument, when applied bluntly without an acompanying attempt to actually quantify the cost/benefit faced by the buyer, could have been generally applied many times in the past to rule out most of what people routinely consume today:
Hundreds of years ago who would have dreamt that people would want to bathe daily and wash their clothes after a single use? Who would ever need more than one pair of shoes?
Why could anyone bother to get their family photo taken (or portrait painted) more than couple times? Who needs multiple bathrooms in their house? Who really needs a separate bedroom for each of their children? Why would you need more than one powerpoint?
Who needs more than 1 variety of bread or wine on the store shelves? Why would anyone go through the logistical hell of stocking pineapples and mangos in a Canberra supermarket?
What kind of normal consumer needs to send more than a few letters per week? Who would want more than 10 books in their house?
What possible use could anyone have for gigabit internet speeds? Why would anyone need gigabytes of ram and a multi core CPU with a clock cycle in the gigahertz?
Healthcare exhibits diminishing returns too - whatever new drugs come to market soon are unlikely impact human healt more than clean water/soap/the smallpox vaccine. And yet, for good reason, you're still bullish on people using AI and investing a lot of resources generally in order to develop better treatments.
This is similar to when Keynes predicted his grandkids would only work 15 hour weeks. GDP per capita did go up, but demand rose to the occasion.
The vacuum robot is the best example so far given it's input/action space is genuinely small.
There's still a long way to go until vacuum robots never get stuck, never get tangled in the tressles of a rug, never miss a spot, delay turning on when you're having a sleep in, notify you in an effective way when they pass over an unexpectedly wet floor and notice that rain must be comming through an open wiindow, cleverly avoid rooms people are currently occupying, keep themselves safe from curious todlers etc.
Given the value of your time, a robovac which demands your intervention at even a slightly lower rate would still be worth paying frontier-AI prices to run
But yeah, maybe we really can saturate the task of robot vaccum control with models not that much better than today's fable.
At that point, a superior form factor for a domestic cleaning robot probably starts becoming viable. You won't be satisfied with your current robovac 5 years from now
@John_Hempton@JosephNWalker@TheStalwart@tracyalloway But like, the point I'm trying to gesture at is if you're optimize the value of a function by searching over an unfathomably high dimensional input space, we should expect the global maximum to be an extreme value, and we should expect that value to be very hard to find
Easier to use: - definitely (we should wrap that into definition of "good")
Cheaper: - orgs will definitely care about cost control (this is actually what I'm working on at my job) and non-frontier models will have lots of use cases, but not in a way that canibalises demand for the frontier. That's only going up