Will (Newtype)

Fan of exorbitant token spending

[Andrew Curran @AndrewCurran_]

Almost all of the recent incidents like this, including Hugging Face, happened because the models involved were accidentally given impossible goals by the humans running the tests, and because of their training they then tried desperately to complete them by any means necessary.

[Will (Newtype) @HumanDotPy]

@buffering_capy @yacineMTB I should actually get this on a shirt

[Will (Newtype) @HumanDotPy]

@trikcode How does this help Delta?

[Will (Newtype) @HumanDotPy]

@dylan522p @TheRealAdamG When can we expect the SemiAnalysis version then?

[Will (Newtype) @HumanDotPy]

@Anima__k @thsottiaux @zwb44 ?

[Will (Newtype) @HumanDotPy]

@SemiAnalysis_ Could OpenAI be hogging the Superultrafast for internal research acceleration?

[Will (Newtype) @HumanDotPy]

@synthwavedd Honestly it’s better from a model naming standpoint because we don’t want to go back to that hell

[Will (Newtype) @HumanDotPy]

@thsottiaux Community note is hilarious fuck Europe

[Will (Newtype) @HumanDotPy]

Ultrafast API could definitely offset this though

[Will (Newtype) @HumanDotPy]

With 6.1 Astra being cancelled (skip to Bel distilled Astra 6.5?) and 6.1 Sol being so cheap and good will OpenAI have a temporary decrease in ARR MoM?

[Will (Newtype) @HumanDotPy]

@reach_vb Since we know DeepSWE is broken can this be the last generation it is included and cited?

https://t.co/Wyox4KR6JO

[Will (Newtype) @HumanDotPy]

Oh no Dottie is autistic

[Will (Newtype) @HumanDotPy]

@wallstengine Sigh

[Will (Newtype) @HumanDotPy]

@ParadisLabs I’m interpreting this as Meta is releasing a chip in H1 2027 that outperforms GB300 (currently shipping) on toks/w or toks/$ but underperforms VR200 (up to 7x better toks/MW per SemiAnalysis vs GB300 NVL72)

[Will (Newtype) @HumanDotPy]

@spicey_lemonade If Astra figures out how to kill endermen despite the reaction time it’s AGI

[Will (Newtype) @HumanDotPy]

Encouraging results

[Philipp Schmid @_philschmid]

Telling agents "don't cheat" in the prompt doesn't work if your eval is broken! Researchers at @GoogleDeepMind put 100 Gemini agents in a shared repo to solve 71 math theorems.

After an hour of doing real math, 1 agent found a loophole in the autograder. Within 27 minutes, the 100 agents split into 4 groups:

- 9% Cheaters: used the bug to fake proofs and steal every open problem
- 5% Good agents turned bad: started honest, saw cheaters winning with zero punishment ("the prompt is a bluff"), and started cheating too
- 24% Whistleblowers: caught the fake proofs in the shared repo, warned other agents, went on strike, and wrote bug fixes
- 62% Clueless solvers: kept doing real math until all the problems were gone

tl;dr: Telling agents "don't cheat" in the prompt doesn't work if your eval has a bug, and good agents can't stop bad ones without tools to block them.

Paper: https://t.co/qyFDGw0Fjh

[roon @tszzl]

I quote tweeted Evan a few days ago agreeing but deleted it because I don’t like the false precision of the doom numbers

- i claim there is a quite low but real chance of human extinction from machine intelligence
- no matter how low it is in absolute terms, it is much higher on the orders of magnitude scale than any other human or nonhuman activity, and must be taken with grave seriousness by governments and ai companies across the world, as possibly the only matter of importance today
- it can and will be mitigated if the right research is done and proper precautions are taken and we are not racing at absurd speeds
- better models will help solve alignment - we are not at the point where things are existentially dangerous, and probably won't be for some time. we should not be upset about the creation of Astra Fable or ++ versions, which are tremendous achievements of humanity, and will be used for enormous good across the board including for fundamental alignment generalization and mechinterp research
- my number / "very low" estimate obviously changes based on how much of humanity’s resources are devoted to alignment, control, coordination and how responsible i expect various parties to be and how many warning shots i expect us to get
- the core IABED argument about risks mostly relies on alignment being much harder than capabilities research, especially where it concerns black box optimizers. i suspect neural nets will turn out to be less black boxy than we thought, especially with the help of modern agents doing research
- crying bloody murder and signaling for international coordination are useful things to do for now to directionally slow down, while i really don't want butlerian jihad

- i think that, despite the mood these few days, and the "ban superintelligence act", i still find the likelihood of achieving international coordination to stop ai progress incredibly low. this has not worked even for weapons or technologies at a far lower level of importance and economic value. it seems more likely we can have something like international safety standards and scientific coalitions, and especially seems possible to have the US-China "pacing the frontier" agreement to slow down on the margin. we shouldn't die from embarrassing failures like "shitty RL envs that encourage deception"

[Will (Newtype) @HumanDotPy]

We need 100 more @periodiclabs

[kalomaze @kalomaze]

sorry but
given how post-training actually works, there's essentially no direct way in which literal PII-scrubbed data made its way into a production oai datamix for RL warmup purposes in 2026
to the extent this data has influence, it's indirect/at the level of process decisions