Psyho

Humanity's Last Programmer; Game Designer; Problem Solver; past: OpenAI (Dota), Pro Competitive Programmer, Poker

[Psyho @FakePsyho]

Humanity has prevailed (for now!)

I'm completely exhausted. I figured, I had 10h of sleep in the last 3 days and I'm barely alive.

I'll post more about the contest when I get some rest.

(To be clear, those are provisional results, but my lead should be big enough)

[Psyho @FakePsyho]

worth noting that
1.9x faster improvement in AI capabilities
is not
1.9x faster improvement in AI capabilities of the best publicly available model

[METR @METR_Evals]

1. Benchmarks provide one piece of evidence for acceleration. Since Oct 2025, AECI scores for Anthropic’s models have grown at 1.9x [95%: 1.5–2.8] the earlier rate. Analysis choices affect this estimate, and it remains unclear how benchmark scores map to real-world capabilities.

[Psyho @FakePsyho]

to be fair, the main offender here is twitter's "algorithm"

there's currently way too much incentive to QT anything that's going viral; you've interacted with a single viral tweet? enjoy those 200 quote tweets flooding your feed for the next 24 hours

[Psyho @FakePsyho]

I'm honestly surprised to see so much hate directed at the authors of the Weight-Space Programming blogpost for overstating the results in their headline, yet so little directed at the mindless AI influencers who blindly amplified it.

Sadly, it's the same game. As long as people happily spread any bs that might go viral, researchers will have a very strong incentive to oversell their work.

[Psyho @FakePsyho]

> i agree that the reception of this blog has been more inflated than we expected

I mean, I don't blame you. If anything, I blame people who blindly share it without reading anything past the headline. And choosing headlines sucks because it's basically a lose-lose situation:
use very safe headline/abstract -> no one cares
use very bold headline/abstract -> hopefully people share it enthusiastically (which is only an issue when it goes viral)

I liked the experiments!

[Psyho @FakePsyho]

meh, the more you dig into each experiment, the less magical it gets; it feels like a loose collection of cherry-picked demonstrations with a grand narrative built around them

Exp 1-2 are tiny (~20K params) with unusually easy to interpret architecture; Exp 1 is an MLP, where nearly all params are used in the first layer, so you can easily tell what kind of objects the network is trying identify
Exp 5 is about designing a game strategy (we already know that Astra is extremely good at this) and then converting it to a tiny (~20K params) MLP; entirely doable by a human, but of course would require a lot of effort
Exp 3 & 8 are closer to automated mechinterp using python and established techniques, rather than direct weight interpretation (at least according to chat)

to be clear, those are genuinely cool results, but I think people are reading too much into them

I don't think any of this should meaningfully change anyone's intuition about what models are capable of

[Rikiya Takehi @rikiyatakehi]

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.

We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: https://t.co/bOWCcFhCVq

[Psyho @FakePsyho]

@KinasRemek I miss the times when you didn't post AI slop

[Psyho @FakePsyho]

@emirhanboge well, it's not even an interp blogpost, it's just a collection of loosely related experiments

[Psyho @FakePsyho]

@scaling01 as usual, a viral headline is all you need

[Psyho @FakePsyho]

@scaling01 @EpochAIResearch wtf, this is way more shocking than the crypto hacks

congrats!

[Psyho @FakePsyho]

@JaydenFugassi 🥲

[Psyho @FakePsyho]

when streaming Order of the Sinking Star, I try to explain my thought process whenever I find a solution interesting, here's one example

notes:
- that was near the end of my stream and I was running very low on energy, so you might want to watch this at 1.5-2x speed; sorry for broken English, this is all improvised
- if you don't know the rules: the goal in this level is to remove all of the goblins, they kill each other then they're on adjacent tiles; we can move them around by teleporting them through the mirror; our hero is rogue and unlike in the classic sokoban, rogue pulls objects instead of pushing them

[Psyho @FakePsyho]

@patrycjuszwyzga to może byc najbardziej chaotyczny wywiad który miałeś u siebie 😅 mam wielką nadzieję że mówiłem bardziej składnie niż wydaje mi się że mówiłem

[Psyho @FakePsyho]

staying at an airbnb
took me 10 minutes to find the switch for the lamp above the bed

this is some escape room-ass level design

[Psyho @FakePsyho]

@michalwarda @KinasRemek Congrats, although I don't remember saying anything even remotely useful 🥲

[Psyho @FakePsyho]

@adfontes_ @scaling01 of course, you can safely assume I know nearly everything that happened in this space
Jack was an amazing and prolific puzzle designer

[Psyho @FakePsyho]

a small announcement: I'm almost done with a puzzlescript-based benchmark; you can think of it as ARC-AGI-3.5, maybe even ARC-AGI-4; I expect it to be the most reliable way of assessing agents' abstract reasoning skills in a game environments

I went through over 300 games, handpicked the most interesting and diverse set of 40, fixed a ton of bugs, went through multiple iterations of prompt design, etc

most games can be completed in around 1h, some of them are significantly harder; models are tested using standard harnesses (codex/claude code) with no code generation allowed; action limits are very generous and unfortunately the benchmark is already somewhat saturated, but the models are still insanely action inefficient

FYI, I won't release the game set to the public, but I guess I might share runs from some of the rejected games.

btw, if anyone knows, what do I have to do in order to not pay $10K out of my own pocket for testing, I'm all ears

[Psyho @FakePsyho]

- I'm not aware of a single reasoning benchmark that is not 100% saturated and tests abstract reasoning skills; it feels that this should be an actually useful benchmark
- it's not that badly saturated; I've included some shorter & easier games in order to differentiate weaker models; the difficulty distribution is quite broad, there are a few games where the best models finished less than 20% of these games
- action efficiency is still very bad; on very hard levels, best models usually perform aimless random walk for 80% of actions; the final metric will probably combine both accuracy and action efficiency
- I'm interested in evaluating weaker models as well

I also started working on another benchmark based on larger games (so something like multiple mazebenches), I'm probably like 10% done, but that's going to be a much larger project

[Psyho @FakePsyho]

tbf, all of the data from EnigmaEval was publicly available pretty much from day 1 since they named their sources and all of those problems are on the internet

creating a benchmark based on puzzle hunt problems is probably still meaningful today - it's an interesting mix of testing deep research, persistence and potentially large context

[Psyho @FakePsyho]

@RbdJellyfish omg 🫠
my story with the game so far is that I either overcomplicate things or forget about an interaction that I encountered one hour ago

[Psyho @FakePsyho]

@Darpinian yeah, last year was too early
gpt 5.4 is probably the first model that was capable of playing decently some of the games; and then sol 5.6, opus 5 & astra 6 were obviously huge jumps

[Psyho @FakePsyho]

@PrzemekUrbanski there's a ton of interesting observations honestly
I hope to do a proper report on that with the release

probably not the next week though, as I'm traveling/eventing

[Psyho @FakePsyho]

MazeBench is hard, like very hard; many puzzles are not directly brute force'able (i.e. state space is too large); it's also a bit confusing for the agent due to large size (context too small, or at least agents are not efficient with managing)

btw, I'm also started working on another benchmark (actually I got 3 more ideas while working on this one) that should be much larger and more general than mazebench, but that's a larger project that will take more time