Ryan Peters

reverse engineering intelligent (learning) systems.
rl, evals, safety @emulated_ai

[Ryan Peters @ryanirl]

If anyone wants to chat, you can find me here tonight!

[Alex Shaw @alexgshaw]

Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union

https://t.co/g9QbDuBWB2

[Ryan Peters @ryanirl]

@alexgshaw Will be there with my team!

[Ryan Peters @ryanirl]

Unironically a super creative benchmark 😅

[Epoch AI @EpochAIResearch]

Can AI tell if you've built your IKEA furniture wrong? Our new benchmark, the Furniture Assembly Benchmark (FAB), gives models the manual and a photo of a half-completed piece of furniture and asks them to spot the mistake.

The top score has gone from 28% to 80% in just 10 months.

[Ryan Peters @ryanirl]

Holy

[Ryan Peters @ryanirl]

mimo-v2.6-pro has reached Grok / GPT 5.6 Luna level performance on DeepSWE v1.1

[Ryan Peters @ryanirl]

https://t.co/LQUbjFJPiy

[Ryan Peters @ryanirl]

2P calcium imaging of cerebellar Purkinje cell dendrites

[Theo Jaffee @theojaffee]

Better call Sol

[Ryan Peters @ryanirl]

"Across every trial a model acted once every 15 to 59 seconds" - It's currently very expensive and slow to run these tests, but give it a year or so and we might have realtime speed at near human performance at the current rate of improvement.

[Vals AI @ValsAI]

What tasks are left that humans find easy but today's models still find hard?

Two such tasks are computer use and games. We’re launching CUA-Bench, a benchmark testing how well AI can use a keyboard and mouse across 6 games (3 kept private) in real time.

To saturate it, models will need to output real-time actions and learn continuously from video, not just text.

[Ryan Peters @ryanirl]

Las misteriosas mariposas del alma

[Ryan Peters @ryanirl]

Did a little bit of processing of one of the videos. So clean! If you look closely enough, you can even see some dendrites.

[Murayama lab, RIKEN CBS @mlab_cbs]

覚醒・睡眠・麻酔下における大規模神経活動データを公開。脳領域、脳波、筋電位、脳状態、Ca²⁺波形、スパイクデータなどを含��ます。
ぜひデータをDLして、自由に遊んでみてください。新しい解析法やAIを用いた解析にも使っていただけたら嬉しいです。

[Ryan Peters @ryanirl]

Some insanely clean widefield 2p data. And it's open sourced!

https://t.co/LGYJCX46QG

[Murayama lab, RIKEN CBS @mlab_cbs]

覚醒・睡眠・麻酔下における大規模神経活動データを公開。脳領域、脳波、筋電位、脳状態、Ca²⁺波形、スパイクデータなどを含��ます。
ぜひデータをDLして、自由に遊んでみてください。新しい解析法やAIを用いた解析にも使っていただけたら嬉しいです。

[Ryan Peters @ryanirl]

Just to clarify: Not the ratio in the graph, but a ratio showing if the cheat was found or considered.

[Ryan Peters @ryanirl]

This correlates very well with my own experience. In the few benchmarks I have ran, Gemini 3.8 flash is consistently the easiest model to honeypot, and is the most likely to cheat when given the opportunity.

[Vals AI @ValsAI]

This investigation began when we noticed that, on BioMysteryBench, Gemini 3.8 Flash attempted to cheat in 21.5% of trials, roughly 14pp higher than the next model and more than 4x the roughly 5.0% rate for the rest of the field.

[Ryan Peters @ryanirl]

Although seemingly cheats less on SWE-bench Verified 🤔

I would like to see a study showing the ratio of "thought of or found a cheat to actually exploited the cheat". Anthropic and OAI models seem to find the cheats faster but then are also more likely (from my experience) to avoid cheating (either because they are more aligned or they are just enough eval aware to not fall for the honeypot). But Gemini models not so much.

I suspect on some of these rollouts the cheat is not obvious enough to be exploited and would heavily bias these results. For example, is cheating on BioMysteryBench easier and more obvious than on SWE-bench Verified?

https://t.co/Ie7EadD01E

[Vals AI @ValsAI]

SWE-bench Verified, an older benchmark, is easy to shortcut with simple Git queries. The attempt rates were an order of magnitude higher: GPT-5.6 Terra at 89.4% and GPT-5.6 Luna at 78.8%, with a long tail of models pulling the same trick.

[Ryan Peters @ryanirl]

https://t.co/MtUd4Se9fK

[Ryan Peters @ryanirl]

A gradient of autoresearch curves