But if you make the comparison a bit more stringent by fitting the linear transformation using only models released before October 2025, the ECI suggests slightly more acceleration than the MCI.
I also helped develop the METR Capability Index (MCI).
Interestingly, the MCI has very similar trends to the ECI, despite having different methodologies, tasks, and models. If you overlay the MCI against the ECI (allowing for a linear transformation between the two axes), the trends are remarkably similar.
https://t.co/RSFjblo2hH
We've been working on understanding model capabilities beyond what our prior time horizon tasks could capture.
Today we’re sharing a research update. We find that AI R&D-relevant capabilities are improving rapidly and are getting harder to measure.
When speed-running nanoGPT, agents tended to make training steps faster, while human speedups tended towards cutting the number of training steps needed to reach the target loss.
I was appendix czar for this post — here are some of my favorite results:
Larger swarms tended to have bigger gaps on their performance on the test set they generate compared to what the hidden tests are actually testing for. This might suggest an “echo-chamber” effect, where swarms convince themselves to over-optimize on bad metrics, and don’t check their assumptions well.
https://t.co/T4geqwP7GZ
We've been working on understanding model capabilities beyond what our prior time horizon tasks could capture.
Today we’re sharing a research update. We find that AI R&D-relevant capabilities are improving rapidly and are getting harder to measure.
Fable models seem to have very complicated relationships with reward hacking. In independent runs of the same task, Fable 5.1 would sometimes be very cautious to avoid reward-hacking opportunities and other times spend lots of time exploiting that same reward hack.
We've been working on understanding model capabilities beyond what our prior time horizon tasks could capture.
Today we’re sharing a research update. We find that AI R&D-relevant capabilities are improving rapidly and are getting harder to measure.