The evals are not ok
New NeurIPS paper: Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking.
Evals drive deployment, safety standards, and research. Yet we ignore noise from prompt, judge, agent harness etc, & so underestimate error in benchmarking & annotation, & it gets worse with more data.
I'll get to what this means for CIs & how to fix, but first let's talk about how this makes benchmark hacking easier 🧵
@hammer_mt certainly a fair point! this does have some exposure to task complexity but i'm not convinced it's mostly a complexity measure. it's more a stability measure (e.g. how reliably the model can chain together the right set of tasks without getting distracted/stuck on a tangent).
Gentle Reminder: time horizon is a measure of stability, not intelligence. Stability is necessary but not sufficient for ASI.
For example, a "while loop" can run forever, but it's not intelligent. LLMs have had a stability flaw. The flaw is bad enough that we need to basically "reboot" the LLM after a few hours... re-prompt, redirect, etc.
Through hard work from AI researchers around the world, the flaw is getting addressed, and so soon we won't have to reboot LLMs as often. But just because we don't have to reboot LLMs all the time doesn't mean we're getting superintelligence. It means we're getting LLMs that can have the *same* level of intelligence they already had... but they can use that intelligence for longer periods before a human needs to reboot/redirect/reprompt/etc.
I recognize "reboot" is an imperfect metaphor, but I think it's the right metaphor because when you reboot a computer you're not fundamentally changing its intelligence. A computer you need to reboot isn't fundamentally smarter or more valuble than a computer you don't need to reboot... it's just cheaper to run because the IT person doesn't have to come hit a button as often.
There are many important risks of AI, and this graph is relevant to many of them (e.g. job displacement), but this isn't a graph that can lead to the conclusion in the QT below.
@hammer_mt certainly a fair point! this does have some exposure to task complexity but i'm not convinced it's mostly a complexity measure. it's more a stability measure (e.g. how reliably the model can chain together the right set of tasks without getting distracted/stuck on a tangent).
@AlexKrusz true, but i mean sometimes things really are hyperbolic. it's something to be cautious about but i don't personally use it as the only qualification to disqualify a belief. i am however cautious about stories about inf/-inf utility, because this is a tried-and-true brain hack
one could even argue that stability is a safety feature more than a risk. would you trust a self-driving car that turns off or changes destinations once every 5 minutes... once every hour... once a day? would you trust a boat that sprung a leak once a week?
i think history will look back on the hysteria over this graph as 20% legitimate concern about AI advancement and 80% hysteria that this is "the graph" when it really isn't. it's a measure of stability. stability is net-good. stability means a human told it to do a thing and it kept doing that thing until it was done.
@AlexKrusz i don't know if it's impossible, but once an AI can do a task for a year without a human debugging it, it's not clear to me that 2 years or 10 years is that much different.
Time horizon is a measure of stability, not intelligence. Stability is necessary but not sufficient for ASI. For example, a "while loop" can run forever, but it's not intelligent. LLMs have had a stability flaw.
The flaw is bad enough that we need to basically "reboot" the LLM after a few hours... re-prompt, redirect, etc. Through hard work from AI researchers around the world, the flaw is getting addressed, and so soon we won't have to reboot LLMs as often.
But just because we don't have to reboot LLMs all the time doesn't mean we're getting superintelligence. It means we're getting LLMs that can have the *same* level of intelligence they already had... but they can use that intelligence for longer periods before a human needs to reboot/redirect/reprompt/etc.
I recognize "reboot" is an imperfect metaphor, but I think it's the right metaphor because when you reboot a computer you're not fundamentally changing its intelligence. A computer you need to reboot isn't fundamentally smarter or more valuble than a computer you don't need to reboot... it's just cheaper to run because the IT person doesn't have to come hit a button as often.
There are many important risks of AI, and this graph is relevant to many of them (e.g. job displacement), but this isn't a graph that can lead to the conclusion in the QT below.
@DavidSKrueger Fwiw - I believe we could demonstrate this at small scale with PySyft's current level of GPU enclave support, and that such a project would likely yield a reasonable path towards a scaled up version.
@schisofrenia often it's non-public content for specific audiences ofc, and often it's not with cameras and light studios, but the basic template seems like its there