In simple terms, yes, but there’s a bit more nuance to it. Information delta is a function of understanding what data we’re capturing and how much of it actually provides a signal that the model doesn’t already have.
The earlier in the pipeline you can understand that, the higher the value.
As AI continues to advance, extracting meaningful information and signals from the real world will become increasingly difficult. So will finding the right experts who can provide them.
One metric I increasingly care about is the information delta which is essentially what does an expert know, observe, or understand that frontier models don't and what is the best way for us to extract that information.
I believe that gap will become one of the most valuable things in AI development.
Expert data has a selection bias problem.
In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵
Expert data has a selection bias problem.
In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵
Today we're open-sourcing GlobeBench, 1,632 tasks that test whether LLMs can simulate RL environments well enough to train agents on.
The top model, GPT-6 Astra, gets 49.6% right. No other model of the 20 we tested reaches 30%. 🧵