Expert data has a selection bias problem.
In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. π§΅
@karpathy@scaling01@Tim_Hua_@ArnhavD@VarunGangal Arnhav and Varun study how models reason and act. We're hiring researchers and engineers in San Francisco to do behavioral research like this with them.
https://t.co/XSD3xQ5Bak
βNeed inspect repo. Grep.β
Last year open models said βLet me start by exploring the repository.β Now several write exclusively in clipped, article-free shorthand.
Across 12 open models Γ 20 coding tasks, filler words fell 25β50% between releases.
But terser isnβt cheaper. π§΅
@karpathy@scaling01@Tim_Hua_ Why waste time say lot word when few word do trick?
Few Word Do Trick, by @ArnhavD and @VarunGangal. The full writeup has the per-family charts, the filler rules and the limitations.
https://t.co/kZQZhAOYwP
Today we're open-sourcing GlobeBench, 1,632 tasks that test whether LLMs can simulate RL environments well enough to train agents on.
The top model, GPT-6 Astra, gets 49.6% right. No other model of the 20 we tested reaches 30%. π§΅