A few caveats. Everything is measured relative to O*NET's task list.
Coverage for 770 of the occupations is a lower bound. And coverage tells you what benchmarks test, not what AI can do.
If you work on AI and labor, we'd love your review.
https://t.co/8c2z2kxjYY
We estimate benchmarks touch the tasks behind 36% of US working time.
Only 4.5% is spent on tasks a benchmark tests closely, and the remaining 64% on tasks no benchmark tests at all.
Nearly two thirds is spent on tasks no benchmark touches at all.
Our goal is a score for every job that reflects real work, with model performance weighted by each step's share of working time and its wages.
Coverage, time and wages are already linked to every step. Per-step model results are the missing piece.
What do AI benchmarks actually test about real work?
Introducing Atlas of Work.
Our initial launch goes broad: we mapped 923 US occupations into 271k steps and searched 2.7k AI benchmarks for coverage.
Our findings, evidence, vision and methodology:
https://t.co/hnkRkj7IEQ
We have 1,000 benchmarks for coding and math, and roughly zero for what an AP specialist does all day.
Atlas is a map of human work, broken into workflows, with the coverage gaps showing. If you’re building envs, this is where to look.
Early access: https://t.co/SOCVcyF6ZQ
The easiest job to automate is the one you've never done.
Ask someone who does it what a smart outsider would get wrong. That conversation is a better starting point for building an agent than a job description.
We're building Atlas around that knowledge.
Atlas is a map of AI and human work, and we’re getting close to opening it.
Start with a profession, explore its workflows and see the research connected to them.
The waitlist is open now.
https://t.co/xcDbk8ySZI
We compared six real-work benchmarks and found five different ways of grading them.
A score tells you more once you know what counted as success and who or what made that call.
https://t.co/3ufLMSMXMI
What would you look up in Atlas first?
Give us a workflow rather than a job title, like reconciling a disputed invoice or investigating a late delivery, and tell us what you’d want to understand about it.
The world is rapidly changing we just saw a flying car yesterday. But this change is hitting AI the fastest and there’s a lot of uncertainty on how this affects real human work. We’ve taken on this challenge by building Atlas ( https://t.co/jSew3RfBp1 or check out @rlsupplyai ) to bring more clarity for people on how AI maps to real work and much much more. We want to help everyone navigate these crazy, scary but exciting times. As we fine tuned Atlas and get it ready for launch we would love to on-board more researchers, experts or anyone related in the space to try it out! Sign up on our waitlist or send me a message I can fast track you!
Follow one workflow far enough and you reach what a job title leaves out: the trigger, the decisions that change its course, and what counts as finished.
That’s the level of work we’re making explorable in Atlas.
Join the waitlist https://t.co/OLi17fl73A
In our O*NET analysis, 82% of occupations with both task labels had a Supplemental task rated more important than at least one Core task.
If you’re choosing what to train on, the labels alone won’t tell you which work matters most.
https://t.co/wvfh7biyx4
@NVIDIAAI Nemotron 3.5 ships open datasets, training recipes and RL environments alongside the model, which gives researchers a lot more to inspect and reuse.
The useful next step is checking which parts of real work those environments actually represent..
Meet NVIDIA Nemotron 3.5 Super VL
Our new open model combines reasoning and tool use with text, image, document, and video understanding for long-running AI agents.
120B total parameters with 12B active 🧵