Rubric-based LLM evals are everywhere. But how do you know your rubric is any good?
RIFT names 8 ways rubrics quietly break + automated diagnostics to catch them, great work by @SnorkelAI. Had a lot of fun building a repo to implement it and rerun the experiments.
🛠️ Repo: https://t.co/s5sTPlWU4T
📄 Paper: https://t.co/fqMAuAvL0Q
Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment:
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods
Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.
Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.
We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: https://t.co/E5kfZQ0Im5
I'm excited to announce @SnorkelAI's $350M Series E at $3.5B, led by @insightpartners and @S32_VC.
We've grown 18x+ in the last 12 months since launching our Data-as-a-Service offering, passing $375M ARRR this week.
As AI advances to superhuman capabilities, AI data & environment development must advance with it - and basic staffing and crowdsourcing approaches are not enough.
AI progress now requires deep research and technology work that combines human expertise with specialized AI in compounding ways. @SnorkelAI is building the RSI data engine and frontier data lab for this next phase.
We're honored to have the support of existing investors Addition, @lightspeedvp, @GreylockVC, @GVteam, P7, Factory, @WellsFargo, Walden Catalyst Ventures, and new investors @ThirdPointLLC, @MarchCPs, @BlumbergCapital, @AllegisCapital, @Frontlinevc, and @standard_vc.
–
@SnorkelAI started as a research project a decade ago at @StanfordAILab.
Our thesis was simple: AI progress would become increasingly data-centric – and therefore data development should be studied as a true research and technology problem, not just a staffing and crowdsourcing one.
Today, as AI capabilities verge on superhuman, building the data and environments to safely measure and train AI is becoming too hard for even the smartest human experts to do alone.
Only humans and AI agents, collaborating together in compounding ways, can meet the accelerating needs of the frontier, and keep humans in the driver’s seat of AI progress for decades to come.
At @SnorkelAI, we are building the data lab to define the shape of this new “Data 2.0” frontier, and the new paradigms of human-computer interaction needed to advance it.
Our key focus is building the RSI engine for data, where specialized AI models accelerate and improve human expert output, and in turn, scaled human supervision is used to continuously evaluate and improve these models – creating a powerful compounding loop to keep pace with an accelerating RSI frontier.
With this round of funding, we are also doubling down on our commitments to support data development for open benchmarking and evaluation (more news here soon!); an increasingly diverse ecosystem of general and specialized intelligence; and a path to safe, well-aligned AI built on robust training and evaluation data.
Data development will guide and drive the next stages of AI – and must do so in a human-centric, AI accelerated, open, diverse, and safe way. We are excited to support this mission in the next decade of research ahead at @SnorkelAI.
More thoughts here: https://t.co/Dzk6olqmAc
Excited for our @SnorkelAI Frontier Data Summit!
Most importantly to this moment (when robust, independent evaluation is appropriately top of mind): the core of this summit is a poster session with leading open benchmark developers, sponsored by @Snorkel Open Benchmarks Grants!
Frontier Data Summit: Oct 8, SF. @fchollet, @sanmikoyejo, @nikogrupen, @ajratner, @alexgshaw, @StevenDillmann, and others onstage plus 25+ posters with the teams behind Agent's Last Exam, OSWorld 2.0, Terminal Bench Science, Terminal Bench, CollusionBench, PostTrainBench, T² Scaling Laws, SlopCodeBench, LONGRUN, SREGym, BizBench, and more.
@XFreeze Glad to see this getting attention! This chart is from the MedAgentBench leaderboard at Medical Sphere! 🙂
For anyone interested, full results and all model runs on our platform: https://t.co/kjpuLKQx3f
@elonmusk Glad to see this getting attention! This chart is from the MedAgentBench leaderboard at Medical Sphere! 🙂 For anyone interested, full results on our platform: https://t.co/kjpuLKQx3f https://t.co/Gnu3sVWbaS
Cool to see Elon sharing our work at @MedicalSphereAI, Grok 4.6 made a pretty compelling case! 🙂
Medical Sphere is built to address the widening gap between rapidly advancing AI model capabilities and their independent and objective validation in real-world settings. And that gap is even more critical in a domain like healthcare.