Lori

[vincent sunn chen @vincentsunnchen]

Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment:
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods

Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.

Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.

We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: https://t.co/E5kfZQ0Im5

[Lori @lsdah1]

@ajratner @SnorkelAI @insightpartners @S32_VC Exciting, building complex data for frontier labs and enterprise agentic usage needs a specialized team who views quality as the highest priority.

[Lori @lsdah1]

@ajratner @SnorkelAI @insightpartners @S32_VC Exciting, building complex data for frontier labs and enterprise agentic usage needs a specialized team who views quality as the highest priority.

[Alex Ratner @ajratner]

I'm excited to announce @SnorkelAI's $350M Series E at $3.5B, led by @insightpartners and @S32_VC.

We've grown 18x+ in the last 12 months since launching our Data-as-a-Service offering, passing $375M ARRR this week.

As AI advances to superhuman capabilities, AI data & environment development must advance with it - and basic staffing and crowdsourcing approaches are not enough.

AI progress now requires deep research and technology work that combines human expertise with specialized AI in compounding ways. @SnorkelAI is building the RSI data engine and frontier data lab for this next phase.

We're honored to have the support of existing investors Addition, @lightspeedvp, @GreylockVC, @GVteam, P7, Factory, @WellsFargo, Walden Catalyst Ventures, and new investors @ThirdPointLLC, @MarchCPs, @BlumbergCapital, @AllegisCapital, @Frontlinevc, and @standard_vc.

–

@SnorkelAI started as a research project a decade ago at @StanfordAILab.

Our thesis was simple: AI progress would become increasingly data-centric – and therefore data development should be studied as a true research and technology problem, not just a staffing and crowdsourcing one.

Today, as AI capabilities verge on superhuman, building the data and environments to safely measure and train AI is becoming too hard for even the smartest human experts to do alone.

Only humans and AI agents, collaborating together in compounding ways, can meet the accelerating needs of the frontier, and keep humans in the driver’s seat of AI progress for decades to come.

At @SnorkelAI, we are building the data lab to define the shape of this new “Data 2.0” frontier, and the new paradigms of human-computer interaction needed to advance it.

Our key focus is building the RSI engine for data, where specialized AI models accelerate and improve human expert output, and in turn, scaled human supervision is used to continuously evaluate and improve these models – creating a powerful compounding loop to keep pace with an accelerating RSI frontier.

With this round of funding, we are also doubling down on our commitments to support data development for open benchmarking and evaluation (more news here soon!); an increasingly diverse ecosystem of general and specialized intelligence; and a path to safe, well-aligned AI built on robust training and evaluation data.

Data development will guide and drive the next stages of AI – and must do so in a human-centric, AI accelerated, open, diverse, and safe way. We are excited to support this mission in the next decade of research ahead at @SnorkelAI.

More thoughts here: https://t.co/Dzk6olqmAc

[vincent sunn chen @vincentsunnchen]

https://t.co/aNuEf6Yu9j

[Fred Sala @fredsala]

The coolest trend for AI is shifting from conversation to action—less talking and more doing. This is also a great opportunity for evals: we need benchmarks that measure utility, including in an economic sense.

@terminalbench is my favorite effort of this type!

[Alex Ratner @ajratner]

Scale alone is not enough for AI data. Quality and complexity are equally critical. Excited to support all of these for LLM developers with @SnorkelAI Data-as-a-Service, and to share our new leaderboard!

—

Our decade-plus of research and work in AI data has a simple point: scale alone is not enough. AI success is all about the quality, complexity, and distribution of data—in addition to volume. We’re excited to be powering leading LLM developers with @SnorkelAI Expert Data-as-a-Service, our white glove service for custom, expert-level AI datasets—and to now preview some of what we’re building via our new Expert Data Leaderboard (🔗 in 🧵) + upcoming OSS dataset releases!

Snorkel Expert Data-as-a-Service is built to meet the rapidly evolving data needs of the agentic AI world—where success is built on the quality, complexity, and distribution of datasets, in addition to size and scale.

This kind of high-quality, frontier AI data can only come from a union of technology and human expertise. With Snorkel Expert Data-as-a-Service, we’re powering frontier LLM developers across agentic, expert knowledge, reasoning, coding, multi-modal, and other task types via the combination of these two key components:
- (1) The Snorkel Expert Network: A global team of subject matter experts focused wholly on specialized knowledge–spanning thousands of topics in STEM/academic, vertical/professional, and consumer/lifestyle domains.
- (2) @SnorkelAI Data Development Platform: Our unique programmatic data curation and quality control platform, accelerating and improving expert authoring and review through principled techniques developed over the last decade of R&D.

Now: we’re incredibly excited to showcase some of the power of Snorkel Expert Data-as-a-Service via the new Snorkel Leaderboard—putting frontier models to the test in complex, agentic, and reasoning settings inspired by real industry scenarios (not esoteric puzzles)!

We’ll be releasing new leaderboards and accompanying expert-verified open source datasets (coming soon!) regularly. To start, we’re sharing three initial ones in preview:
- SnorkelFinance: Q&A over financial documents requiring agentic tool-calling and reasoning
- SnorkelUnderwrite: Agentic insurance tasks requiring industry-specific reasoning and tool use
- SnorkelSequences: Mathematical tasks requiring compositional multi-step reasoning

[Lori @lsdah1]

Was in class of the recorded sessions and it was absolute great contents and must have.

[Stanford NLP Group @stanfordnlp]

#NLProc and LLMs: Ready for some summer learning? The 2024 version of cs224n is out, with new content on pre-training, post-training, benchmarking, reasoning, agents, and more

https://t.co/PzRGTWUwPj

Want a cohort class experience? Also available (paid):
https://t.co/bDh197g0X6

[Andrej Karpathy @karpathy]

An attempt to explain (current) ChatGPT versions.

I still run into many, many people who don't know that:
- o3 is the obvious best thing for important/hard things. It is a reasoning model that is much stronger than 4o and if you are using ChatGPT professionally and not using o3 you're ngmi.
- 4o is different from o4. Yes I know lol. 4o is a good "daily driver" for many easy-medium questions. o4 is only available as mini for now, and is not as good as o3, and I'm not super sure why it's out right now.

Example basic "router" in my own personal use:
- Any simple query (e.g. "what foods are high in fiber"?) => 4o (about ~40% of my use)
- Any hard/important enough query where I am willing to wait a bit (e.g. "help me understand this tax thing...") => o3 (about ~40% of my use)
- I am vibe coding (e.g. "change this code so that...") => 4.1 (about ~10% of my use)
- I want to deeply understand one topic - I want GPT to go off for 10 minutes, look at many, many links and summarize a topic for me. (e.g. "help me understand the rise and fall of Luminar"). => Deep Research (about ~10% of my use). Note that Deep Research is not a model version to be picked from the model picker (!!!), it is a toggle inside the Tools. Under the hood it is based on o3, but I believe is not fully equivalent of just asking o3 the same query, but I am not sure.

All of this is only within the ChatGPT universe of models. In practice my use is more complicated because I like to bounce between all of ChatGPT, Claude, Gemini, Grok and Perplexity depending on the task and out of research interest.

[Forbes @Forbes]

Snorkel AI Raises $100 Million To Build Better Evaluators For AI Models https://t.co/b4YrqNF7Ta https://t.co/b4YrqNF7Ta

[Delip Rao @deliprao]

It’s been amazing to watch @ajratner build and grow @SnorkelAI from a neurips presentation and an open source repo to one of the highly valuable AI companies building impactful product features for data labeling. Very timely that they are bringing in that expertise to agentic workflows! If you still wondering why you need data in this world of high performing models, see the next tweet.

[Percy Liang @percyliang]

High quality data is key. Excited to work with Snorkel to improve our Marin models!

[Lori @lsdah1]

Snorkel AI Announces $100 Million Series D and Expanded Platform to Power Next Phase of AI with Expert Data - https://t.co/qfeynRPnSL

[Alex Ratner @ajratner]

Agentic AI will transform every enterprise–but only if agents are trusted experts.

The key: Evaluation & tuning on specialized, expert data.

I’m excited to announce two new products to support this–@SnorkelAI Evaluate & Expert Data-as-a-Service–along w/ our $100M Series D!

---

Snorkel Evaluate is our new data-centric agentic AI evaluation platform for specialized, mission-critical enterprise settings where vibe checks and out-of-the-box metrics driven by simple LLM prompts are not enough.

Snorkel Expert Data-as-a-Service is our white glove service for expert-level AI datasets, powering frontier LLM developers in areas like expert knowledge, reasoning, agentic action and tool use, and more!

Both built on top of @SnorkelAI’s Data Development Platform, using our programmatic technology to drive higher-quality expert data, faster– for getting specialized AI to real production value.

If you’re building enterprise AI and want to partner around the key ingredient in AI today–the data–book a demo and let's talk! https://t.co/w0J8izpn8p

Finally, see thread for details on 🧵👇
- 📽️ A walkthrough of Snorkel Evaluate and Expert Data-as-a-Service on an agentic AI enterprise task
- 📅 An upcoming event on Enterprise Agentic AI with innovators from @Accenture @BNY @Comcast @Stanford @QBE & others
- 📊 An upcoming series of benchmark datasets and model artifact releases

👀 Want early access to the full agentic AI dataset? Retweet this post and we'll send you the link!

[Lori @lsdah1]

#动物森友会 #AnimalCrossing #ACNH #NintendoSwitch

[Lori @lsdah1]

#动物森友会 #AnimalCrossing #ACNH #NintendoSwitch

[Lori @lsdah1]

#动物森友会 #AnimalCrossing #ACNH #NintendoSwitch

[Lori @lsdah1]

#动物森友会 #AnimalCrossing #ACNH #NintendoSwitch

[Lori @lsdah1]

#动物森友会 #AnimalCrossing #ACNH #NintendoSwitch