rakesh

please clap.

[Henry Kiss Ehrenberg @henryehrenberg]

We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns.

Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions.

We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that.

Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.

[Alex Ratner @ajratner]

Agentic AI will transform every enterprise–but only if agents are trusted experts.

The key: Evaluation & tuning on specialized, expert data.

I’m excited to announce two new products to support this–@SnorkelAI Evaluate & Expert Data-as-a-Service–along w/ our $100M Series D!

---

Snorkel Evaluate is our new data-centric agentic AI evaluation platform for specialized, mission-critical enterprise settings where vibe checks and out-of-the-box metrics driven by simple LLM prompts are not enough.

Snorkel Expert Data-as-a-Service is our white glove service for expert-level AI datasets, powering frontier LLM developers in areas like expert knowledge, reasoning, agentic action and tool use, and more!

Both built on top of @SnorkelAI’s Data Development Platform, using our programmatic technology to drive higher-quality expert data, faster– for getting specialized AI to real production value.

If you’re building enterprise AI and want to partner around the key ingredient in AI today–the data–book a demo and let's talk! https://t.co/w0J8izpn8p

Finally, see thread for details on 🧵👇
- 📽️ A walkthrough of Snorkel Evaluate and Expert Data-as-a-Service on an agentic AI enterprise task
- 📅 An upcoming event on Enterprise Agentic AI with innovators from @Accenture @BNY @Comcast @Stanford @QBE & others
- 📊 An upcoming series of benchmark datasets and model artifact releases

👀 Want early access to the full agentic AI dataset? Retweet this post and we'll send you the link!

[brooke bowman @gptbrooke]

don’t want to dox anyone but this is a beautiful venue and an even more beautiful moment

pulling for everyone who wants to get married to have their own moments like this

[helen @helen]

kids these days will never know what it’s like to have another scorcher with tomorrow being even hotter and ur husband says he’ll call sears and u mention that he said that yesterday and tell him to call now. so he calls now

[P.E. Moskowitz @_pem_pem]

the older i get the less cynical i get the more i am willing to admit that sometimes it’s nice to sit in a brewery with lots of beers in downtown cleveland or whatever

[rakesh @guharakesh]

@brian_curtin @dlberes it's a slippery slope after dolan

[rakesh @guharakesh]

@rber ☹️

[heems (visit veena.nyc) @HIMANSHU]

Let’s discuss the contradiction.

[rakesh @guharakesh]

@KaitlinNotKate

[Alex Ratner @ajratner]

1/ ‘22 was a big year for @SnorkelAI! We
- Introduced new foundation model & data-centric AI paradigms
- More than tripled our customer base, incl. 5/10 top US banks + top-10 in 7 other industries, accelerating their AI dev by 10-100x+
- Welcomed 40 new Snorkelers

And much more!

[rakesh @guharakesh]

the ocean is so beautiful

[Snorkel AI @SnorkelAI]

[1/5] Today, we’re excited to introduce Data-centric Foundation Model Development, a new paradigm for enterprises to use foundation models to solve complex, real-world problems. https://t.co/AqH32NPfd8

[rakesh @guharakesh]

@DookiePoopsicle @coffee_dad if you want i can just text you whenever i'm Having #a coffee

[rakesh @guharakesh]

@_SeanGroff @kentcdodds @reactjs DM? we'd love to have you at @SnorkelAI

[rakesh @guharakesh]

@DookiePoopsicle

[rakesh @guharakesh]

@rber i got a bunch of huel and immediately started cooking again out of shame 😞