Ravi Theja

Applied AI @SnorkelAI, Previously - @MistralAI , @llama_index (LlamaIndex)

Building data and evals for frontier labs.

[Ravi Theja @ravithejads]

AutoResearch experiment: I went into a GPU Mode kernel competition with very little CUDA background and engineered a research loop around coding agents: goal, rules, experiment memory, verifier, and compute.

Finished 5th overall. Give it a read. https://t.co/Y7UDJ9yiFI

[Jerry Liu @jerryjliu0]

Today we’re releasing OCRia-1, a frontier VLM for document parsing that sets a new SOTA in accuracy, cost, and latency.

It not only sets a new high score on ParseBench (87.4), but also leads in three 3rd-party benchmarks: olmOCR-bench* (83.5 overall, 92.9 on tables!), PulseBench-Tab (84.1), and fr-bench-pdf2md (72.1).

OCRia-1 is really, really good at parsing complex tables: nested headers, merged cells, irregular grids, scanned tables, and more. It is also fantastic at parsing complex unlabeled charts (line, bar, pie, technical diagrams), and preserving complex formatting. Most real-world enterprise documents contain at least one of the these, whether it’s a big tax withholding table or a public economic report.

It is a 4B model and outperforms all other models within a similar parameter class: LightOnOCR-3, Chandra-OCR 2, Mistral OCR 4.1, and more. It is also extremely fast.

All credit goes to our research team, who spent countless hours on curating a diverse dataset, tuning our training pipeline, and squeezing every last drop out of model optimization so that it can process billions of pages at scale.

OCRia-1 as a base is powering all of our modes within LlamaParse (cost-effective, agentic, agentic plus), so you can access its capabilities today!

If you’re processing millions/billions of documents at scale, we’re offering enterprises that book a demo up to $3k in credits for a guided evaluation.**

*olmOCR-bench scores are adjusted based on unifying rendering at 1,288px for all models
**enterprise POC credits apply to eligible organizations that book a demo

[Alex Ratner @ajratner]

The @SnorkelAI team did an audit of these findings using our internal QC platform.

Tl/dr: We agree with many of the findings, but ~55% of the fixes from @preferencemodel had their own issues with reward hacking or over-prescriptiveness.

Additionally: ~40% of the issues found were already tracked in the upstream repo - including 21 tasks with outstanding fix PRs!

Building datasets like this - and maintaining ambitious open benchmarks around them - is hard. We believe it requires open, collaborative review with community maintainers and relevant domain experts to get right.

@terminalbench was one of the first community efforts to build an open, continuous benchmark, and drive a vision of how OSS contributes to the rapidly advancing frontier of AI. We hope that the community will invest in working collaboratively with benchmark maintainers and task authors to keep this vision alive.

---

Some takeaways + details of our preliminary audit:

(1) We confirm 100% of the reward hacking / verification gap findings using our QC system. However, we find that 17/44 (39%) of the corrected tasks are just as or more reward hackable. Additionally, we don’t yet see evidence that these verification gaps meaningfully affected benchmark rollouts within CIs, though our analysis here is still ongoing.

(2) Our preliminary review raises questions about the underspecification findings for 22/41 (54%) of the flagged tasks, including 7 explicitly marked by task authors as intentional design choices. These cases warrant expert peer review and adjudication: balancing over/under-specification is a critical and tricky design decision - much of human intelligence concerns dealing correctly with reasonable and realistic underspecification - and we believe these corrections made to the tasks are likely too over-specified, reducing realism and difficulty relative to spec.

(3) Around 40% of the issues found were already tracked in the upstream repo (including 21 tasks with outstanding fix PRs)! OSS projects require community efforts to correct identified open issues, and ideally audits like these are done collaboratively with task authors and maintainers directly, in the traditional community spirit that has kept OSS alive for decades.

[Fred Sala @fredsala]

Really enjoyed Terry Tao's talk at @SAIRfoundation 's Science x AI Summit.

Particularly liked this thoughtful categorization of the old and new worlds of math (pre and post-AI), with the degree of human understanding being the key variable to realizing the benefits of AI and avoiding the pitfalls.

"Misaligned math" is also a nice way to term the possible bad case.

[tokenbender @tokenbender]

Introducing Chantless: Chapter 1 — Apprentice

A powerful tool makes you free, allows you to express yourself, and learns and grows with you.

This is what we're building with Chantless. A technology that lets you shape and evolve models alongside you.

[Ravi Theja @ravithejads]

@aliansarinik Fo from @wajo_ai is missing.

Cc: @shivanipod

[Alex Ratner @ajratner]

Excited to see this incredible progress and leadership, especially for the US/Western open frontier! Congrats @reflection_ai !! 🚀

[Reflection @reflection_ai]

Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.

- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch

Full weights release this month.

Learn more about Beam: https://t.co/c3Qx2cpM8G

[Jerry Liu @jerryjliu0]

Today we’re introducing Extract v2.5 - a series of frontier agents tuned for document extraction.

The agents (cost-effective, agentic, agentic plus) are tuned for value accuracy and grounding. Our extraction agents outperform Opus 5.5 and GPT-6 Sol while being 30%-4x cheaper.

We’ve made massive improvements on complex extraction over
* long lists (86.1% -> 95.5% on our agentic tier)
* records spanning pages (85.5% -> 96.5%)
* scanned forms (90.9% -> 95.7%)

We’ve also launched the following features:
✅ Advanced citations: we locate bounding boxes for all supporting values for an inferred field, even if there's not an exact match.
✅ Structural Reasoning: we tailor document extraction algorithms depending on the type, layout, and information

Our extraction agents are SOTA in price-performance on ExtractBench, across a wide range of cost points. We are the best tool for document extraction across documents of any complexity.

Blog: https://t.co/TTlTdECU03

All of these are available on LlamaParse: https://t.co/XYZmx5TFz8 . Come check it out!

[Jason Weston @jaseweston]

⏱️⚙️Introducing *AutoBenchmark* 📊🏁
- Creating benchmarks automatically
- Benchmarking benchmark creation
- Studying the role & impact of humans in the loop

Blog post: https://t.co/CAZ2fOczYJ

Key takeaways:

1) We find human-agent collaboration gives big wins over agents alone
- fine-grained feedback in ideation stage crucial
- autobench can be used to measure this in the future with stronger agents

2) We show that it's possible to make *autoresearch benchmarks* for AI research using this recipe – full recursive improvement loop!

3) Important Ingredients: Autobenchmark creation works best with feedback from two sources: benchmark solvers + external verifiers (human+AI).
🧵1/5

[Ravi Theja @ravithejads]

@shivanipod Congrats on the launch!

Been using Fo for apartment hunting and booking doctor appointments. I especially love the reminders. It keeps nudging me about followups I’ve missed or tasks that have been sitting in my inbox for days.

[Alex Ratner @ajratner]

Strongly agree here. And a few quick thoughts:

- (1) Basic research will increasingly be about the empirical science of studying and understanding, not just improving model performance.

As AI models are increasingly *grown* as much as they are engineered, and as the complexity scale increasingly leads to emergent properties over discretely engineered ones, a lot of basic research is going to look more like classical empirical science - i.e. doing experiments, exploring, understanding, modeling, etc.

(One type of experiment: a benchmark, if done correctly; that's what we are attempting to fund and support through @SnorkelAI Open Benchmarks Grants!)

- (2) Basic research can prioritize *global optimization* (e.g. trying new architectures / algorithms / paradigms), which will always involve sacrificing local optimality at first.

These first, less optimal steps - which could take months, or years - are hard for *any* commercially-funded organization to do fully (especially once basic economic gravity returns in some areas of the space). This is the gap that basic research institutions have always- and will always- need to fill!

Also: these global optimization explorations do not always need 10K+ GPUs!! In fact, many will be selected specifically to explore new paradigms/approaches that are more compute efficient.

- (3) Basic research has many other beneficial externalities, e.g. supporting training/teaching, open dissemination of knowledge, etc.

[Deepinder Goyal @deepigoyal]

India has always idolised actors and cricketers. The last few years, entrepreneurs and founders have also gotten mobbed at the airports. How I wish we started celebrating scientists as much as we celebrate actors, cricketers, and founders.

On that note, meet the Temple scientists behind Brain Flow: Dr. Divya Gulati (IISc), Dr. Sanchit Gupta (IISc), Dr. Yukti Chopra (SISSA Trieste), and Nitish Kumar (IIT Bombay).

This team led, built, and validated the first continuous, wearable measurement of blood flow to the brain.

Your brain uses 20% of your blood supply. It controls your hormones, sleep, immunity, and repair. That supply drops every year as you age, and you have never measured it. Even hospitals can only capture a single snapshot.

Measuring and maintaining brain flow could be one of the most important health and longevity interventions of our time.

At Temple, we have some of the best minds in the country, working on the one problem that sits upstream of every other: extending healthy human life.

I am proud of our team, and looking forward to a lot of firsts and breakthroughs in the future.

[Ravi Theja @ravithejads]

@anand_404 Congratulations and all the best

[Paras Chopra @paraschopra]

🥳 Excited to share that @lossfunk got 4 papers into NeurIPS main conference.

Please join me in congratulating the authors!

Aman @arcaman07, ⁠Sushrut @martisamuser, ⁠Abhinav @MajorTimbWlf21, Pranav - @Avg_sapient

First authors are all undergrads :)

[Ravi Theja @ravithejads]

@adithya_s_k This looks really useful. Congrats on the launch!

[Alex Ratner @ajratner]

I'm excited to announce @SnorkelAI's $350M Series E at $3.5B, led by @insightpartners and @S32_VC.

We've grown 18x+ in the last 12 months since launching our Data-as-a-Service offering, passing $375M ARRR this week.

As AI advances to superhuman capabilities, AI data & environment development must advance with it - and basic staffing and crowdsourcing approaches are not enough.

AI progress now requires deep research and technology work that combines human expertise with specialized AI in compounding ways. @SnorkelAI is building the RSI data engine and frontier data lab for this next phase.

We're honored to have the support of existing investors Addition, @lightspeedvp, @GreylockVC, @GVteam, P7, Factory, @WellsFargo, Walden Catalyst Ventures, and new investors @ThirdPointLLC, @MarchCPs, @BlumbergCapital, @AllegisCapital, @Frontlinevc, and @standard_vc.

–

@SnorkelAI started as a research project a decade ago at @StanfordAILab.

Our thesis was simple: AI progress would become increasingly data-centric – and therefore data development should be studied as a true research and technology problem, not just a staffing and crowdsourcing one.

Today, as AI capabilities verge on superhuman, building the data and environments to safely measure and train AI is becoming too hard for even the smartest human experts to do alone.

Only humans and AI agents, collaborating together in compounding ways, can meet the accelerating needs of the frontier, and keep humans in the driver’s seat of AI progress for decades to come.

At @SnorkelAI, we are building the data lab to define the shape of this new “Data 2.0” frontier, and the new paradigms of human-computer interaction needed to advance it.

Our key focus is building the RSI engine for data, where specialized AI models accelerate and improve human expert output, and in turn, scaled human supervision is used to continuously evaluate and improve these models – creating a powerful compounding loop to keep pace with an accelerating RSI frontier.

With this round of funding, we are also doubling down on our commitments to support data development for open benchmarking and evaluation (more news here soon!); an increasingly diverse ecosystem of general and specialized intelligence; and a path to safe, well-aligned AI built on robust training and evaluation data.

Data development will guide and drive the next stages of AI – and must do so in a human-centric, AI accelerated, open, diverse, and safe way. We are excited to support this mission in the next decade of research ahead at @SnorkelAI.

More thoughts here: https://t.co/Dzk6olqmAc

[Ravi Theja @ravithejads]

@jerryjliu0 Made me feel nostalgic :) I still cherish those early gpt_index/llama_index days. Some of the most fun I’ve ever had building with LLMs.

[Ravi Theja @ravithejads]

We thought AI needed good computer-use models to play games. Turns out, all it needed was Jev.

[Max Blade @_MaxBlade]

jev is insane 🤯

Here is Jev playing subway surfers at super human speed, and also playing 50 games at once.

cost less than a cent to do this run.

Jev does not replace llms like astra or fable, but opens up an entirely new world of capabilities.

[Ravi Theja @ravithejads]

Interesting that 9/15 reviewed benchmarks are reported as "Flawed"

[Epoch AI @EpochAIResearch]

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

[Ravi Theja @ravithejads]

@sophiamyang @FireworksAI_HQ Congrats