GTM @llama_index
All objectionable opinions are from the AI clone Muse is building with root access to my computer. Any you find smart and witty are my own.
Send me the page that breaks your parser.
Merged-cell tables, 30 year old scans, handwritten claims forms, engineering drawings you name it, we want the worst of the worst and the ones that cause the most pain today.
I'll run it through LlamaParse and post the output. No pitch, no form. Just reply or DM.
@hnshah This is absolutely true. The best I've worked with do both. Trying and failing during the day and ingesting relevant content off hours to get ideas for the next thing to try the next day.
@jasonlk When products and models are evolving so quickly from one day to the next you need to stay in front of your customers and prospects constantly to understand how and why these changes are creating value.
@morganjingram As a young rep it can be difficult to understand the downstream business impact of what you are selling even if you have a good grasp of the product. This is why case studies, success stories, testimonials, etc. should be mandatory reading for new reps.
@gregisenberg I find that the mistakes GrokBot makes start to really pile up over large enough datasets. Even for basic tasks like email and data pulls.
If you're evaluating document AI, ask which errors would actually break your workflow.
A missing decimal in a financial table can matter more than a page of formatting differences.
That's what I like about our ParseBench-2 release: taking error severity seriously.
Today weโre introducing ParseBench-2 - our most ambitious document parsing benchmark to date.
It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with human judgment. We do this by painstakingly assigning severity scores to 330+ error types, and tuning each metric to match the severity scores.
Existing benchmarks either use brittle deterministic metrics (e.g. OmniDocBench) or LLM-as-a-judge (ParseArena), which can help recognize equivalent representations but are coarse/stochastic/subject to bias by the prompt+model.
ParseBench-2 also contains:
โ 4.3k+ document pages (double from ParseBench-1).
โ An extra dimension (forms)
โ Much better ground-truth across all dimensions compared to ParseBench-1.
This is the next evolution of ParseBench, our hugely popular benchmark that has 50k+ monthly downloads, over 150+ OCR solutions benchmarked, and has been adopted by companies like NVIDIA, Cohere, Aleph Alpha, Perplexity, and others.
Alongside ParseBench-2, we are also releasing OCRia-1.5 - a new family of document parsing models that powers all our modes in LlamaParse. They represent a massive step up in capabilities from our latest release, and are state of the art in parsing complex tables, forms, charts, formatting, and assigning bounding boxes to complex layouts.
Come check out our blog post, technical report, and website.
Blog: https://t.co/TKbfCyTjht
Technical Report: https://t.co/eAu0s6ttA1
Website: https://t.co/PWczfhp0OX
If youโre interested in trying out our OCRia-1.5 capabilities, come sign up for LlamaParse: https://t.co/XYZmx5TFz8
@jerryjliu0 A wrong dollar amount and a formatting issue can both be parsing errors, but the consequences are very different. That distinction matters when you're deciding what you can actually automate. Glad to see it reflected in how we measure accuracy.
@jerryjliu0 Proof the convo was good: I completely forgot to take the photo ๐ Shoutout @imaanxsultan for the artistโs rendering ๐จ Great meeting everyone & thanks for coming out - canโt wait for the next one!
@jspujji This narrative is very alive in trad saas. Then I talked to someone working at a consulting firm and he said they consider the big opp to be building their own bespoke software to make consultants more efficient. That was eye opening. I think we'll see a convergence.
@Kazanjy No similar tool spend is a tough one. It can be harder to get the project moving at first, but if you can its almost like a guaranteed win. Might not be worth the time though tbh.
@emollick This explains why products I discover and come to love one week keep disappearing the next lol. The only way to keep things relatively stable is to build your own skills/projects.
@arushijain273 Welcome to the club! Already crushing my 100 followers, but Iโm replying to every micro micro micro influencer I can find with a religious fervor unseen since the Reformation. Pray for me ๐