Resume parsing isnβt as simple as it seems.
I'm excited to share our work on ResumeBench: a frontier benchmark of how well AI systems parse and extract data from resumes.
@llama_index leads the charge, with @reductoai and frontier AI models close behind.
Yet even the best system gets only 15% of resumes extracted perfectly π§΅
@raunakdoesdev / @aditabrm would love to see if ResumeBench will fail your test too. And I suppose the bigger question is what should a benchmark author do about something like this?
(We just launched ~5 mins ago and @reductoai is #2 on accuracy (Deep Extract), and #1 on speed (Extract): https://t.co/JSfcuDdb6g
Resume parsing isnβt as simple as it seems.
I'm excited to share our work on ResumeBench: a frontier benchmark of how well AI systems parse and extract data from resumes.
@llama_index leads the charge, with @reductoai and frontier AI models close behind.
Yet even the best system gets only 15% of resumes extracted perfectly π§΅
This work is personally close to my heart as we strive to build better systems that can reliably extract resume content!
If you'd like to add your system to the leaderboard, my DMs are open and we'd love to get feedback on our work!
You can learn more at: https://t.co/aeRJ17baOi
In terms of metrics, we've done a comprehensive deep dive into the tradeoffs between accuracy, cost, and latency.
@llamaparse Extract Agentic Plus tops the leaderboard with an F1 score of 87.16 and extracts 15% of resumes perfectly, with Extract Agentic being in the same weight class at half the cost.
@reductoai Extract is by fast the fastest while suffering on accuracy.
@Alibaba_Qwen is the cheapest (~$2 / 1000 pages) for > 80% accuracy.
@0xSigil Have you benchmarked Woof on any CUA/BUA benchmarks to see how well it can actually execute tasks compared to the others? Would love to get you on Online Mind2Web leaderboard!
@sachpatro97 We have built multiple cool trajectory viewers where we can also highlight "key steps" and decision points. Happy to trade notes.
This is because we build CUA datasets and also run Online Mind2Web leaderboards.