Introducing RecruiterBench, the people-search benchmark most aligned with recruiting workflows.
At Wrangle, when developing our internal models, we realized that all existing people-search benchmarks (that we’ve seen) are flawed:
- They only score 10 candidates
Real recruiting workflows require sourcing through 100 or more profiles to make a hire. Evaluating accuracy across 10 people doesn’t provide an adequate picture of how a tool will perform on a req.
RecruiterBench measures precision over 200 returned candidates for each role, which we believe accurately represents search capabilities and performance.
- The queries are overly simplistic and contrived
Accuracy over “Product managers in Austin” doesn’t matter when you’re looking for someone with a specific skillset.
We built RecruiterBench on 290 complete active reqs on Ashby, spanning 58 unique industries. This also allows us to find unique insights, such as what industries and role types each tool excels or struggles with.
For example, some strong points we identified are:
Wrangle deep on AI infrastructure: p@50 of 60.4%, versus Metaview 39.6%, Wrangle Turbo 52.2%, Exa 21.8%, and Treg 9.6%.
Wrangle deep on food/agriculture: p@50 of 79.4%, versus Wrangle Turbo and Metaview at 56%, Exa 24.6%, and Treg 9.2%.
Juicebox on legal: on the same three roles, p@50 was 82.7%, ahead of Turbo 76.3%, Metaview 65.7%, Exa 55.7%, and Treg 11.7%. Wrangle Deep still led at 93.3%.
Metaview on Advertising/marketing tech: p@50 was 78.2%, leading Wrangle Deep's 77.8%, Wrangle Turbo’s 73.8%, Exa’s 45.6%, and Treg’s 17.6%..
- They evaluate with a single LLM judge, without nuance about must-have, dealbreaker, and nice-to-have criteria
RecruiterBench uses a two-part LLM scoring system to first disqualify candidates based on must-haves, dealbreakers, location mismatches, and over/underqualification. Then we use a second judge to determine relevance and give no/partial/full credit.
We’re excited to continue developing more accurate benchmarks, and to build the best recruiting tool in the world!
Check out RecruiterBench on Github: https://t.co/ACdb6A2wYU.
Today, @JudgmentLabs opens for general access.
We believe long running agent systems will generate more exhaust data than the world can currently process.
Hidden within agent traces are rich records of how to make these systems better: what users ask for, when agents stumble, and how agents interact with the external world.
Judgment is the most performant monitoring and improvement layer for long-horizon agents.
We’re excited to see what you build!
Introducing RecruiterBench, the people-search benchmark most aligned with recruiting workflows.
At Wrangle, when developing our internal models, we realized that all existing people-search benchmarks (that we’ve seen) are flawed:
- They only score 10 candidates
Real recruiting workflows require sourcing through 100 or more profiles to make a hire. Evaluating accuracy across 10 people doesn’t provide an adequate picture of how a tool will perform on a req.
RecruiterBench measures precision over 200 returned candidates for each role, which we believe accurately represents search capabilities and performance.
- The queries are overly simplistic and contrived
Accuracy over “Product managers in Austin” doesn’t matter when you’re looking for someone with a specific skillset.
We built RecruiterBench on 290 complete active reqs on Ashby, spanning 58 unique industries. This also allows us to find unique insights, such as what industries and role types each tool excels or struggles with.
For example, some strong points we identified are:
Wrangle deep on AI infrastructure: p@50 of 60.4%, versus Metaview 39.6%, Wrangle Turbo 52.2%, Exa 21.8%, and Treg 9.6%.
Wrangle deep on food/agriculture: p@50 of 79.4%, versus Wrangle Turbo and Metaview at 56%, Exa 24.6%, and Treg 9.2%.
Juicebox on legal: on the same three roles, p@50 was 82.7%, ahead of Turbo 76.3%, Metaview 65.7%, Exa 55.7%, and Treg 11.7%. Wrangle Deep still led at 93.3%.
Metaview on Advertising/marketing tech: p@50 was 78.2%, leading Wrangle Deep's 77.8%, Wrangle Turbo’s 73.8%, Exa’s 45.6%, and Treg’s 17.6%..
- They evaluate with a single LLM judge, without nuance about must-have, dealbreaker, and nice-to-have criteria
RecruiterBench uses a two-part LLM scoring system to first disqualify candidates based on must-haves, dealbreakers, location mismatches, and over/underqualification. Then we use a second judge to determine relevance and give no/partial/full credit.
We’re excited to continue developing more accurate benchmarks, and to build the best recruiting tool in the world!
Check out RecruiterBench on Github: https://t.co/ACdb6A2wYU.
“Less clicks, more output is essentially what I’m looking for. And that’s what Wrangle does really well.” - Ryan Taft, Talent Lead at Serval
Read the full case study: https://t.co/4YpyAkWuRK
Today, we’re announcing Zeroset’s $5.2M pre-seed round to deploy world models into enterprises, co-led by @GradientVC and @2048vc, with participation from Leblon Capital and others.
Frontier intelligence is capable of enormously complex tasks. And yet with an abundance of general capability and mandate, automation still requires enormous handwork. The models and systems that exist today lack an understanding of the world they act in.
Robotics faced a similar problem. Building capable robots required more than compute-scaling on policies; they needed representations of their environments, encoding state and how it changed under actions. Over time, those representations shifted from hand-specified states toward learned models of the world.
What is missing in the enterprise is a general system that represents work and learns from it. At Zeroset, we believe the same paradigm that enabled progress in robotics can be applied to represent enterprise “worlds.”
We open with Nebula, a system that captures the state and traces created as people and agents work across systems, using them to model how an enterprise works and changes over time.
These models contain the decisions, dependencies, and unwritten processes bespoke to a company. They are derived from the enterprise, and they should belong to the enterprise.
Companies should be able to use the best models for their work and train them on their own operating knowledge freely. The underlying system does not need to change.
Nebula is currently in closed research preview. If your team is automating complex work or exploring how to learn from the work your organization already does, please contact us.
We are building the system through which that intelligence becomes increasingly specialized to the enterprise, allowing agents to act through it, and models to learn from it.
If you’d like to join us in building this future, we’re hiring researchers and systems engineers.
Read more here: https://t.co/sAae9FFGzH