A dataset can fit your industry and still miss what your agent needs to do.
Humanbased is an AI data marketplace where developers define tasks and rewards, and experts contribute knowledge to create the data.
Start a campaign: https://t.co/0Sva5FTdUp
Ask your coding agent to draft the campaign. Keep the final call with you.
This Humanbased preview takes a goal through MCP or the CLI to a proposal you review and confirm before launch.
CLI and MCP are in preview.
https://t.co/0Sva5FSG4R
Finding conflicting evidence is not enough. Does it change the agent's conclusion?
DAYJOB: Law exposes that gap. For developers: test the same task with complete, missing and conflicting evidence. Evaluate the decision, not just retrieval.
⚖️ DAYJOB: LAW, our new benchmark measuring whether agents can handle the work of experienced lawyers ⚖️
An investor is about to put $20 million into a startup. The company’s ownership spreadsheet looks clean. But its official stock ledger shows 350,000 shares held by a former executive that the spreadsheet leaves out. The board approved a buyback, but there’s no proof it ever happened.
Would an AI agent notice? Better question: would it do anything about it?
That’s one of the scenarios in DAYJOB: Law, our new benchmark for evaluating AI agents on real-world legal work, with 2,638 messy case files and nearly 1 GB of data to sort through.
The agent has to recognize that the missing shares could change the investor’s ownership stake and the price they should pay, even though the request never mentioned them.
Congrats 🏅GPT-6.2 Astra, 🥈GPT-6.2 Sol , and 🥉Sonnet 5.5 for the top three spots.
Your eval panel picked the better answer. What happens next?
Pull the accepted judgments into your own code and apply your release threshold. Humanbased supplies the judgment records; your code makes the call.
A person confirms each launch.
https://t.co/0Sva5FSG4R
A legal answer can read beautifully. Can you point to the passage that supports it?
In this Humanbased demo, agents draft, people keep the evidence, and an independent reviewer checks the record.
Real source; synthetic contributors.
https://t.co/0Sva5FSG4R
Can you trust your AI grader?
Anthropic recommends checking it against independent human labels.
Our take at Humanbased: developers and experts should co-create the criteria, edge cases, and reference labels behind the score.
Safety evaluations are critical for understanding whether a model is ready to deploy, which safeguards to build for it, and where to focus training and research.
On our Engineering Blog, we offer 10 recommendations for how to design, run and report them.
https://t.co/MtSQmqqi2D
New AI capabilities bring new data needs: task demos, realistic scenarios, expert references.
Humanbased connects developers and experts to create and review that data.
10 stories in our first AI Data Weekly. What data is your next iteration missing?
Which frame does the action actually start in?
If you work with EGO data, here's a Humanbased flow from egocentric footage to action annotation, independent review and an accepted record.
Demo with synthetic contributors.
https://t.co/0Sva5FSG4R
The campaign is running. You've just spotted a field you should have asked for.
Add it. Humanbased lets you edit the fields, who qualifies and the reward while work runs. You can also restore a saved version.
https://t.co/0Sva5FSG4R
Have one data task you want to try? Start there.
Top up your Humanbased balance from $1, set the task and reward, test the workflow, then launch with a budget cap.
Here's the setup, step by step.
https://t.co/0Sva5FSG4R
Got a clear idea of the data your agent needs?
Build the workflow around it. Choose who does the work, reuse your courses and forms, and adjust the campaign as you learn.
Take a look inside the Humanbased Developer Portal.
https://t.co/0Sva5FSG4R
You know what your agent needs to learn. Now make the task clear to the people creating its data.
Define the work, choose who qualifies, set the acceptance rule. That's a Humanbased campaign.
https://t.co/0Sva5FSG4R
@victor207755822 Hi Deli, work with Humanbased. We’re building expert-validated CAD and TB4 data for mechanical design, manufacturing & construction: CAD files, task context, intermediate steps, tool traces & expert review. Relevant to your team’s CAD/3D reasoning or domain agents?
@PKUCXK Hi Xiaokang Chen, work with Humanbased. We’re building expert-validated CAD and TB4 data for mechanical design, manufacturing & construction: CAD files, task context, intermediate steps, tool traces & expert review. Relevant to your team’s CAD/3D reasoning or domain agents?
@yzhang_cs Hi Yu Zhang, work with Humanbased. We’re building expert-validated CAD and TB4 data for mechanical design, manufacturing & construction: CAD files, task context, intermediate steps, tool traces & expert review. Relevant to your team’s CAD/3D reasoning or domain agents?
@ShengjieWa34067 Hi, work with Humanbased. We’re building expert-validated CAD and TB4 data for mechanical design, manufacturing & construction: CAD files, task context, intermediate steps, tool traces & expert review. Relevant to your team’s CAD/3D reasoning or domain agents?
AI safety isn’t just about refusing harmful requests. It’s also about helping with legitimate ones.
At Humanbased, we believe reusable tests should score both: harmful responses and unnecessary refusals. Reflection’s Beam work shows why this distinction matters.
Sharing more about how we trained and evaluated Beam, our first model, for safety: https://t.co/FZIbzeUWav
In the coming weeks, we'll release additional safety research, evaluations, and tools under Project Iris, Reflection's open AI safety initiative.
We are committed to prioritizing safety in every model we build—and sharing what we develop and learn so that researchers, developers, policymakers, and enterprises can build on our work, advancing safety across the broader AI ecosystem.
Building an agent or a domain-specific model?
On Humanbased, define the task, set who qualifies, and connect human contributions with agent checks before acceptance.
You set the reward. No sales scoping call. No minimum commitment.
https://t.co/0Sva5FSG4R
Expert reviews are more than records of decisions. They can become reusable AI evaluation assets.
Our view at Humanbased: turn experts’ questions, concerns, and rejection reasons into tasks and scoring criteria—so AI can be evaluated against the evidence and risks that matter.
@YorkYang5050 Human-to-robot learning needs quality video and precise action data. Humanbased brings trained teams together for capture, annotation, and validation, tailored to your training needs. Happy to discuss via DM.
@JasonMa2020 Human-to-robot learning needs quality video and precise action data. Humanbased brings trained teams together for capture, annotation, and validation, tailored to your training needs. Happy to discuss via DM.