By popular demand we’re open-sourcing CWE-Bench on Hugging Face.
AI practitioners can now access frontier cyber evals and understand what makes cyber data high-quality. We would love feedback and contributions on results and improvements.
Each CWE-Bench task gives an agent a real open-source repository and an audit instruction. The agent must find and fix the vulnerability without being told what it is or where it occurs. Deterministic verifiers check that the fix blocks the exploit and preserves legitimate behavior.
This is the largest Cyber release in the AI community!
Learn more on the HF blog.
Excited to see what you build.
Glad to see CWE-bench in the mix. We built it to see how well coding agents can find and fix security vulnerabilities in real codebases.
If you’re curious how yours does, we’ve open-sourced 100 tasks on Hugging Face. Linked below.
Congrats to @TencentHunyuan on taking #1 on CWE-Bench-AA with Hy4 at 76%.
We built CWE-bench to see how well agents handle real vulnerabilities. Thanks to @ArtificialAnlys for bringing it to the Cyber Index.
Tencent's Hy4 ranks #3 on the Artificial Analysis Cyber Index and #1 on CWE-Bench-AA
@TencentHunyuan's Hy4 scores 58% on the Artificial Analysis Cyber Index, 1 point ahead of DeepSeek V4.1 Flash 1017 (max). It trails Kimi K3.1 Pro (64%) and the trusted-access GPT-6 Sol (Daybreak Blue, max) (69%). API pricing is $0.834/$2.501 per 1M input/output tokens.
Key results:
➤ #1 on CWE-Bench-AA (76%). Hy4 leads DeepSeek V4.1 Flash 1017 (max) by 6 points on our implementation of @CollinearAI's CWE-bench v1, which tests agents' ability to find and fix real software vulnerabilities
➤ Strong capability for the cost. At $0.92 per task, Hy4 sits on the Capability vs. Cost Pareto frontier. It scores 2 points above Grok 4.7 (xhigh) at 1/11th the cost and 1 point above DeepSeek V4.1 Flash 1017 (max) at 2x the cost
➤ Fewer output tokens than peers. Hy4 uses 64k output tokens per task: 43% fewer than DeepSeek V4.1 Flash 1017 (max) and 37% fewer than Grok 4.7 (xhigh), but 49% more than Kimi K3.1 Pro and 56% more than GPT-6 Sol (Daybreak Blue, max)
Fable 5.2 takes #1 on CWE-bench v2: 75.0% pass@1 across 250 tasks, 7.9 points ahead of Claude Opus 5.5.
It scores 78.5% pass@1 on public tasks and 72.7% on held-out tasks, with 82.8% pass@4 overall.
Explore the leaderboard: https://t.co/24a7AYnW6B
Agreed! The future of computer-use data seems to be pushing on 3 fronts:
a) longer horizon
b) reasoning and making decisions like a domain expert under ambiguity (the longer the horizon the less detailed the task is)
c) preference-based, e.g. good literature survey or a good analytics report
Yesterday’s Diwali celebration at Collinear ✨
We hope everyone who celebrated had a wonderful Diwali, filled with light, joy, and time with loved ones.
By popular demand we’re open-sourcing CWE-Bench on Hugging Face.
AI practitioners can now access frontier cyber evals and understand what makes cyber data high-quality. We would love feedback and contributions on results and improvements.
Each CWE-Bench task gives an agent a real open-source repository and an audit instruction. The agent must find and fix the vulnerability without being told what it is or where it occurs. Deterministic verifiers check that the fix blocks the exploit and preserves legitimate behavior.
This is the largest Cyber release in the AI community!
Learn more on the HF blog.
Excited to see what you build.
Where do frontier models actually fail?
CWE-Bench is now open source on Hugging Face. We publish open benchmarks and technical breakdowns behind the scores.
Follow @CollinearAI for our next evaluation.
We build the tasks, environments, and verifiers that turn agent failures into training data.
We’re excited to bring them to the @FireworksAI_HQ Frontier Ecosystem as a launch partner.
Build with Collinear: https://t.co/dCE9MufDSO
Explore the ecosystem: https://t.co/eIdeZpaLZ6
Introducing Collinear Zero: a Pareto-optimal defensive cyber agent.
Zero adds independent attack testing, one fix round, and a final recheck to security patching.
On CWE-bench v1, adding Zero to OpenCode takes Opus 5.5 from 67.1% to 72.9% pass@1, using DeepSeek V4.1 Flash 1017 as the tester. DeepSeek with Zero reaches 69.4% at $0.213 per attempt.