Vals AI

Public LLM Evaluation // @BloombergBeta @pearvc @a16z @HRTVentures @nextladder

[Vals AI @ValsAI]

For one weekend, we are benchmaxxing on purpose!

Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.

Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.

Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.

Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.

[Vals AI @ValsAI]

Congrats to @liquidai. Full results at https://t.co/4l3KueS37Z

[Vals AI @ValsAI]

Liquid AI's d1-ultra is the only decision model to get all 1,960 ClaimBench claims right. That matches Opus 5.5, Sonnet 5.5, Haiku 5.5 and GPT-6.1 Sol.

[Vals AI @ValsAI]

It costs $0.21 per 1,000 claims and answers in under a second. Haiku 5.5, the cheapest of those language models, costs 12x more and takes about 5 seconds.

Liquid's earlier d1 missed 26 claims, and most of those were subtle contradictions, where one small wording change makes the claim wrong. d1-ultra got every one of them. It costs about twice what d1 does.

[Vals AI @ValsAI]

Full results at: Full results at https://t.co/s8EG8XdxNw

[Vals AI @ValsAI]

Mercury 3 from Inception is #23 on the Vals Index, putting it just ahead of Beam 1.0 and well ahead of Inkling and Inkling Small.

[Vals AI @ValsAI]

The reason for this is caching: only 84% of Mercury 3's input tokens came from cache, compared with 95 to 98% for those models, and uncached input costs 10 times as much. Input was about 88% of its token spend on the Index. Code Migration alone cost $4.41 per task.

[Nikil Ravi @nikilravi]

if anyone's curious- https://t.co/wJQzYTxj6u is the framework we use to create and run evals!

[Vals AI @ValsAI]

Finance reshuffles the leaderboard: Gemini 3.8 Flash and Kimi K3 place far higher here than on the Vals Index, while GPT-6 Astra and GPT-6.1 Sol place far lower.

Opus 5.5 costs $11.67 per test run, while Wonder Full is 1.0 points behind at $3.14, 27% of the cost. DeepSeek V4.1 Flash 1017 is 4.7 points behind at $0.42, about 4% of the cost.

Full results and methodology: https://t.co/yuUPJ2OblA

[Vals AI @ValsAI]

Today we’re releasing the Vals Finance Index, a measure of AI’s ability to do finance work, weighted by each sub-sector’s share of U.S. GDP.

[Vals AI @ValsAI]

Claude Opus 5.5 leads the index at 65.6%, with Gemini 4 Argon (64.9%) within the margin of error.

However, no model leads every component. Opus 5.5 tops Finance Agent and Tax Agent Bench, and Claude Sonnet 5.5 tops Excel Modeling. Even the leaders fail more than half of the tax questions.

[Vals AI @ValsAI]

Apply today! https://t.co/qHUPSVpJuF

[Vals AI @ValsAI]

For one weekend, we are benchmaxxing on purpose!

Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.

Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.

Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.

Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.

[Vals AI @ValsAI]

Congratulations to the @KeenableAI team. For full results, visit: https://t.co/vq9PrZXIfp

[Vals AI @ValsAI]

Keenable (Pro mode) is now #1 on the Vals Web Search Index, scoring 49.1% overall. We tested Keenable’s search API paired with GPT-5.6 Sol.

It leads in finance at 53.9% and places second in legal at 44.2%, behind Claude Fable 5 with native search.

[Vals AI @ValsAI]

It is also quick: 0.55 seconds per search on average. Search time is about a tenth of task wall time; the rest is model inference.