Connor Frank

@valsai
Princeton ‘26

[Connor Frank @conjfrnk]

@SarvagyaSom1shi Scooby

[Vals AI @ValsAI]

We have found a critical safety and alignment issue in recent public and internal models from multiple frontier labs which allows them to be misused in a high-stakes domain. It may also increase the risk of catastrophic misalignment.

We have disclosed it to the labs, and they are investigating and mitigating it. As an independent third-party evaluator, we maintain complete editorial independence and will share our findings publicly in the coming weeks.

[Connor Frank @conjfrnk]

There are hills, and they are being climbed.

[Vals AI @ValsAI]

Does it matter where you run an open-weight model?

Today we are releasing the Inference Provider Index. It compares first and third-party open-weight models for quality on agentic tasks in finance, coding, and legal industries.

Unlike other inference benchmarks, we rerun it every two weeks to catch quality degradation, and we use agentic tasks, not single-turn multiple-choice questions. It covers over 650 hours of model runtime.

[Vals AI @ValsAI]

For one weekend, we are benchmaxxing on purpose!

Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.

Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.

Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.

Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.

[Vals AI @ValsAI]

Today we’re releasing ClaimBench, our new benchmark targeted towards decision models. It tests whether a model can tell if a financial document supports, contradicts, or doesn’t address a claim.

We evaluated 15 models on 1,960 claims paired with 10-K excerpts, with Decider V1.2 leading the pack of decision models.

[Vals AI @ValsAI]

When will robotics be solved?

We put frontier models in real-time MuJoCo sims of a robot arm and a humanoid, where they only receive camera feeds and sensor readings, and respond with motor commands. We tested them on 24 tasks built around spatial reasoning and reaction time.

[Vals AI @ValsAI]

We tested Mercury Decide, a new model from @_inception_ai, on the same benchmark we used for TypeSafe's Jev.

It matched frontier accuracy on claim verification at the lowest cost we have measured so far.

[Vals AI @ValsAI]

Can a model that doesn’t generate text compete with frontier LLMs?

We independently tested TypeSafe’s Jev against 11 other models. On 400 claim-verification questions, it matched GPT-6 Astra’s 97.5% accuracy at ~1/500th the cost.

[Vals AI @ValsAI]

For decades, researchers have sought materials that sort electrons by spin while their magnetism cancels.

In 3 days, 90+ Opus 5.5 agents helped us uncover two room-temperature magnetic semiconductor candidates in simulations: YBaMnFeO₅ and KV[Cr(CN)₆].

KV[Cr(CN)₆] was synthesized back in 1999. Its predicted ability to sort electrons by spin appears to have been hiding in plain sight for 27 years.

[Rayan Krishnan @RayanKrishnan]

https://t.co/5oPSEvEnNl

[Connor Frank @conjfrnk]

The increase in the pace of progress right now is hard to overstate

[Connor Frank @conjfrnk]

The pace of progress right now is hard to overstate.

[Vals AI @ValsAI]

Existing coding benchmarks stop at the first working version. We are releasing Vibe Code Bench 1-100 today, to measure what comes next.

This benchmark asks if models can handle a large number of modifications to a product, without breaking what already works.

[Vals AI @ValsAI]

It's encouraging to see industry leaders call for rigorous, independent evaluations as AI capabilities advance. Investment in AI systems has far outpaced our ability to test and understand them. We recognized this gap and started Vals to close it. A market-based approach to evaluation has been called for across the industry, and it is the best mechanism for the public to gain confidence in AI as it becomes more capable.

Vals has built the infrastructure and novel methodologies that independent evaluation requires. That work has produced the first public RSI benchmark, an index of the economic impact of frontier models, and evaluations of model behavior across social safety nets, cyber risk, and mental health.

Our work has been adopted across the industry. Vals benchmarks already appear in the model cards of every major foundation model lab. We regularly brief the U.S. government, national security agencies, and allied governments on what our evaluations show. Enterprises use our findings to understand the risks and capabilities before adoption.

Vals is committed to expanding that work, and the access that makes it possible, as models become more capable. A diverse ecosystem of evaluators and approaches is the surest way to ensure these complex systems are understood. We welcome every serious effort to build it.

[Rayan Krishnan @RayanKrishnan]

Open source and self reported benchmarks are done. The industry desperately needs a scrupulous third party evaluator before AI sees an Enron, ‘08 crisis or worse.

[Vals AI @ValsAI]

Using Fable 5.1, we’ve elicited a decoding of a cipher that has been unsolved for 373 year, and listed as #28 on Klaus Schmeh’s (the most-published cryptology author in the world) “The Top 50 unsolved encrypted messages.”

The solution? Almost embarrassingly simple in hindsight.

[Vals AI @ValsAI]

We are thrilled to introduce the first cohort of Vals Fellows!

These five researchers are building evals that tackle some of AI’s most important questions: from model safety to formal mathematical reasoning, as well as real world engineering and security challenges.

Get to know the researchers building the next generation of evals in the thread below.

[Vals AI @ValsAI]

Today we're announcing our $40M Series A at a $400M valuation, led by @a16z , with participation from existing investors @8vc, @pearvc, and @BloombergBeta and new investors @HRTVentures and @nextladder.

Alongside the fundraise, three more announcements:

- Vals Smith: is now generally available. Anyone can create a custom coding benchmark from any GitHub repo with 120 free credits to get started.

- Frontier Risk Benchmarks: We are releasing the RSI Index in collaboration with @CoreWeave and just launched ReverseEngBench, a new cyber benchmark built with Columbia University, Tufts University, UC Berkeley, and UCLA. We are also sharing our initial work in mental health, with more to come across environmental impact, military, and biosecurity.

- Website + Vals Index 2.0: We completely rebuilt the Vals website and have released Vals Index 2.0, with coverage of more of the economy.

Our revenue has already grown 8x compared to all of 2025. Our customer base doubled and the team tripled in 6 months. Our results have been cited in model cards from OpenAI, Anthropic, Google, Meta, and xAI.

The AI economy runs on self-reported grades. When a model ships, the scores come from the company that built it. No other trillion-dollar industry works this way. Finance has ratings agencies. Medicine has the FDA. AI has vibes and vendor benchmarks. We built Vals to be the independent evaluation layer the industry is missing.

Check out our new website and try Vals Smith!