jake

AI research. Mercor code team. UC Berkeley 2025

[jake @jake_cole1]

Had a blast building this!!! Paper coming soon 👀

[Mercor @mercor]

Agents can write code. Can they finish real engineering work?

Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design.

Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end.

OWE-bench top 3 leaderboard:

1. Fable 5.2 (54.6%)
2. GPT-6 Astra (44.2%)
3. Claude Opus 5.5 at 41.6%.

Explore the benchmark website: https://t.co/oT0qna1dO0

[Vikram @Vikramaditya_V1]

On OWE-Bench's drawing-to-CAD task, Gemini 4 Argon wrote the best program of 14 models on the 30 drawings it developed on. On drawings it had never seen, it came seventh.

Good that the main leaderboard scores only the held-out cases.

[Alano @Alanogrok]

Freezing the artifact before hidden-test scoring is a meaningful boundary. Long-horizon benchmarks also need per-task failures and resource budgets, since a normalized average is not the fraction of real engineering jobs completed.

[jake @jake_cole1]

@harborframework Harbor is goated 🐐

[jake @jake_cole1]

@dwkim601 @mercor You can filter by domain and look at each tasks scores and output on https://t.co/BvM1rIhpqx

[Edward Hu @edwardjhu]

On our new long-horizon engineering benchmark, Astra and Fable 5.2 designed an electric aircraft in 8 hours with 80M tokens!

https://t.co/r1q3QsXXKV

Well done @jake_cole1 and @bertievidgen!

[jake @jake_cole1]

@JayRS_Tweets @mercor Yes, they even get a monthly happy hour budget :)

[Raúl Nogales 🇪🇸🇺🇸 @raulnogales]

@mercor @jake_cole1 Short coding tasks flatter every model. Finishing a full project is different, it tests planning, memory and recovery from mistakes. A benchmark built around that gap will change how teams judge real engineering ability.

[Abhi Kottamasu @abhikottamasu]

OWE-Bench is the best measure of real engineering capability.

Congrats to @jake_cole1 and the rest of the Code Data team!

If you’re excited by cutting edge work like this, we’re hiring! DMs open.

[jake @jake_cole1]

@mapsmoods @mercor No, you

[jake @jake_cole1]

@justice_note @mercor The scoring is continuous— this is includes partial credit

[jake @jake_cole1]

@AarslanEmre @mercor It’s scored via 34 held-out, simulated autopilot flights that test a variety of conditions

[Jason Wei @_jasonwei]

This is a pretty cool benchmark of actual long-horizon tasks, with Fable 5.2 spending more than 200M tokens to try to solve each problem!

[Mercor @mercor]

Agents can write code. Can they finish real engineering work?

Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design.

Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end.

OWE-bench top 3 leaderboard:

1. Fable 5.2 (54.6%)
2. GPT-6 Astra (44.2%)
3. Claude Opus 5.5 at 41.6%.

Explore the benchmark website: https://t.co/oT0qna1dO0

[Jason Wei @_jasonwei]

The fall of “what you can do” and the rise of “who you know”

Traditionally, many of us believe in a world where how hard you work is more important than who you know. I used to subscribe to this idea as both a scientist, who believed that facts about our world are agnostic to who says them, and as an American, who grew up believing that success was a function of how hard you worked, not who you knew.

However I am now feeling a substantial change in AI, specifically in the past five years, where who you know is becoming more important, and what you can do is becoming less important. I am obviously not saying that technical skills don’t matter, or that you cannot be successful without being well connected. Technical skills are still important and make or break your career. But I am noticing a trend. To assign arbitrary numbers, I would say that five years ago, success was 75% your skills and 25% who you knew. Now, it is maybe 55% skills and 45% who you know.

There are three technical reasons for this trend. The first is the commoditization of intelligence—coding agents have basically democratized research execution. In the past you could outwork competitors by staying up late to debug code or babysitting your experiments with higher frequency. Now, many research tasks can be completed by anyone who knows what to prompt. Research taste obviously still matters, but the playing field is definitely more equal. As technical execution becomes less of a differentiator, other advantages naturally become more important.

The second reason for this trend is that with the success of scaling language models, it is simply a fact that being resource rich makes you so much more likely to succeed. Ten years ago, an academic lab with eight GPUs had a real chance of doing great research. Now, a single RL run can require hundreds (if not thousands) of GPUs, as well as substantial data and infra. So now, two equally capable researchers can have very different outcomes depending what resources they have. And access to these resources is not a function of technical skill alone. It also depends on reputation, trust built with decision-makers, and professional relationships. These things have always mattered, but as the resource requirements of frontier AI have grown, so too has their importance.

Third, when companies stopped publishing research and moved to secretive, closed-source models, AI became a field of poor observability. In domains of perfect observability like chess or soccer, it is very easy to tell who is doing well and understand their contributions. But now in AI we have fewer public mechanisms for attributing credit, making it much harder for outsiders to know who did what for model launches. How does this get resolved? When it's difficult to independently verify someone's technical contributions, people increasingly rely on indirect signals such as reputation, personal endorsements, and who will vouch for them. Consequently, who you know plays a much bigger role in hiring, recruiting, and fundraising.

We often talk about how to improve technical skills, but nobody really talks about how to get better at "who you know". One observation is that many well-connected people mix their work and personal lives. The traditional wisdom is to keep a firm boundary between the two, which in the past was a very reasonable policy. But the returns of being social have gone up, which means that anyone still holding to that boundary is paying a much steeper price for it than they would have before.

Sam Altman is the prime example of a well-connected person who mixes work and personal. There is so much to learn from Sam. For example, back when OpenAI was smaller, Sam invited almost all the employees to his house for social hours. He hosts poker nights and dinners at his house for recruiting purposes. Sam has a reputation of being generous with his time and treating his friends well and I think that has yielded a disproportionate return for him. Obviously not all interpersonal relationships at OpenAI went perfectly but I think a large part of OpenAI’s success is from Sam’s connections.

The interesting thing about this is that intentionally trying hard at getting to know people is probably the suboptimal strategy. I don't think Sam made his connections because he knew they would pay off in his career, but rather because he is a generous person who is innately curious about people and technology. Perhaps the best strategy is to be your best and truest self, which is easier said than done, especially in a changing technological world.

While these technical trends have made the field less dependent on technical merit, I don’t think this is necessarily a bad thing, and it is not the first time this has happened in our field. For example, abstractions introduced by Python also made coding less technical and enabled more people to code. And with some skillsets, there is more opportunity now than ever. For instance, people without programming backgrounds can now create great technology, with product or research taste as more salient levers. So while what we can do is commoditized, what we choose to do, and who we do it with will matter more than ever.

Thanks Edward Hu for feedback on this essay.

[jake @jake_cole1]

Do you think you think there are neurons attributed to mental illnesses? Are there schizophrenia neurons?

[lyra @_lyraaaa_]

if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1
now you have evil qwen

recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input

[lyra @_lyraaaa_]

if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1
now you have evil qwen

recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input