Abhi Kottamasu

Head of Code Data, Applied Engineering @mercor

[Jason Wei @_jasonwei]

This is a pretty cool benchmark of actual long-horizon tasks, with Fable 5.2 spending more than 200M tokens to try to solve each problem!

[Abhi Kottamasu @abhikottamasu]

OWE-Bench is the best measure of real engineering capability.

Congrats to @jake_cole1 and the rest of the Code Data team!

If you’re excited by cutting edge work like this, we’re hiring! DMs open.

[Mercor @mercor]

Agents can write code. Can they finish real engineering work?

Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design.

Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end.

OWE-bench top 3 leaderboard:

1. Fable 5.2 (54.6%)
2. GPT-6 Astra (44.2%)
3. Claude Opus 5.5 at 41.6%.

Explore the benchmark website: https://t.co/oT0qna1dO0

[Brendan (can/do) @BrendanFoody]

We raised our $750 million Series D at a $20 billion valuation from @benchmark, @generalcatalyst, @nvidia, @salesforce, and @zoom.

@mercor scaled from $1M to $3B in run-rate revenue in 27 months, the fastest in history to $3B.

For 2 years, we've organized human intelligence to build evals and training data for every frontier lab.

The labs are the leading indicator for AI startups, then the Fortune 500.

Our growth has accelerated over the past 6 months as enterprises start evaluating and training their own models. For many, owning their intelligence is existential.

[Cognition @cognition]

Founded by three 22-year-olds in 2023, @mercor crossed $2B ARR in June, only 4 months after hitting $1B.

Today, millions of experts on the platform bring their knowledge of medicine, law, engineering, and beyond to frontier AI - helping models take on increasingly complex and consequential work in the real world.

Mercor's engineers move at extraordinary speed.

They give Devin long-running tasks before bed, during meetings, or on their way to the gym. They come back to tested features and PRs ready to merge.

[Kevin Cherian @cheriankevin]

Nobody ever wanted a ticket. They want their problem solved.

Today @HajarnisSanjeet I are beyond excited to announce our $4M seed round for @decimalcx, co-led by @khoslaventures and @kearnyjackson to bring engineering-level understanding to every customer interaction.

Since inception, we’ve partnered with some of the best AI-native companies: @meetgranola, Omnea, Resilinc, @tealium, @GoBuildOps, @Lucidworks and a Fortune 5 company to help their support and engineering teams resolve customer problems.

[Mercor @mercor]

Grok 4.5 from @spacexai places #2 on the APEX-SWE leaderboard at 51.2% Pass@1 (±6.0), behind Fable 5 (65.5% ±6.2) on our benchmark for real-world software engineering work.

It leads Integration (65.0% Pass@1) and places #2 in Observability (37.3% Pass@1), covering multi-step build tasks and diagnosis/debugging respectively. The Integration lead maps directly to the agentic workflows Grok 4.5 was built for: multi-step coding tasks run in collaboration with Cursor.

Grok models have improved 30.2 pp in a year on this benchmark: Grok 4 (21.0% Pass@1) to Grok 4.5 (51.2% Pass@1).

Congratulations to the xAI and Cursor teams.

[Abhi Kottamasu @abhikottamasu]

Congrats @spacexai! 🚀

This is a huge step up from Grok 4.5.

Great to see APEX-SWE on the model card too.

[SpaceX Super Intelligence @SpaceXSI]

Introducing Grok 4.6.

It delivers frontier intelligence and is a significant improvement over Grok 4.5 at the same price.

[Abhi Kottamasu @abhikottamasu]

Agreed! SWE Marathon 1.0 had a great selection of product-clones. Glad to have built on top of this research.

Congrats @rishi_desai2 and @abundant_labs on the launch of SWE Marathon v1.1!

Excited to collaborate in the future!

[Rishi Desai @rishi_desai2]

Small correction: SWE-Marathon does include four full-stack product-clone tasks (Slack, Excel, S3, Mastodon) evaluated end-to-end with CUA-based verifiers.

Will Mercor’s tasks be released? I’d be happy to incorporate them into SWE-Marathon v1.2!

[Elon Musk @elonmusk]

Grok places second after Fable on real-world software engineering

[Ben Pan @ybenpan]

At Cognition, we care deeply about the end-to-end software engineering experience. That’s why we partnered with Mercor to build APEX-SWE, a benchmark that reflects real-world engineering tasks like observability and integration.

When we evaluated SWE-1.6 Preview on APEX-SWE, it scored highly among frontier models, despite not being trained specifically for this benchmark. We see this as a signal of real-world generalization, and a testament to the progress our research team is making.

[Silas Alberti @silasalberti]

software engineering != coding. we collaborated with Mercor on APEX-SWE, a benchmark that includes interacting with external tooling & MCPs like telemetry, Slack, cloud deployments, help desk software.

it was also surprising to see that SWE-1.6 Preview performed quite well on this benchmark even though we haven't trained on this distribution yet

[Mercor @mercor]

.@cognition reviewed a subset of tasks, pressure-testing how real production systems actually break and get fixed.

They see firsthand that real engineering is not just writing functions, it’s provisioning infrastructure, integrating services, and debugging live systems.

[Abhi Kottamasu @abhikottamasu]

Research Paper: https://t.co/F5YniMJRVj
Leaderboard: https://t.co/JTMSkquFvo
Hugging Face: https://t.co/PPYmeZ7eON
Harness: https://t.co/nzFExhrb2z

[Abhi Kottamasu @abhikottamasu]

Excited to launch APEX-SWE, a new benchmark for real-world software engineering!

Huge shoutout to the team at Cognition, this was a deeply collaborative effort, and together we’ve set a new standard for measuring real software engineering work.

[adarsh @adarsh_exe]

Traditional coding benchmarks do not reflect how software is actually built and maintained.

That's why we built a new benchmark, APEX-SWE, in partnership with @cognition. It measures whether AI models can perform complex, real-world software engineering work to ship systems that work and debug them when they don't.

@OpenAI GPT 5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1.

[Cognition @cognition]

We've collaborated with @mercor on APEX-SWE, a new benchmark that evaluates AI models on realistic software engineering tasks.

[Mercor @mercor]

Introducing APEX-SWE, in collaboration with @Cognition. They see firsthand that real software engineering is not just writing code anymore. It's deploying systems, integrating with tools and debugging when things break.

On APEX-SWE, every model fails to reliably solve the real production software engineering tasks.

@OpenAI GPT-5.3 Codex (High) tops the leaderboard at 41.5% on Pass@1, followed by @AnthropicAI Opus 4.6 (High) at 40.5%.

Every frontier model fails on nearly 60% of real production tasks.

[Kevin Cherian @cheriankevin]

Not true that you can’t buy this off the shelf.

We’re building exactly this (and more) at https://t.co/8j4OfMb7TU.

@klarnaseb is right - for companies like Klarna (any complex environment/software/hardware product), support isn’t trivial RAG over public docs or a billing refund bot.

Software & hardware customers are sophisticated. They ask complex questions that require real product understanding & investigation - code, logs, data, and context.

Support is strategic infrastructure.

But that doesn’t automatically mean you should build the entire system in-house.

If you’re thinking about this problem, I’d love to chat.

[Abhi Kottamasu @abhikottamasu]

@BrendanFoody Grok 4 is a surprising 🥈

[Abhi Kottamasu @abhikottamasu]

@cryptoshaneajg factsss

[Abhi Kottamasu @abhikottamasu]

@cryptoshaneajg @solcitiesnft @BTDiaz @SidPittie What is this?