Harsh Raj

[Razvan Dumitru @RazvanDuu]

Glad to have helped @harshraj1728 with this project at Scale! A super useful taxonomy for anyone trying to debug AI agents in production.

[Vipul Gupta @vipul_1011]

Highest alpha move as an industry researcher: hire a great intern.

Kudos to @harshraj1728 for taking this from an idea to a finished paper in under 2 months.

[Harsh Raj @harshraj1728]

Very cool to see our work featured by @omarsar0! 🫡

Glad the taxonomy is resonating with people building agents in practice. Would love to hear feedback and see where people take it from here!

[elvis @omarsar0]

// Model or Harness //

Great paper if you are building with agents in production.

(bookmark it)

It organizes 41 agent failure modes by the interaction they originate in. Each mode gets assigned to an edge between two components (model, harness, user, tools, memory, environment) plus a fault side naming where the repair belongs.

Attributing failures to edges rather than to single components matches how agent bugs actually present. Most of them live in the seam between a model and its scaffolding.

The schema also holds up under automation. Across four frontier models, the strongest judge reaches Cohen's kappa of 0.76 against human category labels, so the labeling can run continuously over production traces instead of one postmortem at a time.

Harness engineering became the main lever for agent builders this year without a shared vocabulary for where a harness bug ends and a model bug begins.

Paper: https://t.co/3U3Nzr44Qd

Track more trending AI papers in our academy: https://t.co/1e8RZKs4uX

[Vipul Gupta @vipul_1011]

Appreciate the shoutout from @omarsar0 🫡

Glad people are finding it useful.

Really looking forward to seeing what people build with this. Would love to hear any feedback to improve it further.

Amazing project led by @harshraj1728

[elvis @omarsar0]

// Model or Harness //

Great paper if you are building with agents in production.

(bookmark it)

It organizes 41 agent failure modes by the interaction they originate in. Each mode gets assigned to an edge between two components (model, harness, user, tools, memory, environment) plus a fault side naming where the repair belongs.

Attributing failures to edges rather than to single components matches how agent bugs actually present. Most of them live in the seam between a model and its scaffolding.

The schema also holds up under automation. Across four frontier models, the strongest judge reaches Cohen's kappa of 0.76 against human category labels, so the labeling can run continuously over production traces instead of one postmortem at a time.

Harness engineering became the main lever for agent builders this year without a shared vocabulary for where a harness bug ends and a model bug begins.

Paper: https://t.co/3U3Nzr44Qd

Track more trending AI papers in our academy: https://t.co/1e8RZKs4uX

[Harsh Raj @harshraj1728]

7/7 We’ve already started using this taxonomy as a lens for studying frontier-agent failures and asking a more interesting question: What can the structure of these failures tell us about current agents—and which interventions actually help? Some fun results coming soon. :)

[Harsh Raj @harshraj1728]

1/7 Excited to share our new paper from my internship at @ScaleAILabs:
Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
https://t.co/ahRrgvOov5 🧵

[Harsh Raj @harshraj1728]

6/7 I’m very grateful for the opportunity to work on this at @ScaleAILabs, and especially thankful to the amazing people I had the chance to learn from and collaborate with
@vipul_1011 @nas_mahmoud_ @RazvanDuu @DarvinIsTrying @aakashsabharwal @yunzhing

And there’s more coming👀

[Harsh Raj @harshraj1728]

As a summer intern at @ScaleAILabs, I've been building an audit framework for AI benchmarks.

Auditing @harborframework's Harbor-Index, we found a few benchmark issues that the team quickly resolved. Excited to contribute to a fast-moving, collaborative open-source ecosystem!

[Kevin Li @kevin_x_li]

Benchmarks are software, and like software, they require continuous maintenance.

Thanks to feedback from community members like @Harsh15810401 and Aaron Schneider, we've already shipped 3 new releases of Harbor-Index in the 2 weeks since launch.

A major highlight: Aaron discovered a valid alternative solution for an ARC-AGI task by analyzing our published agent trajectories! We've confirmed his findings and updated the verifier to account for this solution in our upcoming v1.4 release: https://t.co/zTfDpiUscT

PRs and issues are always welcome. Let's keep improving Harbor-Index in the open, together!

[Kevin Li @kevin_x_li]

Benchmarks are software, and like software, they require continuous maintenance.

Thanks to feedback from community members like @Harsh15810401 and Aaron Schneider, we've already shipped 3 new releases of Harbor-Index in the 2 weeks since launch.

A major highlight: Aaron discovered a valid alternative solution for an ARC-AGI task by analyzing our published agent trajectories! We've confirmed his findings and updated the verifier to account for this solution in our upcoming v1.4 release: https://t.co/zTfDpiUscT

PRs and issues are always welcome. Let's keep improving Harbor-Index in the open, together!

[Ryan Marten @ryan_marten]

We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work.

Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort.

Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%

[Vipul Gupta @vipul_1011]

Benchmarks tell us where a model fails. They rarely tell us why and what to train on next.

In our new @ScaleAILabs paper, we tackle this.

CRAFT turns rubric criteria into a hierarchical capability tree, identifies model weaknesses, and generates targeted fine-tuning data.🧵

[Harsh Raj @harshraj1728]

@madiator Congratulations 🎉

[Richard Zhuang @RichardZ412]

We openly share our FULL stack:

Paper: https://t.co/h6lgENPkhx
Model: https://t.co/aZJPpMGZvS
Data: https://t.co/Kiko3fqUXT (main dataset), https://t.co/xpOPHPKner (full traces used during our experiments), https://t.co/9dl1ryKeGO (full tasks)
Full artifacts: https://t.co/ge7GDH555y
Code: https://t.co/UdKZXwj67F

Read on for the highlights👇(2/n)

[Richard Zhuang @RichardZ412]

🧑‍🤝‍🧑Of course, none of this happens without the support of my wonderful collaborators: @NeginRaoof_ @etash_guha @marnezhurina @JJitsev @AlexGDimakis @FeuerBenjamin @lschmidt3 @ryan_marten @charlie_ruan @alexgshaw @hbXNov @ekellbuch @artemg314 @HeckelReinhard @xdotli @Shlok_Natarajan @Harsh15810401 @nick11roberts @ethanlshen @nishadsinghi @michaelsiux @suvarna_ashima @realleonlc @kobe0938 @StevenDillmann @LiBoxuan91538 @LinShi592021 @siyan_zhao @WanjiaZhao1203 @StevenDillmann and many more contributors helping behind every ablation study!! (7/7)

[Victor May @MrColeslaw972]

Grateful to @TmlrCert for featuring our work on MixtureVitae!