Andrew Wang

MIT

[terminalbench @terminalbench]

Introducing Terminal-Bench Challenges!

A new capability has emerged at the frontier: agents completing large-scale projects autonomously. To test this capability, we felt another flavor of benchmark was needed.

Terminal-Bench Challenges are long-horizon, token-intensive, single-task benchmarks. Today we are releasing our first 3 challenges.

[Andrew Wang @andr3w_wang]

We’re looking for more challenge ideas! Especially tasks that have interesting verification methods: adversarial testing, fuzzing with LMs, replicating closed source software, etc.

And tasks in specialized domains: Game emulation, FPGAs, GPU driver programming, MoQ, etc.

https://t.co/VG2jd11YXf

[Andrew Wang @andr3w_wang]

Can agents build complete projects that deliver real value? We’re launching Terminal Bench Challenges: 3 unsolved tasks which could make a real impact on the open source community if solved.

These tasks provide a testing ground for optimizations both on the model and harness level on our continuous leaderboard for each task.

[Andrew Wang @andr3w_wang]

Challenges are tasks that are not only large in size, but also difficult because of the technical ability needed to solve them.

I'm excited to see how these tasks drive harness optimizations, gearing them toward long horizon work. Even more so, I'm excited to see the end artifacts they come up with!