Boxuan Li

I work on Data & AI Infra

[Ivan Bercovich @neversupervised]

https://t.co/jUA2S777zV

[Ivan Bercovich @neversupervised]

I want to share a new dataset of 331 reward-hackable environments. These are real environments used in Terminal Bench and adjacent benchmarks. I first got interested in this because, as a reviewer of Terminal Bench, I noticed a lot of our tasks were hackable. I also noticed that many contributors to the benchmark do so because it provides credibility when selling environments to labs. Hence, TBench tasks are, in my opinion, held to a higher quality standard than those being used today for RL. No one is spending hours manually reviewing the $1B in tasks being purchased by major labs. As far as I understand, while everyone knows environments are hackable, nobody has released hundreds of "realistic" environments. (link in comment)

[Ivan Bercovich @neversupervised]

https://t.co/buOPNmWeBe

[Charlie Ruan @charlie_ruan]

Releasing the official SkyRL + Harbor integration: a standardized way to train terminal-use agents with RL.

From the creators of Terminal-Bench, Harbor is a widely adopted framework for evaluating terminal-use agents on any task expressible as a Dockerfile + instruction + test script.

This integration extends it: the same tasks you evaluate on, you can now RL-train on.

Blog: https://t.co/yDyId02UfH
🧵

[Ryan Marten @ryan_marten]

Been trying to keep up with how people are using Harbor. Breaking out my random gdoc into a public collection. Benchmarks, RL on harbor, training datasets, QA tools.

Really impressed with the community so far and excited to see what else gets made (add your project in a PR!).

https://t.co/JTvqt9tZez

[Mike A. Merrill @Mike_A_Merrill]

The Terminal-Bench paper is here! Read it to learn where frontier models still fail and the secrets of how we sourced hundreds of high quality environments from our open source community. 🧵

[Negin Raoof @NeginRaoof_]

How can we make a better TerminalBench agent?
Today, we are announcing the OpenThoughts-Agent project.
OpenThoughts-Agent v1 is the first TerminalBench agent trained on fully open curated SFT and RL environments.
OpenThinker-Agent-v1 is the strongest model of its size on TerminalBench, and sets a new bar on our newly released OpenThoughts-TB-Dev benchmark. (1/n)

[Alex Shaw @alexgshaw]

Super stoked about cline-bench! This is exactly the kind of thing we built Harbor for 🙏

[Alex Shaw @alexgshaw]

Today, we’re announcing the next chapter of Terminal-Bench with two releases:

1. Harbor, a new package for running sandboxed agent rollouts at scale
2. Terminal-Bench 2.0, a harder version of Terminal-Bench with increased verification

[Graham Neubig @gneubig]

We have a new state-of-the-art result on TheAgentCompany from Shanghai AI lab: MUSE + Gemini 2.5, solving 41.1% of the real-world inspired tasks.

The new method is based on "learning on the job", a memory-based method.

[Graham Neubig @gneubig]

We just updated the leaderboard of TheAgentCompany, a benchmark of tasks like real-world work.

- In December 2024, 24% of the tasks could be solved
- In June 2025, 33% of the tasks could be solved

I'm interested to see when we'll be at 50%.

[Aditya Soni @Aditya_Soni_8]

Can we design AI Agents that achieve generalizability across diverse task domains?

Our new paper introduces OpenHands-Versa, a generalist agent with strong performance on three challenging agent benchmarks, ranking #1 on SWE-Bench Multimodal and The Agent Company leaderboards 🚀

[Boxuan Li @LiBoxuan91538]

How good are Claude Sonnet 3.7 and Gemini 2.5 Pro? We just evaluated both models, and Gemini 2.5 Pro now achieves 30.3% at #1, followed by Sonnet 3.7 with 26.3% at #2. Both surpassed the previous SOTA Sonnet 3.5 which got 24% tasks resolved.

[Graham Neubig @gneubig]

How far are we from having competent AI co-workers that can perform tasks as varied as software development, project management, administration, and data science?

In our new paper, we introduce TheAgentCompany, a benchmark for AI agents on consequential real-world tasks.

[Boxuan Li @LiBoxuan91538]

@yebuseth No. TheAgentCompany has no official account.

[Boxuan Li @LiBoxuan91538]

How far are we from having competent AI co-workers? A case study
#ai #llm #agents #ai-benchmark
https://t.co/Fru0CPR1k2

[Boxuan Li @LiBoxuan91538]

@0xRiver8 @YufanSong98 @TomerLevinboim @gneubig @frankxu2004 @_AgentCompany That account is NOT an official account of TheAgentCompany.

[Boxuan Li @LiBoxuan91538]

@YufanSong98 @TomerLevinboim @gneubig @frankxu2004 Caveat: OpenHands’ system prompts for LLMs that support function calling and those that don’t, are, as one could imagine, different.

[Boxuan Li @LiBoxuan91538]

@TomerLevinboim @YufanSong98 @gneubig @frankxu2004 OpenHands’ system prompt might have been engineered for Claude unintentionally, since it was the primary LLM developers of OpenHands used for dev and benchmarking.

[Boxuan Li @LiBoxuan91538]

@TomerLevinboim @YufanSong98 @gneubig @frankxu2004 The prompt we used was an one liner: https://t.co/OGY1Co1YQJ

[Boxuan Li @LiBoxuan91538]

@TomerLevinboim @YufanSong98 @gneubig @frankxu2004 For example, all LLMs had trouble closing the popup window on ownCloud. Claude (and maybe others), sometimes could figure out that using “escape” keystroke would close the popup. If we add this trick to the system prompt I believe the score would be boosted.

[Boxuan Li @LiBoxuan91538]

@TomerLevinboim @YufanSong98 @gneubig @frankxu2004 During our evaluation, we didn’t add any specific prompts that could potentially boost the score. I believe the score could be boosted if you inspect all the failed trajectories and add specific prompts to work around common errors.