@Amit_T18@csaba_kissi Exactly! We wanted to capture that real-world moment when you pick up someone else’s work and have to figure out what still needs doing.
@qhoxie@svpino Good point about follow-through. The full spec helped both models get more of the requested behavior right. The traces show where things can still go wrong between getting a requirement and verifying the patch.
@thesimaximalist@omarsar0 That’s the idea behind Oracle-Spec. We gave the agent the full requirements up front and held the model, tasks, tests, and time budget fixed. Both models passed more checks for new behavior and fully completed the same number of tasks as in CoWork.
@_mahns@omarsar0 The agent starts with a short handoff note and a repo checkout. From there, it uses MCP tools to search the board and docs and message teammates as questions come up.
@lautaroseth@svpino Fair question! GPT-5.6 Terra plays the colleagues. However the environment controls what each teammate knows and can share. The leaderboard scores come from hidden tests run on the submitted patch.
@rohanpaul_ai Thanks, Rohan! We wanted the benchmark to reflect what it takes to ship a change in a real codebase. The new feature has to work, and so does the code around it.
Can a coding agent take over a teammate’s work in progress?
Introducing SWE-CoWork, a benchmark for coding agents working as digital employees on long-horizon software engineering tasks.
Each assignment starts with a short handoff in a simulated workplace modeled on real-world engineering workflows. Agents search task boards and versioned docs, ask teammates for missing details, and revise their code when a reply changes what needs to be built.
Open questions and confirmed requirements can both be lost before the agent hands the work back. SWE-CoWork lets us trace them through the conversation, the patch, and the tests.
101 assignments across 101 real open-source repositories.
26 simulated teammates per workplace.
11 programming languages.
12 application domains.
17 workplace tools.
🌐 Website: https://t.co/vyddiAIGDE
📄 Paper: https://t.co/EPBQ5RjlGy
💻 Code: https://t.co/zU8wPB0Wir
🤗 Hugging Face Daily Papers: https://t.co/tAbAQZ4pKR (If you find it useful, we’d appreciate an upvote!)
@BuilderCulture_ An explicit queue with timed follow-ups would be worth testing. We also think a capable agent should recognize the need for such tracking and maintain it autonomously. That's one of the capabilities we aim to evaluate with SWE-CoWork.
@aaliya_va That’s what we were aiming for! We wanted to test it like joining a real engineering team, where part of the job is figuring out who knows what.
@omarsar0 Agreed. Tracing a failed check back to the handoff helps us see where agents can do better. Here’s to agents that make teamwork a little easier for everyone.
To see where delivery breaks down, we analyzed a set of CoWork runs from Claude Opus 5.5.
We traced 979 failed checks for new behavior back to specific requirements. In 44% of those failures, the agent never received the requirement. In 56%, the agent had received it and the check still failed.
In one run on carapace-bin, the agent received all 25 requirements held by teammates. Still, 22 of 35 checks for new behavior failed. The trace showed incorrect CLI flags and a build failure the agent left unchecked.
If you’re building coding agents, pick a failed check and trace it back through the agent’s conversations, code changes, and tests.
[9/9]
Thank you to all our OSWorld-Science collaborators:
Dingyuan Dai, Heli Qi, Lei Liu, Yinxi Li, Baiding Chen, Zijun Dou;
Qingcheng Zeng (@SteveZeng7), Qi Kang, Oliver Kunyang Sun, Eric Wang, Bo Zhou, Haixin Wang, Yufan Du, Shi Bo, Ruihan Lin, Mengqi Yuan, Dunjie Lu, Steven Dillmann (@StevenDillmann), Yiming Shi, Tina Su, Amy Xin, Minghao Liu, Xi Wang, Xu Huang;
Ge Zhang (@GeZhang86038849), Pengyu Nie, Zhen Yang, Jie Tang (@jietang), Juanzi Li, Weihao Xuan, and Tianyu Liu (@meSuper8).
Full author list and affiliations: https://t.co/XK0B87AGzB
[1/9]
What are the limits of AI agents in scientific research? Can they use professional software as effectively as scientists?
Meet OSWorld-Science: a benchmark and evaluation environment for computer-use agents, with tasks grounded in real-world scientific research needs—from pathology image analysis to physical simulation.
146 tasks.
7 scientific domains.
17 tools.
12 VLMs evaluated.
Tasks are developed through expert proposals and iterative human–AI co-design, with selection guided by scientific value and difficulty.
By combining GUI/CLI interaction, artifact-based evaluation, and trajectory analysis, we examine how agents carry out scientific workflows and where execution breaks down.
TokenWave is glad to be part of this research collaboration.
🌐 Project: https://t.co/XK0B87AGzB
📄 Paper: https://t.co/c4RHtcKPxn
💻 Code: https://t.co/HW28bONUTr
📊 Data: https://t.co/95y8wzLghC
[8/9]
Help expand OSWorld-Science:
• Contribute tasks grounded in real research needs across scientific, clinical, and engineering workflows.
• Bring cross-application and interdisciplinary tasks, including drug-discovery workflows.
• Debug and verify tasks and evaluators.
• Support testing with API access or sandbox resources.
Researchers and collaborators welcome.
Submit a task: https://t.co/tTmQ4po3WD