Bhuvan Koduru

I try :)
Safety | Agentic Evals | Harnesses | AI safety research @SimularAI | NLP @SCSatCMU |

[arXiv Bangers @arXivBangers]

Evolving Agent Interfaces

Letting an agent autonomously evolve its own interfaces from task experience improved Qwen3.8-27B's success on OSWorld V2 from 5.6% to 21.3%, showing interface self-design can dramatically boost computer-use capabilities.

https://t.co/8AQOmBg6Gt

[Bhuvan Koduru @Bhuvan_Koduru]

@chalo2000 🎉👏

[Xin Eric Wang @xwang_lk]

People long believed in human-like computer use. But software (even terminals) is built for humans, not AI agents. Why not let agents build their own?

Meet 𝗔𝗴𝗲𝗻𝘁 𝗦𝟰, an agent scientist evolving its own interfaces, letting an open 27B model beats GPT-5.5 & Opus 4.8.

With its evolved interfaces, the same model improves 𝟯.𝟴× and costs <𝟱% as much per task as either frontier model.

Agents today read screenshots and act through click, scroll, type, maybe a shell. So an agent fills a spreadsheet cell by cell when one line of code would do, or prints two thousand lines of a file to find one function.

Every new app or model release means a human reads trajectories, diagnoses failures, and hand-writes new primitives. Skill-distillation methods can reorder what the interface allows, but they can’t extend it. Agents compensate with more test-time compute instead of a better interface.

𝗠𝗼𝗱𝗲𝗹𝘀 𝗶𝗺𝗽𝗿𝗼𝘃𝗲; 𝗶𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲𝘀 𝘀𝘁𝗮𝘆 𝗳𝗿𝗼𝘇𝗲𝗻.

“We want AI agents that can discover like we can, not which contain what we have discovered.”
— Rich Sutton, The Bitter Lesson

Hand-designed interfaces are what we have discovered. Agents can read their own trajectories, spot what’s missing, and write and test code. 𝗜𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲 𝗱𝗲𝘀𝗶𝗴𝗻 𝘀𝗵𝗼𝘂𝗹𝗱 𝗯𝗲 𝗮𝗻 𝗮𝗴𝗲𝗻𝘁 𝗰𝗮𝗽𝗮𝗯𝗶𝗹𝗶𝘁𝘆.

That’s what Agent S4 does.

Agent S4 makes a 𝗿𝗲𝗽𝗼𝘀𝗶𝘁𝗼𝗿𝘆 𝗼𝗳 𝗶𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲𝘀 𝗮𝗻𝗱 𝘂𝘀𝗮𝗴𝗲 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 the unit of evolution. A task agent attempts synthetic tasks with no success labels or verifiers. A meta-agent studies the trajectories, forms hypotheses about which actions, observations, or knowledge would help, and builds descendants through mutation or crossover. Validation tasks the meta-agent never sees decide which repositories survive.

The model and harness stay fixed, and the meta-agent 𝗻𝗲𝘃𝗲𝗿 𝘀𝗲𝗲𝘀 𝗮 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝘁𝗮𝘀𝗸. 𝗢𝗻𝗹𝘆 𝘁𝗵𝗲 𝗶𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲 𝗲𝘃𝗼𝗹𝘃𝗲𝘀.

With the evolved repository, Qwen3.8-27B goes from 𝟱.𝟲% to 𝟮𝟭.𝟯% on OSWorld V2, 𝟯.𝟴× the same model with fixed interfaces and a new state of the art for open-source computer-use agents.

It beats GPT-5.5 (13.0%, $25.50/task) and edges past Opus 4.8 (20.6%, $72.40/task) at $𝟭.𝟭𝟬 per task.

On ScienceBoard, the same model goes from 41.4% to 58.6%.

The analysis is the more interesting part. Evolution mostly deepens interfaces the agent already has: 4 new interfaces and 127 extensions. Give the meta-agent a scratch VM to test its code, and it builds 12 new interfaces and 253 extensions, reaching music scores, audio projects, and 3D scenes. The best repository came from crossing two earlier branches.

In one case study, the seed agent hand-builds a browser client mid-task and debugs through failure after failure; the evolved agent calls page_text and reaches the same message at tool call 𝟭𝟭 instead of 𝟯𝟳.

Human-like computer use was a starting point, not the destination. The interface an agent acts through is a second axis of improvement, beyond better models and harnesses, and agents can now design it themselves. 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 𝗮𝗴𝗲𝗻𝘁 𝗶𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲𝘀 𝗶𝘀 𝗮 𝗻𝗲𝘄 𝗱𝗶𝗺𝗲𝗻𝘀𝗶𝗼𝗻 𝗼𝗳 𝗮𝗴𝗲𝗻𝘁 𝘀𝗲𝗹𝗳-𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁.

[DAIR.AI @dair_ai]

Banger paper on evolving agent interfaces.

Pay close attention to this new trend if you build with agents.

Like optimizing harnesses, it's remarkable what you can achieve with agents that have access to better interfaces.

Computer-use agents usually act through interfaces that people design by hand, so an agent may type a spreadsheet cell by cell when one line of code would do. Simular's Agent S4 has a meta-agent that builds those interfaces instead. It reads the task agent's trajectories, with no success labels, and writes new tools and notes on when to use them.

With the evolved interfaces, an open 27B model reaches 21.3% on OSWorld V2, 3.8 times that of the same model with hand-built interfaces.

That is above GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their cost.

Population-based selection keeps the interface repositories that do best on validation tasks the meta-agent never sees, so the changes carry over to new tasks.

On ScienceBoard, the evolved interfaces raise success from 41.4% to 58.6% with the same model. The code is open source.

https://t.co/GugEMxwuzg

Chat with Paper: https://t.co/5stmPhni6v

[Bhuvan Koduru @Bhuvan_Koduru]

@niloofar_mire @humansand So cool! 🤯

[Saaket Agashe @ COLM @saa1605]

Computer-use agents are limited by the interfaces humans design, typing a spreadsheet cell by cell when one line of code would do.

Can we let them discover their own interfaces through evolutionary search? Come to our poster and find out.

📍Continental Ballroom 6 at 2.30 PM at the Lifelong Agent (LLA) Workshop.

#COLM2026 #LLA, 2:30pm
https://t.co/wNzN2lefbx

[Bhuvan Koduru @Bhuvan_Koduru]

The model says no in chat. The agent does it anyway.
Here's what I worked on at @SimularAI this summer, where we found refusal drops sharply once models get a mouse and keyboard, and steering that fixes chat refusal barely carries over to the agent.

[Simular @SimularAI]

Our new agent safety research: The model says no in chat. The agent does it anyway.

A model that says no in chat might not exercise the same harmful refusal when it's given a mouse and keyboard in computer use.

It might tell you no but still goes ahead to do harmful things.

How did @sai_borg do?

Running the OS-Harm benchmark, Sai declined 69% of the tasks on its first turn, before taking any action on the computer.

Then we added a short safety section to Sai's instructions: its default agent mode declined 82% of the misuse tasks on its first turn, against 71% for the same build without it, three runs each.

[Bhuvan Koduru @Bhuvan_Koduru]

@anshuman1 👋

[Bhuvan Koduru @Bhuvan_Koduru]

Excited to be at #COLM2026 this week!
I’ll be presenting Heard but Not Heeded, where we study how audio-language models encode paralinguistic information, yet often fail to actually use it at the output. 📄 arXiv:2609.00727

Also excited that CUES: Computer Use with Evolving Software was accepted to the Workshop on Lifelong Agents! We explore agents that build task-specific interfaces in code and evolve reusable software from experience.

I’m currently looking for full-time roles in agents, agent harnesses, AI safety & evals. If you’re working on these problems and will be at COLM, I’d love to meet!
#AI #AIAgents #ML