Yes, an on-demand restore would be a meaningful improvement. In our collected coding sessions, when a pruned tool would have helped with the current task, what we observed was that the agent most often still reached its goal with the core tools we deliberately protect from pruning (the shell, file reading and editing, and search). So the loss was more on efficiency, typically a few extra turns, rather than task success.
Do you know that past experience can substantially reduce the token cost of your coding agent?
Introducing our recent work TokenEconomizer: 18.4–30.9% lower cache-aware cost on SWE-Bench, with comparable task success. We deployed it as a plugin for mainstream coding agents - feel free to try it out and let us know your feedback!
Plugin: https://t.co/LkbfVdqPfR
Paper: https://t.co/K6MMDSSjLO
Project page: https://t.co/c5WJstLdIz
(1/3)
TokenEconomizer learns from past sessions in your codebase to:
• Reuse practical lessons as micro-skills
• Prune irrelevant tools
• Redirect agents stuck in failure loops
It works externally, without changing the model or agent harness.
(2/3)
We are really excited to share our recent work, BenchEvolver, a framework for transforming saturated benchmark tasks into challenging ones for frontier models.
We hope our work encourages the community to rethink benchmarks not as fixed datasets, but as adaptive evaluation systems. For many verifiable domains, automated and reproducible pipelines can continuously generate, validate, and calibrate new tasks as frontier models improve.
Our results are promising: some beginner coding tasks, such as LeetCode Easy problems, can be evolved into extremely challenging problems comparable to the hardest tasks in top competitive programming contests, with estimated Codeforces ratings around 3000. Beyond evaluation, these synthesized tasks also enable self-improvement of frontier models.
Paper: https://t.co/sXDrlIkuFv
Project Page: https://t.co/vckZJsXJMn
Static benchmarks are dying — they tend to get saturated quickly.
Evaluation and training data should co-evolve with frontier models.
We released BenchEvolver — a framework that automatically evolves saturated problems into harder, verified tasks for evaluating frontier models, which can also serve as useful self-improvement signals for RL.
New work from UC Berkeley @berkeley_ai@BerkeleyRDI@BerkeleySky
Project Page: https://t.co/PL1KpGyd87
Paper: https://t.co/gBQOXrZbAV