Top Tweets for #cocoabench

What makes us humans general agents? 🤔 It’s probably not that we’ve mastered every app or UI. New tools appear constantly, yet we become productive quickly. What really transfers are our cognitive abilities: how we perceive, reason, and use memory. 🧠 Introducing CocoaBench, an evaluation framework for general agents with compositional cognitive abilities. 👉 Features 1⃣Complex, realistic tasks🧩 • Human-crafted tasks that are long-horizon, understandable, and span diverse scenarios and domains • Assume only a small set of general tools (browser, terminal, file system; no per-task APIs) • Challenging for existing agent systems. ChatGPT Agent reaches only 44% success rate. • Check out our example tasks on the website (link in reply) and see if you can solve them. They’re fun 🙂 2⃣Covering diverse cognitive abilities 🧠 CocoaBench covers different choices in the following dimensions of cognitive architecture: • Perception: how the agent gathers and preprocesses information from websites, terminal outputs, files, and images • Reasoning : planning, deductive/inductive/abductive inference, and both symbolic & visual reasoning skills • Memory : managing working memory in the long horizon, saving and loading procedural memory (skills), etc. 3⃣CocoaAgent framework🛠️ • Seamless integration with the AIO Sandbox for isolated browser / terminal / file operations • Easy to plug in any models or your own custom agent • Flexible evaluation functions This is just our first release, and we’re actively expanding CocoaBench with more tasks, analyses, and agents. Excited to see what the community builds on top of it! Project website + more details in the thread👇