With DiagEval, a Gemini 3 Flash Preview-powered GUI evaluator surpasses the Opus 4.6 baseline in accuracy while costing less than half per case.
๐ Paper: https://t.co/1XS58XePBo
๐ป Code: https://t.co/wbeSVryinj
GUI agents are now the default for evaluating vibe-coded apps.
But when an agent says โtest failed,โ is the app actually broken, or did the agent just fail?
We introduce DiagEval: a post-failure diagnostic framework that treats failed rollouts as evidence, not final verdicts.
As a first step, we ask whether executable dynamics can be learned from collected I/O traces alone, without instrumented program state. Concretely, we model CLI and GUI execution as video rollout from instructions, pixels, and user actions.
Early demos show signals of learned runtime primitives: terminal rendering, execution structure, I/O alignment, short-horizon control, and GUI interaction.
๐๐๐ ๐๐๐ฅ๐จ๐ฐ.
๐ซฑ Introducing ๐๐๐ฎ๐ซ๐๐ฅ ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐๐ซs:
๐ฐ๐ก๐๐ญ ๐ข๐ ๐๐ ๐๐จ๐๐ฌ ๐ง๐จ๐ญ ๐ฃ๐ฎ๐ฌ๐ญ ๐ฎ๐ฌ๐ ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐๐ซ๐ฌ ๐๐๐ญ๐ญ๐๐ซ, ๐๐ฎ๐ญ ๐๐๐ ๐ข๐ง๐ฌ ๐ญ๐จ ๐๐๐๐จ๐ฆ๐ ๐ญ๐ก๐ ๐ซ๐ฎ๐ง๐ง๐ข๐ง๐ ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐๐ซ ๐ข๐ญ๐ฌ๐๐ฅ๐?
Beyond today's conventional computers, agents, and world models, Neural Computers (NCs) are new frontiers where computation, memory, and I/O move into a learned runtime state.
We ask: whether parts of runtime can move inward into the learning system itself. This is our first step toward the Completely Neural Computer (CNC): a general-purpose neural computer with stable execution, explicit reprogramming, and durable capability reuse.
Work done with Mingchen Zhuge (@MingchenZhuge), Changsheng Zhao, Haozhe Liu (@HaoZhe65347 ), Zijian Zhou (@ZijianZhou524 ), Shuming Liu (@shuming96 ), Wenyi Wang (@Wenyi_AI_Wang ), Ernie Chang (@erniecyc ), Gael Le Lan, Junjie Fei, Wenxuan Zhang, Zhipeng Cai (@cai_zhipeng ), Zechun Liu (@zechunliu ), Yunyang Xiong (@YoungXiong1 ), Yining Yang, Yuandong Tian (@tydsh ), Yangyang Shi, Vikas Chandra (@vikasc), Juergen Schmidhuber (@SchmidhuberAI)
You Donโt Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation
The paper introduces RealDevWorld, a novel evaluation framework that combines RealDevBench, a diverse set of open-ended software engineering tasks, and AppEvalPilot, an agent-based evaluation system, to effectively assess the functionality and usability of production-ready software generated by Large Language Models through dynamic user interactions.
Yutong Bian @yutong_bian , Xianhao Lin, Yupeng Xie, Tianyang Liu @LtyLeoii22 , Mingchen Zhuge @MingchenZhuge , Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang @didiforx , Jiaqi Chen, Xiangru Tang @XiangruTang , Yongxin Ni, Sirui Hong, Chenglin Wu @AlexanderWu0
DeepWisdom; Fudan University; HKUST; UC San Diego; KAUST; Westlake University; Stanford University; Yale University; NUS
https://t.co/r9pL7XhWZY
๐ค How can AI Agents learn from past experiences across different tasks, instead of starting from scratch?
๐ We just released Agent KB! Our framework uses a teacher-student mechanism to let agents reuse experience, enabling cross-task knowledge transfer.
๐ฅ The results are stunning: A new open-source SOTA on GAIA!
ยท GPT-4.1 performance +18.8% (55.1% โ 73.9%)
ยท Claude-3.7 on SWE-bench solve rate +12%
๐ Paper: https://t.co/zD0FK3XRwh
๐ป Code: https://t.co/L74M9952sQ
Like the idea of experience reuse? A GitHub โญ is much appreciated!
๐ง 264 pages and 1416 references chart the future of Foundation Agents.
Our latest survey dives deep into agentsโcovering brain-inspired cognition, self-evolution, multi-agents, and AI safety.
Discover the #1 Paper of the Day on Hugging Face๐:
https://t.co/CvtqGKyCYb
1/3
20 Months: 0 โ 7 papers (2 ICLR orals) & 40+ institution collabs.
With a clear vision, we're building the open-source foundation for tomorrow's agents.
We also release MGX (https://t.co/AyzVUF9dH3) and commit to open-source its core soon.
Check threads for what we've built!
1/8