Chris Rawles

Thinking and reasoning at @GoogleDeepMind

[Chris Rawles @thecrawles]

1/ 🤖 Personal assistants of the future will be able to operate computers just like humans — by controlling user interfaces. To help make this vision a reality, we are excited to introduce AndroidWorld, a new benchmark for building and evaluating computer control agents. 🌍📱

[Yu Su @ysu_nlp]

Life update: I moved to silicon valley to tackle agents' biggest challenges: plasticity and reliability.

Today's agents are smart but brittle. They lack plasticity (continual learning and adaptation) and reliability (stable, predictable behavior with bounded failures). These two traits define whether agents become critical infrastructure or remain clever demos.

Plastic systems like to change. Reliable systems resist change. Is it even possible to have both of these seemingly conflicting traits? Fortunately, humans are a living example of that. We are constantly learning and adapting while staying remarkably dependable (for the most part, at least). The real question is, how can we achieve the same harmony within a different cognitive substrate?

We've brought together some of the world's best agent experts whose work (Mind2Web, MMMU, LLM-Planner, SeeAct, UGround) helped shape the modern agent field. Now we are taking on the new mission: unlocking plasticity and reliability for every agent.

We are looking for cracked researchers and engineers to join us in person in the bay area! If you strongly resonate with the mission, send your CV and thoughts to: [email protected]

I will be at #neurips2025. Happy to chat over coffee!

[Sundar Pichai @sundarpichai]

Our new Gemini 2.5 Computer Use model is now available in the Gemini API, setting a new standard on multiple benchmarks with lower latency. These are early days, but the model’s ability to interact with the web – like scrolling, filling forms + navigating dropdowns – is an important next step in building general-purpose agents.

Developers can try these capabilities via API in @googleaistudio + Vertex AI.

[Chris Rawles @thecrawles]

Headed to ICLR to present AndroidWorld. Reach out if you want to connect!

[Chris Rawles @thecrawles]

@BowenWangNLP Super cool idea. Nice work!

[Chris Rawles @thecrawles]

@jiayi_pirate voice to text for iMessage is so clutch

[Chris Rawles @thecrawles]

Leaderboard:
https://t.co/x6Lm4lLzCz
Project page: https://t.co/sjpj46bjgM

[Chris Rawles @thecrawles]

Take a break from election news and check out the new AndroidWorld leaderboard! (Shamelessly copied the design from @shuyanzhxyc et al.'s WebArena leaderboard 😄)

[Xing Han Lu @xhluca]

Now we have an environment benchmark for every interface/platform:

Web: {visual}webarena, WorkArena, etc.
Ubuntu/Mac: OSWorld
Windows: WindowsArena
Android: AndroidWorld

[Chris Rawles @thecrawles]

Super cool to see this. I am curious how it will perform on mobile screens and tasks. Soon, hopefully there will be numbers on AndroidWorld https://t.co/xvKta6b6zl

[Alex Albert @alexalbert__]

Computer use API

We've built an API that allows Claude to perceive and interact with computer interfaces.

You feed in a screenshot to Claude, and Claude returns the next action to take on the computer (e.g. move mouse, click, type text, etc).

[Yu Su @ysu_nlp]

People into agents, let me pitch something to you:

🌟 An agent that works across every platform (web, desktop & mobile)
🌟 Visual perception only, no messy & often incomplete HTML or a11y tree
🌟 SOTA performance across 6 agent benchmarks

Sounds too good to be true? Continue ⬇️

🖥️ Human-like embodiment for digital agents
Most humans perceive the digital world visually and act via keyboards, mice, or touchscreens. In principle, the embodiment of a digital agent should already be complete if it can

1) visually perceive the GUI renderings, and
2) have effectors equivalent to a keyboard for typing and equivalent to a mouse or touchscreen for pixel-level operations like clicking and hovering.

However, existing agents, even those using multimodal LLMs, still require text-based representations (HTML/accessibility trees/etc.), which are noisy, often incomplete, and costly to obtain and encode, adding to latency and compromising user experience. Occam's razor tells us to seek a minimalist approach: visual observation only + pixel-level operations directly on the screen.

📌 Visual grounding was the blocker
In our SeeAct agent ('GPT-4V is a Generalist Web Agent, if Grounded' at ICML'24), which helped popularize multimodal agents, we have identified visual grounding as the key challenge. Multimodal LLMs like GPT-4o can often visually perceive a complex screenshot correctly and generate plausible textual plans (e.g., 'go back to the homepage') about what to do, but how to turn the textual plans into precise coordinates on the screen that an agent should act on (e.g., pixels corresponding to that house icon at the top left of the screen)? A few recent works have attempted this visual grounding problem, but the performance was far from practical use, which was blocking the development of digital agents with human-like embodiment.

📢 Introducing UGround, a universal grounding model for GUI agents
We need a universal visual grounding model that can:

- Generalize across platforms: different websites, desktop (Windows, Linux, macOS), mobile (Android, iOS), etc.
- Plug and play in different planning models (e.g., multimodal LLMs)
- Support diverse input image resolutions (up to 1-2k pixels, portrait & landscape) and referring expressions (visual, positional & functional)

We show that a simple recipe, which includes web-based synthetic data and a slight adaptation of the LLaVA architecture, is surprisingly effective for training such universal grounding models. Using our data synthesis pipeline, we construct a large grounding dataset with 10M (screenshot, referring expression, element coordinates) triplets and train UGround with a 7B LLaVA-like architecture.

💥SOTA in most comprehensive agent eval to date

While most existing agent work evaluates on 1-2 benchmarks, to show the universality of UGround, we evaluate on 6 agent benchmarks covering all major platforms and 3 different settings:
- GUI visual grounding (ScreenSpot)
- Offline agents (Multimodal-Mind2Web, AndroidControl & OmniAct)
- Online agents (Mind2Web-Live & AndroidWorld)

Using UGround, agents with human-like embodiment (visual perception only + pixel-level operations) achieve SOTA performance across the board, often outperforming prior SOTA by a large margin, despite prior work using additional text-based observations! These results make a strong case for human-like embodiment for GUI agents.

Another surprising finding: even though the training data is predominantly web-based (because the web has the richest metadata for synthesis), the model generalizes fairly strongly to desktop and mobile settings. This shows the shared design principles underlying different GUIs do exist and UGround can capture that.

Final remarks
We started working on this not too long after SeeAct. It actually didn't take long to get to SOTA performance on visual grounding (ScreenSpot), but what we really care about is to make the model useful in real agents, not just on grounding benchmarks. That took another few months of hard work. I'm glad that the final solution is still quite simple, elegant, and works like a charm.

This work is led by the amazing @BoyuGouNLP from @osunlp (can you believe this is just his 1st year in PhD?). @OrbyAI (Yanan, Cheng, Will, Peng, etc.) has been an invaluable collaborator on this: compute, eng support, research brainstorming, and incredible patience in tolerating my nitpicking that keeps delaying the release 😜

[Chris Rawles @thecrawles]

@ChrisVernonShow @KevinOConnorNBA Love you Verno. Will miss you guys :(

[Rogerio Bonatti @rogerio_bonatti]

AI assistants have changed the way we use computers to work and search for information. As LLMs become more powerful, what’s next? Agents.

Excited to introduce Windows Agent Arena, a benchmark for evaluating AI models that can reason, plan and act to solve tasks on your PC.

🔗Blog: https://t.co/ZnocYBAb1a
🌐Webpage: https://t.co/v0Sleh9TTT
📃Paper: https://t.co/DL68XDiLVZ
💻Code: https://t.co/X5lusmxOLh

🧵👇

[Chris Rawles @thecrawles]

@random_walker Nice work -- I really like the proposed baselines. For UI control, we observed high variance in success rate using temperature=0 and it's even worse when changing the task seed/parameters. Fig 3 from https://t.co/gfjeyUB90h

[Jiayi Pan @jiayi_pirate]

🎉Release day!
We develop RL techniques / infra to post-train VLM agents for device control.
Our 2B VLM, when post-trained with an autonomous evaluator (reward model), improves its success rate on Android device-control tasks from 17% to 67%.

[Chris Rawles @thecrawles]

@pan_jiayipan @xhluca This is an exciting line of work! Will be interesting how accurate we can make autoraters

[Xing Han Lu @xhluca]

Another solid work by the Android agents team at @GoogleDeepMind. 800+ apps interactions is massive.

It would probably cover most of the use cases we can see on Android (so one can spend less time on thinking about generalization and focus on doing well on validation).

[fly51fly @fly51fly]

[LG] Latent State Estimation Helps UI Agents to Reason
W E Bishop, A Li, C Rawles, O Riva [Google Research] (2024)
https://t.co/V6NlNWRqU9

- Latent state estimation is important for agents operating in real-world environments with non-deterministic action outcomes and noisy observations.

- This work investigates latent state estimation for large language models (LLMs) in the context of autonomous user interface (UI) agents.

- LLMs can form point estimates of latent state by assigning high probabilities to completions that best explain observations, leveraging intuitive knowledge encoded in the weights.

- 5 aspects of latent state are estimated for UI agents: previous actions, screen summaries, progression, mistakes, and completion.

- LLM accuracy at estimating these is 76.8%-97.3%, matching or exceeding human performance.

- Incorporating latent state estimates improves performance of LLM-based UI agents by 1.2-1.6x across 3 reasoning methods and 3 benchmarks.

- Analysis shows the cause of failure shifts from action selection to other factors like grounding when latent state is used.

[Chris Rawles @thecrawles]

@frankxu2004 The APIs are described here:
https://t.co/9qelri4msL

We opted not to use them for M3A, but we made this API designed for agents to consume to understand how to use the tools:
https://t.co/D1ihNdjXfX

[Chris Rawles @thecrawles]

1/ 🤖 Personal assistants of the future will be able to operate computers just like humans — by controlling user interfaces. To help make this vision a reality, we are excited to introduce AndroidWorld, a new benchmark for building and evaluating computer control agents. 🌍📱

[Chris Rawles @thecrawles]

9/ 🎉 We're excited to share AndroidWorld with the research community and are looking forward to seeing what people build on top of it!

[Chris Rawles @thecrawles]

8/ We also tested a Web Agent on AndroidWorld to explore cross-domain automation. While web agents can solve Android tasks, they significantly lag behind Android-optimized agents like M3A. More work will be needed to build effective cross-domain agents for optimal performance. (Thanks to @ysu_nlp, @boyuan__zheng, @BoyuGouNLP, and @hhsun1 for their work on the web agent!)