Our test cases are plain sentences , the everyday tasks people ask the agent to do, written the way they would say them.
Through the @the_agi_company MCP, Claude runs each one on a real Android phone, in the real app. When a screen changes, the sentence reads the same and still runs.
Mujgan, our product manager, wrote the requirements.
They go far beyond any synthetic benchmark, and she checks them on a phone the same day, with no engineering time involved.
Any mobile team can do this on Android today, and we are working on bringing it to iOS.
Our agent is the one that initially determines a task is successful when it reaches a finished state. When the agent finishes a task, we can determine success using our predefined heuristics, calibrated on the agent’s successes and failures.
For cases that don’t exactly match our heuristics, they are sent to our judge agent, which can classify them as success or failure.
The agent is good at screen understanding and understanding the user flows within the app.
If the task in the sentence is possible for the agent to take, then it can do it, even if a real person would not do it that way.
Otherwise, it is able to understand the task is impossible and notifies the user of this.
The agent is still under constant development, and we keep working on making its reasoning better
@Mihir_Malde@the_agi_company Yes, this is where our agent shines. Since we don’t overfit the agent on the UI, the agent is good at understanding the UI and taking actions based on the user flows.
It is able to understand how to navigate across different UI flows even if the app updates
The more capable a phone agent gets, the more places it should refuse to act on its own. Ours can use any app on your phone, but it’s instructed to stay out of banking apps and stops at every checkout before it spends your money.
Where to draw those lines turned out to be one of the biggest things we had to work out in testing.
Say you ask the agent for a flight to Japan, which might be a $1,000 flight. You don't want it to book one on its own and tell you about it afterwards.
So instead:
> it shows you the top five options
> stops at checkout
> then asks, "this is what it's going to cost you, do you want to proceed?"
Banking apps are off limits for the same reason. We don't want people using it for financial transactions autonomously.
The hardest case is when the instruction DOESN’T come from you. Say someone hides an invisible instruction to the agent inside an email, telling it to send money to a crypto link. It is very difficult to force a real phone into a situation like that, so we simulate a lot of these scenarios instead.
That's also why the list of blocked apps is small. Most of the safety goes into training, so the agent learns to refuse a task that looks risky or malicious and hands control back to you, without us hard coding every case.
We sit down with Div Garg, CEO at @agi_inc, to talk about running AI agents entirely on mobile devices, optimizing models for edge computing chips, and building safety mechanisms into autonomous app interactions.
https://t.co/MYNWhYKBLw
The AI that sees everything on your phone should live on your phone.
Think about what is on there. Your messages, your bank details, your health info, your photos, the people you talk to every day.
Whatever reads all of that and acts on it is the most personal piece of software you will ever use, and I do not think it should belong to a company that rents it back to you.
You already own the phone. You should own the model on it too.
One model for one person, trained to you, getting better every month, with your data staying where it already lives. We train our own models for this, and with every new generation of chips more of what we do will run on-device.
That model also has to do what you asked, inside your own apps, and get it right every time. If it gets each step right 97% of the time, it finishes a twenty-step task about half the time. Most of our work goes into the boring long tail of reliability.
We build this with the chip makers, phone makers, carmakers and telcos, so it comes with the device, under their brand, on millions of devices.
A billionaire has an army of people who do things for him. I want everyone to have that help, from a model they own.
@eo_build spent 48 hours with our team.
The full episode is out.
"The difference between you and a billionaire is the billionaire has an army of people who can go and do things for him. But now if we can have an army of agents that can go and do everything for you, that gap gets removed."
@divgarg is building a personal AI that runs on your own phone, where you own the model and the data.
In this episode of ODDS, we went inside @the_agi_company in San Francisco, from a new hire's first day on production code to an agent ordering Chipotle on DoorDash hands-free, and hear why Div turned down a million-dollar offer from OpenAI, why frontier labs won't ship AI that runs on your device, and why the team hires slow and fires fast.
0:00 Your own Jarvis on your phone
2:51 Starting Stanford's first Transformers course
4:10 Cold-emailing Geoffrey Hinton
4:57 Turning down $1M from OpenAI
6:01 Why trust comes before capability
9:05 Why frontier labs won't ship this
10:12 Live demo: ordering Chipotle hands-free
13:49 Demo isn't a product
15:52 A trial week of real production code
22:05 Hire slow, fire fast
23:36 Less screen time, more real life
I want the AI on my phone to be mine.
And we’re building it at @the_agi_company now.
AI that stays private and does things for me in my own apps.
@eo_build made a short documentary about the team and why we started AGI, Inc.
The full episode comes out Sunday and tonight we watch it with friends at @AGIHouseSF: https://t.co/xeEoWeySiA
@Vidiyala99@Techweek_@a16z @agi_inc sorry we missed you, the store wrapped a couple weeks ago. dm me, happy to talk shop, reliability is most of the job for us too
We're taking over a coffee shop on Union st. for 3 days of SF @Techweek_ by @a16z .
The goal is to make it the busiest pop-up in the city this week.
Mon to Wed, 9am to 4pm, franko & friends in the marina (next to the equinox): https://t.co/c9oSxrzpxV
Coffee and matcha on us.
What's there: our agent running live on android, the smart glasses checkout steve and i did on stage at snapdragon summit (look at a thing on the shelf, the agent buys it, confirmation on your watch), and our merch in the new branding.
two parties.
monday 4 to 6pm is the private opening: dj, drinks, friends, family, partners, VCs. invite only, dm me.
wednesday 4 to 6pm we close it out with a happy hour.
and some more serious stuff on thursday: devices x ai pitch night, 5 to 8pm. speakers from @Qualcomm, @deutschetelekom and @docomo, then founders building for phones, glasses, cars and robots pitch live.
rsvp here: https://t.co/Neil7GpiTt
we're hiring engineers and researchers, and we want to meet device makers.
come say hi, bring a +1.
@raily@Techweek_@a16z @agi_inc late on this, but the agent does the checkout: glasses spot it, it buys in the shopping app on your phone, you confirm on your watch. finding each other was what the coffee was for ☕️
Day 2 of our @Techweek_ wearables pop-up. a few hundred people tried the demos yesterday and every single one said the coffee is incredible. We’re building incredible things, but fine, the coffee.
Open today and tomorrow, 9 to 4, Franko & Friends on Union st (next to Equinox). come say hi and ask our agent to do something for you on glasses, a phone or a watch.
I want to see what you come up with.
And i want the busiest pop-up of #SFTechWeek
Stop by: https://t.co/c9oSxrzpxV
I’m very bullish on wearables. We started with phones and cars, but the form factor is going to change. Smart glasses, pins, and more are coming.
Last week we let anyone in SF walk into our @agi_inc wearables pop-up and try our agent on smart glasses. You’d think everyone here had tried a pair already.
Most hadn’t, and even I was surprised by how excited people got.
The shift happened when the glasses did something for them instead of taking a photo or playing a song. Look at something on the shelf, and it’s bought.
Watch the faces.
Last day of Wearables pop-up at @Techweek_ and there's no tomorrow.
We have a line every day to test our agent, and we're showing something special: @agi_inc running on @Meta Ray-Ban glasses.
You look at an item, say "buy it", and it's done.
@Meta built the glasses. We made them do things.
Coffee and demos till 4, happy hour 4 to 7, Franko & Friends on Union St.
If you've been meaning to come, this is the last chance you get.