Added spatial context to my desktop agent.
Now I can just point at something on my screen and ask it to act on it.
No need to explain what I’m referring to. It knows exactly where I pointed.
Added spatial context to my desktop agent.
Now I can just point at something on my screen and ask it to act on it.
No need to explain what I’m referring to. It knows exactly where I pointed.
@yash_darbar16 The MCP gives Claude visual context screenshots plus the exact element/coordinates we’re pointing at so it can understand the reference and act on it accurately.
One thing I kept running into with coding agents was how hard it is to give them visual context.
I ended up building an MCP server that lets the agent see what I'm looking at. I can point at something on the screen, talk through it naturally, and it understands exactly what I'm referring to.
It's a surprisingly small change, but it makes working with a coding agent feel much more natural.
Today we're shipping spatial context.
This lets you draw directly on your screen + pass it as context to the model.
AI can easily analyze screenshots. But, it has no idea where exactly you want it to look. Now you can just show it.
It's very intuitive.
Demo:
It's kind of wild watching Fable work for 47 minutes straight on a complex task. Feels a lot more hands free than before you can actually focus on something else while it keeps making progress...
@FarzaTV Been building around this too. The biggest thing I've noticed is how much time gets wasted just trying to explain what you're looking at. Once the agent has that context, everything feels a lot smoother.
Demo:
One thing I kept running into with coding agents was how hard it is to give them visual context.
I ended up building an MCP server that lets the agent see what I'm looking at. I can point at something on the screen, talk through it naturally, and it understands exactly what I'm referring to.
It's a surprisingly small change, but it makes working with a coding agent feel much more natural.
I've been building a desktop agent that can understand intent and complete multi-step tasks across your desktop.
The part I find most exciting is that it works in the background, so you can keep using your computer while it gets things done.
I've been building a desktop agent that can understand intent and complete multi-step tasks across your desktop.
The part I find most exciting is that it works in the background, so you can keep using your computer while it gets things done.