Yes, I would agree that for clicking, modern LLMs are good enough to handle it on it's own and are quite cost efficient, however, when you are trying to understand and extract more complex data from screenshots with lots of elements, it becomes harder and harder to consistenly rely on LLMs.
Self-hosted grounding models never change weights on it's own, and offer much better predictability with quicker times, so in the instance that you cannot rely on the LLM, grounding models are an excellent alternative.
Having worked at a yc-backed startup building CUAs for over a year, I learned a thing or two.
This article goes over everything you need to know, with code examples and an open-source version of a CUA-building platform as well. https://t.co/wnmKUM2dq6
@VyomKulsh26 Frontier LLMs should be able to understand when the window is out of focus. In the instance that it is not, I would recommend:
1. trying to concisely provide context on how to prevent this issue should it arise, or:
2. try to deterministically avoid dialog popups
It's movements whilst manipulating the VM can seem erratic, as it does move very fast, however in terms of the steps it takes to complete the process, is all based on how you define the tasks.
Agents nowadays can do their own discovery process and naturally find good ways to complete the tasks.
Computer-use, by Openly. Build your own computer-using agents, with or without code.
100% free.
100% open-source.
No-code tools supported.
https://t.co/Crj9e75k1E