computer use agents are slow because they think too much
they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow.
my mac now remembers.
the first time i say something new, like "search youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec:
{"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"}
every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA
say it once, it's slow. say it again, it's instant.
the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said.
and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second.
everything it learns lives in one plain text file on my disk.
i've been playing around a lot with @trycua by @francedot and peekaboo by @steipete . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.