Logesh Kumar

Machine learning Research and Engineering @ Blackbox AI | LLMs, Code generation , Code Assistants

[Logesh Kumar @logesh_umapathi]

Streamrender-H3 Project page : https://t.co/6yXrD3tgTW

Code video model Project page: https://t.co/XKdVn3fzVq

[Logesh Kumar @logesh_umapathi]

Fascinated by code -> video (world) model research. Code brings determinism and physics, which are challenging for video models. Video generation adds the aesthetics and polish that code-based engines lack.

1. Streamrender-H3 : Converts the visual from a code game engine and renders it via streaming diffusion model at ~230 ms latency

2. Code video model: turn prompts into code-driven video previews. Then the video previews are rendered in high fidelity with video generation models

Links below.

[Logesh Kumar @logesh_umapathi]

@KeithZhai Thats a good point! didn't explicitly measure the overhead added by puppeteer yet. Expecting it to be ~20ms range.

[Logesh Kumar @logesh_umapathi]

🤔 Current generation computer-use agents are frustratingly slow. Can we make computer use agent 3x faster by treating them as decision models?

I wanted to explore this, and it turns out we can!

I trained GUI-Decisions, a set of vision-language decision models that predict actions and coordinates without autoregressively decoding them as text.

Instead of generating coordinates, the models learn to represent each coordinate as a probability distribution over 256 bins. We read these distributions directly from fixed slots.

The results:
🚀 31B model: 146 ms per grounding step — ~3.5× faster than conventional decoding (472–546 ms).

🚀 2B model: 93 ms per step — ~3.5× faster than its native-resolution baseline (323 ms).

With reasoning enabled, the conventional 31B baseline takes 2.6–3.1 seconds per step, making the slot-reading approach dramatically faster for the evaluated grounding tasks.

Of course, this doesn't solve the entire latency problem for multi-step computer-use agents. But it removes a substantial amount of unnecessary decoding overhead from the grounding step.

I have open-sourced both models, a live demo, and a detailed write-up covering the approach, experiments, failures, and limitations.

#huggingface #ai_agents #computer_use

[Logesh Kumar @logesh_umapathi]

@smykx @JiahuaDong Thank you so much !

[Logesh Kumar @logesh_umapathi]

@Joshua_WD Thank you!

[Logesh Kumar @logesh_umapathi]

@Joshua_WD You could also test it live here - https://t.co/bpcD8cywnJ

[Logesh Kumar @logesh_umapathi]

🖼️ 𝗧𝗿𝘆 𝗶𝘁 𝗹𝗶𝘃𝗲 𝗵𝗲𝗿𝗲 : https://t.co/bpcD8cywnJ

📝 𝗕𝗹𝗼𝗴 𝗽𝗼𝘀𝘁: https://t.co/2hmx0gDU20

🔗 𝗠𝗼𝗱𝗲𝗹𝘀 : https://t.co/Hr2dw7N42Y

[Logesh Kumar @logesh_umapathi]

Blog post: https://t.co/2hmx0gDU20

Models: https://t.co/GYSrGWYXBa , https://t.co/Zu7mfs5WyB

Try it live now: https://t.co/bpcD8cywnJ

[Logesh Kumar @logesh_umapathi]

Introducing GUI-Decisions.

An open-source decision model for GUI grounding that is 3x faster than its non-thinking counterpart and 25x faster than its thinking counterpart.

Here is a thread on how this is trained 🧵👇

[Logesh Kumar @logesh_umapathi]

Results:

We get >3x faster latency and better GUI grounding accuracy than the base - Gemma4-31B model

[Logesh Kumar @logesh_umapathi]

@natolambert @trillium_labs Awesome. Congrats ! !

[Logesh Kumar @logesh_umapathi]

You can try it here today: https://t.co/Kr4346bS4t

[Logesh Kumar @logesh_umapathi]

Here is a glimpse of Ambient's upcoming release. The ambient agent is shaping up to be a generalist video agent. Video in, any outcome out.

Excited about the release.

[Logesh Kumar @logesh_umapathi]

Repo: https://t.co/sYPGSxaqOb

[Logesh Kumar @logesh_umapathi]

🚀 𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺 𝗢𝗻𝗲 : System One typed decisions endpoint over existing decoder-only models and multimodal support

Jev and Laya are awesome; they enable a new paradigm of use cases. But they don't support multimodal inputs and require us to use new models. It turns out that existing decoder-only models (like Qwen3.5, Gemma, etc.) are good zero-shot classifiers (no surprise) and can be extended without training for typed decisions like Jev or Laya.

So, I built a layer that can expose zero-shot system one decisions endpoint over existing inference services. It supports vLLM / Sglang / even OpenRouter (experimental).

• 𝗠𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗰𝗼𝗺𝗲𝘀 𝗳𝗼𝗿 𝗳𝗿𝗲𝗲. With vision enabled models like gemma 4 or qwen 3.5 - the typed decisions endpoint supports images / video / audio, Jev or Laya doesn't support images or video yet.
• 𝗧𝗵𝗲 𝗹𝗮𝘁𝗲𝗻𝗰𝘆 𝗮𝗻𝗱 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 𝗶𝘀 𝗰𝗼𝗺𝗽𝗮𝗿𝗮𝗯𝗹𝗲 𝘁𝗼 𝗝𝗲𝘃 𝗮𝗻𝗱 𝗟𝗮𝘆𝗮 - and in some cases better on self hosted models.
• This approach builds a zero-shot classification wrapper around existing decoder-only models, enabling us to use them for typed-decision tasks.
• Therefore, the existing world knowledge of these models is preserved, and we can use them for typed-decision tasks.
• Experimental support for fine-tuning with the RLCD objective to improve accuracy.

The attached demo shows self-hosted Gemma-31B playing Snake and a maze game with only image as input over a system one decision endpoint at just ~150 ms latency per step.