How the model in my computer-use project works.
It sees a screenshot of the screen.
It outputs one action, like a click or typing text.
The action runs, and the model gets a new screenshot.
In this example it reads the NameError, types has_digit, and clicks Run.
The loop stops when the model says done.
A checker then inspects the files on disk to decide pass or fail.
I post-trained Qwen2.5-VL-7B to use a computer for a simple coding loop. The model has to open Jupyter, open a notebook, run the code, and fix errors that show up.
I kept a locked set of 24 tasks out of training and evaluated on those. The untrained model scored 0/24. On this NameError example, it lingered on the Jupyter launcher or pasted a new function into the wrong line and left the typo has_digi in place.
The clip is that same task before and after training. After training, the model opens the notebook, reads NameError: name 'has_digi' is not defined, changes it to has_digit, and the second run prints True.
I post-trained Qwen2.5-VL-7B to use a computer for a simple coding loop. The model has to open Jupyter, open a notebook, run the code, and fix errors that show up.
I kept a locked set of 24 tasks out of training and evaluated on those. The untrained model scored 0/24. On this NameError example, it lingered on the Jupyter launcher or pasted a new function into the wrong line and left the typo has_digi in place.
The clip is that same task before and after training. After training, the model opens the notebook, reads NameError: name 'has_digi' is not defined, changes it to has_digit, and the second run prints True.
Part 2: I added reinforcement learning to my Chrome Dino agent.
It started from the model that copied my play.
The only reward was β1 for crashing.
5-minute runs went from 26/50 to 38/50 on the same courses.
16 courses got better and none got worse.
@TeslaSI
I taught an AI to play Chrome Dino from pixels only, just by watching me play.
It kept landing on cacti.
The fix was one number in training.
Median score: 208 β 2625.
How I built it, the CNN, how it failed, and how I fixed it.
@TeslaSI https://t.co/xnmEyvZpJn
I taught an AI to play Chrome Dino from pixels only, just by watching me play.
It kept landing on cacti.
The fix was one number in training.
Median score: 208 β 2625.
How I built it, the CNN, how it failed, and how I fixed it.
@TeslaSI https://t.co/xnmEyvZpJn
Pixels β actions, part 3.
My Dino agent kept landing on wide cactus groups.
I saved a clip of every crash and found it was jumping too early.
I measured it: I jump when the cactus is 18 px away, and the agent jumped at 30 px.
The cause was one number in training.
Jumps are only 4% of frames, so I made a missed jump cost 24x more than a wrong jump.
That made the agent jump whenever it was even a little unsure.
I tried 1x, 3x, 5x, 8x and 24x on the same 50 courses.
1x jumped too late and 24x jumped too early.
5x won.
Games lasting 2.5 minutes went from 0/50 to 35/50.
Median score went from 208 to 2625.
Code: https://t.co/OIeigtu509
Pixels β actions, part 2.
I ran my Dino agent for 50 games and saved a clip of every crash.
Median score: 259. Best: 2794.
I went through all 50 crashes.
About half were wide cactus groups. It jumps too early and lands on them.
About a third were birds. It jumps into them.
My training data barely had either one. I rarely reached birds when I recorded.
Next: record more of those cases, retrain, run the same 50 games again.
Pixels β actions, part 2.
I ran my Dino agent for 50 games and saved a clip of every crash.
Median score: 259. Best: 2794.
I went through all 50 crashes.
About half were wide cactus groups. It jumps too early and lands on them.
About a third were birds. It jumps into them.
My training data barely had either one. I rarely reached birds when I recorded.
Next: record more of those cases, retrain, run the same 50 games again.
This got me curious, so I built a tiny version. Pixels in β actions out.
I trained an agent to play Chrome Dino from raw pixels only. Just screen in, key out.
8.5 min of my own gameplay β small CNN β it now outscores me.
Me: avg 374
Agent: avg 910 (5 games)
1 ms per decision on a MacBook.
Bottom of the video: what the AI actually sees.
This got me curious, so I built a tiny version. Pixels in β actions out.
I trained an agent to play Chrome Dino from raw pixels only. Just screen in, key out.
8.5 min of my own gameplay β small CNN β it now outscores me.
Me: avg 374
Agent: avg 910 (5 games)
1 ms per decision on a MacBook.
Bottom of the video: what the AI actually sees.
Tesla is hiring a Machine Learning Engineer for Realtime Games, Digital Optimus, in Palo Alto. The job: build agents that play AAA games, browse the web and use pro apps at superhuman levels. Why games? Digital Optimus, the Tesla and xAI project Musk announced in March, reads screen video and acts with a keyboard and mouse. AAA games are a safe, endless, real-time proving ground for exactly that, the same approach DeepMind's SIMA uses. Tesla's posting says it will benchmark how skills transfer across games, general computer use, FSD and Optimus. https://t.co/1eJkapzr4g
Just wrapped up an incredible build day at #AgentsEverywhere! Our project, Turno, is a multilingual AI phone receptionist that understands natural language across English, Hindi, Spanish, and Hinglish. The real magic? It's seamlessly integrated into a phone line, making booking appointments effortless for patients. Huge thanks to @OpenAI for the amazing platform and to @aitinkerers for organizing this global event! So proud of what we shipped. #AI #Hackathon #VoiceAI
7/
Ended up keeping the GELU kernel and dropping the LayerNorm one. Output's identical to stock GPT-2.
Code + a longer writeup:
https://t.co/1kZRiiBFkb
Next I want to try torch.compile / CUDA graphs for that idle time.
1/
Wrote my first Triton kernel. Started by profiling GPT-2 to find the real bottlenecks, then tried to fix them with fused
kernels. Got a small win (+12.5%), Here's how it went.
6/
Reason: PyTorch already fuses LayerNorm, so mine was just adding overhead on top. GPT-2's GELU is written as a bunch
of separate ops that don't get fused, so there was actually something to gain there.