Forja

Software Engineer / BJJ practitioner

[Forja @ImForja]

As a challenge to learn about reinforcement learning, I decided to train an agent to beat Pokémon Red.

The goal is to defeat the Pokémon League without giving the agent cheats like infinite money or automatic HM use.

So far, the agent can win against Misty, but there are still plenty of challenges ahead.

I'll stream the latest attempts on Twitch and share the successes and failures here on X.

[Forja @ImForja]

The capacity to verify that software you didn't write is correct is probably the most valuable skill right now.

[Forja @ImForja]

@0xSero Which of those would you say works best on two RTX PRO 6000 Max-Q?

[Forja @ImForja]

All solo drills for skill acquisition in jujitsu, and all combat sports in fact, are a waste of time.

Every movement you make should be a result of what your opponent is giving you. If you remove the opponent, there is no information for you to act upon, and your training environment is no longer representative of your performance environment.

Simple ecological dynamics.

[Forja @ImForja]

Took me a while to try it out, but the Codex desktop app is better than the CLI app. Especially if you're doing FE work.

[Forja @ImForja]

@mindinpanic Did those views convert into paid users?

[Forja @ImForja]

@leoncastilloSM Thanks! It's not a public app, just something I vibe-coded to help me stay focused.

[Forja @ImForja]

An issue I've had with tools that block distractions is that it's hard to categorize what is a distraction. E.g a YouTube video can be a distraction or a useful tutorial.

Turns out small AI models are great for this. Just explain why you need the app/website for the task and let the AI decide if you really need it or not.

[Forja @ImForja]

Nothing is as annoying in BJJ as that guy who is always stopping the roll to "teach" you something. Don't be that guy.

[Forja @ImForja]

It's easier than ever to build inaccessible apps.

[htmx.org / CEO of the Anti-Squeeslop League (same) @htmx_org]

the replacement of basic usability w/squeeslop animations continues apace

note the search element triggering the pointless animation (changes click targets to no benefit, btw)

button? search input? who knows? who cares? nothing matters. AGI has been achieved.

[Forja @ImForja]

@GuruAnaerobic Same goes for people who do martial arts.

[Forja @ImForja]

I've been going to the gym for 15 yrs now, and while improving a PR feels nice it doesn't come even close to hitting a cool move in a sport.

Still proud of this scramble and sub I hit a while ago in BJJ.

[Forja @ImForja]

@levelsio I lived in the Netherlands for a while. From time to time, I'd wake up in the middle of the night feeling very sick and with an urge to puke. It disappeared completely once I stopped drinking tap water.

[Forja @ImForja]

Bought two RTX PRO 6000s last November for around €7k each.

The same retailer lists them at €15k each now.

I should have bought four.

[Forja @ImForja]

@LandsharkRides BJJ gyms have a systematic hygiene problem. I've met only a few gyms where cleanliness is taken seriously.

[Forja @ImForja]

@levelsio I've been making my own electrolyte drink for a while as well for both the gym and BJJ practice. I recommend adding magnesium and potassium.

My recipe so far:
- 1L water
- 750 mg sodium
- 150 mg potassium
- 50 mg elemental magnesium
- 30g sugar
- Lemon/Lime

[Forja @ImForja]

Spent some time with Jev trying to see how it played Pokémon Red.

The TLDR is that playing a JRPG is not a good use for Jev, which, to be fair, is in line with their description that the model is suited for system 1 tasks, not system 2.

That's not to say it's impossible to beat Pokémon Red with Jev. If you build a good harness around it, where instead of Game Boy buttons as decisions you give it general actions like "Navigate to Oak's house" or "Pick Bulbasaur" and also short-term milestones to progress the game like "Pick Oak's parcel at Viridian City Pokémart" it's probably doable.

But at that point, the real engineering feat would be building a harness that makes general enough suggestions to allow Jev to be creative while still being possible to reach the goal of finishing the game. Otherwise, we would just be doing a scripted run with extra steps.

In case you're curious, here is a video of Jev playing Pokémon Red till the point of leaving the player's house. It cost 2.9 cents to get here, but to be fair, I could probably have made some optimizations.

[Forja @ImForja]

Got early access to Jev from TypeSafe AI. Looking forward to seeing how well it plays Pokémon Red 👀

[Forja @ImForja]

@psypharmacopeia What do your training sessions look like? 15m warm up + 45m static drills + 15m sparring?

[Forja @ImForja]

Update on beating Pokémon Red by training an AI agent from scratch with RL:

The architecture I went with had several policies, mostly one for each milestone of the game. E.g. Policy A to go from Oak's Lab to defeating Brock, Policy B to get to Misty, etc.

This allowed me to tackle the challenge incrementally. Instead of trying to train a single policy for days, I can train multiple for hours and see the AI progress day by day.

A weakness of this approach is a dependency of the later policies on the earlier ones. For example, the policy that beats Misty expects the previous policy to have chosen Charmander because that's part of the environment it was trained on. This becomes a problem when you realize you need to change a behavior in earlier sections to make the game easier to beat.

This problem became apparent now that I need the AI to deal with Flash. To get Flash in Pokémon Red, you need:
- To have caught 10 Pokémon
- Have a Pokémon that can learn Flash.
As of now, none of the policies have learnt to catch Pokémon, so I'm a bit stuck 🥲

I'm not 100% sure what I'm going to do yet to solve this, but I think decomposing the game into hard milestones to be achieved no matter what is not the way.

I'm more inclined to believe that training the AI to achieve exploratory goals like catching as many new Pokémon as possible and exploring all the zones might yield better results.

[Forja @ImForja]

The hard part so far of training RL agents to beat Pokémon Red is reliability.

It's not difficult to get an agent to reach a goal once. What's hard is to get it there at least 50% of the time.

The latest milestone is reaching Rock Cave. It's not reliable yet, but it's close.