Joseph Suarez 🐡

I build sane open-source RL tools. MIT PhD, creator of Neural MMO and founder of PufferAI. DM for business: non-LLM sim engineering, RL R&D, infra & support.

[Joseph Suarez 🐡 @jsuarez]

Releasing PufferLib 5.0: Train agents in under a second

[Joseph Suarez 🐡 @jsuarez]

@elonmusk @cradrrat Our small-model RL stack is all C / CUDA C. Trains up to 60M samples per second on a single GPU! https://t.co/2ErwF6cjWq

[Joseph Suarez 🐡 @jsuarez]

@yacineMTB You joke but I invited a brilliant guy from Africa to give an RL presentation during my PhD, and he had to reschedule because he got malaria

[Joseph Suarez 🐡 @jsuarez]

@raysan5 You ignore them

[Joseph Suarez 🐡 @jsuarez]

I had this exact shit happen with VCs. They could not understand "I am making this open source because it needs to exist and it can probably make money anyways." Even when I said exactly that. It was like they were looking for the calibrated strategy under which my open source profit model was 10 cents more profitable than selling drugs to orphans.

[Joseph Suarez 🐡 @jsuarez]

@NeuralSpace98 @istoica05 @berkeley_ai Nope, IS and KL pens do less than nothing. The only reason it works on LLMs is because they don't have a good bootstrap to destroy in the first place

[Daphne Cornelisse @daphnesolves]

I will be at CoRL next week! Come find me at poster session 3 on Wednesday, 11/11 (#60).

Decided to riff on the cooking and spice analogy for the poster; here is a little sneak peek.

[Joseph Suarez 🐡 @jsuarez]

It's never a vacuum. You want to know why it took me so long to figure out "hey why is our RL 1000x slower than the simplest possible napkin math?" ... Because I was surrounded by PhDs making the algos and envs SLOWER every year, until we got double-digit SPS. Oh, and if you actually make something fast, rejected from all conferences not sample efficient go fuck yourself

[Joseph Suarez 🐡 @jsuarez]

@RarityOS I'm down to send a few evenings ET. 90 kc scy/bow/zcb sick of WDR "oops I have 100 kc and still skydive"

[Joseph Suarez 🐡 @jsuarez]

@yacineMTB You said you were always right so I just threw a legitimately hard research question at you

[Joseph Suarez 🐡 @jsuarez]

What do you think is the ideal way to set up a state-based curriculum, given that you can't sample states the agent hasn't seen, that the useful state to return to cascades backwards from measurable value/adv spikes, and that most envs don't have easily exploitable level-based structure? Oh, and also that partial state sets are messy in many agent envs

[Joseph Suarez 🐡 @jsuarez]

@takyon236 I had to remake my first character because, being 10, I had never played an RPG and didn't understand that you don't spread your stat points around. I also at once point had an enu with -29 wisdom and couldn't figure out why leveling took so long.

[Joseph Suarez 🐡 @jsuarez]

Wouldn't you know, the guy who did his PhD on RL in MMOs plays a bunch of MMOs. Not much WoW, but here's a super niche emergent ready check story. Back in ~2006, I played a French MMO called Dofus. Your character had a right-click option to "slap" which did nothing but make your character say "Aie!" ... this became the ready check, to the point where players with their hands already on the keyboard would often type "aie" instead of clicking the menu option.

The game was turn-based and as a result was playable with terrible internet. A typical evening would involve spending 3 hours finding a barely passable party with massive level gaps and poor role coverage, an hour or two doing the rooms to get to the boss, and then anywhere from 10 minutes to an hour on the boss itself, depending on how janky your setup was. You had to click to join the fights, even within your own dungeon party. If you lagged out or went afk, tough, night ruined for everyone. So you wouldn't tell everyone to ready up, you'd tell them to Aie!

[World of Warcraft @Warcraft]

✅ or ✅✅?

[valo @ValtteriValo]

working on open world multi agent support for the OSRS env in pufferlib!

there are cool projects here on X with people running language model controlled bots on private servers.

this is a bit different (and i think even cooler) since it's a whole OSRS world simulated in fast C, 32 agents in this demo, and current policy is a 7.7 million param net as opposed to the LLMs needed to coherently control these bots being at least hundreds of billions of params.

these policies were trained in 4 minutes on a single gaming GPU, but i know there's plenty of throughput to gain still!

btw they immediately figured out dying is just free healing so they don't bother with eating...

[Joseph Suarez 🐡 @jsuarez]

@takyon236 Owned the whole bont jeweler's market for about a week once. Everything crafted, maged, or both by SealExpert

[Joseph Suarez 🐡 @jsuarez]

Both! Started when 1/2 crit fail rate Iop's wrath. Must have been before 1.29, though I don't know how far back... probably around 19 years ago. I started playing again fresh several times, on touch and on modern. I'd typically run a team of 4-6. If you want a really old bit of history, I think I was the first one to figure out bow skill + creizy stuff wand spam and to use it to 1-turn bherb

[Joseph Suarez 🐡 @jsuarez]

@typedfemale There is nothing peaceful about manual fusion stream overlapped cudagraph hell

[Joseph Suarez 🐡 @jsuarez]

@FakePsyho We hold our own accountable. Shitfluencers are on mute anyways

[Joseph Suarez 🐡 @jsuarez]

@yacineMTB @elyxlz https://t.co/Te4V4mNw1h. I do not know if it actually works

[Joseph Suarez 🐡 @jsuarez]

@SableVex @fraserpricee Not exactly. There is human playtest capability so you can see it works, but it is not built out to the point where people would want to actively play it in a persistent instance

[Joseph Suarez 🐡 @jsuarez]

@SableVex @fraserpricee All in (high-fidelity, game-accurate) sim. This is a research project. We do not deploy live bots to ruin my favorite game.