Jeff Schomay

Software engineer at the intersection of games, narrative, and AI

[Jeff Schomay @jschomay]

My latest exploration - letting AI play my game https://t.co/93Mh4TJyd4

[Jeff Schomay @jschomay]

What if crosswords had images for clues instead of words? I made Visual Crossword as a quick weekend experiment in blending AI with classic game mechanics. Play it at https://t.co/lnydBDBQGr

[Jeff Schomay @jschomay]

I turned ASCII game graphics into full-frame visuals in real-time using AI 🎮

100ms generation times at 10 FPS - here's how I solved the latency challenge with @FAL's LCM models and WebSocket optimization.

Full technical breakdown: https://t.co/9MYnk0glw1

#AI #GameDev

[Geoffrey Litt @geoffreylitt]

tired: Claude Code ports my personal website to a new framework

wired: Claude Code generates a command center UI where I can execute the port! with live feedback, including a side-by-side preview of old / new site 😎

this is an example of what I call an "AI HUD" -- a "heads-up display" that gives me visibility into what's going on. i can see:

- what terminal commands are running
- what files are being created on disk
- a live running preview of the old / new website (!)

this HUD was actually my third attempt at doing this task, and I liked it the best. here's why I landed there:

I wanted to move from middleman to astro for my personal site. These are both static site generators that output html, so I suspected an AI agent could do a good job with a port -- since it could compare old / new build output and try to match.

Attempt 1: Agent
I just had Claude Code do the port. It went pretty well.

But it was hard for me to review what had happened! The git diff was dozens of new files and it was hard to tell how they mapped to the old site. I think the LLM also tried directly writing some files rather than doing deterministic copying, which felt dangerous.

Attempt 2: Script
I realized a better approach was to have Claude Code write a script that would do the port. The script would execute shell commands like "copy all the markdown files from this directory into this other directory in the new site".

This worked much better because I could review the *process* rather than the *results*. Still, even with a nicely commented script, it felt like I was doing a lot of work to review. I wondered: how could I make it ridiculously easy to review this?

Attempt 3: HUD
Enter the command center! This is an entire web app that I had Claude build just for this one task. I can click buttons to run each step of the port, and watch the new website gradually materialize.

It has a node server that runs on my local machine that executes commands on my filesystem and coordinates dev servers for the old/new site. And then a web UI that talks to that server to visualize terminal commands, file trees, and live running previews. It's a HUD because I feel like I can ambiently see everything that the port process is doing 🧐

Was this whole thing worth making for this one-time task? I'm not sure, maybe it was overkill 😅 But it did help me feel more confident that I understood what was going on, and total time was still less than it would have taken me to do the whole port by hand!

One way to look at this experience is: "AI can put absurd amounts of effort into a pull request description."

What do I mean by that? Well, a PR description is a way to communicate a code change to a human. Typically we do that by writing a bit of text, because that's all we have time for.

But given more time and effort, we can do better! An interactive command center / HUD can help a person understand what a code change is actually doing. Historically you'd never imagine investing the effort to do that for a single PR. But now it's within the realm of possibility.

So, next time you have an agent do something for you, and you wish you understood better what it was doing, consider: instead of just asking for textual descriptions, maybe ask the agent to make you a HUD!

[Jeff Schomay @jschomay]

What do you get when you combine a classic roguelike game engine with a metroidvania, a sci-fi theme, and fit it all into a single screen? You get "Edge of Known Space" - my latest game that I am excited to release! Play for free at https://t.co/eH0SQoKS3N

[Jeff Schomay @jschomay]

@jxnlco We do at @mech_orchard to build agents that help modernize legacy COBOL mainframe systems.

[Jeff Schomay @jschomay]

@jxnlco we finally started using Instructor at work and I wanted to reiterate how awesome it is and shout out to all the tutorials too. I wrote a piece on how we use it to eloquently implement advanced LLM patterns in our agents.
https://t.co/hw73Rovs5O

[Jeff Schomay @jschomay]

Article I wrote at @mech_orchard on "How to Make Your AI Less Flaky" https://t.co/29xx9Pcrad

[Jeff Schomay @jschomay]

Well that didn't take long! @leonardoai just added the live draw feature I mentioned. Pretty sweet. https://t.co/RcEJAyOU1k

[Jeff Schomay @jschomay]

A tip I learned for getting more control over AI generated art
https://t.co/sBw98KwO3T

[Jeff Schomay @jschomay]

@jdklose Thanks Johannes! Hey, BTW Flowtastic is pretty sweet, nice work.

[Jeff Schomay @jschomay]

A bit late, but sharing my experience while at AI Engineer Summit: In Search of AI @aiDotEngineer https://t.co/4Rx265r2q9

[Jeff Schomay @jschomay]

Nice! My article got featured on @hashnode :)

[Hashnode @hashnode]

AI-generated games are endless, unique, and unpredictable 🎲

Join @jschomay in creating a game where no two paths are the same. Gamers and AI enthusiasts – this one's for you!

🔗 https://t.co/YrKvcFI7q5

[Jeff Schomay @jschomay]

@rushing_andrei @yoshtm1 Nice, getting reminders of our waymo experience...

[Jeff Schomay @jschomay]

@jxnlco @aiDotEngineer I haven't had the chance to use it or read through the docs yet, but at first glance I appreciate how comprehensive they are! I love that you include a cookbook. I saw the talk in person live and I thought it was great the whole way through.

[Jeff Schomay @jschomay]

This was one of my favorite talks at @aiDotEngineer sumit: Pydantic is all you need: Jason Liu https://t.co/agJgBHRl2d @jxnlco

[Jeff Schomay @jschomay]

The generative ai asset pipeline and server code is open source at https://t.co/KY9yrLK9rm. Feel free to fork it and make your own variations of infinite worlds to explore!

[Jeff Schomay @jschomay]

Just finished my write-up on the talk I gave at @aiDotEngineer Summit - "Lost in an Infinite Maze: Building a Real Time Generative AI Game Assets Pipeline" #ai #gamedev
https://t.co/3l3DLd9pON

[Jeff Schomay @jschomay]

You can play the final game at https://t.co/q2YJ9xkHdf

[Jeff Schomay @jschomay]

Just got back from @aiDotEngineer AI Engineer Summit 2023. It was great! My favorite talks:
- Pydantic is all you need
- The Hidden Life of Embeddings
- Climbing the Ladder of Abstraction (on the day 2 video)
- Open Questions for AI Engineering
https://t.co/ZXlErPT7YK