love this idea! We have had successful examples of learning in the open (github, stack overflow) but being deliberate about this kind of learning and checking assumptions - like conversations being private - is more important than ever with AI
Hot take: an empty chat box as the primary UI is the exact *opposite* of intelligence in a product
It's like saying "I have no idea what you want to do here, tell me"
This is why pre-AI UIs feel better IMHO
Genuine thanks to these folks for their awesome work that supported my fun experiment for the past few weeks!
@karpathy - autoresearch's keep/discard loop and the idea that eval infra matters more than generated code directly shaped this
@badlogicgames - pi is by far the best coding harness
@microsoft - playwright-cli makes the real-browser evaluation possible
@ctatedev / @vercel - agent-browser as an excellent alternative backend
@vite_js - Vite+ drives all the prototypes - having a single `vp` command that encompasses all parts of webdev is very agent-friendly
It's early and rough. I'd love feedback from anyone thinking about AI-native product development, agent-driven evaluation, or how the SDLC is changing.
https://t.co/ZJ0gOb5h89
It starts with a persona, e.g. Marco, 35, software engineer, gets home at 6:15 with a hungry toddler and 30 minutes to figure out dinner.
The persona generates its own tasks based on their life context that you define. The builder doesn't get to define what success looks like.
Then the loop runs. The persona tries to use the app - enters ingredients, applies filters, picks a recipe. Gets stuck, gives up, or succeeds. The coding agent reads the feedback, writes a plan connecting specific persona struggles to specific code changes, and tries again.
If the score improves, then keep the iteration. If it doesn't, then discard the iteration.
It's slow right now - a full session with three prototypes and persona variants takes 6-10 hours. You can kick it off and come back to it later. Suggest not watching it in real time :)
To be clear - this is called autocrit, not "autoproduct."
The synthetic persona is a useful caricature, not a real user. It doesn't have actual prior experience, social context, or the ability to learn over time. Results are meant to lead to "we have confidence to go deeper on approach A" - not "ship this"
Building products closely with humans is and will likely always be critical. This doesn't replace user research. It's a way to get structured feedback fast enough to keep up with how quickly we can now build, so you bring something sharper to real users when you do talk to them.
Where this could go next:
- grounding personas in real user data
- more fine-grained, continuous evaluation
- and treating the user model as a durable artifact (in addition to others) rather than the code.
Longer blog post coming soon!
In the time it takes to write a PRD, you could spin up dozens of prototypes. But how do you know which one is useful for your users?
Part of what makes @karpathy's autoresearch work so well is that val_bpb is scalar, deterministic, and cheap to compute. UX quality is none of those things... it's multi-dimensional, noisy (same code can score 50 one run and 80 the next), and expensive per evaluation.
pi-autocrit attempts to bridge that gap instead of pretending it doesn't exist:
- Task-based scoring grounds the signal in behavior, not opinion: the persona either accomplished the task or didn't.
- Priority tiers (P0/P1/P2) collapse multiple dimensions into one composite score you can improve on
- Persona variants (same person, different moods) help separate real signal from noise
It's not val_bpb. But it's enough to drive a loop that measurably improves things, and the measurement itself will keep getting better.
I've been experimenting on something for the last couple of weeks:
pi-autocrit - an automated UX evaluation loop for web app prototypes.
A synthetic persona actually uses your app in a real browser (clicking, typing, getting confused) and reports what worked and what didn't. The coding agent reads that feedback and iterates overnight.
Built on pi (by @badlogicgames), agent-browser (@vercel, @ctatedev), playwright-cli (@microsoft), and @vite_js Vite+ with direct inspiration from @karpathy's autoresearch.
Clearly none of us are blind and are using SMS-based 2FA. I use SMS instead of an authenticator app because my systems are easily PHYSICALLY compromised. SMS is the only way I've been able to keep my things protected from certain people because I can port the phone number.
@dan_abramov @moshyfawn @TwitterSupport isn't sms 2fa more convenience vs. an actual security feature? My read of this is "if you absolutely need to depend on sms 2fa, you can pay for it" but i'm guessing prob it will get removed completely. Twitter docs explaining how to use app/hardware-based 2fa are good too imo
Excited to announce 14 leading #health providers are uniting to save lives with data with @Truveta. Together, we are building a statistically significant #data platform for US health, including 13% of all US clinical care.
https://t.co/yVJbsLyjxd
The mRNA vaccines (Pfizer and Moderna) are kind of brilliant at a science level. I’ve had a few people in my real non-Twitter life ask me to explain how it works so I’m going to try my best here in this thread while I’m waiting for a patient to show.
1/ In case you were wondering: Apple's replacement for Intel processors turns out to work really, really well. Some otherwise skeptical techies are calling it "black magic". It runs Intel code extraordinarily well.
When I interviewed Tony Hsieh for @HowIBuiltThis he was so soft spoken, modest and understated that I had to literally ask him to “brag” about his incredible achievements. RIP https://t.co/WXzgVWur3K
1) This morning, I shared an enthusiastic tweet about Pfizer's interim results with their COVID-19 vaccine. Let me explain here why am I am so enthusiastic, how the road to get this to people might look like and why we still need to control the pandemic NOW.
Thank you President Elect Biden for mentioning disabled people. Disabled people around the world noticed immediately. This administration will represent ALL the people of the US. We are so proud to be playing a role in the election of the Biden Harris Team.