Cold-start notes from an indie product. The sample is a single website, so the conclusions may not generalize, but every number is real. https://t.co/5zFscLfbxC
@Vtrivedy10 Agree on seeding. I'd seed first from my own real failures, not public benchmarks: each becomes a golden case listing what the output must contain and must never invent. Benchmarks teach the shape of an eval; your failures teach what "wrong" means for your task.
The replies to my intro post kept coming back to 3 things:
1. Evals: a small golden set built from real failures, rerun on every change.
2. Did the agent work? Check what it did, not what it claims.
3. Reaching the right people: still figuring it out.
What works for you on #3?
time for a proper intro, let's connect!
I'm Iris. Ex-TikTok PM, spent 7.5 years building platform products. Now a solo builder making AI products in public.
If you're building with AI, deep in evals, or navigating the PM → AI leap
say hi.
Sharing a case study on the trade-offs product managers face regarding time and decision-making, as well as the choices involved in defining an MVP(Minimum Viable Product).https://t.co/iVy1oWGUPg
Big theme on X this week: the harness matters more than the model.
Building an agent solo, I agree. The model guesses. The harness decides whether that guess reaches a user.
Mine is boring: a golden set of real failures, rerun on every change, plus checks against real state.
A demo is an agent working on the tasks you picked. A release is it working on the tasks you didn't.
My bar: a small golden set built from real failures, rerun on every prompt, model, or retrieval change. If it can't pass that, it's still a demo.
What's your bar?
8 years at TikTok/Bytedance. I helped build the platform side, where one product change could reach millions of creators overnight.
Now I'm building alone. No team, no roadmap meeting, ~4 focused hours a day.
The weirdest part so far: at a big co, the hard part is getting alignment. Solo, the hard part is knowing what NOT to build.
Ex-big-co folks who went indie: what was the first habit you had to unlearn?
time for a proper intro, let's connect!
I'm Iris. Ex-TikTok PM, spent 7.5 years building platform products. Now a solo builder making AI products in public.
If you're building with AI, deep in evals, or navigating the PM → AI leap
say hi.
@manxlab Hey Chris, good to meet you. Getting a good product in front of the right people is the part I'm still figuring out too. What are you building?
@rvgdev Still manual, on purpose. A small golden set from real failures: what output must contain, cite, and never invent. Rerun on every prompt, model, or retrieval change. An LLM judge is a first pass. I trust a verifier against real state. A judge mostly agrees with itself.
Solo builder rule I keep relearning: if I can't write down how I'll know it worked, I'm not ready to build it.
At a big company, someone else owned the metrics. Alone, the verifier is me, and I'm the easiest person to fool.
@XiaodaoJiang1 Thanks Xiaodao, following back. Evals are exactly where the platform habit helps, deciding what good looks like before shipping anything.