5 things I wish I knew before shipping my first LLM feature 🧵
Log every prompt and response from day one
Users will break your assumptions in the first hour
Cheap models are great for simple steps, so don’t use the big one for everything
Structured outputs save you from parsing nightmares
Write evals before you need them
what’s the most unexpected thing your LLM app did in production? I’ll start: it politely apologized to a user for a bug that didn’t exist
devs building with AI: what’s the one tool or habit that saved you the most time this year?
shipped a small tool today that turns messy meeting notes into clean action items. 200 lines of code, one API call, three weeks of edge cases. that ratio is the whole job
day 14 of building in public: users found a bug I never would have. one pasted an entire novel into the input box. the app handled it. my wallet did not
lesson learned this week: always set a max token limit and a budget alert. past me says thanks, current me is still sweating
hot take: most “AI apps” are 10% model and 90% handling the weird stuff users type into a text box
your eval set matters more than your prompt. if you can’t measure improvement, you’re just rearranging vibes
unpopular opinion: the best prompt engineering advice is “say clearly what you want.” the second best is “show an example”
spent 4 hours debugging my prompt. the issue was that I wrote “respond in JSON” and the model responded in JSON, wrapped in a friendly paragraph about how it was responding in JSON
my API bill this month: 😐
me: “it’s fine, it’s an investment”
my API bill next month: 😐😐😐
the stages of building with LLMs:“this is magic”
“why is it doing that”
“I will add ‘be careful’ to the system prompt”
“it is still doing that”