Your app is fine with 10 users and falls over at 1,000. Somebody says "just add a load balancer" and the explanation ends there.
What part of system design is blocking you right now? Someone else here is probably stuck on the same thing.
Let's connect.
@notaryannnn I'd compare its scores against human evaluations regularly.
Bigger concern is when the judge confidently approves something a human would reject.
Same input, different output is fine for an LLM feature. Until you have to write the test for it.
If you test LLM features, what do you check and how? Exact match, a rubric, another model as judge, eyeballing it?
Would like to connect with people working on this.
Say an agent has 50 customer emails to answer.
Would you approve its plan once, or check each reply before it sends?
The plan tells you the approach, not the exact words. Each-message approval means 50 reviews.
Does your choice change for refund disputes?
Which task would you hand to an agent first, without approving each step?
Drafting a reply leaves you the send button. Booking an appointment commits a time slot.
Before changing models, save the inputs your current one gets wrong.
For invoice extraction, keep a few it gets right too. Run both models on the same set and compare field by field, including anything that got worse.
Which AI tool did you quietly stop opening? You paid for it, told a friend it was great, then it slid off your dock.
What made it fade? I want to follow people who say this out loud.
@aksenHQ Hey, MagnetX looks nice. was searching for the pricing page.
Currently I am using a hermes setup to automate posting. Might be useful for me, will try out tomorrow.
Let's connect!
Bookmark this before your next agent build.
Anthropic says find the simplest solution possible and only add complexity when needed. For many apps, a single LLM call with retrieval and examples is usually enough.
https://t.co/fmYO6IMKzO
What are you building that your non-tech friends would actually use?
How would you explain it to them? Something like "it keeps track of who owes what after a trip."
What do they do instead right now, a spreadsheet, a group chat?