Luis Henrich-Bandis

Building OptBench: benchmarks for teams shipping AI agents. Founder of StudyPDF (140k+ students). MSc AI at Imperial. Posting about benchmarks and evals.

[Luis Henrich-Bandis @luisnhenrich]

@brennanzambo with that many starter workflows, how do you validate the review checks? are you testing them against actual outputs, including ones that should be rejected?

[Luis Henrich-Bandis @luisnhenrich]

@Aditya_pate_l what made you pick the model for the WhatsApp receptionist? curious how you compared answer quality, response time and cost before putting it in front of customers.

[Luis Henrich-Bandis @luisnhenrich]

@therobertta_ when you take an agent to production, how do you choose which traces become eval cases? curious whether you include normal traffic alongside reported failures.

[Luis Henrich-Bandis @luisnhenrich]

@itsHimelz what are the yes or no calls in your agent loops running on today? curious what you'd test before moving them to a cheaper model.

[Luis Henrich-Bandis @luisnhenrich]

@StoryHelm with 61 agents reviewing manuscripts, how do you check that the contradictions they flag are real? curious whether you keep manuscripts with known issues as a test set, or mostly learn from author feedback.

[Luis Henrich-Bandis @luisnhenrich]

@vsaietta what will you check the Gemini Decision swap against? curious whether you have F24s you've already sorted by hand to compare it on.

[Luis Henrich-Bandis @luisnhenrich]

@Khalil_Preview versioning the prompts makes changes traceable. how do you check whether a new version is actually better before deploying it?

[Luis Henrich-Bandis @luisnhenrich]

@Abbylester when an edge case shows up after launch, does it become a saved case you rerun on the next change? curious how you manage that across client projects.

[Luis Henrich-Bandis @luisnhenrich]

@XiaodaoJiang1 what are most of your small calls actually doing? curious what examples you'd test ember on before switching, beyond the overall index score.

[Luis Henrich-Bandis @luisnhenrich]

@hailports the blind comparison is useful. how does your router decide a request needs the premium model rather than the local one, and how are you testing that decision?

[Luis Henrich-Bandis @luisnhenrich]

@themuscleai curious about the next step for the ticket router: once the llm labels the escalated tickets, how will you check those labels before using them for training?

[Luis Henrich-Bandis @luisnhenrich]

@SparkTsaiX in the apps that passed their tests but weren't usable, what was the first thing a real user hit? curious whether those failures then became checks for the next version.

[Luis Henrich-Bandis @luisnhenrich]

@SrichandanKota the release gates across 10 agents and 73 tools caught my eye. what did a gate actually check: end task success, individual tool calls, or both?

[Luis Henrich-Bandis @luisnhenrich]

@AditSavani16 how do you decide the smallest model is still good enough for a feature? curious what you actually test before switching, and how much time that takes.

[Luis Henrich-Bandis @luisnhenrich]

@JunaidKFarooqi @hasantoxr half filled forms and changed minds are hard to invent up front. do those cases get saved as tests when a real user hits them, or do you mostly find them by trying to break it yourself?

[Luis Henrich-Bandis @luisnhenrich]

@_aminparth when you audited your own apps, how did you check which calls could move to a smaller model without losing quality? curious what the side by side testing looked like in practice.

[Luis Henrich-Bandis @luisnhenrich]

@Super_PK_ for the dog health alerts, how do you check whether an alert was right? do you get vet outcomes back, or is it mostly owner feedback so far?

[Luis Henrich-Bandis @luisnhenrich]

@mehulmpt clicking the link and then saying it would never sign in is quite a gap between actions and explanation. has that incident become a test you rerun when changing the inbox agent's model or permissions?

[Luis Henrich-Bandis @luisnhenrich]

@Yashcsp22 for the voice agents, where do the golden cases come from? curious how much is built from real calls versus scenarios the team writes.

[Luis Henrich-Bandis @luisnhenrich]

@DhruvJain08 and run that eval on opus at higher effort first, since that is the cheap comparison they are pointing at. if opus on high already clears the bar, there is no reason to pay fable prices for that job.