Finished Chapter 4 of AI Engineering by Chip Huyen. It takes the evaluation methods from Chapter 3 and applies them to choosing models and testing an application.
The starting point is defining what a useful result looks like before building.
That means checking the capabilities your task needs, the quality and safety of the output, whether the model follows instructions, and what it costs in money and time.
A summary can sound clear but invent a detail. It can also be faithful to a document that contains incorrect information.
Checking support against a source and checking real-world accuracy are different jobs.
Model selection starts with constraints like privacy, licensing, hardware, and required features.
Then you can shortlist candidates using public benchmarks and test them on your own examples.
Self-hosting gives more control, but it also means paying for engineering and operations. An API can save that work while limiting what you can change.
Public rankings need care too. The benchmarks included, how their scores are combined, and whether their data appeared in training can all affect the result.
A high overall score may hide poor performance on the particular task you're building for.
The evaluation pipeline needs to check individual components as well as the complete user task. If a resume assistant extracts the wrong employer, the mistake might have started when reading the PDF. Scoring only the final answer won't tell you which step failed.
Clear rubrics and realistic test data matter. Look at results by language, input length, or user group instead of relying only on the average.
Check how much scores vary, and track changes to the test set and evaluator so they don't get mistaken for application improvements.
Main takeaway is to connect evaluation to a decision. Is the system useful enough to launch? Which requests still need a person? Is a more expensive model worth the gain? Keep checking those answers as the application and its users change.
I finished Chapter 3 of AI Engineering by Chip Huyen, which covers evaluation methods.
Different scores answer different questions.
A model can score well on one measure and still fail at the task you need it to do.
Perplexity measures how well a model predicts text.
Lower perplexity can be useful when comparing language models under the same conditions, but it doesn't tell you whether an assistant follows instructions or gives a useful answer.
Post-training can improve those behaviors while increasing perplexity.
For tasks like code generation, you can run tests to check whether the output works. Comparing words with a reference is a different check. "The payment was approved" and "The payment was not approved" share almost all their words, but report opposite outcomes.
Even semantic similarity needs care around those differences.
AI judges can help when there isn't a simple test. The judge includes its prompt and settings as well as the model.
If you change the rubric, a higher score might come from a more lenient evaluator rather than a better application.
Judges can also favor longer answers or whichever answer appears first.
Comparing two responses can be easier than scoring each one independently.
That helps with ranking models, but a preferred answer may still be wrong.
Being better than another model doesn't establish that either one is good enough for your use case.
My main takeaway is to decide what a successful result looks like before choosing a metric.
For something testable, check the outcome. For subjective qualities, define the criteria and check the judge too.
I want to know what a score measured before treating it as evidence of improvement.
I got tired of searching for one function in a codebase with 847 matches for the word i was looking for 😭
So i built CodeLens..
you can search by meaning or exact symbol names, jump straight to the code, and even find similar code.
The fun part? it runs 100% offline. no APIs, no cloud, no internet.
Built the whole thing and literally killed it. 🍀
Repo: https://t.co/rQ6oIhmUMT