Tagged
#evaluation
12 articles
Test your prompts like code
A prompt is code that ships to users. Treat it that way — with test cases, a baseline, and a regression check before every change.
What a "frontier model" actually means — and why benchmarks mislead
"Frontier model" is a moving label, not a spec. Here is what it really points to, why leaderboard scores rarely tell you what you need, and how to choose well anyway.
Prompt engineering fundamentals that still matter
Trends in prompting come and go. A small set of fundamentals keeps working across models and releases. Here they are, with the reasoning behind each.
Document Q&A that actually works: patterns and pitfalls
Asking questions over your own documents is the most useful AI demo and one of the easiest to get quietly wrong. Here are the patterns that survive real use.
Choose the right model size for a task
Bigger is not always better. A practical method for picking a model size that matches the task, the budget, and the latency you can live with.
Set up a feedback loop to improve answers
An AI feature that never learns from its mistakes stays stuck. How to capture signal, turn it into examples, and close the loop that makes answers better.
Evaluation beyond benchmarks: human and model judges
Benchmarks measure what is easy to score. For open-ended work you need judgment — from people, or from a model standing in for them. Both can mislead.
How models are evaluated: benchmarks, and why they lie
Benchmark scores look like measurements, but they are arguments. Here is how model evaluation actually works, and why a high number can still mislead you.
Reduce hallucinations: a practical checklist
Models invent facts when the task invites them to. This checklist covers the moves that cut hallucinations without pretending you can eliminate them.
Measuring quality: how to set up a basic eval
Vibes don't scale. A small, honest evaluation turns 'this feels better' into a number you can trust — here's how to build one from scratch.
Evaluating AI tools: a checklist that survives the demo
AI tools are designed to dazzle in a demo. This checklist helps you judge them on the durable questions that decide whether they hold up in real use.
Emergent abilities: real or mirage?
Big models seem to suddenly "get" skills smaller ones lack. Is that a real phase change, or a trick of how we measure? The honest answer is: both.