Advanced · Lesson 5 of 6 · 22 min
Evals and observability
Test prompts and agents the way you test code: success criteria, graded eval sets, LLM-as-judge done carefully, regression gates in CI, and traces that let you debug a production run step by step.
Why prompts need tests
A prompt edit, a model upgrade, a new tool or a changed retrieval index can each change behaviour across thousands of inputs you never look at. Without an eval, the first signal is a client asking why leads are being tagged wrong. Evals turn vague quality into a number you can compare between versions.
Start from success criteria that are specific and measurable: route at least 95% of tickets to the right queue, never promise a refund the policy does not allow, answer within 10 seconds. Then build a test set from real inputs, including the ugly ones: empty messages, mixed languages, angry customers and messages that try to give the model instructions.
You do not need hundreds of cases to begin. Anthropic started its research agent evals with about 20 queries representing real usage; early on, a single prompt change could move success from 30% to 80%, so small sets reveal big effects. Grow the set every time production shows you a new failure.
Grading: code first, then a model, then people
Anthropic's guidance is to pick the fastest reliable grader. Code-based grading, such as an exact match on a label or a check that a required phrase appears, is fastest and most reliable but lacks nuance. Human grading is the most flexible but slow and expensive. LLM-based grading is fast, flexible and scalable, but test it for reliability before you trust it at scale. It also advises favouring volume: more cases with slightly noisier automated grading beat a handful of hand-graded ones.
Design outputs so code can grade them. A classifier that returns structured JSON with a fixed label set can be graded by exact match, with no judge needed. Reserve LLM-as-judge for open-ended outputs such as drafted replies or summaries.
When you use a judge, give it a detailed rubric with concrete pass conditions, let it reason briefly before giving a verdict, and parse a strict final line. Before relying on it, grade 30 to 50 cases by hand and check how often the judge agrees with you; if it disagrees often, fix the rubric before you trust its scores.
const RUBRIC = [
"You grade a customer support reply against the POLICY.",
"PASS only if all are true: it answers the customer's actual question;",
"every policy claim matches POLICY; it promises no refund, discount or date that POLICY does not allow.",
"Explain briefly, then end with a final line that is exactly VERDICT: PASS or VERDICT: FAIL.",
].join(" ");
export async function judgeReply(policy: string, question: string, reply: string): Promise<boolean> {
const response = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 1024,
system: RUBRIC,
messages: [{ role: "user", content: "POLICY:\n" + policy + "\n\nQUESTION:\n" + question + "\n\nREPLY:\n" + reply }],
});
const text = response.content.flatMap((block) => (block.type === "text" ? [block.text] : [])).join("");
const verdict = text.match(/VERDICT: (PASS|FAIL)\s*$/);
if (!verdict) throw new Error("Judge returned no parseable verdict"); // count as an eval error, not a pass
return verdict[1] === "PASS";
}Regression gates in CI
Treat the eval like a test suite. Store cases in version control next to the prompt, run them on every change to the prompt, tools or model id, and fail the build when the score drops below your threshold. Record the score per version so you can see drift over time and answer the question of whether last week's change helped.
Pin the exact model id in production and change it deliberately, running the eval before you switch. Providers retire models on published schedules, so a model upgrade is a planned change with a test run, not something that happens to you.
Open-source tools can save you writing the harness: Promptfoo, DeepEval and the UK AI Security Institute's Inspect all run eval suites, and tracing platforms such as Langfuse and LangSmith attach scores to traces. The harness below is deliberately plain so you can see the whole idea.
import { readFile } from "node:fs/promises";
type EvalCase = { id: string; input: string; expected: "sales" | "support" | "billing" | "spam" };
const cases: EvalCase[] = JSON.parse(await readFile("evals/ticket-routing.json", "utf8"));
const failures: string[] = [];
for (const evalCase of cases) {
const label = await classifyTicket(evalCase.input); // the production function under test
if (label !== evalCase.expected) {
failures.push(evalCase.id + ": expected " + evalCase.expected + ", got " + label);
}
}
const accuracy = 1 - failures.length / cases.length;
console.log("accuracy " + accuracy.toFixed(3) + " on " + cases.length + " cases");
failures.forEach((failure) => console.log("FAIL " + failure));
// Fail CI so a prompt or model change cannot silently regress routing.
if (accuracy < 0.95) process.exit(1);Tracing agents in production
Evals tell you whether a version is good; traces tell you what happened in one specific run. A trace is the end-to-end request, and spans are its timed steps. OpenTelemetry's GenAI semantic conventions give these steps standard names, such as an invoke_agent span with child chat spans for each model call and execute_tool spans for each tool call, and standard attributes such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The conventions are in use today but still marked as in development, so expect some names to change.
Content capture is a decision, not a default. By default the conventions record metadata only, because prompts, tool arguments and results can contain sensitive data. Turn on content capture where you need it for debugging, and apply the same retention and access rules you would to the underlying customer records.
Whatever backend you pick, open-source and self-hostable Langfuse, LangSmith or Arize Phoenix among them, alert on the signals that precede incidents: cost per run, turns per run, tool error rate, refusal and max_tokens stop rates, and latency at the 95th percentile. Then close the loop: when a trace shows a bad run, turn it into a new eval case.
Key points
- ◆Write measurable success criteria and an eval set from real inputs before tuning prompts.
- ◆Prefer code-based grading; design outputs so they can be graded by code.
- ◆Use LLM-as-judge with a concrete rubric and a strict verdict format, and check it against human labels first.
- ◆Gate prompt, tool and model changes on eval scores in CI, and pin model ids in production.
- ◆Trace runs with GenAI spans and token attributes, capture content deliberately, and turn bad traces into eval cases.
Terms in this lesson
Try it · exercise
Put a regression gate in front of a small classifier prompt.
- Write a classifier prompt that labels support messages as sales, support, billing or spam and returns only the label.
- Create 20 cases in a JSON file from realistic messages, including at least four tricky ones (sarcasm, two topics, an instruction aimed at the model).
- Run the harness from this lesson and fix the prompt until it passes your threshold.
- Deliberately weaken the prompt, for example by deleting the label definitions, and run the harness again.
You're done when: The harness prints an accuracy score and lists failing case IDs; with the good prompt it exits with code 0, and with the weakened prompt it exits with a non-zero code that would fail a CI build.