Agent Evals and Observability in 7 Days: Turning "Seems Fine" Into Numbers You Can Reproduce

Why You Cannot Ship an Agent on "I Tried It a Few Times"

Start with the three things that make agents harder to evaluate than single-turn systems, settle the five terms this course runs on, and then turn non-determinism into two numbers you can put in a report: the odds of succeeding at least once, and the odds of succeeding every single time.

Benchmark Sets: Turning the Incidents You Already Had Into Reproducible Tasks

The ceiling of an evaluation is set by task quality, not by the metric formula. Today you mine support tickets and failure logs for tasks, settle what makes a task good, add the negative cases almost everyone forgets, and validate every item with a reference solution before it ever judges an agent.

Let a Model Judge, Then Put the Judge Through an Exam

Anything code can decide should never go to a model, but open-ended quality genuinely has to. Today you write rubrics and pairwise comparison, measure the judge's three systematic biases with a position-swap experiment, and decide whether it is fit to use with an agreement coefficient rather than accuracy.

Trajectory Evaluation: It Reached the Destination, but How Many Things Did It Hit

Looking only at the outcome lets a whole class of failures through: the answer was right, but it cost ten times as much, took seven detours, and touched a tool it had no business touching. Today you score the trajectory itself with tool-sequence checks, loop detection and step budgets, then extend single-turn evaluation to multi-turn with a simulated user.

Observability: Putting a Flight Recorder on Every Run

Offline evaluation only covers the tasks you thought of; production is where the real users are. Today you instrument the target against the open generative AI semantic conventions, turn one run into a queryable span tree, aggregate cost and latency, and report the evaluation scores themselves as telemetry.

The Regression Gate: Blocking a Drop Before It Merges

An evaluation only prevents incidents once it runs in the pipeline. Today you freeze a baseline into a snapshot, set a threshold on a score that moves every run without either missing regressions or crying wolf daily, and solve the two problems that actually bite: what a run costs, and who decides whether a red build is real.

Examining the Suite Itself: Saturation, Broken Tasks and Grading Defects

The last day turns the lens on the evaluation system. A suite approaching a perfect score has stopped carrying information, and a single mis-written grader can make a good model look fifty points worse. Today you health-check the suite, establish the discipline of reading transcripts, and assemble seven days of work into a portfolio project.