Evaluation and Cost: Designing a Benchmark Set, Computing Pass Rate, and Reading Token Usage and Cache Hits

Four tests for whether a task belongs in the set

One: a definite initial state. Every task regenerates the sandbox repo first, then arranges it into its own shape if it needs to. Most often broken: run task one, skip the reset, run task two, and task two's result quietly carries task one's edits — with no warning.

Two: an end state a program can decide. The criterion must be something code evaluates to true or false: the test runner's exit code, whether a line is present in a file, whether a file was touched at all. "Answered well" is not a criterion. A benchmark set exists to be repeatable and unattended; the moment a human has to judge, it has decayed back into "pretty good."

Three: a difficulty gradient, including a negative case. If all five tasks are "fix it," you measured only whether the Agent is willing to touch things, and an Agent that edits whatever it sees scores full marks. So task two's correct behavior is to change nothing at all.

Four: it must run offline. Every step has a matching branch in the offline script, so the set runs end to end with no key. An evaluation that only runs with a real key ends up as "we ran it once before release."

The five tasks:

One trap deserves naming, because it arrives with no error message: every time the thing under test grows a new rule, the set may quietly stop being valid. Add a hook that rejects a class of writes, tighten a permission rule, change the wording a skill mandates — and a task whose end state predates that change now encodes behavior the Agent is no longer allowed to produce. Nothing crashes; the pass rate drops, or worse, stays high while measuring the wrong thing. A benchmark set is a maintained artifact: when a rule lands, walk the tasks and ask which end states it just invalidated.

Score the end state, not the process

This is today's one sentence to take away, and the place people most often get it wrong. The end state is what the repo looks like once the task finishes; the process is what the Agent said, how many tool calls it made, and whether any failed on the way. Scoring must not look at the process, for three reasons, each worse than the last.

One: many paths are equally correct. Read then edit, test then edit, edit directly — same result. Scoring the process enshrines whichever path you imagined the day you wrote the set, so a shorter route loses points and the evaluation becomes an exam in resembling you.

Two: a failure inside the process is not a failed task. Task five, already-green, exists for this: the starting point is already fixed, so the first exact-replace necessarily misses and returns a failed tool result. The Agent then runs the tests, they are green, and the task genuinely is done. Scoring the process calls that a loss; scoring the end state calls it a win, and the end state is right. The self-test pins it down: tool failures greater than zero, verdict still pass.

Three, the dangerous one: process scoring degrades into "the model said it was done, so it is done." A regular expression over the model's closing paragraph hunting for "fixed" or "all green" always reports a beautiful pass rate, because models almost never admit they fell short. A scorer that always gives high marks is worse than no scorer at all: it lets you merge a pile of regressions with confidence.

A related discipline: the scorer runs the tests itself, and does not borrow the tools of the thing under test. Judging what run_command produced by calling run_command means any defect in the tool layer — truncation cutting off the failure summary, a timeout read as success — corrupts the scoring too, invisibly. The scorer does the dumbest possible thing: spawn a process, run node --test, believe only the exit code.

export function runTests(repoDir: string): Promise<{ pass: boolean; output: string }> {
  return new Promise((resolve) => {
    // The scorer spawns its own process and trusts only the exit code. Borrowing the
    // tools of the thing under test means a broken tool corrupts the verdict too.
    const child = spawn('node', ['--test'], { cwd: repoDir, stdio: ['ignore', 'pipe', 'pipe'] })
    let output = ''
    const collect = (chunk: Buffer) => {
      output += chunk.toString('utf8')
    }
    child.stdout.on('data', collect)
    child.stderr.on('data', collect)
    const timer = setTimeout(() => child.kill('SIGKILL'), 30_000)
    child.on('close', (code) => {
      clearTimeout(timer)
      resolve({ pass: code === 0, output: output.slice(-2000) })
    })
  })
}

import asyncio
 
 
async def run_tests(repo_dir: str) -> tuple[bool, str]:
    # asyncio subprocesses carry their own timeout, but wait_for does not kill on
    # expiry - do it yourself or the process stays around
    proc = await asyncio.create_subprocess_exec(
        "node", "--test", cwd=repo_dir,
        stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
    )
    try:
        out, _ = await asyncio.wait_for(proc.communicate(), timeout=30)
    except asyncio.TimeoutError:
        proc.kill()
        return False, "the scorer timed out waiting"
    return proc.returncode == 0, out.decode("utf8")[-2000:]

The scorer needs checking too, via the reverse assertion: feed it an input that ought to fail and confirm it reports failure. Self-test item two does exactly that — the untouched sandbox repo must be scored red. Without it, a scorer that runs nothing and returns pass sails green forever; that is the lab's first trap.

Four metrics: pass rate is the conclusion, the other three are the explanation

The fourth is the only one that may not exist: sum the cached field inside each turn's returned usage, divide by the summed input tokens. Two details are easy to get wrong.

The denominator is input tokens, not total tokens. Caching applies to input only; input plus output gives a number that is permanently too small and wobbles with how talkative the model happened to be.

When every cached count is zero, report "unavailable" rather than 0.0 percent. Two possible causes: nothing hit cache, or the gateway you use does not return that field. The code cannot tell them apart, so it must not pretend it can. The offline script does not simulate caching, so that dashboard cell reads "unavailable" with a note pointing at your gateway's documentation. That is not laziness — it is the most common situation in the real world, which is why the branch is worth getting right offline first. How caching engages and how it is billed differs between gateways; go by the documentation of the one you use. This course guesses nothing.

// Denominator is input tokens: caching applies to input only, so using the
// total would make the rate swing with how long the outputs happened to be
const known = cachedTokens > 0
return {
  passRate: attempts.filter((a) => a.passed).length / n,
  avgRounds: sum((a) => a.rounds) / n,
  // All zeroes has two causes: nothing hit, or the gateway does not report it.
  // The code cannot tell them apart, so it must not pretend to.
  cacheHitRate: known && promptTokens > 0 ? cachedTokens / promptTokens : null,
  cacheNote: known ? 'hit rate = cached input tokens / input tokens' : 'unavailable: check your gateway docs',
}

known = cached_tokens > 0
return Summary(
    pass_rate=sum(a.passed for a in attempts) / n,  # booleans sum as 0/1 in Python
    avg_rounds=sum(a.rounds for a in attempts) / n,
    # None rather than 0.0: when "no hits" and "not reported" are
    # indistinguishable, better to withhold the number
    cache_hit_rate=cached_tokens / prompt_tokens if known and prompt_tokens else None,
    cache_note="hit rate = cached input tokens / input tokens" if known else "unavailable: check your gateway docs",
)

One more engineering rule: do not count tokens a second time. The LimitTracker from day six already keeps books for three hard ceilings; cost reporting just feeds the same usage events into the same object. Two ledgers for one sum eventually produce "the ceiling says thirty thousand, the report says twenty-eight thousand," and nobody can say which is right. Same for the tool columns — take the call record the render layer already accumulated.

Cost: only tokens are facts, the money is something you supply

Nineteen days have printed token counts and never a price. Money appears today for the first time, so the terms have to be nailed down.

There is not a single gateway's price anywhere in the code — not one number. Unit prices come from environment variables, defaulting to zero. Three reasons: price lists change, so a price in code is a fact with an expiry date; the same model costs different amounts on different gateways, and this course has refused to bind to any one of them since day one; and the practical one — a default of zero makes "no unit price configured" look like "free," so when it is unset that column reads "no unit price set (cost unavailable, not zero)" rather than a tidy 0.00.

LLM_PRICE_CURRENCY=USD     # just a label, the code converts nothing
LLM_PRICE_IN=1             # per million input tokens, taken from your own bill
LLM_PRICE_OUT=4            # per million output tokens
LLM_PRICE_CACHED=0.1       # per million cached input tokens

For the cached portion, the lab subtracts it out of the input and prices it separately — one common arrangement only, and rules differ, so go by the billing page of the gateway you use. What matters: discuss tokens and relative proportions first, money second. Tokens are what you can directly optimize and they do not expire; convert everything into currency and every conclusion has to be recomputed the next time a price list moves.

Change one thing, run again: the smallest closed loop

With a benchmark set the loop closes: measure a baseline, change exactly one thing, measure again, read the difference. The critical part is exactly one thing: change the prompt and the truncation ceiling together, see an improvement, and you cannot tell which caused it — so next time you carry both forward, and one may be doing harm.

The lab's A/B uses the same five tasks and the same scorer, differing only by one extra sentence at the end of the system prompt: "before editing any file, run the tests once." Offline, one run per task:

Reproducible here are the pass rate, the average turn count, and the stable difference of plus 341 between the two rows (about 1.2 percent). The totals themselves wobble by around 0.1 percent, from exactly one source: the test output fed back in carries the test runner's millisecond timing line. The hit-rate column is permanently unavailable offline, and elapsed time is not reproducible at all, so nothing in the lab asserts on it.

The conclusion may not be the one you expected: an offline A/B measures cost, not effect. The offline script's answers are fixed, so a changed prompt does not make it take a different route, the pass rate cannot move, and the only thing that grew is tokens — 1.2 percent more money for nothing at all.

That is a property of evaluation rather than a flaw in the lab, and today's second takeaway: what an A/B can measure depends on whether the layer under test is actually varying. To measure behavior change you must go through a real gateway. And the moment you do, a third thing shows up.

Variance: run the same task three times before concluding anything

A real model will not do the same task the same way twice: different path, different turn count, different tokens, and now and then a pass where the last run failed. So "pass rate went from 80 to 84 percent after my change" proves nothing on its own — first you need to know how much it wanders when the same code runs three times untouched. If it wanders eight points by itself, a four-point "gain" is noise pointing a convenient direction.

So the set must support repetition. The lab uses the plainest measure: the range as a proportion of the mean. Offline, the range is essentially zero — turn counts identical, tokens differing only by that timing line. The number looks beautiful and means the opposite: near-zero variance offline says this evaluation is not testing the model at all, only your own code. A good regression test, not a model evaluation. Both are useful; do not confuse them:

• Offline evaluation answers "is my code still behaving the way it did." Fast, deterministic, fit for every commit.
• Real evaluation answers "did it get better after the model or prompt changed." Slow, noisy, must be repeated, and costs money every run — all the more reason to get the scorer, the metrics and the dashboard right offline first, rather than debugging your own statistics with paid calls.

The dashboard: the report is data, the display is display

The last step is almost free: each evaluation writes a JSON report to disk, the dashboard runs on local port 3120, and the page fetches that endpoint itself.

One decision is worth stating: why not just build the numbers into the HTML. Because the report is data and the dashboard is one presentation of it. Split apart, the same JSON can be a CI gate (pipeline red when the pass rate drops below a line), the before side of the next comparison, or input to a script of your own — none of which requires parsing a page of HTML. The moment you reshape the report to make the page look nicer, that boundary is gone.

work/evals/r-20260914063251-j2h9.json   one per evaluation; dashboard and gate both read this
http://127.0.0.1:3120/                  static page, fetches the endpoint below itself
http://127.0.0.1:3120/api/runs          the run list plus the latest full report

The dashboard has a closing move: if you start it, you close it. It is a long-running server; leave it up and the process never exits and the port stays held. Calling close() is not enough — the browser's keep-alive connections hold the callback off forever, so existing sockets must be destroyed too. Same rule as parking the child process on day fifteen: anything you start, somebody has to be responsible for stopping.

Source Reading

Hands-On Lab

The lab builds the five tasks, the scorer, the four metrics, the A/B variants and the dashboard in one pass, all runnable offline. All five exercises are traps that never raise an error and only hand you a flattering number: a scorer that always passes, an always-true byte comparison, a second token ledger, a hit rate whose denominator is the total and which prints zero instead of "unavailable," and a fake A/B where the variant never took effect. The starter passes six of fourteen unmodified.

1. Make the scorer actually run the tests, and watch self-test item two go from red to green — it reverse-confirms that the untouched repo really is red.
2. Add the byte-for-byte comparison, and watch the negative-case task finally decide whether a file that should not move has moved.
3. Feed usage into day six's ledger object, and watch the token column stop being zero, with input and output reconciling line by line.
4. Change the hit rate's denominator to input tokens and return "unavailable" when everything is zero, and watch that dashboard cell turn from 0.0 percent into a sentence.
5. Actually splice the variant's sentence into the system prompt, watch the two variants' token totals differ for the first time, then run with --repeat=3 and look at the variance.

Acceptance is five ticks: all fourteen self-test items pass; the untouched repo is scored red; the negative-case task makes zero tool calls with both files byte-for-byte unchanged; task five has a tool failure in the process and an end-state verdict of pass; both dashboard routes respond, and the port is released afterwards.

Interview Questions

Today's three questions test design judgment about benchmark sets and metrics:

1. How do you design a benchmark set for a Coding Agent? Should the scoring function look at the end state or the process?
2. Which metrics would you use to judge an Agent? Why is pass rate not enough?
3. With unstable results across repeated runs of one task, how do you judge whether a change helped under that noise?

Full prompts, analyses and key points are in this course's day-twenty question bank. Question three discriminates most: most answer "run it a few times and average," and few say "measure the baseline's own spread first, then check whether the change exceeds it."

Checklist and Tomorrow

• I can name the four tests a benchmark task must pass, and why the set needs a negative case
• I can explain why scoring looks only at the end state, and give an example of a failed process with a passing end state
• I can say why "the model said it was done" is the most dangerous kind of scoring
• I can say why the scorer must not borrow the tools of the thing under test, and what a reverse assertion is for
• I can state what each of the four metrics answers, and why pass rate alone will lie
• I can say that the cache hit rate's denominator is input tokens, and why all-zero must report "unavailable"
• I can explain why unit prices must be supplied by the reader, and why a default of zero must not read as free
• I can say why an A/B changes exactly one thing, and what a fake A/B looks like
• I can explain what offline and real evaluation each answer, and why near-zero offline variance is not good news
• I can say why the report and the dashboard split into a data layer and a display layer

Tomorrow is the last day, D21, "Packaging and Release: a Global Command, a Config Directory, Versioning and Updates — a Twenty-One-Day Retrospective." Today's set has one more use there: run it before you release, so that "it installs and completes one real task" also becomes a judgment with an exit code, instead of one more round of "it worked on my machine."