Productionizing and Retrospective: Writing Evals for Tools, Versioning, Publishing to npm and a Registry, Observability, and a Capstone Project

How to write an eval set

An eval set has one design point: all three categories of case are mandatory.

Everybody writes the first two, and the third is the watershed.

An eval set of positives alone gives you the illusion of 100%: it cannot detect calling when nothing should be called, and most production complaints are exactly that — the user says offhand "thanks, no need to look in the docs" and the assistant goes and searches the docs. That is over-triggering, and it hurts the experience more than a missed call because it has side effects.

How do you pick trap cases? Find phrasings adjacent in topic but different in intent. "Thanks, no need to search the docs" (contains search and docs, intent is negation), "how do you say documentation in French" (contains documentation, is a language question), "what can you do" (small talk, and calling a tool is wrong).

Then the assertion. There is one assertion and it judges in both directions:

// When a tool is expected, only selecting it passes; when no call is expected, only selecting nothing passes.
// Treating null as "anything goes" is the most common authoring error in an eval set — it makes negatives
// permanently green, which is worse than having none, because you believe you tested.
export function judge(expected: string | null, actual: string | null): boolean {
  return expected === null ? actual === null : actual === expected
}
 
// All three categories are mandatory, and this throws rather than logging a warning:
// an eval set is a line of defense, and a broken defense must stop the line
export function loadCases(raw: { cases: EvalCase[] }): EvalCase[] {
  for (const kind of ['happy', 'edge', 'trap'] as const) {
    if (!raw.cases.some((c) => c.kind === kind)) throw new Error(`the eval set has no ${kind} cases`)
  }
  return raw.cases
}

# When a tool is expected, only selecting it passes; when no call is expected, only selecting nothing passes.
# Treating None as "anything goes" is the most common authoring error in an eval set — it makes negatives
# permanently green, which is worse than having none, because you believe you tested.
def judge(expected: str | None, actual: str | None) -> bool:
    return actual is None if expected is None else actual == expected
 
 
# All three categories are mandatory, and this raises rather than logging a warning:
# an eval set is a line of defense, and a broken defense must stop the line
def load_cases(raw: dict) -> list[EvalCase]:
    cases = raw["cases"]
    for kind in ("happy", "edge", "trap"):
        if not any(c["kind"] == kind for c in cases):
            raise ValueError(f"the eval set has no {kind} cases")
    return cases

One practical recommendation: an eval set must run in one command and finish in under a minute. A slow eval set is no eval set, because whoever edits a description will not wait. Run it on a cheap small model such as claude-haiku-4-5-20251001; picking a tool does not need the most expensive one.

Versioning: changing a description is more dangerous than renaming a field

Look at how the protocol versions itself first. MCP's version number is of the form YYYY-MM-DD, with an unusual meaning: it marks the date of the last breaking change. Backward-compatible changes do not advance it, so one version number can keep evolving for a while. Deprecation has an explicit policy too: a feature marked deprecated stays in the spec for at least twelve months (at least ninety days even through the expedited path) before it may be removed.

That practice is worth copying. Your server should have its own list of what counts as breaking, and the definition given in the official material on extension evolution can be used directly: removing or renaming a field, changing a field's type, changing the semantics of existing behavior, and adding a required field are all breaking.

But a tool has one more category than an ordinary API, and it is the one most easily missed:

Changing one sentence of a description is a behavioural change.

The incident above already explained why: the description is the model's only basis for selecting a tool. Renaming a field makes callers error immediately — an error is loud and somebody finds you within five minutes. Changing a description errors nothing; it quietly drops the selection rate by a few points and surfaces two weeks later as "it has got dumber lately." A loud error is far easier to handle than a quiet degradation, so a description change needs a gate more than anything else.

The gate is the eval set from the last section: a changed description must run it, and a dropped selection rate means no merge.

Compatibility technique at the field level is the same as for any API, with two MCP-specific rules:

• Add fields as optional, because older clients will not send new arguments.
• Change the name when changing semantics, rather than editing in place. Mark the old one deprecated, name the replacement in its description, and leave it for a while before deleting — you do not know how many people hard-coded that name into a prompt.

The approach for extensions follows the same thinking: prefer a capability flag or a version field in the extension's settings when a change is needed, and when a breaking change is unavoidable, use a new extension identifier.

Publishing: npm and the registry are two different things

Clear up a relationship most people get wrong the first time: the official registry stores metadata only, not artifacts.

Artifacts live in a package registry — npm, PyPI, Docker Hub. The official MCP registry stores a server.json: what this server is called, where to find it (which npm package, or which remote address), how to start it, and which environment variables it needs. It is positioned for downstream aggregators (the various MCP marketplaces) to consume rather than for hosts to query directly.

So publishing is publish the package first, then register, and the order cannot be reversed.

Three things must line up, and getting one wrong means rejection:

The namespace is verified by ownership: authenticating with GitHub, the name must start with io.github.yourusername/; using your own domain goes through DNS verification. That is also the registry's main defense against impersonation.

That version rule needs care: one version number can be published only once, and metadata cannot change after publication. One typo means bumping to a new version. And strings that look like ranges — ^1.2.3, 1.x, >=1.2.3 — are rejected outright, which is a deliberate safety catch. Semantic versioning is recommended, and a local server should keep server.json's version aligned with the package version so the two do not disagree.

A few things about the package itself: it needs an executable entry point (bin), or users cannot start your server with npx; files must include the build output directory; and the README's first screen must state what it is, how to install it, and which environment variables it needs — the reader's budget is three minutes and they do not intend to clone and run it.

Observability: check the error rate first, then the selection rate

Once a server is live there are only a handful of questions you need answerable. Few, accurate metrics beat a pile of unwatched curves.

Four metrics suffice:

1. Calls per tool — which tool carries the load and which was never called (a tool never called should either be deleted or has a description problem).
2. Error rate per tool — and count the two error classes separately: a protocol error is your bug and a tool execution error is the model supplying bad arguments. Persistently high rates of the latter usually mean not that the model is dim but that your schema descriptions are unclear.
3. Latency distribution per tool — look at P95, not the mean. On a remote server, a tool degrading from 200 milliseconds to 8 seconds may barely move the mean.
4. Selection rate — the eval set's score, run regularly and watched as a curve.

The investigation order is fixed too: check the error rate first, then the selection rate. A normal error rate with users complaining usually means it is not being selected; only a spiking error rate sends you to the code and upstream.

For tracing, the protocol provides the place without prescribing the rules: a request's _meta is an open field whose keys must carry a prefix (a reverse domain, the same naming rule as extension identifiers). So passing trace context through inside your own _meta key is both conformant and general.

// Client: put the trace context into a _meta key under your own namespace.
// The key must carry a prefix (reverse domain), the same naming rule as extension identifiers
const TRACE_KEY = 'com.example/trace'
 
function withTrace(params: Record<string, unknown>, traceparent: string) {
  return { ...params, _meta: { ...(params._meta as object), [TRACE_KEY]: { traceparent } } }
}
 
// Server: pull it out and attach your span; start a fresh one when absent rather than erroring on a missing field
function startSpan(params: Record<string, unknown>, toolName: string) {
  const meta = params._meta as Record<string, { traceparent?: string }> | undefined
  const parent = meta?.[TRACE_KEY]?.traceparent
  return tracer.startSpan(`tools/call ${toolName}`, { parent })
}

# Client: put the trace context into a _meta key under your own namespace.
# The key must carry a prefix (reverse domain), the same naming rule as extension identifiers
TRACE_KEY = "com.example/trace"
 
 
def with_trace(params: dict, traceparent: str) -> dict:
    meta = {**params.get("_meta", {}), TRACE_KEY: {"traceparent": traceparent}}
    return {**params, "_meta": meta}
 
 
# Server: pull it out and attach your span; start a fresh one when absent rather than erroring
def start_span(params: dict, tool_name: str):
    parent = params.get("_meta", {}).get(TRACE_KEY, {}).get("traceparent")
    return tracer.start_span(f"tools/call {tool_name}", parent=parent)

One last thing to trip over: on stdio a server's logs may only go to stderr. To view metrics, open a separate HTTP endpoint rather than printing to stdout — only protocol messages belong there, and one extra character makes the client fail to parse.

The extension ecosystem: three pieces beyond the core

There is a layer beyond the spec called extensions. They are optional, identified by a reverse domain such as io.modelcontextprotocol/, negotiated in both sides' capability declarations, and off by default and enabled explicitly. When one side supports an extension and the other does not, the supporting side either falls back to core behavior or errors explicitly.

Three official ones exist today, each solving something the core protocol does not:

• The authorization extension: the machine-to-machine client credentials flow and centralized access control for enterprises. The core OAuth set assumes a user sitting at a browser, and these two cover "no user" and "there is an IT administrator."
• Application interfaces (MCP Apps): letting a server render an interactive interface in the host — charts, forms, players. A tool returning text has limits.
• Tasks: long-running asynchronous tasks with polling, supplemental input midway, and durable handles. A twenty-minute job should not occupy one request.

One more is on the way: distributing skills over MCP, currently a working group and not yet a published official extension. To use skills today, the route remains their own directory convention — which is another course's home turf.

To decide whether to touch extensions, one sentence: look only when the core protocol cannot solve it and you genuinely have that problem. Extensions being off by default is reasoned, and every one you enable adds a layer of compatibility burden.

Retrospective: what you hold after this week

Put this course back on the map first. Three things in agent engineering get conflated constantly, separated in one line:

MCP handles the wiring, Skills handle the experience, and context engineering handles the trade-offs.

MCP solves how tools and data get wired in; Agent Skills solves turning working experience into reusable capability; context engineering solves what belongs in the context window. The three do not conflict and cannot substitute for one another. This week you learned the first.

Then count the spoils. After seven days you should hold these:

The suggestion for a portfolio is to combine D4's server, D5's client, and D7's eval and publishing configuration into one repository, with an architecture diagram, a quick start that runs in three commands, and a key design decisions section in the README. That last section is the only part nobody can copy and where interviewers pick their follow-ups — every decision must state what was given up.

As for every number, write only what you measured yourself and note the conditions. "Two replicas locally, offline mode, 9 of 9 self-checks green" is plain and stands up; write one unmeasured scale figure and you invalidate not just that line but every genuine number you gave.

Source Reading

Hands-On Lab

The template's three tools are deliberately confusable: search, read by slug, and create. The lines between them rest entirely on a few sentences of description, which makes them a perfect target for an eval set. Run as-is, starter shows 3 green and 4 red.

1. Run the solution's eval once to see what the score table looks like, particularly why those three trap cases should be judged as no call.
2. In the starter, fix judge's assertion first and then add the negatives — reverse the order and every negative you add will be falsely green.
3. Add metrics for all three paths to the server's tools/call branch, not omitting the isError-true one.
4. Complete the publish check, then deliberately break mcpName once and confirm it really does report it.
5. Finally run pnpm build and npm pack --dry-run, and check the manifest for dist and server.json.

If you have a key, drop MOCK=1 and run the eval once against a real model, then change one sentence of a tool description and run it again — that alone makes it obvious why a description change counts as a breaking change.

Interview Questions

Today's 3 questions are in the question bank below, weighted toward designing tool evals, which changes count as breaking, and the order of investigation in production. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Write a re-runnable eval set for a tool, and name the three categories of cases it should cover
• Judge whether a change is breaking, and explain why a description change counts as one too
• Add tracing and key metrics to a server, and say which one to check first when something goes wrong
• Explain the division between npm and the official registry, and the three fields that must line up
• All 5 acceptance criteria of the lab pass
• Answer at least 2 of the 3 interview questions without looking at the key points

Seven days end here. Look back at day 1's question — what makes those tools usable by somebody else's program — and your answer is no longer one sentence but a server that runs, a client that connects, a security checklist you filled in, and an eval set that catches degradation.

Where to go next depends on what you lack: to turn how-to-do-things experience into something reusable, take Agent Skills; if wiring in more tools left you short of context, take context engineering. Wiring, experience, and trade-offs — with all three in hand, you can genuinely build agents.