A Design Method: Distilling From Repeated Tasks, Checklist Style vs. Reference-Manual Style, Four Anti-Patterns, and Trigger Testing

Distilling from one real conversation

The correction-exists criterion is not only a filter; it is also the content's direct source. The method is do it once, then distil: work through the task with the model end to end, correcting it normally throughout, then look back over that conversation and copy out four kinds of thing.

First, the sequence of steps that worked. Write down, in order, the path that eventually went through. Note that it is the path that worked, not every path you tried — branches abandoned midway do not belong in the skill, where they would only give the model more options.

Second, every correction you made. "Use this library, not that one." "Check for null here." "Wrong order, validate before writing." Those are the densest content in the whole skill, because they mark precisely the gap between the model's default behavior and your expectation.

Third, what the input and output actually look like. What format the input file is in and what the output should look like. Paste samples for this rather than describing them in prose.

Fourth, the background you supplied. Facts the model could not possibly know and that you told it on the spot: this field has three different names across three systems, this health check endpoint only checks the process is alive and not whether the database is reachable. Facts like these are extremely valuable, and they are exactly the raw material for yesterday's pitfalls section.

Beyond the conversation there is one badly underrated source: material that already exists in your project. Internal documentation and runbooks, endpoint specs and configuration files, review comments that keep recurring, and the fix commits in the history. That last one is especially valuable — it records what was actually changed rather than what was meant to be done. A data pipeline skill distilled from real incident records will always beat one generated from a "data engineering best practices" article.

Two forms: checklist style and reference-manual style

With the material gathered, the organization comes next. Mature skills land in one of two forms, and getting it right makes writing flow while getting it wrong makes everything awkward.

Checklist style suits tasks where steps depend on each other, the order cannot change, and missing one breaks things. The body is mainly a numbered list of steps, often with a progress checklist for the model to tick. Release procedures, form filling, and data migrations all belong here.

This form has a very useful variant: plan first, validate, then execute. Have the model produce an intermediate plan (a file mapping fields to values, say), run a validation script comparing that plan against ground truth, and execute for real only once it passes. The key is not the first or third step but the validation in the middle — with an error message stating "the field signature_date does not exist; available fields are…," the model has enough information to fix it itself. Bulk operations and destructive operations should all take this route.

Reference-manual style suits knowledge-dense tasks consulted on demand with no fixed order. The body is mainly a collection of what-to-do-when entries, plus a set of reference files cut small. Handling endpoint error codes, looking up format specs, and domain terminology mappings all belong here.

The criterion between the two is one sentence: does this thing have an order that must be followed? If yes, checklist style; if no, reference-manual style. Mixing them is common, but mix by section rather than listing steps and piling up entries within one section — that reads like a manual whose table of contents got shuffled.

Four anti-patterns

The four below are what beginners most often produce, and each is recognizable in one sentence.

First, covering everything. One skill tries to cover all the work in a domain: a "database skill" handling queries, index creation, and backup and restore. Its problem is that it cannot be triggered precisely — the description has to be written broadly enough to cover all that, and broad means spuriously triggered everywhere. The test: if you cannot resist words like "and so on" or "and related matters" while writing the description, the scope is already too large. Split it.

Second, shattered into fragments. The reverse: one thing split into five skills, one reading files, one validating, one converting. Its problem is that doing one thing activates five at once, five bodies enter the context together, and they may even contradict each other. The test: if two skills are almost always used together, they were one skill.

The scale for splitting and merging is the same as for splitting functions: a skill should encapsulate one cohesive unit of work and compose with other skills. "Query the database and format the results" can be one unit; "query the database and also administer the database" is two things.

Third, restating common knowledge. The body spends paragraphs explaining what a PDF is, how HTTP works, what a database migration means. The model knows all of it, and writing it in only dilutes attention. The test is the same line: without this, would the model get it wrong? If not, delete it.

Fourth, a menu with no default. "You may use any of the four libraries A, B, C, or D." Handed a pile of peer options, the model can only pick one, and it may pick differently each time. The right form is one default plus one fallback: "use A; switch to B for scanned documents going through OCR." This one often gets mistaken for preserving flexibility, when it is really offloading the decision cost onto the model.

A related point while we are here: a skill is not better for being more complete. A skill covering every edge case tends to make the model waste steps on irrelevant instructions — reading a rule that does not apply to the current task, it will still try to comply. Concise step-by-step guidance plus one usable example usually beats exhaustive documentation. When you notice yourself covering the seventh edge case, ask first: would this one be better left to the model's judgment?

Trigger testing: replacing feel with numbers

Now to today's hardest section. Everything above was about how to write; this is about how to know you wrote it right.

Yesterday you tried three sentences, which is spot-checking. Turning it into a ruler takes three things.

First, a set of labeled queries. About 20, of which 8 to 10 should trigger and 8 to 10 should not. The should-trigger batch has to spread across four dimensions: phrasing formal, casual, and even with typos; some naming the domain directly and some only describing the need; some very short and some carrying a pile of file paths and background; some single-step tasks and some burying this thing inside a long stretch of other work. The most valuable positive cases are the queries where the skill genuinely should be used but the wording does not say so — a query that recites the skill's function verbatim will be matched by any description and measures no difference at all.

The should-not batch takes even more design. A sentence with no overlap at all, like "what is the weather today," measures nothing. What is genuinely useful is the near-miss negative: sharing keywords or concepts with the skill while actually needing something else. For a CSV analysis skill, "help me fix a formula in my Excel budget sheet" and "write a script to push each CSV row into the database" are both good negatives — the first is Excel editing rather than CSV analysis, and the second is data movement rather than analysis.

Second, repeated runs. A model's behavior is non-deterministic, and the same query may trigger this time and not the next. Run each 3 times and compute a trigger rate: triggers divided by runs. A should-trigger query passes above 0.5, and a should-not query passes below 0.5. Twenty queries times 3 runs is 60 calls, which at that scale demands a script.

Third, a train and validation split. This is what beginners most often skip and most often pay for. Tune the description against all 20 and you will produce a description tuned specifically for those 20, which collapses on a fresh batch. The method is to split the query set roughly sixty-forty: use only the training set to find problems and guide rewrites, and never look at the validation set, then use the validation pass rate at the end to pick which version of the description is best. Keep the positive-negative ratio close in both halves, and fix the split once made rather than reshuffling every round.

Judging whether a query triggered is the core of this apparatus. The principle is surprisingly simple: assemble every skill's name and description into a catalog, hand it to the model along with the user's sentence, and ask which one to use or none. That is what a client does in stage one, and we are simply pulling it out to run on its own.

type SkillMeta = { name: string; description: string }
 
// This prompt deliberately mirrors what a client does in stage one: names and descriptions only, no bodies
function buildJudgePrompt(skills: SkillMeta[], query: string): string {
  const catalog = skills.map((s) => `- ${s.name}: ${s.description}`).join('\n')
  return [
    'Below is the catalog of available skills, each with only a name and a description.',
    catalog,
    '',
    `The user says: ${query}`,
    '',
    'Answer with one skill name only; if none clearly applies, answer none. Do not explain.',
  ].join('\n')
}
 
export async function judgeOnce(skills: SkillMeta[], query: string, target: string): Promise<boolean> {
  if (process.env.MOCK === '1') {
    // Offline simulation: approximate the model's judgment by keyword overlap so the chain runs without a key
    const hit = target.split('-').some((w) => query.includes(w))
    return hit
  }
  const answer = await callModel(buildJudgePrompt(skills, query))
  return answer.trim().toLowerCase() === target.toLowerCase()
}
 
// Run it `runs` times for a trigger rate, because the model's judgment is itself non-deterministic
export async function triggerRate(skills: SkillMeta[], query: string, target: string, runs = 3) {
  let hits = 0
  for (let i = 0; i < runs; i++) if (await judgeOnce(skills, query, target)) hits++
  return hits / runs
}

import os
from dataclasses import dataclass
 
 
@dataclass
class SkillMeta:
    name: str
    description: str
 
 
# This prompt deliberately mirrors what a client does in stage one: names and descriptions only, no bodies
def build_judge_prompt(skills: list[SkillMeta], query: str) -> str:
    catalog = "\n".join(f"- {s.name}: {s.description}" for s in skills)
    return "\n".join(
        [
            "Below is the catalog of available skills, each with only a name and a description.",
            catalog,
            "",
            f"The user says: {query}",
            "",
            "Answer with one skill name only; if none clearly applies, answer none. Do not explain.",
        ]
    )
 
 
def judge_once(skills: list[SkillMeta], query: str, target: str) -> bool:
    if os.environ.get("MOCK") == "1":
        # Offline simulation: approximate the model's judgment by keyword overlap so the chain runs without a key
        return any(w in query for w in target.split("-"))
    answer = call_model(build_judge_prompt(skills, query))
    return answer.strip().lower() == target.lower()
 
 
# Run it `runs` times for a trigger rate, because the model's judgment is itself non-deterministic
def trigger_rate(skills: list[SkillMeta], query: str, target: str, runs: int = 3) -> float:
    hits = sum(1 for _ in range(runs) if judge_once(skills, query, target))
    return hits / runs

With the trigger rate in hand, what remains is statistics and the split. The code below runs a set of labeled queries into a table: expectation, trigger rate, and pass or fail per row, then the pass rate summarized separately for the training and validation sets. The key is computing the two sets separately — the training numbers guide your edits and the validation numbers decide which version you ship.

type Case = { query: string; shouldTrigger: boolean; split: 'train' | 'validation' }
 
export async function evaluate(skills: SkillMeta[], target: string, cases: Case[]) {
  const rows = []
  for (const c of cases) {
    const rate = await triggerRate(skills, c.query, target)
    // Threshold 0.5: should-trigger must be above it, should-not must be at or below it
    const passed = c.shouldTrigger ? rate > 0.5 : rate <= 0.5
    rows.push({ ...c, rate, passed })
  }
  const rateOf = (split: Case['split']) => {
    const subset = rows.filter((r) => r.split === split)
    return subset.length === 0 ? 0 : subset.filter((r) => r.passed).length / subset.length
  }
  // Training numbers guide the rewrite; validation numbers decide which version you ship
  return { rows, train: rateOf('train'), validation: rateOf('validation') }
}

from dataclasses import dataclass
 
 
@dataclass
class Case:
    query: str
    should_trigger: bool
    split: str  # "train" or "validation"
 
 
def evaluate(skills: list[SkillMeta], target: str, cases: list[Case]) -> dict:
    rows = []
    for c in cases:
        rate = trigger_rate(skills, c.query, target)
        # Threshold 0.5: should-trigger must be above it, should-not must be at or below it
        passed = rate > 0.5 if c.should_trigger else rate <= 0.5
        rows.append({"case": c, "rate": rate, "passed": passed})
 
    def rate_of(split: str) -> float:
        subset = [r for r in rows if r["case"].split == split]
        return len([r for r in subset if r["passed"]]) / len(subset) if subset else 0.0
 
    # Training numbers guide the rewrite; validation numbers decide which version you ship
    return {"rows": rows, "train": rate_of("train"), "validation": rate_of("validation")}

Once you have the results, there are two hard rules for editing. A should-trigger query that did not trigger means the description is too narrow, and what you add is the class of phrasing rather than that one sentence — copying the failing query's exact words into the description is where overfitting begins. A should-not query that triggered means the description is too broad, and what you add is a boundary: state explicitly what it does not do, or spell out the line between it and neighboring capabilities.

Two more lessons worth recording. Stop after about five rounds: if the pass rate stops moving, the problem is most likely the query set itself (too easy, too hard, or mislabelled) rather than the description. The best version is not necessarily the last: the later rounds are quite likely overfitting to the training set, so pick the version by validation pass rate rather than by iteration order.

Source Reading

Hands-On Lab

There is real code to write today. Before starting, have yesterday's commit-message skill description ready, since it is the subject under test. When stuck on inventing near-miss negatives, go back to the two CSV examples in the trigger testing section and build from them — shared keywords, different target verb, which is the only criterion.

1. Get the solution running and use MOCK=1 to see the per-row hit table and the two pass rates from a labeled query set.
2. Complete the description-matching judgment in the starter so it returns a hit or miss plus one line of reasoning per query.
3. Complete the trigger rate statistics and the separate summaries for the training and validation sets.
4. Add three near-miss negatives of your own, rerun, and observe how the trigger rates change.
5. Rewrite the description per the training set's failures, run again, and compare the two versions' validation pass rates.

Interview Questions

Today's 3 questions are in the question bank below, weighted toward the criteria for choosing a skill's subject, the trade-off between the two content forms, and the design of trigger testing and overfitting. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Judge whether a repeated task is worth turning into a skill, using three criteria
• Distinguish checklist style from reference-manual style, and pick the right form for a task
• Design a set of positive and negative example queries, measure description's trigger rate, and rewrite it accordingly
• Name the four anti-patterns on the spot, with one example each you have written or seen
• All 5 acceptance criteria of the lab pass
• Answer at least 2 of the 3 interview questions without looking at the key points

Tomorrow (D4) we give a skill hands and feet. That crucial validation inside today's plan-validate-execute is a script; today's "the model reinvents the same logic every time" is likewise a signal to write one. Tomorrow settles when to switch from writing instructions to writing code, how a script's dependencies become self-contained, how an interface is designed for an agent rather than a person, and one unavoidable question — exactly how risky is it to let a model run your script.