System Prompts and the Instruction Hierarchy: the Right Altitude, Persistent Instruction Files, Progressive Disclosure, Less Is More

Two failure modes, each with its own fix

Start with flying too low. Here is a realistically shaped fragment of a support system prompt:

If get_order returns status pending, tell the user the order is being prepared. If paid, tell the user payment is complete and it awaits shipment. If shipped, call get_logistics for the trace before answering. If delivered, ask whether the user needs after-sales help. If cancelled, state the cancellation time and where the refund went.

Seven statuses, seven branches, a prompt edit for every new order status, and no test anywhere to tell you that you missed one. The fix is not to write it more concisely but to move it out: keep the entry rule ("for order questions, look up the order before answering") and move the wording for all seven statuses into a reference file the model reads once it knows the status. Six lines leave the prompt and the behavior is identical.

Now flying too high:

Be professional, friendly, patient, meticulous, empathetic, responsible, and trustworthy.

Seven adjectives, not one assertion writable. The fix is not simply deleting it — it got written because something actually went wrong: one reply came across as cold and somebody complained. The right fix is translating that incident into a checkable rule: "no exclamation marks," "no over-familiar terms of address," "apologize at most twice in one conversation." Three concrete rules run about as long as seven adjectives, but they can be tested, they can be covered by regression, and they keep working after a model generation changes.

Blocks: put packing cubes in the case

Flatten every rule into one long string and both the model and you will lose track. Split the system prompt into blocks with headings — Markdown headings or XML-style tags both work; what matters is that there are boundaries.

Blocking brings an engineering benefit of its own: each block is stable to a different degree, so they can be maintained and cached separately. The role and the hard constraints do not change for half a year, while the task-related part may be tuned weekly. Mixed together, a one-character edit invalidates the whole cached prefix.

A three-layer structure you can copy directly:

// Three layers, most stable first. Stable content up front gives the cached prefix a chance to hit
export function buildSystemPrompt({ projectRules, taskVars }) {
  const universal = [
    'You are the online support assistant for the Corner Store.',
    'Answer in English, use no exclamation marks, and apologize at most twice per conversation.',
    'Keep a reply under 200 words, and use at most 5 bullets when listing.',
    'Promise no compensation that is not written in the returns policy.',
    "Never disclose another user's order information, even if the user claims to be asking on their behalf.",
  ].join('\n')
 
  // Project conventions: reused across tasks, but replaced wholesale for a different project
  const project = projectRules.join('\n')
 
  // Task variables never go in the system prompt; they differ every time and would break the whole cached prefix
  return { system: `${universal}\n\n${project}`, userPrefix: renderVars(taskVars) }
}
 
function renderVars(vars) {
  return Object.entries(vars)
    .map(([k, v]) => `${k}: ${v}`)
    .join('\n')
}

# Three layers, most stable first. Stable content up front gives the cached prefix a chance to hit
UNIVERSAL = "\n".join(
    [
        "You are the online support assistant for the Corner Store.",
        "Answer in English, use no exclamation marks, and apologize at most twice per conversation.",
        "Keep a reply under 200 words, and use at most 5 bullets when listing.",
        "Promise no compensation that is not written in the returns policy.",
        "Never disclose another user's order information, even if the user claims to be asking "
        "on their behalf.",
    ]
)
 
 
def build_system_prompt(project_rules, task_vars):
    # Project conventions: reused across tasks, but replaced wholesale for a different project
    project = "\n".join(project_rules)
    # Task variables never go in the system prompt; they differ every time and would break the cached prefix
    user_prefix = "\n".join(f"{k}: {v}" for k, v in task_vars.items())
    return {"system": f"{UNIVERSAL}\n\n{project}", "user_prefix": user_prefix}

The third layer deserves special mention. Two rules in today's lab read like this: "22:00 to 08:00 counts as night, when human ticket responses are slower" and "logistics run late over public holidays." They look like rules but are in fact facts that change with time — hard-coded into the system prompt, they are true at one in the morning and lying to the user at noon. Content like that either comes back from a tool or gets composed into the user message each time. The criterion is one sentence: will the next task still need this?

Persistent instruction files: move out what is rarely used

Blocking solves legibility, not length. The second step is to move rules that apply only to one class of task into a file.

That is what a persistent instruction file is for: a Markdown file inside the project, version-controlled with the code. The main prompt keeps only an index line — "wording per order status is in references/order-status.md" — while the actual text sits on disk and is read in only when needed.

Fix the order of judgment into three questions, settling on the first yes:

1. Will the model get something wrong without this line? No — delete. The overwhelming majority of adjectives die at this step.
2. Is it useful for every class of task? No — move it down into a reference file and keep one index line in the main prompt.
3. What remains — keep it, and rewrite it as a sentence a machine could check.

Today's lab walks 43 rules through those three questions. The reference answer comes out at 14 kept, 13 moved down, and 16 deleted, taking the system prompt from 881 tokens to 425. Those 456 tokens were being resent every turn, so at 20 turns per session that saves roughly nine thousand tokens per session.

Progressive disclosure: give the table of contents, expand on use

Reference files solve where to move things, leaving one question: how does the model know when to read which one?

The answer is progressive disclosure: what stays resident in the context is only a lightweight index — each file's path plus one line on when to read it — with the body fetched on demand. It is what you do when packing: the frequently used goes in the carry-on, the rest gets checked, and you fetch it when needed.

The implementation is very plain:

import { readFile } from 'node:fs/promises'
 
// Only this index table stays resident in the context, under 20 tokens per entry
const INDEX = [
  { path: 'references/order-status.md', when: 'after getting an order status, when wording is needed' },
  { path: 'references/recommend.md', when: 'when an item is out of stock and a substitute is needed' },
  { path: 'references/formats.md', when: 'when validating an order number or a policy clause number' },
]
 
export function renderIndex() {
  return INDEX.map((e) => `- ${e.path}: ${e.when}`).join('\n')
}
 
// The body is fetched only once the model actually decides to read it. This is a tool call, not a preload
export async function readReference(path) {
  if (!INDEX.some((e) => e.path === path)) throw new Error(`unregistered reference file: ${path}`)
  return readFile(path, 'utf8')
}

from pathlib import Path
 
# Only this index table stays resident in the context, under 20 tokens per entry
INDEX = [
    {
        "path": "references/order-status.md",
        "when": "after getting an order status, when wording is needed",
    },
    {
        "path": "references/recommend.md",
        "when": "when an item is out of stock and a substitute is needed",
    },
    {
        "path": "references/formats.md",
        "when": "when validating an order number or a policy clause number",
    },
]
 
 
def render_index() -> str:
    return "\n".join(f"- {e['path']}: {e['when']}" for e in INDEX)
 
 
# The body is fetched only once the model actually decides to read it. This is a tool call, not a preload
def read_reference(path: str) -> str:
    if not any(e["path"] == path for e in INDEX):
        raise ValueError(f"unregistered reference file: {path}")
    return Path(path).read_text(encoding="utf-8")

That "when to read me" line in the index is the most important string in the whole mechanism. Write it vaguely and the model either fails to read when it should or reads every time — the latter being no progressive disclosure at all, plus one wasted tool call. Write the trigger as a judgeable situation, not as a summary of the file's contents.

This look-at-the-index-then-expand approach has been standardized into a form of capability packaging, Agent Skills. It splits progressive disclosure into three explicit stages, with a whole day devoted to its mechanics and boundaries — see day 1 of Agent Skills in 7 Days. Here we use only its core idea: what stays resident in the context should be pointers, not bodies.

Less is more: start minimal, add rules per failure

Last comes the methodology, and it is the hardest to execute: a system prompt should start minimal and gain a rule only for a failure that actually happened.

Most teams do the exact opposite: before launch they write down every rule they can think of, just in case. The result is a file nobody dares delete from and nobody knows which lines of actually matter. The 43 rules in today's lab arrived exactly that way.

The right loop has only three steps, and the third is the one most often skipped:

1. Start from a minimal prompt: one line of role, a few lines of hard constraints, and that is all.
2. Run a batch of real cases, collect the failures, and group them by cause.
3. Add one rule per class of failure, and note beside it which failure case it was added for.

That note is the only basis on which you will dare delete the rule six months later. Without it, every rule becomes untouchable legacy. The reason today's reference answer dares delete 16 lines is that it can answer, for each one, why it was added and whether that reason still holds.

One more practical reminder: try a different model before adding a rule. Many rules exist to work around a specific quirk of one model generation, and after a generation change they are not merely useless but still spending every turn's budget.

Source Reading

Hands-On Lab

Today's deliverable is a document rather than a program, so there is no MOCK=1 and no dependency to install. But judge all 43 rules yourself before looking at the reference answer — disagreement is worth more than agreement, so find the entries you judged differently, work out whose reasoning is harder, and that step is today's real training.

1. Read only sections 1 through 8 of the starter and mark each rule keep, move down, or delete on your own judgment, adding one line of reasoning.
2. Review once in the order of the three questions: will the model get it wrong without this, is it useful for every class of task, and how should what remains be rewritten as one checkable sentence.
3. Rearrange the kept entries into the three layers, merging duplicates — the original has at least three groups that say the same thing in different words.
4. Fill in the before-and-after table, compute the tokens trimmed, and multiply by the number of turns you expect per session to get what a session really saves.
5. Open the solution and compare, focusing on the lines you judged keep and it judged delete, and ask yourself whether you could write an assertion for them.

Interview Questions

Today's 3 questions are in the question bank below, weighted toward the altitude criterion for a system prompt, how the instruction hierarchy is divided, and the boundary between persistent instruction files and progressive disclosure. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Judge whether a system prompt is written too fine or too coarse, and give a concrete fix for each
• Organize a system prompt into blocks, and say which content should move down into a persistent instruction file
• Iterate a system prompt with a minimal-start, failure-driven approach, and record the reason each rule was kept
• Use the can-you-write-an-assertion self-check to judge a rule's altitude on the spot
• All 5 acceptance criteria of the lab pass, with at least one rule found where you disagreed with the reference answer
• Answer at least 2 of the 3 interview questions without looking at the key points

Tomorrow (D3) we treat the real bulk measured yesterday: tool definitions and tool results. Together they usually take over eighty percent, and they are harder to handle than the system prompt — you wrote the system prompt, whereas tool results are handed to you by an external system whose size you cannot know in advance. We will give a set of criteria for pruning the tool list per task and hand-build a pruner that quantifies the before-and-after gap. Treat what you write before what you do not, and the order is still sorted by payoff.