Context Is the Scarcest Resource: the Window, Attention Decay, and Cost — From Prompt Engineering to Context Engineering

Today's Goals

1. Name the four pieces that make up a model request's context, and work out each one's token share
2. Explain attention budget and context rot in your own words, and name the three costs they bring
3. Judge whether a problem can be solved with prompt engineering or must be escalated to context engineering

This course solves exactly one thing: what should actually go into the request every time you call a model. Today is about seeing clearly what is in there now — without that, all the pruning, compaction, and isolation of the next four days is guesswork. Once you have read it, scroll back to the top and tick off the three goals.

Plain-Language Walkthrough

Packing a carry-on suitcase before a trip

A carry-on suitcase has fixed capacity. You are away five days and want to bring clothes, a laptop, chargers, a contract, an umbrella, and samples for a client. Start packing and you find the hard part was never whether the case is big enough, but three questions: what has to go in now; what can be bought or borrowed once you arrive; and what looks useful but will never once be opened.

The clever move is not buying a bigger case. The bigger the case, the more you stuff into it, and finding a pen at the hotel takes five minutes of digging. What actually works is three habits: list what goes in before packing, take things out along the way as you need them, and repack at a layover to throw out whatever is finished.

A model's context window is that suitcase. Its capacity is counted in tokens — one token is roughly three or four letters of English. Every call, you have to fill the case again from scratch and hand it over, because the model itself has no memory: it does not recall what was said last turn, and if you do not pack it again, it does not know.

So what to pack turns from a detail into a discipline. It has a name — context engineering — and a one-line definition: before each inference, pick the smallest set of high-signal tokens that lets the model do the right thing. Note the two words in that sentence: smallest and high-signal. They conflict often, and making the trade-off inside that conflict is what these five days practice.

Start by remembering one counter-intuitive conclusion: packing more does not mean working better. The third section explains why.

What is actually in one request

Open the case and look. The context an agent request carries is almost always made of four pieces:

flowchart LR
  A[System prompt] --> E[The context of one request]
  B[Tool definitions] --> E
  C[Conversation history] --> E
  D[Tool results] --> E

• The system prompt: role, procedure, hard constraints. Resent verbatim every turn, with essentially constant length.
• Tool definitions: each tool's name, description, and parameter shape. Also resent every turn, with length growing with the number of tools.
• Conversation history: what the user said and what the model said. A few dozen tokens per turn, growing linearly.
• Tool results: the data tools hand back. One call can add two thousand tokens, growing in steps.

Those four grow in completely different ways, so they have to be treated separately. Look at them as one total token count and you will never know where to cut.

The difference in growth has a less obvious consequence: the payoff from changing each piece differs entirely. The system prompt is resent verbatim every turn, so halving it benefits every turn of the whole session — the piece with the highest compounding. Tool results are added once and then sit there, so pruning one saves only that one, but their single-shot volume is the largest, which makes their absolute payoff the highest. Conversation history grows slowest and is usually the smallest of the four. Those three properties set the order of the next four days, and day five puts a real bill on the table to show it.

The easiest thing to get wrong: tool results are written inside messages whose role is user, which makes them look like part of the conversation history, but they are not something the user said. They are data, not dialogue. Count them inside the history and the table is worthless. When splitting, go by content block type, not by message role:

// Split into four pieces by content block type: tool results get their own bucket, never mixed into history
export function splitContext(system, tools, messages) {
  const history = []
  const toolResults = []
 
  for (const msg of messages) {
    const kept = []
    for (const block of msg.content) {
      if (block.type === 'tool_result') {
        toolResults.push({ id: block.tool_use_id, content: block.content })
      } else {
        kept.push(block)
      }
    }
    // Do not put back a message left with no blocks; the API rejects an empty message
    if (kept.length > 0) history.push({ role: msg.role, content: kept })
  }
 
  return { system, tools, history, toolResults }
}

# Split into four pieces by content block type: tool results get their own bucket, never mixed into history
def split_context(system, tools, messages):
    history = []
    tool_results = []
 
    for msg in messages:
        kept = []
        for block in msg["content"]:
            if block["type"] == "tool_result":
                tool_results.append({"id": block["tool_use_id"], "content": block["content"]})
            else:
                kept.append(block)
        # Do not put back a message left with no blocks; the API rejects an empty message
        if kept:
            history.append({"role": msg["role"], "content": kept})
 
    return {"system": system, "tools": tools, "history": history, "tool_results": tool_results}

Today's lab runs that code over a fictional e-commerce support session. Spoiler on the result: system prompt 280, tool definitions 308, conversation history 218, tool results 2191 — tool results alone are 72.9%. That proportion is very typical of real projects, and almost everybody guesses the bulk is conversation history before they measure.

Attention budget and context rot

"So I switch to a model with a bigger window and I am fine?"

That is everyone's first reaction and the first intuition this course dismantles. A bigger window does solve "it does not fit," but it does not solve another problem: the more you pack, the more the model overlooks.

That phenomenon has a name, context rot: as context grows longer, the model's ability to recall information inside it accurately declines. Its origin is architectural — in a Transformer every token has to relate to every other token, so n tokens mean on the order of n-squared pairwise relations. The longer the context, the thinner the same fixed attention is spread. This is not a cliff but a gentle slope: no particular length suddenly collapses, but every extra thousand irrelevant tokens costs a little accuracy.

There is a more practical reason too: models see mostly shorter sequences during training, so the parameters dedicated to very long-range dependencies are few to begin with. "A nominal 200K window" and "stable behavior across 200K tokens" are two different statements; the nominal number is capacity, not a guarantee.

So the right mindset treats attention as a budget rather than treating the window as capacity. Capacity thinking asks whether there is still room; budget thinking asks whether this thousand tokens is worth spending. The latter is this course's way of thinking.

One judgment you can use immediately: if you cannot say which specific decision a piece of content will change, it should not be added. That criterion comes back repeatedly on day two when trimming the system prompt and day three when pruning tool results.

Three bills: money, latency, accuracy

A longer context makes you pay three bills, and most people count only the first.

The first is money. Input tokens are billed by volume, and the model having no memory means everything before gets resent every turn. In a 20-turn session, turn 1 might send 3,000 tokens while turn 20 sends 11,000, and the total input for the session is a running sum rather than the value of the last call. Today's lab has you compute that sum yourself.

Putting numbers on it hits harder. Say a session runs 20 turns, turn 1 sends 3,000 tokens, and each later turn adds some as history and tool results accumulate, reaching 11,000 by turn 20. What you pay for is not 11,000 but the sum of all 20 — roughly 140,000 tokens on a linear estimate. Plenty of people quote that 11,000 in a cost estimate and thereby understate the session by an order of magnitude. Day five works this bill out in full from today's real lab data, including how caching rewrites it.

The second is latency. The longer the input, the slower the first token. What a user feels is not "thirty percent more expensive" but "it spun for four seconds." And this bill gets amplified in an agent setting: one task calls the model many times over, each a little slower, and the accumulation is a distinctly perceptible "this thing is sluggish."

The third is accuracy, and it is the most easily overlooked. As the last section said, irrelevant content dilutes attention. What makes this bill frightening is that it does not raise an error: the model will not tell you it missed a constraint because the context was too cluttered; it will simply give a plausible-looking answer that violates the system prompt. By the time you notice, it is usually a production incident.

The line between prompt engineering and context engineering

Prompt engineering cares about writing one passage well: how the role is set, how the task is stated, how the format is constrained. Context engineering cares about how the whole suitcase is packed: that passage is one item of luggage, and beside it sit the tool definitions, the history, and the tool results, all of which change every turn.

The line is actually clear. Ask yourself one question: is my problem that a single call was not written clearly enough, or that things were not kept clean across many turns?

• The same question is answered wrongly once and answered correctly after a rephrasing — that is prompt engineering.
• The first five turns are fine and by turn twenty it starts violating the original constraints — that is context engineering.
• The data was clearly retrieved and the model says it does not have that information — also context engineering; the information is in the window, only drowned.

A harder criterion: prompt engineering deals with content, context engineering with budget allocation. If your change is "make this sentence more precise," it is the former; if it is "how much should this piece get, and when should it be thrown out," it is the latter.

This course assumes you can already write prompts. If that part is shaky, start with Prompt Engineering in 5 Days; the basics of writing an agent loop are in day 5 of the 30-day course, which covers the full round trip of a tool call — the origin of this course's tool definitions and tool results.

What each of the five days fixes

Of the four pieces, only three are actually actionable, and the five days are ordered by their payoff:

1. Today (D1) is measurement: split the four pieces, work out the shares, and find where your bulk is. Without that table, everything later is a guess.
2. D2 treats the system prompt: resent every turn, it has the highest compounding of the four. Altitude, layering, and progressive disclosure cut it in half.
3. D3 treats tool definitions and tool results: usually the bulk. It gives criteria for pruning the tool list and a pruner that quantifies the before-and-after gap.
4. D4 treats conversation history: a long-running task will fill the window. Compaction, external notes, and subagent isolation each fit different situations.
5. D5 closes the loop: turn the first four days' techniques into a measurable cycle, work out the bill, watch two utilization metrics, and triage against four failure modes.

Mind the order: measure before changing, and treat what is resent before what merely grows. Many people start with compaction because it sounds the most sophisticated; but compaction treats the history, and the history is often the smallest of the four — you work at it for an afternoon and the bill does not budge.

Source Reading

Hands-On Lab

The first four criteria need no key, and only the fifth needs the real endpoint. Before you start, settle one thing: which piece do you think is largest? Write your guess down before running, because the moment you find it wrong is where this lab's real value is.

1. Get the solution's MOCK=1 pnpm start running, look at the share table for a session split four ways, and compare it against your guess.
2. Go back to the starter and complete exercise 1's splitting function, then run once to confirm the tool results row is no longer 0 and the four pieces sum to the total.
3. Complete exercise 2's offline estimator, switching from counting characters to counting by character class, and watch the total drop from seven thousand-odd back to around three thousand.
4. Complete exercise 3's differential counting and exercise 4's share calculation, then rerun with TOOL_COUNT=8 and watch which piece loses control of its share first.
5. Configure ANTHROPIC_API_KEY, drop MOCK=1, run once, subtract the two totals, and record the direction and size of the estimator's error.

Interview Questions

Today's 3 questions are in the question bank below, weighted toward what makes up the context window, attention budget and context rot, and the boundary between context engineering and prompt engineering. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Name the four pieces that make up a model request's context, and work out each one's token share
• Explain attention budget and context rot in your own words, and name the three costs they bring
• Judge whether a problem can be solved with prompt engineering or must be escalated to context engineering
• Explain why tool results must be counted apart from conversation history rather than both as messages
• All 5 acceptance criteria of the lab pass, with the estimator's error against the real count recorded
• Answer at least 2 of the 3 interview questions without looking at the key points

Tomorrow (D2) we start work on the piece with the highest compounding: the system prompt. It is resent verbatim every turn, so changing it once saves money on the whole session. We will cover how to judge whether a system prompt is written too fine or too coarse, how to layer it, and what content should not be in it at all. Treat what is resent before what merely grows — that order is not casual, it is sorted by how much each change saves.