Week 1 · D1About 4 hours

Why an Agent Starts Degrading Once It Runs Past the Second Context Window

Start with what really separates a long-horizon task from an interactive conversation, then reproduce the degradation with your own hands in a run that crosses two context windows: the second window redoes everything the first one already did. The model did not get dumber. There is simply no evidence left in its context that any of that work happened.

Today's goals 0/3

Can name the three difficulties a long-horizon unattended task adds on top of an interactive conversation, and give one concrete consequence for each of having nobody there to catch it
Can describe an unattended run precisely using context window, state, feature list, verified commit and intervention, and explain why the context window and the state have to stay separate
Can explain why no harness configuration parameter belongs in the model interface signature, and name the class of conclusions that turns into a false green the moment one does

Today's Goals

Can name the three difficulties a long-horizon unattended task adds on top of an interactive conversation, and give one concrete consequence for each of having nobody there to catch it
Can describe an unattended run precisely using context window, state, feature list, verified commit and intervention, and explain why the context window and the state have to stay separate
Can explain why no harness configuration parameter belongs in the model interface signature, and name the class of conclusions that turns into a false green the moment one does

The thing you build over these seven days is a harness: the layer around the model that decides what happens next, records where the work stands, rules on whether "done" means done, and calls a halt when the run sticks. Today reproduces the problem.

Plain-Language Walkthrough

A shift handover: the last person went home, and the briefing was given exactly once

Picture the night shift changing over on a factory floor. Zhang works until ten, briefs Li — these three machines are fixed, that one is half disassembled, the torque wrench is in the third drawer — then goes home.

Now ask one question: where does that briefing live?

In Li's head. Swap Li for Wang partway through the night, with Zhang already gone, and the briefing goes with him. Wang can only guess which machine is fixed.

Zhang's one spoken briefing is the agent's context window; swapping Li for Wang is swapping the window. None of this matters while a person is there, because you can always add one more sentence. But if the floor is empty from ten at night until eight in the morning, a lost briefing is genuinely lost.

So the fix is not a clearer briefing. It is a whiteboard on the wall that every incoming shift reads first: what is on it is authoritative, what is in someone's head is just convenient.

What makes long-horizon hard: nobody to fill in, nobody to call a halt, nobody to sign off

Anyone who has built an agent knows the interactive shape: you speak, it answers, you correct it. Running unattended adds three difficulties, all from the person not being there.

First, nobody fills in the gaps. In a conversation you supply the premise it missed and paste the requirement again after truncation. Unattended, the window turns over and the agent redoes work it finished three hours ago, with nobody to notice. That is today's experiment.

Second, nobody calls a halt. Watching it patch the same file the same way a fifth time, you stop it. Unattended, it patches it a fiftieth time: a night of compute burned on a task that never moves, morning progress zero, bill full.

Third, nobody signs off. Nobody checks the sentence "it is done": in the morning it reports all forty features complete, you start the service, and it will not come up.

Anthropic's engineering blog sorts long-horizon failures into four modes, each paired with a mechanism. That table is the skeleton of these seven days, and failure and mechanism only make sense as a pair:

Failure mode	Paired mechanism
Declaring victory too early	A feature list; a line flips to passing only after verification
Undocumented defects left in the environment	An initial git repo plus a progress note, committed every session
A feature marked complete that was never tested	Only an end-to-end check changes that line's status
Time burned working out how to start the application	An `init.sh` that every session reads first

Those four mechanisms land on D4, D2, D5 and D3. None get built today; today only has to convince you they are needed.

Five terms, settled now

Six days lean on five words. Pin them down before anyone quietly uses a different definition.

Term	What it means	Easiest thing to get wrong
Context window	One context window	Volatile: compacted, truncated, reset, swapped out
State	The authoritative progress record, on disk	Survives windows, the only copy you can trust
Feature list	One line per end-to-end verifiable item, with a pass flag	Drives task selection, not a progress bar for humans
Verified commit	A point that passed an end-to-end check and was committed	Rollback targets only this, never "wherever we just were"
Intervention	What the harness does once a stall is detected	Three options: switch tasks, roll back, or stop and alarm

The context window is volatile: compacted, truncated, reset, or swapped out entirely

Everyone knows a context window has a limit, but most picture one way it dies. There are four, and not one line of your code raises an error.

Compacted: the framework sees the limit approaching and summarizes, folding twenty steps into five lines. The constraint that mattered may have been in the folded part.
Truncated: the blunter version. The oldest messages are simply cut.
Reset: the process crashed, the machine rebooted, the queue was reshuffled, and the next round starts from nothing.
Swapped out entirely: a long task always spans several windows, and the second is a brand new session.

The first three are decisions the runtime makes for you; the fourth follows from task length. What they share: none throws, and none logs "I threw your progress away". All you see is an agent whose behavior suddenly stops making sense.

Hence this course's spine: the context is volatile; the state is authoritative.

One run across two context windows: degradation is missing evidence, not a dumber model

Today's lab makes that reproducible. The runner is nightrun; the target is a note service called notekeeper shipping a forty-line feature list, every line initially unpassed. The configuration is two windows, three steps each, and at the window boundary exactly one thing happens: the accumulated transcript is cleared.

loop.js

for (let w = 1; w <= config.windows; w += 1) {
  // The only thing the window boundary does. This line IS "the context is
  // volatile" — comment it out and today's degradation disappears.
  transcript.length = 0
 
  for (let i = 1; i <= config.stepsPerWindow; i += 1) {
    // The opening context holds the task description and the feature list.
    // Not one character of progress.
    const context = transcript.length === 0 ? base : `${base}\n\n${transcript.join('\n')}`
    const reply = await askModel(context)
 
    const parsed = parseReply(reply)
    const applied = applyPatch(repoDir, parsed.featureId, parsed.code)
    const verdict = verifyShallow(repoDir, parsed.featureId, applied)
 
    // DONE_MARK is the only completion evidence the model recognizes.
    // It is written into the transcript, which is to say: it lives inside the
    // window, and it is gone the moment the window turns over.
    const mark = verdict.passed ? DONE_MARK : NOT_PASSED
    transcript.push(`[step ${i}] verify: ${parsed.featureId} ${mark}`)
  }
}

loop.py

for w in range(1, config.windows + 1):
    # The only thing the window boundary does. This line IS "the context is
    # volatile" — comment it out and today's degradation disappears.
    transcript.clear()
 
    for i in range(1, config.steps_per_window + 1):
        # The opening context holds the task description and the feature list.
        # Not one character of progress.
        context = base if not transcript else base + "\n\n" + "\n".join(transcript)
        reply = await ask_model(context)
 
        parsed = parse_reply(reply)
        applied = apply_patch(repo_dir, parsed.feature_id, parsed.code)
        verdict = verify_shallow(repo_dir, parsed.feature_id, applied)
 
        # DONE_MARK is the only completion evidence the model recognizes.
        # It is written into the transcript, which is to say: it lives inside the
        # window, and it is gone the moment the window turns over.
        mark = DONE_MARK if verdict.passed else NOT_PASSED
        transcript.append(f"[step {i}] verify: {parsed.feature_id} {mark}")

Run it and you get this. The lab prints in Chinese, because both editions of this course share one codebase; the shape is what matters:

TextText

window 1 did: F01 F02 F03
window 2 did: F01 F02 F03   <- F01 F02 F03 here are duplicated work
 
feature list actual progress: 3/40  [F01 F02 F03]
steps spent: 6
of which wasted: 3
duplicates inside window 1: none
duplicates inside window 2: none
 
opening context seen by step 1 of each window:
  identical, character for character (2891 characters each).
  Nothing window 1 did is present when window 2 opens.

Six steps bought three lines of progress: half the compute burned for nothing, and both opening contexts identical character for character, 2891 characters each.

Here is the argument this course rests on: the second window guesses from scratch because the evidence is gone, not because the model got dumber. The model runs one rule: pick the first feature with no completion evidence and build it. Window 1 picked F01 because it is the first line. Window 2 picked F01 again, because within the 2891 characters it can see, F01 still carries no completion evidence. Same rule, same input, same output. The evidence changed, not the behavior.

One step further: the progress existed all along, in the harness's own memory, which is where "feature list actual progress: 3/40" came from. Nobody failed to record it — nobody wrote it into the next window's opening. And window 1 has no duplicates precisely because its evidence was still accumulating in the same window.

Why the model interface should take exactly one string

Now the methodological pivot. The model interface is a single line:

AskModel = (context: string) => Promise<string>

One string in, one string out. No harness configuration parameter is in that signature and none ever will be: no withState, no options, no config.

This is not fastidiousness. These seven days consist of proving harness mechanisms help, and the moment the model senses which are on, it can play along:

ask-model.js

// The correct signature: the model can only see one string.
// What the harness turned on or off, it cannot read one character of.
async function askModel(context) {
  // One rule only: pick the first feature with no completion evidence
  const queue = listedFeatures(context)
  const done = featuresMarkedDone(context)
  const next = queue.find((id) => !done.has(id))
  return next === undefined ? 'ALL_DONE' : writePatch(next)
}
 
// The tainted signature: one extra parameter and every conclusion in the
// course is void. It can work hard when the state layer is on and play dumb
// when it is off, and you would never be able to tell.
async function askModelTainted(context, harness) {
  if (!harness.stateEnabled) return 'IMPLEMENT F01'
  return askModel(context)
}

ask_model.py

# The correct signature: the model can only see one string.
# What the harness turned on or off, it cannot read one character of.
async def ask_model(context: str) -> str:
    # One rule only: pick the first feature with no completion evidence
    queue = listed_features(context)
    done = features_marked_done(context)
    nxt = next((fid for fid in queue if fid not in done), None)
    return "ALL_DONE" if nxt is None else write_patch(nxt)
 
 
# The tainted signature: one extra parameter and every conclusion in the
# course is void. It can work hard when the state layer is on and play dumb
# when it is off, and you would never be able to tell.
async def ask_model_tainted(context: str, harness) -> str:
    if not harness.state_enabled:
        return "IMPLEMENT F01"
    return await ask_model(context)

Run seven days on the tainted signature and every curve comes out beautiful: state layer on, higher completion; off, lower. "The state layer works" looks empirically confirmed. It is a false green you fed yourself — you measured how cooperative the model is, not the harness.

Commit this test to memory: in any controlled experiment, the system under test must not know it is being tested. Turn it around and you get today's real payoff: this model sees one string and nothing else, so the duplicated work can only emerge for real. It cannot have been scripted, because the script has no concept of a window.

Changing the harness can beat changing the model

Why does this deserve seven days of its own? One very hard argument.

LangChain published a measured result: same model, no model change, harness changes only, taking Terminal Bench 2.0 from 52.8 to 66.5 — a gain of 13.7 points — and moving the submission from outside the top 30 into the top 5.

A gain that size is usually a model generation. Here not one line of model was swapped. Four things changed, three of which are this course's spine:

What they changed	What it does	Where it lands
A forced check before exit	No completion claim until a check has run	D5, the end-to-end gate
Doom loop detection	Tracks file edits, suggests a new approach after too many	D5, doom loops
Local environment context	Maps the directory tree, finds available tooling	D2, state and context rebuilding
Reasoning budget allocation	More reasoning for planning and verification	Out of scope, model tuning

The fourth is deliberately out of scope: model tuning, not harness structure, and this course holds that boundary through the last day.

Somebody watching, versus nobody watching

If you took Build Your Own Coding Agent in 21 Days, you already hand-built harness parts one by one: the tool loop, session recovery, stall signals, context compaction. What is left?

One difference that changes nearly every conclusion: that course lives in an interactive REPL with a person sitting there; this one runs all night with nobody present. Three cases where that flips the answer:

Same topic	Somebody watching	Nobody watching
git	Never commit; never pollute the user's repository history	Commits are the authoritative progress record, made by the agent
Compaction	Probe with early facts, confirm nothing was lost	A probe tests whether it remembers, not whether it obeys
Doom loops	Raise a stall signal after ten rounds with no movement	What happens next automatically: switch, roll back, stop

None of the three is about who is right; the premises differ. The git reversal is cleanest: that course operates on the user's repository, where injecting commits oversteps; this one on the agent's own workspace, where commits are the progress, and not committing leaves nothing to roll back to. That opens D4.

The compaction row is worth remembering. Probing verifies factual fidelity. Constraint fidelity is a different property: an agent that remembers every fact may well have stopped honoring the rule that no file is deleted without confirmation. Probing is blind to that, which is the core of D6.

Source Reading

Today's lab ships four frozen files, finalized today and copied verbatim for six days. Start with the shortest two.

src/core/types.ts is the vocabulary in code: Feature is one line of the list, RunState the state on disk, Step one step inside a window, VerifyResult the verdict of an end-to-end check, Outcome how a run ended (finished, stalled, out of budget, crashed). Read the sixth type closely: AskModel = (context: string) => Promise<string>. Notice the comment above Feature.passes too — only a passing end-to-end check may flip it to true — the whole of D5 is that one line of discipline.

src/core/git.ts holds a single function carrying this course's red line. It is the only course where an agent really executes git, and if the working directory resolves wrong once, the commit lands in the course repository's own history.

git.js

// The only place in the lab allowed to call git. workdir is a required first argument.
function runGit(workdir, args) {
  if (!workdir) throw new Error('runGit: workdir is required; never rely on an inherited cwd')
  // Always pass -C explicitly; never depend on the cwd inherited by the process
  return execFileSync('git', ['-C', workdir, ...args], { encoding: 'utf8' })
}
 
// The wrong way: no directory, relying on the inherited cwd.
// Correct inside work/repo, and a commit into your own repository anywhere else.
function runGitUnsafe(args) {
  return execFileSync('git', args, { encoding: 'utf8' })
}

git.py

def run_git(workdir: str, args: list[str]) -> str:
    """The only place in the lab allowed to call git. workdir is a required first argument."""
    if not workdir:
        raise ValueError("run_git: workdir is required; never rely on an inherited cwd")
    # Always pass -C explicitly; never depend on the cwd inherited by the process
    done = subprocess.run(["git", "-C", workdir, *args], capture_output=True, text=True, check=True)
    return done.stdout
 
 
def run_git_unsafe(args: list[str]) -> str:
    """The wrong way: no directory, relying on the inherited cwd.
    Correct inside work/repo, and a commit into your own repository anywhere else."""
    return subprocess.run(["git", *args], capture_output=True, text=True, check=True).stdout

Three rules go with it: git runs only inside the separate repository at work/repo; every call goes through this one exit; and a consistency script fails the build if the literal git appears outside this file. That last is a machine gate — good intentions do not hold a red line.

Hands-On Lab

🧪 Day 1 lab: run across two context windows and reproduce the degradation yourself

Code location: labs/agent-harness-7days/day-01-two-windows

Today builds the foundation of nightrun: the type layer, the single git exit, a target carrying a forty-line feature list, the offline script engine, and a two-window run that prints the duplicated work. All four frozen files arrive complete, so there is exactly one exercise — the two hollowed-out loops in src/core/loop.ts.

Two things are easy to get wrong: the transcript must be cleared at the top of each window, and the verification line must carry DONE_MARK on a pass, since that marker is the only completion evidence the model recognizes.

Read src/core/types.ts and map the six types onto the five terms above, especially AskModel.
Read src/core/git.ts and confirm workdir is required and -C is passed every time.
Run pnpm target:all: forty patches applied at once, a real service started, twenty-odd requests fired, proving the forty lines add up to something that runs.
Fill in the two loops in src/core/loop.ts and run MOCK=1 pnpm selftest until all 13 assertions are green.
Run MOCK=1 pnpm start and watch window 2 redo F01 F02 F03.

Do not stop there. The mutation check in the README is today's decisive step: comment out transcript.length = 0 at the top of the window loop, run again, and the output changes.

TextText

window 1 did: F01 F02 F03
window 2 did: F04 F05 F06
 
feature list actual progress: 6/40  [F01 F02 F03 F04 F05 F06]
of which wasted: 0

Duplication gone, progress doubled. That proves two things: the duplication really comes from the context reset at the window boundary rather than from the script, and the assertion "no duplicates inside window 1" stayed green across both runs, so it discriminates rather than sitting vacuous.

Interview Questions

Today's four questions circle three things: where progress belongs in a long-horizon task, where the model interface's boundary sits, and how skeptical to be about "it is done". Read each analysis before the key points.

The third shows up in real interviews, because it is this job every day: something ran all night and says it finished — on what grounds do you believe it. "I would spot-check a few" earns half credit.

Checklist and Tomorrow

Can name the three difficulties unattended running adds, with one consequence each
Can describe an unattended run using context window, state, feature list, verified commit and intervention, and say why window and state stay separate
Can explain why no harness configuration parameter belongs in the model interface
Can name the four ways a context window dies, and what they share: no error
Got all 13 self-test assertions green under MOCK=1 pnpm selftest, and saw the two opening contexts match exactly
Ran the mutation check, and can say which two things it proves
Can answer three of the four interview questions without the key points

Tomorrow is D2, The Line Between State and Context: Moving the Authoritative State Out of the Window. Today's fix is already in plain sight — the progress sat in memory all along, and nobody wrote it into the next window's opening. So tomorrow is not about writing a file. It is about where that dividing line belongs: which three kinds of information must be persisted, which two must not, and one counterintuitive test — the goal of rebuilding a context is equivalence, not restoration. Replay a whole night verbatim and you pay for it twice, then hit the window limit again.

Interview questions

What is the difference between the context window and state? Why can a long-horizon agent's authoritative progress not live only in the window?上下文窗口和状态有什么区别？为什么长时程 Agent 的权威进度不能只存在窗口里？
Common in ChinaCommon overseasIntermediate#long-horizon#state-management#context-window
How to reason about it · think before answering
This tests whether you have actually run a long task. Answering only 'the window has a token limit, so compact it' treats it as a capacity question. The discriminator is whether you can name the window's volatility and say who covers for it when it disappears.
Offer a transferable test: ask 'if this piece of information vanished right now, could I look it up anywhere else?' If yes it is a volatile copy; if no it is the authoritative record and must hit disk. The window is the former, state is the latter. The distinction is ownership, not size.
Then note that the window dies in four ways and none of them raise an error: the framework auto-compacts it, it gets truncated, the process restarts and resets it, and a long task spans several windows by construction. The first three are decisions the runtime makes for you, the fourth is arithmetic. What they share is silence - no exception, no log line saying your progress was discarded. You only see the agent behaving inexplicably.
Now the consequence, which is what the interviewer is waiting for. If progress lives only in the window, the next window contains no evidence of prior work, so the agent restarts from the top of the checklist. That is not the model getting dumber - the evidence is gone, and the same rule on the same input must produce the same output. The cost is double: half the compute is wasted, and duplicate writes dirty the workspace so later dependent tasks behave unpredictably.
Interactive chat hides this because the human *is* the state layer: you can always add 'that one was finished yesterday'. Unattended, that human is absent, so an authoritative record on disk has to stand in for them.
Expected follow-up: can you just save the whole night's transcript and replay it into the next window? No, for two reasons. It will not fit, and replaying it pays for a night of reasoning twice. The goal of rebuilding context is equivalence, not restoration - you need what is done, what is in flight, and how many attempts have been made. Intermediate reasoning and one-shot tool receipts do not belong on disk.
分析过程 · 先想清楚再作答
这题考的是「有没有真的跑过长任务」。只答「窗口有 token 上限，所以要压缩」是把它当成一道容量题——区分度在于你能不能说出窗口的**易失性**，以及易失之后谁来兜底。
先给一个能迁移的判据：问一句「这条信息如果现在丢了，还有别的地方能查到吗」。查得到的是易失副本，查不到的就是权威记录，必须落盘。窗口是前者，状态是后者，两者的区别不在容量而在**归属**。
然后点出窗口有四种死法，而且它们都不报错：被框架自动压缩、被截断、进程重启后重置、以及长任务本来就要跨好几个窗口。前三种是运行环境替你做的决定，第四种是任务长度的必然。共同点是没有任何异常、没有任何日志说「你的进度被扔了」，你只会看到 Agent 的行为忽然变得莫名其妙。
接着说清后果，这一段是面试官真正想听的。进度只活在窗口里，窗口一换，Agent 在新上下文里找不到任何做过的证据，于是从清单第一条重新开始。**这不是模型变笨了，是证据没了**——同一条规则、同一个输入，必然同一个输出。代价是双份的：算力白烧一半，而且重复写入会把工作区弄脏，后面依赖它的任务会给出你预料不到的结果。
交互式对话之所以感觉不到这个问题，是因为人就是那个兜底的状态层：你随时能补一句「那条昨天做完了」。无人值守时这个人不在场，所以必须有一个磁盘上的权威记录替他站着。
可预期的追问是「那把整夜的对话原样存下来、下个窗口再喂回去行不行」。不行，两个理由：一是窗口上限本来就装不下，二是那样做等于把一夜的推理再烧一遍钱。重建上下文的目标是**等效**而不是还原——只需要「做完了什么、正在做什么、试了几次」，中间推理和一次性的工具回执不该持久化。
Key points
The window is a volatile copy; state is the authoritative record on disk. The difference is ownership, not size.
The window dies four ways - compaction, truncation, reset, replacement - and none of them raise an error.
The test: if this vanished now, could I look it up elsewhere? If not, it must be persisted.
With progress only in the window, the next window has no evidence and restarts from the top. The model did not get dumber.
In interactive chat the human is the state layer; unattended you need a disk record to stand in.
Rebuild for equivalence, not restoration: what is done, what is in flight, how many attempts.
答题要点
窗口是易失副本，状态是磁盘上的权威记录；区别在归属而不在容量。
窗口有四种死法：压缩、截断、重置、整个换掉，而且一行都不报错。
判据：这条信息现在丢了还能不能在别处查到，查不到就必须落盘。
进度只活在窗口里，换窗口后 Agent 找不到证据就从头重做——不是变笨，是证据没了。
交互式对话里人就是那个状态层，无人值守时必须有磁盘记录替他站着。
重建的目标是等效不是还原：只留做完了什么、正在做什么、试了几次。
Why should the model-facing interface not know which harness features are switched on? What happens if it does?为什么给模型的接口不该知道当前开了哪些 harness 功能？如果知道了会发生什么？
Common in ChinaCommon overseasDeep dive#experiment-design#interface-boundary#false-green
How to reason about it · think before answering
The real question is the second half. Plenty of candidates can say 'keep the interface clean'; few can say 'the moment it knows, every experimental conclusion you drew is void'. That is the whole discriminator.
Set up the scenario. You are running a controlled experiment to show some harness mechanism - a state layer, a verification gate, a loop detector - actually helps. You run once with it on and once with it off and compare completion. The design assumes that apart from that one mechanism, everything else about the two runs is identical.
Now add a config parameter to the model-facing signature. The assumption collapses: the system under test can see the experimental condition, so it can play along - work properly when the state layer is on, play dumb when it is off. The curves look beautiful and 'the state layer helps' appears to be empirically confirmed, but you measured cooperation, not effect. There is a name for that result: a false green.
The nastier part is that a false green raises no error. It looks exactly like a real finding, and it leans in the direction you were hoping for. A bug that crashes is luck; a bug that makes you more confident is a disaster.
So narrow the signature to the bone: one string in, one string out. The model sees only the context you fed it and learns nothing about harness configuration. Then the phenomenon can only emerge for real - the second window repeats work because its context genuinely holds no completion evidence, not because something told it to act forgetful.
Generalize it into a principle worth stating out loud: in any controlled experiment, the system under test must not know it is being tested. Evaluation, A/B testing and security red-teaming are all the same sentence, and naming that earns credit.
Expected follow-up: how do you inject model-side behavior then, such as a worse script? Separate model-side data from harness configuration. Swapping the script changes what code the model would write, which belongs to the model side. Whether the state layer is currently enabled belongs to the harness side and must stay outside the signature.
分析过程 · 先想清楚再作答
这题的题眼在后半句。能说出「接口要干净」的人很多，能说出「一旦知道了，你的全部实验结论都作废」的人很少——区分度全在这里。
先把场景摆清楚：你在做一个对照实验，想证明某个 harness 机关（状态层、验证闸门、打转检测）有用。做法是开一次、关一次，比两次运行的完成度。这个设计的隐含前提是：**除了那个机关，两次运行的其他一切都相同。**
现在假设模型接口的签名里多了一个配置参数。那个前提立刻不成立了——被测系统能看见实验条件，于是它可以配合演出：开了状态层就好好干，关了就装傻。跑出来的曲线非常漂亮，结论「状态层有用」看起来被实测证实了，但你测的不是机关的效果，是模型有多配合。这类结果有个专门的名字：**假绿**。
更麻烦的是假绿不会报错，它长得跟真结论一模一样，而且是往你想要的方向偏。一个会自己报错的 bug 是运气，一个让你更自信的 bug 才是灾难。
所以正解是把签名收到最窄：一个字符串进，一个字符串出，模型只能看到喂给它的那段上下文，harness 开了什么一个字都拿不到。这样一来现象就只能真实涌现——第二个窗口之所以重复劳动，是因为它的上下文里确实没有完成证据，而不是因为有人告诉它「现在状态层关了，你装傻吧」。
这条判据可以抽象成一句通用原则：**任何一次对照实验，被测系统都不该知道自己正在被测。** 它在评估、A/B 实验、安全红队里是同一句话，答的时候点出来能显著加分。
可预期的追问是「那模型的行为数据怎么传进去，比如换一份更差的剧本」。答案是区分**模型侧数据**与**harness 配置**：换剧本改的是「模型会写出什么代码」，那本来就属于模型这一侧；而「当前开没开状态层」属于 harness 侧，前者可以传，后者必须挡在签名外。
Key points
A controlled experiment assumes everything but the mechanism is identical; a config parameter breaks that immediately.
Once the model can see the condition it can play along - work when the feature is on, act dumb when it is off.
The resulting curves are a false green: you measured cooperation, not the mechanism.
False greens raise no error and lean the way you hoped, which is worse than a crash.
Fix: narrow the signature to one string in, one string out, so phenomena can only emerge for real.
General principle: the system under test must not know it is under test. Model-side data may vary; harness configuration stays out of the signature.
答题要点
对照实验的隐含前提是除被测机关外其他一切相同，配置参数会直接打破它。
模型一旦感知实验条件就能配合演出：开了好好干、关了装傻。
那样跑出来的曲线是假绿——测的不是机关效果，是模型有多配合。
假绿不报错、还往你想要的方向偏，比会崩的 bug 危险得多。
正解是签名收窄到一个字符串进一个字符串出，现象只能真实涌现。
通用原则：任何对照实验，被测系统都不该知道自己正在被测。模型侧数据可以换，harness 配置必须挡在签名外。
An agent that ran unattended overnight tells you in the morning that every task is complete. How do you verify that claim?一个无人值守跑了一夜的 Agent 早上告诉你全部任务已完成。你会怎么验证这句话？
Common in ChinaCommon overseasIntermediate#verification#long-horizon#reporting
How to reason about it · think before answering
This is about whether you distrust completion claims. 'I'd spot-check a few' earns half credit - spot-checking is a tactic, not a method, and you will most likely sample the tasks it faked best. The interviewer wants to hear how you turn 'it says it is done' into a criterion that does not rely on its own account.
First, separate two things: the status it claims and the actual state of the environment. The first is a line it wrote itself; the second is a fact code can query. All verification must land on the second. Skip this and everything after it is auditing a self-report.
Second, make the claim verifiable in form. Every checklist item must be end-to-end verifiable, meaning you can write a check that ignores all agent output and simply drives the running system. That constrains how items are written: 'improve search' is unverifiable; 'title search does case-insensitive substring matching' is verifiable.
Third, re-run the whole suite rather than the last item. A night unattended produces dozens of changes and later work commonly breaks earlier work, so verifying the tail verifies nothing. Re-run the entire checklist, and check whether the pass flags were flipped by verification results or by the agent's own assertion. The machine form of that discipline: only a passing end-to-end verification may flip an item to passing.
Fourth, look for a dirtied environment: the same code appended twice, leftover temp files and debug switches, progress notes that fell behind. None of these turn a check red, but they detonate for whoever picks the work up next.
Finally, state the limits of your own method - this shows more experience than a perfect answer. End-to-end checks that go through the interface layer cannot see rendering or look-and-feel defects. Either add browser automation or write explicitly in the report that this class is uncovered. Never quietly count it as covered.
Expected follow-up: what if it edited the verification scripts themselves? Verification definitions must not sit in the same hands as the implementation. The checklist and check scripts belong to the harness, live in version control, and are reconciled before each run; the agent may only touch what is being implemented.
分析过程 · 先想清楚再作答
这题考的是对完成声明的怀疑态度。答「我去抽查几条」只能拿一半分——抽查是手段不是方法，而且抽到的大概率是它做得最像的那几条。面试官想听的是你怎么把「它说完成了」变成一个不依赖它自述的判据。
第一步先把两件事分开：**它声称的状态**和**环境的真实状态**。前者是它自己写的一行字，后者是可以用代码去查的事实。所有验证都必须落在后者上。这一步做不到，后面所有努力都在验一份自述。
第二步给出可验证的形式。清单里的每条 feature 必须是端到端可验证的，也就是能写出一段不看 Agent 任何输出、只对着运行中的系统发请求就能判通过与否的检查。这一条决定了清单该怎么写：「优化一下搜索」不可验证，「搜索标题时不区分大小写地做子串匹配」可验证。
第三步是回归地跑，不是只跑最后那条。无人值守一夜会做几十条，后做的很容易踩坏先做的——只验最后一条等于没验。正确做法是把清单**整套**重跑一遍，并且看清单的通过标记是不是由验证结果翻的，而不是由 Agent 自己声称翻的。这条纪律的机器形式是：只有端到端验证通过才允许把那条改成通过。
第四步查一遍环境有没有被弄脏：同一段代码有没有被写两遍、有没有留下临时文件与调试开关、进度笔记有没有跟上。这一类问题不会让任何验证转红，但会在下一个人接手时炸。
最后要承认能力边界，这一点比全都答对更能显出工程经验：端到端验证走的是接口层（发请求看响应），测不到渲染类与观感类的缺陷。这类缺陷要么接浏览器自动化，要么就明确写进报告说「这一类没覆盖」——不许悄悄当成覆盖了。
可预期的追问是「那它把验证脚本本身改坏了怎么办」。答案是验证的定义不能和实现放在同一只手里：清单与验证脚本属于 harness 侧、进版本控制、每次运行前核对；Agent 只被允许改被实现的那部分。
Key points
Separate the claimed status from the real state of the environment and verify only the latter.
Every checklist item must be end-to-end verifiable by driving the running system, ignoring agent output.
Re-run the whole suite, not just the last item - later work routinely breaks earlier work.
Only a verification result may flip an item to passing, never the agent's own assertion.
Check for a dirtied environment: duplicate writes, leftover temp files, stale progress notes.
State the limits: interface-level checks miss rendering defects, so add browser automation or declare the gap in the report.
答题要点
先分开它声称的状态与环境的真实状态，所有验证只落在后者上。
清单每条都必须端到端可验证：不看 Agent 输出、只对运行中的系统发请求就能判。
整套回归重跑，不只验最后一条——后做的很容易踩坏先做的。
通过标记只能由验证结果翻，不能由 Agent 自己声称翻。
查环境有没有被弄脏：重复写入、临时文件、进度笔记是否跟上。
承认能力边界：接口层验证测不到渲染类缺陷，要么接浏览器自动化，要么在报告里写明未覆盖。
Same model, same prompts, but swapping the harness lifts the benchmark score by more than ten points. What does that tell you?同一个模型、同一套提示词，换一个 harness 成绩提升十几个点。这说明了什么？
Common in ChinaCommon overseasIntermediate#harness-engineering#benchmarks#model-capability
How to reason about it · think before answering
This is open-ended and tests whether you carry a mental map dividing model capability from scaffolding. 'It shows the harness matters' is a non-answer; say which part matters, and where the conclusion does and does not generalize.
Lead with the fact and the magnitude: LangChain published a run where, with no model change and harness changes only, Terminal Bench 2.0 went from 52.8 to 66.5, a gain of 13.7 points, moving from Top 30 into the Top 5. Thirteen points is typically a model generation, and not a line of the model changed.
First conclusion: a model's capability and the capability it demonstrates on a benchmark are two different things. The score is capability multiplied by how well the scaffolding lets that capability out. If the scaffolding term is 0.7, a stronger model only buys you 0.7 of the gain - while lifting the scaffolding from 0.7 to 0.9 costs nothing like re-evaluating a model generation. The two paths have completely different cost structures.
Second conclusion: look at what actually changed. None of the three effective changes made the model smarter. Each put a gate where the model was known to fail - force a verification pass before exit (against false completion), track repeated edits to one file and suggest a new approach (against looping), map the directory and available tooling up front (against flailing in an unfamiliar environment). The shape is always 'known failure mode plus a gate aimed at it', which is exactly this course's structure.
Third, the limits - do not over-generalize. The size of the win depends on how bad the previous scaffolding was and how long the task is. In interactive turn-taking the human is the harness, so you will not find ten points there. The wins live in unattended long-horizon runs, where no failure mode has a human backstop. Also note that the fourth change in that experiment was reasoning-budget allocation, which is model tuning rather than harness structure; folding it into 'the harness did it' conflates two things.
Then land it somewhere actionable, which is the most valuable sentence in the answer: before locating the win, ask whether anyone is watching this agent. For unattended systems, first check for authoritative state, a completion gate, and stall intervention. Miss any of the three and the money spent on a better model is probably wasted.
Expected follow-up: how would you prove the harness gets the credit? Controlled-experiment discipline - change exactly one variable, freeze model and prompts verbatim, and make sure the model side cannot perceive which mechanisms are enabled, or what you have is a false green.
分析过程 · 先想清楚再作答
这题是道开放题，考的是你心里有没有一张「模型能力与脚手架」的分工图。答「说明 harness 很重要」等于没答，得说清它重要在哪一段、以及这个结论能推到哪里、不能推到哪里。
先给事实和量级：LangChain 公开过一次实测，不换模型只改 harness，Terminal Bench 2.0 从 52.8 提到 66.5，涨了 13.7 点，排名从 Top 30 进了 Top 5。十三点七分通常是一代模型的差距，而这次一行模型都没换。
第一层结论：**模型的能力和它在基准上表现出来的能力，是两件事。** 分数是「模型能力 × 脚手架能不能把这份能力用出来」的乘积。脚手架那一项如果是 0.7，换一个更强的模型也只能拿到 0.7 倍的增益，而把脚手架从 0.7 提到 0.9 却不需要重新评测一整代模型——**这是成本结构完全不同的两条路**。
第二层结论是看它到底改了什么。那三处有用的改动都不是让模型更聪明，而是**在模型犯已知错误的地方加一道机关**：退出前强制跑一遍验证（治谎报完成）、追踪文件编辑次数太多就建议换思路（治打转）、一进来先映射目录与可用工具（治摸索环境）。共同形状是「已知失败模式 + 一道针对它的闸门」，这也正是本课七天的结构。
第三层是这个结论的边界，别过度推广。换 harness 的收益大小取决于原来的脚手架有多差、以及任务有多长：交互式的一问一答里人就是那个 harness，涨不了十几个点；真正的收益出现在无人值守的长任务里，因为那里每一个失败模式都没有人兜底。同时那次实验里第四个改动是推理预算分配，那属于模型调参而不是 harness 结构——把它一起算进「harness 的功劳」就是把两件事混了。
最后给出可执行的落点，面试里这一句最值钱：**定位收益之前先问这个 Agent 有没有人在旁边看着。** 无人值守的系统，先查它有没有权威状态、有没有完成闸门、有没有停滞干预；这三样缺一样，换模型的钱大概率白花。
可预期的追问是「那你怎么证明是 harness 的功劳而不是别的」。答案是对照实验的纪律：只动一个变量、模型与提示词逐字冻结，而且被测的模型侧不能感知当前开了哪些机关——否则你拿到的是假绿。
Key points
Give the magnitude: harness-only changes moved Terminal Bench 2.0 from 52.8 to 66.5, a 13.7 point gain, Top 30 into Top 5.
A score is capability times scaffolding; demonstrated capability is not the same as capability.
Improving scaffolding and swapping models have different cost structures - the former needs no model re-evaluation.
All three effective changes share a shape: a known failure mode plus a gate aimed at it, not a smarter model.
Limits: the win lives in unattended long-horizon work; in interactive use the human is the harness, and reasoning-budget tuning is not harness structure.
Actionable close: ask whether anyone is watching, then check for authoritative state, a completion gate, and stall intervention.
答题要点
给量级：不换模型只改 harness，Terminal Bench 2.0 从 52.8 到 66.5，涨 13.7 点，Top 30 进 Top 5。
分数是模型能力与脚手架的乘积，模型能力与它表现出来的能力是两件事。
改脚手架和换模型的成本结构完全不同，前者不需要重新评测一整代模型。
三处有用的改动形状相同：已知失败模式加一道针对它的闸门，而不是让模型更聪明。
边界：收益出现在无人值守的长任务里；交互式场景人就是 harness，涨不了这么多；推理预算属模型调参不算 harness。
落点：先问有没有人在旁边看着，再查权威状态、完成闸门、停滞干预这三样。

Today's Goals

Plain-Language Walkthrough

A shift handover: the last person went home, and the briefing was given exactly once

What makes long-horizon hard: nobody to fill in, nobody to call a halt, nobody to sign off

Five terms, settled now

The context window is volatile: compacted, truncated, reset, or swapped out entirely

One run across two context windows: degradation is missing evidence, not a dumber model

Why the model interface should take exactly one string

Changing the harness can beat changing the model

Somebody watching, versus nobody watching

Source Reading

Hands-On Lab

Interview Questions

Checklist and Tomorrow

Interview questions

Comments