Dayward AI
Week 1 · D7About 4 hours

Putting It Together: Let It Run All Night, Then Check the Work in the Morning

Merge six days of modules into a runtime you could hand to a colleague, run it long-horizon for real, then produce an overnight report a person can read in three minutes: what got finished, where it is stuck, why it stopped, and what a human has to decide next.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Can merge the six modules into one runtime and confirm each of them is still doing its job, rather than trusting a final report that happens to be green
  2. Can design an overnight report that lets a person say in three minutes what got finished, where it is stuck and why it stopped
  3. Can name the silent failure most likely to appear when independently built modules are merged into one system, and give a way to find it

Six days of modules are complete. Today does two things: run the whole stack unattended for a real long-horizon night, and then audit those six layers one at a time. When you finish, scroll back up and tick off the three goals.

Plain-Language Walkthrough

The first thing you do in the morning is not read what it said

You get in at eight. The agent has been running for ten hours. What do you look at first?

A beginner reads the last sentence it produced: all tasks complete. Somebody who has done this a few times looks at three other things instead: what the code looks like now, how many items on the feature list flipped to passing, and how many points in the commit history you could safely fall back to.

That distinction runs through the whole course. D5 already made it once — the model saying it is done is not grounds for flipping any state. Today is the last application of the same sentence, except the speaker has changed. It is now the harness itself: the report claiming forty items finished is not grounds for believing forty items are finished, either. You have to be able to recheck it on the spot.

So today's report carries one hard rule: every number in the report must be traceable to the authoritative state or to the feature list. The report is not a third record. It is a view of those two.

Who the report is for: the last comparison with D2

D2 wrote a summary. D7 writes a page of report. Both look like the same activity — turn state into prose — but the reader is different, and almost every choice comes out inverted.

The D2 rebuild summaryThe D7 overnight report
Readerthe model in the next context windowthe person checking the work in the morning
Purposemake it take the correct next steplet a person decide in three minutes whether to step in
Styleconclusion first, one fact per line, numberedconclusion first, then evidence, bad news at the top
Lengthas short as possible (you pay for it once per context window)long enough is fine (it gets read once)

But the two draw their numbers from the same source — one authoritative state and one feature list — so they cannot contradict each other. That is the last time the discipline held since D2 pays off.

Four writing disciplines for the report

One: bad news goes first. People read the report to decide whether to intervene, not to admire progress. So where it is stuck has to sit above what got finished.

Writing a chronological diary is the most common mistake. Step 1 did F01, step 2 did F02, and so on — none of those words help with the decision, and they will drown the three lines that do.

Two: every piece of bad news has to carry its next step. F07 failed is useless. F07 was attempted twice and then swapped away is useful — the first only tells you a problem exists, the second tells you what the harness already did and what is left for you.

Three: give an executable way to reproduce. The report has to end with how to continue the run and how to inspect the scene, or the reader will finish it and still have to go hunting for paths.

Four: no number that exists only in the report. This one gets broken the most because it reads as counterintuitive. An average seconds-per-step line looks professional, but if it cannot be traced back to the state, it is a third record that will drift. The D4 rule that the code and the list go into one commit, and the D6 rule that the ledger and the workspace are held at the same instant, both guard against the same failure. Today is the third instance.

Silent failure: the real danger once six layers are stacked

Now the actual point of the day.

When six modules get merged into one system, the dangerous outcome is not a crash — a crash tells you immediately. The dangerous outcome is silent failure: one layer has stopped doing anything, and the system keeps running, the report keeps printing, and not one error is raised.

This is not a hypothetical. It happened while building another course in this repository: a newly added hook dropped the benchmark pass rate from five of five to two of five, and the report came out anyway, with no error — because that report only counted how many runs completed, never whether the ones that used to pass still passed.

There is exactly one way to find it, and it is very plain:

Switch each module off, one at a time, and see whether the difference comes back.

That is the per-layer audit today's lab prints. Measured output:

TextText
(everything on, baseline)    16 features finished
D3 initialization phase      4 probes (baseline 1)                               still doing its job
D4 one commit per feature    peak uncommitted work 4 features (baseline 0)       still doing its job
D5 end-to-end gate           7 premature completion claims (0 with the gate on)  still doing its job
D6 constraint restatement    1 of 4 windows kept the rules (baseline 4 of 4)     still doing its job
D6 budget circuit breaker    outcome budget-exhausted, 5 steps (baseline 16)     still doing its job
D2 state layer               16 features finished (baseline 16)                  no difference when off

One line to take away: every layer needs a switch that turns it off

This is the most portable thing in the seven days:

Every layer you add needs a switch that turns it off. A layer with no switch is a layer you can never prove is still working.

It is also why this course ran a mutation check every single day. Today only gathers seven days of mutation checks into one table — and that table is itself the thing you should build the next time you face any layered system.

One thread through seven days

Looking back, the seven days were seven faces of one idea.

DayWhat moved out of the context windowWhere it lives instead
D1(first, prove the problem is real)none yet
D2progressthe authoritative ledger on disk
D3knowledge of the environmentinit.sh, the progress notes, the first commit
D4grounds for choosing what to work onthe feature list file and the verified commits
D5the criterion for donethe end-to-end acceptance cases
D6the rulesconstraints restated in every context window
D7the conclusionone page written for a person

The one-line thesis has not changed since day one: the context is volatile; the state is authoritative. Everything the seven days did was move one more thing out of that volatile place and give it a carrier somebody else can open.

What this course deliberately does not touch

The last thing to state is the boundary, which is both honest and a map for where to go next.

TopicWhere it belongs
Sandboxes, container isolation, egress allowlists, permissionsAgent Security in 5 Days
Evaluation method: pass rates over repeated trials, model judges, telemetryAgent Evals and Observability in 7 Days
Context compression itself, what to drop and what to keepContext Engineering in 5 Days; this course only teaches one consequence of compaction
The tool loop, session recovery, todo renderingBuild Your Own Coding Agent in 21 Days

This course only ever answered one question: with nobody watching, how do you get it through the night. Every other question has a better home.

Source Reading

Today adds one module. Two places repay a slow read.

Position one: the first line. For many readers it is the only line they will read, so it has to answer three things at once.

headline.js
// How long it ran, how much it finished, why it stopped -- all three, always.
// Writing only "completed" says nothing at all.
function headline(input) {
  const done = input.state.done.length
  const total = input.features.length
  const dur = minutes(input.finishedAt - input.startedAt)
  const why =
    input.outcome.kind === 'completed'
      ? 'feature list fully complete'
      : input.outcome.kind === 'stalled'
        ? `halted: too many features swapped away (${input.skipped.join(' ')})`
        : input.outcome.kind === 'budget-exhausted'
          ? `halted: ${describeBreach(input.breach)}`
          : `crashed: ${input.outcome.error}`
  return `Ran for ${dur}, finished ${done}/${total} features, ${why}.`
}

Notice that the four outcomes each get their own wording, never folded into a vague not finished. What you need to know in the morning is whether the clock ran out or whether nothing got built at all, and those two call for completely different responses.

Position two: how the audit table is produced. The shape of the code is the habit made concrete.

checkup.js
// Run the baseline once, then switch off exactly one layer per run and watch
// whether that layer's own metric gets worse. The point is that every layer
// has its own metric, instead of all of them being judged by how many features
// finished -- some layers do not move that number at all. What they move is
// how much you lose when something goes wrong.
const base = await once(root, NIGHT)
const noInit = await once(root, { ...NIGHT, initMode: 'inline' })
const noCommit = await once(root, { ...NIGHT, commitMode: 'per-window' })
const noPin = await once(root, { ...NIGHT, pinConstraints: false })
 
report('D3 initialization phase', noInit.probeCount > base.probeCount)
report('D4 one commit per feature', noCommit.maxUnsaved > base.maxUnsaved)
report('D6 constraint restatement', keptWindows(noPin) < keptWindows(base))

Every layer has its own metric, and that is the load-bearing part. One commit per feature does not change how many features finish at all — D4 measured it directly, nine against nine. What it changes is how much you lose when something goes wrong. An audit table that only reads the finished count would rule that layer useless, and somebody would delete it at three in the morning.

Hands-On Lab

🧪 Day 7: Run It Through the Night, Check the Work in the Morning

Code location: labs/agent-harness-7days/day-07-overnight

Two exercises, both in src/report/overnight.ts: the headline and the full page. Every module from the previous six days carries over verbatim.

  1. Read the comment block at the top of src/report/overnight.ts, and map the four differences against the D2 summary.
  2. Fill in headline. All three facts are required, and each of the four outcomes gets its own wording.
  3. Fill in renderReport. Bad news first, and every number traceable to the state.
  4. Run MOCK=1 pnpm selftest until all 100 self-test assertions are green. The starter is red on 6 of them to begin with.
  5. Run MOCK=1 pnpm start: read the overnight report first, then the audit table.

The measured night: 40 of 40 features finished across 7 context windows and 40 steps, with 40 verified commits, constraint survival reading 3 -> 3 -> 3 -> 3 -> 3 -> 3 -> 3 across the seven windows, and 0 features failing the recheck that reverifies every claimed item.

Then do the manual acceptance item in the README: read only the first line of the report, and ask yourself whether you need to step in. If the first line leaves you unable to answer, that line is written wrong. It is the smallest possible test of the three-minute standard.

Interview Questions

Today's four questions close out the course: the silent failure that appears when modules are merged, the tradeoffs in a report, how to judge whether an overnight result was any good, and which single module you would keep if you could only keep one.

The third is worth rehearsing. Eighteen of thirty finished overnight, good or bad — the correct answer is to ask three questions before concluding anything, not to produce a verdict. That is precisely what this course has argued against from D1 through D7: the better-looking number very often maps to the worse reality.

Checklist and Tomorrow

  • Can merge the six modules into one runtime and confirm each of them is still doing its job, rather than trusting a final report that happens to be green
  • Can design an overnight report that lets a person say in three minutes what got finished, where it is stuck and why it stopped
  • Can name the silent failure most likely to appear when independently built modules are merged into one system, and give a way to find it
  • Can state the four writing disciplines, especially why no number may exist only in the report
  • Can say, for each of the seven days, what moved out of the context window and where it lives instead
  • Got all 100 self-test assertions green with MOCK=1 pnpm selftest, and read the overnight report and the audit table with your own eyes
  • Can name the four topics this course explicitly leaves alone, and where each one belongs
  • Can answer at least three of the four interview questions without looking at the key points

Seven days end here. What you hold now is a runtime that can go through the night on its own: progress lives on disk, knowledge of the environment lives in files, the grounds for choosing live in a list, the criterion for done lives in acceptance cases, the rules live in the opening of every context window, and the whole thing hands you one page a person can read in three minutes.

Where to go next depends on where your own next bottleneck turns out to be. If the agent starts touching external systems and untrusted input, go to the security course. If you need to answer whether this agent is actually any good, go to the evals course. If the context itself becomes the bottleneck, go to the context engineering course. And if you want to put this stack on something real, you can start right now — swap the target for your own repository and the forty-item list for your own requirements, and not one line of the seven-day structure has to change.

Interview questions

  • Six modules developed independently get merged into one system - what worries you most, and how would you find it?六个独立开发的模块合并进一个系统,你最担心出现什么问题?怎么发现它?
    Common in ChinaCommon overseasDeep dive#silent-failure#module-merge#switch-experiment

    How to reason about it · think before answering

    1. Start by naming the right worry: a crash is not the dangerous case. A crash gives you an error, a stack and a timestamp - it is the cheapest kind of failure because you cannot miss it. The expensive kind is silent failure: one layer has stopped doing anything, while the whole thing still runs, the report still comes out, and nothing reports an error. 'I would write integration tests' is not wrong but not enough - what breaks on a merge is precisely the path where every module passes its own tests and nobody covers their interaction.
    2. Why merges are especially prone to this deserves its own sentence: while developed separately, each module's tests rest on the assumptions of its own day. After the merge, a newly added gate can block an older flow while that older flow's report stays green - what a gate blocks does not become an error, it becomes a smaller number, and nobody knows what that number should have been. This class of problem first exists at the moment of the merge, so no amount of separate testing finds it.
    3. Only one discovery method really works: switch each module off one at a time and see whether the difference comes back. Looking at the total score is useless - the total is the combined effect of six layers, any one of which can fail while another masks it, and the masked result often looks better. A switch experiment turns 'is this layer doing anything' into an observable delta instead of an assumption.
    4. The checkup table in this course's D7 was built exactly that way, and five measured rows are worth memorizing: switch off D3's initialization stage and environment probing rises from 1 to 4 times; switch off D4's commit-per-feature and peak uncommitted work rises from 0 to 4 items; switch off D5's end-to-end gate and false completions rise from 0 to 7; switch off D6's constraint restatement and the windows that keep the rules drop from 4/4 to 1/4; tighten the budget and the outcome becomes a breaker trip after 5 steps against a baseline of 16. Every row is a delta that comes back.
    5. A layer that makes no difference when switched off is a finding, not noise. In this checkup, D2's state layer is exactly that case: switch it off and the completed count is still 16, identical to the baseline. Digging in shows it is not broken but superseded by D4 - once task selection moved into the harness (pick the next item by reading the checklist off disk), 'redo the work because the context holds no evidence of progress' went from not happening to being impossible. That has been written into the self-test as a positive assertion, so nobody later 'fixes' it without realizing what they are undoing.
    6. So the habit to take away is one sentence: every layer you add needs a switch that turns it off. A layer without a switch leaves you permanently unable to tell whether it stopped working long ago. That is also why this course runs a mutation check every single day - switch off the layer that day introduced and confirm the symptom returns. D7 merely collects seven days of mutation checks into one table, so every layer is worth a delta you can reproduce on the spot.
    7. Expected follow-up: does that not mean running everything seven times? Yes, and one checkup costs far less than one wasted night, and it automates naturally. Also the switch must be a real switch - a configuration flag, not a few lines commented out by hand. A checkup that depends on commented-out code is itself a thing somebody has to remember, and remembering is what this whole course argues against.

    分析过程 · 先想清楚再作答

    1. 这题第一步是把担心的对象说对:**最危险的不是崩。** 崩了有报错、有栈、有时间点,是最便宜的一类失败,你一定会知道。真正贵的是**静默失效**——某一层其实已经不起作用了,而整体照样跑、报告照样出、一个错都不报。答「我会写集成测试」不算错但不够:合并之后出问题的恰恰是每个模块单独测都绿、而互相影响的那条路径谁也没覆盖到。
    2. 为什么合并特别容易出这个,值得单独说一句:各模块单独开发时,它们的测试都建立在自己那一天的前提上。合并之后**新加的闸门可能让旧流程过不去,而旧流程的报告仍然是绿的**——闸门拦下来的东西不会变成错误,只会变成一个更小的数字,而没有人知道那个数字本来该是多少。这类问题在合并那一刻才第一次存在,所以单独测多少遍都测不出来。
    3. 发现方法只有一条真正管用:**逐个模块开关一遍,看差别回不回来。** 只看总分没用——总分是六层共同作用的结果,任何一层失效都可能被别的层掩盖,而且掩盖之后总分往往更好看。开关实验把「这一层有没有在起作用」变成一个可观测的差值,而不是一句「应该在吧」。
    4. 本课 D7 的体检表就是这么做出来的,五行实测可以直接背:关掉 D3 的初始化阶段,环境探测从 1 次涨到 4 次;关掉 D4 的一条一提交,未提交工作峰值从 0 条涨到 4 条;关掉 D5 的端到端闸门,谎报从 0 条涨到 7 条;关掉 D6 的约束复述,保住规矩的窗口从 4/4 掉到 1/4;把预算收紧,结局变成熔断、5 步就停(基准 16 步)。每一行都是一个能回来的差值。
    5. **一个关掉之后没差别的层,要当成发现而不是当成噪声。** 本课体检里 D2 的状态层就是这种情况:关掉它完成条数仍然是 16 条,与基准一模一样。查下去发现它不是坏了,是**被 D4 取代了**——任务选取上收到 harness 之后(读磁盘清单挑下一条),「上下文里没有进度证据就重做」这件事从「不再发生」变成了「不可能发生」。这条已经写成一条**正向断言**进了自检,防止将来有人把它「修好」而不自知。
    6. 所以带走的习惯是一句话:**每加一层,就要有一个关掉它的开关。没有开关的层,你永远不知道它是不是早就不干活了。** 这也是这门课每天都做变异检验的原因——关掉当天新增的那层,确认现象重现;D7 只是把七天的变异检验汇总成同一张表,让每一层的价值都变成一个可以当场复现的差值。
    7. 可预期的追问是「那不是每次都要跑七遍吗」。是的,但一次体检的成本远低于一夜白跑的成本,而且它天然可以自动化。另外开关必须是**真开关**(配置项,而不是临时注释掉几行代码)——靠注释做的体检本身就是一件要靠人记住的事,而这门课从头到尾都在反对靠人记住。

    Key points

    • The worry is not a crash but silent failure: a layer stops working while everything runs, reports and raises nothing.
    • Merges invite it: a new gate blocks an older flow while that flow's report stays green.
    • Find it by switching each module off and checking whether the difference returns - far better than reading the total.
    • Measured checkup: probing 1 to 4, uncommitted peak 0 to 4, false completions 0 to 7, constraints 4/4 to 1/4, tightened budget trips the breaker.
    • A layer with no difference is a finding: D2's state layer was superseded by D4 - degradation went from not happening to impossible.
    • That was written into the self-test as a positive assertion so nobody silently 'fixes' it later.
    • Habit: every layer needs an off switch, and it must be a config flag rather than commented-out lines.

    答题要点

    • 最担心的不是崩,是静默失效:某一层不起作用了,而整体照样跑、报告照样出、一个错都不报。
    • 合并特别容易出这个:新加的闸门让旧流程过不去,而旧流程的报告仍然是绿的。
    • 发现方法是逐个模块开关一遍看差别回不回来,比只看总分有用得多。
    • 实测体检表:初始化 1→4 次探测、一条一提交 0→4 条未提交峰值、闸门 0→7 条谎报、约束复述 4/4→1/4、预算收紧则熔断。
    • 关掉之后没差别的层要当成发现:D2 状态层被 D4 取代了,退化从不再发生变成不可能发生。
    • 那条已写成正向断言进自检,防止将来有人把它「修好」而不自知。
    • 习惯:每加一层就要有一个关掉它的开关,而且必须是配置项不是注释掉几行。
  • Design the morning report for an unattended run - what goes in, and what gets cut?设计一份无人值守运行的早间报告,你会放哪些内容、砍掉哪些内容?
    Common in ChinaCommon overseasIntermediate#report-design#human-readable#single-source

    How to reason about it · think before answering

    1. State the criterion before the contents: one glance should show what got done and where it is stuck. Once the criterion is fixed, the contents follow. 'Format the log nicely' misses - a log's reader is someone already debugging, while this report's reader is a person who just woke up, is not yet in context, and only wants to know whether to step in. The same material written for different readers has a completely different structure.
    2. This course puts it side by side with the window-rebuild summary, and all four properties differ: reader (the model in the next window versus the person doing the morning check), purpose (make the right next move versus decide in thirty seconds whether to intervene), style (conclusion first, one thing per line versus conclusion then evidence, bad news first), length (as short as possible, since every window pays for it, versus long enough, since it is read once). Yet both draw their numbers from the same source - the same authoritative state and the same checklist - so they cannot contradict each other.
    3. Discipline one: bad news first. Writing a chronological play-by-play is the most common error, and the easiest to fall into because a log is already in that order. 'Step 1 did F01' contributes nothing to the decision of whether to intervene. Putting the whole 'where it is stuck' section ahead of 'what got done' is the only real structural decision in this report.
    4. Discipline two: every piece of bad news must carry the next step. 'F07 failed' is useless; 'F07 was attempted twice, rolled back to the last green point, and skipped' is useful. The test is handy: after reading this line, does the person know what to do next? If not, the line is unfinished - it has merely moved a question from the machine to the human.
    5. Discipline three: give an executable way to reproduce. The final section must answer how to resume the run and how to inspect the scene, otherwise the reader has to go hunting for the state files and the repository path. This one is almost free to add, and leaving it out costs five minutes no matter how good the rest of the report is.
    6. Discipline four is the one most often missed: no number may exist only in the report. Every figure must be traceable to the state or the checklist, or it becomes a third record that will eventually diverge. The last line of this course's report is a recheck: the checklist claims 40 items, and re-verifying each one finds 0 that fail. The report is not a new record, only a view over the authoritative state and the checklist, so it cannot fight with them.
    7. Say what gets cut too: the step-by-step play-by-play, everything the model said, and the intermediate reasoning - all of it. Expected follow-up: then how do you debug? Debugging happens in the logs and at the scene, and the report's job is only to lead the person there, which is exactly what discipline three buys. Stuffing debugging material into the report makes the just-woken reader read logs on the machine's behalf, and the three-minute standard dies on the spot.

    分析过程 · 先想清楚再作答

    1. 先把判据说出来,再谈内容:**一眼能看出干了什么、卡在哪里。** 判据定了,内容自己就排出来了。答「把日志格式化一下输出」是跑偏的——日志的读者是正在排查问题的人,报告的读者是**一个刚睡醒、还没进入状态、只想知道要不要介入的人**。同一份材料写给不同的人,结构完全不一样。
    2. 本课把它与窗口重建摘要摆在一起做对照,四项全都不同:读者(下一个窗口里的**模型** 对比 早上来验收的**人**)、目的(让它做对下一步 对比 让人在三十秒内决定要不要介入)、写法(结论前置、一行一件事 对比 先结论再证据、坏消息放前面)、长度(越短越好,每个窗口都要付一次钱 对比 够用就行,只读一次)。但两者**数字同源**——都来自同一份权威状态与同一份清单,所以它们不可能互相矛盾。
    3. 第一条纪律:**坏消息放前面。** 按时间顺序写流水账是最常见的错误,也是最容易犯的,因为日志天然就是那个顺序。「第 1 步做了 F01」对「我要不要介入」这个决定毫无帮助。把「卡在哪里」整节排在「做完了什么」前面,是这份报告唯一真正的结构决定。
    4. 第二条:**每条坏消息都要带得起下一步。**「F07 失败」没用;「F07 试了 2 次、已回退到上一个绿点、已跳过」才有用。判据很好用:读完这一条,人知不知道下一步该做什么。不知道,这条就还没写完——它只是把一个疑问从机器转移给了人。
    5. 第三条:**给可执行的复现方式。** 最后一节必须回答「怎么接着跑」和「怎么看现场」,否则人读完报告还得自己去翻目录找状态文件和仓库路径。这一条便宜到几乎不用想,但漏掉它会让前面写得再好的报告都多花五分钟。
    6. 第四条也是最容易被忽略的:**不许出现报告独有的数字。** 报告里每个数字都要能在状态或清单里找到出处,否则它就是**第三份会分叉的记录**。本课那份报告最后一行是一句复核:清单声称 40 条,逐条重验之后跑不通的有 0 条——报告不是新记录,它只是权威状态与清单的一个视图,所以它不可能和它们打架。
    7. 砍掉什么同样要说清:逐步流水账、模型说过的话、中间推理,全砍。可预期的追问是「那出了问题怎么排查」——排查看日志和现场,报告只负责把人**导到**现场,这正是第三条纪律的用处。把排查材料塞进报告,等于让那个刚睡醒的人替机器读日志,三分钟标准立刻作废。

    Key points

    • One criterion: a glance shows what got done and where it is stuck; the reader just woke up and is not in context.
    • It differs from the rebuild summary on reader, purpose, style and length, yet shares its numbers, so the two cannot contradict.
    • Bad news first: a chronological play-by-play is the most common mistake.
    • Each piece of bad news must carry a next step: 'F07 failed' is useless, 'F07 tried twice and skipped' is useful.
    • Give an executable way to resume the run and inspect the scene.
    • No number may exist only in the report, or it becomes a third diverging record; the real report ends with an item-by-item recheck.
    • Cut the play-by-play, the model's utterances and the intermediate reasoning; debugging lives in the logs, the report only points there.

    答题要点

    • 判据一句话:一眼能看出干了什么、卡在哪里;读者是一个刚睡醒、还没进入状态的人。
    • 与窗口重建摘要四项全不同(读者/目的/写法/长度),但数字同源,所以不可能互相矛盾。
    • 坏消息放前面:按时间顺序写流水账是最常见的错误。
    • 每条坏消息要带得起下一步:「F07 失败」没用,「F07 试了 2 次、已跳过」才有用。
    • 给可执行的复现方式:怎么接着跑、怎么看现场。
    • 不许出现报告独有的数字,否则它就是第三份会分叉的记录;实测报告最后一行是逐条复核。
    • 砍掉逐步流水账、模型说过的话、中间推理;排查看日志,报告只负责把人导到现场。
  • An agent ran overnight and finished 18 of 30 items - is that a good result or a bad one?Agent 跑了一夜完成了三十条里的十八条。这个结果算好还是不好?
    Common in ChinaCommon overseasDeep dive#acceptance-criteria#false-progress#stop-reason

    How to reason about it · think before answering

    1. The right answer is to ask three questions before concluding, not to hand over a verdict. Any reply that opens with 'not bad' or 'pretty poor' is already wrong, because the ratio 18 of 30 carries no information you can act on. Judging by completion count alone is exactly what this course argues against from D1 through D7 - a better-looking number often corresponds to a worse reality.
    2. Question one: are the remaining twelve items untouched, or attempted and failed? On a dashboard the two look identical, both simply incomplete, but they mean opposite things. Untouched means the budget ran out, and another night or a bigger budget continues the work. Attempted and failed means something is stuck, and any additional budget burns in the same place. What separates them is the attempt counts and the skipped set, not the completion count.
    3. Question two: are those eighteen genuinely usable, or merely marked done on the checklist? D5 of this course answers that with a measurement: same code, same model, same script, with the only difference being what counts as verification. The shallow-check run claimed 33 items, re-verification found only 24 actually working, and 9 were false. The end-to-end gate run claimed 20, had 20 working, and 0 false. The run that claimed more produced less.
    4. Those numbers carry a further lesson: of the 9 false completions, only 2 came from genuinely bad patches - the other 7 were correct work dragged down by an earlier item. False completions are therefore not isolated events; one unverified feature contaminates a stretch of the items after it, which means 'how many of the eighteen are fake' cannot be answered by spot-checking one or two.
    5. Question three: why did it stop? The four outcomes call for completely different handling. Finished (the checklist is complete - go and accept the work), budget exhausted (it stopped at the line, so the next step is more budget or less scope), halted (too many items were skipped, so the harness is saying a premise is wrong and continuing only burns money), and crashed (look at where it crashed before deciding to resume). Only the first two are safe to resume blindly; the other two need a human look first. That is why the first line of the report must answer why it stopped.
    6. Only after all three questions does a conclusion mean anything, and it lands at one of two extremes. If the twelve were untouched, the eighteen all survive re-verification, and it stopped at the budget line, this is a good result and the next step is simply more budget. If the twelve were attempted and failed, several of the eighteen were never really verified, and it halted on too many skips, then a premise is broken and more budget only burns more money. One 18 of 30, two opposite next steps.
    7. Expected follow-up: which numbers should you actually look at? Three. The count that still works after item-by-item re-verification, which is the only trustworthy progress figure; the skipped set, which shows what class of problem it is stuck on; and the stop reason, which decides whether the next step belongs to a person or a machine. All three belong in a decent overnight report, and all three trace back to the authoritative state and the checklist.

    分析过程 · 先想清楚再作答

    1. 这题的正解是**先问三个问题再下结论**,不是直接给一个答案。任何张口就说「不错」或「太差」的回答都已经错了,因为 18/30 这个比值本身不携带任何可用于决策的信息。只看完成数,正是这门课从 D1 到 D7 一直在反对的事——**更好看的数字往往对应更差的现实。**
    2. 问题一:**剩下那十二条是没轮到,还是试过没成?** 这两种在报表上长得完全一样(都是「未完成」),但含义天差地别。没轮到只是预算不够,加一夜或加预算就能继续;试过没成说明有东西卡住了,再给多少预算都会烧在同一个地方。能区分它们的是尝试次数与被换掉的集合,不是完成数。
    3. 问题二:**那十八条是真的能用,还是清单上写着完成?** 本课 D5 有一组实测正好回答这个:同一份代码、同一个模型、同一份剧本,唯一的差是拿什么当验证——浅检查那趟清单声称 33 条、逐条重验只有 24 条真的能用、谎报 9 条;端到端闸门那趟声称 20 条、能用 20 条、谎报 0 条。**声称得多的那趟,真实产出反而少。**
    4. 这组数字还有一层值得说:那 9 条谎报里只有 2 条是真正的坏补丁,另外 7 条本来都是做对的,是被前面一条连累的。所以谎报不是孤立事件——**一条没真验的 feature 会污染它后面的一串**,这意味着「18 条里有几条是假的」这个问题不能靠抽查一两条回答。
    5. 问题三:**它为什么停下来?** 四种结局的处理完全不同:跑完了(清单全部完成,直接验收)、预算熔断(到线就停,下一步是加预算或砍范围)、停机(被换掉的条目太多,harness 在说前提出了问题,继续跑只是烧钱)、崩了(先看崩在哪,再决定要不要续跑)。只有跑完与熔断适合无脑续跑,另外两种都要人先看一眼。所以报告的第一行必须回答「为什么停」。
    6. 三个问题问完,结论才有意义,而且会落到两端之一:如果十二条是没轮到、十八条逐条重验都能用、停因为预算到线——那是一个**很好的结果**,下一步只是加预算;如果十二条是试过没成、十八条里有若干没真验、停机是因为换掉太多——那是**前提出了问题**,加预算只会烧更多的钱。同一个 18/30,两种完全相反的下一步。
    7. 可预期的追问是「那到底该看哪几个数」。三个:**逐条复核之后仍然能用的条数**(唯一可信的进度)、**被换掉的集合**(卡在哪类问题上)、**停机原因**(决定下一步是人做还是机器做)。这三个数都在一份合格的夜跑报告里,而且都能在权威状态与清单里找到出处。

    Key points

    • The right answer is three questions before a verdict; the ratio itself carries nothing you can act on.
    • One: are the remaining twelve untouched or attempted and failed - budget shortfall versus something stuck.
    • Two: are the eighteen usable or merely marked done - D5 measured 33 claimed, 24 usable, 9 false.
    • False completions cluster: only 2 of the 9 were bad patches, 7 were dragged down by an earlier item, so spot checks cannot answer it.
    • Three: why it stopped - finished, budget, halt or crash; only the first two are safe to resume blindly.
    • Then conclude: usable items plus a budget stop is a good night; stuck items plus false completions plus a halt means a broken premise.
    • The three numbers to read: items surviving re-verification, the skipped set, and the stop reason.

    答题要点

    • 正解是先问三个问题再下结论,18/30 这个比值本身不携带可用于决策的信息。
    • 问题一:剩下十二条是没轮到还是试过没成——前者只是预算不够,后者是有东西卡住了。
    • 问题二:十八条是真的能用还是清单上写着完成——D5 实测浅检查声称 33 条、实际只有 24 条能用、谎报 9 条。
    • 谎报不是孤立的:那 9 条里只有 2 条是坏补丁,另外 7 条是被前面一条连累的,所以抽查一两条回答不了。
    • 问题三:为什么停——跑完 / 熔断 / 停机 / 崩了,四种处理完全不同,只有前两种适合无脑续跑。
    • 问完才有结论:都能用且停在预算线上就是好结果;有卡点又有谎报还因换掉太多而停机,就是前提出了问题。
    • 该看的三个数:逐条复核后仍能用的条数、被换掉的集合、停机原因。
  • Looking back at the whole harness, if you could keep only one module, which one and why?回顾整套 harness,如果只能保留一个模块,你留哪个?为什么?
    Common in ChinaCommon overseasDeep dive#tradeoff-reasoning#trust-vs-throughput#course-recap

    How to reason about it · think before answering

    1. There is no standard answer here, but there are better and worse ways to reason, and the reasoning is what the interviewer wants. Naming a module and justifying it with 'it matters most' scores nothing whichever one you pick, because that sentence is true of all six. What the question really asks is: on what basis do you rank these layers?
    2. Give the criterion before the answer, never the other way round. The criterion is one sentence: which module, when absent, makes the other modules' output untrustworthy rather than merely smaller? Smaller can be fixed with budget - run another night, allow more steps, and the number climbs back. Untrustworthy cannot be fixed, and it is worse than having no number at all, because plausible fake progress stops people from doubting until much later, when the whole line has to be redone.
    3. By that criterion, the first answer is to keep the end-to-end gate: without it, everything the other modules produce is untrustworthy progress. The state layer faithfully records a batch of fake completions, the green points on the checklist are fake green points, and the report computes a better-looking number from that data - which is exactly the shape of the D5 measurement (33 claimed, 24 usable, 9 false, with false completions dragging down items that were originally correct). The gate is the one module that decides whether the numbers are real.
    4. The second answer, to keep the state layer, works just as well: without it the run does not survive a single night. The measured contrast is 3 items against 9, and 6 wasted steps against 0 - swap the window and it guesses from scratch, so running longer only spins in place. Trustworthy but unfinished versus finished but untrustworthy: which is worse depends on the setting, which is precisely why this question can be answered either way.
    5. So the right move is to state the conditions rather than hedge. If the night's output goes straight into use, keep the gate, because wrong output costs far more than no output. If this is an exploratory long run whose output a person will review item by item anyway, keep the state layer, because getting it to finish comes first and correctness is backstopped downstream. Saying under which condition you would switch answers shows more thinking than the answer itself.
    6. It is worth citing the seven-day thread, which explains why these six modules are not interchangeable: what gets moved out of the window is progress (D2), environment knowledge (D3), the basis for choosing (D4), the criterion for done (D5), the rules (D6) and the conclusion (D7) - six different things, each given a carrier somebody else can open. The through line stays the same: context is volatile, state is authoritative.
    7. Expected follow-up: what is still missing from this stack? Volunteer the boundaries: sandboxing and permissions belong to the security course, evaluation metrics to the evaluation course, and context compaction techniques themselves to the context engineering course - name the owner, do not expand. Being able to state where your own stack ends is far more convincing than reciting one more module name.

    分析过程 · 先想清楚再作答

    1. 这题没有标准答案,但**推理方式有好坏**,而且面试官要的就是推理方式。上来就报一个模块名、理由是「它最重要」,无论选哪个都拿不到分——因为那句话对任何一个模块都成立。这题真正在问的是:你凭什么给这六层排序。
    2. 先给判据,再给答案,顺序不能反。判据是一句话:**哪一个模块缺席时,其余模块的产出会变成不可信,而不是变少?** 变少还能靠加预算补——多跑一夜、多给几步,数字会涨回来;**不可信没法补**,而且比没有数字更糟,因为一个看起来合理的假进度会让人停止怀疑,直到很久以后才发现整条线都要重来。
    3. 按这个判据,第一个答案是**留端到端闸门**:没有它,别的模块产出的都是不可信的进度。状态层会忠实地记下一批假的完成,清单上的绿点是假的绿点,报告会基于这些数据算出一个更好看的数字——D5 那组实测正是这个形状(声称 33 条、实际能用 24 条、谎报 9 条,而且谎报会连累后面本来做对的条目)。闸门是那唯一一个决定「数字是不是真的」的模块。
    4. 第二个答案是**留状态层**,同样说得通:没有它连一夜都跑不下来。实测对照是完成条数 3 条对 9 条、白干步数 6 步对 0 步——窗口一换就从头乱猜,跑得再久也只是在原地打转。可信但跑不完,与跑得完但不可信,哪个更糟取决于场景,这正是这题可以两边答的原因。
    5. 所以正确的做法是**给出场景条件而不是骑墙**:如果这一夜的产出会被人直接拿去用,选闸门(错误的产出比没有产出贵得多);如果这是一个探索性的长跑、产出无论如何还会被人逐条过一遍,选状态层(先让它能跑完,正确性由后面的人兜底)。**说清楚在什么条件下换答案,比答案本身更能说明你想清楚了。**
    6. 回答里值得顺带引一条七天的线索,它能说明这六个模块为什么不是可互相替代的:搬出窗口的分别是**进度**(D2)、**环境知识**(D3)、**判断依据**(D4)、**完成的判据**(D5)、**规矩**(D6)、**结论**(D7)——六样不同的东西,各自有一个能被别人打开的载体。主线一句话:**上下文是易失的,状态才是权威的。**
    7. 可预期的追问是「那这套东西还缺什么」。要能主动说清边界:沙箱与权限归安全那门课,评估指标归评估那门课,上下文压缩技术本身归上下文工程那门课——**只说归属,不展开**。能说出自己这套东西的边界在哪,比多背一个模块名有说服力得多。

    Key points

    • No standard answer, but the reasoning is judged: give the criterion first, then the pick.
    • Criterion: which module's absence makes the rest untrustworthy rather than smaller - smaller is fixable with budget, untrustworthy is not.
    • Answer A, keep the end-to-end gate: without it the state records fake completions, the checklist holds fake green points, and the report computes a prettier lie.
    • Answer B, keep the state layer: without it the night does not survive at all - 3 items against 9, 6 wasted steps against 0.
    • State the condition instead of hedging: output used directly favors the gate, an exploratory run reviewed afterwards favors the state layer.
    • The seven-day thread: progress, environment knowledge, the basis for choosing, the criterion for done, the rules, the conclusion - context is volatile, state is authoritative.
    • Volunteer the boundaries: sandboxing to the security course, evaluation metrics to the evaluation course, compaction itself to the context engineering course - owner only, no expansion.

    答题要点

    • 这题没有标准答案,但推理方式有好坏:先给判据再给答案,顺序不能反。
    • 判据:哪个模块缺席时其余模块的产出会变成不可信,而不是变少——变少能靠加预算补,不可信没法补。
    • 答案 A 留端到端闸门:没有它,状态层记的是假完成、清单是假绿点、报告算出更好看的假数字。
    • 答案 B 留状态层:没有它连一夜都跑不下来,实测完成 3 条对 9 条、白干 6 步对 0 步。
    • 给场景条件而不是骑墙:产出直接拿去用就选闸门,探索性长跑且后面有人逐条过就选状态层。
    • 七天线索:搬出窗口的是进度、环境知识、判断依据、完成的判据、规矩、结论;主线是上下文易失、状态权威。
    • 主动说边界:沙箱与权限归安全那门课、评估指标归评估那门课、压缩技术本身归上下文工程那门课,只说归属不展开。

Comments