The Regression Gate: Blocking a Drop Before It Merges

Interview questions

• Evaluation scores fluctuate on every run. How do you set a gate threshold that neither misses regressions nor fires false alarms every day?评估分数每次都在抖，你怎么定一个既不漏报也不天天误报的门禁阈值？

  Common in ChinaCommon overseasDeep dive#ci#regression#thresholds

  How to reason about it · think before answering

  1. This tests whether you know thresholds must be measured. Answering 'block if it drops more than five points' earns half credit regardless of the number, because the number was guessed.
  2. Step one is to measure the noise: hold the task set, trial count, model and prompt fixed, vary only the random seed, run twenty or thirty rounds, and record the maximum, minimum and mean pass rate. The width of that band is how much you can lose to luck alone.
  3. Step two places the thresholds: a warning line near the band width and a blocking line clearly above it, say one and a half to two times the width. The point is the justification - the threshold must sit outside the noise, or you are blocking bad luck rather than regressions.
  4. Step three notes that a threshold is a local property of this task set and this trial count. Change the tasks, the k, or the model and you must measure again. Copying somebody else's threshold is the same as having none. A bonus point: trial count drives noise width directly, so a small smoke tier is noisier and needs a wider threshold.
  5. Step four adds the third state: rather than forcing a verdict in the gray band, rerun that one run with more trials and decide afterwards. Pass, warn and block fit the shape of noisy data better than a binary gate.
  6. Close on the cost of false alarms, which is the real discriminator: too tight is more dangerous than too loose. Daily false alarms train the team to ignore the gate, and the end state is identical to having no gate while everyone believes they are protected.

  分析过程 · 先想清楚再作答

  1. 这题考的是「知不知道阈值要量出来」。回答「下降超过 5 个百分点就拦」的，无论那个数字是多少都只能拿一半分——问题不在数值，在于它是拍脑袋来的。
  2. 第一步是**量噪声**：固定任务集、试次数、模型、提示词，只换随机种子，连跑二三十轮，记下通过率的最高值、最低值和均值。区间宽度就是「什么都不改、纯靠运气能掉多少」。
  3. 第二步才是定位置：警告线压在区间宽度附近，拦截线明显更高（比如区间宽度的一点五到两倍）。关键是要说出判据——阈值要在噪声之外，否则拦的是运气不是退化。
  4. 第三步要点出「阈值是局部属性」：它属于你这套任务集和这个试次数，换任务集、换 k、换模型都要重新量。**抄别人的阈值等于没定阈值。** 顺带能说出「试次数直接决定噪声宽度，任务少的冒烟档噪声更大、阈值必须更宽」，就是加分项。
  5. 第四步给出两态之外的做法：落在灰区的那一次不强行下结论，而是加大样本复跑一次再判，三态（通过/警告/拦截）比两态更贴合噪声的真实形态。
  6. 最后要说清楚误报的代价，这是这题真正的区分点：**太紧比太松更危险**。天天误报会把团队训练成无视门禁的人，最终结果和没有门禁一样，但过程中所有人都以为自己有防护。

  Key points

  • Measure noise first: vary only the seed across twenty or thirty runs to get the pass-rate band.
  • Put thresholds outside the band: warning near the band width, block clearly above it.
  • A threshold belongs to this task set and this trial count; re-measure after any change, never copy.
  • Use three states, resolving the gray band by rerunning with more trials instead of forcing a verdict.
  • A high false-alarm rate teaches the team to bypass the gate; too tight is worse than too loose.

  答题要点

  • 先量噪声：固定一切只换种子连跑二三十轮，得到通过率的波动区间。
  • 阈值放到区间之外：警告线贴近区间宽度，拦截线明显更高。
  • 阈值是这套任务集与这个试次数的属性，换任何一项都要重新量，不能照抄。
  • 做三态而不是两态，灰区靠加大样本复跑来判，而不是强行下结论。
  • 误报率过高会让团队学会跳过门禁，太紧比太松更危险。
• Beyond the pass rate, what else goes into a baseline snapshot? What breaks if you leave something out?基线快照里除了通过率，你还会记录什么？少记了会出什么问题？

  Common in ChinaCommon overseasIntermediate#ci#baseline#reproducibility

  How to reason about it · think before answering

  1. The answer is not a list of fields but what each field rules out. Reciting names collapses under the follow-up 'what happens if the prompt version is missing'.
  2. Start with the shape: a snapshot has two halves. The environment fingerprint - model, prompt version, task-set version, eval framework version, trial count, tier - and the scores, overall plus per task. The fingerprint decides whether two results are comparable at all; the scores decide by how much they differ.
  3. Go through the costs one by one. No model field and a model swap reads as a regression. No prompt version and a one-sentence edit produces a drop with no commit to blame. No task-set version and two newly added hard tasks look like a worse agent. No trial count and changing k from five to three silently changes the noise structure. Every case has the same shape: the number moved and nobody can say who moved it.
  4. Call out the prompt version specifically. It is a string, editing it carries none of the ceremony of code review, and many teams keep it in a config console. Versioning it costs almost nothing and converts a whole class of unexplainable drift into something traceable.
  5. The random seed is the interesting exception: record it, but keep it out of the comparability check. Its purpose is reproduction - when the gate fires you must be able to replay that exact run - while scores from different seeds are precisely the samples you use to estimate noise. Putting the seed in the fingerprint makes every seed change incomparable, which deletes non-determinism from the evaluation.
  6. Finish with the correct behavior on a fingerprint mismatch: refuse to compare and block, rather than computing a delta and annotating that the model changed. The latter produces a meaningless result that looks entirely normal. Requiring a human to re-record the baseline is the safe path.

  分析过程 · 先想清楚再作答

  1. 这题的答案不是罗列字段，而是「每一项各自排除了哪一种解释」。只背字段名的回答，在追问「那少记提示词版本会怎样」的时候会卡住。
  2. 先说骨架：快照分两半。一半是**环境指纹**，包括模型、提示词版本、任务集版本、评估框架版本、试次数与档位；另一半才是分数（总体通过率加逐任务通过率）。指纹决定两次结果**能不能比**，分数决定**比出来是多少**。
  3. 少记的代价可以逐项说：少了模型，换了模型的分数会被当成退化；少了提示词版本，改一句话导致的下降查不到任何对应的提交；少了任务集版本，加了两条难题会被当成 Agent 变差；少了试次数，k 从 5 改成 3 会让分数的噪声结构完全不同却看不出来。**四个问题的共同点是：分数变了，但没人能说出是谁动的。**
  4. 提示词版本要单独强调：它是一个字符串，改它没有代码评审的仪式感，很多团队还放在配置后台里随时可改。给它编个版本号成本几乎为零，收益是把一整类查不出原因的波动变成可查的。
  5. 随机种子是个有意思的例外：**要记，但不参与可比性判断**。它的用途是复现——门禁红了要能一字不差重跑那一次；但换种子跑出来的分数恰恰是估计噪声的样本，把它塞进指纹会让每次换种子都被判成不可比，等于把非确定性从评估里删掉了。
  6. 最后说指纹不一致时的正确行为：**拒绝比较，判拦截**，而不是「算个差值再标注一下模型变了」。后者会产出一个毫无意义却看起来很正常的结论，要求人明确表态重新记录基线，才是安全的。

  Key points

  • Two halves: environment fingerprint (model, prompt version, task-set version, framework version, trial count, tier) plus scores.
  • Each field rules out one explanation; omitting any leaves you with a moved number and no suspect.
  • Prompt version is the most commonly missed and the most damaging, because editing it bypasses code review.
  • Record the random seed but keep it out of the fingerprint: it exists for reproduction, and including it makes seed changes incomparable.
  • On a fingerprint mismatch, refuse to compare and block rather than computing a meaningless delta.

  答题要点

  • 快照分两半：环境指纹（模型、提示词版本、任务集版本、框架版本、试次数、档位）与分数。
  • 每一项各排除一种解释，少记任何一项都会变成「分数变了但说不清是谁动的」。
  • 提示词版本最容易漏也最容易出事，因为改它不需要代码评审。
  • 随机种子要记但不进指纹：它用于复现，进指纹会让换种子被判成不可比。
  • 指纹不一致时正确行为是拒绝比较并拦截，不是硬算一个差值。
• Running the full evaluation on every commit is too expensive. How would you tier it, and on what basis?每次提交都跑全量评估太贵，你会怎么分层？依据是什么？

  Common in ChinaCommon overseasIntermediate#ci#cost#strategy

  How to reason about it · think before answering

  1. This probes cost awareness and how you express trade-offs. Answering 'sample a random subset each time' misses: random subsets cover different failure classes each run, so the gate becomes intermittently blind.
  2. The basis for tiering is coverage divided by cost, not importance. The smoke tier should cover the most failure classes with the fewest tasks, and the full tier carries the long tail. So the smoke tier is curated, not sampled.
  3. A workable three-tier split: smoke on every commit to catch major breakage in minutes; full on tags, pre-release, or a nightly schedule to catch small regressions and the long tail; expensive human review quarterly on a sample, to calibrate the automated graders themselves.
  4. State the cost of tiering, which is the discriminator here: fewer tasks means fewer trials, and fewer trials means wider noise. For the same agent, going from eight tasks to three can widen the noise band from twelve points to twenty. The smoke tier can therefore only catch large failures; small regressions are invisible to it, and a green smoke run does not mean no regression.
  5. That implies a practice: measure thresholds separately per tier. Sharing one threshold either makes the smoke tier alarm constantly or makes the full tier far too insensitive.
  6. Expected follow-up: how do you pick the smoke tasks? By failure-class coverage - one representative per known class, plus the task corresponding to the most recent production incident - and revisit the list periodically, because failure classes shift as the product changes.

  分析过程 · 先想清楚再作答

  1. 这题考成本意识与取舍表达。回答「随机抽一部分任务跑」的方向就偏了——随机抽样每次覆盖的故障类别都不一样，门禁会变得时灵时不灵。
  2. 分层的依据不是「任务重不重要」，而是**覆盖面除以成本**：冒烟档要用最少的任务盖住最多的故障类别，全量档负责长尾。所以冒烟档是挑出来的，不是抽出来的。
  3. 一条可用的分层是三档：冒烟档每次提交跑，几分钟内拦住大事故；全量档在打 tag、发版前或每晚定时跑，抓小幅退化与长尾；昂贵的人工评审按季度抽样跑，用来校准自动评分器本身。
  4. 必须主动说出分层的**代价**，这是这题的区分点：任务变少意味着试次变少，试次变少意味着噪声变宽。同一个 Agent，任务从八条减到三条，噪声区间可能从十二个百分点涨到二十个。**所以冒烟档只能抓大事故，小幅退化它根本看不见**，冒烟档绿了不等于没有退化。
  5. 由此还能推出一条实践：两档的阈值要分别测，不能共用一套。共用一套的后果要么是冒烟档天天误报，要么是全量档过于迟钝。
  6. 可预期的追问是「怎么决定哪些任务进冒烟档」。答案是按故障类别覆盖去选：每一类已知的故障至少留一条代表，加上最近一次线上事故对应的那条；并且这份名单要定期复核，因为故障类别会随产品变化。

  Key points

  • Tier by coverage divided by cost; the smoke tier is curated, never randomly sampled.
  • Three tiers: smoke per commit, full on release or nightly, human review quarterly on a sample.
  • Fewer tasks means fewer trials and wider noise, so a green smoke run does not prove there is no regression.
  • Measure thresholds per tier; a shared threshold either alarms constantly or goes blind.
  • Select smoke tasks for failure-class coverage and include the task from the latest production incident.

  答题要点

  • 分层依据是覆盖面除以成本，冒烟档要挑选而不是随机抽样。
  • 三档：冒烟档每次提交、全量档发版或每晚、人工评审按季度抽样。
  • 任务少则试次少、噪声更宽，冒烟档只能抓大事故，绿了不等于没退化。
  • 两档的阈值必须分别测量，共用一套要么天天误报要么过于迟钝。
  • 冒烟档按故障类别覆盖来选，并把最近一次线上事故对应的任务放进去。
• Your evaluation gate keeps firing, and the team has learned to skip it. How would you change the process?评估门禁红了，团队却越来越习惯直接跳过。你会怎么改这条流程？

  Common in ChinaCommon overseasDeep dive#ci#process#culture

  How to reason about it · think before answering

  1. This is barely a technical question; it asks whether you have actually maintained such a gate. Answering 'communicate more' or 'mandate investigation before merge' earns nothing - that asks people to fight the incentives instead of changing them.
  2. First, admit that skipping is rational. If eight of ten red runs go green on a rerun, skipping is the probabilistically correct move. So measure the false-alarm rate; it is usually the root cause, and it is a technical problem, not an attitude problem.
  3. Second, lower the cost of fixing. A red gate must not emit only a number: give the seed that reproduces the run, which tasks dropped, and which failing transcripts to read first. Cutting investigation from thirty minutes to five removes most of the incentive to skip.
  4. Third, raise the cost of skipping while keeping a legitimate exit. Intentional behavior changes really do need the bar moved, and the correct expression is 'update the baseline and say why', not 'ignore this run'. Keep the baseline file in version control so every loosening shows up in a diff and gets reviewed. Never ship a one-click skip button - one that exists eventually becomes the default path.
  5. Fourth, fix the verdict logic itself: three states with a gray-band rerun stop the genuinely ambiguous run from being blocked outright. That removes most false alarms without giving up sensitivity to real regressions.
  6. Close with the counterintuitive one: if false alarms truly cannot be brought down, demote the block to a warning rather than keep a blocking gate everyone routes around. A bypassed gate supplies false confidence, which is worse than a gate that honestly says it only warns.

  分析过程 · 先想清楚再作答

  1. 这题几乎不考技术，考的是有没有真的维护过这类闸门。答「加强宣导」「规定必须查清楚才能合并」的拿不到分——那是在要求人对抗激励，而不是改激励。
  2. 第一步要承认一件事：**跳过是理性行为**。如果门禁十次红里有八次重跑就绿，那么跳过在概率上是对的。所以要先量一下误报率——它多半就是根因，而且它是一个技术问题，不是态度问题。
  3. 第二步是**降低修的成本**。门禁红的时候不能只丢一个数字，要直接给出：用哪个种子能复现、哪几条任务掉了、先读哪几条失败试次的轨迹。排查成本从半小时降到五分钟，跳过的诱因就少了一大半。
  4. 第三步是**提高跳过的成本，但保留合法出口**。有意的行为变更确实需要放宽标准，它的正确表达是「更新基线并说明原因」，而不是「本次忽略」。把基线文件放进版本库，放宽就会出现在 diff 里、需要有人评审。**绝不提供一键跳过按钮**——存在的跳过按钮最终一定会变成默认路径。
  5. 第四步是把判定本身修对：引入三态与灰区复跑，让真正模棱两可的那一次不再强行拦截。这直接砍掉大部分误报，而且不牺牲对真实退化的灵敏度。
  6. 最后要说一条反直觉的：如果误报确实压不下来，**宁可先把拦截线放宽成警告**，也不要留着一条大家都在绕过的拦截。一条被绕过的门禁提供的是虚假的安全感，比一条明确说「我只警告」的门禁更危险。

  Key points

  • Accept that skipping is rational and measure the false-alarm rate; it is usually the root cause and it is technical.
  • Lower the cost of fixing: emit the reproducing seed, the tasks that dropped, and which failing transcripts to read.
  • Raise the cost of skipping but keep a legitimate exit: baseline updates go through version control and review, never a one-click skip.
  • Adopt three states with a gray-band rerun to remove ambiguous false alarms without losing sensitivity.
  • If false alarms persist, demote blocking to warning rather than keep a gate everyone bypasses.

  答题要点

  • 先承认跳过是理性行为，去量误报率——它通常就是根因，且是技术问题。
  • 降低修的成本：红的时候给出复现种子、掉分的任务、该读哪几条失败轨迹。
  • 提高跳过的成本但保留合法出口：更新基线要进版本库、要被评审，绝不做一键跳过。
  • 引入三态与灰区复跑，砍掉模棱两可那部分误报而不牺牲灵敏度。
  • 压不下误报时宁可把拦截降级成警告，也不要留一条大家都在绕过的门禁。

Comments