Why You Cannot Ship an Agent on "I Tried It a Few Times"

Interview questions

• You run the same task five times and get three passes and two failures. How do you report this agent's quality to your manager?同一个任务跑五次，三次对两次错。你会怎么向老板汇报这个 Agent 的质量？

  Common in ChinaCommon overseasIntermediate#evaluation#metrics#non-determinism

  How to reason about it · think before answering

  1. This tests whether you can characterize a non-deterministic system, not whether you can divide. Answering 'a 60% success rate' earns half credit at best: the number is arithmetically right but carries no confidence interval and no context.
  2. Start with sample size. Three out of five is a point estimate with a very wide spread; five trials cannot distinguish a true rate of 40% from one of 80%. The honest report is 'we only have five trials, this number is not yet decision-grade'.
  3. Then separate two different questions: the probability of succeeding at least once, and the probability of succeeding every time. With a human reviewing the output, a 60% per-run rate means three attempts will almost certainly yield something usable. If it runs unattended, three consecutive successes happen with probability 0.6 cubed, about 21.6% - broken most of the time. The same 60% supports opposite conclusions in the two settings.
  4. Give a next action rather than stopping at the number: raise the trial count to something decision-grade, read the transcripts of both failures and attribute them, and freeze this task into the benchmark set so the next change has a baseline.
  5. Close on comparability: the number must carry the model version, prompt version, task-set version, and random seed. Without those, next week's number cannot be compared with this one.
  6. Expected follow-up: how many trials are enough? There is no fixed answer - it depends on the difference you need to detect. Separating 60% from 65% needs far more trials than separating 60% from 90%. State the question first, then size the sample.

  分析过程 · 先想清楚再作答

  1. 这题考的是「会不会把一个非确定性系统的表现讲清楚」，不是算术。张口就报「成功率 60%」的只能拿一半分——那个数字本身没错，但它既没有置信度，也没有说清适用场景。
  2. 第一步先把样本量的问题摆出来：五次里成三次，60% 这个点估计的波动范围很大。真实成功率是 40% 还是 80%，五次样本根本分不开。所以汇报时要说的是「目前只有五次样本，这个数字还不能用来做决策」，而不是直接把 60% 报上去。
  3. 第二步要区分两个问题：至少成一次的概率，和每次都成的概率。如果这个 Agent 后面有人 review，60% 的单次成功率意味着跑三次几乎一定能拿到一个可用结果；如果它是自动执行的，那么连成三次的概率只有 0.6 的三次方，约 21.6%，等于绝大多数时候是坏的。**同一个 60% 在两个场景下的结论完全相反。**
  4. 第三步给出下一步动作，而不是停在报数：把样本量加到足够判断的规模、把两次失败的轨迹读一遍做归因、并把这条任务固化进基准集，这样下次改动才有得比。
  5. 最后补一句可比较性：这个数字要带上模型版本、提示词版本、任务集版本和随机种子，否则下周再报一个数，没人知道是系统变了还是环境变了。
  6. 可预期的追问是「那你要跑多少次才够」。答案不是一个固定数字，而是取决于你要分辨多大的差异：想区分 60% 和 65% 需要的样本量，远大于区分 60% 和 90%。先说清楚要回答什么问题，再定样本量。

  Key points

  • Lead with sample size: five trials cannot pin the true rate to a useful interval.
  • Distinguish 'at least once' from 'every time' and map each to a deployment scenario.
  • For unattended execution compute the consecutive rate: 0.6 cubed is about 21.6%, a very different conclusion.
  • Propose next actions: more trials, read both failure transcripts, freeze the task into the benchmark set.
  • Always report model, prompt, task-set version and random seed, or the number is not comparable.

  答题要点

  • 先说样本量不足：五次样本无法把真实成功率定位到一个有用的区间。
  • 区分「至少成一次」与「每次都成」，并说明两者适用于不同场景。
  • 自动执行场景要算连续成功率：0.6 的三次方约 21.6%，结论与 60% 完全不同。
  • 给下一步动作：加样本、读失败轨迹做归因、把任务固化进基准集。
  • 报数必须带模型版本、提示词版本、任务集版本与随机种子，否则不可比。
• Why should you evaluate an agent against the final state of the environment rather than what its last message says?为什么评估 Agent 时要看环境的最终状态，而不是看它最后一条回复说了什么？

  Common in ChinaCommon overseasBasic#evaluation#graders#outcome

  How to reason about it · think before answering

  1. This looks easy; the discriminator is whether you can name the concrete mechanism by which a correct-sounding reply accompanies no action, rather than just saying 'models hallucinate'.
  2. The mechanism: the model has seen enormous amounts of customer-service dialogue and knows exactly what to say after a refund request. It can therefore produce 'your refund has been processed, expect it in three business days' without ever calling the refund tool. Textually that sentence is identical to the one it produces when the refund really happened.
  3. So a text-matching grader is blind here - it cannot separate the two cases. Checking the environment can: whether a row exists in the refunds table is a binary, settled fact.
  4. This is also the cleanest illustration of 'if code can judge it, do not ask a model'. Reading one database row is fast, cheap and objective; asking a second model to judge the reply's truthfulness is expensive and injects fresh uncertainty.
  5. Add the more serious consequence: a text grader is not merely wrong, it is systematically optimistic. An agent that learned to talk well without acting scores highly, and optimizing against that score trains it to talk even better.
  6. Expected follow-up: what about open-ended outputs such as a research report? Layer it - whatever can be reduced to state (do the cited links resolve, are the required points covered) stays with code, and only the genuinely subjective remainder goes to a model judge, which is Day 3.

  分析过程 · 先想清楚再作答

  1. 这题看着简单，区分度在于能不能举出「回复正确但事情没做」的具体机制，而不是只说一句「模型会幻觉」。
  2. 机制是这样的：模型在训练里见过大量客服对话，它非常清楚在「用户要求退款」之后应该说什么。于是它可以在**完全没有调用退款工具**的情况下，流畅地说出「已为您办理退款，预计三个工作日到账」。这句话与真正退了款时说的那句，在文本层面可以一模一样。
  3. 所以基于文本比对的评分器在这里是失效的：它判不出这两种情况的区别。而查环境可以——退款记录表里有没有这一行，是一个二值的、确定的事实。
  4. 反过来说，这也是「能用代码判就别用模型判」这条原则的最佳例证：查一行数据库记录既快又便宜又客观，而让另一个模型去读回复判断真假，既贵又会引入新的不确定性。
  5. 还要补一个更严重的后果：文本评分器不只是判错，它是**系统性地偏向乐观**。一个学会了说漂亮话但不干活的 Agent，在文本评分下会拿高分。如果你再拿这个分数去做优化，就是在训练它更会说漂亮话。
  6. 可预期的追问是「那开放式的回答怎么办，比如一份研究报告」。答案是分层：能落到结果态的部分（引用的链接是否真实存在、要覆盖的要点是否都在）仍然用代码判，剩下真正主观的部分才交给模型裁判，那是 D3 的内容。

  Key points

  • A model can produce a reply identical to the successful case without calling any tool.
  • Text comparison cannot separate the two; checking state can, because it is a binary fact.
  • Best illustration of 'prefer code graders': faster, cheaper, and objective.
  • Text graders are systematically optimistic; optimizing against them rewards better-sounding lies.
  • For open-ended output, layer it: state-checkable parts to code, the subjective remainder to a model judge.

  答题要点

  • 模型能在不调用工具的情况下说出与真正执行时一模一样的回复。
  • 文本比对判不出这两种情况，查环境可以——它是二值的确定事实。
  • 这是「能用代码判就别用模型判」的最佳例证：更快、更便宜、更客观。
  • 文本评分器会系统性偏向乐观，拿它做优化等于训练模型更会说漂亮话。
  • 开放式输出要分层：可落到状态的用代码判，剩余主观部分才交给模型裁判。
• When should you report pass@k, and when must you report pass^k? Give an example of each and state the cost.什么场景该用 pass 的 at k，什么场景必须用 pass 的 k 次方？各举一个例子并说明代价。

  Common in ChinaCommon overseasIntermediate#evaluation#metrics#pass-at-k

  How to reason about it · think before answering

  1. This tests whether you match metrics to product shape. Reciting definitions earns nothing; give the test: is there a human backstop?
  2. With a backstop, use pass@k. Code completion or generating image candidates: produce ten, a human picks one, and the interaction succeeds if any one of them works. Reporting pass@1 here badly understates the system's value.
  3. Without a backstop, pass^k is mandatory. Automated refunds or automated ops commands: nobody checks each one, and a single error is real money or a real incident. The question is not 'can it succeed' but 'will it ever fail'.
  4. State the costs. pass@k hides instability: a 30% agent scores 97% at pass@10, which looks great but means users retry three times on average. pass^k is harsh - it collapses as k grows, and teams may dismiss it as unachievable and stop tracking it.
  5. In practice report both, and always label k. A report with only one of them is incomplete, and pass^k without a stated k is not a number at all.
  6. Expected follow-up: how do you choose k? From real usage. Set k for pass@k to how many times users actually retry, and k for pass^k to how many consecutive runs the business requires to be clean. Do not default to 10.

  分析过程 · 先想清楚再作答

  1. 这题考的是「指标与产品形态的匹配」。只背定义拿不到分，要给出判据：**后面有没有人兜底。**
  2. 有人兜底的场景用 pass@k。典型例子是代码补全或生成图片候选：一次生成十个方案，人来挑一个，只要十个里有一个能用，这次交互就是成功的。这时候报 pass@1 会严重低估系统的实际价值。
  3. 没人兜底的场景必须用 pass 的 k 次方。典型例子是客服自动退款、自动执行运维命令：没有人逐条检查，错一次就是一笔真钱或一次事故。这时候你关心的不是「它能不能做对」，而是「它会不会有一次做错」。
  4. 代价要说清楚。pass@k 的代价是它会掩盖不稳定性：一个成功率 30% 的 Agent 在 pass@10 下能拿到 97%，看起来很好，但它意味着用户平均要试三次以上。pass 的 k 次方的代价是它非常严厉，k 稍微一大数字就塌下去，容易让团队觉得「怎么努力都没用」而放弃这个指标。
  5. 所以实践里两个一起报，并且明确标注 k 是多少。只报一个的评估报告都是不完整的，报的时候不写 k 更是没有意义——脱离 k 的 pass 的 k 次方不是一个数。
  6. 可预期的追问是「那 k 取多少」。答案是按真实使用形态取：用户平均会重试几次，就把 pass@k 的 k 设成几；业务要求连续多少次不出错，pass 的 k 次方就取几。不要随手取一个 10。

  Key points

  • The test is whether a human backstop exists: reviewed output takes pass@k, unattended execution requires pass^k.
  • Backstopped: code drafts, candidate generation. Unattended: automated refunds, automated ops.
  • pass@k hides instability - a 30% agent reports 97% at k equals 10.
  • pass^k is harsh and collapses as k grows, so teams tend to abandon it.
  • Report both with k labeled, and choose k from real retry behavior or the business continuity requirement.

  答题要点

  • 判据是后面有没有人兜底：有人 review 用 pass at k，无人值守用 pass 的 k 次方。
  • 有兜底例子：代码草稿、生成候选；无兜底例子：自动退款、自动运维。
  • pass at k 的代价是掩盖不稳定性：30% 的 Agent 在 k 等于 10 时能报到 97%。
  • pass 的 k 次方的代价是过于严厉，k 一大就塌，容易被团队放弃。
  • 两个一起报并标注 k；k 要按真实重试次数或业务连续性要求来取。
• A newly written evaluation task has a 0% pass rate over one hundred trials. What is your first reaction?一个新写的评估任务，跑一百次通过率是零。你的第一反应是什么？

  Common in ChinaCommon overseasIntermediate#evaluation#debugging#task-quality

  How to reason about it · think before answering

  1. This is a trap question about whether you will suspect your own evaluation. Answering 'the agent cannot do it, go optimize the model' starts in the wrong place.
  2. The correct first reaction is that the task itself is probably broken. A hundred straight zeros is an extreme signal - even a hard but solvable task usually gets lucky at least once in a hundred. Zero looks like a wall, not a slope.
  3. Debug from the evaluation side toward the model side. Start with grading: a mistyped expected value, a strict equality comparison on floats, a case or whitespace mismatch. These are extremely common - one brittle string comparison can fail a perfectly correct answer.
  4. Then the task description: is it ambiguous enough that the agent solved a different problem, or does it reference something absent from the environment, such as an order id never seeded in?
  5. Then reproducibility: does the task contain randomness that changes the correct answer each run while the grader compares against one fixed answer?
  6. Only last comes 'the agent genuinely cannot do it'. The standard way to establish that is a reference solution - do the task correctly by hand, feed it to the grader, and see whether it passes. If the reference solution fails, the problem is one hundred percent in the evaluation. That practice is the core of Day 2.
  7. Expected follow-up: what about a task that passes 100% immediately? Usually the task is too easy or the criterion too loose; it carries no information and needs fixing too.

  分析过程 · 先想清楚再作答

  1. 这题是个陷阱题，考的是「会不会怀疑自己的评估」。回答「说明 Agent 做不到这个任务，要去优化模型」的，方向就错了。
  2. 正确的第一反应是：**这条任务本身多半是坏的。** 一百次全零是个极端信号——即便是很难的任务，如果它确实可解，一百次里通常会蒙对至少一次。全零更像是一堵墙，而不是一个斜坡。
  3. 排查顺序应该是从评估侧往模型侧走。先看判分：是不是判据写错了，比如期望值拼错、浮点数做了严格相等比较、大小写或空格不一致。这类问题非常常见，一个字符串比对写死就能让一个完全正确的答案判失败。
  4. 再看任务描述：是不是有歧义，导致 Agent 理解成了另一件事；是不是依赖了环境里不存在的东西，比如引用了一个没有被 seed 进去的订单号。
  5. 然后看可复现性：任务里有没有随机成分，导致每次的正确答案都不一样，评分器却拿着一个固定答案在比。
  6. 最后才是「确实是 Agent 做不到」。而验证这一点的标准做法是写一个参考解——人工把这条任务正确地做一遍，喂给评分器，看它判不判通过。参考解都过不了，那 100% 是评估的问题。这个动作是 D2 的核心内容。
  7. 可预期的追问是「反过来呢，通过率一上来就是 100%」。那通常说明题目太简单或者判据太松，这条任务提供不了任何信息，同样需要修。

  Key points

  • Suspect the task first, not the model.
  • A hundred straight zeros is extreme: a solvable hard task usually succeeds at least once.
  • Debug evaluation-side first: broken grading, ambiguous description, missing environment fixtures, irreproducible randomness.
  • Strict equality comparison on floats or strings is the most common grading defect.
  • Validate with a reference solution: if a hand-crafted correct answer fails the grader, the fault is in the evaluation.

  答题要点

  • 第一反应应该是怀疑任务本身，而不是去优化模型。
  • 一百次全零是极端信号：真正可解的难任务通常会蒙对至少一次。
  • 排查顺序从评估侧到模型侧：判分写错、任务描述有歧义、依赖了环境里没有的东西、随机性不可复现。
  • 严格相等比较（尤其是浮点与字符串）是最常见的判分缺陷。
  • 用参考解验证：人工做对一遍喂给评分器，过不了就一定是评估的问题。

Comments