System Design Deep Dive: Agent Platforms / Customer-Support Agents / Multi-Tenancy / Cost Control

Interview questions

• You get 35 to 40 minutes for a system design round. How do you budget that time, and why is drawing the architecture not step one?系统设计环节只有 35 到 40 分钟，你会怎么分配时间？为什么第一步不是画架构图？

  Common in ChinaCommon overseasBasic#system-design#interview-process

  How to reason about it · think before answering

  1. This question tests pacing, not knowledge. Interviewers ask it because the previous candidate spent 25 minutes on the architecture diagram and left five each for deep dives and trade-offs — which is exactly where the rubric puts most of the weight.
  2. Give the structure with explicit time boxes: 5 minutes clarifying requirements, 3 minutes on capacity and cost estimation, 8 minutes sketching the architecture, 15 minutes going deep on two or three areas, 5 minutes on trade-offs. Naming actual minute counts is itself worth points, because it shows you have rehearsed against a clock.
  3. Then answer the 'why not draw first' half head on: a one-line prompt leaves five things unknown — daily actives, latency budget, cost budget, multi-tenancy, and failure tolerance — and every one of them changes the architecture materially. Drawing first means at best you guessed right, at worst the interviewer realizes twenty minutes in that you solved a different problem. An analogy lands it: the client said 'we need an office building' and you unrolled construction drawings before hearing whether the budget is twenty million or two hundred million.
  4. Add the situation that comes up almost every time: you start asking and the interviewer says 'just assume something'. That is not permission to skip clarification, it is an invitation to state a number and its justification. The right reply is 'then I will assume 10k daily actives at five turns each, and I will flag in the final step what changes at 100k'. You keep the pacing and turn the assumption into a traceable premise.
  5. Close by explaining how step four is prepared: those 15 minutes cannot be improvised. Have three deep-dive packages ready — state and ordering, cost and rate limiting, failure and retry — so any pick is covered. Saying you prepared three directions signals rehearsal better than winging one.
  6. Expect the follow-up: what if you run out of time? Cut step three, never step five. An unfinished sketch can be closed with 'the rest follows the standard pattern, happy to come back to it', but dropping the trade-off section makes you indistinguishable from someone who memorized an architecture.

  分析过程 · 先想清楚再作答

  1. 这题考的不是知识，是节奏感。面试官问它，通常是因为上一位候选人在架构图上讲了 25 分钟，深入和权衡各剩五分钟——而评分表上分数最重的恰恰是后两步。
  2. 先给结构，五步加时间盒：需求澄清 5 分钟、容量与成本估算 3 分钟、架构草图 8 分钟、深入 2 到 3 个点 15 分钟、权衡与取舍 5 分钟。给得出具体分钟数本身就是分数，因为它说明你掐过表。
  3. 然后正面回答「为什么不先画图」：一句话的题干里，日活、延迟预算、成本预算、是否多租户、失败可容忍度这五件事全是未知的，而它们每一个都会实质改变架构。不问就画，最好的结果是运气好蒙对，最坏的结果是二十分钟后面试官发现你解的是另一道题。用一个类比说清：甲方只说「我要一栋办公楼」，你就展开施工图，而他连预算是两千万还是两个亿都没讲。
  4. 补一条几乎每次都会遇到的现场情况：你开始问，面试官说「你先自己假设一个」。这不是让你别问了，是让你自己给一个数并说出依据。正确接法是「那我按日活 1 万、人均 5 轮算，如果实际是十万级我会在最后一步说明哪里要改」——既守住了节奏，又把假设变成了可追溯的前提。
  5. 最后主动交代第四步的准备方式：深入的 15 分钟不能临场想，要提前备好三个「深入包」（状态与保序、成本与限流、失败与重试），面试官挑哪个都有货。说得出「我提前准备了三个方向」，比现场硬讲一个更能体现你练过。
  6. 可以预期的追问：如果时间不够怎么办？答案是砍第三步而不是砍第五步——草图讲不完可以说「其余按常规做，需要的话我们回头补」，但权衡那 5 分钟一旦砍掉，你就和一个只会背架构的人没有区别。

  Key points

  • Five steps with time boxes: clarify 5, estimate 3, sketch 8, deep dive 15, trade-offs 5
  • Do not sketch first because DAU, latency budget, cost budget, multi-tenancy and failure tolerance all change the architecture
  • When told to 'just assume something', state a number with its justification instead of skipping clarification
  • Fill the 15-minute deep dive from three pre-prepared packages: state and ordering, cost and rate limiting, failure and retry
  • If time runs short, cut the sketch, never the trade-offs — almost nobody does that section, so doing it stands out

  答题要点

  • 五步加时间盒：澄清 5 分钟、估算 3 分钟、草图 8 分钟、深入 15 分钟、权衡 5 分钟
  • 不先画图，是因为日活、延迟预算、成本预算、是否多租户、失败可容忍度这五件事都会实质改变架构
  • 面试官说「你先假设一个」时，要自己给数并说出依据，而不是跳过澄清
  • 深入的 15 分钟要靠提前备好的三个「深入包」：状态与保序、成本与限流、失败与重试
  • 时间不够时砍草图不砍权衡——权衡那 5 分钟几乎没人做，做了就是加分
• System design: design an e-commerce customer support agent. It looks up orders and shipments, drafts refunds by policy, answers product and policy questions, and escalates to a human when it cannot resolve the issue.系统设计：请设计一个电商客服 Agent。它要能查订单和物流、按规则拟退款方案、回答商品与政策问题，并在搞不定时转人工。

  Common in ChinaCommon overseasDeep dive#system-design#customer-support#escalation

  How to reason about it · think before answering

  1. Start by separating this from 'design an agent platform', or you will answer an infrastructure question. The platform question is about running execution reliably; this one is about not trapping users inside a bot. The rubric lives in the business exits, not the message bus. So pin the thesis in your first sentence: every conversation must end in exactly one of three exits — self-served, handed to a human, or filed as a ticket.
  2. Clarify for 5 minutes, asking four things: daily actives and concurrent sessions (does execution need to be split out), whether human agents work nights (does exit three exist), whether the agent executes refunds or only drafts them (do you need an approval tier), and how large the knowledge base is and how often it changes (is retrieval the center of this problem). The third question matters most: it decides whether this system has irreversible side effects.
  3. Estimate for 3 minutes, out loud: 2000 input plus 500 output per turn, input is 2000 over a million times $0.15 which is $0.0003, output is 500 over a million times $0.60 which is also $0.0003, so about $0.0006 per turn. 10k daily actives at five turns is 50k turns, roughly $30 a day and $900 a month in model spend, machines excluded. For concurrency, a peak factor of 3 and 6 seconds per turn gives about 11 in-flight executions at peak, which is 3 worker replicas at 4 concurrent each. State the arithmetic before the result — the interviewer's next line is always 'where did that number come from'.
  4. Sketch for 8 minutes, four blocks: ingress does auth, rate limiting, persistence and publish, then returns immediately; execution pulls from the bus, runs the agent loop, and streams sequenced fragments back; storage is sessions, runs and messages plus a chunk table for the knowledge base; observability is tracing plus a cost ledger. Then mark on the diagram which node decides between the three exits — that single annotation tells the interviewer you are answering the support question rather than the generic platform one.
  5. Go deep for 15 minutes, starting with escalation because that is the crux. The criteria must be quantified, any one of four triggering a handoff: two consecutive unresolved turns, an explicit user request, an amount above the auto-execution ceiling (50 CNY in our setup), or a sentiment keyword hit. Then describe the handoff payload: not forty turns of raw transcript, but a structured summary — the user's ask in one line, verified facts, actions already taken, and the failure reason, with a link to the full transcript. Cover the knowledge base in one line (hybrid search, rerank, inline citations) and spend the weight on 'when the citation set comes back empty, take exit two or three rather than letting the model invent an answer' — that is the sentence they will push on. Cover multi-turn in one line too: compress once history passes 70% of budget, cut on a turn boundary, and merge a change of mind within 30 seconds into the same execution.
  6. Trade-offs for 5 minutes, three points: bias the escalation threshold toward escalating, because a false handoff costs one human conversation while trapping a user costs a churned customer and a bad review — different orders of magnitude. Drafting refunds instead of executing them trades one human approval for an entire class of irreversible incidents. And name what breaks the design: once the agent team is large enough to need skill-based routing and queueing, escalation stops being a boolean and becomes its own scheduling system.
  7. Expect, in rough order: does 'I want to file a complaint' count as a sentiment hit (yes, and track that class separately — it is a product signal); should the agent keep listening after handoff (yes, to summarize and prompt the human, but not to speak); how do you stop users being bounced repeatedly (allow one handoff per conversation, then file a ticket); and what happens to old answers when the knowledge base changes (cite chunk ids and versions so you can trace which revision was wrong).

  分析过程 · 先想清楚再作答

  1. 先说这题和「设计一个 Agent 平台」的区别，否则你会把它答成一道基础设施题。平台题考的是怎么把执行跑稳，这题考的是**怎么保证不把用户困在机器人里**——面试官心里的评分点在业务出口上，不在消息总线上。所以主线要一开口就钉死：任何一通会话最后只能落到三条出口之一，自助解决、转人工、留工单。
  2. 第一步澄清 5 分钟，问四件事：日活与并发会话数（决定要不要拆执行层）、人工坐席有没有夜班（决定出口三存不存在）、退款是 Agent 直接执行还是只拟方案（决定要不要人工确认档）、知识库有多大且多久更新一次（决定检索是不是本题的重点）。第三个问题尤其关键，它直接决定这道题是不是带副作用。
  3. 第二步估算 3 分钟，现场算：单轮 2000 输入加 500 输出，输入 2000 除以一百万乘 0.15 等于 0.0003 美元，输出 500 除以一百万乘 0.60 也等于 0.0003 美元，一轮约 0.0006 美元；日活 1 万、人均 5 轮就是 5 万轮，一天约 30 美元、一个月约 900 美元，模型费不含机器。并发按峰谷比 3、单轮 6 秒算，峰值在途约 11 次执行，每个 worker 并发 4 就是 3 个副本。报数字之前先报算式，面试官插的那句一定是「这个数怎么来的」。
  4. 第三步草图 8 分钟，四块：接入层只做鉴权、限流、落库、投递并立刻返回；执行层从消息总线取活跑 Agent 循环、片段带序号回传；存储是会话、执行、消息三张表加一张知识库切块表；可观测是 tracing 加成本台账。在这张图上额外标出三条出口的分叉点在哪一个节点上——这是本题独有的一笔，画上去面试官立刻知道你答的是客服而不是通用平台。
  5. 第四步深入 15 分钟，优先讲转人工这一支，因为它是本题的题眼。判据必须量化，四条任一命中就转：连续 2 轮未解决、用户明确要求、涉及金额超过自动执行上限（本课口径 50 元）、情绪词命中。接着讲交接形状——不是把 40 轮原文丢给客服，而是一段结构化摘要：用户诉求一句、已核实事实几条、Agent 已做过的动作、失败原因，附原始对话链接。知识库那一支一句话带过混合检索加重排加引用，重点落在「引用为空时走出口二或三，而不是让模型编一个答案」，这是最容易被追的一句。多轮那一支同样一句话：历史超七成预算触发压缩且切口对齐到一轮开头，用户中途改口则 30 秒内合并进同一次执行。
  6. 第五步权衡 5 分钟，说三件事：转人工的判据宁可偏松，因为误转的代价是一次人工会话，把用户困住的代价是一个流失客户加一条差评，两者不在一个量级；退款只拟方案不直接执行，是拿一次人工点头换掉一整类不可逆事故；以及什么规模会推翻这个设计——坐席团队大到需要技能路由和排队策略时，转人工就不再是一个布尔判断，而是另一套调度系统。
  7. 可以预期的追问，按频率排：用户说「我要投诉」算不算情绪词命中（算，且这一类要单独统计，它是产品问题的信号）；转人工之后 Agent 还要不要继续在旁边听（要，用来生成小结和给坐席提示，但不允许再发言）；怎么防止用户被反复转来转去（同一通会话只允许转一次，第二次直接留工单）；以及知识库更新后旧答案怎么办（回答里带引用编号和版本，出问题能倒查是哪一版说错的）。

  Key points

  • Thesis: every conversation ends in exactly one of three exits — self-served, escalated to a human, or filed as a ticket
  • Clarify four things: concurrent sessions, whether humans cover nights, whether refunds are executed or only drafted, and knowledge base size and churn
  • Estimate with arithmetic: about $0.0006 per turn, so 10k DAU at five turns is roughly $30/day and $900/month; peak concurrency about 11, meaning 3 worker replicas
  • Quantify escalation: two consecutive unresolved turns, an explicit request, an amount over the auto-execution ceiling, or a sentiment keyword
  • Hand over a structured summary — ask, verified facts, actions taken, failure reason — plus a transcript link, not forty raw turns
  • When retrieval returns no citations, take exit two or three instead of letting the model improvise; answers carry citation ids
  • Reuse compression and 30-second merge for multi-turn; draft refunds rather than executing them, trading one approval for a class of irreversible incidents
  • Trade-off: bias toward escalating, because a false handoff and a trapped user cost different orders of magnitude

  答题要点

  • 主线一句话：任何一通会话只能落到三条出口之一——自助解决、转人工、留工单
  • 澄清必问四件事：并发会话数、人工有没有夜班、退款是执行还是只拟方案、知识库规模与更新频率
  • 估算带算式：单轮约 0.0006 美元，日活 1 万人均 5 轮约 30 美元一天、900 美元一月；峰值并发约 11、3 个 worker 副本
  • 转人工判据必须量化，四条任一命中：连续 2 轮未解决、用户明确要求、金额超自动执行上限、情绪词命中
  • 交接给人工的是结构化摘要（诉求、已核实事实、已做动作、失败原因）加原始对话链接，不是 40 轮原文
  • 知识库检索不到时走出口二或三，绝不让模型自由发挥编答案；回答带引用编号
  • 多轮沿用压缩与 30 秒打断合并，退款只拟方案不直接执行，用一次人工点头换掉一类不可逆事故
  • 权衡：判据宁可偏松，因为误转和困住用户的代价不在一个量级
• For a multi-tenant agent service, how do you design data isolation and billing isolation, and when do you move from a shared table to a dedicated database per tenant?一个多租户的 Agent 服务，数据隔离和计费隔离要怎么设计？什么时候该从共享表升级到独立库？

  Common in ChinaCommon overseasIntermediate#system-design#multi-tenancy#isolation

  How to reason about it · think before answering

  1. The hinge is that 'isolation' is plural. Plenty of candidates answer only data isolation, but the layer that actually breaks in production is resources: one tenant's spike starves everyone else, no rows leak, and users still complain. Open with all three — data, resources, billing — and note that each missing layer maps to its own class of incident.
  2. On data, one sentence separates people who shipped this from people who read about it: 'every query carries tenant_id' versus 'row-level security is the backstop'. The first eventually misses a query, and the one it misses is always the newest, least-tested feature. The correct framing is that RLS is the gate and the application-level where clause is just an optimization — the same reasoning as idempotency being adjudicated by a database unique constraint. Whatever the data layer can enforce should not depend on everyone remembering.
  3. On resources, give two concrete things: a rate-limit bucket per tenant, and workers sharded by a hash of the tenant id. That is the same sharding mechanism used to preserve per-user ordering, with a different hash input and a different purpose — containing spikes rather than serializing. Noisy neighbors hurt more in agent workloads because a single execution can run thirty seconds, so a thousand queued items from one tenant leaves everyone else waiting.
  4. Billing is the simplest and the most often forgotten: add a tenant column to the token usage ledger and tag every write. Invoicing, quotas and over-budget degradation all hang off it. Bring the cost figures too — roughly $0.0006 per turn at 2000 in and 500 out, about $900 a month at 10k daily actives and five turns each — because quoting per-tenant economics shows you actually ran the numbers.
  5. Then the escalation criteria, the second discriminator. Three tiers: shared table with a tenant column, schema per tenant, database per tenant. The trigger is not tenant count, it is whether a single tenant can starve the rest and whether there is a hard compliance requirement. 'Split the database past a hundred tenants' is guesswork: a hundred small tenants share a table happily, while one regulated enterprise customer may require physical separation on its own. State the cost too — a database per tenant looks clean, but migrations, backups, monitoring and connection pools all multiply by tenant count, so operational cost jumps rather than scaling linearly.
  6. Expect the sharpest follow-up: does the idempotency key change under multi-tenancy? The algorithm does not, but its scope must include the tenant id. Without it, two tenants whose clients independently produce the same string — both using order id order-1024 — collide, and the later request is rejected by the unique constraint as a duplicate. One tenant's write is swallowed by another tenant's history, both logs look perfectly normal, and it is the hardest class of multi-tenant bug to find.

  分析过程 · 先想清楚再作答

  1. 这题的题眼在「隔离」是复数。只答数据隔离的候选人非常多，而多租户翻车最多的其实是资源那一层——一个租户的洪峰打穿别人的处理能力，数据一条都没串，用户照样投诉。所以第一句先把三层摆出来：数据、资源、计费，缺哪一层对应一类事故。
  2. 数据这一层，判断一个人有没有真做过就看一句话：他说「每条查询都带 tenant_id」还是「靠数据库的行级安全兜底」。前者迟早会漏一处，而漏掉的那处通常是最新加、最没被测过的功能。正确说法是行级安全是闸门，应用层那句 where 只是优化——这和幂等的最终裁判必须是数据库唯一约束，是同一种思路：能在数据层强制的，不要指望每个人写代码时都记得。
  3. 资源这一层给两件具体的东西：每个租户一个独立限流桶，以及 worker 按租户标识哈希分片。分片这一招和「按用户哈希保住同一用户顺序」是同一套机制，只是哈希的输入换了，目的从保序变成隔离洪峰。Agent 场景里噪声邻居格外突出，因为单次执行可能跑三十秒，一个租户灌一千条进来，别人就得排队。
  4. 计费这一层最简单也最容易漏：token 用量台账加一列租户标识，写入时打标。账单、配额、超支降级三件事全靠它。顺带说一句成本口径——单轮 2000 输入加 500 输出约 0.0006 美元，日活 1 万人均 5 轮约每月 900 美元，能报出这个量级说明你真的算过每租户成本。
  5. 然后回答升级判据，这是本题的第二个区分点。三档是共享表加租户列、schema 级、库级；**判据不是租户数量，是「有没有单个租户能把别人拖垮」和「有没有合规硬要求」**。答「超过一百个租户就该分库」是典型的凭感觉，因为一百个小租户共享一张表毫无问题，而一个受监管的大客户哪怕只有一个也可能必须物理隔离。代价要一起说：库级隔离看着干净，但迁移脚本、备份、监控、连接池全部乘以租户数，运维成本是陡增不是线性。
  6. 可以预期的追问，也是最见功力的一问：幂等键在多租户下要不要变？答案是键的算法不用变，但**作用域必须带上租户标识**。不带的话两个租户的客户端各自生成了同一个字符串（都用订单号 order-1024），后来那个会被唯一约束当成重复请求挡掉——一个租户的写入被另一个租户的历史请求吞掉，两边日志都完全正常，是多租户里最难查的一类 bug。

  Key points

  • Three parallel layers, each missing one causing its own class of incident: data, resources, billing
  • Data isolation is backstopped by row-level security; the application where clause is only an optimization
  • Resource isolation is a per-tenant rate-limit bucket plus sharding workers by tenant hash, aimed at noisy neighbors
  • Billing isolation is a tenant column on the usage ledger, powering invoices, quotas and degradation
  • Escalate to schema or database isolation based on starvation risk and compliance mandates, not tenant count
  • Per-tenant databases multiply migrations, backups, monitoring and connection pools — operational cost jumps
  • The idempotency key algorithm stays, but its scope must include the tenant id or identical keys across tenants collide

  答题要点

  • 三层隔离并列，缺一层对应一类事故：数据、资源、计费
  • 数据靠行级安全兜底，应用层的 where 只是优化——能在数据层强制的不要靠人记得
  • 资源是每租户独立限流桶加按租户标识哈希分片，防的是噪声邻居而不是数据串
  • 计费是台账加一列租户标识，账单、配额、超支降级全靠它
  • 升级到 schema 级或库级的判据是「单租户能否拖垮别人」与「有没有合规硬要求」，不是租户数量
  • 库级隔离的代价是迁移、备份、监控、连接池全部乘以租户数，运维成本陡增
  • 幂等键算法不变，但作用域必须带租户标识，否则两个租户的同名键会互相挡掉请求
• An agent system's model spend is out of control. Which levers do you pull, in what order, and roughly how much does each save?一个 Agent 系统的模型成本失控了，你会从哪几个层面着手控制？每一层大概能省多少？

  Common in ChinaCommon overseasIntermediate#system-design#cost#capacity-planning

  How to reason about it · think before answering

  1. The reflex answer is 'switch to a cheaper model', and it is also the easiest one to get killed on: the follow-up is 'how do you know quality did not drop', and without an offline eval set and a comparison run you are exposed. The right opening is 'look at the ledger first' — slice by user, by day and by model to find which dimension is growing. Locate before you act.
  2. Second, put the baseline on the table, because cost talk without a baseline is noise. At 2000 input and 500 output tokens per turn, input is 2000 over a million times $0.15 which is $0.0003, output is 500 over a million times $0.60 which is also $0.0003, so about $0.0006 per turn. 10k daily actives at five turns is 50k turns, roughly $30 a day and $900 a month.
  3. Then give five layers ordered by the cost you pay, not by the savings: caching and prompt caching, tiered model routing, context compression, step and tool budget caps, rate limiting and degradation. The ordering is part of the answer, because it also communicates your rollout sequence.
  4. Attach a number derived from the baseline to each layer. Caching at a 15% hit rate takes $900 to roughly $765. For tiered routing, state the precondition honestly: it is the only layer that can change the order of magnitude, but only if your baseline runs a flagship model — if you already run the cheapest tier there is nothing left to squeeze. Saying that out loud is far more credible than inventing a savings percentage. Compression takes input from 2000 to 1200 tokens, so $0.00048 per turn, about $720 a month, a 20% cut.
  5. Layer four is usually mis-sold as savings; what it actually buys is predictability. With a cap of five tool calls per subtask, per-turn cost finally has a ceiling: five calls each feeding back 800 tokens pushes input to 6000, so $0.0012 per turn, exactly double the baseline — and with no cap there is no ceiling at all. The right phrasing is 'this does not save money, it makes the bill predictable'.
  6. Layer five is rate limiting and degradation, last because it costs the most: $900 across 10k daily actives is about $0.09 per user per month, so a $1 monthly hard cap is invisible to real users and only stops scripted abuse. The nuance is degrade before refusing — this is the only layer users can feel.
  7. Expect the follow-up: which layer first? Say layers one and three, because they only touch your own code, change no product promise, and need no quality re-validation, whereas tiered routing needs an eval set and rate limiting needs product buy-in.

  分析过程 · 先想清楚再作答

  1. 这题最容易脱口而出的答案是「换个便宜模型」，也是最容易被追死的答案——面试官紧跟着就问「你怎么知道换了质量不掉」，答不出离线评估集和对比实验就露馅了。正确的第一句是「先看台账」：按用户、按天、按模型各切一刀，找出是哪一维在涨。先定位再动手，这是工程习惯。
  2. 第二步是把基准摆到桌上，没有基准的成本讨论全是废话。单轮 2000 输入加 500 输出，输入 2000 除以一百万乘 0.15 等于 0.0003 美元，输出 500 除以一百万乘 0.60 也等于 0.0003 美元，一轮约 0.0006 美元；日活 1 万、人均 5 轮就是 5 万轮，一天约 30 美元、一个月约 900 美元。
  3. 然后给五层，排序的依据是**你要付出的代价从小到大**，不是省钱多少：缓存与 prompt cache、模型分级路由、上下文压缩、步数与工具预算上限、限流与降级。这个顺序本身就是答案的一部分，因为它同时说明了你的落地顺序。
  4. 每层配一个从基准推出来的数字。缓存按一成半命中估，900 降到 765 左右。分级路由要诚实说清前提：它是唯一能改数量级的一层，但前提是你的基准用的是旗舰模型；基准已经是最便宜那档时这一层榨不出东西——主动说破这一条，比硬编一个省钱比例可信得多。上下文压缩把输入从 2000 压到 1200，单轮变成 0.00048 美元，一个月 720 美元，降两成。
  5. 第四层最容易被讲成「省钱」，其实它买的是**可预测**：给每个子任务设 5 次工具调用上限之后，单轮成本才有上界——调满 5 次、每次结果回灌 800 token，输入涨到 6000，单轮 0.0012 美元，正好是基准的两倍；没有上限时这个数字没有上界。这一层的正确说法是「我不是靠它省钱，我是靠它让账单可以被预测」。
  6. 第五层是限流与降级，代价最大所以放最后：900 美元摊到 1 万日活是每人每月 0.09 美元，给单用户设 1 美元硬顶，正常用户碰不到，挡的是脚本刷接口那种极端户。要点是超预算先降档再拒绝，而不是直接拒绝——它是五层里唯一用户能感觉到的一层。
  7. 可以预期的追问：这五层里哪一层最先做？答「第一层和第三层」，因为它们只改自己的代码、不动产品承诺、也不需要重新验证质量；而分级路由要配离线评估集，限流要配产品沟通，都不是当天能上的。

  Key points

  • Open with 'look at the ledger', not 'use a cheaper model': slice by user, by day and by model to locate the growth
  • Set a baseline: about $0.0006 per turn, roughly $30/day and $900/month at 10k DAU and five turns
  • Five layers ordered by cost to you: caching and prompt cache, tiered routing, context compression, step and tool budget caps, rate limiting and degradation
  • Tiered routing is the only order-of-magnitude lever, but only if the baseline is a flagship model — say so when it is not
  • Compression from 2000 to 1200 input tokens gives $0.00048 per turn, about $720/month, a 20% cut
  • Tool budget caps buy predictability: with a cap the per-turn ceiling is $0.0012, without one there is no ceiling
  • Rate limiting comes last because users feel it; degrade before refusing

  答题要点

  • 第一句不是「换便宜模型」，是「先看台账」：按用户、按天、按模型各切一刀定位是哪一维在涨
  • 先立基准：单轮约 0.0006 美元，日活 1 万人均 5 轮约每天 30 美元、每月 900 美元
  • 五层按代价从小到大：缓存与 prompt cache、模型分级路由、上下文压缩、步数与工具预算上限、限流与降级
  • 分级路由是唯一能改数量级的一层，但前提是基准用的是旗舰模型；基准已经最便宜时要诚实说没得省
  • 上下文压缩把输入从 2000 压到 1200，单轮 0.00048 美元、每月 720 美元，降两成
  • 工具预算上限买的是可预测：有上限时单轮上界是 0.0012 美元，没上限时没有上界
  • 限流降级放最后，因为它是唯一用户能感觉到的一层；超预算先降档再拒绝

Comments