Model Integration and System Prompts: a Multi-Provider Abstraction With Fallback, Overriding the Default Persona (dg P03/P04/M04)

The unified calling layer: gathering three vendors' quirks behind one interface

The answer is a layer, and its job is one sentence: expose one function upward, and absorb every difference downward.

The everyday analogy is a travel power adapter. Your laptop accepts one kind of plug, the world has more than a dozen socket standards, and the adapter's value is not cleverness — it is that it holds the world's mess outside your device. A unified calling layer is identical: whether OpenAI's format, Anthropic's format, or some open-weight model's compatibility endpoint sits underneath, the layer above always receives the same shape.

Start by defining what that same shape is. In: a messages array, an optional temperature, an optional timeout. Out: the reply text, which provider actually served it, and how many tokens it consumed. Note that "which provider actually served it" in the output is not debug information, it is this layer's core product — fallback and cost accounting both depend on it later.

const PROVIDERS = [
  { name: 'fast', model: 'openai/gpt-4o-mini', timeoutMs: 8000 },
  { name: 'strong', model: 'anthropic/claude-3.5-sonnet', timeoutMs: 15000 },
  { name: 'backup', model: 'google/gemini-2.0-flash-001', timeoutMs: 8000 },
]
 
// An error type with a structured status field: let callers read a field instead of
// parsing the error message string
class ProviderError extends Error {
  constructor(message, status) {
    super(message)
    this.name = 'ProviderError'
    this.status = status
  }
}
 
async function callOne(provider, messages) {
  const res = await fetch('https://openrouter.ai/api/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ model: provider.model, messages }),
    signal: AbortSignal.timeout(provider.timeoutMs),
  })
  if (!res.ok) throw new ProviderError(`${provider.name} returned HTTP ${res.status}`, res.status)
  const json = await res.json()
  return {
    text: json.choices[0].message.content,
    provider: provider.name,
    tokens: json.usage.total_tokens,
  }
}

import os
from dataclasses import dataclass
 
import httpx
 
 
@dataclass
class Provider:
    name: str
    model: str
    timeout: float
 
 
PROVIDERS = [
    Provider("fast", "openai/gpt-4o-mini", 8.0),
    Provider("strong", "anthropic/claude-3.5-sonnet", 15.0),
    Provider("backup", "google/gemini-2.0-flash-001", 8.0),
]
 
 
def call_one(provider: Provider, messages: list[dict]) -> dict:
    res = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={"model": provider.model, "messages": messages},
        timeout=provider.timeout,
    )
    # The HTTPStatusError httpx raises carries response.status_code, so it is
    # structured by construction
    res.raise_for_status()
    body = res.json()
    return {
        "text": body["choices"][0]["message"]["content"],
        "provider": provider.name,
        "tokens": body["usage"]["total_tokens"],
    }

// A record is a natural fit for immutable configuration like this and is cleaner
// than a pile of getters
record Provider(String name, String model, Duration timeout) {}
record Reply(String text, String provider, int tokens) {}
 
// ObjectMapper is thread-safe and not cheap to construct, so share one per process
static final ObjectMapper MAPPER = new ObjectMapper();
 
// An exception with a structured status field: let callers read a field instead of
// parsing the getMessage() string
static class ProviderException extends IOException {
    final int status;
 
    ProviderException(String message, int status) {
        super(message);
        this.status = status;
    }
}
 
static final List<Provider> PROVIDERS = List.of(
    new Provider("fast", "openai/gpt-4o-mini", Duration.ofSeconds(8)),
    new Provider("strong", "anthropic/claude-3.5-sonnet", Duration.ofSeconds(15)),
    new Provider("backup", "google/gemini-2.0-flash-001", Duration.ofSeconds(8)));
 
static Reply callOne(Provider p, ArrayNode messages) throws Exception {
    var payload = MAPPER.createObjectNode();
    payload.put("model", p.model());
    payload.set("messages", messages);
 
    var request = HttpRequest.newBuilder(URI.create("https://openrouter.ai/api/v1/chat/completions"))
        .header("Authorization", "Bearer " + System.getenv("OPENROUTER_API_KEY"))
        .header("Content-Type", "application/json")
        .timeout(p.timeout())
        .POST(HttpRequest.BodyPublishers.ofString(payload.toString()))
        .build();
 
    var res = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() >= 400) {
        throw new ProviderException(p.name() + " returned HTTP " + res.statusCode(), res.statusCode());
    }
    JsonNode json = MAPPER.readTree(res.body());
    return new Reply(json.at("/choices/0/message/content").asText(), p.name(),
        json.at("/usage/total_tokens").asInt());
}

// Define the request and response types inline so this compiles as written
struct Message: Codable { let role: String; let content: String }
struct ChatRequest: Encodable { let model: String; let messages: [Message] }
 
struct ChatResponse: Decodable {
    struct Choice: Decodable { let message: Message }
    let choices: [Choice]
    let usage: Usage
}
 
struct Provider {
    let name: String
    let model: String
    let timeout: TimeInterval
}
 
struct Reply {
    let text: String
    let provider: String
    let tokens: Int
}
 
// badStatus carries the status code as an associated value so the next section's
// shouldTryNext has something to judge on
enum RouterError: Error {
    case badStatus(Int)
    case allFailed(String)
}
 
// The response body uses snake_case total_tokens, so map it explicitly to camelCase
struct Usage: Decodable {
    let totalTokens: Int
 
    enum CodingKeys: String, CodingKey {
        case totalTokens = "total_tokens"
    }
}
 
let providers = [
    Provider(name: "fast", model: "openai/gpt-4o-mini", timeout: 8),
    Provider(name: "strong", model: "anthropic/claude-3.5-sonnet", timeout: 15),
    Provider(name: "backup", model: "google/gemini-2.0-flash-001", timeout: 8),
]
 
func callOne(_ p: Provider, messages: [Message]) async throws -> Reply {
    var request = URLRequest(url: URL(string: "https://openrouter.ai/api/v1/chat/completions")!,
                             timeoutInterval: p.timeout)
    request.httpMethod = "POST"
    let key = ProcessInfo.processInfo.environment["OPENROUTER_API_KEY"]!
    request.setValue("Bearer \(key)", forHTTPHeaderField: "Authorization")
    request.setValue("application/json", forHTTPHeaderField: "Content-Type")
    request.httpBody = try JSONEncoder().encode(ChatRequest(model: p.model, messages: messages))
 
    let (data, response) = try await URLSession.shared.data(for: request)
    guard let http = response as? HTTPURLResponse else { throw RouterError.badStatus(-1) }
    guard http.statusCode < 400 else { throw RouterError.badStatus(http.statusCode) }
    let decoded = try JSONDecoder().decode(ChatResponse.self, from: data)
    return Reply(text: decoded.choices[0].message.content, provider: p.name,
                 tokens: decoded.usage.totalTokens)
}

One detail here is easy to miss: the timeout lives in configuration, and it differs per vendor. A strong model is slow by nature, and giving it 8 seconds kills a large number of requests that would have succeeded; a fast model that has not produced its first token after 8 seconds is essentially a lost cause. Timeouts must be set per provider — a single global timeout is the most common configuration mistake beginners make.

One more detail no language escapes: vendors do not agree on field naming. The OpenAI lineage uses snake_case total_tokens, while Swift and Java model objects habitually use camelCase. When decoding you either write the mapping explicitly (CodingKeys in Swift, @JsonProperty in Java) or turn on a global snake_case strategy on the decoder. The Swift version above takes the former route — do not expect field names to line up by themselves; this is the most frequent low-grade bug when wiring up a second provider.

Finally, draw a boundary: the unified layer is responsible only for getting a request out and a result back — do not put business logic inside it. People take shortcuts and write something like "in the support scenario, append this extra prompt" into this layer, and then a new business line means editing shared code, and every edit forces a regression pass across three callers. The thinner this layer, the more callers it can serve.

Fallback: when to switch, and when not to

You go to an ATM and it prints a slip. If the slip says the machine is out of cash, another machine solves it. If it says your card has been reported lost, visiting every ATM in town will not help. It is the same action — try another machine — and whether it is worth doing depends entirely on the reason for the failure.

Model calls are identical. The hard part is not the for loop, it is judging whether an error is worth trying another vendor for. The intuitive trap is: they all failed, so retry them all. Wrong — classification has to happen before the retry, and the basis for classifying is not the status code itself but one question: could another vendor plausibly do better?

• Switching will not help: 400 for a malformed body, 403 blocked by a safety policy. The request itself is the problem, and retrying just commits the same bug twice.
• Switching will most likely help: 408 timeout, 429 rate limit, 5xx server fault — the textbook "their problem."
• The two most often filed in the wrong bucket are 404 and 402. A 404 usually means the model id you wrote has been retired or renamed — one of the everyday failures listed at the top of this chapter. It looks like your problem, and switching vendors may well succeed. A 402 is insufficient balance, and switching to an account with money in it obviously rescues it. Filing all 4xx as "do not retry" is the most common error here.
• A 401 depends on how you manage credentials: with all three sharing one gateway key, switching changes nothing; with separate keys per vendor, A's being revoked while B still works means switching rescues it entirely.

One more thing to get right: judge on structured fields, not by parsing the error message string. Something like err.message.startsWith('HTTP ') fails silently the day you reword a log line. So each version below defines an error type carrying a status first.

// 402 insufficient balance and 404 model retired also belong to "another vendor might
// do better" - do not leave them out
const RETRIABLE_STATUS = new Set([402, 404, 408, 429])
 
function shouldTryNext(err) {
  if (err.name === 'TimeoutError') return true
  if (err instanceof ProviderError) {
    return RETRIABLE_STATUS.has(err.status) || err.status >= 500
  }
  // 401 lands in false here on the premise that all three vendors in this course share
  // one OpenRouter key; with independent credentials, judge 401 as "switch vendors"
  return false // 400 and 403: your own problem, nobody else can help
}
 
async function callModel(messages) {
  const errors = []
  for (const provider of PROVIDERS) {
    try {
      return await callOne(provider, messages)
    } catch (err) {
      errors.push(`${provider.name}: ${err.message}`)
      if (!shouldTryNext(err)) throw err
    }
  }
  throw new Error(`Every provider failed: ${errors.join('; ')}`)
}

# 402 insufficient balance and 404 model retired also belong to "another vendor might
# do better" - do not leave them out
RETRIABLE_STATUS = {402, 404, 408, 429}
 
 
def should_try_next(err: Exception) -> bool:
    if isinstance(err, httpx.TimeoutException):
        return True
    if isinstance(err, httpx.HTTPStatusError):
        code = err.response.status_code
        return code in RETRIABLE_STATUS or code >= 500
    # 401 lands in False here on the premise that all three vendors in this course share
    # one OpenRouter key; with independent credentials, judge 401 as "switch vendors"
    return False  # 400 and 403: your own problem, nobody else can help
 
 
def call_model(messages: list[dict]) -> dict:
    errors = []
    for provider in PROVIDERS:
        try:
            return call_one(provider, messages)
        except Exception as err:
            errors.append(f"{provider.name}: {err}")
            if not should_try_next(err):
                raise
    raise RuntimeError("Every provider failed: " + "; ".join(errors))

// 402 insufficient balance and 404 model retired also belong to "another vendor might
// do better" - do not leave them out
static final Set<Integer> RETRIABLE_STATUS = Set.of(402, 404, 408, 429);
 
static boolean shouldTryNext(Exception err) {
    if (err instanceof HttpTimeoutException) return true;
    if (err instanceof ProviderException pe) {
        // read the structured field rather than parsing getMessage()
        return RETRIABLE_STATUS.contains(pe.status) || pe.status >= 500;
    }
    // 401 lands in false here on the premise that all three vendors in this course share
    // one OpenRouter key; with independent credentials, judge 401 as "switch vendors"
    return false; // 400 and 403: your own problem, nobody else can help
}
 
static Reply callModel(ArrayNode messages) throws Exception {
    var errors = new ArrayList<String>();
    for (var provider : PROVIDERS) {
        try {
            return callOne(provider, messages);
        } catch (Exception err) {
            errors.add(provider.name() + ": " + err.getMessage());
            if (!shouldTryNext(err)) throw err;
        }
    }
    throw new IllegalStateException("Every provider failed: " + String.join("; ", errors));
}

// 402 insufficient balance and 404 model retired also belong to "another vendor might
// do better" - do not leave them out
let retriableStatus: Set<Int> = [402, 404, 408, 429]
 
func shouldTryNext(_ err: Error) -> Bool {
    if let urlError = err as? URLError, urlError.code == .timedOut { return true }
    // Cast to the concrete type before matching a case: pattern-matching an enum against
    // the Error existential does not compile
    if let routerError = err as? RouterError, case .badStatus(let code) = routerError {
        return retriableStatus.contains(code) || code >= 500
    }
    // 401 lands in false here on the premise that all three vendors in this course share
    // one OpenRouter key; with independent credentials, judge 401 as "switch vendors"
    return false // 400 and 403: your own problem, nobody else can help
}
 
func callModel(messages: [Message]) async throws -> Reply {
    var errors: [String] = []
    for provider in providers {
        do {
            return try await callOne(provider, messages: messages)
        } catch {
            errors.append("\(provider.name): \(error)")
            guard shouldTryNext(error) else { throw error }
        }
    }
    throw RouterError.allFailed(errors.joined(separator: "; "))
}

That loop is a dozen lines and it costs you three separate bills.

The first is money. Switching to the next vendor means paying twice for the same prompt, and a three-vendor chain costs triple in the worst case. You will not notice normally, because the happy path calls one vendor — but the moment the primary provider gets shaky the bill climbs while you are unaware, which is why "how often fallback fired" has to be a monitored metric.

The second is latency. A serial fallback's total time is the sum of the timeouts: three vendors at 15 seconds each means a worst case of 45 seconds for the user, when failing outright at 20 would have been kinder. So on top of per-vendor timeouts you need a total budget.

The third is the most expensive and it is called a cascade. The instant the primary provider rate-limits, all of your traffic lands on the backup within the same second — and the backup's quota was requested on the basis of taking a slice of traffic, so it cannot absorb a sudden doubling, starts returning 429 itself, and the traffic moves on to the third. All three fall over in sequence, which is worse than having had one. The fix is a circuit breaker: keep a consecutive-failure count per provider, skip it temporarily past a threshold, then let a trickle through after a cooldown and only restore it on success. It shares its lineage with the circuit breakers in a microservice gateway, and it comes up in interviews constantly.

The system prompt: the framework's default persona is a time bomb

When you hire a new support agent, you start with a job description: which company they represent, what they may and may not answer, how long a reply should be, which system to look up an order in. Push them to a desk with none of that explained and they will still serve customers — using whatever they picked up somewhere else. The part you did not write is never blank; it is a default somebody else filled in.

You may not have noticed while getting the Pi SDK running on day three: you wrote no system prompt, and the agent still behaved like an assistant. That persona is the default the framework filled in for you.

At the demo stage that is thoughtful. In production it is a hazard. A default persona usually says something generic along the lines of "you are a helpful AI assistant," which brings three concrete problems.

First, it does not know your business boundary. A user asks it to write a resignation letter and a generic assistant will happily oblige; but if your product is e-commerce support, that reply is completely off-topic and wasted tokens too.

Second, its output format is outside your control. A default persona will not constrain "answer in at most three sentences" or "no Markdown headings," so a level-two heading suddenly appears inside a chat bubble and your frontend styling falls apart.

Third, and worst — it changes when the framework upgrades. Every behavior you tested rests on a piece of text you did not write, cannot see, and which may quietly change after the next pnpm update. That class of bug is exceptionally hard to trace, because the code never moved.

So the rule is: always write the system prompt explicitly, even if it is one sentence. In production it is usually not a hard-coded string but an assembled template:

You are the after-sales assistant for a certain e-commerce platform.
 
[Scope]
- Only answer questions about orders, returns and exchanges, and shipping
- When asked about anything else, politely explain that you only handle
  after-sales matters, and do not attempt an answer
 
[Output requirements]
- At most 3 sentences per reply; no Markdown headings or lists
- Anything involving amounts or timings must come from a tool call; never answer
  from memory
 
[Current context]
- Current time: 2026-09-04 14:30
- Customer tier: gold
- Available tools: query_order, apply_refund

Look at the structure: persona plus scope plus output requirements plus dynamic context. The first three blocks are a static template; the last is assembled per request. Putting the current time in there is a frequent interview point — the model has no clock, and if you do not tell it today's date it cannot work out which day "the order I placed three days ago" refers to.

Routing: let the cheap model do the rough work

With a unified layer and fallback in place, the last step is choosing a model actively rather than waiting passively for a failure.

The core fact is that model prices differ by more than fifty times, while a large share of your tasks does not need the strongest model at all. Deciding whether a user's sentence is about an order is 99% accurate with the cheapest small model; drafting a dispute clause from three contracts requires the strongest one you have. Using a flagship model for intent classification is driving a sports car to pick up a parcel downstairs.

There are three common routing dimensions:

Here is a number you can feel immediately. Suppose a support agent handles 10,000 conversation turns a day at an average of 2,000 tokens per turn. Send everything to a flagship model at 15 per million tokens and that is about 300 a day; but at least six in ten of those turns are rough work like intent classification and information extraction, and moving them to a small model at 0.3 per million tokens brings the daily cost down to roughly 125 — over sixty thousand a year, with users unable to tell the difference. So when an interviewer asks how you control cost, the first answer they want to hear is usually not "compress the context" but "tiered routing."

The implementation is not complicated: add a tier parameter to callModel, let the caller declare whether this call wants fast or strong, have the routing layer pick the starting provider accordingly, and reuse the fallback logic wholesale. Start with static tiers by task type; do not jump straight to dynamic routing where the model decides which model to use — that scheme costs an extra model call of its own, and both the latency and the cost may not repay it.

Source Reading

Hands-On Lab

starter/ has four exercise points cut out of it and runs fully offline under MOCK=1: the mock layer manufactures timeouts, rate limits, and 400s according to mockFailure, so you can verify fallback without actually rate-limiting a real account. The fake model in there also reads the system prompt you assembled — with no declared scope it answers anything, and only a declared scope makes it decline. buildSystemPrompt's default return value is precisely the phoned-in persona this chapter criticizes, so run the poem request against it once as-is and watch it go off-topic with your own eyes.

1. Extract the three providers' configuration into an array, each with its own timeout value, and write the unified callOne.
2. Implement shouldTryNext, classifying by whether another vendor could plausibly do better: 400 and 403 throw immediately, while 408, 429, 5xx, and the easily forgotten 402 and 404 continue to the next vendor. Judge on the status field of ProviderError; parsing the error message string is not allowed.
3. Ask an out-of-scope question with the default persona first and remember how it goes off-topic; then write a custom system prompt (persona plus scope plus output requirements plus the current time) and run the same command to see it decline instead.
4. Use MOCK=1 to make the first provider time out and the second return 429, verify the third one succeeds, and check that the log shows the full downgrade path.
5. Print the provider name and token count for every call, ask five questions in a row, and observe the cost distribution.

Interview Questions

Today's four questions are in the bank below, covering the motivation for multiple providers, the fallback trade-offs, what a system prompt is for, and the cost-versus-latency decision. Expand a question and read the analysis before the key points — the follow-up on question 2, about circuit breakers and cascades, is the spot in this chapter you are most likely to be asked about, so do not skip it.

Checklist and Tomorrow

• Implement a model-calling layer that supports at least 3 providers, switchable by configuration
• Write a custom system prompt for an agent that overrides the framework's default persona
• Name the factors a fallback strategy must weigh when a model call fails (latency, cost, availability)
• Say clearly which errors deserve a fallback and which do not, and explain why retrying a 400 is waste while a 404 should switch vendors
• The lab's downgrade log shows the full chain from timeout to 429 to success
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D5) attention moves from how you call the model to how the model calls you. You will fit the agent with a real tool system: validate arguments against a schema, feed an error message back to the model when a tool fails so it can correct itself, then use event subscription to expose every internal step. Today's fallback handles "external services are unreliable"; tomorrow's error feedback handles "the model itself makes mistakes" — and only stacked together do those two become the real source of a production agent's stability.