Messages, Context Engineering and Compression, Session Storage/Recovery/Forking (dg M06/M08/M09/M10)

Context engineering: deciding what goes in a fixed budget

D1 used a desk to explain what a context window is; today we go straight to the next question — the desk is this big, so what happens when it is full? Change the scene: packing a cabin bag before a trip. The volume is fixed, and clothes, laptop, and papers all want to come, so packing is not about how much fits but about ranking priorities inside a fixed volume while leaving room for what you buy on the way back. Context engineering is exactly that: you decide what goes into each request, and how much.

The budget splits into at least four parts, and most people only count one:

The second row is the one people forget. Tool definitions are resent every round, billed by volume just like the history, and they silently get longer the day you register your eleventh tool. That is the other face of D5's principle: the more detail you write, the more accurately the model uses a tool, and every word is paid for once per round.

So a 128k window never means you have 128k available. Subtract the first three rows and what is genuinely left for the history is often only six or seven tenths — and you must keep a safety margin, because hitting the ceiling returns a hard error from the model and the user sees outright failure rather than slightly worse recall. The move is to give the history its own budget line and act the moment it is touched. So, act how?

Context compression: keep the decisions, drop the transcript

Nobody sends the verbatim transcript of a two-hour meeting to the group chat. They send minutes: who owns what, due when, which points went unresolved. The ten minutes that went off-topic should be thrown away the moment the meeting ends — what gets used later is never the process, it is the conclusion.

Context compression is minute-taking for a conversation, and it answers three questions: when to fire, what to drop, what to keep.

When to fire. Use a threshold, not a timer, and never wait for an error. The common practice is to fire once the history exceeds seven tenths of its budget. Why not nine tenths? Compression itself is a model call to write the summary, so it has latency and it can fail; leave it until nine tenths and one timeout or rate limit means the very next round hits the wall. Seven tenths is the rescue window you leave yourself.

What to drop. Prioritize the process-shaped material in the earliest stretch: the model's intermediate reasoning, raw tool results that have already been consumed, requirements the user later overruled. What they share is that their value has already settled into the conclusions that follow, so keeping the originals is just weight.

What to keep. This is the part that, written wrong, gives the model instant amnesia. The system prompt is always kept verbatim (it is not part of the history). The most recent handful of messages keep their original text, because the model leans hardest on what just happened. And constraints or facts the user stated ("I am in Shanghai," "budget under two thousand") must be written into the summary rather than discarded as process. One more thing goes wrong constantly: the cut has to land at the start of a round. Cut between an assistant carrying tool_calls and its matching tool result and the next request contains a dangling call with no result, which nearly every vendor's API answers with a 400.

const MAX_TOKENS = 8000 // the history's own budget, not the model's window ceiling
const TRIGGER_RATIO = 0.7 // compress at seven tenths, leaving rescue room if the summary call fails
const KEEP_RECENT = 6 // the last 6 keep their original text: the model leans on what just happened
 
// Walk back to the nearest position that is not a tool result, so the cut lands at a round's start
function alignToTurnStart(msgs, index) {
  let i = index
  while (i > 0 && msgs[i].role === 'tool') i--
  return i
}
 
async function compressIfNeeded(history, summarize) {
  if (estimateTokens(history) < MAX_TOKENS * TRIGGER_RATIO) return history
 
  const system = history.filter((m) => m.role === 'system')
  const rest = history.filter((m) => m.role !== 'system')
  const cut = alignToTurnStart(rest, Math.max(0, rest.length - KEEP_RECENT))
  if (cut === 0) return history
 
  // The summary is itself a model call: it costs money and takes time, so do not
  // leave it until the last moment
  const digest = await summarize(rest.slice(0, cut))
  return [...system, { role: 'user', content: `[Summary of earlier conversation] ${digest}` }, ...rest.slice(cut)]
}

MAX_TOKENS = 8000  # the history's own budget, not the model's window ceiling
TRIGGER_RATIO = 0.7  # compress at seven tenths, leaving rescue room if the summary call fails
KEEP_RECENT = 6  # the last 6 keep their original text: the model leans on what just happened
 
 
def align_to_turn_start(msgs: list[dict], index: int) -> int:
    """Walk back to the nearest non-tool position so the cut lands at a round's start."""
    while index > 0 and msgs[index]["role"] == "tool":
        index -= 1
    return index
 
 
async def compress_if_needed(history: list[dict], summarize) -> list[dict]:
    if estimate_tokens(history) < MAX_TOKENS * TRIGGER_RATIO:
        return history
 
    system = [m for m in history if m["role"] == "system"]
    rest = [m for m in history if m["role"] != "system"]
    cut = align_to_turn_start(rest, max(0, len(rest) - KEEP_RECENT))
    if cut == 0:
        return history
 
    # The summary is itself a model call: it costs money and takes time, so do not
    # leave it until the last moment
    digest = await summarize(rest[:cut])
    return [*system, {"role": "user", "content": f"[Summary of earlier conversation] {digest}"}, *rest[cut:]]

// Dependencies: java.util; Summarizer is a functional interface returning summary text
static final int MAX_TOKENS = 8000; // the history's own budget, not the model's window ceiling
static final double TRIGGER_RATIO = 0.7; // compress at seven tenths, leaving rescue room
static final int KEEP_RECENT = 6; // the last 6 keep their original text
 
// Walk back to the nearest position that is not a tool result, so the cut lands at a round's start
static int alignToTurnStart(List<Message> msgs, int index) {
    while (index > 0 && "tool".equals(msgs.get(index).role())) index--;
    return index;
}
 
static List<Message> compressIfNeeded(List<Message> history, Summarizer summarize) throws Exception {
    if (estimateTokens(history) < MAX_TOKENS * TRIGGER_RATIO) return history;
 
    var system = history.stream().filter(m -> "system".equals(m.role())).toList();
    var rest = history.stream().filter(m -> !"system".equals(m.role())).toList();
    int cut = alignToTurnStart(rest, Math.max(0, rest.size() - KEEP_RECENT));
    if (cut == 0) return history;
 
    // The summary is itself a model call: it costs money and takes time, so do not
    // leave it until the last moment
    String digest = summarize.apply(rest.subList(0, cut));
    var compressed = new ArrayList<>(system);
    compressed.add(new Message("user", "[Summary of earlier conversation] " + digest, null, List.of()));
    compressed.addAll(rest.subList(cut, rest.size()));
    return List.copyOf(compressed);
}

let maxTokens = 8000 // the history's own budget, not the model's window ceiling
let triggerRatio = 0.7 // compress at seven tenths, leaving rescue room if the summary call fails
let keepRecent = 6 // the last 6 keep their original text: the model leans on what just happened
 
// Walk back to the nearest position that is not a tool result, so the cut lands at a round's start
func alignToTurnStart(_ msgs: [Message], _ index: Int) -> Int {
    var i = index
    while i > 0, msgs[i].role == "tool" { i -= 1 }
    return i
}
 
func compressIfNeeded(
    _ history: [Message],
    summarize: ([Message]) async throws -> String
) async throws -> [Message] {
    guard Double(estimateTokens(history)) >= Double(maxTokens) * triggerRatio else { return history }
 
    let system = history.filter { $0.role == "system" }
    let rest = history.filter { $0.role != "system" }
    let cut = alignToTurnStart(rest, max(0, rest.count - keepRecent))
    guard cut > 0 else { return history }
 
    // The summary is itself a model call: it costs money and takes time, so do not
    // leave it until the last moment
    let digest = try await summarize(Array(rest[0..<cut]))
    return system + [Message(role: "user", content: "[Summary of earlier conversation] \(digest)")] + rest[cut...]
}

Here is the effect. Take that 30-round, 30,000-token example with no compression along the way. Afterwards you hold one roughly 600-token summary plus the last 6 messages in full. By this section's own arithmetic, 4 messages per round is about 1,000 tokens, so 6 messages is a round and a half, roughly 1,500. Total about 2,100 — a fourteenth of the original. And the saving is continuous: every subsequent round is billed at the compressed length, so you save not once but on all remaining rounds. The price is one extra model call plus one irreversible loss of information.

Session persistence: a receipt or a monthly statement

A supermarket receipt prints line after line and never erases a mistake to reprint it; a bank's monthly statement overwrites one fresh summary each month. Saving a session is a choice between those two. Append-only is the receipt: every message produced appends one line to the end of a file, and history is never modified. A snapshot is the statement: at the end of every round the whole session is overwritten.

Prefer append-only, for three fairly concrete reasons: write cost is independent of history length; it can naturally be replayed to any step, which is the foundation for the next two sections on recovery and forking; and when something goes wrong you have a complete audit trail. A snapshot's value is speed — replaying thousands of messages in full is too slow — so the common combination is append-only as the source of truth plus periodic snapshots.

For format, JSONL with one JSON object per line is the least trouble: readable, tail-able, and appendable without parsing the whole file. Metadata (who the parent session is, which message it forked at, how far compression has reached) goes in a small separate file:

.sessions/
  s-8f3a.jsonl        # one message per line, append-only, never rewritten
  s-8f3a.meta.json    # session metadata: id, parentId, forkedAt, compression watermark

{"seq":1,"role":"user","content":"When will the order I placed three days ago arrive?","ts":"2026-09-04T10:00:00Z"}
{"seq":2,"role":"assistant","content":"","tool_calls":[{"id":"c1","name":"query_order"}],"ts":"2026-09-04T10:00:01Z"}
{"seq":3,"role":"tool","tool_call_id":"c1","content":"...order JSON...","ts":"2026-09-04T10:00:02Z"}

In production the files become a database (a sessions table and a messages table in Postgres) with the structure unchanged.

Session recovery: load the right save point

Loading a game save, you expect to pick up exactly where you were. But if the save recorded your position and not your inventory, you reappear on the spot with nothing — the quality of a recovery is determined by what was saved in the first place. Sessions are the same, and two traps catch nearly everybody.

The first: persisting the system prompt of the moment along with the history. D4 set the rule, and the last block of its structure is dynamic context, which holds live things like the current time. A session saved three days ago and loaded today makes the model believe it is still three days ago, and it computes dates wrong. The right approach: the system prompt does not go into the persisted history. Assemble a fresh one with the current time on every recovery, then attach the conversation read off disk.

The second is subtler: the save landed in the middle of a tool call. The process died after the model returned tool_calls and before the tool result was written back, so the last line on disk is a dangling call, and sending that on the next recovery produces the previous section's 400. So after loading you must run a completeness check: if the tail is a tool_calls with no matching result, either append a tool message saying the tool execution was interrupted and to call it again, which reuses D5's error-feedback mechanism, or discard that incomplete tail.

The third thing to restore is the compression watermark: how far this session has already been compressed. Without it, the threshold check counts the already-summarized stretch again, so you either compress repeatedly or never compress at all.

Session forking: a new line from a save point

One of the great pleasures in games: save before a pivotal choice, play through option A, and if you dislike it, load and take B. Both lines share the earlier story and diverge after, and however A plays out, the save itself is unchanged.

Session forking is that, and the definition is clean: copy the first k messages of a history into a new session, remember which session is the parent and which message it was cut at; the parent is not modified in any way. That last clause is the line between forking and rolling back — a rollback lops off a stretch of history and continues, which is destructive, while a fork grows a new branch and both sides can continue.

It maps to three high-frequency product scenarios: regenerate (the user dislikes a reply and retries, which is a fork from the second-to-last message), prompt A/B experiments (attach two versions of a system prompt to the same real history and run each, which is far more credible than fabricated data), and reproducing a production issue (fork the offending session into your own account and poke at it however you like without touching the original).

import { randomUUID } from 'node:crypto'
 
// Forking = copy the first k messages plus remember where you came from. The parent is
// read-only, which is the fundamental difference from a rollback.
function forkSession(parent, atIndex) {
  const cut = alignToTurnStart(parent.messages, atIndex)
  return {
    id: randomUUID(),
    parentId: parent.id,
    forkedAt: cut,
    // A deep copy is mandatory: referencing the parent's message objects directly means
    // the two branches will eventually contaminate each other
    messages: structuredClone(parent.messages.slice(0, cut)),
    createdAt: new Date().toISOString(),
  }
}

import copy
import uuid
from datetime import datetime, timezone
 
 
def fork_session(parent: dict, at_index: int) -> dict:
    """Forking = copy the first k messages plus remember where you came from.
    The parent is read-only, which is the fundamental difference from a rollback."""
    cut = align_to_turn_start(parent["messages"], at_index)
    return {
        "id": str(uuid.uuid4()),
        "parent_id": parent["id"],
        "forked_at": cut,
        # deepcopy is mandatory: a slice copies only the outer list, and the dicts inside
        # are still the same objects, so editing one side changes both
        "messages": copy.deepcopy(parent["messages"][:cut]),
        "created_at": datetime.now(timezone.utc).isoformat(),
    }

// Message is a record (immutable), so no per-element cloning is needed here:
// copying a list of immutable elements is already a safe deep copy
record Session(String id, String parentId, int forkedAt, List<Message> messages, Instant createdAt) {}
 
static Session fork(Session parent, int atIndex) {
    int cut = alignToTurnStart(parent.messages(), atIndex);
    return new Session(
        UUID.randomUUID().toString(),
        parent.id(),
        cut,
        List.copyOf(parent.messages().subList(0, cut)),
        Instant.now());
}

// Message is a struct (a value type), so assigning an array slice already copies,
// and the "two branches sharing the same objects" trap simply cannot happen —
// safety handed to you free by value semantics
struct Session {
    let id: UUID
    let parentId: UUID?
    let forkedAt: Int
    var messages: [Message]
    let createdAt: Date
}
 
func fork(_ parent: Session, at index: Int) -> Session {
    let cut = alignToTurnStart(parent.messages, index)
    return Session(
        id: UUID(),
        parentId: parent.id,
        forkedAt: cut,
        messages: Array(parent.messages[0..<cut]),
        createdAt: Date())
}

Two of those comments are worth reading side by side: JavaScript and Python need an explicit deep copy, because a slice copies only the outer container while the same mutable objects sit inside; Java's record and Swift's struct hold immutable or value-type elements, so copying the list is enough. That is not a syntax difference, it is a data-model difference — design messages to be immutable and forking goes from "remember to deep-copy" to "hard to get wrong."

There are three engineering costs. Storage amplification: every fork copies the first k messages in full, so five forks means six histories; at scale you switch to storing a parent reference plus a cut point and stitching on read, at the price of a more complex read path. Cost attribution: each branch spends separately, and the bill has to aggregate back up a tree by parentId, or what you see is a pile of orphaned sessions. The third is the one people miss: the session has already been compressed and the user wants to fork from a stretch that was summarized away, so the original text is gone. That is exactly why the previous callout insists the raw history be append-only. How deep you can fork depends on whether you kept the originals.

Classifying memory: what follows the session, and what follows the person

A tour guide carries two things. One is today's itinerary — meet at nine, lunch here — which is waste paper the moment the group disperses. The other is the agency's customer file, noting that this traveler avoids spice and travels with a child, still useful when they come back next year.

That is the boundary. Short-term context is this session's messages array: void when the session ends, and sent in full with every request. Everything today — compression, recovery, forking — manages that. Long-term memory is cross-session facts and preferences about a user; it does not live in messages but in an external system, retrieved on demand with the few relevant hits injected into the request.

Deciding where a piece of information belongs takes three questions: will it still be needed after this session? will it expire with time? can retrieval bring it back? "The user is in Shanghai" is needed, does not expire, and is retrievable, so it goes to long-term memory. "Rewrite that second paragraph in three sentences" is not needed later and expires immediately, so it stays short-term. Anything in between stays short-term first and settles into a long-term preference once it recurs.

The engineering differences are total: short-term context is billed by token, bounded by the window, and deleted with the session; long-term memory is stored per item, bounded by retrieval quality, and has to handle expiring an old memory when the user changes their mind. The most common misuse is stuffing long-term memory in as context all at once — six months of accumulated preferences, two hundred items, jammed into a request both bursts the window and distracts the model with irrelevant memories. The right posture is retrieving only the three to five most relevant. That retrieval machinery is RAG, D12's main subject. Today it is enough to draw the boundary: today is about this conversation remembering; D12 is about this person being remembered.

Source Reading

Hands-On Lab

starter/ has four exercise points cut out of it and runs fully offline under MOCK=1: the mock layer manufactures multi-round conversations with tool calls so the history reaches the threshold quickly, and you see compression fire without paying for thirty real rounds. All four blanks default to something that runs but is visibly wrong — tokens always zero, the history never shortens, a restart reads zero messages, a fork contaminates its parent — so run it as-is once and read those four warning lines.

1. Run MOCK=1 pnpm start unmodified first and note the four exercise-incomplete lines; that is your to-do list.
2. Implement appendMessage persistence: append each message as JSONL to .sessions/main.jsonl, then cat the file and confirm one message per line with the tool message present.
3. Implement estimateTokens and compressIfNeeded: once past seven tenths of the budget, summarize the early messages into one, aligning the cut with the provided alignToTurnStart, and finish with a comparison line showing the reduction.
4. Run MOCK=1 pnpm start chat "I live in Shanghai, please note that", then start a separate process for MOCK=1 pnpm start chat "where did I say I live?"; the second must print that it recovered N messages from disk and must answer correctly.
5. Implement the deep copy in forkSession, fork with fork 4, append a round to it with --session=, then show main and confirm the parent is untouched.

Interview Questions

Today's four questions are in the bank below, weighted toward context management, long conversations, and memory classification. Read the analysis before the key points — the follow-up on question 1, about which model should write the summary and what to do when it fails, draws the most follow-ups.

Checklist and Tomorrow

• Draw a message's complete path from user input to being written into history
• Implement a simple context-compression strategy that fires automatically as the window nears full
• Implement persisting a session, resuming it to continue the conversation, and forking a new session from a given step
• Name the four parts of the context budget, and say why the compression cut must align to a round's start
• All 5 acceptance criteria of the lab pass
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D7) we assemble the last six days into a real service: expose an SSE endpoint with Fastify, package it with Docker, and run the week-one retrospective. The order is deliberate — today's sessions live in a local file and survive a restart, but that road closes the instant you become a service: several instances each write their own disk, the load balancer sends the user's second request to a different machine, and the history vanishes. Session state has to move out of the process and off the single machine, and that is the first unavoidable hurdle on the way from script to service.