Session Persistence and Recovery: an Append-Only Event Log, Resume and Forking, Week One Retrospective

Why an event log and not a session object: appendable, auditable, forkable

The first temptation is serializing the whole session into one JSON file: an object with a message array, usage statistics and current state, rewritten at the end of each turn. Five lines of code.

It has three problems, all of the "fine until it is not, and then it is ugly" kind:

1. Every write reads and writes the whole thing. It slows as the session grows, and writing requires holding the entire content in memory.
2. There are endlessly many failure modes. Crash halfway through a whole-file write and you get truncated JSON — not "the last message is missing" but a file that will not parse at all. More insidious is a half-successful overwrite: new content shorter than old leaves the tail of the old file behind.
3. There is no history, only the present. You cannot answer "which turn made that edit" or "how many tokens did the previous turn cost." Those questions always come up in a post-mortem or a cost review.

An event log is the opposite: append only, never rewrite. Every time something happens, add a line at the end.

The second row is today's sentence worth remembering: appending has exactly one failure mode, so it is the only one you can think through in advance. Section five deals entirely with that one.

One event per line: sequence, timestamp, type — three fields suffice

Use JSONL: one JSON object per line, with no syntactic relationship between lines. The cost is assembling on read; the benefit is touching only the file's end on write.

Each line has three mandatory fields — seq, ts and type — followed by that type's own fields:

{"seq":1,"ts":"2026-09-07T10:00:00.000Z","type":"message","message":{"role":"user","content":"run the tests and see which case fails"}}
{"seq":4,"ts":"2026-09-07T10:00:01.200Z","type":"message","message":{"role":"assistant","content":"Let me run the tests first.","toolCalls":[{"id":"call_test_1","name":"run_command","args":"{\"command\":\"node --test\"}"}]}}
{"seq":5,"ts":"2026-09-07T10:00:01.900Z","type":"tool_end","id":"call_test_1","result":{"ok":false,"content":"exit code 1..."}}

Each field is irreplaceable: seq locates and forks ("branch from event 8"), ts audits ("how long did this step take"), and type determines what other fields follow. The lab has five types: message, tool_end, usage, error and fork.

export type SessionEvent =
  | { seq: number; ts: string; type: 'message'; message: Message }
  | { seq: number; ts: string; type: 'tool_end'; id: string; result: ToolResult }
  | { seq: number; ts: string; type: 'error'; message: string }
  | { seq: number; ts: string; type: 'fork'; from: string; atSeq: number }
 
async append(event: NewEvent): Promise<SessionEvent> {
  this.seq += 1
  const full = { seq: this.seq, ts: new Date().toISOString(), ...event } as SessionEvent
  // One appendFile is one O_APPEND write. This is the persistence layer's only write path:
  // being the only one is what keeps the failure mode to "the last line is incomplete"
  await fs.appendFile(sessionPath(this.id), `${JSON.stringify(full)}\n`, 'utf8')
  return full
}

@dataclass(frozen=True)
class SessionEvent:
    seq: int
    ts: str
    type: Literal["message", "tool_end", "usage", "error", "fork"]
    payload: dict[str, Any]
 
 
class SessionLog:
    def append(self, kind: str, **payload: Any) -> SessionEvent:
        self._seq += 1
        event = SessionEvent(self._seq, datetime.now(timezone.utc).isoformat(), kind, payload)
        line = json.dumps({"seq": event.seq, "ts": event.ts, "type": kind, **payload})
        # "a" is O_APPEND: the only write path, so the only failure mode is a partial last line
        with open(self.path, "a", encoding="utf-8") as handle:
            handle.write(line + "\n")
        return event

Two small decisions worth stating:

• A session id looks like s-20260907-100000-ab12 (date, time, four random characters). The time makes "sort by filename" equal "sort by time" — which is how --resume without arguments picks the most recent one, with no separate "current session" pointer to maintain. The randomness is because two sessions can start in the same second.
• Persistence happens when an event occurs, not at the end of a turn. Batching to the end of a turn loses everything if the process is killed mid-turn; and the log's order becomes "all events, then all messages," leaving the reader to reorder mentally. The lab's order is naturally correct: the assistant message, its tool_end, then the tool result message.

Replay recovery: which events rebuild messages and which are only for display

Recovery is replay: read the events from the start and rebuild the message array. There is one rule, and it is counterintuitive —

Only message events participate in the rebuild; none of the other four do.

tool_end clearly holds the tool result, so why not rebuild the tool message from it? Because the tool result already landed in the log as a message with role tool, and that is what gets sent to the model. What tool_end additionally records is elapsed time, character count, whether it was truncated, and whether it was stopped by the approval gate or by loop detection — that is for humans: auditing, costing, post-mortems.

export function replay(events: SessionEvent[]): Replayed {
  const messages: Message[] = []
  let replayed = 0
  let skipped = 0
 
  for (const event of events) {
    if (event.type === 'message') {
      messages.push(event.message)
      replayed += 1
    } else {
      // tool_end / usage / error / fork: for humans only, not part of the rebuild
      skipped += 1
    }
  }
  return { messages, replayed, skipped }
}

def replay(events: Iterable[SessionEvent]) -> Replayed:
    """Only message rebuilds. Mixing other types in makes tool results appear twice."""
    messages = [event.payload["message"] for event in events if event.type == "message"]
    skipped = sum(1 for event in events if event.type != "message")
    return Replayed(messages=messages, replayed=len(messages), skipped=skipped)

Separating the two classes pays directly: the rebuild has one rule and never forks over "which field is authoritative"; and adding a statistics column to the terminal only adds another non-message event type, with the replay code unchanged. Conversely, mixing both into one event type bills you on day twelve's compaction — you would have to distinguish "this part goes to the model" from "this part is for humans" inside one object.

In the lab, one turn produces 8 events, 5 of them message; replay yields 5 messages with 3 not participating. Those numbers are reproducible under MOCK=1 (the script is fixed). The cumulative token count on that line is not — tool results carry the command's elapsed time ("exit code 1 (65ms)"), and a different millisecond count changes the estimated tokens. So the self-test asserts it is greater than zero, not a specific value.

On version compatibility in passing: logs live a long time and your event types will grow. So do not crash on an unknown type when reading; the lab treats it as one of the non-rebuilding kinds. Conversely, the meaning of a field already written must never change — that would be rewriting history.

What to do about a corrupted log: a truncated last line is normal, so stop at the previous one

Now for that single failure mode.

A process can be Ctrl+C'd, killed, or have its terminal closed at any time. If it stops midway through an appendFile, the file's last line is half a JSON object. That is normal, not exceptional — over a year of sessions it happens many times.

There is one way to handle it: stop at the line before the bad one; do not skip, do not repair.

export async function readEvents(id: string): Promise<ReadResult> {
  const lines = (await fs.readFile(sessionPath(id), 'utf8')).split('\n')
  if (lines.at(-1) === '') lines.pop() // a well-terminated file ends with an empty string
 
  const events: SessionEvent[] = []
  for (let i = 0; i < lines.length; i += 1) {
    const parsed = parseLine(lines[i] as string, events.at(-1)?.seq ?? 0)
    // Stop here rather than continue: skipping assumes the middle can be corrupt too,
    // and appending never produces that
    if (!parsed.ok) return { events, stoppedAtLine: i + 1, reason: parsed.reason }
    events.push(parsed.event)
  }
  return { events, stoppedAtLine: null, reason: null }
}

def read_events(path: Path) -> ReadResult:
    events: list[SessionEvent] = []
    with open(path, encoding="utf-8") as handle:
        for lineno, line in enumerate(handle, start=1):
            if not line.endswith("\n"):
                # No trailing newline means the last line was not finished: stop, do not continue
                return ReadResult(events, stopped_at=lineno, reason="the last line was cut short")
            parsed = parse_line(line, events[-1].seq if events else 0)
            if parsed is None:
                return ReadResult(events, stopped_at=lineno, reason="this line cannot be read")
            events.append(parsed)
    return ReadResult(events, stopped_at=None, reason=None)

Why not skip the bad line and keep reading? Because skipping assumes the middle can be corrupt. Appending never produces that — and if it truly happens, something else touched the file, which is even less reason to guess. Stop, report the line number and the reason, and let the user decide.

Do not validate only that the JSON parses, either. The lab checks two more things: seq must strictly increase (a non-increasing one means two processes are overwriting each other, or the file was concatenated), and type must be recognized. The Python version exploits one more signal — a line with no trailing newline is the unfinished one — which is earlier and more precise than waiting for a parse failure.

Forking: copy to a new file and truncate, far safer than rewriting in place

With sequence numbers comes a new capability: branch from event N into a new session.

Its practical uses are more numerous than they sound. At event eight it proposed a direction and you want to try another without losing the current one; or you want two different attempts from the same prelude for comparison — the embryo of day twenty's benchmark evaluation.

The implementation has two steps: copy the lines with seq at or below the fork point into a new file, then append a fork event recording the lineage (which session and which step it came from). Three rules:

• Copy to a new file; never truncate in place. In-place truncation is a rewrite, and crashing halfway loses both the parent session and the fork. The cost is one extra copy on disk.
• Do not renumber seq. Keeping the original numbers is what lets a fork point map straight back to the parent session.
• Not every line is a fork point; only clean boundaries are. You may branch only after a user message or an assistant message with no tool calls. Truncating after an assistant message that requested tools yields a message array where a tool was requested with no result, and the next request is immediately invalid — that is the invariant fixed on day three. So the terminal must list the available positions rather than making the user count lines:

you > /fork
  You can fork from these positions (clean boundaries only):
    /fork 2  user: run the tests and see which case fails
    /fork 8  assistant: the failing case is divide by zero: div...
    /fork 9  user: then fix it
    /fork 19  assistant: all four cases green, the divide-by-zero case is fixed...

One last distinction, revisited on day fourteen: forking is not rolling back. Forking grows a new branch from that step, leaving the old one untouched; rolling back moves the current branch back to that step. And today's log covers only the conversation, not files — forking a new session does not restore the already-edited src/calc.js on disk. File rollback needs snapshots, day fourteen's subject. The log restores the conversation, snapshots restore files, the two are independent, and you can roll back only one side.

Week one retrospective: with seven layers stacked, it already completes a real fix

Week one ends here. Looking back, each day added a layer, and the order was not arbitrary:

flowchart TB
  D1["D1 REPL and gateway abstraction<br/>put the model in the terminal"] --> D2["D2 streaming and event layering<br/>StreamDelta and AgentEvent"]
  D2 --> D3["D3 read-only tools<br/>loop, fragment merging, truncation"]
  D3 --> D4["D4 write tools and shell<br/>first real code change"]
  D4 --> D5["D5 approval gate<br/>three states, a pause not an exception"]
  D5 --> D6["D6 errors and self-correction<br/>feedback, backoff, cancel, limits"]
  D6 --> D7["D7 session persistence<br/>append-only log, recovery, forking"]

Each layer depends only on those below it, deliberately:

• D2's layering (what the gateway said versus what the loop did) let the next five days add capabilities without touching the render layer.
• D3's read-only flag became D5's "read-only tools pass by default," one field saving an allowlist.
• D5 made approval "return a result" rather than throw, so D6 needed no new code for refusals — a refused call looks like a tool failure.
• D6 cleaned up within-turn failures first, and only then did D7 discuss cross-process recovery. Reversed, "this turn errored" and "last time did not finish" tangle in the same code, and their handling is entirely different.

Today's mca completes a real job: run tests in a real repository, read code, edit files, run again to prove the fix, asking where it should ask, stopping where it should stop, getting up after falling, and continuing after the process exits. That is a Coding Agent's core loop — the remaining fourteen days all add "work longer, work smarter, distribute better"; the core will not change again.

Week two changes direction from tomorrow: the first seven days solved whether it can work, the next seven solve whether it can avoid detours. The first item is not making it search the whole repository itself.

Source Reading

Hands-On Lab

Today leaves five exercises, two of which are "looks tidier, is actually a rewrite" traps: skipping bad lines and continuing, and truncating the parent session while forking. The starter passes four of eleven unmodified.

Recovery must be verified across two processes, so the self-test's parent spawns two children: the first opens a new session and establishes a fact, the second continues with --resume. Replaying inside one process proves nothing — the message array was in memory anyway.

1. Define the five event types and the sequence rule, writing messages, tool calls and usage as events; after one turn, use /log to check that seq runs contiguously from 1 and that tool_end sits exactly between the assistant message and the tool result.
2. Implement appending and replay recovery; start a different process with --resume and see whether the model's first sentence cites the case name established last turn.
3. Manually cut the log's last line in half and confirm recovery stops at the previous complete event and reports the line number.
4. Implement forking from a given sequence number, and have /fork with no argument list the available positions; after forking, confirm the parent session's file is byte-identical.
5. Run the self-test: MOCK=1 SELFTEST=1 pnpm start should print 11/11 passed. Event counts, seq values and the number of fork points are reproducible; the session id and each line's ts are not, so do not assert on them.

Acceptance is five ticks: the self-test prints 11/11 passed; the log persists line by line in the correct order after turn one; a different process with --resume cites last turn's finding; a truncated last line still recovers and reports the line number; and /fork lists no assistant message with tool calls, with the parent session untouched after forking.

Interview Questions

Today's three questions test persistence trade-offs, not "what is event sourcing":

1. For session persistence, do you choose an event log or a session snapshot? What does each cost?
2. How does an append-only log handle a last line cut short by a crash?
3. What is the difference between forking a new session from a historical step and rolling back to it?

Full bilingual prompts, analyses and key points are in this course's day-seven question bank. Question two looks smallest and discriminates most — it asks whether you have thought about failure modes, not whether you can write try/catch.

Checklist and Tomorrow

• I can name the session object's three costs and why an event log has only one failure mode
• I know why each line needs seq, ts and type, and why a session id carries a timestamp
• I can explain why only message events rebuild, and the rest are for display and audit
• I know why a bad line means stopping at the previous one, and why "helpfully repairing" a log is forbidden
• I can explain why forking copies to a new file and why seq is not renumbered
• I can name the positions that are not clean fork points and what breaks if you truncate there
• I can state the difference between forking and rolling back, and what the log and snapshots each cover

Tomorrow is D8, "Reference Injection: Parsing @ for Files, Directories, URLs and Images, and Reporting How Much Was Injected," starting week two. Over these seven days, every time it wanted to see a file it had to glob, then grep, then read across three or four turns — while you often knew which file mattered all along. Tomorrow builds @ references: files, directories and web addresses resolved straight into injected context, with the terminal explicitly printing how many characters and roughly how many tokens were injected. It leads week two because the next few days (project instructions, memory, compaction) all push things into the context, and @ references are the simplest and most controllable kind, so use it to establish the discipline that injection must be accounted for.