lazy documentationv0.22

Task State Machine and Crash Recovery

This document describes the task status state machine and the crash/resume lifecycle in Lazy.

Task Status State Machine

Status Definitions

Lazy tasks can be in one of the following statuses:

Status Categories

Terminal statuses (task is finished, core fields frozen):

Blocked statuses (waiting for human action):

Active statuses (has active worktree that should not be merged into):

Valid State Transitions

backlog
  → working       lazy start (creates session, launches agent)
  → abandoned     lazy close (task canceled before starting)

working
  → blocked       reconciler (agent turn completes, response.json processed)
                  | reconciler (supervisor reported an unrecoverable agent failure —
                    see "Agent Failure Classification" below; NOT auto-resumed)
  → conflict      reconciler (agent turn completes with file permission violations)
                  | lazy accept (pre-accept turn ends, task restored to its pre-accept status)
  → submitted     lazy accept (same restore, for a task that was submitted)
  → interrupted   reconciler (container dies without response.json)
  NOTE: working cannot transition to pairing or abandoned — the agent is running.
  NOTE: accept moves the task through `working` for the pre-accept turn and
        restores the status it ACTUALLY had — see the merging note below.

blocked
  → working       lazy unblock (human gives feedback)
  → conflict      any park that re-derives the label and finds a pending
                  violation set (see "conflict is DERIVED" above)
  → pairing       lazy pair (human wants to work interactively)
  → merging       lazy accept (begins merge process)
  → abandoned     lazy reject (rejects the work) | lazy close (canceled without accept/reject)
  → backlog       reconciler migration (blocked task with no session → backlog)
  NOTE: blocked cannot go directly to complete — must go through merging first.

conflict
  → working       lazy unblock (human gives feedback to fix violations)
  → blocked       any park that re-derives the label after every violation was
                  resolved (approved or rejected)
  → pairing       lazy pair (human wants to work interactively on violations)
  → abandoned     lazy reject | lazy close

interrupted
  → working       auto-resume (reconciler, circuit breaker allows) | lazy resume
                  | reconciler stale-completed-response sweep (recovers a response
                    written after the task was marked interrupted; routes through
                    working, then working → blocked)
  → pairing       lazy pair (human investigates interactively)
  → abandoned     lazy reject | lazy close
  NOTE: interrupted cannot go to merging or complete — must unblock/review first.

pairing
  → blocked       pairing ends (Claude Code exits) | reconciler stale pairing sweep
  → conflict      same two paths, when the task still owes a decision on
                  file-permission violations

merging
  → complete      lazy accept (checks pass, merge succeeds) | reconciler (merge completed)
  → blocked       lazy accept (checks fail) | reconciler (PR/MR closed externally)
  → conflict      lazy accept (merge phase aborts on a task that was in conflict)
  → submitted     lazy accept (merge phase aborts on a task that was submitted)
  NOTE: merging cannot go to abandoned — merge either succeeds or fails back to
        the status the task held before the accept. A task in `conflict`
        (unresolved violations) or `submitted` (open PR awaiting review) is not
        `blocked`; rewriting it to `blocked` on an aborted accept would destroy
        that signal, so every abort restores the TRUE prior status.
  NOTE: an accept that DIES mid-merge (daemon restart, crash, kill) leaves the
        task in `merging` with no owner. That is recovered, not transitioned
        away: `lazy reject`, `close`, `submit` and `unblock` return the task to a
        resting status first and then do their job, and the reconciler sweeps
        such tasks on every tick — so a stranded merge escapes without anyone
        needing to know an incantation. A merge that is genuinely in flight (an
        accept holding the task's lifecycle lock) is refused instead, and a
        forge-pending merge is left for remote-sync. See
        `src/daemon/stranded-merge.ts`; `lazy doctor` reports anything sitting
        in `merging`.

zombie (system-only)
  → complete      reconciler (merged branch sweep completes the task)
  NOTE: any non-terminal → zombie (system actor only). Terminal statuses cannot go to zombie.

complete
  → blocked       lazy reopen --reason "..." (reopen accepted task)
  → backlog       lazy reopen (reopen accepted task with no session)

abandoned
  → backlog       lazy reopen (reopen rejected/closed task with no agent work)
  → blocked       lazy reopen (reopen rejected/closed task with agent work)

Actors (turn & transition provenance)

Every turn and status transition records an actor — who caused it. This is provenance metadata: it never gates behavior (a save is a save regardless of actor), it only records who did the thing.

There are five actors (Actor in src/types):

The discriminator is the channel, not the content source. A command that arrives over MCP is builder; the same command over the CLI is human. This is deliberate: when the builder relays a human's feedback via lazy_unblock, the turn is still builder — the actor records who submitted (pressed the button), not who authored the words. The human's content is persisted as written either way (modulo control-character escaping, below); the tag is orthogonal to it.

Mechanically: CLI commands default to getActor() (env-var / human). MCP tool handlers set the actor at origination — MCP_ACTOR (builder) for the builder, AGENT_ACTOR (agent) when the tool context carries a task id — and thread it through the RPC layer, because lifecycle ops persist their records in the daemon — a shared process that can't see the caller's channel from its own environment, and whose getActor() therefore reports human for every caller. Everything an operation writes carries that one threaded actor: the turn, every status transition it makes, and any comment it leaves. That covers the turn-creating ops (start, unblock, ask, resume, stop, sync) and the ones that never launch an agent (reject, close, submit, reparent), plus task creation — whose channel lands on the initial backlog status-changelog entry — and journal entries. Read surfaces (lazy show, the web dashboard, the MCP lazy_show turns section, fidelity / report digests) label a human-role turn by its authoring actor so builder and supervisor turns are distinguishable from what a person typed.

An absent actor is not a bug — it means one of two things. On a role: 'agent' turn, an actor is never written: the field records who submitted a command, and an agent's own reply is not one (lazy show labels those by role). On a human-role turn or a status change, absent means the default channel — a CLI/human action, or a record written before the actor was populated. Only the non-default channels (builder, agent, system, supervisor) are stamped, so legacy records and new CLI records read the same way. Turn-creating commands are the exception: they always write an explicit actor on the human-role turn, human included.

Text intake is sanitized at the boundary

Every prompt lazy sends an agent is passed as an argv element (claude -p <prompt>). A raw NUL byte in argv is illegal, so a single NUL in feedback used to kill the spawn instantly, trip crash-loop detection, and — via an auto-resume that restarts on a generic prompt — silently drop the feedback.

Every text intake (CLI unblock/comment/create, the $EDITOR and piped-stdin paths, every MCP tool argument, the daemon's unblock/ask RPCs, and auto-delivered comment/CI text) therefore runs through sanitizeUserText() (src/utils/sanitize-text.ts) before persistence. Non-printable control characters — C0 except tab/LF/CR, DEL, and C1 — are replaced with their printable escapes (a NUL becomes the six characters \u0000).

For free-text arguments that become prose in a prompt (feedback, message, prompt, note, reason, question, goal_context), a short note is appended disclosing the substitution, so it is visible rather than a silent rewrite. Strings that are not prose are escaped without that note: short single-line fields like a task goal, and strings nested inside array arguments — files, approved_files — whose elements become git argv. Appending an explanatory paragraph to a file path would corrupt the path.

The policy is sanitize and deliver, never reject: rejecting would discard feedback at exactly the moment the human finished typing it, which the never-lose-human-feedback invariant forbids. Nothing is dropped — only re-encoded. Ordinary text is passed through byte-for-byte.

The delivery side is guarded too, as defense in depth: the argv builders escape (and log loudly) rather than fail, and the src/utils/spawn.ts wrappers refuse a NUL-bearing argv with an actionable error instead of the opaque args[N] must be a string without null bytes.

Transition Triggers

The CLI commands below are the human-channel triggers; each has an MCP equivalent (lazy_start, lazy_unblock, …) that drives the same transition but records the actor as builder (or agent, when the caller is a task agent acting inside its own subtree).

Human Actions (CLI commands)

Agent Actions

System Actions (reconciler)

The reconciler runs automatically before lazy list, lazy blocked, lazy active commands. It performs several sweeps:

  1. Working task sweep — Process tasks in 'working' status:

    • If response.json exists → record turn, transition to blocked
    • If container running and no response → leave as working
    • If container stopped without response → transition to interrupted, maybe auto-resume
    • If no container and no response → transition to interrupted, maybe auto-resume
  2. Interrupted response sweep — Process stale responses for interrupted tasks:

    • Race condition fix: supervisor may write response.json after reconciler marks task interrupted
    • If interrupted task has response.json → process it, transition to blocked
  3. Terminal container sweep — Clean up orphaned containers:

    • Tasks in complete/abandoned may have lingering containers if cleanup failed
    • Remove container, clear container_name from session
  4. Merged branch sweep — Detect zombie tasks (branch merged but status not updated):

    • For non-terminal tasks with sessions: check if branch merged into target
    • If merged → set outcome='accepted', transition to zombie (system only) → complete
    • Uses the zombie status as an intermediate state to maintain transition table integrity
    • Prevents orphaned state where lazy accept succeeded but crashed before updating task
  5. Stale pairing sweep — Recover tasks stuck in 'pairing':

    • If pairing PID no longer exists → transition to blocked
    • Cleans up pairing lock file
  6. Blocked-to-backlog migration — One-time migration:

    • If blocked task has no session → transition to backlog
    • Separates "unstarted" (backlog) from "waiting for review" (blocked)

Accept observability

An accept can run for minutes — a pre-accept agent turn, remote pushes, an LLM-written merge description, the merge itself. It used to print nothing for that whole window, which is indistinguishable from a hang. It now narrates.

The phase table is the single source of truth. src/daemon/progress.ts declares every accept phase (id + label + whether it is optional) and the two plans built from it: the fresh-accept plan (acceptPhasePlan, which includes the pre-accept turn only when that step is enabled) and the remote-merge re-entry plan (acceptReentryPhasePlan: check remote state → finalize → clean up). The accept path executes phases from that same table, so the announced plan cannot drift from what actually runs.

How narration travels. The daemon's PhaseReporter emits ProgressEvents into the heartbeat envelope described in src/daemon/heartbeat.ts — the same NDJSON framing that already carried liveness, extended with a {"progress": …} line rather than forked:

Line Meaning
{"lazyEnvelope":1} preamble; the response is framed
{"heartbeat":<ms>,"phase":"Merge"} still alive, currently in this phase
{"progress":{…}} a plan announcement, or a phase start/done/skipped/failed
{"status":…,"body":…} the result

A progress line also feeds the client's liveness callback: the daemon wrote it because its handler is running now, so a phase change resets any client deadline just as a heartbeat does.

Both client surfaces consume the same events:

Surface Rendering
lazy accept src/cli/phase-display.ts — plan header, then one line per phase; on a TTY the open phase repaints with a live elapsed counter, off a TTY it is append-only
MCP lazy_accept src/daemon/mcp-proxy.ts relays each event as notifications/progress

Narration is strictly observational: a throwing or absent emitter cannot change the outcome of an accept, and an accept with nobody listening runs identically.

Pre-flight is a prelude, not a plan entry. It runs before the plan is known (it is what decides which plan applies), so it is narrated with no [n/m] position and the plan is announced immediately after it.

Status during an accept. The narration answers "what is it doing" for the caller who is watching; the task's own status answers it for everyone else (lazy list, lazy show, MCP):

Accept phase Status Substate
Pre-accept validation turn working agent:pre-accept
Merge phase (description, merge, finalize) merging —
Aborted at any point the status held before the accept —

agent:pre-accept (src/utils/working-substate.ts, derived from the supervisor's command_type) exists because a bare working during an accept is indistinguishable from a human having unblocked the task by hand.

Waiting on subtasks

An agent that decomposes its work into subtasks drives them with blocking lazy tools — lazy_wait (long-poll until a subtask finishes its turn) and lazy_ask (resume a subtask's agent and wait for its answer). For that whole stretch the parent agent is doing nothing, yet every read surface showed it as working(agent): indistinguishable from an agent thinking hard for twenty minutes.

The signal is the call itself, not the agent's output. Every agent tool call — container runner and host-process runner alike — is forwarded by src/daemon/mcp-proxy.ts to the daemon's POST /mcp/:taskId/:toolName route and authenticates with a per-session MCP token. So handleMcpToolCall already knows which task is blocked, on what, and since when. Nothing here parses agent stdout.

Piece Where
Which tools park the caller BLOCKING_WAIT_TOOLS in src/daemon/wait-registry.ts (lazy_wait, lazy_ask)
Wrapping a blocking call trackWait(), called from handleMcpToolCall
Live marker readers see waiting.json in the task's protocol dir (src/protocol/waiting.ts)
Substate derivation + label src/utils/working-substate.ts ({ kind: 'waiting' })
Durable intervals Storage.recordWaitStart/recordWaitEnd/readWaitIntervals

Rendering. working(waiting on fix-foo (2m10s)) across lazy list, lazy active, lazy status, lazy show, lazy watch, and the MCP substate field — all of them go through the shared derivation, so they cannot drift. Two concurrent waits list both labels; a larger fan-out summarizes (waiting on a, b +2).

Precedence. Within an agent phase, waiting outranks agent:answering and agent:pre-accept: those say what the turn is, waiting says what it is doing this second. A harness phase outranks waiting — there the supervisor, not the agent, is the active thing. A dead run still reads not-alive; a wait marker never resurrects a stranded task.

Clearing is on settle, not on response delivery. trackWait clears from a finally, so a call whose MCP client already disconnected — which the daemon finishes anyway (complete-anyway semantics) — still clears. The marker also carries the writing daemon's pid: a reader that finds that pid dead treats the whole file as stale and reports no waits, so a SIGKILLed daemon degrades to the pre-existing working(agent) rather than to a lie. cleanProtocol removes the file at turn teardown, and a clean daemon stop clears every marker it owns.

Persisted wait intervals

The same bookkeeping writes a durable record, because waited time is not the agent's time: a turn that spent two hours blocked on a subtask must not bill those two hours to the agent in duration or economics reports.

Intervals live in <storagePath>/waits/intervals.jsonl — shared JSONL used by every backend (the same reasoning as trace spans: telemetry-shaped, append-only data earns no table and no migration, and the cross-backend row shape is byte-identical by construction).

The file is event-structured: a wait writes one start line when it begins and one end line when it settles, folded on read. That is what makes a crash readable — an interval whose end never arrived reads back with ended_at: null and outcome: null, the documented "died mid-wait" shape, rather than vanishing. Consumers should treat such an interval as open, bounded above by the turn's end.

turn_sequence is best-effort by construction: agent turns are only written when the turn ends, so at wait time the only thing available is the next unused sequence. A consumer that needs certainty should attribute by time overlap with the turn's own window and use the field as a hint.

Every write is wrapped in a catch: a wait that cannot be recorded still waits. An agent must never lose lazy_wait because an observability write failed.

Agent-reported progress

The working substate answers who is active — agent, harness, a wait, nothing. It cannot answer doing what: working(agent) looks identical for an agent reproducing a bug, running a migration, and re-reading the same file for the ninth time. lazy_update_progress is the complementary channel — the agent posts a short human-readable line and every surface that already renders the substate folds it in.

Ephemeral, latest-wins, never history. A progress line is worthless five minutes after it was written, so it never reaches Storage: permanent storage of a self-reported status blurb would be a second, worse turn log. Each call replaces the previous message; there is no history and there must not be one. What is durable about a turn (rationale, decisions, deferrals) belongs in the journal.

Piece Where
The tool lazy_update_progress (src/mcp/tools.ts) — agent-only, requires a task context
Daemon-side writer recordProgress() in src/daemon/progress-registry.ts
Live marker readers see progress.json in the task's protocol dir (src/protocol/progress.ts)
Substate decoration + label src/utils/working-substate.ts (WorkingSubstate.progress)

Who writes it. The daemon, for the same reason it writes waiting.json: every agent tool call authenticates with a per-session MCP token, so handleMcpToolCall already knows which task is reporting. Nothing parses agent stdout, and no client writes the marker directly.

Rendering. working(agent: running migration 3/7) across lazy list, lazy active, lazy status, lazy show, lazy watch, and the MCP substate field — one shared derivation, so the surfaces cannot drift. The line is whitespace-collapsed and capped at 120 characters at the write boundary (and again on read, so a foreign file cannot break a table cell). Over-length messages are truncated, never rejected, and the truncation is echoed back to the agent rather than being silent.

Precedence. The progress line decorates only the kinds where the agent is the active thing: agent, agent:answering, agent:pre-accept, and waiting on …. It is deliberately absent from harness:<phase> — that phase is the supervisor's work, and showing the agent's last line there would report a claim about a turn phase that has already ended — and from not-alive.

Clearing is structural, not remembered. writeCommand deletes progress.json, and every turn begins with a command, so a new turn always begins with no progress: a message from a finished turn cannot linger. Belt and braces around that: the substate only renders while a live status.json exists, cleanProtocol removes the file at turn teardown, a clean daemon stop clears every marker it owns, and the marker carries the writing process's pid so a reader that finds it dead disbelieves the file and reports no progress.

Bookkeeping never breaks the call it observes. Every write in the registry is wrapped in a catch. A progress post that cannot be recorded is a lost status line; losing a status line must never cost an agent its tool call, let alone its turn.

Agent Failure Classification

Not every agent failure deserves the same response. A rate limit clears on its own; a revoked credential never does. The supervisor therefore never inspects error text itself — it asks the agent what went wrong and acts on the class it gets back.

This is the runtime half of credential handling. The start-time half is the daemon credential gate (src/daemon/credential-gate.ts): a daemon with no credential at all never comes up in the first place, on any start path, so the fatal_auth class below is about credentials that were present at start and stopped working — not about a daemon that never had one.

The taxonomy

Each agent implements classifyFailure() (src/agent/interface.ts) and maps its own raw stderr / stdout error / exit code onto the shared vocabulary in src/agent/failure-taxonomy.ts:

Class Examples Supervisor behavior
fatal_auth 401/403, invalid API key, missing credential, credit balance exhausted Stop on the first failure
fatal_config unknown flag, invalid model, agent binary missing (exit 127) Stop on the first failure
transient_overload 429, 529, 503, "overloaded" Retry forever, 5s→60s
transient_network ECONNRESET, ETIMEDOUT, socket hang up, fetch failed Retry forever, 5s→60s
transient_unreachable ECONNREFUSED, ENOTFOUND Retry 5s→60s, bounded at 12 attempts (~9 min), then treated as unrecoverable
unknown anything unrecognized Retry 15s→60s; the crash-loop detector still applies

A classifier that does not recognize a failure returns unknown rather than guessing. A wrong fatal_* blocks a task that would have recovered — the more expensive mistake.

Agent-specific dialects (Claude Code's "Invalid API key · Please run /login", Cursor's "not logged in") are matched in that agent's implementation; shared HTTP/socket signals live in classifyCommonFailureSignals().

Retry pacing

src/supervisor/retry-policy.ts is a pure function of (class, attempt) and is the only place cadence is decided. Transients use 5s → 10s → 20s → 40s → 60s (capped) — roughly 60 retries an hour. The previous uniform ladder (30s → 300s) managed about 12, which in a live incident meant two attempts in five minutes on a condition that could never recover.

The fast-fail crash-loop detector (3 failures under 10s each) is disabled for transient classes: a 429 comes back in milliseconds, so three in a row would abort a turn that was about to succeed. It remains the bound for unknown.

transient_unreachable is the deliberate middle ground for the ConnectionRefused case: a refused connection to a local proxy can heal if the proxy restarts, so it is retried generously, but a proxy that never returns must not spin forever. After the attempt cap it escalates to a stop.

What "stop" means

When the policy says stop, the supervisor throws FatalAgentError and writes the class, reason, and attempt count into response.json (failure_class, failure_reason, failure_attempts). The reconciler treats a set, non-unknown failure_class as the signal to move the task to blocked — not interrupted — and records the classified reason as an interrupt. This matters because maybeAutoResume only ever fires on interrupted tasks; blocked is what actually stops the reconciler from relaunching into a dead condition, and puts the task in front of a human with the reason attached.

The crash-loop backstop reports, it does not judge

The fast-crash-loop detector (3 failures under 10s) throws CrashLoopError, which writes the same three fields. Its class is always unknown — the detector runs for no other class (see appliesFastFailDetection) — and unknown on the wire is diagnosis only: the task still goes to interrupted with auto-resume, because many crash-loop causes are transient. Before this, the backstop threw a bare Error and the whole classification was dropped, so the recorded turn said only Crash loop detected: … with no class or attempt count.

Agent classifiers must therefore keep unrecoverable conditions out of unknown. Cursor's ActionRequiredError (plan quota spent, spend limit needed) is fatal_auth for exactly this reason; Claude's usage limit reached is a self-healing 5-hour window and stays transient_overload. Both live in each agent's own classifier, never in the shared signals.

The Cursor verdict is narrower than the wording alone: quota phrasing is fatal only when nothing in the message says the wall clears by itself. A rate-limit marker, or a reset horizon stated as a short duration ("resets in 20 minutes"), falls through to the shared signals and stays transient_overload — a horizon stated as a date ("when your monthly cycle ends on 9/19/2026") does not, because no retry ladder outlives it. Ambiguity resolves toward retrying: a wrong fatal_* blocks work that would have finished.

Live retry state (class, reason, next delay) is also projected into status.json (retry_failure_class, retry_failure_reason, retry_next_delay_ms) for presentation surfaces to render.

Rendering "what is being retried"

phase=retrying on its own tells a human nothing — the question in a live stall is always retrying what, and why. src/utils/retry-summary.ts is the single formatting seam: formatRetrySummary() turns the status.json retry projection into one line (attempt 7 (transient_overload): API Error: 529 overloaded), truncating the error snippet to a bounded length. Every surface renders through it, so they cannot drift:

Surface Where
lazy watch / lazy status header src/cli/status-header.ts
lazy list / lazy active substate src/utils/working-substate.ts (harness variant carries retry)
lazy show "Retry State" block src/cli/commands/show.ts
Supervisor log (Phase: retrying …) src/supervisor/retry-status.ts
MCP lazy_show → retry_status src/mcp/tools.ts (buildRetryStatus)

The MCP field exists because a builder driving tasks over MCP has no host CLI: without it, a task stuck in the retry loop is indistinguishable from a healthy working one.

Surfaces that render both a header and a substate suffix (watch) or a status word and a detail block (show) strip retry from the secondary copy — the detail is printed once per line, not two or three times.

Watchdog kills are outside the taxonomy

A no-progress watchdog kill has no error text to classify — lazy killed the process itself, after [agent] watchdog_output_timeout_ms (default 30 minutes, and the timer resets on every completed step) passed with no forward progress. So it is not routed through classifyFailure(); it is decided by one question, in decideWatchdogRetry():

Did the turn capture anything? Behavior
Yes — a final result was on the wire, or the turn added commits Not retried. The work is already on disk, so relaunching would repeat it or wedge the same way; the turn ends and a human reads what was captured
No — no result, no new commits Relaunched in-turn with transient-style backoff (5s → 10s), bounded at 3 attempts. Each attempt costs a full no-progress window, so the bound is far tighter than UNREACHABLE_MAX_ATTEMPTS

The zero-work branch exists because the "work is already on disk" rationale is false in the case that hurts most: during a provider incident a task sat 45 minutes in working with its first model call hung — nothing captured, nothing to repeat — and a relaunch is exactly what heals it. If the commit probe cannot read git, the kill is treated as zero-work (fail toward retrying).

A watchdog kill carries no failure_class, so when it does end the turn the reconciler routes it to interrupted with bounded auto-resume, not blocked. The recorded turn and the session's interrupt_reason are rendered by src/utils/watchdog-turn.ts — one place, shared by work turns and the ask path — and say which guard fired, what its limit was, that keep-alives are not progress, and whether lazy already relaunched.

Crash/Resume Lifecycle

When an agent crashes mid-turn, Lazy automatically detects the failure and attempts to resume the task. This section describes the full lifecycle.

1. Detection

The reconciler runs on every lazy list, lazy blocked, lazy active invocation. It scans tasks in 'working' status and checks:

Grace period: Newly-working tasks (last_interaction_at within 30 seconds) are skipped to give the container time to start. This prevents a race where reconciliation runs before the container finishes launching.

If a working task has:

Then the task is marked as interrupted.

2. Classification: Clean vs Dirty Worktree

Once a crash is detected, the reconciler checks hasUncommittedChanges(worktreePath) to classify the crash:

This classification determines the auto-resume strategy.

3. Recording the Interrupt

The reconciler transitions the task to 'interrupted' and records diagnostics:

await storage.updateTaskStatus(taskId, 'interrupted', 'system');
await storage.recordInterrupt(session.id, {
  reason: exitCodeToReason(exitCode),  // e.g., "OOM killed or SIGKILL (exit code 137)"
  exit_code: exitCode,
  logs: runner.getRunLogs(containerName, 50),  // Last 50 lines
});

The recordInterrupt call increments the consecutive_interruptions counter, which is used by the circuit breaker.

4. Circuit Breaker Check

Before attempting auto-resume, the reconciler checks:

if (session.consecutive_interruptions >= MAX_CONSECUTIVE_INTERRUPTIONS) {
  // MAX_CONSECUTIVE_INTERRUPTIONS = 3
  logger.warn('Circuit breaker triggered, not auto-resuming');
  return;
}

If the circuit breaker fires, the task stays in 'interrupted' state until a human manually intervenes with lazy resume or lazy unblock.

Rationale: Prevents infinite crash loops. If an agent crashes 3 times in a row without completing a turn, something is fundamentally wrong (OOM, broken dependencies, infinite loop, etc.). A human needs to investigate.

5. Auto-Resume: Clean Worktree Path

If the worktree is clean (no uncommitted changes), the auto-resume flow is:

  1. Resolve parent branch (same logic as normal unblock):

    • For child tasks: parent's branch (lazy/<parent-ref>)
    • For root tasks: task.metadata.remote_target_branch or 'main'
  2. Merge upstream (INVARIANT: every unblock merges upstream):

    • Set sync_before_work: true in the UnblockCommand
    • Supervisor merges parent/main into task branch before agent resumes
    • Prevents branch drift (see CLAUDE.md architectural invariants)
  3. Build crash context prompt:

    You are being resumed after a crash. Upstream has been merged into your branch
    since your last turn. Don't assume your previous state is intact — verify before
    continuing.
    
  4. Write UnblockCommand to protocol directory with:

    • parent_branch set (for upstream merge)
    • sync_before_work: true
    • Crash context prepended to the resume prompt
  5. Record synthetic human turn:

    await storage.createTurn({
      sessionId: session.id,
      sequence: nextSeq,
      role: 'human',
      content: '[system] Session interrupted and auto-resumed',
      actor: 'system',
    });
    
  6. Transition to working:

    await storage.setAutoResumed(session.id, true);
    await storage.updateTaskStatus(task.id, 'working', 'system');
    
  7. Launch supervisor (or write command if already running):

    • If container exists: just write UnblockCommand, supervisor picks it up
    • If container gone: launch new supervisor container

The agent resumes with full Claude Code /resume context (conversation history), but filesystem state reflects the crash — it must verify assumptions before continuing.

6. Auto-Resume: Dirty Worktree Path

If the worktree has uncommitted changes, the auto-resume flow is:

  1. Skip upstream merge:

    • Set sync_before_work: false
    • Merging upstream on a dirty worktree would fail (git merge refuses) or create confusing state (stashed changes, lost edits)
  2. Build crash context prompt:

    You are being resumed after a crash. There are uncommitted changes in your worktree
    from your interrupted turn. Review them, decide what to keep, commit or discard,
    then continue your work.
    
  3. Write UnblockCommand with:

    • No parent_branch (skip merge)
    • sync_before_work: false
    • Crash context prepended to resume prompt
  4. Record synthetic human turn, transition to working, launch supervisor** (same as clean path)

The agent resumes and sees uncommitted changes. It must decide what to do with them (commit, discard, edit further) before making progress.

7. Session Resume and Conversation Context

When the supervisor launches, it:

  1. Looks for the Claude session ID in the sandbox's .claude/projects/ directory
  2. Passes agent_session_id to the capture layer
  3. Claude Code's /resume functionality restores the full conversation history

Filesystem state depends on crash timing:

The conversation transcript is preserved, but the agent must verify filesystem state — the crash may have interrupted file writes, git operations, or tool calls.

7b. Feedback Redelivery (never lose human feedback)

INVARIANT (CLAUDE.md): a turn whose feedback was recorded but never consumed is re-delivered verbatim when the task resumes — for any crash cause.

The gap this closes: feedback is persisted before the container launches (save first, act second). If the work phase then crashes, the feedback exists in the store but the agent never read it. Resuming with the generic "your previous session was interrupted, carry on" prompt leaves that feedback available only implicitly via turn-history injection — and in practice the agent does not act on it.

Tracking. Delivery state lives explicitly on the turn (Turn.feedback_delivery):

Value Meaning
absent The turn carries no redeliverable feedback (system resume notices, supervisor sync/nudge turns, stop reasons). Never triggers redelivery.
pending Feedback is persisted but no agent response has consumed it.
consumed An agent response completed after this feedback was delivered.

Turns created with carriesFeedback: true start as pending: the initial task prompt (task-launcher), lazy unblock, lazy ask, and auto-delivered comments/CI output (auto-deliver — actor system, but the content is human).

Why an explicit marker and not "is there an agent turn after it?" Because a crash records an agent error turn. That turn consumed nothing, so the positional proxy would mask exactly the case redelivery exists for.

Marking consumed. storage.markFeedbackConsumed(sessionId) flips every pending turn in the session to consumed. It is called wherever an agent response completes normally — handleCompletedResponse, the stranded-session recovery, and the ask recorder — and deliberately not in the error paths. Clearing the whole backlog at once is what makes redelivery idempotent: a turn that did consume its feedback can never be re-delivered into.

Redelivering. Both resume paths (autoResumeTask and lazy resume) call findPendingFeedback(); when it returns a turn, the redelivery block replaces the generic resume context in the prompt. The newest pending turn is reproduced verbatim (never summarized or truncated); if older pending turns exist, the prompt says how many so a queue is never silently collapsed. The synthetic resume turn itself does not carry feedback, so a crash during the resume re-delivers the same feedback again.

Rows written before this field existed read as "carries no feedback" — the correct reading, since resurrecting them now would be a stale redelivery.

8. Three Crash Scenarios in Detail

Scenario A: Crash Before Any Edits

Agent starts turn → reads files → analyzes → CRASH

State:

Auto-resume flow:

Scenario B: Crash Mid-Edit with Uncommitted Changes

Agent starts turn → edits file A → edits file B → CRASH (file B half-written)

State:

Auto-resume flow:

Scenario C: Crash After Commit But Before response.json

Agent starts turn → edits files → commits → CRASH (before writing response.json)

State:

Auto-resume flow:

Reconciler behavior: On next successful turn, the reconciler will detect the "new" commit (it compares session.git_start_sha with current HEAD) and record it.

9. Special Cases and Edge Cases

Merge-and-Fix Failures

If the supervisor crashes during the merge_and_fix phase (upstream merge failed with conflicts), auto-resume is disabled:

if (response.phase === 'merge_and_fix') {
  logger.warn('merge-and-fix failed, not auto-resuming (task needs human investigation)');
  // Task stays interrupted
}

Rationale: The task cannot make progress without a successful upstream merge. Auto-resuming would start a new turn on a stale branch, diverging further from upstream. A human must investigate the merge conflict.

Error Responses

The supervisor can write an error response.json (instead of a completed response):

{
  "status": "error",
  "phase": "capture" | "merge_and_fix" | "supervisor",
  "error": "...",
  "exit_code": 137,
  "stderr": "..."
}

When the reconciler sees an error response:

  1. Record agent error turn (visible in lazy show)
  2. Transition to interrupted
  3. Record interrupt diagnostics
  4. Auto-resume if allowed (unless merge-and-fix failure)

This gives visibility into crash details while still attempting recovery.

Interrupted Response Sweep (Race Condition Fix)

Problem: The reconciler moves a task to 'interrupted' due to a supervisor error, but the supervisor may have already picked up the next command (written by lazy resume or lazy unblock) and completed it. The new response.json sits unconsumed because the reconciler only looked at working tasks.

Solution: After the primary working task sweep, the reconciler runs a secondary sweep over interrupted tasks:

const interruptedTasks = await storage.listTasksWithOptions({ interruptedOnly: true });
for (const task of interruptedTasks) {
  const response = readResponse(protoDir);
  if (response?.status === 'completed') {
    // Stale response found — process it, transition to blocked
    await handleCompletedResponse(...);
  }
}

This ensures responses are never lost, even when races occur between reconciler and supervisor.

Consecutive Interruptions Counter Reset

The consecutive_interruptions counter is reset to 0 when:

This means the circuit breaker is lenient: if the agent succeeds once, the counter resets. Only sustained consecutive crashes trigger the breaker.

10. Manual Recovery

If auto-resume fails or the circuit breaker fires, humans can recover manually:

Interrupt diagnostics are stored in the session:

interface Session {
  interrupt_reason: string | null;        // "OOM killed or SIGKILL (exit code 137)"
  interrupt_exit_code: number | null;     // 137
  interrupt_at: number | null;            // timestamp
  interrupt_logs: string | null;          // last 50 lines of container logs
  consecutive_interruptions: number;      // circuit breaker counter
  auto_resumed: boolean;                  // was this session auto-resumed?
}

This gives humans full visibility into what went wrong.

Code References

Key files implementing the state machine and crash recovery:

Testing

State machine behavior is tested in:

When modifying state transitions, ensure tests cover: