Runs — durable execution

A run is one execution of a graph from trigger to terminal node. It is a durable state machine: the checkpoint is the truth, and no process needs to stay alive for the run to continue.

Checkpoints#

Every step boundary writes a checkpoint into Postgres. If a worker dies, the lease on the step expires and the step re-runs from the checkpoint — no loss, no double-commit. Runs pause at human approvals for weeks without holding compute; the state is rows, not processes.

What is captured#

Every non-deterministic interaction is recorded in the ledger as it occurs:

• Model responses (with request/response hashes)
• Tool results
• Timestamps
• Retrieval hits
• Policy verdicts

This capture is what makes replay possible.

Deterministic replay#

Any run can be replayed step by step with byte-level fidelity from the recorded inputs. Replay is the primary artifact a model-risk officer uses to answer “under what exact conditions did this decision get made?”

Time-travel fork#

From any checkpoint, you can create a fork: change one or more variables (an input value, a policy version, a threshold) and re-run the branch. The fork stays linked to the original in the ledger with identical inputs except the changed variable. Both runs are on the record — the original is never altered.

RUN r_01J8F3T9K2 · 14 steps · ledger SEQ 0388–0417
● ● ● ● ● ● ● ● ● ● ● ● ● ●   ← replay: byte-fidelity to production
              ↑
           FORK from SEQ 0121 · amount 12,400 → 500
● ● ● ● ● ● ● ● ● ○ ○ ○ ○     ← alternate outcome: [ALLOWED, no gate]

Failure semantics#

• Infrastructure failure: the run resumes from the last checkpoint.
• Approval timeout: the gate escalates per its SLA configuration.
• Policy denial: the step is denied and recorded; the run continues per graph definition or halts — your choice at author time.

See also#

Every non-deterministic input captured here lands in the hash-chained ledger and is exportable as an evidence bundle. What agents learn across runs is governed separately — see Memory consolidation.