ADR-033 — Persistence & State: checkpoints, hydration and recovery¶
| Field | Value |
|---|---|
| Status | Accepted |
| Version | v.5 |
| Date | 2026-08-06 |
| Owner | Ruslan Gabitov |
| Refines | SAD-001 v.1 §10 (save/restore as P0: "goroutines are the execution medium, persistence is the state of record"), ADR-001 v.6 (the runtime whose state this makes durable), ADR-007 v.2 (the in-memory long-wait model — dehydration & wake-on-trigger — whose durable half this owns) |
| Related | ADR-006 v.4 (subscriptions), ADR-013 v.2 (facts vs state), ADR-014 v.1 / ADR-016 v.1 (correlation state), ADR-017 v.1 (the loop as sole state owner), ADR-021 v.1 (the job queue's own durability), ADR-023 v.3 (scopes, child instances), ADR-025 v.2 (iteration state), ADR-026 v.1 (the completion ledger) |
This ADR is the Persistence & State ADR that ADR-001, ADR-007 and the Repository skeleton have been deferring to. It decides what a Process Instance's durable state is, when it is written, how an instance dehydrates (long waits release their goroutines), how it hydrates (restart recovery, wake-on-trigger), and what suspend/resume means on top — and what keeps several engines sharing one store safe. Implementation rides the accompanying SRDs, sliced smallest-first: checkpoint/save/restore, then hydration, then the operator surface (cluster-safe locking rides the same slices — the lease is part of the record from the first one).
1. Context & problem¶
- The standard prescribes the state machines, not the storage. BPMN
2.0 defines the observable lifecycles — the activity lifecycle
(§13.3.2, the vendored extract
docs/bpmn-spec/state-machines/activity-lifecycle.md: "State transitions are observable points where engine MUST persist state, emit events, and apply data assignments") and the process lifecycle (process-lifecycle.md). It is silent on persistence mechanics — the storage model is an engine concern, but the transition points are normative: they are where persisted state must be consistent. - The engine's own commitments. SAD-001 §10: save/restore is P0; checkpoints align with lifecycle transitions; on restart the runtime queries the Repository for in-flight instances and rehydrates — "recovery should be straightforward and bounded, not a fragile dance". ADR-001's invariant (relocated into ADR-007 §3): a track's continuation state is fully described by position, track/step state, Scope data and lineage — no hidden state on a token object; per-node resumable state is reached through a per-node state contract, never stored on the shared, immutable node definitions.
- What exists today. The Repository extension is a deliberate
skeleton (save/load/delete/list-in-flight over an opaque record)
whose package doc defers the durable contract here. Long waits park
a goroutine on a channel (ADR-017's model) — cheap, but nothing
survives a restart, and a UserTask waiting three days still holds
its goroutine. Suspend/resume does not exist; the observability
taxonomy reserves
Paused/Resumedslots for it (ADR-013 v.2). - The state to capture has grown. Since the runtime core landed, an instance's in-flight state spans: the scope tree's data (ADR-010 v.2 / ADR-011 v.7), the track table, event subscriptions and armed handlers (ADR-006, ADR-023), conversation keys (ADR-016), iteration state (ADR-025), the completion ledger with its data snapshots (ADR-026), parked human tasks and enqueued jobs (ADR-020/021). Deciding which of these is state-of-record and which is derivable is the heart of this ADR.
2. Decision¶
2.1 The checkpoint document — minimal state, derived arming¶
An instance's durable state is one checkpoint document, written atomically per instance, containing only what cannot be re-derived:
- Identity & pin: instance id, the process reference version- pinned (latest-at-launch resolution happened at start — recovery re-clones the SAME registered version; a missing version fails recovery loud).
- Status: the persisted lifecycle status —
Active,Suspended, or a terminalCompleted/Terminated(the Repository's status set gainsSuspended). - The scope tree's data: every open scope's committed data (path, datum name, value, state), parent links — the ADR-011 data plane at its last consistent commit. Values serialize through the engine's canonical value model (single values, arrays, records, maps — ADR-011 v.7); an unserializable payload is a loud checkpoint error, not a silent skip.
- The track table: per live track — position (node id), the track/step state, lineage (fork parentage), and its scope path. Tokens are projections (ADR-001) — persisting the track table IS persisting the tokens.
- Wait descriptors — the per-node state contract of ADR-007 §3, realized as serializable descriptors: a timer's absolute deadline and cycle position; a parked human task's id and announced/taken state; a message/signal wait's subscription key and correlation keys (ADR-016's conversation key-set); an event gateway's armed alternatives; partially-complete iteration state (ADR-025's counters and collected outputs — for both driving strategies: a sequential pass's position and collected outputs no less than a parallel group's open set, §2.10).
- The completion ledger (ADR-026): entries with their ordinals and data snapshots — compensability must survive a restart. When a compensation throw is resolving, the ledger alone is not the state: the document also records the sweep position — the remaining queue and the entry being undone (§2.9) — because the sweep consumes entries out of the ledger as it runs.
- In-flight call linkage: a parked Call Activity's descriptor — the awaited child instance id and the call node. The child is its own checkpoint record carrying the reverse link (parent instance id + call node id); §2.10 makes the pair restorable.
Everything else is derived at hydration by re-walking the re-cloned graph: armed boundaries and event-sub-processes, routing tables, subscriptions re-registered from the wait descriptors, condition re-evaluation wiring. Node definitions are shared and immutable — they are never serialized; the registered process (at the pinned version) is the schema, the checkpoint is the data. The document carries a schema version for forward migration.
2.2 Checkpoint policy — the loop writes, transitions gate¶
- The loop is the single writer. All instance state mutates on the instance's event loop (ADR-001/ADR-017), so a checkpoint taken by the loop between event applications is a consistent cut by construction — no locks, no torn state.
- Checkpoints align with observable lifecycle transitions (the
normative persist points of §13.3.2): instance created / completed /
terminated / failed; a node reaching
Completed(its data outputs committed); a track parking on a long wait; a scope opening/closing; a data commit outside those (a standaloneDataChange) rides the next transition. Mid-step state (a half-executed service call) is never checkpointed — a step is atomic between transitions, and recovery re-enters the node, not the half-step. Re-enter applies to steps, not to composite constructs: a construct whose position is recorded (§2.1 items 5–7) re-enters only what the recorded position says remains — never the whole construct from the top (§2.10). - The write mode is a policy seam with a safe default: synchronous write-through on terminal and wait transitions (the states that must never be lost), write-behind batching allowed for intermediate progress transitions. The embedder tunes durability vs throughput without touching the engine; the zero-config in-memory default makes every checkpoint a cheap in-process store.
2.3 Effects are at-least-once; state is the record¶
A checkpoint and the effects around it (outbound messages, worker jobs, observer facts) cannot be atomic across seams, so the model is explicit:
- State (checkpoints) is exactly-once by construction — one writer, one record, replace-on-write with a version check (a stale overwrite is a loud error, the split-brain guard).
- Effects are at-least-once. Recovery may re-emit an outbound message or re-announce a task whose pre-crash send raced the checkpoint. The receiving seams already tolerate this: message correlation dedups by key (ADR-016), the job queue reconciles by lock state (ADR-021 owns its own durability and reclaim), the task distributor re-announces parked tasks idempotently (the announce/take model of ADR-020 v.1).
- Facts are observability, not state (ADR-013): the observer stream is never replayed from storage and never read back to rebuild state; recovery emits fresh facts about what recovery did.
2.4 Dehydration — the durable half of the long-wait model¶
ADR-007's in-memory seed (subscription registered → goroutine ends → fresh track on trigger) becomes the memory projection of the same model this ADR makes durable:
- A track reaching a long wait writes its wait descriptor into the checkpoint (§2.2's wait transition), registers the in-memory subscription, and its goroutine ends. The instance keeps bookkeeping only; with every track dehydrated the whole instance may release its loop until a trigger arrives (idle instances cost memory, not goroutines).
- The trigger path is unchanged for a resident instance (ADR-006 delivery); for a released instance the trigger hydrates first (§2.5), then delivers.
- ADR-007 is finalized (Draft → Accepted) by the hydration SRD as the in-memory statement of this section — one model, two residency levels.
2.5 Hydration & recovery¶
- Wake-on-trigger: an arriving trigger for a non-resident instance loads the checkpoint, re-clones the pinned process version, replays the derivation walk (arming, routing), respawns tracks at their recorded positions, then delivers the trigger through the normal path. Hydration is the same code path recovery uses for one instance.
- Restart recovery: the runtime lists in-flight instances and hydrates each — eagerly for instances with due work, lazily (on-trigger) otherwise; the policy is an engine option, the default eager-and-bounded.
- Recovery semantics per wait kind: an overdue timer fires
once, immediately (missed cycle repetitions collapse into one firing
— recovering yesterday's every-5-minutes timer must not fire 288
times); a parked human task is re-announced through the
distributor (a
Takentask keeps its taken state; completion after recovery follows the normal path); a message/signal wait re-registers its subscription — messages that arrived while down were never accepted, senders retry (at-least-once, §2.3); an in-flight job is the dispatcher's concern — its queue survives by its own contract (ADR-021), and the engine's recovered node waits for the job outcome exactly as it did before the restart. - Recovery failures are loud and per-instance: one corrupt checkpoint or missing process version marks THAT instance failed-to-recover (observable, operator-visible) and never blocks the rest.
2.6 Suspend & resume — the operator surface on top¶
- Suspend (per instance): stop dispatching new steps; in-flight
steps run to their next transition (quiesce, bounded by the step
granularity of §2.2); checkpoint; mark
Suspended; release the runtime state (a suspended instance is a dehydrated instance with a status that refuses triggers). Triggers arriving while suspended are refused to the sender's retry (messages) or held by their seams (timers re-arm on resume relative to their absolute deadlines; human tasks stay taken/announced but completion is refused until resume). - Resume: hydrate (§2.5) + status back to
Active+ deliver whatever is due (overdue timers fire once, §2.5). - Engine-level pause (the whole runtime) is the same quiesce
applied to all resident instances; it fills the reserved
Paused/Resumedobservability slots (ADR-013 v.2). Suspension is state (it survives a restart); pause is runtime posture (it does not).
2.7 The Repository is one port among peers — the storage composition rule¶
The Repository is the engine's instance-checkpoint port, nothing more — one narrow port among peers, not the system's persistence facade. The rule generalizes across every storage-backed module:
- One port per consumer. Each subsystem defines its own domain-shaped storage port: the engine's checkpoint store (this Repository), the dispatcher's job store when it goes durable (ADR-021 owns its queue), the DataStore port, an AuthN/Z plugin's own store, a future history sink. No port serves two masters, so no port ever needs generic query/DDL surface — a "universal Repository" would degenerate into either an unqueryable blob store or a homemade data-definition language (§4).
- The shared thing is the backend handle, and it is the user's.
Storage-backed modules accept their backend handle at construction
(a
database/sqlhandle, a driver pool, a document-store client) — the stdlib and driver ecosystems ARE the "db connector", the engine never wraps or owns one. One handle, created by the embedder, feeds the engine's checkpoint store and any plugin's store alike — the existing composition rule promoted from adapters to all storage users (ADR-003 §4.4: no cross-module imports; the user composes shared resources at construction time). - Namespaced schemas, per-module migrations. Every storage-backed module owns a namespaced slice of the shared database (a schema or table prefix) with its own embedded migrations — DDL and dialect live inside the module's storage implementation, where they are legitimate; the engine core stays storage-blind (the SAD-001 v.1 §3 G2 dependency posture — the zero-config engine keeps its in-memory defaults with no database anywhere).
-
The
Migratorcapability convention (optional, one method: prepare your own objects) lets bootstrap walk the wired extensions and have each create its schema — the disciplined form of "every module manages its own db objects"; never a core DDL framework. -
Tenancy is recorded from day one — the tenant-linkage principle. Tenants partition data ownership, orthogonally to §2.8's engine groups (which partition work): every instance — and, through its checkpoint, all its data — belongs to exactly one tenant, while one engine MAY serve many tenants. An instance with no tenant configured belongs to the default tenant, so single-tenant embedders stay configuration-free. The default is designated, not named: the tenant registry marks exactly one tenant per engine group as the default (a flag under a uniqueness guarantee), never a reserved id — a magic id could collide with an operator-chosen one. Storage-backed modules keep a tenant registry (a tenants table in their namespaced schema) that instance records reference — referential integrity today, the attachment point for tenant metadata tomorrow. This layer guarantees only that the linkage is durable from the first schema version (retrofitting a tenant column into a shipped adapter is a breaking migration); tenant-scoped APIs, enforcement/isolation rules and the tenant lifecycle are the future Multi-tenancy ADR's territory (§5).
The skeleton itself grows only what this model needs: the Suspended
status; a record version for compare-and-set saves (the §2.3
split-brain guard) plus the §2.8 ownership lease, the engine's
group (§2.8 — listing is group-scoped) and the instance's
tenant on the record (the engine stamps the default tenant until
the Multi-tenancy ADR wires the real assignment); the checkpoint
document as an opaque, schema-versioned byte payload (the
serialization model is the engine's, the storage's job is bytes);
listing filtered by status and ownership. History stores, inboxes and
pagination stay out — a future history ADR's territory.
2.8 Many engines, one store — cluster-safe locking¶
Several engine processes MAY share one database. This layer owns the correctness of that sharing; the distribution of work stays with the Distribution & Scale ADR (§5). The model:
- Engines are partitioned into named groups. The group is part of the engine's configured identity (engine id + group name), not of the storage wiring: every record an engine creates carries its group, and the recovery listing and wake-on-trigger claims are group-scoped — an engine never sees, let alone claims, another group's instances. A cluster is therefore the set of engines sharing one store under one group name; several clusters MAY share one database without interference. An engine with no configured group forms a single-engine group under its own engine id — clustering is explicit, never accidental: only engines deliberately sharing a group name form a cluster, and accidental co-tenancy of one database is inert (no cross-claims). The engine id is therefore a stable, operator-chosen identity — which it already is: it names the lease owner. The zero-config posture stays configuration-free: a restarted engine, keeping its id, recovers its own instances without naming a group. The store keeps a group registry alongside the records: the first engine of a group establishes it (idempotently), records reference established groups only, and an engine MAY assert membership in an existing group — a join asserted against an absent group fails loud at startup, turning the typo that would otherwise silently mint a fresh partition into a refusal. Deployment parity (below) becomes a per-group contract — parity is owed within a group, never across groups.
- Instance ownership is a lease. An engine claims an instance before running it (at start, hydration, or wake-on-trigger): a per-instance lease record — owner engine id + incarnation, expiry — written with the same CAS discipline as the checkpoint. Lease renewal rides the instance's checkpoint writes plus a low-frequency heartbeat for long-quiet instances; completion, dehydration and suspension release the lease.
- CAS is the fencing token. Every checkpoint save carries the record version AND the owner's lease incarnation; a save from an engine whose lease expired or was reclaimed fails loud — a zombie engine (paused VM, network partition survivor) can never overwrite the new owner's state. The failed engine drops the instance and re-reads (its local state is disposable — the store is the record, §2.3).
- Claim-first wake semantics. Recovery listing returns unowned or lease-expired instances only; wake-on-trigger claims before hydrating — the first claimer wins, losers drop the attempt and the trigger's at-least-once seam (message retry, timer re-check, task re-announce) routes it to whoever owns the instance. Overdue-timer scans in a fleet are therefore naturally deduplicated: firing requires the claim.
- Orphan recovery is lease expiry. A crashed engine's instances become claimable when their leases lapse — no coordinator, no membership protocol; the store's CAS is the only synchronization primitive this layer needs. Lease reclaim is observable (an operator-visible fact naming both engines).
- Deployment parity is the operator's contract. A checkpoint pins its process version (§2.1); an engine can only claim instances whose pinned version it has registered — a claim against an unregistered version is refused loud. Distributing definitions themselves is the Distribution & Scale ADR's territory.
2.9 Observability¶
Recovery and residency are operator-relevant, low-volume milestones: hydration/dehydration and suspend/resume/recovery emit instance-scoped facts at lifecycle volume (never per-checkpoint — a checkpoint accompanies an already-observable transition; a per-checkpoint fact would double the stream for zero information). The exact kinds/phases/details ride the SRDs under ADR-013 v.2's open taxonomy and masking rules (names and counts, never payload values).
2.10 Composite constructs capture faithfully — child instances are durable¶
The §2.1 document is only as good as its coverage: a construct whose runtime position lives outside the re-cloned graph and outside the document restores wrongly by omission — the instance returns looking healthy and re-executes or abandons work, which is worse than a refusal because nothing signals it. Hence the rule set:
- Faithful capture is the norm. Every composite construct the engine runs — a composite scope, sequential and parallel multi-instance iteration, a Standard Loop, a resolving compensation sweep, an in-flight Call Activity — records its position in the checkpoint document (§2.1 items 5–7), and hydration rebuilds the construct at that position: a half-complete iteration resumes at its next pass with its collected outputs, a mid-sweep compensation continues with the remaining queue, a composite body whose child scope had drained resumes its host exactly once — restore must never double-execute completed work. An early deferral posture ("defer the checkpoint while a construct is in flight") was an implementation stopgap, never this contract: it silently widens the loss window and does not compose (a construct-dense process might never find a capturable instant). It is retired; a checkpoint refusal remains legitimate only for genuinely unserializable data (§2.1 item 3), and it stays loud.
- Child instances are durable, symmetrically linked. Every instance the engine runs — root or called child — checkpoints under the same repository, the same engine group and the same discipline; a child is never a volatile appendage of its parent. The parent's record carries the call descriptor (§2.1 item 7), the child's record carries the reverse linkage, and recovery restores both ends: the parent re-establishes its completion watch on the recovered child, and the child's terminal outcome reaches the restored parent exactly as it would a resident one. The cancel cascade (a caller's termination terminates the child) survives the restart with the re-link.
- A missing counterpart fails loud — a child instance is state, not an effect. The at-least-once posture (§2.3) covers effects; a child instance is recorded state, and duplicating one is designed out. A restored parent whose awaited child record does not exist — or a child whose parent vanished — fails its restore loudly (the per-instance recovery failure of §2.5), never silently re-launches the call.
- A call tree recovers as a unit — recovery affinity (v.5). A parent and the children it awaits are one recoverable unit: an engine that claims a parent claims that parent's recorded call tree in the SAME sweep, transitively, and a child is never revived on its own — recovery reaches a child only through its caller's claim. Two engines of one group can therefore never split a call pair between them — the split that would leave a correctly-recovered parent unable to re-link a correctly-recovered child, failing a restore that nothing is actually wrong with. Affinity is a claiming rule, not a distribution policy: which engine gets the tree is still whoever reaches it first (§2.8's claim-first semantics, the group scoping unchanged); only the granularity of a claim grows from an instance to a call tree. A child whose lease is genuinely held elsewhere (a live engine still running it) is not claimable and the parent's recovery fails loud as above — affinity removes the self-inflicted split, not the legitimate conflict.
A child whose caller is terminal is therefore not revived either: a parent completes only after its call returns, and a terminating parent terminates its children, so a terminal caller with a live child is an interrupted cancel cascade (§2.7's restart contract read through ADR-023's cascade). Recovery finishes the cascade — it writes the child's terminal record and reports it — rather than reviving an instance whose outcome has no consumer, or leaving a permanent resident of every future listing. Never silent, either way.
Not decided here: re-linking a parent and child that live on DIFFERENT engines — a remote child handle, cross-engine completion delivery and cancel cascade. That is a distribution concern (§5) and needs its own decision; affinity's contract is that a healthy group never needs it.
3. Grounding¶
| Claim | Source |
|---|---|
| Transitions are the normative persist points ("engine MUST persist state" at transitions) | the vendored extract, state-machines/activity-lifecycle.md (BPMN §13.3.2, Fig. 13.2) |
| The process-level lifecycle recovery re-enters | state-machines/process-lifecycle.md |
| Save/restore is P0; checkpoint-at-transition; bounded recovery | SAD-001 v.1 §10 |
| Track state = position + state + scope data + lineage; per-node state contract; immutable shared definitions | ADR-007 v.2 §3 (relocated from ADR-001) |
| The loop as sole owner of instance state (the consistent-cut premise) | ADR-001 v.6, ADR-017 v.1 |
| Correlation keys / dedup-by-key on redelivery | ADR-014 v.1, ADR-016 v.1 |
| The job queue's own durability, lock reclaim | ADR-021 v.1 §2.4/§2.7 |
| The completion ledger must survive (compensability of completed work) | ADR-026 v.1 §2.7 |
| The BPMN standard is silent on storage mechanics | no clause governs persistence in §13–§14; the extract covers only the state machines |
| Instance-level distribution rides sticky ownership + persistence rehydration | SAD-001 v.1 §13 (preliminary; the future Distribution & Scale ADR's home) |
| Composition at construction; no cross-module imports | ADR-003 §4.4 (the depguard-enforced rule §2.7 promotes) |
4. Alternatives considered¶
| Alternative | Pros | Cons | Decision |
|---|---|---|---|
| A. Event sourcing — append-only fact/command log, state rebuilt by replay | perfect audit; natural history | replay demands deterministic re-execution, which service tasks, scripts and external calls break; unbounded recovery time; the observer stream would become load-bearing state, inverting ADR-013 | ❌ rejected — checkpoints are bounded and "not a fragile dance" (SAD §10); the fact stream stays observability |
| B. Persist everything incl. derived arming/routing | hydration = pure load, no derivation walk | duplicates what the immutable graph already encodes; every arming change becomes a schema migration; bigger documents, more drift surface | ❌ rejected — minimal state + derivation keeps the checkpoint stable across engine evolution |
| C. Goroutine release first, durability later (pure ADR-007 now) | quick memory win for huge waiting populations | rewires the same wait bookkeeping twice; invents an ephemeral wait-state shape the durable model would redefine; resume path never exercises recovery | ❌ rejected — this ordering discussion is what triggered this ADR; the wait descriptor is designed once, durably |
| D. A universal Repository serving every module's persistence (engine state, authz, jobs, history through one interface) | one seam to learn | irreconcilable data shapes force generic query/DDL surface — a homemade data-definition language and de-facto ORM; every consumer's schema churn becomes interface churn | ❌ rejected — one narrow port per consumer (§2.7) |
| E. A shared db-driver seam in core (modules speak SQL through an engine-owned connector) | full query power per module; one connection story | picks a storage paradigm for everyone (non-SQL backends become second-class); the zero-config in-memory engine becomes the special case; core inherits a dialect against the SAD G2 posture; and the ecosystem already ships the connector (database/sql + drivers) |
❌ rejected — the handle is the user's, shared at construction (§2.7) |
| F. Checkpoint snapshot + derived arming + per-node wait descriptors (chosen) | one writer, consistent cuts for free; bounded recovery; ADR-007 becomes a projection | requires the serialization discipline of §2.1 (canonical values, loud on unserializable) | ✅ chosen |
5. Deferrals¶
- Work distribution (sticky routing, rebalancing, cluster-wide signal broadcast and correlation indexes, definition distribution) — the Distribution & Scale ADR (SAD §13), on top of this layer's state of record and §2.8 locking. The boundary: THIS ADR makes engines sharing a store safe; that ADR makes them cooperative.
- History / audit store (queryable execution history beyond the live checkpoint) — a future ADR; the observer stream remains the live audit today.
- Cross-version instance migration (#95) — the checkpoint pins its process version; migrating a pinned instance is its own workstream.
- Multi-tenancy (tenant-scoped APIs, enforcement/isolation rules, tenant lifecycle and how a tenant is assigned at registration/launch) — a future ADR; this layer records the instance→tenant linkage (§2.7) so the storage never needs a tenancy retrofit.
- Queryable incident store (#80 → the ops-console epic) — incidents are already durable per instance: they ride the checkpoint document and recovery, with an incident-holding persisted status, since the incidents landing. What remains deferred is the cross-instance, queryable surface — enumerating incidents without decoding every checkpoint payload, and incident visibility outliving the instance record — which needs its own extraction/index design in the storage adapters (per §2.7's composition rule).
- Ledger nesting fidelity — a completed child scope's ledger is folded into its parent's entry (ADR-026 v.1 §2.1); the checkpoint flattens the fold into ordinal-ordered siblings, which preserves the reverse completion order the sweep runs by (§2.10 restores correctly) but loses the nesting and the display names. Cosmetic; restored when a consumer needs it.
The accompanying SRDs (smallest-first): the checkpoint document + save/restore + restart recovery over the Repository seam; then dehydration/hydration (goroutine release, wake-on-trigger, ADR-007 finalized); then suspend/resume and the engine pause.
Document History¶
| Version | Date | Author | Change |
|---|---|---|---|
| v.5 | 2026-08-09 | Ruslan Gabitov | Accepted (landed via SRD-087: the unconditional child rule, the transitive claim, the finished cascade; make ci green, diff-coverage 96.3%). §2.10 gains recovery affinity: a parent and the children it awaits are ONE recoverable unit, so an engine claiming a parent claims its recorded call tree transitively in the same sweep, and an engine meeting a child first defers it to the parent's claim. This closes the multi-engine split that left a correctly-recovered parent unable to re-link a correctly-recovered child (#308): the failure was self-inflicted by claim granularity, not by anything wrong with either instance. Affinity is a claiming rule, not a distribution policy — §2.8's claim-first semantics and group scoping are unchanged, only the granularity of one claim grows. A child whose lease is genuinely held by a live engine stays unclaimable and the loud refusal stands. Cross-engine re-link proper (remote child handles, cross-engine completion and cancel) is explicitly NOT decided here and stays a §5 distribution concern. Implementation by the accompanying SRD. |
| v.4 | 2026-08-06 | Ruslan Gabitov | Composite-construct fidelity (new §2.10, motivated by the capture-deferral stopgap and its silent siblings): faithful capture is the norm — a composite scope, sequential/parallel multi-instance, a Standard Loop, a resolving compensation sweep and an in-flight Call Activity all record their position in the document, and hydration rebuilds each at that position (never re-executing completed work); the deferral posture is retired (refusal remains only for unserializable data, loud). Child instances are durable and symmetrically linked: every instance — root or called child — checkpoints under the same repository/group; the parent records the call descriptor, the child the reverse linkage; recovery restores both ends (completion watch re-established, cancel cascade survives); a missing counterpart fails the restore loud — a child instance is state, not an effect, so duplicating one is designed out. §2.1 grows: item 5 covers both iteration strategies, item 6 adds the sweep position, new item 7 the call linkage; §2.2 clarifies re-enter applies to steps, not recorded composites. New §5 deferral: ledger nesting fidelity (flattening preserves sweep order; nesting/names cosmetic); the incident-store deferral refreshed to current truth — incidents are durable per instance since the incidents landing, what stays deferred is the cross-instance queryable surface. |
| v.3 | 2026-08-04 | Ruslan Gabitov | Two partitioning principles, motivated by the first durable Repository adapter (postgres) — partitioning must be in the port's contract before a second implementation freezes it. Engine groups (§2.8): the group is part of the engine's configured identity (not storage wiring); records carry their creator's group and the recovery listing / wake claims are group-scoped, so several clusters may share one database without interference; an unnamed engine forms a single-engine group under its own id — clustering is explicit opt-in, never accidental (zero-config restart recovery unchanged); deployment parity becomes a per-group contract. Tenant linkage (§2.7): tenants partition data ownership orthogonally to groups — one instance belongs to exactly one tenant, one engine may serve many; unspecified → the default tenant (flag-designated in the registry, exactly one per engine group — never a reserved id); storage keeps a tenant registry (tenants table) that records reference; only the durable linkage lands here — tenant-scoped APIs, enforcement and lifecycle are a future Multi-tenancy ADR (new §5 deferral). §2.7's skeleton-growth list adds the group and the tenant to the record. |
| v.2 | 2026-07-27 | Ruslan Gabitov | Pin refresh: ADR-007 authored in full and Accepted (v.2 — the in-memory dehydration & wake-on-trigger mechanism this ADR's §2.4/§2.5 delegated to), so the §Refines and §3-grounding pins move v.1 → v.2. No content change to the durable model. |
| v.1 (Accepted) | 2026-07-27 | Ruslan Gabitov | Accepted with the first landing slice (the accompanying checkpoint/recovery SRD): the checkpoint document, the grown Repository (CAS + lease), consistent-cut capture, restart recovery with re-enter semantics and the recorded-deadline timers — all proven live, incl. the §2.8 fencing (a zombie engine's saves rejected) and the ADR-005-style incremental plan: dehydration/wake-on-trigger and suspend/resume ride the remaining slices. One §2.8 sharpening surfaced by the landing: deployment parity covers ELEMENT IDENTITY too — recovery requires stable node ids across engines (pinned ids or a serialized model). |
| v.1 | 2026-07-26 | Ruslan Gabitov | Initial draft — the deferred Persistence & State conception: one checkpoint document per instance (identity + pinned version, status incl. Suspended, scope-tree data, the track table, per-node wait descriptors, the completion ledger; armed/routing state derived at hydration, never stored; schema-versioned, loud on unserializable values); the loop's consistent-cut checkpoint at the normative lifecycle transitions with a pluggable write mode; exactly-once state / at-least-once effects (correlation dedup, job-queue reclaim, idempotent re-announce); dehydration as the durable half of ADR-007's model (one model, two residency levels); wake-on-trigger hydration = single-instance recovery, per-wait-kind semantics (overdue timers fire once, tasks re-announce, subscriptions re-register); suspend/resume as status over the same machinery, engine pause filling the reserved observability slots; the Repository grows CAS + Suspended + opaque schema-versioned payloads. Event sourcing, persist-everything and hydration-before-durability rejected. The storage composition rule: the Repository is the checkpoint port only — one narrow port per storage consumer, the backend handle user-owned and shared at construction (no universal Repository, no db-driver seam in core — both named rejected alternatives), namespaced schemas with per-module migrations and the optional Migrator capability. Cluster-safe sharing (§2.8): per-instance ownership leases with CAS fencing, claim-first wake, lease-expiry orphan recovery, loud deployment-parity refusals — safety here, work distribution deferred to the Distribution & Scale ADR. Implementation rides the accompanying SRDs. |