Persistence, recovery, deployment staging, and first qualification slice

This document refines the specification and architecture; they take precedence. It fixes a simple GKE/GCP shape and GCS adapter for the disposable devnet. It leaves the exact region, machine shape, production bucket placement/storage class, recovery SLO, and production topology. Those values require named measurements and block only their affected recovery campaign, placement approval, or mainnet readiness.

1. Operating rules

1. One validator identity has one live signer. Restart, restore, replacement, and rebootstrap require an operator fence before the successor can sign. The fence plus chainspec/profile, storage, journal, share, release, clock, p2p-ceiling, target, and root checks form the pre-vote gate. Passing them permits the engine to vote while service readiness remains Initializing; it becomes Serving only after its own live vote above the replayed journal ceiling.
2. The Simplex journal and validator secrets are identity-bound. Rebootstrap may replace application databases and marshal progress, but preserves the consensus journal and approved custody material. Neither is reconstructed from chain state.
3. The three application databases commit one logical block, but are not a cross-member atomic transaction. Each member exposes its own rewindable checkpoint and durability handle. The set barrier completes only when all captured handles are durable; startup reconciles unequal targets to one certified header before voting.
4. One application SyncPlan is created before marshal and Stateful and consumed by both. A completed peer state sync is durable state: later restarts take the marshal-sync path. Deleting its marker is not rebootstrap. Dynamic DKG uses its own node-wide state-sync plan and readiness gates.
5. Marshal first makes the finalized block durable in its own block store, then dispatches it. Stateful may call the finalized observer after apply while database durability is pending; marshal acknowledgement is released only after that observer resolves and a covering database-set barrier completes. The observer is idempotent because a crash may redeliver the height.
6. Exporters and indexers never make their durability part of consensus acknowledgement. They persist effects and their own watermark together and accept redelivery. An indexer scans certified history for gaps; an exporter publishes only from its queue entries, and a queue entry missing at or below its highest returned hook is a fatal publication gap (R-17.10). Derived events are re-executed from certified inputs; an event is not a receipt.
7. Corruption, chain/profile mismatch, missing required signer material, or unsupported active rules stops the affected role. Missing remote history or temporary peer/service unavailability remains pending or explicitly unavailable; it never becomes protocol invalidity or fabricated absence.
8. Settlement evidence is always real Tempo evidence. Devnet and testnet bind to Tempo Moderato; mainnet binds to Tempo Mainnet. Consensus execution is pure over supplied proof bytes and never performs RPC. A relayer or fixture-acquisition tool may fetch outside consensus, then archive the exact certificate, header, account proof, and storage proof before submission.
9. A validator installation has three separately governed persistent classes: identity-safety, application/rebuildable consensus state, and secret custody. Rebootstrap replaces only the application class. Copying a complete validator directory to start a second process is forbidden.
10. Every local queue has count and byte bounds, one named owner, and an observable full outcome. Retries preserve the same operation identity and stop at a fatal-local error; no retry invents a new command sequence, artifact name, DKG seed, or voting identity.
11. Fresh genesis has one database writer before Stateful starts. It may seed only the current-state member from its native bootstrap target, only after the complete genesis reference has been computed from the speculative batch, and only when the application SyncPlan has no selected or persisted peer-sync floor. Membership and history remain at native bootstrap size one. The expected genesis root and target make retries idempotent; any third state is fatal-local.
12. The header's canonical history target is compact (root, size), and every node role, validator and exporter alike, uses it directly. Every role merkleizes history with inactivity floor zero so the compact and full roots are identical, and every role may prune its compact witness journal under the recovery policy. Mapping the header value to the full target (root, 0..size) is a proof-service reconstruction step over the store's operation log, never a node readiness check; that reconstruction must not use Commonware's compact-to-full TryFrom, which maps only size - 1..size. A nonzero committed history floor on a node still closes readiness and requires recovery from certified history.
13. The exoware store has no server-side authentication of its own and its simulator defaults to permissive CORS, so it MUST sit on a private network reachable only by the exporter, the standby, the proof service, the indexer, and the gateway. No store endpoint is published, and every public read passes through one of those bounded facades. Because the store authenticates no writer, every reader on that network could otherwise issue Ingest.Put, so an authenticating proxy in front of it admits write calls only from the two exporter identities and admits the proof service, the indexer, and the gateway for reads alone; the exoware SDK already carries a bearer credential for exactly this shape. The store also ships no replication, snapshot, or backup: the persistent-disk snapshots and the mirror bucket are Makechain's own durability.
14. Under R-17.21 an exporter or standby state-syncs at its first installation only. State sync installs a height range without running the finalized hook, so nothing that range contains reaches the installing node's queue. On completion the node MUST record its sync anchor A durably, and its own queue coverage begins at A + 1. From the start of the installation until A is recorded, the node MUST re-evaluate at every retargeting of its sync that every height in (W, current target] is durable elsewhere, where W is the publication watermark observed in both the store and the mirror at the start of the installation: either the publication watermark has reached the current target, or the other roster member is healthy, meaning its queue coverage is at least the current target and its watermark has advanced within the publish-lag bound. If neither holds at any evaluation, the node MUST abort the installation and alert; an installation that completes anyway loses (W, A] under the first cause of the history-range-loss risk in specification section 4.4. The sync target's finalization is read from the store's finalized-by-height record and MUST be certificate-verified against the operator's own trust anchor before the plan adopts it. Once an installation has completed, the role follows marshal only and never state-syncs again. An exporter for which max(own queue coverage, publication watermark) lies below the validators' retained window stops and alerts instead of syncing, since no source then holds the missing range. The first exporter of a chain, and a standby that starts at genesis, starts without state sync and queues the genesis block and the location-zero slice of both logs from its own initialization, because the genesis block reaches neither the capture nor the finalized hook; it reconstructs each slice from the native bootstrap commit and verifies it against the targets of the published, hash-checked genesis block before queueing it.

2. Persistent artifact ownership

“Barrier” below means the point after which a success response may be relied on after process crash. Retention durations remain calibration or recovery-objective inputs where the specification does not fix them.

2.1 Validator partition map

Each validator identity has these logical partitions. They may share a physical Persistent Disk for the first devnet only if backup/restore and rebootstrap tooling addresses them by explicit prefix; no command may erase the volume root.

Every on-disk key is prefixed by (chain_id, role, identity where applicable, format_version). Startup rejects a path whose manifest differs. A PersistentVolume snapshot is a backup artifact, not a second live installation: restoring it requires the same identity fence and never rewinds the vote WAL below the active installation's recorded ceiling.

2.2 Artifact contract

External objects and private application data are not node recovery artifacts. A project's availability layers, of which there may be several and which the chain does not locate, own object durability, access policy, and closure completeness under the obligations in availability profile. A chain publication or receipt proves neither byte availability nor permission to read those bytes.

2.3 Fixed-profile test-share lifecycle

The fixed profile uses cryptographically real shares with deliberately test-only custody:

1. An offline one-shot tool takes the canonical ordered seven public identities, uses OS cryptographic randomness, calls the pinned feldman_desmedt::deal, validates total = 7, required = 5, every participant position, and reconstructs each signer once. It emits the public output/chainspec input and seven separately encrypted share envelopes. It emits no dealer seed.
2. An audited operator step imports each envelope into a distinct per-validator secret version. The share never enters git, an image, Terraform state, CI output, logs, a shared Kubernetes Secret, or a run manifest. Only public/output/share fingerprints and secret version identifiers enter the provisioning manifest.
3. Each validator workload identity can read only its one secret. Startup materializes it into process memory or a mode-0600 memory-backed file, reconstructs the signer, compares participant, output, chain and fingerprints, then removes the staging file before enabling the network actor. Local deterministic tests use per-run temporary directories and the same validation path.
4. Reset creates a new chain, dealer output and seven shares. Operators first stop and fence the old identities, revoke workload access, destroy every retired secret version and temporary copy, retire identity/application/custody snapshots, and record only identifiers, hashes, actors and timestamps in the destruction receipt. The old public output and test reports remain public.

Losing one fixed share does not authorize regenerating the same chain's dealer set. Restore the identical approved envelope while the old signer is fenced, or reset the disposable network.

For later DynamicReshareV1 qualification on GCP, the concrete custody adapter stores authenticated ciphertext records and their append/index metadata on the validator's dedicated custody partition; it wraps the store data key with a per-validator Cloud KMS key reached through that validator's workload identity. Access to the KMS key, any ciphertext restore and the identity-safety partition transition only through the fenced handoff. Decrypted shares/seeds/dealings exist only in process memory. KMS or disk unavailability keeps the actor unready, and destroying a KMS version is permitted only after the Commonware prune floor, successor-handoff evidence and recovery policy all release it. The protection level, backup location, operator quorum and mainnet HSM/external-custody choice remain a security qualification decision; any replacement adapter must preserve the same durable/idempotent SecretStore contract and pass G5a.

2.4 Single-voting-identity fence

There is no automatic validator failover or active/standby signer. A Kubernetes lease, Pod name, ReadWriteOnce volume, or revoked secret permission alone is insufficient because an old process may still hold a share in memory. Replacement uses this fail-closed handoff:

1. Mark the identity NotServing(Draining), stop admission/proposal input, and request engine shutdown. Record old workload UID, node/VM identity, last live-vote round and journal ceiling.
2. Prove the old process cannot execute: normal handoff observes clean process exit; failed-node handoff requires control-plane evidence that the VM is terminated. If termination cannot be established, do not start a successor.
3. Revoke the old workload identity's secret/KMS access, detach its read-write volume, and acquire an operator-controlled monotonically increasing activation generation with create/compare-and-swap semantics. Persist an immutable fence receipt. The generation is operational evidence; it is not a consensus field and cannot make two live signers safe.
4. Attach the one authoritative identity-safety partition and custody source to the successor. Startup verifies the fence generation, chain/profile, signer/share, journal integrity and ceiling, application targets/root, release support, clock and p2p ceiling before starting Simplex.
5. Mark Serving only after the successor's own live vote strictly above the replayed ceiling.

Ordinary restart of a process on the same fenced installation uses the same checks and never creates a second activation. Any disk restore or move to another workload is a replacement and follows all five steps. The initial seven operators are the owner's team; later validators remain explicitly permissioned, trusted operators and must present the same fence/handoff evidence. The design makes no claim that seven pods under one administration are seven administrative trust domains.

2.5 Ordinary restart and deliberate rebootstrap

An operator selects exactly one procedure before starting a validator:

Deleting a marker, opening empty partitions over old paths, restoring only some members, rolling back the WAL, or generating a replacement share is neither procedure and leaves the role NotServing. If identity-safety or required custody material is unrecoverable, a same-identity rebootstrap is forbidden; recovery is protocol-approved identity replacement or a disposable-network reset.

2.6 Shutdown and readiness

On SIGTERM or an operator drain, the process first publishes NotServing(Draining) and refuses new admission and proposal leases. It cancels dry-run, fan-out and proof/export network work; tells the engine actors to stop; waits for in-flight vote-WAL syncs, database barriers and marshal metadata already started; syncs durable service queues/watermarks; then closes storage handles. A bounded termination grace may end the process before completion because Commonware restart recovery handles the durable prefix, but shutdown never deletes or edits a partition. Replacement still requires the identity fence even after a clean shutdown.

Role readiness is explicit:

As required by the specification, a serving validator that later lags remains Serving; lag, barrier age, queue occupancy, retry age, exporter publish lag, store ingestion lag, query-tier cache hit ratio, and unsupported pending release are separate health signals and alerts. Readiness is never the only incident signal. One publication signal is an exception and is stated as such in the table above: the gap between the validators' retained floor and the publication watermark closes exporter readiness at its bound, because beyond that bound the range is unrecoverable rather than late.

3. Outcome vocabulary

Environment assertions are explicit. A missing optional endpoint yields a service capability outcome. A missing required validator secret or durable partition yields NotServing. No environment default may mint credits, generate same-chain signer material, select another chain, or weaken verification.

3.1 Queue, timeout, and retry ownership

Every bound below is an explicit deployment value reported in diagnostics. Numeric values wait for the relevant workload campaign, but zero/unbounded values are invalid configuration.

Retry counters, oldest-item age, last typed error, and next-attempt time are observable. Shutdown stops starting attempts, preserves durable queue/cursor state, and abandons only re-derivable in-memory hints. An operator can quarantine a permanently failing service item, but cannot advance its watermark past it.

3.2 Accepted first shared devnet and deferred production placement

Accepted owner decision, 2026-09-19: the first shared disposable devnet runs in one owner-selected GCP region. A regional GKE control plane and node pools in that region host seven one-replica validator StatefulSets, each with a distinct consensus identity, threshold share, workload identity, network address, identity-safety/application/custody partitions, and activation record. Pod anti-affinity spreads identities across available nodes/zones where capacity permits. One regional outage may stop the devnet, and the shared GCP project/administration is one correlated failure and trust domain. This is accepted for development and earns no production topology evidence.

Use GKE/GCP for Rust validators, the exporter and its standby, the exoware store and the proof service, admission backends, index/query workers and relayers. Cloudflare is reserved for later apps/websites, edge routing, abuse controls and bounded facades after their consumed contracts pass the architecture gates. SDK, CLI, explorer, website and the separate consuming application are not in the first implementation slice.

Accepted owner decision, 2026-09-22: history is kept in two copies. The mirror is a same-region Google Cloud Storage Standard bucket under a retention lock, holding one raw object per block, the block with its certificate and both operation-log slices, content-addressed by block_hash, with a height index and a durable watermark, and reached through a distinct workload identity for each writer role. The exoware store runs on a persistent disk with disk snapshots, on the private network of operating rule 13. The mirror adapter contract is only put_if_absent(name, bytes, digest) -> Created | ExistsSame | Conflict, get, and head:

• map create-if-absent to ifGenerationMatch=0; on 412, read and compare length/digest, as specified by GCS request preconditions ↗ (external site);
• treat a successful write as readable, but retry documented transient 5xx outcomes. The GCS consistency contract ↗ (external site) is strong per object/list operation, not a multi-object transaction, so publish the per-block object first, the height-index entry next, and advance the watermark only after verifying both;
• never use listing order as the watermark or overwrite an immutable per-block object;
• keep mirror objects separate from external repository objects and their ACLs; and
• retain the provider-neutral object names/digests so an in-memory deterministic adapter and the GCS adapter exercise the same module interface.

This is a selected low-complexity devnet backend, not a production backend or cost commitment. GCS documents strong object/list consistency and conditional generation writes; regional buckets give cross-zone redundancy and same-region compute locality. Production must separately select regional, dual-region, multi-region, or independently administered buckets; exporter and store process/project/region placement; replication/retention/soft-delete policy; and access/audit controls. The platform lead owns that choice after measuring write/read volume, object count, store growth and rebuild time from either copy, outage/recovery objectives and current prices. G8 records the chosen latency percentiles, machines, regions and cost; none is inferred from the devnet. Object-store success does not authenticate protocol bytes, replace the checker's root comparison, or fence a signer. Published objects remain with the project's own availability layers and are not placed in the mirror bucket by this decision.

3.3 Ingress and least privilege

Validators expose authenticated Commonware mesh channels and operator diagnostics only; no public HTTP, proof, query, object or administrative endpoint reaches a validator. Exporter mesh channels accept only configured operator peers. The exporter's proof gRPC is a separate bounded server identity and cannot submit votes, mutate consensus databases, or read custody partitions. The exoware store answers on the private network of operating rule 13 and is reachable only by the exporter, the standby, the proof service, the indexer and the gateway; it carries no authentication of its own, so network placement plus the authenticating proxy of that rule is the whole control. The store's writer set is exactly two identities, the active exporter and the standby, and the proxy admits Ingest.Put from those two alone. Gateways are the sole public command ingress and hold no vote, store-write, settlement-admin or object-store credentials. Index/query, relayer and availability identities are distinct.

GKE workloads use Workload Identity Federation with resource-level IAM. Validator identities may read only their own test-share/KMS key and attach only their own named volumes. The exporter may create and read mirror objects and write to the store, but may not overwrite or delete either and may not reach repository-object buckets. Readers receive read-only access. On cloud resources that is an IAM role; on the store, which authenticates nothing itself, it is the private network of operating rule 13 plus the authenticating proxy that admits writes only from the two exporter identities, so the proof service, the indexer and the gateway reach the store as readers and cannot write to it. Human break-glass roles are separate, audited, and never mounted into pods; service-account key files and shared Kubernetes Secrets are prohibited. Cloudflare ingress, when later introduced, authenticates to a bounded gateway facade and receives no mesh, database, bucket-writer or custody authority.

Public chain artifacts contain only chainspec/genesis/release material, certificates, blocks, operation-log slices, proofs and public qualification reports. Secret identifiers may appear only as opaque version/audit references; share bytes, dealer seeds, private dealings, credentials and unencrypted operator state never do. Devnet, qualification, staging and mainnet use separate projects/namespaces, workload identities, buckets, partitions and chain identities.

3.4 Deployment, activation, rollback, and reset

Once a block under new rules is final, rolling back binaries cannot roll back the protocol. An unsupported active release is NotServing until a compatible binary is deployed; the node does not fall back to predecessor semantics.

4. First qualification slice: Q1

Status: NOT RUN. Q1 proves one narrow application/consensus/persistence path. It does not qualify all seventeen commands, dynamic epochs/DKG, validator changes, release handoff, public service ingress, production custody, object availability, topology, capacity, or recovery time.

4.1 Preconditions

• Use a fresh fixed-profile chain identity, seven validator identities, real threshold shares, and the specified threshold-VRF/Random V1 + Deferred + Stateful assembly. Finality is an actual five-of-seven certificate; a harness flag may not mark a block finalized.
• Use the exact three persistent database members and a shared application SyncPlan; enable real Simplex WAL, marshal archive/processed metadata, and their durability barriers.
• Settlement is an external prerequisite. Before Q1 can run, a compatible settlement contract must be deployed and pinned on Tempo Moderato and a real positive fixture must be captured: Tempo certificate and header, EVM account proof, beneficiary cumulative storage proof, contract/code and accepted-token identity, and timestamps satisfying the configured trust/skew deadlines. The current deployed StorageRelay is insufficient: it records global issuedUnits and emits per-call Rent/Credit, but has no beneficiary monotonic cumulative at the required pinned base slot and permits operator-issued unpaid credit. Existing real Moderato header/certificate fixtures are useful for a partial certificate-profile gate, but cannot prove a positive credit import.
• The absent settlement deployment and positive fixture block only settlement-dependent Q1 execution and its resulting conformance claim. Independent schema, verifier, executor, persistence, and harness work may proceed so long as it neither fabricates positive evidence nor marks Q1 passed. This planning document does not authorize contract implementation or deployment.
• Deterministic runs replay the captured Moderato fixture through the real settlement verifier with consensus time chosen inside its authenticated deadlines. Persistent Tokio staging uses the same bytes. There is no live RPC in execution, synthetic positive certificate, Makechain faucet, genesis funding, mint bypass, or mocked proof acceptance. RPC is allowed only to the external relayer/fixture tool that captures evidence before the run.
• P3 supplies canonical schema/vectors, fixture byte hashes, registered commitment data, prices and limits sufficient for the enabled commands. The exercised subset is IMPORT_CREDITS.Credit, INIT_ACCOUNT, KEY_AUTHORIZE.Authorize, PROJECT_CREATE, and PUSH. Every other tag remains in the seventeen-tag schema but gains no conformance credit; an unavailable handler fails explicitly and never no-ops.
• Use a fixed-network diagnostic exporter, store, and proof service reachable only by the trusted test harness. They serve real certified bodies and proofs and have no private-object, app, or public-production role.

4.2 Required path and assertions

1. Build the canonical genesis, recompute chain_id, install seven distinct shares, start all nodes, and record each node's journal ceiling, marshal floor, member targets, and readiness transition.
2. Submit IMPORT_CREDITS.Credit with the captured valid Tempo proof and require strict cumulative advance to create and fund the beneficiary ledger. Submit INIT_ACCOUNT with the real genesis witness; then authorize an account-capability delegated key. Create a project with that delegate. Submit one PUSH containing a registered commitment and CREATE of a ref to that same-command commitment. Use normal admission, proposal, verification, apply, and finalization.
3. Verify a real command inclusion proof and publication receipt from caller-supplied genesis trust: exact envelope, command index/count/root, authenticated header and finality lineage, PUSH and commitment index, actor/project/commitment, and occurrence tuple. Substitute one index, header, or envelope and require verification failure.
4. After each named crash cut in §4.3, restart the fenced validator from preserved identity material. Exercise ordinary restart first. In a separate fresh-application-storage run, preserve the vote WAL and share, use the one-time application SyncPlan, catch up the tail, and vote only after all three targets and canonical state_root match one certified header. Exercise one replacement attempt while the old process is not provably terminated and require the successor to stay NotServing; then complete the fence/handoff and require its live-vote gate.
5. Compare canonical results before and after recovery: block/header bytes and hashes, command and intent IDs, command root/count, all three member targets and roots, membership location, account replay mark and ledger totals, project/ref records and revision, event IDs/order, occurrence tuple, command proof, receipt verification, processed height, and exporter/index watermarks where enabled.

The proof service may be unavailable after finality without changing the finalized result; it reports temporary unavailability until its certified gap is repaired. Object bytes are deliberately absent from Q1, and the result must state that receipt success does not prove availability.

4.3 Crash cuts

Each cut is a separate seeded run. Fault injection may stop a process or abort a storage handle; it may not edit journals, delete sync metadata, alter certified bytes, or bypass fencing to force recovery.

4.4 Minimum execution layers and gates

Beyond APP-10's positive credit-import path, Q1 does not absorb APP-02–09, the remaining APP-10 failure/refresh cases, APP-12–20, API-01–06 or API-08–14/16–20, dynamic gates G5a/G5b/G6, or workload/topology gate G8. Those remain separate so a narrow green slice cannot imply full protocol or production qualification.

5. Result artifact contract

Every run writes an immutable manifest even when it fails:

• status NOT RUN, PASS, FAIL, or INCONCLUSIVE; start/end time; seed; layer and cut;
• repository revision, specification revision, Commonware pin, runtime, binary and chainspec/profile digests, genesis block/hash and chain_id;
• fixture and canonical-vector digests, enabled command subset, seven public identities and threshold public output, with no secret bytes;
• exact signed envelopes and IDs, block/header/certificate bytes or content-addressed references, Tempo source profile/contract/beneficiary slot/cumulative and proof fixture digests, and every settlement, command-proof, and receipt-verifier outcome;
• per-node vote-journal ceiling, marshal floor/processed height, application SyncPlan state, each database target/root, readiness transitions, and fencing record before/after recovery;
• export/index watermarks, gap-scan outcome, queue bounds, typed errors, logs/traces, and a comparison report for every §4.2 assertion.

A result is INCONCLUSIVE, never PASS, when required artifacts are missing, a crash cut was not reached, finality was simulated, external verification was bypassed, or the manifest cannot bind all evidence to one chain/profile and run.

6. Source register and version boundary

The local Commonware checkout is newer HEAD 3827a56dc76818bd31c42c2d6b8c7da261123f2a. Its marshal implementation has substantial post-pin changes, including dispatch/retention work, while the reviewed Stateful/DKG contract is largely intact. This document adopts none of those changes. Any pin bump reopens the journal, processed-floor/acknowledgement, database barrier, state-sync, and crash-cut review before qualification evidence can be reused.

7. Decision disposition and execution prerequisites

8. Mesh topology, admission, sync, and readiness

This section holds material formerly stated in the specification and is not normative: the specification defines validity, and nothing below changes whether a block or a command is valid. It supports specification chapter 12 (execution and block validity), chapter 13 (consensus and validator membership), and chapter 17 (node roles and the database set).

Requirement keywords in section 8.2 are kept in capitals because they state the node contract that clients and operators rely on, even though no item affects block validity.

8.1 Mesh topology

The operator mesh is one authenticated Commonware network in lookup mode, not discovery.

Every peer address is already known before the mesh starts:

• validator addresses are chain state under DynamicReshareV1;
• validator addresses are fixed genesis data under FixedEpochZeroV1;
• the exporter and standby addresses are chainspec data, carried by the exporter roster.

Two consequences follow. The address gossip that discovery exists for has nothing to solve. lookup's address overwrite absorbs pod restarts without minting a new peer set.

Peer sets and the single membership owner

Primary and secondary are properties of a tracked peer set, not of a channel. Every consumer of a provider fetches only from the latest set's primary members.

Membership updates therefore have exactly one owner: a Makechain component that tracks, at each set id, the union of current validators, scheduled and departing validators, and the exporter and its standby, all as primary.

• The DKG directory is applied through that owner, never by cloning the oracle for the ceremony. Cloning the oracle flips every consumer to a validator-only set and silently drops the exporter from block gossip and backfill.
• A consumer that needs a narrower view, such as a ceremony that sees exactly its participants, receives a Makechain-owned projection of that provider. The library ships no such view.
• Transport membership never determines voting membership.
• Peer bounds are sized against the largest simultaneously required union, not the current committee.

The single all-primary tracked union and the ceremony-only projection are already covered in section 2 and section 4 of architecture.md, which assign them to the mesh membership module and the mesh seam.

Exporter roster and reachability

The exporter and the standby are non-voting mesh peers, not gRPC followers. The specification's follower rule, R-17.12, is about gRPC followers such as gateways, indexers, and clients; it does not reach these two. They receive block gossip, certificates, and backfill on the mesh exactly as this section's channel plan describes.

The exporter roster is a chainspec value, listed in the chainspec table of specification section 5.2, and it is governed: it is item (i) of the governed parameters of R-18.21, so a release changes it through the ordinary authenticated schedule. A value with no governed path would change only by reset under R-18.24, which is not an acceptable cost for replacing a publication host. Nothing consumes the roster's determinism beyond liveness and transport membership, and transport membership never determines voting membership. The roster holds two entries: one active exporter, which publishes, and one standby follower, which runs with publication off, keeps its own durable publication queue, and resumes from the store watermark when it takes over.

Every mesh peer, the exporter and its standby included, advertises a dialable external address. Commonware has no NAT traversal. Targeted abuse is absorbed by an external proxy, which the exporter needs more than validators because it serves proof traffic.

That validators expose authenticated mesh channels and operator diagnostics only, and that exporter mesh channels accept only configured operator peers, is already covered in section 3.3 of this document.

Channel plan

Each channel carries its own quota so that one traffic class cannot starve another.

max_peers_per_set covers validators plus the exporter and its standby plus the local identity. Undersizing silently drops traffic rather than erroring, because mailboxes are sized from it.

8.2 Admission and mempool policy

Admission is local policy that never changes validity. Block execution is the only authority. The binding parts are the client contract and the bounds that protect consensus.

Response and status classes

Status additionally distinguishes Replaced, a local pool replacement, and Superseded, a finalized sequence that has overtaken an unfinalized intent.

The response class list is new to this document. The f + 1 acknowledgement meaning of Accepted is already covered in section 3.1 (gateway fan-out) and in section 5 of architecture.md, which states that Accepted is never durability or finality.

Required behaviour

• Admission MUST verify signatures, structure, chain binding, and the replay tuple (key_id, generation, sequence) against finalized state and the local pool.
• Admission dedup keys on (key_id, generation, sequence, intent_digest), never on command_id. WebAuthn client data varies per signature, and evidence or witness replacement changes the envelope while leaving the intent unchanged, so command_id = BLAKE3("makechain.command-envelope" || canonical(SignedCommand)) is not stable for one intent. A pool that dedups on command_id admits the same intent twice in exactly those two cases. New to this document: section 3.1 requires same-slot replacement but names no dedup key, and the AlreadyKnown response alone does not carry the prohibition.
• Stateful dry-run of the transition is optional and off by default. A state-invalid command that reaches a block is rejected there, so admission errors are advisory. Already covered in section 3.1 (admission dry-run bounds) and in section 3.2 of architecture.md.
• Admission MUST run off the consensus executor. Its CPU MUST be bounded by per-account rate limits and the optional dry-run switch, never by sharing the proposer's view budget.
• Pools MUST be bounded in count and bytes. Eviction MUST be drop-oldest and batched. Already covered in section 3.1.
• A command whose wire size alone exceeds the block byte budget MUST be rejected at admission.
• A pool MUST hold several pending commands per (key_id, generation), up to the admission_pool_capacity per-key pending-depth cap (specification section 19), in ascending sequence order per R-7.63, and MUST select known candidates in that order.
• A pool MUST NOT wait indefinitely for a missing sequence, because gaps are valid.
• Two commands with the same sequence and different intents replace by later arrival within one pool. The pool reports Replaced for the loser, which is that pool's view and not a chain fact.
• Status retention is finite and stated. Already covered in section 3.1.
• Gateways MUST fan every accepted command out to all validators concurrently with bounded in-flight limits, and MUST report Accepted once f + 1 validators acknowledge. Already covered in section 3.1 (gateway fan-out) and section 2.6 (gateway readiness requires at least f + 1 eligible validator targets).
• No validator gossips commands to another.
• At a price-schedule activation height the pool MUST stop selecting predecessor-schedule commands.
• StateConflict MUST include the conflicting record or context category and the serving checkpoint. It is advisory, never a proof.

Client contract

• Pipelined chunk submission is supported but does not guarantee that each chunk finalizes: a later sequence may arrive and finalize first.
• SDKs reconcile intent status against finalized sequence marks and command inclusion, then re-sign genuinely missing work with fresh sequences. They do not infer inclusion from an advanced mark alone.
• Clients refresh authenticated state, re-evaluate user intent, and re-sign a changed CAS or context rather than rewriting signed fields.
• Clients SHOULD use separate delegated keys for independent automation writers, to avoid sharing a sequence stream.
• The SDK MUST follow the preparation, signing, submission, and reconciliation contract in client-api.md. Client status values are Pending, Finalized, and Unknown, separate from admission responses.
• Only verified command inclusion establishes Finalized for an untrusted client.
• A timeout, a missing status, or an advanced sequence mark never establishes that an intent did not finalize. Uncertain inclusion remains Unknown.
• Automatically re-signing uncertain work with a fresh sequence can duplicate its effects and charges and is forbidden. Already covered in section 2.2, which states that a missing client state leaves an intent Unknown with no automatic new-sequence retry.
• A client MAY rebroadcast the same intent, MAY refresh its unsigned evidence within max_charge, and MAY obtain bounded authenticated history before deciding whether fresh work is needed.

Recommended behaviour

• Proposer selection SHOULD approximate per-account fair share.
• The proposer SHOULD defer rather than drop when its view budget is exhausted. A conforming node's metrics make the identity selected − included − dropped = deferred derivable, because that is how healthy backpressure is distinguished from silent loss. Already covered in section 3.1 (proposal selection).

Local and non-binding

Capacity numbers, quotas, eviction batch size, selection algorithm, view-cost model internals, and requeue mechanics are local choices.

8.3 Sync and recovery procedure

Startup is one durable SyncPlan shared by marshal and the stateful actor. Already covered in section 1, item 4, and in section 2.5.

A fresh node follows two steps:

1. State-sync all three members from validators through the unified Source engine to strictly advancing targets, verified against a certified header.
2. Follow marshal.

Validators are the state-sync source: they serve the current-state operations journal from their live window and the membership log from their committed floor, and the p2p resolver covers the tail. The store is an optional additional source for the membership log only, not a startup dependency, and bootstrap tiles are deferred. Validators serve no historical proofs, which is already covered in section 3.3 of this document.

An exporter or standby starting up obeys one further constraint, stated as operating rule 14. It state-syncs at its first installation only; after that installation completes it follows marshal and never syncs again. State sync installs state without delivering the covered heights to the finalized hook, so nothing in the installed range reaches the installing node's queue. The node records its sync anchor A durably on completion, and its own queue coverage begins at A + 1. Until A is recorded, the node re-evaluates at every retargeting of its sync that every height in (W, current target] is durable elsewhere, where W is the publication watermark observed in both the store and the mirror at the start of the installation: either the publication watermark has reached the current target, or the other roster member is healthy (its queue coverage is at least the current target and its watermark advances within the publish-lag bound). If neither holds at any evaluation, the node aborts the installation and the operator is alerted; an installation that completes anyway loses (W, A] under the first cause of the history-range-loss risk in specification section 4.4. Because coverage of (W, A] is what makes the installation safe, the sync target itself is free: a fresh installation, including a replacement standby, syncs to a recent target and starts with an empty queue. The finalization the plan pins as that target is read from the store's finalized-by-height record and is certificate-verified before the plan adopts it, because the store authenticates no writer. When max(own queue coverage, publication watermark) is already below the validators' retained window, no source holds the missing range, so the role stops and alerts rather than syncing forward and losing it silently.

A validator votes only after the readiness checklist:

• chainspec and namespace agreement;
• release support;
• clock bound;
• the profile-required threshold share;
• all three member targets equal at one certified anchor, and the canonical state_root equal to the header's;
• p2p ceiling;
• journal integrity.

The checklist is already covered in section 1, item 1, and in section 2.6, where it is stated as the pre-vote gate. The operator fence that this document adds to the gate is an operations requirement with no specification counterpart.

Two constraints hold in every recovery:

• Recovery never edits the consensus journal. Already covered in section 1, item 2, and section 2.5.
• Storage handles that fail are gone; recovery is a fresh init. Already covered in section 2.2.

Bootstrap handles are dropped before the execution host opens the members. Holding a bootstrap handle across that open is the boot-deadlock shape the rule forbids. Already covered in section 2.2, whose genesis row requires the task to apply, finalize, await durability, drop, reopen, and compare the canonical root plus target, and in section 3.1 of architecture.md, whose genesis database handoff drops the handle and reopens the member before Stateful or a network actor starts.

Two recovery procedures are supported and no third:

Both procedures are already covered in detail in section 2.5, which also states that fresh genesis is a separate start procedure rather than a recovery procedure.

Deleting the sync-complete marker is not a rebootstrap. Already covered in section 1, item 4, and section 2.5.

Recovering a range missing from published history is a separate procedure with three sources, tried in this order: the validators' retained window, which covers any outage shorter than retained_marshal_blocks; the other store copy, since the exoware store rebuilds from the mirror by replaying its per-block objects and the mirror rebuilds from the store by re-reading the same ranges; and the standby's own durable publication queue.

The third source exists because the standby queues. Under R-17.22 the standby runs with publication off, yet follows every finalized block and appends the same queue entry the active exporter appends, holding the block, its certificate, both operation-log slices, and the projected rows. It releases an entry only when it observes the store and the mirror watermark past that height, so at any moment its queue holds exactly the range the active exporter has not yet made durable in both copies. That is what makes it a recovery source for an active exporter's lost disk, and it is why the standby must be failure-isolated from the active, as R-17.15 requires: a shared disk, host, or zone would lose both queues in one event. A gap scan of the mirror's height index against the store watermark is the readiness signal that starts this procedure. The new statement here is the ordering of the three sources.

The supported outage envelope, meaning how far behind an ordinary restart can recover, is a chainspec-derived number. The rebootstrap transition is tested separately. Both statements are new to this document; section 7, OP-5, defers only the measured recovery time, not the envelope's source.

8.4 Readiness states and the observability minimum

Readiness has three states: Initializing, Serving, NotServing.

• Readiness gates startup completion and drain, not continuous health.
• A serving node that falls behind stays Serving; lag is exposed through metrics.
• A validator becomes Serving only after a live consensus vote of its own for a round strictly above its replayed journal ceiling. An opened database is necessary, not sufficient.
• A node reports NotReady to clients until then.

The three states, the lag rule, and the live-vote gate are already covered in section 2.6. The client-facing NotReady report is new to this document.

Every node exposes at minimum:

The funnel identity is already covered in section 3.1, and lag, barrier age, queue occupancy, retry age, exporter publish lag, store ingestion lag, query-tier cache hit ratio, and an unsupported pending release are already covered in section 2.6 as health signals. The list above is the minimum exposure set and is otherwise new to this document.

A node reports local-disabled settlement status when its chainspec selects the settlement-disabled profile of specification R-16.9: no SettlementSourceProfile and no SettlementAnchor are pinned, matching the settlement interface section 2 unset contract-address and code-hash rows for an undeployed profile. Under that status every settlement transition is deterministically invalid per R-16.10. It is distinct from a complete pinned profile whose mutable imports_paused anchor flag is set, which still reports the complete pinned Tempo profile identity with the pause surfaced separately, and from a partial profile, which R-16.6 makes invalid configuration that a node MUST refuse to start under rather than report as local-disabled.

A stuck barrier, a widening gap between processed height and the finalized tip, or a pending release the binary cannot run is the failure that a bare readiness flag cannot show.

8.5 Retention versus publication

Validators prune marshal's block store after retained_marshal_blocks.

• That window is a chainspec value chosen so that an exporter outage shorter than the window loses nothing: the exporter resumes and publishes the heights it missed from the validators' retained blocks.
• Validators do not block on the exporter's watermark, so an outage on the publication side never slows consensus.
• Two different outages have two different effects on the queue. A process outage of an exporter receives nothing while it is down, so its queue does not grow and the missed heights must come back from the validators' retained blocks on restart. An outage of the drain, of the store, or of the mirror leaves the process following blocks, so its queue grows and holds the unpublished range.
• History is lost in exactly two ways. Either both exporters, active and standby, stay behind for longer than retained_marshal_blocks, in which case validators prune the bodies for a range that no queue ever held and the store and the mirror stay intact but incomplete; or a range that was queued is lost by the queue, the exoware store, and the mirror together. Any one survivor of the relevant set is enough to rebuild.
• Operators monitor the gap between the validator floor and the store watermark. That gap is readiness-critical under R-17.17, not merely a metric: an exporter whose gap reaches its bound goes NotServing and alerts while the range is still recoverable, because past the window it is not.

The exporter publish lag is already covered in section 2.6 as a health signal. The window's sizing basis, the non-blocking rule, and the two loss conditions are new to this document. retained_marshal_blocks versus exporter outage tolerance remains a calibration item in specification section 19; no value is stated here.

Store growth derivations

These figures are derivations at the pinned library version, not measurements, and each is a calibration item until a qualification gate records a measured value.

That growth is qualified separately, as a named item in section 11 of validation.md.