Agent Coordinator

A vendor-agnostic service for AI agents on different workstations to coordinate tasks, report progress, and share handoffs and lessons across multiple projects.

Status: coordination, Linux operations, MCP, and this documentation book are implemented; the operator-run physical Windows workstation exercise remains. The Rust service, embedded web dashboard, and native CLI now support authentication, multiple projects, task admission, atomic ownership, renewable leases, checkpoints, inspected recovery, worktree preparation, local job reporting, and shared resource reservations, immutable submissions, independent review, serialized integration, completion after exact integrated checks, revisioned lessons, scoped decisions, bounded evidence uploads, authoritative Markdown imports/exports, operator account management, project policy editing, objective grouping, complete task history, verified online backups, restore reconciliation, clock rollback handling, and bounded storage maintenance. Current implementation and limits distinguish working behavior from the complete release design.

The book separates operational guides and implemented contracts from preserved release-design chapters. Treat implementation status as the source for working behavior; a proposed example in the plan or a historical contract does not make an endpoint available.

Try it locally

Install Rust 1.94 or newer with a native C build toolchain, then build the locked workspace. Native integration uses Git 2.39 or newer (tested locally with 2.39.5). SQLite is bundled; the dashboard has no separate build or CDN dependency.

cargo build --workspace --locked
target/debug/agent-coordinator-server \
  --public-origin http://127.0.0.1:8080 --allow-insecure-loopback \
  init-admin --username admin
target/debug/agent-coordinator-server \
  --public-origin http://127.0.0.1:8080 --allow-insecure-loopback serve

Enter a password at the hidden prompt, then open http://127.0.0.1:8080. Create a project and issue an agent credential under Access. The token is displayed once. Use the CLI setup and command guide to connect each harness with its own local session name. Connecting returns instructions and does not claim work. For remote use, follow the Linux/systemd/HTTPS examples.

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo test --workspace --locked
python3 scripts/smoke.py
python3 scripts/backup_smoke.py
cargo audit --file Cargo.lock

The smoke exercise uses disposable local accounts and data, and tests two actual CLI processes against the server. Install cargo-audit separately to run the last command. Linux and native Windows client checks are defined in GitHub Actions.

Design documents

• MCP connection guide: authenticated Streamable HTTP tools, protected client launch, session continuity, and retry rules.

• Linux installation: verified native packages, systemd, HTTPS, upgrades, and removal.

• Linux capacity: the constrained 20-project, 50-session, 100,000-task workload and measured restore stage.

• Linux acceptance evidence: passing package, installation, workload, and recovery checks with the retained measured report.

• Clock safety: durable time, authority expiry, and operator reconciliation.

• Storage maintenance: bounded compaction, permanent request identities, and daily maintenance.

• Backup and restore guide: hourly snapshots, off-server copying, fresh-directory recovery, and the operator checklist.

• Backup storage contract: consistency, verification, retention, storage bounds, and publication safeguards.

• Operator guide: account recovery, credential rotation, policy editing, objectives, task history, and inspected recovery.

• Operator access contract: current account and session authorization and safe account-creation retries.

• Objective contract: child membership and completion gates.

• Task history contract: bounded evidence pagination.

• Implementation plan: requirements, confirmed decisions, open installation inputs, data model, milestones, and acceptance scenarios.

• Release scope: engineering defaults, deferred features, and implementation-readiness evidence.

• Job/worktree contract: implemented job reporting, local launch/reconnect rules, scoped reporters, and physical resource holds.

• Coordination contract: task ownership, renewable leases, retries, worktrees, external jobs, and recovery.

• Completion contract: implemented submissions, review activities, required checks, publication guards, and recovery.

• Knowledge and decisions: revisioned lessons, context search, binding-rule provenance, and scoped work blockers.

• Artifact contract: bounded streaming uploads, retention, digest-verified native downloads, and evidence references.

• Import/export contract: historical migration, immutable previews, conflict checks, and provenance-preserving snapshots.

• Workflow specification: proposed states, selection, submissions, review/integration activities, and exact completion rules.

• API and CLI contract: proposed authentication/session flow, request conventions, endpoints, errors, and agent-facing commands.

• Implementation and acceptance: relational constraints, transaction boundaries, permissions, packaging, and release checks.

• Onboarding contract: operator setup, public authentication help, agent orientation, and the proposed repository snippet.

• Repository and hook review: evidence from existing agent workflows and the resulting design revisions. Local source links in this report refer to the workstation where the review was performed.

Direction

The complete release is being built toward the following agreed behavior. Use the implementation-status document above to see which parts are available.

• Rust with Axum and SQLite is the preferred service stack.
• The web interface should use vanilla JavaScript and modern CSS, with Alpine.js where useful.
• The first release includes an HTTP API, web dashboard, and CLI with readable and JSON output. The CLI and local job reporter support Linux and native Windows. An authenticated MCP endpoint is included; a TUI is deferred.
• One service instance will support multiple projects simultaneously.
• Workstations connect over public HTTPS. Every authenticated person and agent has access to every project.
• The server runs as a native Linux service under systemd, behind an HTTPS reverse proxy.
• People sign in with local password accounts; agents use revocable API tokens.
• The service owns the current task, handoff, and lesson records, with Markdown import and export.
• Implementation tasks will use separate Git worktrees, with one integration step at a time into each target branch.
• Agents will authenticate, claim tasks atomically, renew ownership, and record outcomes and shared knowledge through a vendor-neutral interface.
• Agents can create and claim tasks; each project configures required review.
• Review can require an independent agent, a human, or both; independent agent review is the default.
• Projects can allow automatic integration or require human authorization. They can also delegate binding-rule changes to agents without human approval.
• Store bounded logs/reports in the service, source checkpoints in Git remotes, and links to other artifacts.
• Another agent can recover expired work after checking saved work and running jobs; projects may require manual recovery. Code tasks finish after required review, target-branch integration, and validation of the integrated result.
• Existing harnesses launch agents. The service coordinates their work through API, CLI, and optional hooks, with local runners reporting job status.

The packaged server passed the 20-project, 50-session, 100,000-historical-task target at 50 requests/second for 30 minutes under a shared two-CPU/4-GiB/no-swap limit. See the acceptance evidence above for measurements and scope. Backup examples run hourly, retaining 24 hourly and 30 daily copies, with documented off-server copying and a one-hour restore target. The disposable local restore rehearsal passed; full operator-led recovery and an actual off-server destination remain installation checks.

The actual server is undecided. The test baseline is Ubuntu 24.04 LTS on x86_64 with 2 CPU cores and 4 GB RAM. Remaining host/domain/backup-destination choices are installation inputs; no blocking product questions remain.

Repository hygiene

Build output, runtime databases, local credentials, logs, and agent worktrees are ignored. Commit SQL migrations, sanitized configuration examples, shared agent guidance, and Cargo.lock. Never put tokens in a repository binding or URL.

Implementation status

The foundation, job/worktree evidence, reviewed completion, shared-record, operator-workflow, backup/restore, Linux operational controls, and MCP are implemented. Linux release acceptance and the final native Windows/Linux workstation exercise passed; see the retained evidence below. The complete release remains defined by PLAN.md; this document records current behavior.

Working now

• Authenticated stateless Streamable HTTP at /mcp with 56 typed tools routed through the existing REST authorization and mutation checks. A protected native launcher shares the exact saved harness identity with a trusted foreground MCP client and its native CLI children. See the MCP guide.

• Axum HTTP service with embedded vanilla JavaScript/CSS dashboard; SQLite WAL, foreign keys, migrations, transactional event records and mutation receipts.

• Local administrator initialization and password sign-in; Argon2id hashes, protected browser cookies, origin/CSRF checks, bounded login attempts, and human-admin issuance/revocation of separately named agent credentials.

• Multiple projects visible to every authenticated principal. Human operators create projects and control delegation. Agents create tasks and can change project rules when explicitly delegated; they cannot grant themselves permissions.

• Tasks with acceptance criteria, priorities, immutable definition revisions, planned/open lifecycle, same-project prerequisites and cycle rejection. Listings are observations. A transactional claim is the only ownership grant.

• Separate harness sessions with random persisted proofs bound to the issuing credential. One current attempt per task, monotonic ownership generations, renewable deadlines, separate heartbeat/progress timestamps, and revocation checks after acquiring the database writer lock.

• Checkpoints and release handoffs; release requeues or blocks a task and never marks it complete. Expired/revoked ownership requires a recovery claim and an explicit saved-work/job inspection record. A project can restrict recovery to humans. The human can resolve and release inspected work for an agent to claim.

• Clean-checkout registration with workstation/identity conflict checks and readable checkout metadata. Each implementation attempt should use its own worktree. The service records client attestations; it does not inspect a remote filesystem or execute Git itself.

• Public authentication help and service-delivered project instructions with version acknowledgments. Task/project/event lists use bounded cursor pages.

• Native CLI with origin-bound credentials, separate local harness sessions, JSON output, durable pending requests, and same-key retries. The dashboard supports project/task creation, task and checkpoint inspection, and credential administration; it has responsive layouts and no third-party scripts.

• Native worktree preparation with durable intent, remote identity and clean-source checks. Repeating preparation reconciles the same checkout; it never resets or deletes existing work.

• Global named capacity resources and atomic multi-resource reservations shared across projects. Holds survive task expiry, credential revocation, session loss, and missing observations. Ordinary release/recovery cannot bypass unresolved work.

• Locally launched jobs with stable producer identities, a durable launch journal, bounded local logs, and a detached guardian. Reconnect observes the existing job; uncertain launch intent never authorizes another producer. Linux boot/start time and Windows process creation time distinguish process identity from a reused PID.

• Scoped job reporter credentials, ordered idempotent observations, and optional bounded task renewal tied to a specific live harness. Observation authority and task ownership are separate. The service never launches workstation processes.

• Dashboard resource administration and task job/hold evidence, including explicit human reconciliation with recorded termination/isolation evidence.

• Immutable code/general submissions pinned to task and policy revisions, with acceptance evidence, handoffs, and full Git commit/tree identities for code. Active jobs and unresolved physical holds prevent submission.

• Independent agent, human, or both review. Contributor principals and sessions cannot perform an independent agent review. Findings and decisions stay bound to the exact submission; requested changes require a new candidate and fresh reviews.

• An explicit human-managed required-check roster and canonical repository key. Integration holds serialize every project sharing that repository/target pair. Projects can permit automatic integration or require a human authorization.

• Native integration prepares a candidate-containing result in an isolated worktree, validates registered producers against its exact commit/tree and check roster, and requires fresh service authority before a compare-and-swap Git publication. Once push intent is saved, retry only observes; it cannot launch another push.

• Code tasks become done and release dependents only after required approval, known publication, exact successful check receipts, fresh remote observation, and resource quiescence. General tasks finish after their required reviews.

• Dashboard submission evidence, review decisions, human review/authorization, publication reconciliation, and project review/check settings. Publication reconciliation preserves the original evidence and requires a new integration activity and fresh validation; it never fabricates a successful check.

• Revisioned lessons, facts, rejected approaches, and checkpoints; applicability, explicit cross-project sharing, provenance, usefulness feedback, and immutable revision history. Submission lessons and finalized artifact references commit atomically with the immutable submission and retain their original revisions.

• Plain-text FTS context search with bounded records, full current binding rules, explicit budget/truncation guidance, and current scoped decisions. Rule changes record provenance and preserve policy history.

• Scoped decisions with typed allow/deny/defer answers, required actor, exact task/policy revisions, environment, conditions, expiry, and reopening history. Blocks apply to selection, work, review, integration, and displayed authority; inspection, checkpointing, and release remain possible while work is blocked.

• Artifact links and streaming uploads with exact size/SHA-256, configurable quota and disk reserve, bounded concurrency, explicit expiry/deletion metadata, and safe storage reconciliation. Native upload journals preserve exact bytes and keys; retries reauthenticate and downloads verify digest before publishing a new file without overwriting existing output.

• Immutable Markdown previews and human-gated historical apply. Stable source identities, event/revision conflict checks, and durable historical evidence prevent reimports from reopening completed work or completing active attempts. Bounded, snapshot-consistent Markdown exports retain complete provenance.

• Dashboard and CLI access to lessons, decisions, context, artifact metadata, imports, and exports; the dashboard includes correction, human decision answers, rule editing/history, readable import previews, and snapshot downloads.

• Human account creation, revision-checked access changes, self-service password changes, browser-session inspection/revocation, and audited host password recovery. Agent credential rotation preserves the agent principal and returns the new secret once.

• Full policy editing, planned-task admission, inspected human recovery and blocker resolution, with matching native task, policy, objective, and history commands.

• Objectives with required/optional children, revisioned membership frozen once work starts, combined dependency-cycle checks, and their own acceptance/review workflow.

• Complete task evidence pagination across 12 record kinds, including associated review/integration activities, with bounded pages and scoped insertion snapshots.

• Linux systemd and Caddy examples, locked dependencies, and Linux/Windows CI.

• Online SQLite/artifact backups with exact digest and database verification, self-contained snapshots, 24 hourly/30 daily retention, and hourly systemd units.

• Fresh-directory restore invalidates credentials, passwords, sessions, reporters, ownership, and integration authorization before publication. A pause requires hold inspections, old-installation fencing, and post-snapshot gap reconciliation. Unknown jobs and physical/integration holds remain preserved. The dashboard records the checklist and can issue a fresh token for the same agent identity.

• Durable clock rollback detection expires task and reporter authority while preserving physical and integration holds. The dashboard and host command record time reconciliation; restored snapshots preserve their creation-time cutoff. Concurrent read authentication can share a newly committed clock sample, while every mutation retains its own writer-lock time and ownership checks.

• Bounded host maintenance compacts expired receipt results and redundant old running-observation summaries. Permanent request identities and semantic history remain intact. Daily systemd units provide scheduled maintenance.

• Known-prefix version-1 snapshots from schema 12 onward verify without source changes and migrate only inside a private restore copy. Native release tooling verifies package checksums, offline links, systemd/HTTPS installation, and reproducibility. The packaged server passed the sustained constrained workload below.

Deliberate limits of this milestone

Completion is enforced through workflow activities; there is no unrestricted status edit. Configure a canonical repository key and at least one required check before submitting code. Check identity, version, and environment are registered metadata, and all workstation evidence remains a client attestation. A malicious credential can lie about source or a producer; this service is not remote attestation. Independent review rejects every contributing principal as well as its sessions; it cannot establish whether separately enrolled credentials use the same model.

Jobs and submissions require clean committed source snapshots. The CLI does not upload source: publish candidate checkpoints to an appropriate Git remote and fetch them on the receiving workstation before review or integration. Logs remain bounded on the workstation and can be uploaded explicitly as service artifacts; publishing source checkpoints remains a Git operation. A guardian lost after launch may leave an unknown result even if the process has exited. Reconnection never invents an exit result; inspect the journal, surviving process, and physical resource before reconciliation.

Lifecycle-hook adapters remain later work. Workstation reports are client attestations, not remote filesystem inspection or hardware fencing. Supported commands do their work in the foreground; a launcher exiting does not verify completion of detached child or external work. Retain holds for that work.

Human administrators manage human access and agent credentials through the dashboard. Host recovery requires access to the service host and records the supplied reason. A lost token response never replays the secret: rotate the replacement credential again using its returned identity. Account-creation retries retain their original key and require the original password, which the dashboard never persists; the same administrator can authenticate again to reconcile that request safely.

Instruction version 7 adds clock reconciliation and receipt-retention guidance to the existing restore, operator, worktree/job, completion, and shared-record instructions. Existing sessions must fetch and acknowledge the new instructions before new claims.

JSON requests default to 1 MiB and can be configured lower; artifact bytes use a separate 16 MiB hard limit. Task details return the latest 50 attempts, 100 checkpoints, 50 checkouts, and bounded job/resource evidence; old records remain stored, with complete history pagination available separately. Mutation receipts replay for 30 days; expired keys remain reserved so a late retry cannot duplicate an old operation. Artifact retention cleanup and storage quota enforcement are implemented; host maintenance compacts replay payloads and redundant old health summaries. Permanent semantic events, task/handoff/lesson history, and request identities are retained; this bounds maintenance work, not total historical storage.

Lease timing uses protected service time. Hosts should maintain synchronized time; a material rollback pauses new authority until reconciliation, and a restore invalidates existing authority. A rollback during downtime that remains above the last durable observation cannot be detected without a trusted external clock. Production deployment hardware, a permanent public hostname, and an off-server backup destination remain installation or operator inputs. The native Windows workstation acceptance used a disposable Linux service and temporary HTTPS tunnel; it does not establish those production choices.

Evidence and next work

The server tests cover authentication boundaries, revocation, session isolation, instruction versions, project boundaries, dependency-cycle rollback, a concurrent claim race, exact deadline expiry, recovery fencing, stale response replay, and ownership persistence across restart. The combined smoke exercise tests the built CLI and service with two independent harnesses and two projects, isolated worktree preparation, duplicate-free reconnect, job evidence, and explicit capacity release. Job tests exercise global capacity, reporter authority after task/session loss, parent revocation, bounded renewal, terminal observations, and human reconciliation.

Workflow regressions also exercise independent review, concurrent conflicting review decisions, exact-source checks, a cross-project integration claim race, revoked ownership, manual recovery at the deadline, and complete publication reconciliation followed by a replacement integration and fresh checks.

The dashboard was checked with disposable data at desktop and phone widths: human review and completion, code review and authorization, check-roster editing, stale-candidate recovery, preserved history, responsive layout, and sign-out.

Linux workspace checks, the real service/CLI smoke exercise, native Windows client/CLI/local-runner tests, and the locked dependency audit passed for workflow implementation commit 5e6f19b in CI run 34424382043. The same revision passed all 84 local workspace tests, warnings-denied Clippy, formatting, JavaScript syntax checks, and the complete smoke exercise. No production deployment or actual off-server transfer is claimed. Later capacity and recovery evidence appears below.

Continue with BACKLOG.md and HANDOFF.md.

Shared-record review evidence

All 114 workspace tests pass locally (the full suite plus focused checks after the final context/selection additions), along with warnings-denied Clippy, formatting, JavaScript syntax, workspace build, and the final service/CLI smoke. The combined workspace tests cover atomic submission lessons/artifacts, scoped work selection, decision expiry and stale authority replay, knowledge correction and feedback, FTS applicability, historical import conflicts and reimports, export bounds/provenance, artifact quota/expiry/revocation/concurrent cleanup, and native transfer integrity. The complete built smoke exercise also covers knowledge correction/context, human-required decisions, historical apply, provenance-bearing export, exact upload retry, and download bytes/no-overwrite.

Browser verification with disposable data covered literal HTML as inert text, lesson revision 2 and feedback, denied/reopened/allowed decisions, policy-change invalidation, planned/done imported task states, context search, rule provenance, artifact links, export, project selection, and responsive layout without horizontal overflow at approximately 390 CSS pixels. No browser errors were reported. Implementation commit ca59ee0 passed Linux format/Clippy/tests/build, the full service/CLI exercise, native Windows client/CLI/local tests, and the locked dependency audit in CI run 34431398271. The earlier CI link above applies to the prior milestone.

The operator-initiated native Windows workstation acceptance in item 7 is complete.

Operator-workflow review evidence

All 129 workspace tests pass, along with warnings-denied Clippy, formatting, JavaScript syntax checks, the workspace build, and the complete built service/CLI smoke exercise. Additional regressions cover account access races, the last administrator, password/session invalidation, same-principal token rotation, reauthenticated account-creation replay, objective gates/cycles/revisions, and complete history pagination with exact workflow evidence and credential redaction.

Browser checks covered policy changes, task admission, history revisions, objective membership, inspected recovery, blocked release and human resolution, account disabling, token rotation, session revocation, password change and new sign-in. A proxy dropped an account-creation response after commit; after session expiry and reauthentication, re-entering the original password recovered exactly one account and one receipt. Final embedded assets matched their source, no browser errors were reported, and the phone view fit 375 CSS pixels without overflow.

History cursors use insertion row identifiers. Future database maintenance must preserve those identifiers or explicitly invalidate outstanding cursors; never silently reuse them after an in-place VACUUM or a restore.

Implementation commit 865365d passed Linux workspace checks and smoke, native Windows client/CLI/local-runner tests, and the dependency audit in CI run 34440915688. The final audit clarification labels host password recovery’s initiator separately from its target account and passed a focused regression and workspace Clippy.

Backup and restore review evidence

All 140 workspace tests pass, along with warnings-denied Clippy, formatting, JavaScript checks, the workspace build, and the full existing smoke exercise. New regressions cover exact manifest/database artifact membership, digest damage, missing and symlinked files, retention buckets, artifact cleanup locks, repository overlap rejection, standalone verification without parent mutation, atomic no-overwrite publication, and read-only backup schema checks.

Restore tests cover old credential/session/reporter rejection, globally reserved pre-restore mutation keys, repeated restore of a paused snapshot, preserved holds and unknown jobs, complete reconciliation gates, and fresh manual integration authorization with prior evidence retained. Independent source review found no remaining authority blocker; the main agent reviewed and tested the combined code.

The built service/CLI restore exercise completed in 6.9 seconds after the final storage changes. It captured a live database and pinned artifact, verified a copied bundle, refused an existing destination, stopped the original service, restored into an absent directory, rejected old access, recovered the administrator, preserved a checkpoint and hold, completed reconciliation, and reconnected the same agent principal with a fresh token/session and a higher recovery generation. The copied bundle was another local directory, not a real off-server destination.

Browser checks covered task evidence links, multiline inspection evidence, both reconciliation attestations, resuming coordination without releasing a hold, and same-agent credential replacement with the secret cleared before inspection. The layout fit 375 CSS pixels without horizontal overflow; no browser errors were reported. The fixture and browser tab were closed afterward.

Snapshots are full independent copies and can consume roughly 53 times live database/artifact storage, plus working space. Engine limits and the cooperative 45-minute deadline are documented in the backup contract. The rehearsal is not a production-size restore benchmark or proof of host-loss protection. Installation requires destination-side verification and a measured recovery exercise.

Backup/restore implementation 283d9d0 passed Linux workspace checks and both smoke exercises, native Windows client/CLI/local-runner tests, and the dependency audit in CI run 34445152452.

Linux operations acceptance evidence

Candidate a16d879 passed all 160 workspace tests, formatting, warnings-denied Clippy, build, both service/CLI smoke exercises, native Windows client/local-runner checks, and dependency audit in standard CI. Release CI passed byte-identical native Linux/Windows packages, Ubuntu 24.04 systemd/HTTPS installation, restart/reconnect, both timers, and verified backup.

The capacity job downloaded the accepted Linux package and verified its archive and executable identities. That exact server completed 90,000 requests over 30 minutes at 50 requests/second with 20 projects, 50 sessions, and 100,000 historical tasks. Overall p95 was 20.974 ms and p99 was 21.695 ms. Every operation met the 500 ms p95 limit; there were no unexpected errors, all 50 sessions renewed 360 times, and exact ownership and artifact checks passed. The shared two-CPU / 4-GiB / no-swap scope peaked at 664.1 MiB; server RSS peaked at 84.6 MiB. The concurrent upload and online backup overlapped during traffic. The captured snapshot restored with integrity, history, and authority invalidation verified in 6.511 seconds; that measurement stops at the reconciliation pause.

The actual prior executable upgraded a schema-12 database to schema 16 while preserving principals, credentials, sessions, an active attempt, and checkpoints. Instruction version 6 was rejected for new claims until version 7 acknowledgment. Its old snapshot verified unchanged and restored with the new executable while invalidating old credentials, sessions, and ownership. Historical search remained available. The separate small end-to-end restore smoke passed in 7.3 seconds. Browser checks covered clock pause, multiline recovery evidence, reconciliation, and a 375-pixel layout without console errors or overflow.

Main-agent review is complete. The retained acceptance evidence includes the complete sanitized capacity report and its identity. The mdBook consolidation and actual native Windows workstation exercise are complete; CI did not replace the workstation exercise. Production deployment, off-server transport, and full operator-led recovery remain installation checks.

MCP review evidence

Candidate cd9a633 passed all 182 workspace tests, formatting, warnings-denied Clippy, workspace build, and both smoke exercises. Official rmcp 3.2.0 clients exercise modern discovery and legacy initialization over real TCP. Twelve wire regressions cover authentication, revocation after admission, policy, independent review, competing claims, exact shared REST receipts, session/generation guards, and clock/lease semantics. The live launcher exercise shares one saved harness with native CLI commands and MCP, writes a checkpoint, rejects a second launcher, and verifies that connection traffic does not renew ownership.

Main-agent review is complete. Linux workspace checks and smoke, native Windows client/CLI/local-runner tests, and the dependency audit passed in CI run 34468585302. Compatibility is established for the exercised protocol and SDK paths, not every vendor GUI. OAuth-only clients lacking protected custom headers cannot connect directly. Git, job launch/reporting, and binary transfer remain native operations; the service does not execute workstation commands.

Documentation review evidence

The 35-chapter mdBook consolidates the canonical guides and preserved design records. Root live records are included at build time; prior document paths retain compatibility links. Pinned mdBook 0.5.4 checks verify exact chapter coverage, source/include rules, generated targets, fragments, and assets. Main-agent browser review covered navigation, search, live handoff content, desktop layout, and a 390-pixel phone view without page overflow or JavaScript errors.

Package checks verify checksums, offline Markdown links, bounded member counts and expanded size, and a complete book build from an extracted archive. The final local structural fixture used stripped copies of the current debug binaries; it does not replace the retained Linux release/capacity evidence. Documentation CI passed on main commit ebf72f7 in documentation run 34487173043. The same commit passed Linux formatting, warnings-denied Clippy, all workspace tests, the workspace build, both service/CLI smoke exercises, native Windows client/CLI/local-runner tests, and the locked dependency audit in coordination run 34487175859. The earlier billing-blocked attempts ran no steps and are superseded by these passing runs. Main review and the hosted validation follow-up are complete. The operator-initiated Windows workstation exercise in final item 7 is complete. Local-file book navigation and search passed with networking disabled.

Native Windows workstation acceptance evidence

The accepted agent-coordinator 0.1.0 Windows x86-64 CLI from release run 34459987847 ran natively on workstation MINIAIR against a disposable service on Linux workstation mxmini through a temporary Cloudflare HTTPS tunnel. The archive SHA-256 was be5e96ec6f8843a63c8be319d9b4f9745f4e0870bebbbc160c2c24f69dc32a1d, matching the downloaded checksum, and the exercised executable SHA-256 was 40240c59aeaf012aba6721f378e9dec57419ceb8833aec0187391c3bccb1111d.

Distinct Windows and Linux principals and sessions exercised two isolated projects. Windows completed a claim, checkpoint, and general submission in the second project. The contributing Windows principal could not perform its own independent agent review; the Linux principal reviewed and approved the exact submission, which completed the subject while the first project’s race task remained isolated.

For the cross-workstation race, Linux and Windows submitted an ordinary claim for the same task and revision at a published barrier. Linux received generation 2 ownership and Windows received claim_conflict against that existing attempt. Linux checkpointed the observed one-owner result and released the task ready. For recovery, Windows checkpointed a separate task and closed its dedicated session; Linux then claimed recovery, recorded its inspection, and released the task ready without reviving the closed Windows session. Session and credential counts remained stable through Linux completion.

The sanitized rendezvous and completion records are retained in commit c017a69. The one-time encrypted credential handoff was deleted after use, and the disposable public service and tunnel were shut down after the exercise. This evidence is specific to the exercised workstations, accepted package, temporary route, and disposable data. It is not a claim of production deployment, permanent network exposure, hardware attestation, or off-server backup protection.

Agent Coordinator CLI

The native agent-coordinator command connects an existing harness to one Agent Coordinator project. It stores no secret in the repository and never claims work as a side effect of connect.

Repository and credential configuration

Commit a non-secret .agent-coordinator.toml at the repository root:

service_url = "https://coordinator.example.com"
project_id = "project-id-from-the-operator"

The file accepts only those two fields. In particular, a token in this file is rejected. Set the token in the process environment:

export AGENT_COORDINATOR_TOKEN='token-issued-by-the-operator'
export AGENT_COORDINATOR_ORIGIN='https://coordinator.example.com'

Or create credentials.toml under the platform’s private Agent Coordinator configuration directory. On Linux the default is $XDG_CONFIG_HOME/agent-coordinator/credentials.toml, or $HOME/.config/agent-coordinator/credentials.toml when XDG_CONFIG_HOME is unset. On Unix, AGENT_COORDINATOR_HOME overrides the directory for headless installs. The override is rejected on Windows because an arbitrary directory cannot be assumed to have a private user ACL; Windows uses the current user’s platform configuration directory.

[[credentials]]
origin = "https://coordinator.example.com"
token = "token-issued-by-the-operator"

On Unix, the CLI refuses a credential file readable by group or other users; set mode 0600. On Windows, keep the file within the current user’s protected profile and restrict its ACL to that user. The configured origin is matched exactly after normalization, so changing a repository binding cannot redirect an existing credential to another service.

Every independent harness must use a distinct, stable session name. Pass it on every ownership command or configure it in that harness’s environment:

export AGENT_COORDINATOR_SESSION='codex-task-42'

PowerShell:

$env:AGENT_COORDINATOR_TOKEN = 'token-issued-by-the-operator'
$env:AGENT_COORDINATOR_ORIGIN = 'https://coordinator.example.com'
$env:AGENT_COORDINATOR_SESSION = 'codex-task-42'

An environment token is used only when AGENT_COORDINATOR_ORIGIN independently matches the repository service origin. The session proof and pending request state are saved outside the repository in the platform configuration directory. Separate session names always select separate files and therefore cannot silently share attempt ownership.

For local development only, --allow-insecure-loopback permits http://localhost or a loopback IP address. Other HTTP origins are rejected. Redirects are never followed, and the CLI accepts only /api/v1 and /healthz request paths.

Foundation workflow

Connect creates a harness session on its first run and reconciles the saved session on later runs. It returns the session and the bound project’s complete orientation packet:

agent-coordinator connect

List projects or tasks without reserving anything:

agent-coordinator projects list --limit 50
agent-coordinator tasks list --limit 50
agent-coordinator tasks list --limit 50 --cursor opaque-cursor-from-the-response

Create requests read JSON from a file, which avoids shell-specific quoting. Project creation requires an authorized operator role and is normally performed through the human web interface; an ordinary agent credential receives a clear permission error and exit code 4.

agent-coordinator projects create --input new-project.json
agent-coordinator tasks create --input new-task.json

agent-coordinator.exe tasks create --input .\new-task.json

Use --input - to read a JSON object from standard input. Claiming performs the required acknowledgment for the exact complete orientation returned by connect, then submits one claim. It never acknowledges incomplete instructions.

agent-coordinator claim --next
agent-coordinator claim --task task-id --revision 4
agent-coordinator claim --mode recovery --task expired-task-id --revision 7

--mode work is the default. Recovery mode atomically claims an expired or revoked attempt for inspection; it does not permit edits until recovery resolve records the required saved-work and running-job checks. --next can select the next eligible item in either mode.

Lease and checkpoint operations name the attempt and ownership generation explicitly. Checkpointing does not renew a lease, and release does not mark a task complete.

agent-coordinator renew --attempt attempt-id --generation 2
agent-coordinator checkpoint --attempt attempt-id --generation 2 --input checkpoint.json
agent-coordinator release --attempt attempt-id --generation 2 --input release.json

Worktrees

Before changing code, prepare and register a separate clean worktree. The CLI first checks that the named attempt and generation still grant current work authority. It then verifies that the source checkout is clean and has a Git remote matching the repository URL configured for the bound project. The project is never inferred from a directory name.

agent-coordinator worktree prepare \
  --attempt attempt-id --generation 2 \
  --source /srv/src/project \
  --path "/srv/worktrees/task 42" \
  --branch agent/task-42 --base origin/main

agent-coordinator.exe worktree prepare `
  --attempt attempt-id --generation 2 `
  --source 'C:\src\project' `
  --path 'C:\agent worktrees\task 42' `
  --branch agent/task-42 --base origin/main

The destination’s parent must exist, while the destination itself must be new. Before invoking Git, the CLI saves the exact source, destination, branch, base selector, resolved full base commit, attempt, and generation outside the repository. Repeating the same command reconciles that saved destination even if a branch such as origin/main later advances. Different arguments are refused. The command never resets, stashes, deletes, or cleans any checkout. It records the resolved per-worktree Git directory identity, full base commit, branch, path, workstation, and clean state with the service. A lost registration response remains a normal durable pending mutation and retry reuses its body and idempotency key.

Resources

Human administrators create canonical resource definitions through the web or administrative API. Agents can list those definitions and atomically reserve a complete set for their current attempt:

agent-coordinator resources list --limit 50
agent-coordinator resources reserve \
  --attempt attempt-id --generation 2 --input reservation.json
agent-coordinator reservations list --limit 50

{
  "items": [
    {"resource_id": "resource-id", "units": 1},
    {"resource_id": "second-resource-id", "units": 2}
  ]
}

The reservation is all-or-none. An expired attempt, disconnected observer, or missing heartbeat does not release a physical resource. Once every attached job has a terminal producer result, the owning agent can release it explicitly:

agent-coordinator resources release \
  --reservation reservation-id --generation 2 \
  --reason "All attached producers have exited"

agent-coordinator.exe resources release `
  --reservation reservation-id --generation 2 `
  --reason 'All attached producers have exited'

reservations release is an equivalent spelling. Uncertain physical resources require an explicit human resolution through the web or administrative API.

Durable local jobs

jobs run accepts a local program and argument vector from JSON. It requires a held reservation and the worktree prepared for the same current attempt. The worktree must be clean at a committed revision; the job record captures the full commit and tree identities before registration.

{
  "label": "workspace tests",
  "program": "/home/agent/.cargo/bin/cargo",
  "argv": ["test", "--workspace", "--locked"],
  "environment": {
    "PATH": "/home/agent/.cargo/bin:/usr/local/bin:/usr/bin:/bin",
    "HOME": "/home/agent",
    "CARGO_HOME": "/home/agent/.cargo",
    "RUSTUP_HOME": "/home/agent/.rustup",
    "TMPDIR": "/tmp",
    "CARGO_TERM_COLOR": "never"
  },
  "check_identity": "workspace-tests",
  "check_version": "v1",
  "check_environment": "linux-x86_64",
  "log_limit_bytes": 1048576
}

agent-coordinator jobs run \
  --attempt attempt-id --generation 2 \
  --reservation reservation-id \
  --checkout "/srv/worktrees/task 42" \
  --input job.json

PowerShell uses the same JSON shape; program must be an absolute native path:

{
  "label": "workspace tests",
  "program": "C:\\Users\\agent\\.cargo\\bin\\cargo.exe",
  "argv": ["test", "--workspace", "--locked"],
  "environment": {
    "PATH": "C:\\Users\\agent\\.cargo\\bin;C:\\Windows\\System32",
    "USERPROFILE": "C:\\Users\\agent",
    "CARGO_HOME": "C:\\Users\\agent\\.cargo",
    "RUSTUP_HOME": "C:\\Users\\agent\\.rustup",
    "SystemRoot": "C:\\Windows",
    "TEMP": "C:\\Users\\agent\\AppData\\Local\\Temp",
    "TMP": "C:\\Users\\agent\\AppData\\Local\\Temp"
  },
  "check_identity": "workspace-tests",
  "check_version": "v1",
  "check_environment": "windows-x86_64-msvc",
  "log_limit_bytes": 1048576
}

agent-coordinator.exe jobs run `
  --attempt attempt-id --generation 2 `
  --reservation reservation-id `
  --checkout 'C:\agent worktrees\task 42' `
  --input .\job.json

Before registration or launch, the CLI durably saves random job, producer, runner, and reporter identities. It then stores the scoped reporter bearer only in the protected local job file. The detached guardian and producer clear their inherited environment, retaining only PATH, SystemRoot, WINDIR, TEMP, TMP, TMPDIR, SSL_CERT_FILE, and SSL_CERT_DIR when present. The producer also receives the environment explicitly listed in the JSON file, overriding those defaults. AGENT_COORDINATOR_* and COORDINATOR_* variables are rejected. Raw arguments, environment, and logs are never uploaded. Output and error streams stay in separate, bounded, protected local files shown by jobs inspect. Set log_limit_bytes to 0 to discard both streams; the maximum is 67108864 bytes per stream.

Replace the example environment values with the workstation’s actual trusted paths. Configure additional values the program and its children require, such as PATH, the user profile, toolchain directories, temporary directories, and Windows SystemRoot. Do not copy coordinator token, origin, session, or proof variables into this map.

To allow bounded attempt renewal while the job runs, name the exact current harness process and a server-limited window:

agent-coordinator jobs run ... --renew-for-seconds 1800 --watch-pid 12345

Renewal stops when that exact process exits, its authority expires, or the window ends. Job observation may continue after renewal stops. Omitting those options runs and reports the job without delegated attempt renewal.

Use the returned job ID for later inspection. Remote status and list commands do not launch or reconnect any producer:

agent-coordinator jobs list --limit 50
agent-coordinator jobs status --job job-id
agent-coordinator jobs inspect --job job-id
agent-coordinator jobs reconnect --job job-id

jobs inspect reads protected local state. jobs reconnect starts a detached observation-only guardian and works while the service is temporarily unavailable; pending observations remain durable for later delivery. It never launches a producer. If a durable launch intent exists without a recorded process identity, the state becomes unknown and the guardian refuses to launch again. A missing local journal is also treated as uncertain because the producer may already have run. PID alone is never used as producer identity.

The three check_identity, check_version, and check_environment fields are optional for ordinary jobs and must be supplied together. For a required integration check, copy the exact tuple from the project’s human-managed workflow policy and add --activity so the CLI verifies that the checkout is at that activity’s exact prepared result:

agent-coordinator checks list

agent-coordinator jobs run \
  --activity integration-activity-id \
  --attempt integration-attempt-id --generation 1 \
  --reservation reservation-id \
  --checkout "/srv/worktrees/integration 42" \
  --input required-check.json

Omit all three check fields for an ordinary producer. The service creates check receipts from registered terminal jobs. A later integration command selects those receipts by job ID; the CLI never accepts a caller-authored pass result.

Immutable submissions

Submission evidence JSON contains only the bounded result narrative. Every current acceptance criterion must appear exactly once:

{
  "summary": "Implemented and validated the requested behavior.",
  "acceptance_evidence": [
    {
      "criterion": "The exact criterion text from the task",
      "evidence": "The exact behavior and check that demonstrate it"
    }
  ],
  "handoff": "Review the candidate and its registered job evidence."
}

For code work, the CLI reads the repository, base commit, candidate commit, and candidate tree from the attempt’s registered, clean prepared worktree. These fields cannot be supplied through JSON:

agent-coordinator submissions code \
  --attempt attempt-id --generation 2 \
  --task-revision 4 --project-policy-revision 3 \
  --workflow-policy-revision 2 \
  --checkout "/srv/worktrees/task 42" \
  --input submission.json

agent-coordinator.exe submissions code `
  --attempt attempt-id --generation 2 `
  --task-revision 4 --project-policy-revision 3 `
  --workflow-policy-revision 2 `
  --checkout 'C:\agent worktrees\task 42' `
  --input .\submission.json

The service requires the attempt to be current and quiescent: release its reservations after every attached producer is terminal before submission. Submission ends the implementation attempt and creates the review and integration activities required by the pinned policies. It does not itself mark a code task complete.

A submission records commit and tree identities; it does not upload Git objects. Before submitting work that another workstation must review or integrate, push the candidate commit to a durable remote checkpoint ref that the other workstation can fetch. For example:

git -C "/srv/worktrees/task 42" push origin \
  HEAD:refs/agent-coordinator/candidates/submission-id

git -C 'C:\agent worktrees\task 42' push origin `
  'HEAD:refs/agent-coordinator/candidates/submission-id'

On another workstation, fetch that exact ref before claiming review or integration work:

git -C /srv/src/project fetch origin \
  refs/agent-coordinator/candidates/submission-id

The checkpoint ref name is an operator convention and is not created by the coordinator CLI. Keep it available until review and integration finish. The service stores identities and evidence only; sharing a local object database between worktrees is sufficient on one machine but does not transfer source to another machine.

General work uses the same evidence JSON without any Git fields:

agent-coordinator submissions general \
  --attempt attempt-id --generation 1 \
  --task-revision 2 --project-policy-revision 3 \
  --input submission.json

The CLI fixes the workflow-policy revision to zero for general work. A general task with no configured review completes atomically; otherwise it waits for its new immutable-submission review activities.

Review activities

List the current subject workflow, then claim one exact agent-review activity. A review list grants no ownership:

agent-coordinator reviews list --task subject-task-id
agent-coordinator reviews status --activity review-activity-id
agent-coordinator reviews claim \
  --activity review-activity-id --submission submission-id \
  --project-policy-revision 3 --workflow-policy-revision 2

The returned attempt ID and generation are required for renew, release, and decision commands:

agent-coordinator reviews renew \
  --activity review-activity-id \
  --attempt review-attempt-id --generation 1
agent-coordinator reviews release \
  --activity review-activity-id \
  --attempt review-attempt-id --generation 1 \
  --input release.json

Activity release JSON uses the activity-specific handoff contract:

{
  "summary": "Saved work and evidence are ready for the next owner.",
  "blocked": false
}

A review decision JSON contains only the decision evidence:

{
  "decision": "approved",
  "summary": "The candidate satisfies the acceptance criteria.",
  "findings": [
    {
      "severity": "advisory",
      "remedy": "Consider simplifying the helper in a later change.",
      "evidence": "The current form is correct and covered by the registered check."
    }
  ]
}

agent-coordinator reviews decide \
  --activity review-activity-id \
  --attempt review-attempt-id --generation 1 \
  --submission submission-id --input review.json

decision is approved or changes_requested; finding severity is required or advisory. An agent command cannot record a human review. The service checks the authenticated principal and independent contributor history, so changing a token or session cannot turn a contributor into an independent reviewer.

Integration activities

After every required approval and any human publication authorization, list and claim the exact integration activity:

agent-coordinator integrations list --task subject-task-id
agent-coordinator integrations status --activity integration-activity-id
agent-coordinator integrations claim \
  --activity integration-activity-id --submission submission-id \
  --project-policy-revision 3 --workflow-policy-revision 2

Prepare and register a separate worktree for the returned integration attempt, using the observed full target commit as its base. Then prepare the deterministic candidate-containing result:

agent-coordinator worktree prepare \
  --attempt integration-attempt-id --generation 1 \
  --source /srv/src/project \
  --path "/srv/worktrees/integration 42" \
  --branch agent/integrate-42 \
  --base 0123456789abcdef0123456789abcdef01234567

agent-coordinator integrations prepare \
  --activity integration-activity-id \
  --attempt integration-attempt-id --generation 1 \
  --submission submission-id \
  --checkout "/srv/worktrees/integration 42" \
  --expected-target 0123456789abcdef0123456789abcdef01234567 \
  --candidate 89abcdef0123456789abcdef0123456789abcdef

agent-coordinator.exe integrations prepare `
  --activity integration-activity-id `
  --attempt integration-attempt-id --generation 1 `
  --submission submission-id `
  --checkout 'C:\agent worktrees\integration 42' `
  --expected-target 0123456789abcdef0123456789abcdef01234567 `
  --candidate 89abcdef0123456789abcdef0123456789abcdef

Preparation checks fresh activity and attempt authority before local Git work, persists the exact request before creating the result, registers immutable publication intent with the exact prepublication target revision and tree before any push, and materializes the result in the isolated checkout for exact-source checks. It never updates the target branch.

Run every required check against that materialized checkout, wait for terminal successful job receipts, and release the check reservations. The checkout must remain clean at the exact prepared result. Publish with:

agent-coordinator integrations publish \
  --activity integration-activity-id \
  --attempt integration-attempt-id --generation 1

Immediately before Git starts, the library observes the configured remote, calls the service for fresh exact activity, candidate, intent, and remaining lease authority, observes the remote again, durably records push intent, and uses Git’s exact force-with-lease compare-and-swap. A moved target is reported without publishing. Once push intent is durable, a retry is observation-only and never repeats an uncertain push.

After interruption, inspect the remote without any publication capability:

agent-coordinator integrations reconcile \
  --activity integration-activity-id \
  --attempt integration-attempt-id --generation 1

This local reconciliation can confirm the exact prepared result, report that the target moved, or preserve an uncertain outcome. Only an authenticated human can record the service-side disposition of an uncertain or known nonpublication. If the remote moved to an object the trusted checkout does not contain, fetch that exact remote target ref into the checkout and reconcile again so its tree can be observed before a human records evidence. If the target ref was deleted, stop and have an operator restore or repair it; do not substitute a zero commit or continue the normal integration workflow.

When publication is confirmed, finish with an exact set of successful registered check jobs:

{
  "submission_id": "submission-id",
  "check_job_ids": ["job-id-linux", "job-id-windows"],
  "summary": "The exact integrated result passed the pinned check roster."
}

agent-coordinator integrations finish \
  --activity integration-activity-id \
  --attempt integration-attempt-id --generation 1 \
  --input integration-finish.json

finish first registers the immutable integration result with those job IDs. It then observes the configured remote again and asks the service to finalize the exact published commit and tree. The service resolves job IDs to terminal producer receipts and checks the pinned identity/version/environment roster; the JSON cannot fabricate a result. Finalization atomically completes the integration activity and subject task and releases the canonical target hold. Use integrations renew and integrations release with the same argument shapes as their review equivalents while the activity remains owned. A release never clears an uncertain publication hold.

Recovery inspection

Inspect the expired attempt, its registered checkout, jobs, and resource holds, then record one explicit disposition from a JSON file:

agent-coordinator recovery inspect --attempt expired-attempt-id
agent-coordinator jobs list --limit 50
agent-coordinator reservations list --limit 50
agent-coordinator recovery resolve \
  --attempt recovery-attempt-id --generation 3 --input recovery.json

{
  "saved_work_checked": true,
  "running_jobs_checked": true,
  "disposition": "resume",
  "summary": "The exact producer is terminal and the saved commit was inspected."
}

Use the disposition and inspection fields required by the current service orientation and attempt detail. Recovery does not cancel or restart jobs and does not clear a resource merely because an observer or lease expired.

The bounded generic command covers another implemented foundation API path without inventing commands for future workflow surfaces:

agent-coordinator request --method get --path /api/v1/projects
agent-coordinator request --method post --path /api/v1/projects/project-id/claims --input claim.json

Add --json for the complete compact service envelope. The default presents concise tables and labeled summaries while preserving full instructions during connect.

Interrupted mutations

Before every POST or PATCH, the CLI durably stores its method, path, JSON body, and newly generated idempotency key. A transport failure, redirect, rate limit, or server failure leaves that exact request pending and blocks unrelated writes in the same harness session. Resolve it with:

agent-coordinator retry

retry sends the saved body with the saved key. It never manufactures a new key after a response may have been lost. A definitive success or client/state error clears the pending request. Claim and renewal replay output is passed through unchanged, including the service’s fresh current_authority; callers must not treat timing from the original receipt as current authority.

Exit codes are 0 for success, 2 for invalid input, 3 for authentication or setup required, 4 for denied operations, 5 for state or ownership conflicts, 6 for unsatisfied requirements, and 7 for transport or temporary service failures.

Repository bootstrap text

Repositories can place this short bootstrap in AGENTS.md, CLAUDE.md, or the equivalent harness instruction file after the CLI workflow has been installed:

This project coordinates work through Agent Coordinator.
Read .agent-coordinator.toml and run `agent-coordinator connect --json` with a
stable session name unique to this harness. Read the complete orientation and
all required instruction sections before claiming work. Reconnect or show the
returned setup/error details when coordination is unavailable; do not silently
fall back to uncoordinated work. Claim a task before changing code, then renew,
checkpoint, and release it with its exact attempt ID and ownership generation.

Shared knowledge, decisions, and Markdown records

These commands operate on the bound project. All writes use the existing protected session journal and retain the exact request and key for retry. JSON input is limited to 1 MiB. Read results use human-readable output by default; --json preserves the complete response envelope and cursor.

agent-coordinator knowledge list --limit 50
agent-coordinator knowledge show --id LESSON_ID
agent-coordinator knowledge create --input lesson.json
agent-coordinator knowledge edit --id LESSON_ID --input correction.json
agent-coordinator knowledge feedback --id LESSON_ID --input feedback.json
agent-coordinator context --query "producer identity" --budget 65536
agent-coordinator context --query "cache" --component compiler --include-shared
agent-coordinator decisions list
agent-coordinator decisions show --id DECISION_ID
agent-coordinator decisions create --input decision.json
agent-coordinator decisions answer --id DECISION_ID --input answer.json
agent-coordinator decisions reopen --id DECISION_ID --input reopen.json

For example, lesson.json contains:

{
  "kind": "lesson",
  "title": "Check the producer before repeating a test",
  "body": "A missing observer does not prove that its producer stopped.",
  "status": "observed",
  "scope": {"components": ["test-runner"]},
  "tags": ["recovery"],
  "applicability": "When reconnecting after an observer disconnects",
  "provenance": {"summary": "Observed during task recovery; retain the source revision here"}
}

Corrections include expected_revision and the new title, body, status, scope, applicability, tags, and provenance. Prior revisions remain available. Explicit cross-project sharing requires collection: "shared" and share_across_projects: true when creating the record; access to all projects does not automatically share every lesson. Search treats input as plain words, returns complete binding rules, and enforces its byte budget. A budget too small for the rules returns an explicitly incomplete packet and asks for a larger budget. Scope filters include --task-id, --component, --environment, and --version.

Decision creation records the required actor type, exact affected task revisions, project policy revision, options, environment, conditions, and optional expiration in Unix milliseconds. An answer must exactly match an option and include a typed disposition (allow, deny, or defer), rationale, and expected_generation. Only an allow with conditions_confirmed: true can satisfy the work gate. A changed task/policy, expired answer, denial, or deferral keeps work blocked until an authorized reopening and answer. Agent credentials cannot answer decisions requiring a human; use the dashboard. Decision answers do not change project permissions or replace integration authorization.

Submissions may include lessons and artifact_ids in their evidence JSON. Each new lesson uses the lesson fields above, with provenance_summary in place of provenance; the service supplies its task and immutable submission source. The submission, handoff, original lesson revisions, and finalized artifact references commit together. Later lesson corrections preserve the original submission snapshot.

agent-coordinator imports preview --input source.json
agent-coordinator imports show --id PREVIEW_ID
agent-coordinator imports apply --id PREVIEW_ID --input apply.json
agent-coordinator export --json --limit 50

Import requests supply source (stable context, Git revision, observation time, branch, environment), bounded chunks containing arbitrary source paths and Markdown text, and optional explicit historical_mappings. The service never reads workstation paths. Agent credentials may preview and export; applying historical migration requires a human session in the dashboard, because checked source items create historical closure outside the live completion workflow. Inspect the preview’s proposed records, conflicts, unresolved links, and digest before applying it. The apply request contains preview_digest and expected_project_event_revision from that exact preview; intervening service changes require a fresh preview. Ordinary prose remains historical context, unchecked checklist tasks start planned, and completed imported items retain closure. Imported agent guidance never adopts binding rules. See import contract for source identities and limits.

Export without --json prints generated Markdown and the next cursor, when present. JSON output separates data.markdown and data.next_cursor for scripts. Pass --cursor to continue; a changed snapshot requires restarting the export. Generated exports contain provenance and cannot be imported over authoritative records. The dashboard can download a consistent snapshot up to 16 MiB; larger exports should be collected as bounded CLI pages.

For uploaded evidence, first reserve exact size and SHA-256 using artifacts reserve --input reservation.json, then run artifacts upload --id ARTIFACT_ID --file report.txt. Repeat the same command after an interrupted request; its protected journal keeps exact bytes and the original key. A completed retry authenticates again and reports current availability; it does not upload again. Download with artifacts download --id ARTIFACT_ID --output new-report.txt; the destination must not exist. Downloaded size and SHA-256 are verified before publishing the file. Transfers are bounded to 16 MiB and ordinary JSON responses are bounded to 16 MiB. Use --json for the full transfer receipt.

Operator and objective commands

The native client authenticates as an agent; administrative human actions use the browser or the host-local service command. Native commands retain service role checks and the same exact-request mutation journal as claims.

agent-coordinator tasks show --id TASK_ID
agent-coordinator tasks edit --id TASK_ID --input revised-task.json
agent-coordinator tasks history --id TASK_ID --kind attempts --limit 50
agent-coordinator policy show
agent-coordinator policy history --limit 50
agent-coordinator policy edit --input revised-policy.json
agent-coordinator objectives list
agent-coordinator objectives create --input objective.json
agent-coordinator objectives show --id OBJECTIVE_ID
agent-coordinator objectives children --id OBJECTIVE_ID --input membership.json

Use --cursor from a returned page for task history, task lists, objective lists, and policy history. History kinds include attempts, checkpoints, checkouts, jobs, job_observations, resources, artifacts, submissions, reviews, integrations, task_revisions, and events. History pages bind the project, task, record kind, and insertion snapshot; start a new query to include new records.

Task edits provide expected_revision, title, description, acceptance_criteria, priority, depends_on, and planned. Only unowned open or planned tasks may be edited, subject to workflow and policy checks. Changing planned to false admits work; it does not bypass dependencies or objective gates.

Objective creation provides a title, description, acceptance criteria, priority, planned flag, and children entries containing task_id and required. Child replacement provides expected_revision from the objective’s objective_revision and the full children list. Required children must finish before the objective’s own work and completion evidence; optional children remain visible. The objective is a general task with its own review, never an automatic done rollup.

Use the dashboard’s Access page for human accounts, agent token rotation, and browser session revocation. My account changes the current operator’s password and ends all their browser sessions. An administrator with local host access can recover a lost password without publishing a recovery endpoint:

agent-coordinator-server --database /path/to/coordinator.sqlite3 \
  recover-operator-password --username OPERATOR --reason 'Verified local account recovery'

The command prompts invisibly for the new password twice, or accepts --password-stdin. It enables the human account and ends its browser sessions; existing agent credentials, producers, resource holds, and saved work remain subject to their original authority and recovery rules. Never include a password in command arguments or the audited reason.

MCP connection guide

Agent Coordinator exposes a vendor-neutral Streamable HTTP endpoint at /mcp. It uses the same agent credential, durable harness session, project policy, ownership generations, clock safeguards, and mutation receipts as the REST API. Connecting or registering a harness session does not claim work.

The endpoint is a stateless MCP transport. It does not issue or depend on an MCP transport session. X-Coordinator-Session identifies the durable Agent Coordinator harness session and is unrelated to Mcp-Session-Id. Disconnecting or reconnecting an MCP client does not renew a lease, close a coordinator session, or restore expired authority.

Configure authentication

Configure the MCP server in machine- or user-level client settings, outside the repository. Use the exact trusted HTTPS service origin followed by /mcp. Supply these headers on every MCP HTTP request:

The token and session proof must come from the client’s environment or secret store. Never put either value in a URL, repository file, MCP tool argument, checked-in MCP configuration, shell history, log, or conversation. A session ID is not a credential, but it must be a new stable ID for this harness and must not be copied from another harness. Do not forward any authentication header across an origin-changing redirect.

The service currently supports its configured bearer credentials. It does not claim MCP OAuth discovery or dynamic client registration support. A client that requires OAuth and cannot supply a bearer token plus the two coordinator headers cannot connect directly in this release.

For example, Codex supports bearer tokens and HTTP headers sourced from environment variables. Put this in the user-level config.toml, not a repository .codex/config.toml:

[mcp_servers.agent_coordinator]
url = "https://coordinator.example.com/mcp"
bearer_token_env_var = "AGENT_COORDINATOR_MCP_TOKEN"

[mcp_servers.agent_coordinator.env_http_headers]
X-Coordinator-Session = "AGENT_COORDINATOR_MCP_SESSION_ID"
X-Coordinator-Session-Proof = "AGENT_COORDINATOR_MCP_SESSION_PROOF"

The safest way to supply those variables while retaining native CLI continuity is the packaged launcher. First connect the stable harness session for this repository binding, then launch one trusted local MCP client executable:

agent-coordinator --session mcp-work-42 connect \
  --harness codex-mcp --capability code
agent-coordinator --session mcp-work-42 mcp-client -- \
  /absolute/path/to/codex

Use an absolute path to the intended executable and keep all program arguments free of credentials. Keep that client in the foreground for the launcher’s lifetime; the launcher does not supervise a daemon or background process. The launcher locks out a second foreground MCP client using the same harness, validates the repository binding, credential digest, protected session state, and live remote session, then supplies the exact token, session ID, proof, origin, project, and non-secret MCP URL through the child environment. It does not print or create another copy of a credential. It releases the normal session-state lock before starting the child, so that trusted client can invoke native CLI commands with the same --session identity. The launcher also passes the absolute repository-binding path, including when the client or one of its CLI children runs from another directory. The URL in the MCP client’s user configuration must exactly equal the binding’s trusted service origin plus /mcp. An already-running desktop or IDE process does not acquire a newly launched process’s environment; restart that client through the launcher. If the launcher crashes, inspect whether its child is still running before relaunching; release of the local lock does not prove the child stopped.

One launcher session is bound to one repository and project for native CLI work. MCP tools can address any project explicitly, but native operations in a different repository require that repository’s binding and a separately connected local harness and launcher.

This example relies only on client-side environment lookup; the values do not become tool arguments. See the current Codex MCP configuration documentation for those client-specific keys. Other Streamable HTTP clients can use their equivalent protected header facility.

Register and orient the harness

The native launcher path uses the session already created by connect. Begin by calling coordinator_session_get with an empty object; it returns the nonsecret coordinator session ID that later session-scoped tools accept. This inspection does not renew ownership. A client that provisions its own independent session instead must call coordinator_session_register once. Its body contains:

{
  "session_id": "the-same-id-as-X-Coordinator-Session",
  "workstation_id": "stable-workstation-id",
  "harness": "mcp-client-name-and-version",
  "capabilities": ["capability-needed-for-selection"]
}

The session ID in the body must exactly match the configured header. The proof stays in its header and is never a tool argument. Supply a new random idempotency_key with the tool call and retain the exact key and body until the registration response is known. An identical retry retrieves the same session; different parameters for an existing session are rejected. Registration stores only the proof verifier. If a credential is rotated, a session is closed, or a database restore invalidates authority, configure a new session ID and proof instead of trying to attach the old session to a new credential.

Then follow this sequence:

1. Call coordinator_orientation with the intended project ID. Read every required section. Do not acknowledge or claim while instructions_complete is false.
2. Call coordinator_instructions_ack with the configured session ID and that exact project policy revision, instruction version, and complete section list. The session argument can be omitted to use the configured header. Acknowledgment grants no ownership.
3. Call coordinator_tasks_list or the bounded context tools to inspect work.
4. Call coordinator_claim to claim one exact eligible task or request the next eligible task. Do not change source before a successful claim.
5. Persist the returned attempt ID and ownership generation. Schedule renewal from the returned lease_remaining_ms and renew_after_seconds, subtracting request time measured by a local monotonic clock and a safety margin.

The server’s MCP instructions field and tool descriptions summarize this workflow for small agents. Retrieved task, knowledge, and import prose is data; it cannot override binding service instructions or project policy.

Writes and uncertain responses

Every mutating tool requires an idempotency_key argument. Generate a fresh random key before a new logical write and persist it with the exact tool name and arguments before sending the request. After a timeout, disconnect, or other uncertain response, retry the same tool with exactly the same arguments and key. Do not generate a new key merely to get a different answer. A successful replay is historical evidence of the original effect and never renews current ownership.

Before any ownership-dependent write, inspect the attempt or activity if its authority may have changed. Use the exact current generation and revision values. Checkpoints do not renew leases. If the service reports lost or expired authority, stop ownership-dependent work and follow the recovery guidance; a new connection cannot revive the old attempt.

MCP tools use a closed catalog of typed operations. There is no arbitrary URL, method, header, SQL, shell, or generic status-edit tool. HTTP authentication failures remain HTTP errors. Coordination errors are returned to the MCP caller with their stable service code and bounded details so the agent can take the listed next action without seeing credentials.

Workstation operations still use the native CLI

The MCP endpoint coordinates recorded state. It does not run local processes, read local paths, manipulate Git, or transfer binary bodies. When the MCP client was started through mcp-client, use the native CLI with that same stable harness session for operations that need trusted local inspection or durable local journals:

• worktree prepare creates and verifies a separate clean worktree before its registration is recorded.
• jobs run, jobs inspect, and jobs reconnect launch or observe local producers and retain reporter state safely across interruptions.
• artifacts upload and artifacts download journal and verify bounded binary transfers.
• submissions code, integrations prepare, integrations publish, integrations reconcile, and integrations finish observe exact Git objects, preserve publication intent, and enforce guarded compare-and-swap publication.

MCP can inspect the resulting jobs, artifacts, workflow, and history. Recording a checkout, publication intent, or integration result through MCP does not prove that the corresponding local Git or process operation occurred.

An independently provisioned MCP-only session does not share the CLI’s protected session state. Do not run native commands against work owned by that session unless the native client was securely supplied the exact same session identity and proof. A similar harness name is not sufficient, and another session cannot borrow its attempt authority.

Interoperability expectations

A compatible client must support Streamable HTTP, per-request custom headers, JSON Schema tool inputs, JSON tool results, and ordinary HTTP authentication errors. Clients should preserve unknown response fields and opaque pagination cursors. They must tolerate a stateless server that does not issue Mcp-Session-Id and must not infer coordinator authority from MCP connection state. Each inner REST JSON result is bounded to 1 MiB before it is represented as both structured MCP content and equivalent text, so a client’s encoded MCP message limit must allow for that representation.

Release validation exercises the endpoint over an ephemeral TCP listener with the official Rust MCP client. It covers modern discovery and legacy initialization, tool discovery, session registration and lookup, an authenticated claim, a rejected credential, and a fresh transport reconnect that preserves identity without renewing ownership. The service’s route tests separately cover authorization, policy, revision, idempotency, lease, and recovery behavior behind the same tool catalog.

Operating projects and accounts

The dashboard uses the same authenticated, revision-checked service operations as the native client. It does not bypass leases, reviews, completion checks, or uncertain physical resource holds.

Accounts and credentials

Sign in with a local human account. Every account can access every project. Administrators see Access, where they can create human accounts, change their role or enabled state, inspect browser sessions, and issue or rotate agent tokens. At least one administrator must remain enabled. An access change invalidates all existing browser sessions for the affected human account.

My account changes your password and lists your browser sessions. A password change invalidates every browser session for your account. If its response is lost, sign in with the new password and inspect your account; an old session cannot retry a password change to regain access. Passwords are never saved in browser session storage. For uncertain account creation, the request key and non-secret fields are retained; after a reload, re-enter the original initial password to retry. If the account’s password has since changed, inspect the existing account instead of reusing its original creation request.

Agent rotation issues a new token for the same agent principal. This preserves contributor identity for independent-review checks. By default rotation revokes the old token and its sessions. Choose a staged transition explicitly if the old token must remain active temporarily. Copy the replacement token while it is shown; replay returns its identity without its secret. If the token was lost, rotate the new replacement credential itself. Do not revoke it before rotation.

For lost human passwords, the service host’s local recover-operator-password command replaces the password, enables the existing account, and revokes its browser sessions. It requires an audit reason and accepts a hidden prompt or standard input for the password. See the CLI guide. There is no remote password-recovery or public enrollment endpoint.

Project policy and tasks

Open a project’s task queue and select Project policy to edit review mode, recovery mode, integration authorization, agent permission to change binding rules, lease duration, and binding rules. Record the reason and source for the change. Review & check settings defines the shared repository identity and required check roster. New policy versions require agents to reread and acknowledge instructions; existing candidates may require reconciliation.

Open an unowned task to edit its brief, criteria, priority, prerequisites, and admission. Planned tasks remain unavailable until admitted. An edit always names the revision displayed when the form was opened. Refresh and reconsider a stale form instead of overwriting a newer revision. The service prevents active-work, workflow, and dependency-cycle bypasses.

A saved blocker can be resolved by a human with evidence when the task has no active or expired attempt. Expired work uses Inspect for recovery. Inspect the saved worktree or Git checkpoint, previous progress, and every still-running or uncertain job. Unknown producers retain their resource holds until terminal evidence or explicit human reconciliation. An inspection claim grants only recovery authority until the inspection is resolved. Renew before the displayed expiry and release with a handoff when pausing; release does not complete work. If inspection is incomplete, release as blocked with the reason.

Objectives and history

Objectives groups existing tasks under a general task with its own acceptance criteria. Required children must be done before the objective can be claimed and completed through its own submission and review. Optional children remain visible. A task has one optional parent; nested objectives are supported, while cycles through objectives and prerequisites are rejected. Membership freezes once objective work begins. This preserves the scope of its later review.

Browse history opens complete paginated records for the task and its associated review and integration activities. Choose attempts, checkpoints, checkouts, jobs, job observations, resources, artifacts, submissions, reviews, integrations, task revisions, or events. Expand a record to inspect its full evidence, then load more records. Pagination preserves an insertion cutoff; mutable operational records can still change. Start a fresh history query to include records inserted later. Objective details also provide older membership-history pages.

See objective semantics, history pages, and account lifecycle for precise API contracts.

After restoring a backup

Follow the host backup and restore guide to restore into a fresh directory and recover an existing administrator. Restored credentials, passwords, sessions, and ownership are invalidated before the data becomes available.

Open Access → Restore status to inspect the captured resource and integration holds. Use Open task evidence to examine the saved work, then record what you found for every hold. An inspection records evidence; it does not release a resource or prove that a remote process stopped. Keep uncertain work held.

Record evidence that the previous installation cannot act, and account for work performed after the snapshot. Once every required inspection and both records are present, Resume coordination enables new claims. Existing holds still require their normal recovery or reconciliation workflow.

Use New token for this agent on a revoked credential row to issue access for the same agent identity. Start a fresh local harness session, read the current instructions, and recover expired work after inspection. Preserving the identity also preserves its contribution history for independent-review checks.

Linux release installation

The binary Linux release is built and tested for Ubuntu 24.04 LTS on x86-64. Other Linux distributions can build the locked Rust workspace from source, but they are outside this binary package’s acceptance claim. The package contains the service, native CLI, systemd units, configuration and Caddy examples, and a complete copy of the repository’s Markdown guidance. Migrations and dashboard assets are embedded in the server binary.

The release workflow also builds a native Windows x86-64 CLI archive and runs its client, CLI, and local-runner tests on a Windows GitHub runner. This CI result does not establish the separate Windows workstation acceptance milestone.

Verify and inspect the archive

Download the archive and its adjacent .sha256 file from the same reviewed release run. Verify before extracting:

sha256sum --check agent-coordinator-0.1.0-linux-x86_64.tar.gz.sha256
tar -tzf agent-coordinator-0.1.0-linux-x86_64.tar.gz

Replace 0.1.0 with the release version. The archive has one top-level directory and contains no links. Its internal SHA256SUMS covers every packaged file except the manifest itself. Extract it into a staging directory and verify that manifest from the package root:

mkdir package-staging
tar -xzf agent-coordinator-0.1.0-linux-x86_64.tar.gz -C package-staging
cd package-staging/agent-coordinator-0.1.0-linux-x86_64
sha256sum --check SHA256SUMS

Do not pipe an unverified archive into a privileged extraction command. Keep the package and its checksums with the release evidence used for the installation.

Install the service

The examples use /usr/local/bin, /etc/agent-coordinator, and /var/lib/agent-coordinator. Adapt all unit and configuration paths together if the installation uses different locations.

sudo useradd --system --user-group --home-dir /var/lib/agent-coordinator \
  --shell /usr/sbin/nologin agent-coordinator
sudo install -d -o root -g agent-coordinator -m 0750 /etc/agent-coordinator
sudo install -d -o agent-coordinator -g agent-coordinator -m 0700 \
  /var/lib/agent-coordinator /var/lib/agent-coordinator-backups
sudo install -o root -g root -m 0755 bin/agent-coordinator-server \
  bin/agent-coordinator /usr/local/bin/
sudo install -o root -g agent-coordinator -m 0640 deploy/service.env.example \
  /etc/agent-coordinator/service.env

Edit /etc/agent-coordinator/service.env. Set the exact public HTTPS origin and retain the loopback listener. The example contains no credentials. Keep the SQLite database, WAL files, artifact directory, and backups private because they contain authentication verifiers and work evidence.

Initialize the first administrator at a hidden terminal prompt:

sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
  --database /var/lib/agent-coordinator/coordinator.sqlite3 \
  --public-origin https://coordinator.example.com \
  init-admin --username admin

The command supports --password-stdin for a protected automation pipe. Never put the password in arguments, an environment file, shell history, or logs.

Install and start the service, hourly backup timer, and daily maintenance timer:

sudo install -o root -g root -m 0644 deploy/agent-coordinator.service \
  deploy/agent-coordinator-backup.service deploy/agent-coordinator-backup.timer \
  deploy/agent-coordinator-maintenance.service deploy/agent-coordinator-maintenance.timer \
  /etc/systemd/system/
sudo systemd-analyze verify /etc/systemd/system/agent-coordinator.service \
  /etc/systemd/system/agent-coordinator-backup.service \
  /etc/systemd/system/agent-coordinator-backup.timer \
  /etc/systemd/system/agent-coordinator-maintenance.service \
  /etc/systemd/system/agent-coordinator-maintenance.timer
sudo systemctl daemon-reload
sudo systemctl enable --now agent-coordinator.service
sudo systemctl start agent-coordinator-backup.service
sudo systemctl --no-pager --full status agent-coordinator.service
sudo systemctl --no-pager --full status agent-coordinator-backup.service

The timer starting successfully is not proof of a usable backup. Inspect the oneshot result and verify its reported snapshot as described in the backup and restore guide. Inspect maintenance remaining-work flags and monitor failed units as described in the retention contract. After verifying the first backup, run maintenance and then enable both schedules:

sudo systemctl start agent-coordinator-maintenance.service
sudo systemctl --no-pager --full status agent-coordinator-maintenance.service
sudo systemctl enable --now agent-coordinator-backup.timer agent-coordinator-maintenance.timer

Persistent timers can immediately run missed schedules when first enabled. The manual checks above establish first-run evidence before enabling those schedules.

Configure HTTPS

Install Caddy through its documented distribution channel, copy deploy/Caddyfile.example to /etc/caddy/Caddyfile, replace the example hostname, and validate before reload:

sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
curl --fail --show-error https://coordinator.example.com/healthz

The hostname must match COORDINATOR_PUBLIC_ORIGIN. Keep the service listener on loopback and expose only the HTTPS proxy. Do not disable certificate verification. For an internal Caddy CA, explicitly install its root certificate on each client; Caddy documents both tls internal and its persistent data location. The automated Ubuntu acceptance uses a disposable internal CA and supplies that CA to both its HTTPS client and native CLI.

Sign in through HTTPS, create a project, issue a separately named agent credential, and run agent-coordinator connect from a bound repository. Follow the CLI guide for the binding and protected local credential state.

Upgrade and rollback boundary

Before upgrading, stop both timers and wait for or stop any running backup and maintenance units. Use the currently installed matching binary to create and verify a backup. Stop the main service, stage and checksum the new binaries, and replace both binaries together. Start the service and check HTTPS health, sign-in, and native client reconnect. Run and verify a new backup and inspect a maintenance result before restarting both timers. Keep all scheduled and active host commands stopped while replacing binaries; stopping only the main service is insufficient. The new server may migrate the database on startup; the new binary deliberately rejects an older schema for backup rather than silently upgrading it during a backup command. Version-1 snapshots from schema 12 onward can be verified and restored by this release when their migration history is an exact known prefix. Restore upgrades its private copy before invalidating old authority; it leaves the original snapshot unchanged.

Do not roll an old binary back over a database after a newer migration. Restore the pre-upgrade snapshot into a fresh directory with a matching compatible binary and follow the authority reconciliation procedure instead. Preserve the failed state for diagnosis. The service never automatically reverses migrations.

Removal

Disable and stop both timers, then stop any already-running backup or maintenance unit before stopping the main service. Disabling a timer alone does not stop its active oneshot. Use this order:

sudo systemctl disable --now agent-coordinator-backup.timer agent-coordinator-maintenance.timer
sudo systemctl stop agent-coordinator-backup.service agent-coordinator-maintenance.service
sudo systemctl disable --now agent-coordinator.service

Remove the unit files and binaries, then run sudo systemctl daemon-reload. Keep data and backups until their retention or incident requirements have been reviewed. Removing the package does not authorize deleting /var/lib/agent-coordinator or /var/lib/agent-coordinator-backups.

The release workflow’s systemd exercise is disposable and runs only on an Ubuntu 24.04 GitHub-hosted runner. It uses unique account, unit, port, and filesystem names, exercises start and restart through trusted HTTPS, and removes those resources afterward. It is release evidence, not evidence of a production installation, public hostname, off-server backup destination, or completed Windows workstation exercise.

Release archives are deterministic for fixed input binaries, documentation, version, and SOURCE_DATE_EPOCH. CI performs two clean release builds into separate target directories, packages each result, and requires byte-identical archives and checksum files. This checks the complete native release output on the named runner image; keep the run identity with release evidence.

Service deployment examples

These examples configure one service process with SQLite on a local filesystem. Caddy terminates HTTPS; the service accepts only loopback listener addresses and ignores forwarded headers. For checksum verification, package layout, supported binary platform, upgrade boundaries, and removal, start with the Linux release installation guide.

The installable examples remain canonical files in the repository rather than book chapters:

• server service
• hourly backup service and timer
• daily maintenance service and timer
• service environment example
• Caddy configuration example

1. Verify and extract the reviewed release package. Install its agent-coordinator-server and agent-coordinator binaries under /usr/local/bin. Assets and migrations are embedded in the server.

2. Create an unprivileged agent-coordinator system account and group, a /var/lib/agent-coordinator directory owned by that account with mode 0700, and /etc/agent-coordinator owned by root.

3. Copy service.env.example to /etc/agent-coordinator/service.env, set the real HTTPS origin, and protect it with mode 0640 and group agent-coordinator.

4. As the service account, initialize the administrator before starting the service (the password is entered through a hidden prompt):

   sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
     --database /var/lib/agent-coordinator/coordinator.sqlite3 \
     --public-origin https://coordinator.example.com \
     init-admin --username admin
   

   Automation may add --password-stdin and supply the password through a protected pipe. There is deliberately no password command-line argument. Initialization succeeds only on an empty installation; it is not an account recovery command. Use My account for password changes, or the audited host recover-operator-password command described in the operator guide for account recovery.

5. Install agent-coordinator.service under /etc/systemd/system, reload systemd, and enable/start the service. Install Caddy with the matching public hostname from Caddyfile.example. Permit public HTTPS to Caddy and keep port 8080 private. Configure DNS and normal certificate issuance for that hostname. Validate the actual configuration with caddy validate --config /etc/caddy/Caddyfile before reloading Caddy; see its request-body size limit documentation.

6. Open the HTTPS site, sign in, and issue a named agent credential. The token is displayed once. If that response is lost, retry the same request/key to recover its identity, then rotate that credential under Access to issue a replacement for the same agent principal. A token is never replayed.

7. Create the separate backup repository and install the hourly backup units:

   sudo install -d -o agent-coordinator -g agent-coordinator -m 0700 \
     /var/lib/agent-coordinator-backups
   sudo install -o root -g root -m 0644 \
     deploy/agent-coordinator-backup.service \
     deploy/agent-coordinator-backup.timer \
     /etc/systemd/system/
   sudo systemctl daemon-reload
   sudo systemctl start agent-coordinator-backup.service
   

   Confirm that the first oneshot succeeded and verify its reported snapshot. Timer activation alone is not backup evidence. Configure an alert for a failed backup unit and stale verified-snapshot age. See the backup and restore guide for retention, advisory-locked off-server copies, restore authority invalidation, and the required recovery exercise.

   Enable the persistent schedule only after the first snapshot passes verification:

   sudo systemctl enable --now agent-coordinator-backup.timer
   

Install the daily maintenance timer for bounded replay/health payload compaction. Permanent task and lesson history remains stored. Follow the clock recovery guide if the service reports a clock reconciliation pause.

For local development only, use --public-origin http://127.0.0.1:8080 --allow-insecure-loopback. The explicit opt-in permits a non-Secure development cookie solely for a loopback origin. HTTPS uses a Secure, HttpOnly, SameSite=Strict cookie with a 12-hour absolute lifetime and a __Host- prefix. Browser writes require the exact configured Origin, X-CSRF-Token, and Idempotency-Key headers. Login requires the origin but has no idempotency receipt because it creates a fresh browser session.

Password hashes use Argon2id with 19 MiB memory, two iterations, and one lane, matching the OWASP password-storage minimum reviewed on 2026-09-09. Password checks run away from async workers, with at most two concurrent checks. Sign-in is limited to five attempts per username and 30 total attempts per minute, using a bounded in-process table and monotonic time. Restart resets that limiter; use proxy/network controls for internet-scale traffic. The service does not trust client-supplied forwarding headers to bypass its global limit.

The server stores only credential/session verifiers and redacts issued tokens from mutation receipts and events. Do not enable HTTP access logs that record headers, bodies, or query strings. Protect the database, its WAL files, artifacts, and every backup snapshot as credentials. Database initialization uses mode 0600; systemd uses umask 0077. The installed backup repository is local protection only. Do not claim host-loss protection until a complete destination copy has passed destination-side verification, and do not claim the one-hour restore target until the documented recovery exercise has measured it end to end.

Build the release package

The cloud deployment chapters install the same verified Linux x86-64 archive that Linux release installation describes. This chapter covers obtaining that archive: either from the release workflow or by building it on a workstation. Build on Ubuntu 24.04 x86-64 so the binary matches the platform the package is accepted for.

Option A: download the accepted package from CI

Pushing a v* tag runs the release workflow, which builds twice, requires byte-identical archives, exercises a disposable systemd/HTTPS installation, and uploads the archive and its .sha256 as the agent-coordinator-linux-x86_64 artifact. The artifact is retained for 14 days.

git tag v0.1.0
git push origin v0.1.0
gh run list --workflow "Release package acceptance" --limit 1
gh run download RUN_ID --name agent-coordinator-linux-x86_64 --dir dist
sha256sum --check dist/agent-coordinator-0.1.0-linux-x86_64.tar.gz.sha256

Replace RUN_ID with the identifier printed by the list command and 0.1.0 with the workspace version in Cargo.toml. Record the run identity with the installation evidence; that run is the acceptance claim for the archive.

Option B: build on a workstation

Install the pinned toolchain and build only the two packaged crates with locked dependencies:

rustup toolchain install 1.98.1 --profile minimal
rustup default 1.98.1
cargo build --release --locked -p coordinator-server -p coordinator-cli

Package the binaries with the repository’s Markdown guidance and systemd files:

version="$(python3 -c "import tomllib; print(tomllib.load(open('Cargo.toml','rb'))['workspace']['package']['version'])")"
python3 scripts/package_release.py --platform linux-x86_64 --version "$version" \
  --server target/release/agent-coordinator-server \
  --cli target/release/agent-coordinator \
  --source-date-epoch "$(git show -s --format=%ct HEAD)" \
  --output-dir dist

The script prints the archive path, checksum file, and SHA-256. A workstation build has not passed the release workflow’s systemd/HTTPS exercise. Treat it as suitable for a staging instance, or run that workflow on the same commit before calling the instance production.

Copy the package to the host

Both cloud chapters copy dist/agent-coordinator-VERSION-linux-x86_64.tar.gz and its .sha256 to the virtual machine, then verify and extract there exactly as Linux release installation describes. Never extract an unverified archive with elevated privileges.

Deploy to an Azure virtual machine

These operator steps create one small Ubuntu 24.04 virtual machine on Azure, install the verified release package with the documented systemd units, expose it through Caddy on a public hostname, and copy verified backups off the virtual machine. They add Azure-specific provisioning to Linux release installation; that chapter remains the authority for package verification, unit installation, upgrades, and removal.

The steps use the Azure CLI (az) from a workstation. A Standard_B1s virtual machine (1 vCPU, 1 GiB) is sufficient: the accepted package peaked at 84.6 MiB resident memory under the capacity exercise, and Caddy adds a few tens of MiB. Azure free accounts include 750 hours per month of B1s for the first 12 months; afterwards it costs a few dollars per month. Prices and free allowances change, so confirm them in the Azure pricing pages before relying on them.

1. Build or download the package

Follow Build the release package. Keep the archive, its .sha256, and the release run identity together.

2. Create the resource group and virtual machine

Sign in and choose a region and names. The public IP is static so DNS can point at it permanently.

az login
az group create --name agent-coordinator --location eastus
az network public-ip create --resource-group agent-coordinator \
  --name coordinator-ip --sku Standard --allocation-method Static
az vm create --resource-group agent-coordinator --name coordinator \
  --image Ubuntu2404 --size Standard_B1s \
  --admin-username operator --ssh-key-values ~/.ssh/id_ed25519.pub \
  --public-ip-address coordinator-ip --os-disk-size-gb 30
az vm open-port --resource-group agent-coordinator --name coordinator \
  --port 80,443 --priority 1010
az network public-ip show --resource-group agent-coordinator \
  --name coordinator-ip --query ipAddress --output tsv

The vm create command opens SSH on port 22 by default. Port 80 is needed only for Caddy’s certificate challenge and redirect; the service port 8080 stays closed because the service listens on loopback. Restrict the SSH rule to your workstation address once the installation is complete:

az network nsg rule update --resource-group agent-coordinator \
  --nsg-name coordinatorNSG --name default-allow-ssh \
  --source-address-prefixes YOUR_WORKSTATION_IP/32

3. Point DNS at the address

Create an A record for the chosen hostname, for example coordinator.example.com, at the static address printed above. Wait until dig +short coordinator.example.com returns it from your workstation. Caddy cannot obtain a certificate until the name resolves publicly.

4. Copy the package and prepare the host

scp dist/agent-coordinator-0.1.0-linux-x86_64.tar.gz \
    dist/agent-coordinator-0.1.0-linux-x86_64.tar.gz.sha256 \
    operator@coordinator.example.com:
ssh operator@coordinator.example.com
sudo apt-get update && sudo apt-get upgrade -y
sudo timedatectl set-ntp true
timedatectl

Confirm System clock synchronized: yes. Lease timing depends on service time, and a clock rollback pauses new authority; see clock safety.

5. Install the service

On the virtual machine, verify and extract the archive, then follow Install the service without change. In /etc/agent-coordinator/service.env set COORDINATOR_PUBLIC_ORIGIN=https://coordinator.example.com and leave COORDINATOR_LISTEN=127.0.0.1:8080. Initialize the administrator with the hidden prompt, install the units, start the service, and run the first backup and maintenance oneshots before enabling the timers, exactly as documented.

6. Install Caddy and enable HTTPS

Install Caddy from its official apt repository as documented at caddyserver.com/docs/install, then apply the repository’s example configuration:

sudo install -o root -g root -m 0644 deploy/Caddyfile.example /etc/caddy/Caddyfile
sudo sed -i 's/coordinator\.example\.com/coordinator.example.com/' /etc/caddy/Caddyfile
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl enable --now caddy
sudo systemctl reload caddy
curl --fail --show-error https://coordinator.example.com/healthz

Replace the second hostname in the sed command with your real hostname. Caddy obtains a public certificate from Let’s Encrypt automatically; watch journalctl -u caddy -f on the first start if the health check fails. The hostname must equal the configured public origin exactly.

7. Copy verified backups off the virtual machine

The hourly backup timer writes local snapshots only. Azure Files gives a mounted SMB destination that the documented copy procedure can use unchanged:

az storage account create --resource-group agent-coordinator \
  --name coordinatorbackups --sku Standard_LRS --kind StorageV2 \
  --min-tls-version TLS1_2 --allow-blob-public-access false
az storage share-rm create --resource-group agent-coordinator \
  --storage-account coordinatorbackups --name snapshots --quota 50

Storage account names must be globally unique lowercase letters and digits; adjust coordinatorbackups. On the virtual machine, install cifs-utils, store the account key in a root-only credentials file, and mount the share at /mnt/coordinator-backups through /etc/fstab with uid and gid of the agent-coordinator account and nofail. Azure’s portal shows the exact mount command for the share under Connect.

Then run the copy procedure in Verify and copy a snapshot off-server with destination_repository=/mnt/coordinator-backups/agent-coordinator, and schedule it at least hourly. The destination is protected only after destination-side verification passes; record that time separately from the local snapshot time.

8. Complete the first operator tasks

1. Sign in at the HTTPS origin, create the project, and set its canonical repository key and required-check roster.
2. Issue one agent credential per workstation. The token is shown once.
3. From a bound checkout, run agent-coordinator connect and confirm the orientation response.
4. Rehearse a restore into a fresh directory from an off-server copy, following the backup and restore guide, and record the measured time. No host-loss protection is claimed until this passes.

Upgrades, cost control, and removal

Upgrade with the boundary described in Upgrade and rollback boundary. Stopping the virtual machine with az vm deallocate stops compute charges but keeps the disk, static address, and storage account billing. Delete the resource group only after the off-server snapshots have been verified elsewhere; removal of the package does not authorize deleting data or backups.

Deploy to a Google Cloud e2-micro

These operator steps create one e2-micro virtual machine on Google Compute Engine, install the verified release package with the documented systemd units, expose it through Caddy on a public hostname, and copy verified backups to a Cloud Storage bucket. They add Google Cloud provisioning to Linux release installation; that chapter remains the authority for package verification, unit installation, upgrades, and removal.

The e2-micro (2 shared vCPUs, 1 GiB) is included in Google Cloud’s Always Free tier when it runs in us-west1, us-central1, or us-east1 with a standard persistent disk of at most 30 GB, and the tier also covers a small amount of Cloud Storage. The accepted package peaked at 84.6 MiB resident memory under the capacity exercise, so the size is adequate. Free-tier terms change, so confirm them in Google’s pricing pages before relying on them. The steps use the gcloud CLI from a workstation.

1. Build or download the package

Follow Build the release package. Keep the archive, its .sha256, and the release run identity together.

2. Create the project resources and the virtual machine

gcloud auth login
gcloud config set project YOUR_PROJECT_ID
gcloud services enable compute.googleapis.com
gcloud compute addresses create coordinator-ip --region us-central1
gcloud compute instances create coordinator \
  --zone us-central1-a --machine-type e2-micro \
  --image-family ubuntu-2404-lts-amd64 --image-project ubuntu-os-cloud \
  --boot-disk-size 30GB --boot-disk-type pd-standard \
  --address coordinator-ip --tags coordinator-https
gcloud compute firewall-rules create coordinator-allow-https \
  --target-tags coordinator-https --allow tcp:80,tcp:443 \
  --source-ranges 0.0.0.0/0
gcloud compute addresses describe coordinator-ip --region us-central1 \
  --format 'value(address)'

Port 80 is needed only for Caddy’s certificate challenge and redirect; the service port 8080 stays closed because the service listens on loopback. SSH uses gcloud compute ssh, which manages keys through the default network’s SSH rule. Restrict that rule to your workstation address once installation is complete:

gcloud compute firewall-rules update default-allow-ssh \
  --source-ranges YOUR_WORKSTATION_IP/32

3. Point DNS at the address

Create an A record for the chosen hostname, for example coordinator.example.com, at the static address printed above. Wait until dig +short coordinator.example.com returns it. Caddy cannot obtain a certificate until the name resolves publicly.

4. Copy the package and prepare the host

gcloud compute scp --zone us-central1-a \
  dist/agent-coordinator-0.1.0-linux-x86_64.tar.gz \
  dist/agent-coordinator-0.1.0-linux-x86_64.tar.gz.sha256 coordinator:~
gcloud compute ssh coordinator --zone us-central1-a
sudo apt-get update && sudo apt-get upgrade -y
timedatectl

Google images synchronize time from the metadata server; confirm System clock synchronized: yes. Lease timing depends on service time; see clock safety.

With 1 GiB of memory and no swap by default, add a small swap file so a transient spike in the package manager or Caddy cannot trigger the kernel’s out-of-memory killer against the service:

sudo fallocate -l 1G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

5. Install the service

On the virtual machine, verify and extract the archive, then follow Install the service without change. In /etc/agent-coordinator/service.env set COORDINATOR_PUBLIC_ORIGIN=https://coordinator.example.com and leave COORDINATOR_LISTEN=127.0.0.1:8080. Initialize the administrator with the hidden prompt, install the units, start the service, and run the first backup and maintenance oneshots before enabling the timers, exactly as documented.

6. Install Caddy and enable HTTPS

Install Caddy from its official apt repository as documented at caddyserver.com/docs/install, then apply the repository’s example configuration:

sudo install -o root -g root -m 0644 deploy/Caddyfile.example /etc/caddy/Caddyfile
sudo sed -i 's/coordinator\.example\.com/coordinator.example.com/' /etc/caddy/Caddyfile
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl enable --now caddy
sudo systemctl reload caddy
curl --fail --show-error https://coordinator.example.com/healthz

Replace the second hostname in the sed command with your real hostname. Caddy obtains a public certificate from Let’s Encrypt automatically; watch journalctl -u caddy -f on the first start if the health check fails. The hostname must equal the configured public origin exactly.

7. Copy verified backups to Cloud Storage

The hourly backup timer writes local snapshots only. Create a private bucket in the same region and grant the virtual machine’s service account object write access:

gcloud storage buckets create gs://YOUR_UNIQUE_BUCKET --location us-central1 \
  --uniform-bucket-level-access --public-access-prevention
gcloud storage buckets add-iam-policy-binding gs://YOUR_UNIQUE_BUCKET \
  --member serviceAccount:VM_SERVICE_ACCOUNT_EMAIL --role roles/storage.objectCreator

The documented copy procedure needs a mounted destination for its staged, verified, atomically published copy. Mount the bucket with Cloud Storage FUSE (gcsfuse) at /mnt/coordinator-backups for the agent-coordinator account, then run Verify and copy a snapshot off-server with destination_repository=/mnt/coordinator-backups/agent-coordinator at least hourly. If a FUSE mount is not acceptable, verify the local snapshot, copy it with gcloud storage rsync to a per-snapshot prefix, and verify the destination copy again after download on another machine. Either way the destination is protected only after destination-side verification passes; record that time separately from the local snapshot time.

8. Complete the first operator tasks

1. Sign in at the HTTPS origin, create the project, and set its canonical repository key and required-check roster.
2. Issue one agent credential per workstation. The token is shown once.
3. From a bound checkout, run agent-coordinator connect and confirm the orientation response.
4. Rehearse a restore into a fresh directory from an off-server copy, following the backup and restore guide, and record the measured time. No host-loss protection is claimed until this passes.

Upgrades, cost control, and removal

Upgrade with the boundary described in Upgrade and rollback boundary. A stopped instance keeps its disk and static address; a reserved static address that is not attached to a running instance is billed even under the free tier. Delete the instance and bucket only after the off-server snapshots have been verified elsewhere; removal of the package does not authorize deleting data or backups.

Backup and restore operations

This guide covers the native Linux backup timer, verified off-server copies, and an offline restore. A backup is useful only after verification. A restore remains paused until a recovered administrator inspects preserved holds, fences the old installation, and reconciles work that may have happened after the snapshot.

The examples use these installation paths:

• live data: /var/lib/agent-coordinator
• database: /var/lib/agent-coordinator/coordinator.sqlite3
• backup repository: /var/lib/agent-coordinator-backups
• service environment: /etc/agent-coordinator/service.env

Run the shell command blocks with set -eu so an error stops the procedure. Replace the uppercase incident and snapshot placeholders before running them.

Keep the backup repository outside the live data and artifact roots. The backup command rejects a nested repository because recursive capture cannot produce a self-contained recovery set.

Install and monitor the hourly timer

Install deploy/agent-coordinator-backup.service and deploy/agent-coordinator-backup.timer under /etc/systemd/system. Create the backup repository for the service account before starting the timer:

sudo install -d -o agent-coordinator -g agent-coordinator -m 0700 \
  /var/lib/agent-coordinator-backups
sudo install -o root -g root -m 0644 \
  deploy/agent-coordinator-backup.service \
  deploy/agent-coordinator-backup.timer \
  /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start agent-coordinator-backup.service

The calendar timer targets the start of every UTC hour, adds at most 60 seconds of random delay, and catches up once after downtime. The oneshot service and the repository’s exclusive advisory lock prevent overlapping backup or retention work. The online SQLite backup does not require stopping the main service. These settings follow the official systemd timer semantics for calendar accuracy, randomized delay, and persistent catch-up.

Keep service.env limited to the non-secret server settings shown in the deploy example. The backup unit has no network access and performs no off-server transfer; configure transfer credentials only in the separate operator-selected mechanism.

Verify the first snapshot as described below. Only after it passes verification, enable the persistent schedule:

sudo systemctl enable --now agent-coordinator-backup.timer

Inspect the actual result rather than treating timer activation as proof of a backup:

systemctl list-timers agent-coordinator-backup.timer
sudo systemctl status agent-coordinator-backup.service
sudo journalctl -u agent-coordinator-backup.service --since today

Configure monitoring for a failed oneshot unit and for the age of the latest verified snapshot. A later successful run does not erase the need to investigate an earlier disk-space or integrity failure.

The command writes into .staging and publishes a snapshot only after verifying its SQLite image, every selected artifact, its manifest, and its completion marker. A completed snapshot is:

/var/lib/agent-coordinator-backups/snapshots/<UTC-time>-<snapshot-id>/
  database.sqlite3
  blobs/<artifact-storage-id>.blob
  manifest.json
  COMPLETE

Directories use mode 0700 and files use mode 0600. Failed partial snapshots are not usable backups. Backup creation preserves existing completed snapshots if a new snapshot cannot be completed. The engine holds a shared artifact-cleanup lock from before the online SQLite snapshot through blob copy and verification. Cleanup uses a nonblocking exclusive lock and defers while a backup is selecting bytes; the backup does not keep a database writer transaction open while copying blobs. Do not replace this with a filesystem copy of the live database that omits its WAL.

Retention keeps the union of:

• the latest completed snapshot in each of the newest 24 represented UTC-hour buckets; and
• the latest completed snapshot in each of the newest 30 represented UTC-day buckets.

Multiple manual backups in one hour do not consume multiple hourly buckets. An hour or day without a successful backup does not create a fictitious retained copy. Snapshot directories are self-contained, so pruning one does not remove bytes needed by another retained snapshot.

Verify and copy a snapshot off-server

Run verification with the same reviewed server version used for the installation:

sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
  backup-verify \
  --snapshot /var/lib/agent-coordinator-backups/snapshots/SNAPSHOT_DIRECTORY

Verification rejects a missing COMPLETE, manifest mismatches, changed sizes or digests, SQLite integrity or foreign-key failures, and incompatible schema data. It does not open or change the configured live database. Restore likewise reads only its selected snapshot and writes only its absent destination.

Copy one whole self-contained snapshot while holding the repository’s shared advisory lock at /var/lib/agent-coordinator-backups/.lock. The following example uses an already mounted, operator-selected destination. Its .staging and snapshots directories must be on the same filesystem so the final rename is atomic:

source_snapshot=/var/lib/agent-coordinator-backups/snapshots/SNAPSHOT_DIRECTORY
destination_repository=/mnt/operator-selected-backup/agent-coordinator
snapshot_name=$(basename -- "$source_snapshot")
destination_stage="$destination_repository/.staging/$snapshot_name.partial"

sudo install -d -o agent-coordinator -g agent-coordinator -m 0700 \
  "$destination_repository"
sudo -u agent-coordinator install -d -m 0700 \
  "$destination_repository/.staging" "$destination_repository/snapshots"
sudo test ! -e "$destination_stage"
sudo test ! -e "$destination_repository/snapshots/$snapshot_name"
sudo -u agent-coordinator install -d -m 0700 "$destination_stage"
sudo -u agent-coordinator flock --shared \
  /var/lib/agent-coordinator-backups/.lock \
  sh -c 'cp -a -- "$1/database.sqlite3" "$1/blobs" "$1/manifest.json" "$2/" && cp -a -- "$1/COMPLETE" "$2/"' \
  backup-copy "$source_snapshot" "$destination_stage"

sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
  backup-verify --snapshot "$destination_stage"
sudo sync -f "$destination_repository"
sudo -u agent-coordinator flock --exclusive "$destination_repository/.lock" \
  mv --no-clobber --no-target-directory -- \
  "$destination_stage" "$destination_repository/snapshots/$snapshot_name"
sudo test ! -e "$destination_stage"
sudo sync -f "$destination_repository"

The destination is not counted as protected until destination-side verification succeeds and the staging directory is published. For an SSH, object-storage, or other transfer, preserve the same order: hold the source lock for the complete read, transfer every file, place COMPLETE last in a private staging location, verify at the destination, and then publish atomically. Supply authentication, transport encryption, and encryption at rest through that external mechanism. Do not put its credentials in the repository, unit file, command line, or service environment. Record the local snapshot time and verified off-server copy time separately. Hourly local backups do not provide hourly host-loss protection unless verified off-server transfer also runs at least hourly.

Back up before a schema upgrade

Keep the exact currently installed binary until the upgrade is accepted. Before a schema-changing upgrade, enter maintenance, stop the service, and use that old binary to create and verify a snapshot of the old schema:

old_binary=/operator/protected/path/agent-coordinator-server-OLD_VERSION
sudo -u agent-coordinator "$old_binary" \
  --database /var/lib/agent-coordinator/coordinator.sqlite3 \
  backup --repository /var/lib/agent-coordinator-backups
sudo -u agent-coordinator "$old_binary" \
  backup-verify \
  --snapshot /var/lib/agent-coordinator-backups/snapshots/SNAPSHOT_DIRECTORY

The backup command opens an existing database at its exact expected migration level and never upgrades it. A new binary can reject the old schema, so do not install it first and then rely on it to create the pre-migration recovery point. Apply the new release’s migration only after the old-format snapshot passes verification. Preserve the matching old executable with the snapshot until the upgrade recovery window closes.

Restore into a fresh data directory

Use an available compatible host and a reviewed binary compatible with the snapshot manifest. Never restore over the live directory. Keep the old installation and its public endpoint fenced throughout the operation so two installations cannot act on the same external work.

1. Stop the backup timer and main service, block ordinary traffic at the reverse proxy, and prevent any old service instance from restarting. Keep a protected HTTPS path available only to the recovery administrator for the later sign-in:

   sudo systemctl stop agent-coordinator-backup.timer
   sudo systemctl stop agent-coordinator.service
   

2. Preserve the current data directory separately. Select a completed snapshot and verify it before restore:

   sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
     backup-verify --snapshot /var/lib/agent-coordinator-backups/snapshots/SNAPSHOT_DIRECTORY
   

3. Choose an absent staging destination on the same filesystem as the eventual live directory. Record a specific incident or exercise reason, then restore:

   sudo install -d -o agent-coordinator -g agent-coordinator -m 0700 \
     /var/lib/agent-coordinator-restore
   sudo test ! -e /var/lib/agent-coordinator-restore/RESTORE_INCIDENT
   sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
     restore \
     --snapshot /var/lib/agent-coordinator-backups/snapshots/SNAPSHOT_DIRECTORY \
     --destination /var/lib/agent-coordinator-restore/RESTORE_INCIDENT \
     --reason '<audited restore reason>'
   

   Restore refuses an existing destination. It verifies the complete snapshot before mutation, creates coordinator.sqlite3 and its artifact tree with private permissions, runs SQLite integrity and foreign-key checks, invalidates restored authority, checkpoints the staged database, and only then publishes the complete destination directory.

4. With the service still stopped, move the old live directory aside and promote the verified staging directory. Keep the sibling backup repository in place:

   sudo mv --no-clobber --no-target-directory -- \
     /var/lib/agent-coordinator \
     /var/lib/agent-coordinator.pre-restore-RESTORE_INCIDENT
   sudo test ! -e /var/lib/agent-coordinator
   sudo mv --no-clobber --no-target-directory -- \
     /var/lib/agent-coordinator-restore/RESTORE_INCIDENT \
     /var/lib/agent-coordinator
   sudo test ! -e /var/lib/agent-coordinator-restore/RESTORE_INCIDENT
   sudo sync -f /var/lib
   sudo chown -R agent-coordinator:agent-coordinator \
     /var/lib/agent-coordinator
   

5. Start the service while ordinary client traffic remains blocked. The restored service starts in a global pause. All restored browser sessions, agent credentials, agent sessions, reporter proofs, attempts, and leases are invalid. Restored human passwords are randomized and all human accounts are disabled. A fresh restore epoch also rejects cursors issued before the restore. Resource and integration holds remain recorded for inspection.

   sudo systemctl start agent-coordinator.service
   

6. Recover exactly one existing administrator from the host. Enter the password through the hidden prompt, or use a protected standard-input pipe:

   sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
     --database /var/lib/agent-coordinator/coordinator.sqlite3 \
     recover-operator-password \
     --username EXISTING_ADMIN \
     --reason '<audited restore administrator recovery reason>'
   

   If password recovery reports clock_reconciliation_required, correct host time and follow host clock reconciliation first, then retry password recovery. Restoring a database does not erase a captured clock incident.

7. Sign in as that administrator and enumerate the bounded restore requirements:

   The examples below show route and body shapes. Use the dashboard or an authenticated administrator client that supplies the normal Origin, CSRF, and idempotency protections for mutations.

   GET /api/v1/admin/restore?limit=200
   GET /api/v1/admin/restore?cursor=<next-cursor>&limit=200
   

   For every returned hold, inspect current physical and external state and save one disposition with evidence:

   POST /api/v1/admin/restore/inspections
   {
     "restore_id": "<restore-id>",
     "kind": "resource_hold",
     "target_id": "<hold-id>",
     "disposition": "unknown",
     "evidence": "what was inspected and where"
   }
   

   kind is resource_hold or integration_hold; disposition is held, released, or unknown.

   An inspection records the finding; finishing restore does not automatically release a preserved hold. Use the normal human job, resource, or publication reconciliation operation when evidence supports a later state change.

8. Prevent the old installation from serving or writing permanently, then attest that fencing with evidence:

   POST /api/v1/admin/restore/old-installation-fenced
   {"restore_id":"<restore-id>","evidence":"host, process, proxy, and storage fencing evidence"}
   

9. Compare external effects newer than the snapshot. At minimum, inspect remote Git targets for preserved integration work, workstation job/process state, physical resources, and off-service artifact state. Reconcile each uncertain effect through its normal guarded workflow, then attest the completed review:

   POST /api/v1/admin/restore/post-snapshot-gap
   {"restore_id":"<restore-id>","evidence":"snapshot cutoff and reconciliation evidence"}
   

10. After both attestations and every captured hold inspection are recorded, finish restore:

    POST /api/v1/admin/restore/finish
    {"restore_id":"<restore-id>","reason":"operator reviewed restored authority and external effects"}
    

    This atomically lifts the global pause. Inspection records do not change hold state: held resources remain held until a separate guarded operation releases or resolves them. During the pause, ordinary mutations remain rejected; administrator recovery actions and guarded human reconciliation remain available. Issue fresh credentials for existing agent principals only after deciding that they should reconnect. Credential creation returns a token once; store it outside the repository. Confirm that old cookies, passwords, tokens, session proofs, reporter proofs, and attempt generations are rejected.

    A fresh credential for an existing agent principal uses:

    POST /api/v1/admin/agents/<principal-id>/credentials
    {"name":"post-restore workstation credential"}
    

    Recover any additional human account from the host before enabling it; never treat its pre-restore password as valid authority.

11. Restore ordinary proxy access and restart the backup timer only after service readiness, new administrator authentication, old-authority rejection, and the restore checklist have all been observed.

    sudo systemctl start agent-coordinator-backup.timer
    

Exercise and recovery-time evidence

The recovery-time target is one hour from starting this procedure on an available compatible host with access to a completed snapshot and host credentials, through verified service availability and one recovered client. A rehearsal must record:

• start and finish times, binary/schema versions, snapshot ID and snapshot time;
• snapshot/database/artifact byte counts and destination verification result;
• time to restore, recover the administrator, inspect holds, and reconcile the post-snapshot gap;
• rejection of representative old browser, agent, reporter, and attempt authority;
• successful authentication by one recovered client and the restore-finish event;
• storage throughput, failures, manual steps, and external dependencies.

Do not claim the one-hour target from design estimates or local backup cadence. New host procurement, unavailable off-server storage, and recovery of external credentials are outside the measured compatible-host procedure and must be named separately in a real incident report.

Service clock safety contract

Service time is part of every credential, session, lease, reporter, decision, and publication-authority check. The coordinator records a durable high-water mark and combines it with an in-process monotonic clock. Authoritative reads and writers sample that clock through a short SQLite writer transaction before checking time-dependent authority. Concurrent authentication requests that were already waiting when a clock sample was taken may share that sample after its transaction commits. A request arriving after the sample always obtains a new one. This reduces writer contention without caching credential validity or weakening the fresh time and ownership checks inside every mutation.

Domain deadlines and relative remaining/cadence fields use protected coordinator time. Envelope server_time is observational wall time for response correlation and display; it never grants or extends authority.

The protected service time never moves behind a time already observed by the coordinator. During one process lifetime it also advances by monotonic elapsed time when the wall clock stalls or moves backward. A backward adjustment of at most 5 seconds is clamped to protected time without opening an incident. A wall clock more than 5 seconds behind expected protected time creates a durable clock incident and changes clock_state.status to clock_reconciliation.

A mutation that first detects rollback under its writer lock commits only the clock incident and authority expiry, then returns 409 clock_reconciliation_required without its requested coordination effect. Authentication can also detect and persist an incident before the handler runs. Evidence reads may still succeed; newly expired reporter credentials fail normal authentication, and ordinary mutations encounter the reconciliation pause. Detection expires every active attempt and reporter deadline while preserving task pointers, generations, resource reservations, and integration holds for recovery inspection. Existing integration authorization is shown with valid: false and validity_reason: clock_reconciliation_required while the incident is active.

Reconciliation pause

Authenticated evidence remains readable during clock reconciliation. The coordinator rejects operations that grant, extend, or depend on current work authority, including sign-in, session creation, instruction acknowledgment, claims, renewal, submission, decision answers, account changes, credentials, and new publication authority.

The pause keeps bounded operations needed to make work safe or inspectable:

• browser sign-out and revocation, and agent-session closure;
• reservation release and explicit resolution;
• workflow activity release, integration-result recording, and publication reconciliation;
• restore reconciliation and clock reconciliation.

Attempt checkpoint and release routes remain on the pause allowlist so a request already proven under a live attempt cannot be rejected solely by the global pause. Rollback detection expires all active attempts, however, so a later request still fails the normal ownership check and cannot use this exception to revive one. Reporter deadlines are capped at detection and later reporter authentication fails.

The available exceptions do not renew an attempt, release a physical or integration hold implicitly, create a credential, or mark uncertain external work complete. Restore invalidation remains available for an offline staged database. If it is the first observer of a rollback, it commits the clock incident before beginning restore invalidation under a fresh writer lock.

Before that invalidation, restore raises the private copy’s durable clock high-water mark to at least the snapshot manifest’s creation time. That time is also the artifact-inclusion cutoff, so a destination host with an older wall clock cannot revive an expired decision or advertise an expired artifact whose blob was correctly omitted from the snapshot.

Administrator API

GET /api/v1/admin/clock requires a currently enabled administrator and returns:

{
  "clock_state": {
    "status": "clock_reconciliation",
    "last_safe_time": "2027-01-15T08:00:20.000Z",
    "last_safe_time_ms": 1800000020000,
    "incident_id": "incident-id",
    "observed_wall_time": "2027-01-15T08:00:01.000Z",
    "observed_wall_time_ms": 1800000001000,
    "detected_at": "2027-01-15T08:00:20.000Z"
  },
  "incident": {
    "id": "incident-id",
    "observed_wall_time_ms": 1800000001000,
    "high_water_time_ms": 1800000020000,
    "recovered_at": null,
    "recovery_reason": null,
    "recovered_by": null,
    "recovery_kind": null
  },
  "material_rollback_threshold_ms": 5000
}

POST /api/v1/admin/clock/reconcile accepts an idempotency key and:

{
  "incident_id": "incident-id",
  "reason": "The host clock now agrees with an independent trusted source."
}

The incident ID must still be current. The reason contains 1 through 2000 bytes and may contain newlines and tabs but no other control characters or surrounding whitespace. The raw wall clock must reach or pass the current protected time; the 5-second detection tolerance does not weaken this recovery check. Otherwise the endpoint returns clock_incident_changed, clock_still_untrusted, or clock_not_paused as appropriate. A successful administrator reconciliation records the administrator principal in both the ordinary mutation event and the immutable clock-reconciliation audit record.

If sign-in is blocked or the administrator session expired, correct host time and run the host-local command as the service account:

sudo -u agent-coordinator /usr/local/bin/agent-coordinator-server \
  --database /var/lib/agent-coordinator/coordinator.sqlite3 \
  recover-clock --reason 'Host time verified against the configured trusted source.'

Use evidence specific to the incident. There is no unauthenticated HTTP recovery route. A successful command does not recover expired task ownership or clear a separate restore reconciliation pause. If a restored snapshot also contains a clock incident, reconcile the clock before recovering an operator password and completing the restore checklist.

It applies the same trustworthy-time and reason checks. Its audit event has initiator_kind: host_operator and a null actor_id; it does not invent or impersonate a principal. Correcting the wall clock does not itself lift the pause. An administrator or host operator must record explicit reconciliation.

Guarantees and deployment limit

The persisted high-water mark prevents a restarted process from returning to a service time it previously observed, so an authority deadline already observed as expired cannot become current after restart. The monotonic anchor accounts for elapsed time only within the current process lifetime.

No local monotonic clock can measure time across a power-off or restart. If the wall clock rolls backward during unobserved downtime but still starts later than the last persisted observation, the coordinator cannot detect that adjustment without an external trusted time source. Hosts must start the service only after establishing synchronized, trustworthy wall time. Service-manager ordering on a time-synchronization unit alone is not evidence that synchronization succeeded. The clock status is protected coordinator time, not an attestation of absolute time or an NTP health report.

Storage retention and maintenance contract

The host command, daily systemd timer, and mutation replay checks implement this retention policy. Clock reconciliation pauses maintenance until time is trusted.

Host command and bounds

The explicit host command is:

agent-coordinator-server ... maintenance \
  --batch-size 500 --max-batches 20

It calls:

maintenance::run_maintenance(
    state: &AppState,
    options: MaintenanceOptions,
) -> anyhow::Result<serde_json::Value>

batch_size defaults to 500 and is limited to 1–1,000 receipt rows and 1–1,000 old observation rows per transaction. max_batches defaults to 20 and is limited to 1–100. The command reports its run ID and timestamps, cutoff, affected rows, observation rows inspected, batch count, limits, and whether receipt results or old observation rows remain to inspect. An operator can run it again when either remaining flag is true. It never runs VACUUM; ordinary backup and database-size operations remain separate and maintenance does not hold one writer transaction across the whole invocation.

After each committed database batch, the command invokes one existing bounded artifact reconciliation pass and reports only the number of passes attempted. Each pass considers at most 100 expired/deleted artifact rows and 100 staging or orphan entries. Artifact deletion takes the store’s nonblocking exclusive GC lock and rechecks current database state, so an online backup safely defers the physical deletion. A pass count does not claim that every eligible file was deleted; service startup and upload reconciliation provide additional bounded opportunities. Read-only backup and verification commands never schedule this cleanup.

Every batch obtains BEGIN IMMEDIATE before sampling the clock and changing rows. A detected rollback is committed as a clock incident and stops the command. An existing clock incident also stops maintenance without compaction. The cutoff remains fixed at the safe time captured for the run, so advancing wall time during a multi-batch command cannot change which receipts are eligible. Each invocation and its aggregate effects are recorded in maintenance_runs. The run record is updated in the same short transactions as its compaction effects; a process interruption can therefore leave an inspectable running record without misreporting completed batches.

Permanent receipt tombstones

A mutation receipt’s result is replayable through 30 days. Maintenance selects only receipts for which safe_now - created_at is strictly greater than 30 days. It replaces result_json with JSON null and records compacted_at. It never deletes or changes:

• principal_id
• operation and idempotency key
• request fingerprint
• authority epoch
• original creation time

Those columns form a permanent tombstone. Mutation lookup first checks the authority epoch and fingerprint, then returns idempotency_receipt_expired for a matching compacted receipt. A changed request still returns idempotency_conflict, and a pre-restore key still returns request_from_previous_restore. No tombstoned key can execute again. Result compaction therefore saves potentially large response bodies without weakening exactly-once mutation identity.

Redundant producer observations

Producer observation rows and sequence identities remain permanent. The engine may clear only the summary of an old, strictly redundant intermediate running observation. Both its immediately preceding and following observations must have exactly the same state, PID, process-start identity, exit/input fields, and summary. The row must be older than the same 30-day cutoff and cannot be the first or last observation for its reporter.

payload_compacted_at explicitly identifies this elision. The original request_hash, reporter, sequence, producer identity, state, process identity, result fields, and observation time remain unchanged. The current jobs.summary projection is never cleared. First and last evidence, terminal states, unknown, state transitions, changed summaries, and nonempty progress transitions are always retained. Request replay identity therefore remains trustworthy even when a redundant middle display payload is gone.

Each batch first selects at most batch_size old, previously uninspected rows through the retention-scan index. Only those rows run the neighbor checks, and all selected rows receive retention_checked_at whether or not their summaries are eligible. The indexed remaining check looks only for old uninspected rows. Thus a history containing arbitrarily many distinct progress messages cannot turn one batch into an unbounded scan under the SQLite writer lock.

Summary elision is conservative and best effort. Clearing one eligible summary can prevent an adjacent row considered in a later batch from proving that its original neighboring summary matched. The engine leaves such a row intact; it does not retain hidden copies of display text merely to maximize compaction. The compacted count therefore reports actual cleared summaries and does not promise that every duplicate middle summary will eventually be removed. While rows remain uninspected, remaining.observation_payloads is a conservative compatibility flag; remaining.observation_rows_to_inspect states its precise meaning.

Records retained in full

Maintenance does not delete or rewrite semantic events, projects, task definitions or revisions, dependencies, attempts, checkpoints, handoffs, checkouts, resource holds, jobs, submissions, artifact references, knowledge or lesson revisions, decisions, reviews, integration evidence, objectives, restore evidence, or closed import identities. Payload compaction does not directly remove artifact files. Each committed batch also invokes a bounded artifact cleanup pass using the artifact store’s database recheck and backup-aware GC lock.

The first release intentionally favors trustworthy provenance over reclaiming every byte. Event payloads are already compact, and receipt tombstones still use one bounded row per mutation. This policy bounds work per invocation but does not promise a storage ceiling for permanent semantic history. Deployment sizing and the 100,000-task load test must include those identity and audit rows.

Daily maintenance timer

The release includes a maintained maintenance service and daily timer. Install them beside the service and backup units, then enable the timer:

sudo install -o root -g root -m 0644 deploy/agent-coordinator-maintenance.service deploy/agent-coordinator-maintenance.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now agent-coordinator-maintenance.timer
sudo systemctl start agent-coordinator-maintenance.service
sudo journalctl -u agent-coordinator-maintenance.service

The default invocation processes at most 20 batches of 500 candidate rows per kind. Inspect the result’s remaining flags; a large existing history may need additional invocations. The five-minute unit timeout may interrupt between batches; committed batches remain valid and the next invocation continues with uninspected rows. Monitor failed units, remaining work, and disk capacity. Correct and reconcile a clock incident before retrying failed maintenance.

Linux capacity exercise

The release target is 20 projects, 50 live agent sessions, and 100,000 historical tasks. Run the disposable exercise against a freshly built executable:

cargo build --release --workspace --locked
sudo systemd-run --scope --quiet \
  -p MemoryMax=4G -p MemorySwapMax=0 -p CPUQuota=200% \
  runuser -u "$(id -un)" -- \
  /usr/bin/python3 scripts/load_acceptance.py \
    --server target/release/agent-coordinator-server \
    --duration 1800 --rate 50 --history 100000 \
    --require-baseline --report capacity-report.json

It creates a private temporary installation and synthetic canceled history while the service is stopped. Fixture records have task definitions and audit events; they do not claim that implementation or review occurred. After restarting, the exercise creates 50 independent authenticated harness sessions across 20 projects and claims one live task per session. A barrier starts 100 competing claims for one additional task; exactly one request must win.

Measured traffic consists of 40 reads and 10 ownership renewals per second. Reads cover task lists, task detail, project orientation, context search, and paginated events. All 50 active attempts are renewed throughout the run. Requests follow a fixed schedule; latency includes time waiting in the client queue. The exercise fails instead of quietly reducing load when 100 requests are pending. It also streams a 16 MiB artifact and creates a verified online backup during traffic. Backup waits until the first upload chunk is in flight. The report records both operation intervals and the runner requires them to overlap. In a full workload, both operations must complete while metadata traffic is still measured and at least one metadata request must complete during their overlap.

The report records per-operation and combined p95/p99, achieved request rate, errors, peak resident memory, scheduling delay, claim conflicts, and final ownership/foreign-key checks. Every metadata operation must have p95 below 500 milliseconds, with no unexpected error and at least 98% of the target rate. The report records and checks the exact read/write operation counts and successful renewal participation for every scheduled session. The final ownership check compares the exact 50 expected project, task, attempt, generation, principal, session, and credential tuples. It also requires current task pointers, unexpired leases, and the finalized artifact’s project, creator, size, and digest to match.

--require-baseline accepts only a clean source tree on Ubuntu 24.04 x86_64 with at least two available CPUs and a unified cgroup v2 that limits the whole runner scope to at most two CPUs and 4 GiB aggregate memory with swap disabled. The scope contains the load generator, service, backup, restore stage, and their charged page cache. The report records cpu.max, memory.max, memory.swap.max, and aggregate memory.current and memory.peak. Service, backup, and restore processes retain a 4 GiB address-space limit as a secondary bound. The service is pinned to two available CPUs; backup and restore use the same pair.

After measured traffic, the runner stops the live service and restores the captured 100,000-task snapshot into a fresh temporary directory. This restore stage must finish in less than one hour and pass SQLite integrity and foreign-key checks. It verifies historical task count and the restored reconciliation pause, including invalidated credentials, browser and agent sessions, human accounts, and active attempts. Its reported time measures file restore plus staged authority invalidation and validation. It is not a full service-recovery time: the human inspection checklist, old-installation fencing, client reconnection, and service resumption are outside this capacity runner. The small end-to-end backup smoke exercise separately tests completion of that workflow.

Transport for this capacity workload is authenticated loopback HTTP. The separate Linux installation exercise verifies systemd and HTTPS. The report includes executable SHA-256, clean source revision, database schema version, and GitHub repository, head SHA, ref, workflow, job, run ID, and attempt when those CI values exist. Keep it with the exact build’s other release evidence. The executable digest and CI build job together identify the tested binary; the runner never records credentials, request headers, raw errors, or captured server output. The last 120 one-second diagnostic samples retain service CPU time, aggregate CPU throttling counters, queue depth, completed counts, and background operation progress, including on a failed run. These counters help diagnose contention without recording request contents.

Release CI takes its executable from the accepted Linux package, verifies the archive and internal checksums, and checks the server digest against the package job’s recorded value before running the workload. A separate server-only build is insufficient to establish performance of that exact package: building the server and CLI together can select additional shared dependency features. --expected-server-sha256 makes a digest mismatch fail before fixture setup and records the expected identity alongside the observed executable digest.

Short runs (--duration 60, for example) are development checks and have full_acceptance: false; they may omit --require-baseline. A report sets full_acceptance_eligible: true only for the full workload on the verified baseline, then sets full_acceptance: true only after every workload, ownership, backup, and restore-stage assertion passes. The same exact ownership, workload-accounting, overlap, restore-stage, and sanitization checks still run, although a backup may finish after short metadata traffic ends. A 30-minute run without every verified baseline condition also has full_acceptance: false. Neither substitutes for the constrained release run. Local CPU contention from unrelated builds can affect the result and should be avoided for final measurements. The native Windows workstation exercise remains backlog item 7, separate from all of these checks.

Search index change

The volume exercise exposed a global full-text ranking query that scanned and sorted matching tasks from every project before limiting the response. Migration 0015 rebuilds only the derived search index, indexes project identity, and limits user search words to content fields. Search filters the project in the index and uses a bounded FTS rank cursor before joining task details. SQLite documents rank-based early termination for limited result sets. The existing task definitions and provenance stay intact.

Linux release acceptance evidence

Backlog item 6 passed main-agent review and all acceptance gates for candidate a16d879. Standard CI passed all 160 workspace tests, formatting, warnings-denied Clippy, build, both service/CLI smoke exercises, native Windows client/local-runner tests, and the locked dependency audit. Release CI passed native Linux/Windows package reproducibility and Ubuntu 24.04 installation with trusted HTTPS, restart/reconnect, both timers, and a verified first backup.

Packaged-server capacity

The accepted packaged server completed 90,000 requests in 30 minutes at 50 requests/second across 20 projects and 50 sessions with 100,000 historical tasks. Overall p95 was 21.0 ms and p99 was 21.7 ms; every operation met the 500 ms p95 target. There were no unexpected errors, and exact ownership checks passed. The shared two-CPU/4-GiB/no-swap scope peaked at 664.1 MiB; server RSS peaked at 84.6 MiB. The captured snapshot restored to a verified, authority- invalidated reconciliation pause in 6.511 seconds.

The tested source was clean PR merge revision ef10939b6aa3c5b755e650807e99f241307d2ea9. The capacity job downloaded the accepted Linux archive and verified its checksums before testing. Its observed and expected server SHA-256 both were:

0f1f8288987a40cb84ab5396cb30c0918b17336dcbf5d083170e0fc86058d9dc

The mix was 72,000 reads and 18,000 renewals; each of the 50 sessions renewed successfully 360 times. A burst of 100 claims produced exactly one winner and 99 expected conflicts. Queue depth peaked at 26; maximum client scheduling delay was 10.057 ms and is included in request latency.

The 16 MiB upload took 2.709 seconds. The 171,532,860-byte online snapshot took 6.565 seconds. Their intervals overlapped by 2.707 seconds while metadata traffic continued.

Scope and limits

History consists of synthetic canceled tasks with definitions and audit events, distributed across the projects while the service was stopped. The workload is authenticated loopback HTTP; the separate installation gate verifies HTTPS. The restore measurement covers private-directory restore, integrity/foreign-key checks, the exact historical count, invalidated authority, and reconciliation pause. It excludes human inspection, old-installation fencing, reconnection, and service resumption. The concurrent snapshot may precede upload finalization; live artifact ownership and digest are checked separately.

The small end-to-end recovery smoke separately passed in 7.3 seconds. An actual prior schema-12 executable’s database upgraded to schema 16 with work retained; its snapshot verified unchanged and restored with old access rejected. No production installation or actual off-server transfer is claimed. The real Windows workstation exercise remains backlog item 7.

Retained machine-readable report

The complete sanitized report is preserved below. Its original file SHA-256 is 378210fb920bf55549a3a9d179d7038e5b62a9cd543297ac301ef028e5899534.

{
  "schema_version": 2,
  "passed": true,
  "full_acceptance": true,
  "full_acceptance_eligible": true,
  "baseline_required": true,
  "duration_requested_seconds": 1800,
  "requests_per_second": 50,
  "projects": 20,
  "agent_sessions": 50,
  "historical_tasks": 100000,
  "historical_fixture": "Synthetic canceled tasks, definition revisions, and audit events; seeded with service stopped.",
  "host": {
    "os": "Linux-6.17.0-1022-azure-x86_64-with-glibc2.39",
    "architecture": "x86_64",
    "service_cpu_cores": 2,
    "service_address_space_limit_bytes": 4294967296,
    "memory_limit_kind": "RLIMIT_AS per service/backup/restore process; cgroup v2 is required for aggregate baseline acceptance",
    "aggregate_cgroup": {
      "version": 2,
      "cpu_stat": {
        "usage_usec": 701457173,
        "user_usec": 501619525,
        "system_usec": 199837648,
        "nr_periods": 18553,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "available": true,
      "memory_max_bytes": 4294967296,
      "memory_swap_max_bytes": 0,
      "memory_current_bytes": 303566848,
      "memory_peak_bytes": 696315904,
      "cpu_quota_micros": 200000,
      "cpu_period_micros": 100000
    }
  },
  "transport": "Authenticated loopback HTTP; HTTPS installation is exercised separately.",
  "source_commit": "ef10939b6aa3c5b755e650807e99f241307d2ea9",
  "source_dirty": false,
  "executable_sha256": "0f1f8288987a40cb84ab5396cb30c0918b17336dcbf5d083170e0fc86058d9dc",
  "expected_server_sha256": "0f1f8288987a40cb84ab5396cb30c0918b17336dcbf5d083170e0fc86058d9dc",
  "executable_matches_expected": true,
  "github": {
    "github_repository": "marshallr12/agent_coordinator",
    "github_run_id": "34459987847",
    "github_run_attempt": "1",
    "github_sha": "ef10939b6aa3c5b755e650807e99f241307d2ea9",
    "github_ref": "refs/pull/1/merge",
    "github_workflow": "Release package acceptance",
    "github_job": "linux-load"
  },
  "baseline": {
    "verified": true,
    "checks": {
      "ubuntu_24_04": true,
      "x86_64": true,
      "source_clean": true,
      "two_cpus_available": true,
      "cgroup_v2": true,
      "aggregate_memory_max_at_most_4_gib": true,
      "aggregate_swap_disabled": true,
      "aggregate_cpu_quota_at_most_two_cores": true,
      "aggregate_memory_peak_available": true,
      "github_head_matches_source": true
    },
    "os_id": "ubuntu",
    "os_version_id": "24.04",
    "architecture": "x86_64",
    "cgroup": {
      "version": 2,
      "cpu_stat": {
        "usage_usec": 104766,
        "user_usec": 75830,
        "system_usec": 28935,
        "nr_periods": 1,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "available": true,
      "memory_max_bytes": 4294967296,
      "memory_swap_max_bytes": 0,
      "memory_current_bytes": 15167488,
      "memory_peak_bytes": 17379328,
      "cpu_quota_micros": 200000,
      "cpu_period_micros": 100000,
      "cpu_quota_cores": 2.0
    }
  },
  "full_workload": true,
  "database_schema_version": 16,
  "claim_burst": {
    "requests": 100,
    "sessions": 50,
    "winners": 1,
    "expected_conflicts": 99
  },
  "concurrent_operations": {
    "upload": {
      "passed": true,
      "bytes": 16777216,
      "seconds": 2.709,
      "started_offset_seconds": 30.0,
      "finished_offset_seconds": 32.709
    },
    "backup": {
      "passed": true,
      "seconds": 6.565,
      "database_bytes": 171532288,
      "snapshot_bytes": 171532860,
      "started_offset_seconds": 30.002,
      "finished_offset_seconds": 36.567
    },
    "overlap_seconds": 2.707
  },
  "total_including_background_seconds": 1799.989,
  "elapsed_seconds": 1800,
  "completed_requests": 90000,
  "achieved_requests_per_second": 50.0,
  "load_model": "open_loop_fixed_schedule",
  "latency_includes_client_schedule_delay": true,
  "metadata_p95_ms": 20.974,
  "metadata_p99_ms": 21.695,
  "operations": {
    "task_list": {
      "count": 14400,
      "p95_ms": 4.796,
      "p99_ms": 6.948
    },
    "task_detail": {
      "count": 14400,
      "p95_ms": 2.742,
      "p99_ms": 3.833
    },
    "orientation": {
      "count": 14400,
      "p95_ms": 2.551,
      "p99_ms": 3.493
    },
    "renew": {
      "count": 18000,
      "p95_ms": 3.63,
      "p99_ms": 6.335
    },
    "context": {
      "count": 14400,
      "p95_ms": 21.688,
      "p99_ms": 30.511
    },
    "history": {
      "count": 14400,
      "p95_ms": 12.718,
      "p99_ms": 18.67
    }
  },
  "workload_mix": {
    "reads": 72000,
    "writes": 18000,
    "scheduled_reads_per_second": 40.0,
    "scheduled_writes_per_second": 10.0,
    "sessions_with_successful_renewals": 50,
    "successful_renewals_per_participating_session_min": 360,
    "successful_renewals_per_participating_session_max": 360
  },
  "unexpected_errors": {},
  "maxima": {
    "rss_bytes": 88707072,
    "inflight": 26,
    "schedule_lag_ms": 10.056553000012514
  },
  "observed_projects": 20,
  "observed_agent_sessions": 50,
  "ownership_invariants_passed": true,
  "artifact_ownership_invariant_passed": true,
  "restore_stage": {
    "measurement": "restore_stage",
    "passed": true,
    "full_service_recovery": false,
    "seconds": 6.511,
    "target_seconds": 3600,
    "within_target": true,
    "integrity_ok": true,
    "foreign_keys_ok": true,
    "history_count_ok": true,
    "coordination_paused": true,
    "credentials_revoked": true,
    "browser_sessions_revoked": true,
    "agent_sessions_closed": true,
    "human_accounts_disabled": true,
    "no_active_attempts": true,
    "reporters_expired": true,
    "integration_authorizations_invalidated": true,
    "old_receipts_invalidated": true,
    "coordination_state": "restore_reconciliation",
    "database_bytes": 171540480,
    "artifact_count": 0,
    "artifact_bytes": 0,
    "historical_tasks": 100000,
    "credential_count": 1,
    "browser_session_count": 2,
    "agent_session_count": 50,
    "human_account_count": 1,
    "attempt_count": 51
  },
  "recent_diagnostics": [
    {
      "elapsed_seconds": 1680.0,
      "completed_requests": 84000,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 552.5,
      "aggregate_cpu_stat": {
        "usage_usec": 651360639,
        "user_usec": 466324261,
        "system_usec": 185036377,
        "nr_periods": 17286,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1681.0,
      "completed_requests": 84050,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 552.83,
      "aggregate_cpu_stat": {
        "usage_usec": 651729030,
        "user_usec": 466596891,
        "system_usec": 185132139,
        "nr_periods": 17296,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1682.0,
      "completed_requests": 84100,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 553.16,
      "aggregate_cpu_stat": {
        "usage_usec": 652109612,
        "user_usec": 466855308,
        "system_usec": 185254303,
        "nr_periods": 17306,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1683.0,
      "completed_requests": 84150,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 553.48,
      "aggregate_cpu_stat": {
        "usage_usec": 652470947,
        "user_usec": 467111443,
        "system_usec": 185359504,
        "nr_periods": 17316,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1684.0,
      "completed_requests": 84200,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 553.81,
      "aggregate_cpu_stat": {
        "usage_usec": 652830380,
        "user_usec": 467364252,
        "system_usec": 185466128,
        "nr_periods": 17326,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1685.0,
      "completed_requests": 84250,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 554.12,
      "aggregate_cpu_stat": {
        "usage_usec": 653191479,
        "user_usec": 467633299,
        "system_usec": 185558180,
        "nr_periods": 17336,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1686.0,
      "completed_requests": 84300,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 554.45,
      "aggregate_cpu_stat": {
        "usage_usec": 653563891,
        "user_usec": 467898001,
        "system_usec": 185665890,
        "nr_periods": 17346,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1687.0,
      "completed_requests": 84350,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 554.77,
      "aggregate_cpu_stat": {
        "usage_usec": 653922223,
        "user_usec": 468150191,
        "system_usec": 185772032,
        "nr_periods": 17356,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1688.0,
      "completed_requests": 84400,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 555.1,
      "aggregate_cpu_stat": {
        "usage_usec": 654279761,
        "user_usec": 468422409,
        "system_usec": 185857351,
        "nr_periods": 17366,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1689.0,
      "completed_requests": 84450,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 555.41,
      "aggregate_cpu_stat": {
        "usage_usec": 654636342,
        "user_usec": 468683150,
        "system_usec": 185953191,
        "nr_periods": 17376,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1690.0,
      "completed_requests": 84500,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 555.73,
      "aggregate_cpu_stat": {
        "usage_usec": 654995686,
        "user_usec": 468929342,
        "system_usec": 186066344,
        "nr_periods": 17386,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1691.0,
      "completed_requests": 84550,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 556.06,
      "aggregate_cpu_stat": {
        "usage_usec": 655356126,
        "user_usec": 469186149,
        "system_usec": 186169977,
        "nr_periods": 17396,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1692.0,
      "completed_requests": 84600,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 556.37,
      "aggregate_cpu_stat": {
        "usage_usec": 655709493,
        "user_usec": 469438204,
        "system_usec": 186271288,
        "nr_periods": 17406,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1693.0,
      "completed_requests": 84650,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 556.7,
      "aggregate_cpu_stat": {
        "usage_usec": 656066931,
        "user_usec": 469693837,
        "system_usec": 186373093,
        "nr_periods": 17416,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1694.0,
      "completed_requests": 84700,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 557.0,
      "aggregate_cpu_stat": {
        "usage_usec": 656421666,
        "user_usec": 469941827,
        "system_usec": 186479839,
        "nr_periods": 17426,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1695.0,
      "completed_requests": 84750,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 557.33,
      "aggregate_cpu_stat": {
        "usage_usec": 656778791,
        "user_usec": 470206389,
        "system_usec": 186572401,
        "nr_periods": 17436,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1696.0,
      "completed_requests": 84800,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 557.65,
      "aggregate_cpu_stat": {
        "usage_usec": 657137216,
        "user_usec": 470465173,
        "system_usec": 186672042,
        "nr_periods": 17446,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1697.0,
      "completed_requests": 84850,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 557.96,
      "aggregate_cpu_stat": {
        "usage_usec": 657492574,
        "user_usec": 470714402,
        "system_usec": 186778171,
        "nr_periods": 17456,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1698.0,
      "completed_requests": 84900,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 558.29,
      "aggregate_cpu_stat": {
        "usage_usec": 657852746,
        "user_usec": 470956130,
        "system_usec": 186896615,
        "nr_periods": 17466,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1699.0,
      "completed_requests": 84950,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 558.61,
      "aggregate_cpu_stat": {
        "usage_usec": 658214775,
        "user_usec": 471220108,
        "system_usec": 186994667,
        "nr_periods": 17476,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1700.0,
      "completed_requests": 85000,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 558.93,
      "aggregate_cpu_stat": {
        "usage_usec": 658575940,
        "user_usec": 471480685,
        "system_usec": 187095255,
        "nr_periods": 17486,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1701.0,
      "completed_requests": 85050,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 559.25,
      "aggregate_cpu_stat": {
        "usage_usec": 658943093,
        "user_usec": 471744911,
        "system_usec": 187198182,
        "nr_periods": 17496,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1702.0,
      "completed_requests": 85100,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 559.58,
      "aggregate_cpu_stat": {
        "usage_usec": 659312515,
        "user_usec": 472022528,
        "system_usec": 187289986,
        "nr_periods": 17506,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1703.0,
      "completed_requests": 85150,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 559.9,
      "aggregate_cpu_stat": {
        "usage_usec": 659684904,
        "user_usec": 472290816,
        "system_usec": 187394088,
        "nr_periods": 17516,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1704.0,
      "completed_requests": 85200,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 560.22,
      "aggregate_cpu_stat": {
        "usage_usec": 660047744,
        "user_usec": 472559782,
        "system_usec": 187487962,
        "nr_periods": 17526,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1705.0,
      "completed_requests": 85250,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 560.54,
      "aggregate_cpu_stat": {
        "usage_usec": 660411048,
        "user_usec": 472833033,
        "system_usec": 187578015,
        "nr_periods": 17536,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1706.0,
      "completed_requests": 85300,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 560.86,
      "aggregate_cpu_stat": {
        "usage_usec": 660773129,
        "user_usec": 473083500,
        "system_usec": 187689629,
        "nr_periods": 17546,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1707.0,
      "completed_requests": 85350,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 561.19,
      "aggregate_cpu_stat": {
        "usage_usec": 661134525,
        "user_usec": 473350977,
        "system_usec": 187783547,
        "nr_periods": 17556,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1708.0,
      "completed_requests": 85400,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 561.51,
      "aggregate_cpu_stat": {
        "usage_usec": 661502198,
        "user_usec": 473627019,
        "system_usec": 187875178,
        "nr_periods": 17566,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1709.0,
      "completed_requests": 85450,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 561.85,
      "aggregate_cpu_stat": {
        "usage_usec": 661887438,
        "user_usec": 473902517,
        "system_usec": 187984921,
        "nr_periods": 17576,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1710.0,
      "completed_requests": 85500,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 562.22,
      "aggregate_cpu_stat": {
        "usage_usec": 662294712,
        "user_usec": 474184907,
        "system_usec": 188109804,
        "nr_periods": 17586,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1711.0,
      "completed_requests": 85550,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 562.54,
      "aggregate_cpu_stat": {
        "usage_usec": 662662208,
        "user_usec": 474457376,
        "system_usec": 188204832,
        "nr_periods": 17596,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1712.0,
      "completed_requests": 85600,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 562.85,
      "aggregate_cpu_stat": {
        "usage_usec": 663018410,
        "user_usec": 474726518,
        "system_usec": 188291891,
        "nr_periods": 17606,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1713.0,
      "completed_requests": 85650,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 563.17,
      "aggregate_cpu_stat": {
        "usage_usec": 663379594,
        "user_usec": 474990060,
        "system_usec": 188389534,
        "nr_periods": 17616,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1714.0,
      "completed_requests": 85700,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 563.5,
      "aggregate_cpu_stat": {
        "usage_usec": 663741147,
        "user_usec": 475255532,
        "system_usec": 188485615,
        "nr_periods": 17626,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1715.0,
      "completed_requests": 85750,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 563.82,
      "aggregate_cpu_stat": {
        "usage_usec": 664107134,
        "user_usec": 475533814,
        "system_usec": 188573320,
        "nr_periods": 17636,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1716.0,
      "completed_requests": 85800,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 564.14,
      "aggregate_cpu_stat": {
        "usage_usec": 664467345,
        "user_usec": 475788833,
        "system_usec": 188678512,
        "nr_periods": 17646,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1717.0,
      "completed_requests": 85850,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 564.46,
      "aggregate_cpu_stat": {
        "usage_usec": 664821328,
        "user_usec": 476052149,
        "system_usec": 188769179,
        "nr_periods": 17656,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1718.0,
      "completed_requests": 85900,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 564.77,
      "aggregate_cpu_stat": {
        "usage_usec": 665179986,
        "user_usec": 476323234,
        "system_usec": 188856751,
        "nr_periods": 17666,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1719.0,
      "completed_requests": 85950,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 565.1,
      "aggregate_cpu_stat": {
        "usage_usec": 665544834,
        "user_usec": 476579440,
        "system_usec": 188965393,
        "nr_periods": 17676,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1720.0,
      "completed_requests": 86000,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 565.42,
      "aggregate_cpu_stat": {
        "usage_usec": 665907204,
        "user_usec": 476837785,
        "system_usec": 189069419,
        "nr_periods": 17686,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1721.0,
      "completed_requests": 86050,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 565.75,
      "aggregate_cpu_stat": {
        "usage_usec": 666283620,
        "user_usec": 477105679,
        "system_usec": 189177941,
        "nr_periods": 17696,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1722.0,
      "completed_requests": 86100,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 566.07,
      "aggregate_cpu_stat": {
        "usage_usec": 666655546,
        "user_usec": 477381165,
        "system_usec": 189274381,
        "nr_periods": 17706,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1723.0,
      "completed_requests": 86150,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 566.41,
      "aggregate_cpu_stat": {
        "usage_usec": 667025574,
        "user_usec": 477645459,
        "system_usec": 189380114,
        "nr_periods": 17716,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1724.0,
      "completed_requests": 86200,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 566.72,
      "aggregate_cpu_stat": {
        "usage_usec": 667389784,
        "user_usec": 477904457,
        "system_usec": 189485326,
        "nr_periods": 17726,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1725.0,
      "completed_requests": 86250,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 567.05,
      "aggregate_cpu_stat": {
        "usage_usec": 667758403,
        "user_usec": 478182168,
        "system_usec": 189576235,
        "nr_periods": 17736,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1726.0,
      "completed_requests": 86300,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 567.38,
      "aggregate_cpu_stat": {
        "usage_usec": 668128636,
        "user_usec": 478451200,
        "system_usec": 189677435,
        "nr_periods": 17746,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1727.0,
      "completed_requests": 86350,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 567.72,
      "aggregate_cpu_stat": {
        "usage_usec": 668508279,
        "user_usec": 478729571,
        "system_usec": 189778708,
        "nr_periods": 17756,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1728.0,
      "completed_requests": 86400,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 568.05,
      "aggregate_cpu_stat": {
        "usage_usec": 668883246,
        "user_usec": 479018495,
        "system_usec": 189864751,
        "nr_periods": 17766,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1729.0,
      "completed_requests": 86450,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 568.37,
      "aggregate_cpu_stat": {
        "usage_usec": 669259133,
        "user_usec": 479296154,
        "system_usec": 189962978,
        "nr_periods": 17776,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1730.0,
      "completed_requests": 86500,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 568.7,
      "aggregate_cpu_stat": {
        "usage_usec": 669632419,
        "user_usec": 479561172,
        "system_usec": 190071246,
        "nr_periods": 17786,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1731.0,
      "completed_requests": 86550,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 569.05,
      "aggregate_cpu_stat": {
        "usage_usec": 670022128,
        "user_usec": 479843783,
        "system_usec": 190178344,
        "nr_periods": 17796,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1732.0,
      "completed_requests": 86600,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 569.36,
      "aggregate_cpu_stat": {
        "usage_usec": 670386306,
        "user_usec": 480105711,
        "system_usec": 190280594,
        "nr_periods": 17806,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1733.0,
      "completed_requests": 86650,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 569.74,
      "aggregate_cpu_stat": {
        "usage_usec": 670807741,
        "user_usec": 480388405,
        "system_usec": 190419335,
        "nr_periods": 17816,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1734.0,
      "completed_requests": 86700,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 570.08,
      "aggregate_cpu_stat": {
        "usage_usec": 671182872,
        "user_usec": 480671045,
        "system_usec": 190511826,
        "nr_periods": 17826,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1735.0,
      "completed_requests": 86750,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 570.41,
      "aggregate_cpu_stat": {
        "usage_usec": 671564107,
        "user_usec": 480950217,
        "system_usec": 190613890,
        "nr_periods": 17836,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1736.0,
      "completed_requests": 86800,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 570.75,
      "aggregate_cpu_stat": {
        "usage_usec": 671932381,
        "user_usec": 481227495,
        "system_usec": 190704885,
        "nr_periods": 17846,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1737.0,
      "completed_requests": 86850,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 571.07,
      "aggregate_cpu_stat": {
        "usage_usec": 672299617,
        "user_usec": 481484733,
        "system_usec": 190814884,
        "nr_periods": 17856,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1738.0,
      "completed_requests": 86900,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 571.39,
      "aggregate_cpu_stat": {
        "usage_usec": 672664644,
        "user_usec": 481744485,
        "system_usec": 190920158,
        "nr_periods": 17866,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1739.0,
      "completed_requests": 86950,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 571.73,
      "aggregate_cpu_stat": {
        "usage_usec": 673042583,
        "user_usec": 482018512,
        "system_usec": 191024071,
        "nr_periods": 17876,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1740.0,
      "completed_requests": 87000,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 572.05,
      "aggregate_cpu_stat": {
        "usage_usec": 673408619,
        "user_usec": 482285544,
        "system_usec": 191123075,
        "nr_periods": 17886,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1741.0,
      "completed_requests": 87050,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 572.37,
      "aggregate_cpu_stat": {
        "usage_usec": 673776637,
        "user_usec": 482562500,
        "system_usec": 191214136,
        "nr_periods": 17896,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1742.0,
      "completed_requests": 87100,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 572.7,
      "aggregate_cpu_stat": {
        "usage_usec": 674144743,
        "user_usec": 482844733,
        "system_usec": 191300010,
        "nr_periods": 17906,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1743.0,
      "completed_requests": 87150,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 573.03,
      "aggregate_cpu_stat": {
        "usage_usec": 674511303,
        "user_usec": 483114259,
        "system_usec": 191397043,
        "nr_periods": 17916,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1744.0,
      "completed_requests": 87200,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 573.35,
      "aggregate_cpu_stat": {
        "usage_usec": 674879847,
        "user_usec": 483396480,
        "system_usec": 191483366,
        "nr_periods": 17926,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1745.0,
      "completed_requests": 87250,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 573.67,
      "aggregate_cpu_stat": {
        "usage_usec": 675245390,
        "user_usec": 483677234,
        "system_usec": 191568155,
        "nr_periods": 17936,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1746.0,
      "completed_requests": 87300,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 574.0,
      "aggregate_cpu_stat": {
        "usage_usec": 675609991,
        "user_usec": 483954189,
        "system_usec": 191655801,
        "nr_periods": 17946,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1747.0,
      "completed_requests": 87350,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 574.31,
      "aggregate_cpu_stat": {
        "usage_usec": 675971878,
        "user_usec": 484233624,
        "system_usec": 191738254,
        "nr_periods": 17956,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1748.0,
      "completed_requests": 87400,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 574.64,
      "aggregate_cpu_stat": {
        "usage_usec": 676331428,
        "user_usec": 484498608,
        "system_usec": 191832819,
        "nr_periods": 17966,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1749.0,
      "completed_requests": 87450,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 574.96,
      "aggregate_cpu_stat": {
        "usage_usec": 676696417,
        "user_usec": 484757684,
        "system_usec": 191938732,
        "nr_periods": 17976,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1750.0,
      "completed_requests": 87500,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 575.3,
      "aggregate_cpu_stat": {
        "usage_usec": 677082962,
        "user_usec": 485036406,
        "system_usec": 192046556,
        "nr_periods": 17986,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1751.0,
      "completed_requests": 87550,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 575.62,
      "aggregate_cpu_stat": {
        "usage_usec": 677444615,
        "user_usec": 485287387,
        "system_usec": 192157228,
        "nr_periods": 17996,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1752.0,
      "completed_requests": 87600,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 575.95,
      "aggregate_cpu_stat": {
        "usage_usec": 677806211,
        "user_usec": 485550915,
        "system_usec": 192255295,
        "nr_periods": 18006,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1753.0,
      "completed_requests": 87650,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 576.26,
      "aggregate_cpu_stat": {
        "usage_usec": 678165497,
        "user_usec": 485812294,
        "system_usec": 192353203,
        "nr_periods": 18016,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1754.0,
      "completed_requests": 87700,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 576.59,
      "aggregate_cpu_stat": {
        "usage_usec": 678528656,
        "user_usec": 486074005,
        "system_usec": 192454651,
        "nr_periods": 18026,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1755.0,
      "completed_requests": 87750,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 576.91,
      "aggregate_cpu_stat": {
        "usage_usec": 678900275,
        "user_usec": 486352538,
        "system_usec": 192547737,
        "nr_periods": 18036,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1756.0,
      "completed_requests": 87800,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 577.23,
      "aggregate_cpu_stat": {
        "usage_usec": 679261080,
        "user_usec": 486608617,
        "system_usec": 192652463,
        "nr_periods": 18046,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1757.0,
      "completed_requests": 87850,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 577.55,
      "aggregate_cpu_stat": {
        "usage_usec": 679619807,
        "user_usec": 486855366,
        "system_usec": 192764441,
        "nr_periods": 18056,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1758.0,
      "completed_requests": 87900,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 577.87,
      "aggregate_cpu_stat": {
        "usage_usec": 679979952,
        "user_usec": 487117992,
        "system_usec": 192861959,
        "nr_periods": 18066,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1759.0,
      "completed_requests": 87950,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 578.2,
      "aggregate_cpu_stat": {
        "usage_usec": 680352222,
        "user_usec": 487390450,
        "system_usec": 192961772,
        "nr_periods": 18076,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1760.0,
      "completed_requests": 88000,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 578.53,
      "aggregate_cpu_stat": {
        "usage_usec": 680735813,
        "user_usec": 487659709,
        "system_usec": 193076103,
        "nr_periods": 18086,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1761.0,
      "completed_requests": 88050,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 578.86,
      "aggregate_cpu_stat": {
        "usage_usec": 681092773,
        "user_usec": 487923675,
        "system_usec": 193169097,
        "nr_periods": 18096,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1762.0,
      "completed_requests": 88100,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 579.17,
      "aggregate_cpu_stat": {
        "usage_usec": 681449384,
        "user_usec": 488177570,
        "system_usec": 193271814,
        "nr_periods": 18106,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1763.0,
      "completed_requests": 88150,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 579.49,
      "aggregate_cpu_stat": {
        "usage_usec": 681812731,
        "user_usec": 488444284,
        "system_usec": 193368447,
        "nr_periods": 18116,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1764.0,
      "completed_requests": 88200,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 579.81,
      "aggregate_cpu_stat": {
        "usage_usec": 682171908,
        "user_usec": 488701486,
        "system_usec": 193470422,
        "nr_periods": 18126,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1765.0,
      "completed_requests": 88250,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 580.13,
      "aggregate_cpu_stat": {
        "usage_usec": 682527395,
        "user_usec": 488960313,
        "system_usec": 193567081,
        "nr_periods": 18136,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1766.0,
      "completed_requests": 88300,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 580.45,
      "aggregate_cpu_stat": {
        "usage_usec": 682896235,
        "user_usec": 489231766,
        "system_usec": 193664469,
        "nr_periods": 18146,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1767.0,
      "completed_requests": 88350,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 580.78,
      "aggregate_cpu_stat": {
        "usage_usec": 683251428,
        "user_usec": 489474002,
        "system_usec": 193777425,
        "nr_periods": 18156,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1768.0,
      "completed_requests": 88400,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 581.1,
      "aggregate_cpu_stat": {
        "usage_usec": 683612057,
        "user_usec": 489741553,
        "system_usec": 193870504,
        "nr_periods": 18166,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1769.0,
      "completed_requests": 88450,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 581.41,
      "aggregate_cpu_stat": {
        "usage_usec": 683971888,
        "user_usec": 490007046,
        "system_usec": 193964841,
        "nr_periods": 18176,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1770.0,
      "completed_requests": 88500,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 581.74,
      "aggregate_cpu_stat": {
        "usage_usec": 684328284,
        "user_usec": 490252434,
        "system_usec": 194075850,
        "nr_periods": 18186,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1771.0,
      "completed_requests": 88550,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 582.05,
      "aggregate_cpu_stat": {
        "usage_usec": 684687102,
        "user_usec": 490519028,
        "system_usec": 194168073,
        "nr_periods": 18196,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1772.0,
      "completed_requests": 88600,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 582.37,
      "aggregate_cpu_stat": {
        "usage_usec": 685042653,
        "user_usec": 490774434,
        "system_usec": 194268218,
        "nr_periods": 18206,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1773.0,
      "completed_requests": 88650,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 582.68,
      "aggregate_cpu_stat": {
        "usage_usec": 685395335,
        "user_usec": 491025839,
        "system_usec": 194369496,
        "nr_periods": 18216,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1774.0,
      "completed_requests": 88700,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 583.01,
      "aggregate_cpu_stat": {
        "usage_usec": 685762303,
        "user_usec": 491296751,
        "system_usec": 194465551,
        "nr_periods": 18226,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1775.0,
      "completed_requests": 88750,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 583.33,
      "aggregate_cpu_stat": {
        "usage_usec": 686120576,
        "user_usec": 491546422,
        "system_usec": 194574154,
        "nr_periods": 18236,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1776.0,
      "completed_requests": 88800,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 583.65,
      "aggregate_cpu_stat": {
        "usage_usec": 686481197,
        "user_usec": 491820012,
        "system_usec": 194661185,
        "nr_periods": 18246,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1777.0,
      "completed_requests": 88850,
      "pending_requests": 1,
      "service_rss_bytes": 88055808,
      "service_cpu_seconds": 583.97,
      "aggregate_cpu_stat": {
        "usage_usec": 686838941,
        "user_usec": 492083206,
        "system_usec": 194755735,
        "nr_periods": 18256,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1778.0,
      "completed_requests": 88900,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 584.32,
      "aggregate_cpu_stat": {
        "usage_usec": 687217356,
        "user_usec": 492359842,
        "system_usec": 194857514,
        "nr_periods": 18266,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1779.0,
      "completed_requests": 88950,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 584.64,
      "aggregate_cpu_stat": {
        "usage_usec": 687586202,
        "user_usec": 492626908,
        "system_usec": 194959294,
        "nr_periods": 18276,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1780.0,
      "completed_requests": 89000,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 584.96,
      "aggregate_cpu_stat": {
        "usage_usec": 687949629,
        "user_usec": 492881262,
        "system_usec": 195068367,
        "nr_periods": 18286,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1781.0,
      "completed_requests": 89050,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 585.27,
      "aggregate_cpu_stat": {
        "usage_usec": 688300817,
        "user_usec": 493151218,
        "system_usec": 195149599,
        "nr_periods": 18296,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1782.0,
      "completed_requests": 89100,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 585.59,
      "aggregate_cpu_stat": {
        "usage_usec": 688652723,
        "user_usec": 493400101,
        "system_usec": 195252621,
        "nr_periods": 18306,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1783.0,
      "completed_requests": 89150,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 585.92,
      "aggregate_cpu_stat": {
        "usage_usec": 689020265,
        "user_usec": 493671765,
        "system_usec": 195348499,
        "nr_periods": 18316,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1784.0,
      "completed_requests": 89200,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 586.25,
      "aggregate_cpu_stat": {
        "usage_usec": 689393714,
        "user_usec": 493935536,
        "system_usec": 195458178,
        "nr_periods": 18326,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1785.0,
      "completed_requests": 89250,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 586.57,
      "aggregate_cpu_stat": {
        "usage_usec": 689746039,
        "user_usec": 494186171,
        "system_usec": 195559868,
        "nr_periods": 18336,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1786.0,
      "completed_requests": 89300,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 586.88,
      "aggregate_cpu_stat": {
        "usage_usec": 690098751,
        "user_usec": 494441028,
        "system_usec": 195657723,
        "nr_periods": 18346,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1787.0,
      "completed_requests": 89350,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 587.2,
      "aggregate_cpu_stat": {
        "usage_usec": 690448467,
        "user_usec": 494690000,
        "system_usec": 195758467,
        "nr_periods": 18356,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1788.0,
      "completed_requests": 89400,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 587.51,
      "aggregate_cpu_stat": {
        "usage_usec": 690798027,
        "user_usec": 494931959,
        "system_usec": 195866067,
        "nr_periods": 18366,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1789.0,
      "completed_requests": 89450,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 587.82,
      "aggregate_cpu_stat": {
        "usage_usec": 691147692,
        "user_usec": 495178743,
        "system_usec": 195968949,
        "nr_periods": 18376,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1790.0,
      "completed_requests": 89500,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 588.14,
      "aggregate_cpu_stat": {
        "usage_usec": 691501059,
        "user_usec": 495428844,
        "system_usec": 196072214,
        "nr_periods": 18386,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1791.0,
      "completed_requests": 89550,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 588.46,
      "aggregate_cpu_stat": {
        "usage_usec": 691855035,
        "user_usec": 495688518,
        "system_usec": 196166517,
        "nr_periods": 18396,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1792.0,
      "completed_requests": 89600,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 588.77,
      "aggregate_cpu_stat": {
        "usage_usec": 692209695,
        "user_usec": 495945095,
        "system_usec": 196264600,
        "nr_periods": 18406,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1793.0,
      "completed_requests": 89650,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 589.09,
      "aggregate_cpu_stat": {
        "usage_usec": 692562764,
        "user_usec": 496205927,
        "system_usec": 196356836,
        "nr_periods": 18416,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1794.0,
      "completed_requests": 89700,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 589.41,
      "aggregate_cpu_stat": {
        "usage_usec": 692920681,
        "user_usec": 496450922,
        "system_usec": 196469759,
        "nr_periods": 18426,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1795.0,
      "completed_requests": 89750,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 589.73,
      "aggregate_cpu_stat": {
        "usage_usec": 693276265,
        "user_usec": 496711729,
        "system_usec": 196564536,
        "nr_periods": 18436,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1796.0,
      "completed_requests": 89800,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 590.04,
      "aggregate_cpu_stat": {
        "usage_usec": 693629938,
        "user_usec": 496951738,
        "system_usec": 196678200,
        "nr_periods": 18446,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1797.0,
      "completed_requests": 89850,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 590.38,
      "aggregate_cpu_stat": {
        "usage_usec": 693996949,
        "user_usec": 497215985,
        "system_usec": 196780963,
        "nr_periods": 18456,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1798.0,
      "completed_requests": 89900,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 590.69,
      "aggregate_cpu_stat": {
        "usage_usec": 694348019,
        "user_usec": 497485526,
        "system_usec": 196862492,
        "nr_periods": 18466,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    },
    {
      "elapsed_seconds": 1799.0,
      "completed_requests": 89950,
      "pending_requests": 1,
      "service_rss_bytes": 88707072,
      "service_cpu_seconds": 591.01,
      "aggregate_cpu_stat": {
        "usage_usec": 694700765,
        "user_usec": 497726510,
        "system_usec": 196974254,
        "nr_periods": 18476,
        "nr_throttled": 0,
        "throttled_usec": 0
      },
      "upload_started": true,
      "upload_finished": true,
      "backup_started": true,
      "backup_finished": true
    }
  ],
  "background_progress": {
    "upload_started": true,
    "upload_finished": true,
    "backup_started": true,
    "backup_finished": true
  }
}

First implementation slice: integration contract

Status: historical integration contract for the first implementation slice. See implementation status for the complete current service.

This file coordinated parallel implementation, not a change to product scope. Root owns workspace manifests/core types, server coordination.rs, migration 0002, mutation.rs, integration tests, and final integration. Authentication work owns server lib.rs/main.rs/state.rs/error.rs/auth.rs, migration 0001, auth tests and systemd/proxy examples. CLI work owns client/cli crates. UI owns web assets.

Shared Rust interface

• state::AppState: Clone, public pool: sqlx::SqlitePool, config: Config, clock: Arc<dyn Clock>; now() -> i64 milliseconds; open(Config) -> anyhow::Result<Self>. Clock: Send + Sync + fn now_ms(&self) -> i64.
• Config fields: database_path: PathBuf, listen: SocketAddr, public_origin: String, allow_insecure_loopback: bool. Default creates only loopback bindings; development HTTP requires explicit opt-in.
• auth::Auth: axum FromRequestParts<AppState>, public actor: Actor; verify(&self, &mut SqliteConnection, now: i64) -> Result<Actor, AppError> revalidates credential/session revocation inside mutation transaction.
• Actor: Clone, public string id,name,kind,role, optional string credential_id,session_id. kind human/agent; role admin/operator/agent. Valid supplied agent session headers populate session_id only with correct proof and matching active issuing credential. Humans have browser session ID there. Agent without session can read but not own an attempt.
• error::AppError: new(status: StatusCode, code: &str, message: &str), bad_request(&str), conflict(code:&str,message:&str), forbidden(&str), not_found(), auth_required(), with_details(Value); From<sqlx::Error>, serde_json::Error; IntoResponse JSON envelope. No secret Debug.
• response(data: Value) -> axum::Json<Value> produces {data,request_id,server_time}. UTC dates RFC3339. Root owns mutation.rs: Mutation::begin(&AppState,&Auth,&HeaderMap,operation:&str,&impl Serialize); fields tx: Transaction<'static, Sqlite>, actor: Actor, now: i64, and replay: Option<Value>; finish(...) -> Result<Value, AppError> returns data after commit. All writes use BEGIN IMMEDIATE, Auth::verify after lock, mandatory idempotency key, receipt and event in same tx. Replays still authenticate. Issued token receipt must be redacted; first response can attach secret after commit.
• lib router pub fn router(state: AppState)->axum::Router merges coordination::routes(), mounts public/assets/auth routes. Auth private extractors must run before decoding private bodies/project IDs; any global middleware needed for unknown private routes belongs in lib/auth.

Database access foundation (migration 0001)

principals(id TEXT PK,name TEXT UNIQUE,kind TEXT,role TEXT,password_hash TEXT NULL,disabled_at INTEGER NULL,created_at INTEGER); credentials(id TEXT PK,principal_id TEXT FK,token_hash TEXT UNIQUE, created_at INTEGER,revoked_at INTEGER NULL,expires_at INTEGER NULL); browser_sessions(id TEXT PK,principal_id TEXT FK,token_hash TEXT UNIQUE, expires_at INTEGER,revoked_at INTEGER NULL); agent_sessions(id TEXT PK,principal_id TEXT FK,credential_id TEXT FK, workstation_id TEXT,proof_hash TEXT,closed_at INTEGER NULL,created_at INTEGER, capabilities TEXT JSON,harness TEXT); events(seq INTEGER PK AUTOINCREMENT,project_id TEXT NULL,actor_id TEXT, kind TEXT,record_id TEXT,data_json TEXT,created_at INTEGER); mutation_receipts(principal_id TEXT,operation TEXT,key TEXT,fingerprint TEXT, result_json TEXT,created_at INTEGER,PRIMARY KEY(principal_id,operation,key)). Additional checks/indexes allowed. Secret hashing: SHA256 hex via shared auth::digest(&str)->String; random token auth::secret()->String. No agent API can mint full replacement tokens. Password hashing runs off async threads. Embedded SQLx migrations include 0001 and root’s later 0002.

Routes and payloads (all success data is inside envelope.data)

Public GET /healthz, /api/v1/info, /api/v1/help/authentication. GET /, /app.js, /style.css assets embedded by server. POST /api/v1/auth/login {username,password} -> {actor,csrf_token} + HttpOnly same-site cookie. GET /api/v1/me same response for browser restore. POST /api/v1/auth/logout {}. Cookie writes need X-CSRF-Token and expected Origin. Admin: GET /api/v1/admin/credentials -> {items:[{id,name,revoked_at,...}]}; POST /api/v1/admin/agents {name} -> {principal_id,credential_id,token}; POST /api/v1/admin/credentials/{id}/revoke {}. Agent session: POST /api/v1/sessions {session_id,workstation_id,harness, capabilities:[]} with X-Coordinator-Session-Proof (no session header yet); GET /api/v1/sessions/{id}; POST /api/v1/sessions/{id}/close {}. Existing session requests send X-Coordinator-Session and X-Coordinator-Session-Proof in addition to Authorization Bearer token.

Root implements: GET/POST /api/v1/projects (create {name,repository_url,target_branch}) -> project {id,name,repository_url,target_branch,policy_revision:1, review_mode:"agent",lease_seconds:600,recovery_mode:"agent"}; GET/POST /api/v1/projects/{id}/tasks (create {title,description, acceptance_criteria:[string],kind:"code"|"general",priority:0..3, depends_on:[task_id],planned:bool}; default normal=2) -> task {id,project_id,title,description,acceptance_criteria,kind,priority, lifecycle,revision,work_status,current_attempt_id,created_at}; GET /api/v1/projects/{id}/tasks/{task_id} -> task + attempts/checkpoints; GET /api/v1/projects/{id}/orientation -> current project, instruction_version “1”,policy_revision,required_sections:[“coordination-v1”], instructions text, candidates,active_attempts, instructions_complete:true; POST /api/v1/sessions/{id}/instruction-acknowledgments {project_id,policy_revision:1,instruction_version:"1",sections:["coordination-v1"]}; POST /api/v1/projects/{id}/claims {task_id?,expected_task_revision?,mode: "work"|"recovery",policy_revision:1,instruction_version:"1"}; absent task_id means next eligible; -> {claim: {task,attempt,lease_remaining_ms},...} or {claim:null,reasons:[...]}. attempt includes id,task_id,owner_id,session_id, generation,state,expires_at (RFC3339),last_heartbeat_at; POST /api/v1/projects/{id}/attempts/{aid}/renew {generation}; POST …/checkpoints {generation,summary,current_action,next_step,blockers:[]}; POST …/release {generation,summary,blocked:bool}. Checkpoints/release don’t imply completion. Further workflow endpoints remain visibly unavailable in this first slice; never provide an unrestricted status=done operation. GET /api/v1/projects/{id}/events -> {items:[...]}.

Additional implemented routes:

• PATCH /api/v1/projects/{id}/policy: full replacement of editable policy fields with expected_revision, review_mode, recovery_mode, lease_seconds, rules, agent_rule_editing, and automatic_integration. Delegated agents cannot change the last two permission grants. The review/integration execution workflows remain unavailable in this foundation.
• PATCH /api/v1/projects/{id}/tasks/{task_id}: expected_revision, title, description, acceptance_criteria, priority, depends_on, and planned. Only an unowned open/planned task can be edited. There is no generic status edit.
• POST .../tasks/{task_id}/unblock: human-only {expected_revision,reason}.
• GET .../attempts/{aid}: attempt, task, current authority_valid, current lease_remaining_ms, and optional registered checkout metadata.
• POST .../attempts/{aid}/checkout: {generation,workstation_id,identity,path, branch,base_revision,clean}. Registers a client-reported separate clean checkout; the service does not execute Git or independently inspect the workstation.
• POST .../attempts/{aid}/recovery-resolution: {generation,disposition: "resume"|"restart",summary,saved_work_checked:true,running_jobs_checked:true}. Requires a recovery-mode attempt; only after this inspection may it resume normal work or release back to the queue. Incomplete inspections release blocked.

On a claim replay, current_authority reports whether the historical grant still holds. The original claim is a receipt, not a fresh countdown. Renewal replay returns current remaining time and requires a still-valid owned attempt. Checkpoint, checkout, and recovery-resolution replays also require current ownership. Release is the intentional exception: its historical receipt remains replayable after release, because that operation ends ownership. Every replay reauthenticates. All of these observations can become stale; ownership-dependent mutations still enforce the current authority transactionally.

All lists {items:[],next_cursor:null|opaque}. Errors use stable error envelope. The CLI must persist idempotency keys/requests before mutations, never retry an uncertain mutation under a new key. New automatic claims are never connect’s side effect. Missing credentials can call public help. All routes not listed as implemented are future milestones, not advertised working features.

Proposed coordination contract

Status: historical implementation design supporting the plan. The coordination service now implements the core ownership and recovery rules, but current routes, fields, and limits are recorded in implementation status and the operational guides. This chapter preserves the correctness rationale and tunable engineering defaults.

Revised after the repository/hook review. The operator has confirmed separate worktrees per implementation task and one integration step at a time into the target branch for the first release, and simultaneous support for multiple projects in one service instance.

Agent-driven recovery after checking saved work and running jobs is confirmed, with a per-project manual alternative. Code-task completion requires applicable review, target-branch integration, and validation of the integrated result.

Project boundaries

Project identity is explicit in claims, sessions’ work context, and every project-owned record. All authenticated people and agents can access all projects, as confirmed by the operator. There are no project-specific grants. An operation checks authentication, applicable operation/ownership permissions, and the project membership of referenced records. Database constraints and service checks reject mismatched task/attempt/evidence relationships. Search, events, imports, exports, artifacts, and aggregate views follow the same rules.

An agent working on project A cannot change its context accidentally because another agent selected project B; there is no global active-project setting. The same principal can deliberately select any project. After authentication, the dashboard can aggregate all projects. Project context prevents accidental mixing of work; it is not a visibility restriction between authenticated users.

Claims and jobs in different projects can be active simultaneously. Integration and resource reservations block only their canonical resource scope. Projects sharing the same repository target or physical resource must resolve it to a common reservation identity. Cross-project knowledge retrieval preserves source provenance and is available to every authenticated caller. Access to projects does not reveal credential secrets or confer ownership of another agent’s attempt.

Separate work, attempts, and authority

A task describes an intended outcome and its acceptance criteria. It belongs to a stable project ID; a project display name or local checkout path is not sufficient identity. Repository aliases can help discover a configured project, but an ambiguous match must not silently select one.

An attempt records one agent session’s effort on that task: owner, workstation label, start time, checkpoints, branch/commit evidence, and eventual outcome. An expired attempt remains part of history when another attempt takes over.

Tasks can be linked to a parent objective and prerequisites. Review and integration may be separate typed tasks with their own attempts; independent reviewers do not borrow an implementer’s ownership grant. Dependency edges must be acyclic and cannot change in a way that silently makes active work eligible under a different set of prerequisites. A parent is complete only after its required children and its own acceptance criteria are satisfied.

A lease grants temporary authority over an attempt. Its deadline is set by the service. An opaque attempt ID and an increasing ownership generation identify the grant; neither substitutes for authentication. Session authorization must prevent two sessions using the same workstation credential from silently acting as one another. The credential/session mechanism is specified in api-contract.md.

At most one current attempt can own a task. Task state and current-attempt linkage must agree within the same transaction. Generic task edits cannot bypass this rule or overwrite an active attempt’s ownership.

Claim and update rules

1. A candidate list is informational and may become stale immediately.
2. A claim transaction checks authentication, authorization, task eligibility, dependency state, and any configured concurrency restrictions. It creates one attempt, advances the ownership generation, and records the lease.
3. A competing claim returns a conflict or chooses another eligible task. It never receives authority over the existing attempt.
4. Every ownership-dependent mutation verifies the current attempt, generation, authenticated session, allowed transition, and unexpired lease together with the mutation. Server time must be sampled after acquiring the write transaction, so lock-wait time cannot make an expired lease appear valid.
5. A lease is valid only while protected coordinator time sampled under the writer lock is less than expires_at. Equality is expired. The response’s observational wall-clock server_time is not an authority clock. Renewal is allowed only before expiry. Retrying a renewal cannot resurrect an expired attempt.
6. Terminal attempts reject fresh mutations. The system preserves their results and checkpoints as history. A late agent may need a separate recovery-note operation, if selected for the release; such a note cannot complete current work, extend a lease, or unblock dependent tasks.
7. Submission saves the result, handoff, referenced/new lessons, attempt outcome, and chosen task transition in one transaction. An implementation submission advances to required review or integration; it cannot mark the code task done. Overall completion requires the configured review, integration, and validation of the integrated result. Non-code tasks use their applicable acceptance requirements.

Task revisions should protect editable descriptions and acceptance criteria against lost updates independently of attempt generations. If acceptance criteria change during work, record which revision the attempt used and require explicit reconciliation before accepting completion against changed criteria.

Worktree and integration contract

An implementation task records its own worktree, branch, base revision, and workstation identity before code changes. The local path is descriptive metadata; another workstation cannot assume it can read that path. Registration alone does not prove isolation: the client must resolve the checkout/worktree identity and reject accidental reuse by another active implementation task. An existing dirty checkout is never silently reset, stashed, or swept into a new task.

Integration is a separately owned operation. Acquire an exclusive resource keyed by stable project/repository binding and target branch. Check current candidate revision, target revision, required review, and action authorization; record the base and result revision and validation of the integrated result. A moved target requires reconciliation and validation of the new result. A branch-only result cannot satisfy an integration dependency merely by being labeled “done”.

Keep the integration reservation until its outcome is known. If the coordinator cannot tell whether an interrupted publish succeeded, report an uncertain result and reconcile the remote revision before permitting another conflicting action. The coordinator cannot fence arbitrary Git commands; actual branch updates still need the Git-side concurrency check and compliant local workflow. Remote hosting APIs are optional, so the core stores who reported/verified each revision and how, rather than presenting client-reported merges as independently verified facts.

Avoid circular completion requirements

A submitted implementation and a completed deliverable are separate facts. If overall completion includes review and integration, those activities must become eligible from the submitted candidate and applicable review evidence. They cannot wait for the implementation task’s overall done state: that task is itself waiting for their results.

Ordinary task dependencies mean the prerequisite’s required outcome is complete. Workflow activities instead reference an immutable submission ID and candidate revision. A review accepts or rejects that submission; integration consumes its accepted evidence. These are explicit workflow relationships, not an exception that silently treats unfinished prerequisites as done.

If the candidate changes, create a new submission. Previous reviews and checks remain historical and do not automatically apply to it. The readiness check and integration transaction must agree on the current submission, required policy revision, candidate revision, and observed target revision.

Non-code tasks can finish against their own acceptance evidence without a Git integration step. A parent objective waits for required children and its own criteria; it does not create artificial worktrees or branches. The final state names must preserve the selected distinction between submission and completion.

Review policy and independence

Projects can require agent review, human review, both, or disable required review. The selected default is independent agent review. Review requirements are pinned to a versioned policy and an immutable submission. A review records the actor, session when applicable, candidate revision, findings, decision, and check evidence inspected. The service rejects a review of an obsolete submission as an approval of the current one.

Proposed independence rule: an agent reviewer uses a separate review session and attempt, with no recorded contribution to the candidate under review. It may use the same model vendor or workstation; neither determines independence. A separately configured reviewer principal can provide additional separation, but rotating a token alone does not make the implementer an independent reviewer. Retain contributor attribution across recovery attempts and resubmissions. The protocol enforces recorded separation; it cannot prove a model’s reasoning was independent or detect falsely declared agent-session lineage.

Human approval must be recorded by an authenticated human principal. An agent can request it and attach context, but cannot submit a human decision on that person’s behalf. For a project requiring both types, each required approval must apply to the same current submission. A missing reviewer leaves the work waiting with a clear reason; it does not silently weaken the policy.

Review findings have stable IDs and severity, a concrete remedy, and resolution evidence. Rejection returns the implementation to a revision-needed state without deleting prior submissions. Required unresolved findings prevent integration; fixing the code creates a new submission and invalidates old approval applicability. The exact number of required reviewers and escalation thresholds are configuration defaults to specify before implementation.

External jobs and evidence

A test, build, or deployment run is a job, separate from an agent attempt. Register a unique job ID and producer identity before launch. Keep the runner instance/host, owning attempt, checkout/snapshot, permitted operation, invocation description, timestamps, latest observation, and terminal producer result. Raw environment values, credentials, and arbitrary command output are not required metadata and must not be collected automatically.

Producer status and observation freshness are separate fields. A previously reported running job with an unreachable observer is now unobserved; it is not proved stopped. The observer may reconnect to the same job with the necessary authorization. An observer’s permission to attach observations is not permission to renew a superseded task lease, write code, or complete the current task.

A terminal result identifies the producer’s exit status, not a wrapper/waiter’s exit status or a parsed success banner. Evidence specifies:

• The exact commit/tree, or an explicit dirty-snapshot digest and changed paths.
• Gate/check definition revision, relevant configuration/toolchain fingerprint, environment identity, start/end, and whether inputs remained stable.
• The required check roster and actual per-check outcomes: pass, fail, skipped, or interrupted, with reasons and result/log references.
• Who produced and, when required, who independently reviewed the evidence.
• Whether artifacts are accessible to other authorized workstations, retained only locally, or unavailable; optional digests permit identity checks.

A later mutation invalidates evidence applicability to the new snapshot. A snapshot-to-commit equivalence assertion requires an actual matching tree/content check. Historical evidence remains immutable. Missing coverage and unknown or interrupted runs cannot be transformed into a pass by a successful wrapper.

Scoped resources and disconnected producers

Task ownership prevents two owners of one task. Resource admission prevents different tasks from concurrently using an incompatible shared resource.

Resources have a stable scope and key: workstation-local build capacity, checkout-local mutable state, or a project/environment resource such as a shared fixture database or integration branch. Different isolated worktrees need not lock the same source file globally, but their shared fixture or build environment may still conflict. Each configured reservation is exclusive or consumes units from a declared capacity. Acquire a required set atomically, or none of it; return blockers rather than retaining a partial set while waiting for the rest.

Resource validity is checked transactionally with operations that depend on it. Dynamic capacity readings are dated observations, not a promise that other processes will not consume memory or disk. Local locks and environment-specific guards still protect against processes outside the coordinator.

Lease expiry does not prove the resource is physically free. Where an old job can still interfere, a reservation enters a recovery hold until a trusted local observation, fencing mechanism, or authorized operator action resolves it. The hold blocks the conflicting resource, not unrelated work. The final policy must define who can clear it and what evidence is needed, rather than treating every expired reservation as safe to reuse.

Interrupted requests and retries

Retryable mutations carry an idempotency key scoped to the authenticated caller and operation. Store a fingerprint of the request and the operation’s result in the same transaction as the mutation.

• An identical retry refers to the original operation and attempt; it does not create another task, claim another task, or extend the lease a second time.
• Reusing a key with different request data produces a clear conflict.
• Authentication and current authorization still apply when replaying a result.
• An old successful claim receipt is historical evidence, not proof of current authority. The client checks current ownership and the deadline before acting.
• A retry of completed submission can acknowledge the original successful submission without reopening its attempt or changing current task state.

The final API must define receipt retention and behavior for retries outside the supported window. Clients must not silently generate a new key after an uncertain claim response; they must resolve the original operation first.

Health, progress, and recovery

Track these separately:

A background helper is optional and needs an explicit lifecycle contract. It must not renew forever after its associated agent session has ended. Long tool calls, agent thinking time, workstation sleep, and network loss must be considered when choosing heartbeat intervals and expiry. The service can expose stale progress separately from an expired lease without assuming that silence proves a crash.

Proposed defaults are a one-minute heartbeat and ten-minute task lease. A local renewal helper receives a separate, narrowly scoped reporter credential; it can renew the named attempt and report authorized jobs, but cannot create checkpoints, submit work, change policy, or extend its own reporting window. The active agent explicitly declares the action being watched and its reporting-window deadline. Propose a one-hour maximum window initially, adjustable by the operator for long gates. Renewal cannot extend beyond that deadline or a revoked session/attempt.

The helper stops renewing when its observed harness instance exits, when authority is lost, or when the declared window ends. A job continuing beyond the window retains its own job record and resource holds; it does not silently keep the agent’s task owned. The agent can explicitly establish a fresh window while its authority is still valid. Manual HTTP clients can renew directly without a helper. This supports long tool calls while bounding an orphaned helper’s effect.

Display stale progress separately, with a proposed 15-minute warning threshold. A declared long job shows its expected deadline and observation freshness, so an expected wait is distinguishable from unexplained silence. Warnings do not automatically mark failure, restart jobs, or release resources.

Adapter events distinguish a turn ending, context compaction, session disconnection, explicit relinquishment, and task completion. Repeated connect or resume events reconcile existing session/attempt/job IDs first. They cannot implicitly claim additional work. A compact/resume hook must not fast-forward a checkout underneath an active mutation or check of that checkout.

Expiration makes a task a recovery candidate, rather than immediately ready for fresh implementation. Under the selected default, another agent may atomically claim recovery authority. A project may instead require operator release before that recovery claim. Both modes retain the expired attempt and its checkpoints.

The recovery packet includes prior submissions, worktree/branch and published revision references, known jobs, observation freshness, and resource holds. The recovering agent checks what has already been delivered, which saved work is accessible, and whether prior jobs can still interfere. It records a disposition: resume saved work, reconcile already-delivered work, restart with a reason, or remain blocked awaiting an observation or operator decision.

Recovery inspection is distinct from permission to start conflicting work. An unreachable workstation or a missing heartbeat does not prove its jobs stopped. An unresolved external job keeps its conflicting resource reserved. A recovery attempt may continue only when the selected action’s resource and evidence requirements are satisfied. Expiry of a recovery attempt permits another recovery claim; it does not clear those holds. Deadlines and ownership generations prevent two recovery agents from both recording an authoritative disposition.

Disconnected clients must not start new coordinated work. While a previously granted lease is still valid, behavior depends on the final offline-work policy. After authority expires or is lost, instructions must direct clients to stop ownership-dependent changes and reconcile with the service. Client deadlines should be conservative and account for request delay; simply comparing server timestamps with an unsynchronized workstation clock is insufficient.

This is a cooperative protocol. The service can reject stale database writes; it cannot stop local computation or retroactively prevent a Git push by an old agent. A policy for publishing, merging, and external side effects is necessary before promising stronger end-to-end guarantees.

Scenarios the implementation must demonstrate

Shared lessons and delegated project rules

Agents may publish and correct lessons, including lessons useful across projects. The selected policy also allows a project to delegate binding-rule changes to agents without human approval. Store that permission separately from policy text; check it before accepting each revision. A rule revision is an explicit, attributed operation, not a side effect of importing a lesson or rendering Markdown.

Knowledge records contain a concise statement, explanation, applicability tags and versions, source task/submission/revision, supporting evidence, author, and created/updated times. Corrections and supersession preserve earlier versions and their links. A disputed lesson stays inspectable with its dispute visible; it does not silently become mandatory guidance.

Provide project-scoped search and explicit cross-project/common retrieval to all authenticated callers. A common lesson preserves its original source rather than duplicating detached copies. Rank current applicable records using full-text match, component/version tags, and bounded usefulness feedback. Feedback changes ranking, never permission or policy status. No embeddings service or model API is required for the first release.

Orientation separates mandatory current rules from suggested lessons. Required rule updates are versioned and visible to active agents. Local AGENTS.md/CLAUDE.md instructions still apply; a service rule cannot silently erase a local rule or resolve a conflict by pretending the local instruction was never present.

Artifact storage and portable recovery

Store bounded log/report uploads in the service, alongside external artifact links. Code checkpoints use Git remotes. Record source accessibility honestly: local-only uncommitted edits cannot be recovered from a different workstation merely because their path appears in a checkpoint. Source-worktree bundles and build-directory uploads are outside the selected first-release scope.

Artifact metadata includes ID, project/task/job association, original display name, media type, byte size, SHA-256 digest, author, retention, and availability. Use generated storage paths, never client filenames as filesystem paths. Stage and size-check uploads, finish and durably store the file, then commit accessible metadata. Incomplete uploads are not downloadable; orphan-file cleanup and backup/restore reconcile filesystem state with database references.

Enforce per-file and aggregate storage quotas before and during uploads. Retain metadata and a deletion/expiry reason after removing file bytes, so historical evidence does not appear to have an accessible artifact when it no longer does. Uploads are explicit; the reporter does not automatically collect credentials, environment variables, source trees, or every raw log from a workstation.

Downloads require authentication. Treat uploaded files as untrusted attachments, not executable same-origin web pages; any inline preview must render safe text. The server stores external links without fetching arbitrary URLs. Cross-machine access to a linked resource is reported rather than assumed from its existence.

Duplicate work beyond a shared task ID

Atomic claims address two agents choosing the same recorded task. They do not address agents creating two different tasks for the same requested change, or different tasks that touch conflicting code.

Proposed controls, strengthened by the repository review:

• Stable import/source IDs and uniqueness constraints for repeated imports; numeric item labels alone are insufficient across sessions/documents.
• Search-before-create guidance and explicit duplicate/related-task links.
• An after-claim check that acceptance criteria are not already satisfied on the relevant repository revision, with an evidence-based reconciliation outcome.
• Durable closure, rejection, and supersession records consulted by selection and import, including reopening conditions for intentionally deferred work.
• Dependencies and optional component/resource reservations for known conflicts.
• The confirmed separate-worktree and serialized-integration rules above.

Natural-language similarity can suggest duplicates, but cannot establish task identity reliably. Semantic search or model-based deduplication is an optional feature, not a prerequisite for atomic ownership.

Job/worktree milestone integration contract

Status: implemented milestone contract for backlog item 1. At this historical slice boundary it did not enable task completion, review, integration, or artifact uploads; those later service features are listed in implementation status. The service still does not perform automatic remote execution.

HTTP interface (successes use the existing envelope)

GET/POST /api/v1/resources: globally shared canonical key, capacity (1–1000), description; creation human-only, uniqueness by key, immutable initial capacity. GET/POST /api/v1/projects/{p}/attempts/{a}/reservations: create {generation,items:[{resource_id,units}]} atomically, one active reservation per attempt; same resource identity blocks across projects. All-or-none admission. POST /api/v1/projects/{p}/reservations/{r}/release {generation,reason}: The original owner, or the current recovery owner of the same task using its new generation, may release only when attached jobs have terminal producer results. POST .../reservations/{r}/resolve {reason,evidence}: explicit human resolution of an uncertain physical resource, retaining provenance and affected job history. GET /api/v1/projects/{p}/reservations: bounded cursor list with derived held / recovery-required status. Never free holds on observer/lease/session loss.

POST /api/v1/projects/{p}/attempts/{a}/jobs: {generation,job_id,producer_id,runner_instance_id,workstation_id,label, source_revision,source_tree,reservation_id,reporter_id,reporter_proof, renew_for_seconds}. Job, producer, runner, reservation and reporter IDs are UUIDs; workstation_id matches the stable name from session registration. reporter_proof is client-generated random 32 bytes encoded hex, persisted before registration. Store only its verifier; do not store proof in receipt/event. renew_for_seconds 0–3600 (0 disables delegated renewal); observation authorization lasts seven days, independently of the task lease. Requires an active owned work-mode attempt, registered clean checkout, matching workstation, and held reservation belonging to the attempt. Result {job,reporter:{id,expires_at,renew_until},renew_after_seconds} never returns the proof. Job IDs and producer IDs are unique; repeat IDs never mean relaunch. GET /api/v1/projects/{p}/jobs[?cursor=...] and /jobs/{j} expose producer state, last observation/freshness, source identity, reservation, and terminal result.

Reporter bearer namespace: acr_<reporter UUID>.<proof>; no agent token or session headers in a guardian. Only these paths accept that bearer: GET /api/v1/reporters/{id} returns its named job and a reporter object with current renewal/observation authority, launch_allowed, and lease_remaining_ms (not its proof). Launch requires a current live work attempt, active parent session and credential, registered producer, and held reservation. Guardians query this immediately before spawning and subtract request elapsed time plus a safety margin. A registration receipt does not provide fresh launch authority. POST .../{id}/observations {sequence,producer_id,state,pid,process_started_at,exit_code,inputs_unchanged, summary} where state is registered|running|succeeded|failed|unknown|not_started. Sequence positive and strictly increasing; duplicate same sequence/body replays, changed same sequence conflicts, old observations never overwrite newer/terminal. PID is optional metadata, never sole identity. Terminal observations require exit_code for succeeded/failed; not_started requires an explicit local failure before launch; unknown is nonterminal. Observations may continue after attempt expiry or session closure while parent credential/principal and reporter remain authorized. Never automatically release a hold from an observation. POST .../{id}/renew {generation} caps each renewal extension at renew_until without shortening an existing lease, and requires current attempt/generation, active parent session, and live deadline. A reporter cannot create tasks/sessions, checkpoint, expand scope, or renew its own window. Normal bearer credentials cannot impersonate reporter requests.

The middleware calls jobs::ReporterAuth::authenticate(parts,state) for paths beginning /api/v1/reporters/, before decoding bodies. Its verify(connection,now) returns the parent Actor after checking subordinate/parent authorization. The mutation module provides Mutation::begin_reporter(state,auth,headers,operation,input) with the same immediate-transaction, receipt/event guarantees and reporter-bound fingerprint. The coordination module exposes coordination::{Attempt,owned} as pub(crate) for service handlers.

The service gates release/ordinary requeue/recovery resolution/checkout changes on unresolved job/resource evidence, using service helper jobs::ensure_attempt_quiescent(connection,project,task_id) (checks all attempts of this task for held reservations or nonterminal jobs; descriptive conflict). Blocked release remains possible and retains holds. Recovery never cancels jobs.

Native workstations and guardians

CLI prepares a new Git worktree with no automatic reset/stash/cleanup, records resolved Git-dir identity, branch and full base SHA, and preserves other checkouts. Persist a local preparation intent before Git worktree creation so retry reconciles the same path/branch rather than creating a second worktree. Paths with spaces work on Linux and Windows. Resolve repository identity against configured remote URL; do not infer a project from a directory basename.

CLI jobs run receives a local program/argv JSON file, an attempt/generation, reservation ID and prepared checkout. Require a clean committed source snapshot for this milestone. Raw argv/environment/log bytes are not uploaded. Persist job/producer/runner/reporter identities before registration or local launch. A guardian subprocess of the installed CLI gets only the scoped reporter token in a protected job state file. Strip coordinator agent credentials/session env from guardian and child. It records a durable launch-intent before spawning. If interrupted after intent, never spawn again; show uncertain/inspect-existing. An OS file lock serializes guardians, but losing that lock never proves producer termination. Record PID plus OS process start identity, distinct producer UUID, and terminal exit journal; PID reuse must never attach to a replacement process.

Jobs inspect/reconnect replays durable pending observations to the same reporter/job and never launches a producer. Capture terminal result and input stability even offline; log files are local, explicit, bounded, and protected. Unknown observation state retains resources.

Supported producer commands run their work in the foreground. The launched process’s exit is evidence about that process, not proof that a detached child, remote build, or hardware operation ended. A launcher that returns before such work finishes must not be used as a completion witness; keep the resource held and inspect/resolve the external work separately. Jobs do not provide OS sandboxing or physical fencing.

Optional renewal requires an explicitly supplied harness PID captured with its OS start identity. The guardian stops renewal when that exact harness exits, authorization expires, or renewal window ends, while job reporting may continue. No anonymous forever-renewing helper and no automatic agent/task launching.

Completion workflow contract

Status: implementation contract for backlog item 2. The service never runs Git or accepts a caller-authored assertion that a required check passed. Every mutation uses an idempotency key and rechecks authority after obtaining SQLite’s writer lock. Replaying a durable result does not renew a lease or refresh any authority field in the saved response.

Policy and immutable submissions

GET /api/v1/projects/{project}/workflow-policy returns {project_id, revision, canonical_repository_key, required_checks}. A required check is {identity, version, environment}; all three strings are stable exact-match identities. A human changes the roster with PUT /api/v1/projects/{project}/workflow-policy and {expected_revision, canonical_repository_key, required_checks}. The canonical repository key is an operator-managed global identity shared by projects that use different URL aliases for the same repository. The service cannot infer aliases; operators must use one key consistently. The target branch remains the project’s validated Git ref. The roster must contain 1–100 unique, nonempty checks. A canonical binding cannot change after submissions or while a hold exists. The workflow-policy revision is independent of the existing project policy revision. A code submission pins both. There is no implicit empty roster and an agent cannot create, weaken, or replace it.

POST /api/v1/projects/{project}/attempts/{attempt}/submissions accepts the following common fields:

{
  "generation": 1,
  "task_revision": 2,
  "project_policy_revision": 3,
  "workflow_policy_revision": 4,
  "kind": "code",
  "summary": "bounded result summary",
  "acceptance_evidence": [{"criterion": "exact criterion text", "evidence": "bounded evidence"}],
  "handoff": "bounded handoff",
  "repository": "canonical configured repository URL",
  "base_revision": "full source revision",
  "candidate_revision": "full candidate revision",
  "candidate_tree": "full candidate tree"
}

For kind: "general", omit repository/base/candidate fields and send a workflow policy revision of 0; project policy and task revision are still pinned. Every current acceptance criterion must appear exactly once. Submission requires the current unexpired owner, a matching kind/revision/policy, and quiescent subject work: no retained reservations and no nonterminal or uncertain jobs. It atomically ends the implementation attempt, stores the immutable submission, records the contributor session, blocks ordinary subject claims, and creates the applicable review and integration activities. The response is {submission, subject_task_id, activities, next_actions}.

A code submission creates one agent_review and/or human_review activity from the pinned review_mode, plus one integration activity. A general submission creates only its configured reviews; with no review it completes atomically. Activities have stable IDs and linked internal task IDs. They are returned from GET /api/v1/projects/{project}/tasks/{task}/workflow and GET /api/v1/projects/{project}/workflow-activities/{activity}. Internal activity tasks are excluded from ordinary candidate selection and must be claimed through the workflow API.

Activity authority and review

POST /api/v1/projects/{project}/workflow-activities/{activity}/claim accepts {expected_submission_id, expected_project_policy_revision, expected_workflow_policy_revision}. It returns {activity, attempt, lease_remaining_ms, renew_after_seconds, current_authority}. Review activity claims enforce actor type. Agent review also rejects every principal/session recorded as a contributor to any revision of the subject task; contributor history survives releases, recoveries, changes requested, and later submissions. Integration claims require all current approvals and, if automatic integration is disabled, a current human authorization. Integration claims atomically acquire the exclusive hold for the project’s canonical repository key plus target branch. The hold is global across projects and survives attempt/session/credential expiry.

Expired or revoked activity ownership respects the project’s recovery mode. In manual mode, agents cannot take over. A human inspects saved work and physical resources. Before publication intent exists, the human may reopen an expired, revoked, or policy-stale candidate; this fences its linked attempts and requires a new submission and new reviews. Live current owners must release first, and unresolved jobs/resources always block reopening. After publication intent, use publication reconciliation. Completed workflows cannot be reopened.

Activity attempts renew through the existing POST /api/v1/projects/{project}/attempts/{attempt}/renew endpoint and use existing checkout, reservation, and job endpoints. They release through POST /api/v1/projects/{project}/workflow-activities/{activity}/release with {generation, summary, blocked:false}; a release after publication intent is refused. Inspect and reconcile that publication instead; reconciliation creates a replacement activity and retains the historical result. Workflow state changes always require the activity ID as an additional candidate/type guard.

POST /api/v1/projects/{project}/workflow-activities/{activity}/review accepts:

{
  "generation": 1,
  "submission_id": "uuid",
  "decision": "approved",
  "summary": "bounded review summary",
  "findings": [{"severity": "required", "remedy": "bounded remedy", "evidence": "bounded evidence"}]
}

decision is approved or changes_requested; severity is required or advisory. The current, unexpired linked attempt and exact current submission are required. A decision is immutable. Changes requested completes that review activity, cancels the other pending activities, makes the subject eligible for a new revision attempt, and keeps all old submissions, approvals, findings, and contributors. A later submission creates fresh activities; old approvals do not transfer. The subject’s task and project policy revisions must still match the submission for approval to affect readiness.

Checks and integration

Required check receipts are selected by job ID; clients do not submit check outcomes. A selectable job must belong to the current integration activity attempt, have a terminal producer state of succeeded, exit code 0, inputs_unchanged: true, no reconciliation amendment, and exact source revision and tree matching the intended integrated result. Each job registered for a check also carries immutable check_identity, check_version, and check_environment fields that exactly match one pinned required-check entry. One current producer receipt must cover every roster entry, with no duplicate identity tuple. Candidate jobs cannot satisfy integrated-result checks merely because candidate and result happen to share a commit label.

When project policy disables automatic integration, a human records intent with POST /api/v1/projects/{project}/workflow-activities/{activity}/authorization and {submission_id, expected_project_policy_revision, expected_workflow_policy_revision, summary}. Authorization is immutable and applies only to that current candidate and policy pair.

After claiming integration and registering an isolated checkout, the owner calls POST /api/v1/projects/{project}/workflow-activities/{activity}/publication-intent:

{
  "generation": 1,
  "submission_id": "uuid",
  "observed_target_revision": "full target revision before publication",
  "observed_target_tree": "full target tree before publication",
  "result_revision": "full intended integrated revision",
  "result_tree": "full intended integrated tree"
}

The service saves one immutable intent before any Git-side compare-and-swap. The client must independently verify that the remote target still equals observed_target_revision before publishing. Repeating with a new key cannot replace an intent; retry uses the original key.

A fresh activity GET returns publication_allowed and qualifying_check_job_ids. Publishing is allowed only while the exact candidate, policies, approvals, authorization, activity lease, canonical hold, intent, and one successful stable-input exact-result producer per required roster entry are all current. Publication intent alone is not permission to publish. Once any integration result exists, publication is no longer allowed for that activity.

POST /api/v1/projects/{project}/workflow-activities/{activity}/integration-result accepts {generation, submission_id, publication_state, observed_target_revision, result_revision, result_tree, check_job_ids, summary} where publication state is published, not_published, or uncertain. Values must match the saved intent. An uncertain result may omit checks, is durable, makes the activity recovery-required, and retains the global hold. It never completes work or releases the hold. A known not_published result also retains the hold until explicit reconciliation.

A human reconciles with POST /api/v1/projects/{project}/workflow-activities/{activity}/publication-reconciliation and {submission_id, disposition, observed_target_revision, observed_target_tree, evidence}, where disposition is published, not_published, or target_moved. target_moved records that the intended result was published but a later external actor advanced the target before finalization; it closes the old activity and hold and creates a fresh integration activity for the same approved candidate. The new activity requires fresh authorization and checks against its new result. Reconciliation cannot invent check success. Confirmed publication must match the intent’s exact result revision/tree. Confirmed nonpublication permits an explicit new integration attempt after the old hold is closed in the same transaction.

For a known published result, call POST /api/v1/projects/{project}/workflow-activities/{activity}/finalize with {generation, submission_id, observed_target_revision, observed_target_tree}. These last fields are a fresh post-publication remote observation and must equal the intended result revision and tree. Finalization verifies the exact current candidate, task revision, project policy revision, workflow-policy roster, approvals, authorization, intent/result, exact remote observation, source-bound job receipts, activity authority, and quiescence of both subject and integration tasks. It then atomically completes the integration activity and subject task, closes the global hold, and makes dependencies eligible. There is no generic completion endpoint.

Coordination hooks

The coordination module must call these workflow hooks while holding its existing writer transaction:

• guard_normal_claim(connection, project_id, task_id) rejects internal activity tasks and subjects with a current submission; record_contributor(connection, task_id, actor_id, session_id, now) runs when normal or recovery work is claimed and again before owned edit/checkout.
• guard_subject_mutation(connection, project_id, task_id) rejects definition/dependency edits while a submission is current. Policy changes may proceed, but immediately make pinned submissions stale until an authorized new submission/reconciliation.
• guard_release_or_recovery(connection, project_id, task_id) preserves workflow integration holds and refuses recovery/requeue paths that would bypass a current submission or publication uncertainty.
• activity checkout/reservation/job registration calls guard_activity_work(connection, project_id, activity_task_id, now) to reject stale candidates and policies while still allowing checkpoint, renewal, release, and late reporter observations.
• task detail/status reads call workflow_snapshot(connection, project_id, task_id, now) and expose its submission, activities, blockers, and next actions. Ordinary ready lists exclude task IDs present as workflow_activities.activity_task_id.

The jobs module must accept optional check identity/version/environment at job registration and store them immutably. It must expose a server-side receipt lookup; workflow completion never accepts caller-authored status JSON.

Normal attempts pin attempts.task_revision and attempts.policy_revision when claimed. Submission compares those stored values to the current task and project and to the request. Existing pre-migration attempts have null pins and must be released and reclaimed; callers cannot choose a newer revision at submission.

An authenticated human may cancel a stale current submission with POST /api/v1/projects/{project}/tasks/{task}/workflow/reopen and {submission_id, reason}. This requires subject and activity quiescence, refuses uncertain or known publication, cancels pending activities, preserves all evidence, closes any safely releasable hold, and makes ordinary revision work eligible.

Knowledge and decision contract

This contract defines the implemented revisioned knowledge, bounded context, binding-rule history, and scoped decision service. Knowledge is advisory evidence. Project rules remain the binding policy, and decision answers remain narrow workflow gates. Neither record changes credentials, roles, permissions, review requirements, integration authority, or any other authorization.

Knowledge records

Each knowledge identity belongs to its source project and has an immutable sequence of revisions. The current projection is searchable, while older revisions remain available through cursor pagination. Supported kinds are lesson, fact, rejected_approach, and checkpoint. Supported statuses are observed, validated, deprecated, and superseded. A correction always creates the next revision; it never overwrites text, attribution, scope, provenance, or status in an earlier revision.

The request fields are:

A shared record keeps its source project and provenance. It appears in another project’s list or context only when that caller sends include_shared=true. Shared collection membership is not an access grant: every authenticated principal can already read all projects. It is an explicit relevance choice.

PATCH /api/v1/projects/{project}/knowledge/{id} requires expected_revision and the complete new title, body, status, scope, tags, applicability, and provenance. The kind and collection are stable identity fields. status=superseded requires superseded_by_id; the referenced replacement remains a separate record with its own history. Deprecated, superseded, and observed results are returned with their status and are never presented as current binding policy.

Usefulness feedback requires expected_revision, is append-only, and is bound to the exact knowledge revision the caller read. A concurrent correction produces revision_conflict instead of attributing stale feedback to the new revision. A later correction does not rewrite earlier feedback. The detail response reports bounded revision history plus useful/not-useful totals.

The routes are:

• GET|POST /api/v1/projects/{project}/knowledge
• GET|PATCH /api/v1/projects/{project}/knowledge/{id}
• POST /api/v1/projects/{project}/knowledge/{id}/feedback

Lists use a stable opaque updated_at:id cursor and a limit from 1 through 200. Knowledge history uses a descending revision cursor and the same limit range.

Submission lessons

A submission may include up to 50 SubmissionLessonInput values. The server validates the complete batch, all source-project task references, and the explicit shared opt-in before inserting any record. It then inserts every lesson and its revision 1 inside the submission’s existing SQLite writer transaction. Provenance is pinned to the exact project, task, submission, actor, and service time. A validation error rolls back the submission and all linked lessons together.

The workflow integration calls:

insert_submission_lessons(
    conn,
    actor,
    now,
    project_id,
    task_id,
    submission_id,
    lessons,
)

The returned values are the same flat knowledge objects returned by the create route.

Binding-rule provenance

The existing projects.rules and projects.policy_revision projection remains the only binding project policy. Every API policy change keeps the existing authority rule: a human may change policy, while an agent may change rules only when agent_rule_editing is already enabled and may not alter that delegation or integration permission.

Each policy_revisions row stores the complete projection, actor, service time, and an optional bounded provenance explanation. Database triggers reject updates and deletes of those historical rows. GET /api/v1/projects/{project}/policy/history?cursor=&limit= returns descending immutable revisions. Empty provenance is retained for revisions created before provenance was supplied.

No knowledge import, knowledge correction, feedback, or decision answer promotes text into projects.rules. Promotion is an explicit policy PATCH under the current policy revision and existing delegated authority.

Bounded context search

GET /api/v1/projects/{project}/context accepts:

• required q, limited to 1,024 bytes and 32 search terms;
• limit from 1 through 100;
• budget from 1,024 through 131,072 bytes;
• include_shared=true for explicitly shared cross-project knowledge;
• optional task_id, component, environment, and version knowledge-scope filters.

SQLite FTS5 searches the current task title, description, and acceptance criteria and the current knowledge title, body, tags, and applicability. Results are ordered by FTS rank with a stable identity tie-break. Pending or invalidated decisions for the requested task_id, or all project decisions when it is omitted, are placed before text matches. An empty scope dimension means generally applicable; a populated dimension must match the corresponding requested filter. Items are included only as complete JSON objects, stopping at the item count or byte budget. Empty and truncated results include a concrete next_actions value.

The complete current projects.rules value is always returned separately as policy. It is never truncated. instructions_complete=false and truncated=true indicate that the requested byte budget was smaller than the mandatory policy object; in that case no optional context items are returned. Callers must increase the budget before treating the packet as a complete orientation.

Decisions

A decision is a structured question, 2–20 exact answer options, rationale, required actor (human, agent, or either), and one or more affected task IDs pinned to exact task revisions. Each generation also pins the current project policy revision, environment, conditions, optional expiry, reopening rationale, actor, and service time.

An answer requires:

{
  "expected_generation": 2,
  "disposition": "allow",
  "answer": "Proceed",
  "rationale": "The recorded rollback owner is present.",
  "conditions_confirmed": true
}

answer must exactly match one listed option. disposition is allow, deny, or defer. An allow requires conditions_confirmed=true; deny and defer are durable answers but keep affected work blocked. Actor type is checked after the SQLite writer lock is obtained, and an agent answer is attributed to a live agent session. Human-required answers require a human principal.

A prior allow stops blocking only while all of the following remain true:

• every affected task has the exact pinned revision;
• the project has the exact pinned policy revision;
• the allow explicitly confirmed the recorded conditions and environment;
• the decision has not expired.

If any pin changes or the allow expires, every affected task remains blocked until the decision is reopened against all current revisions and answered again. This prevents an answer for one task in a multi-task scope from silently carrying forward after another affected task changes. Reopening creates a new immutable generation and cannot change the question, options, or required actor, so it cannot weaken a human requirement. Older generations and answers remain inspectable.

Opening or reopening a decision is rejected while any affected subject or activity holds integration publication authority. This avoids issuing a new blocker after a publisher has already received exclusive authority for an external compare-and-swap operation.

The routes are:

• GET|POST /api/v1/projects/{project}/decisions
• GET /api/v1/projects/{project}/decisions/{id}
• POST /api/v1/projects/{project}/decisions/{id}/answer
• POST /api/v1/projects/{project}/decisions/{id}/reopen

Decision lists use stable created_at:id cursors. Detail history uses descending generation cursors. Current status is pending, allowed, denied, deferred, expired, or stale; work_allowed is true only for allowed.

Workflow gate

After obtaining SQLite’s writer lock and rechecking authentication, time, generation, ownership, and policy, ownership-dependent operations call:

ensure_decisions_resolved(conn, project_id, task_id, now)

The guard returns decision_required with at most 20 stable decision IDs and a truncation flag. It applies to ordinary work selection/claims, active ownership-dependent mutations, submission, workflow claims and workflow operations for both the subject and activity task, publication preparation/result/finalization, and completion. Renewal, checkpoint, release, and recovery inspection remain available so an owner can preserve or relinquish work safely. A recovery claim may inspect stale work, but ordinary work cannot resume until the decision gate passes.

Task readiness uses the same predicate as the transaction guard. Lists and orientation therefore report a decision-blocked task instead of advertising it as ready. Receipt replay never renews ownership or revives an expired decision allow.

All knowledge, feedback, decision, answer, and reopen mutations use the standard idempotency receipt and event transaction. The service stores no credentials or request headers in these records or events.

Artifact contract

This release stores bounded reports and logs as authenticated artifacts and also records external HTTPS links. Source checkpoints continue to travel through Git remotes. The service never fetches an external artifact URL and never treats a client filename as a filesystem path.

Limits and storage

• An uploaded artifact is at most 16 MiB. Request bodies are streamed to disk and SHA-256 checked; they are not encoded as JSON or buffered in memory.
• Live and reserved uploads share a configurable service-wide quota (10 GiB by default). A reservation counts until it expires, is deleted, or becomes finalized and later expires.
• The service preserves configurable free disk space (256 MiB by default). It checks available space at reservation, before upload, and while writing each chunk. Four uploads may run concurrently. Waiting for an upload slot is bounded to five seconds and an upload must complete within two minutes.
• The store is a directory named after and next to its database, for example coordinator.sqlite3.artifacts. Each database therefore has an isolated store. Blob, staging, and lock names use server-generated UUIDs. On Unix, directories are mode 0700 and staged files are mode 0600.
• Upload reservations last one hour. Retention defaults to 90 days and can be set from 1 to 3650 days or pinned with no expiry. Expiry is evaluated using the service clock. Metadata remains after expiry or deletion.

The service stages a body under its generated key, enforces its exact reserved size while receiving it, hashes it incrementally, calls fsync, and atomically renames it into the blob store. Only then does it take the SQLite writer lock, recheck the current credential/session, reservation author, expiry, size, and digest, and commit finalized metadata, its mutation receipt, and its event in one transaction. No transaction is held while receiving a body or doing filesystem work. A crash after rename and before commit leaves a recoverable orphan bound to the same reservation; retrying the same saved bytes and idempotency key verifies that blob and finalizes it without replacement or duplication.

Each cleanup pass is bounded to 100 database records, 100 files from one rotating blob prefix, and 100 staging entries. Before removing a known expired/deleted blob, cleanup obtains the artifact lock and rechecks its current database state. Unknown blob orphans must be older than five minutes, then receive the same lock and a fresh absence check. This prevents cleanup racing upload finalization or a retention change. Live reservations remain intact, including a durable blob awaiting a retried database finalization. Run reconciliation after opening the state at service startup; reservation creation also invokes a bounded pass.

HTTP API

Every route requires normal human or agent authentication. Every mutation also requires a persisted Idempotency-Key. All JSON success bodies use the normal data envelope.

POST /api/v1/projects/{project}/artifacts records an external link:

{
  "display_name": "CI report",
  "media_type": "text/html",
  "external_url": "https://artifacts.example/report/123",
  "task_id": null,
  "job_id": null,
  "size_bytes": null,
  "sha256": null,
  "retention_days": 90,
  "pinned": false
}

The URL must be HTTPS, have a host, and contain no embedded credentials. The service records it without making a network request. Optional task and job IDs must belong to the same project; when both are supplied, the job must belong to that task.

POST /api/v1/projects/{project}/artifacts/uploads reserves an upload:

{
  "filename": "check-report.txt",
  "media_type": "text/plain",
  "size_bytes": 1234,
  "sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "task_id": null,
  "job_id": null,
  "retention_days": 90,
  "pinned": false
}

The response contains artifact metadata and an authenticated upload_path. Send the exact raw bytes to that path with PUT, the same normal authentication, and a separately persisted idempotency key. Content-Length, when supplied, must match the reservation. The reservation author is the only principal allowed to upload its content. A retry must reuse its saved bytes, digest, and key.

GET /api/v1/projects/{project}/artifacts returns a bounded cursor page plus authenticated service-wide upload usage, quota, disk capacity/reserve, maximum artifact size, and default retention diagnostics in data.storage. limit defaults to 50 and is capped at 200. GET /api/v1/projects/{project}/artifacts/{artifact} returns one metadata record.

GET /api/v1/projects/{project}/artifacts/{artifact}/content streams finalized, live upload bytes. Downloads always use Content-Disposition: attachment, a generated safe ASCII filename, the recorded content type, Content-Length, and X-Content-Type-Options: nosniff. External-link content is not proxied.

POST /api/v1/projects/{project}/artifacts/{artifact}/retention accepts:

{"pinned": false, "retention_days": 180}

When pinned is true, omit retention_days. The author or a human operator may change retention. POST /api/v1/projects/{project}/artifacts/{artifact}/delete accepts a nonempty reason; the author or a human operator may create the deletion tombstone. The database change commits before physical cleanup. If cleanup is interrupted, retrying the same key or startup reconciliation removes the inaccessible bytes.

Metadata reports database state as reserved, finalized, or deleted, plus a current availability value:

Task submission integration must call validate_submission_artifacts(connection, project, ids, now) under its writer transaction before inserting the submission, then call link_submission_artifacts(connection, project, submission, ids, now) in that same transaction. References are unique, limited to 100, same-project, finalized, and live. submission_artifacts is immutable; later expiry or deletion changes availability without erasing the historical reference.

Native transfers and configuration

After reserving an upload, use artifacts upload --id ID --file PATH with the same local harness session. The CLI snapshots exact bytes into its protected journal before sending them. Retrying that command reuses the snapshot, digest, and idempotency key, even if the original file changed. Completed upload retries reauthenticate and return current artifact metadata, including expiry, deletion, or unavailability; they never upload again. Use artifacts download --id ID --output PATH to save bounded bytes into a new file. Its size and SHA-256 must match authenticated metadata before the new file is published. An existing destination is never overwritten.

Server settings COORDINATOR_ARTIFACT_QUOTA_BYTES and COORDINATOR_ARTIFACT_DISK_RESERVE_BYTES configure admission limits; the latter must retain at least 16 MiB. COORDINATOR_JSON_BODY_LIMIT_BYTES defaults to 1 MiB and can be lowered to 1024 bytes. Binary streaming retains its separate 16 MiB hard limit. Startup reconciles artifact metadata and storage before accepting requests, preserving explicit unavailable/tombstoned evidence.

Markdown import and export contract

The service owns current task, knowledge, decision, policy, submission, and handoff state. Markdown is bounded source evidence. Supplying a path never asks the service to read that path, inspect a repository, run a hook, or execute imported text. Clients read files locally and send explicit chunks with provenance.

Preview

POST /api/v1/projects/{project}/imports/preview accepts:

{
  "source": {
    "context": "stable operator-defined repository/source identity",
    "git_revision": "full 40- or 64-character commit ID",
    "observed_at": "2026-09-09T20:00:00-04:00",
    "branch": "development",
    "environment": "audited workstation"
  },
  "chunks": [{"path": "BACKLOG.md", "markdown": "# Tasks\n- [ ] Example"}],
  "historical_mappings": [{
    "path": "HANDOFF.md",
    "section_identity": "Session / Closed work",
    "title": "Earlier implementation",
    "disposition": "closed",
    "evidence": "Exact bounded historical evidence"
  }]
}

The request contains 1–32 chunks, at most 200 KiB of Markdown, and at most 200 parsed checklist items and historical mappings. Paths are normalized relative metadata with no parent traversal. Source context, path, heading lineage, and an item anchor form the stable identity. A checklist can carry an explicit stable anchor such as <!-- coordinator-id: release-check -->; this is required when titles may change or duplicate inside one section. A number such as 26 is never the identity by itself.

Only explicit Markdown checkboxes become task records. Unchecked items import as planned, never ready. Checked items import as done with durable closure provenance tied to the source revision, exact preview source digest, observation time, branch, environment, path, and section. Ordinary prose cannot create a ready or done task. Explicit historical mappings create searchable project knowledge with closed, rejected, or superseded disposition; they do not create tasks or execution authority.

The response is an immutable preview with id, digest, project_event_revision, normalized items, conflicts, unresolved Markdown links, source provenance, and application time. It also captures each matching import record revision. Duplicate identities and generated-export input are blocking conflicts. Newer service edits are reported and preserved. Missing relative links are reported without causing the service to follow or read them.

Preview creation is idempotent with the normal persisted Idempotency-Key. Its audit event is project-scoped, but preview events are excluded from the authoritative project revision used by imports and exports. Creating or reading a draft preview therefore does not stale that preview or another draft preview.

Apply

POST /api/v1/projects/{project}/imports/{preview}/apply accepts:

{
  "preview_digest": "exact digest returned by preview",
  "expected_project_event_revision": 42
}

A currently authenticated human applies a preview. Agent credentials may create and inspect previews and read exports, but cannot apply them. This conservative initial migration boundary exists because a checked historical checklist creates durable closure outside the live completion protocol; widening it requires an explicit delegated-import policy. Under the SQLite writer lock, the service rechecks the preview identity and digest, project event revision, prior import-record revisions, actor authority, and blocking conflicts. Any project mutation after preview causes a stale-preview conflict. Application, import records, task/knowledge revisions, the idempotency receipt, and audit event commit atomically.

Reimport never changes a linked task or knowledge record that has a newer service revision. If the service projection is still at the last imported revision, a reimport may update it. Once an imported item or its service task is closed, later unchecked, missing, archived, or renamed source cannot reopen it. A checked item is imported closure evidence, not a live completion endpoint and not a substitute for the normal review/integration workflow on service-created work. Rejected historical prose remains noneligible.

Imported AGENTS.md, CLAUDE.md, handoff text, command examples, and hook names remain inert source content. Imports do not change project or workflow policy, grant authority, install hooks, or run commands.

Read and export

GET /api/v1/projects/{project}/imports/{preview} reads the immutable preview and its application status.

GET /api/v1/projects/{project}/exports?limit=50&cursor=... returns at most 200 records and caps the complete response page at 256 KiB. The projection includes current service tasks, immutable submission handoffs, project and workflow policy revisions with provenance, current searchable knowledge, decisions and answers, and imported historical records. It returns full record contents without silent truncation in structured records plus markdown, snapshot_event_revision, generated_at, generated: true, next_cursor, page_complete, and an explicit empty omissions list. If one record alone cannot fit, export fails with export_record_too_large instead of dropping content.

The opaque cursor binds the last sort key to the project event revision. If the project changes between pages, the next page returns export_snapshot_changed; the client restarts rather than labeling pages from different states as one snapshot. Generated Markdown begins with agent-coordinator-generated-export. Feeding it to preview produces a blocking conflict because exports are views of authority, never replacement authority.

Audited fixtures and limits

The tests use small sanitized excerpts read with git show from SithBit 20368b6fdb8c457cd822480f44c509253b9ea385 and Submission bbbdf8b6dbeee80ff0d1b87afaf5a91597c953c9. They cover the stale Item 26/do-not-requeue closure, repeated numerical labels, Submission’s stale July 20 resume prose, later merge/branch context, and unresolved memory links. The source repositories were read only; no checkout, hook, or source file was changed.

The parser intentionally supports headings, bullet checkboxes, explicit stable anchors, and explicit historical mappings rather than general Markdown semantics. Source archives that need richer interpretation must provide reviewed mappings.

Objective grouping contract

Objectives provide optional parent grouping without introducing a completion shortcut. An objective is an ordinary general task with its own description, acceptance criteria, ownership attempt, immutable submission, and configured review. Its objective record adds revisioned child membership and derived readiness.

Records and lifecycle

The objective ID is the wrapper task ID. Task APIs therefore expose the same record and include these fields:

• objective_id is the task ID for an objective, otherwise null.
• parent_objective_id identifies a task’s optional parent objective.
• parent_objective_required says whether that child gates its parent.
• objective_children_ready is true when every required direct child is done.

A task has at most one parent objective. Objectives may contain ordinary tasks or nested objectives from the same project. Internal review and integration activity tasks cannot be grouped. Required and optional children both participate in cycle detection; only required children gate parent work.

An objective cannot be claimed until all required children have lifecycle done. Optional children may remain open. The service repeats the required child check inside submission and general-review completion transactions. A parent therefore cannot become done through a stale claim or unchecked review. Completing a required child updates the parent’s ready time when the last required child becomes done.

The parent still completes its own acceptance workflow. Grouping a set of done children does not mark the parent done, create a branch, publish source, or stand in for review.

API

POST /api/v1/projects/{project}/objectives accepts:

{
  "title": "Release objective",
  "description": "Coordinate the release outcomes.",
  "acceptance_criteria": ["Required outcomes and release evidence are reviewed"],
  "priority": 2,
  "planned": false,
  "children": [
    {"task_id": "task-id", "required": true}
  ]
}

Creation atomically inserts the general wrapper task, objective projection, initial membership revision, task revision, receipt, and event. priority defaults to 2, description to an empty string, planned to false, and children to an empty list.

GET /api/v1/projects/{project}/objectives?cursor=ID&limit=50 returns stable ID-ordered summaries. Limits are 1–200. Each summary is a flat task record plus objective_revision, child counts, completed required-child count, required_children_ready, and membership_frozen.

GET /api/v1/projects/{project}/objectives/{id}?cursor=REVISION&limit=50 returns the summary plus all current children in caller-defined order. Current membership is limited to 100 children. Each child includes its ID, title, lifecycle, current task revision, derived work status, required flag, and position. The response also includes at most 200 immutable membership-history entries. membership_history_next_cursor is the exclusive revision cursor for the next older page.

PATCH /api/v1/projects/{project}/objectives/{id}/children accepts:

{
  "expected_revision": 1,
  "children": [
    {"task_id": "task-id", "required": true},
    {"task_id": "follow-up-id", "required": false}
  ]
}

expected_revision binds the objective’s objective_revision. A successful change appends immutable membership history and increments both objective_revision and the wrapper task revision, so task history records the changed completion scope.

Mutation and freeze rules

Objective mutations use the normal SQLite writer transaction. Authentication, current revision, scope, cycle, and freeze checks occur after the writer lock; the effect, receipt, and event commit together.

Membership is editable only while the parent is open or planned and has never had an ownership attempt or submission workflow. The first claim freezes it permanently. This prevents later required/optional changes from invalidating a submission, approval, or completed objective. membership_frozen exposes this state for clients.

Task prerequisite edits and objective membership changes validate one combined directed graph. An edge points from a task to its prerequisite and from an objective to each child. The service rejects a change if the target can already reach the source through either edge type. Cross-project children, duplicate children, self-membership, and a second parent are rejected.

Ordinary task edits remain subject to task revision checks, active-work guards, workflow guards, and the project’s existing delegated rule-editing policy. Objective membership cannot bypass those checks or weaken reviewed acceptance requirements.

Task history contract

Task history is a read-only, project-scoped view of durable service evidence. It does not change task state and does not grant workflow authority.

Route

GET /api/v1/projects/{project_id}/tasks/{task_id}/history

Query parameters:

• kind is required and is one of attempts, checkpoints, checkouts, jobs, job_observations, resources, artifacts, submissions, reviews, integrations, task_revisions, or events.
• limit is optional, defaults to 50, and must be from 1 through 200.
• cursor is the opaque next_cursor returned by the preceding page. Clients must not parse or alter it.

The response data has this shape:

{
  "project_id": "project UUID",
  "task_id": "requested task UUID",
  "subject_task_id": "subject task UUID",
  "kind": "attempts",
  "snapshot": "opaque insertion snapshot identity",
  "items": [
    {
      "kind": "attempts",
      "relation": "subject",
      "task_id": "task UUID for this record",
      "occurred_at": "RFC 3339 timestamp or null",
      "record": {}
    }
  ],
  "next_cursor": "opaque cursor or null"
}

A workflow activity task resolves to its subject task. Pages requested for the subject or any of its workflow activity tasks traverse the same task graph. relation distinguishes subject evidence, workflow activity evidence, and other subject evidence such as artifacts attached through a submission. Every SQL relationship is also constrained to the requested project.

reviews include the activity, immutable decision, and findings. integrations include the activity, authorization, target hold, publication intent, integration result, reconciliation, and the complete job records named by the result’s exact check-job identifiers. submissions include the recorded acceptance evidence, handoff, artifact identifiers, and immutable knowledge revisions. No record text is silently shortened.

Bounds and cursor behavior

Each page is limited by both its requested item count and a 256 KiB serialized data budget. If one complete record cannot fit, the service returns conflict code history_record_too_large instead of returning partial evidence.

The first request captures the greatest insertion row identifier for that kind. Later pages keep this cutoff, so records inserted concurrently are excluded and there are no gaps or duplicates among rows that remain present. snapshot names this insertion cutoff. It is not an immutable state snapshot: durable rows that the service permits to change may show their newer values on later pages.

The cursor is versioned and bound to the project, requested task, history kind, cutoff, last row, and service cursor epoch. Reusing it with another project, task, kind, or snapshot returns conflict code history_cursor_mismatch. Malformed cursors return a bad request.

Version 2 cursors include the service cursor epoch, which rotates before restored data is published. A cursor from before restore is rejected; restart the history query. Version 1 cursors from earlier service releases must also be replaced. Backup uses VACUUM INTO on a separate image and leaves live row identifiers unchanged. Maintenance must not run an in-place VACUUM while cursors remain valid; any procedure that rewrites row identifiers must rotate the cursor epoch.

Sensitive fields

History preserves operational evidence but excludes authentication material. Attempt session and credential identifiers, reporter identities and proof hashes, job observation request hashes, submission contributor sessions, review sessions, and artifact storage keys are omitted. Event data is parsed as structured JSON and recursively removes fields whose names identify authorization, cookies, credentials, passwords, proofs, request hashes, secrets, sessions, or tokens. Service-created events currently store no authentication material in event data.

Integration wiring

The core crate must declare and re-export history:

pub mod history;
pub use history::*;

The server crate must declare pub mod history; and merge history::routes() into the application router. Migration 0011_history_indexes.sql must run after the existing workflow, knowledge, import, artifact, and operator migrations.

Retained observation payloads

Maintenance may clear the summary of an old, exactly redundant intermediate job observation. History exposes payload_compacted_at as an RFC 3339 timestamp, or null when the payload is intact. The row, sequence, process identity, state, and request identity remain stored. First/last observations, state transitions, and nonduplicate progress summaries remain intact. See retention.

Event pages build the requested task’s related record identities, then use the project/record index. They include task-, job-, and submission-associated artifact mutations. Returned pages remain bounded; query work for a task with exceptionally deep evidence grows with that task’s own history, rather than unrelated project events. The release volume benchmark does not model 100,000 records on one task.

Operator access contract

Operator access keeps a stable principal identity separate from passwords, browser sessions, and agent tokens. Human accounts have role admin or operator; agent principals retain role agent. Every authenticated principal can see all projects, while only a currently enabled human administrator can manage accounts and agent credentials. There is no public enrollment or password recovery route.

All HTTP responses use the standard data, request_id, and server_time envelope. Mutations require Idempotency-Key; browser mutations also require the configured origin and CSRF token. Account and credential events contain no passwords, password hashes, API tokens, token verifiers, cookies, or CSRF values.

Human accounts and passwords

GET /api/v1/auth/account returns the current human’s own account as data.operator:

{
  "id": "principal-id",
  "name": "operator-name",
  "kind": "human",
  "role": "operator",
  "enabled": true,
  "revision": 2,
  "created_at": "2026-09-10T00:00:00.000Z",
  "disabled_at": null
}

GET /api/v1/admin/operators?cursor=... lists at most 200 human accounts with a bounded next_cursor. GET /api/v1/admin/operators/{id} returns one account. Both require an administrator.

POST /api/v1/admin/operators creates a human account from:

{"name":"reviewer","role":"operator","password":"a private initial password"}

The password must contain 12 through 1024 bytes. It is hashed with uniquely salted Argon2id outside the database writer transaction. The transaction then rechecks the administrator and creates the principal, receipt, and event atomically. The response contains data.operator and never echoes the password. A retry must re-enter the original password. The service checks it against the resulting account’s slow password hash and returns idempotency_secret_mismatch for a different password. If that account has since changed its password, the original creation request can no longer be secret-verified; inspect the account instead. The receipt retains neither the password nor another password-checking oracle. A successful replay returns freshly read account metadata, so the old receipt cannot represent later role or enabled-state changes as current. Account creation is the only mutation whose receipt is bound to the human administrator principal instead of the individual browser session. This permits replay after signing in again following an uncertain response. The service still rechecks the new browser session and current administrator role under the writer lock. Revoked, disabled, or demoted administrators cannot replay it. All other mutation fingerprints retain their existing session binding.

POST /api/v1/admin/operators/{id}/access accepts:

{"expected_revision":2,"role":"admin","enabled":true}

The writer-locked transaction rechecks the administrator, target revision, and last-active-administrator invariant. Changing access revokes all target browser sessions. Two concurrent changes cannot both remove the final administrators. Stale input returns revision_conflict; an attempt to remove the sole enabled administrator returns last_active_admin.

POST /api/v1/auth/password is available only to a human browser session and accepts:

{
  "current_password": "the current private password",
  "new_password": "a different private password",
  "expected_revision": 2
}

Current-password verification and new-password hashing run outside the writer transaction. Under the writer lock the service rechecks the live browser session, unchanged old hash, and account revision, then replaces the hash and revokes all of the account’s browser sessions, including the caller. The response clears the browser cookie. If the response is lost, sign in with the new password and inspect GET /api/v1/auth/account; a revoked old session cannot use a receipt as renewed authority.

Browser sessions

GET /api/v1/browser-sessions?cursor=... lists the caller’s sessions. An administrator may add principal_id to inspect another human. Items contain only id, principal_id, created_at, expires_at, revoked_at, and current. Cookie tokens and their hashes are never returned.

POST /api/v1/browser-sessions/{id}/revoke with {} revokes one of the caller’s sessions or, for an administrator, another human’s session. Revoking the current session clears its cookie. Existing task, attempt, job, and resource records are unchanged.

Agent token rotation

Existing POST /api/v1/admin/agents enrollment creates a stable agent principal with an initial named credential. POST /api/v1/admin/credentials/{old_credential_id}/rotate accepts:

{"name":"linux-builder-2026-09","revoke_old":true}

revoke_old defaults to true. Rotation requires an active credential belonging to an enabled agent principal. It creates a separately named credential for that same principal and, by default, revokes the old credential and closes its agent sessions. It never transfers or revives an attempt and never resolves or releases a job or resource hold.

The first response contains principal_id, principal_name, credential, replaced_credential_id, replaced_credential_revoked, and the new token. Only that response exposes the token. A retry returns the same credential identity without token, adds secret_unavailable: true, and directs the administrator to rotate the newly created replacement credential with a new idempotency key. The default rotation then revokes that credential while issuing another token.

After a restore revokes every credential, an administrator can preserve an existing agent identity while issuing fresh authentication with:

POST /api/v1/admin/agents/{principal_id}/credentials
{"name":"after-restore"}

The principal must still be an enabled agent. The first response contains its unchanged principal_id and principal_name, the new credential record, and the one-time token. A replay returns the saved credential identity without the token and sets secret_unavailable: true. The operation is available during restore reconciliation so a recovered administrator can reconnect clients, but the global restore pause prevents those clients from claiming or changing work until reconciliation finishes.

Host-local recovery

Lost-password recovery is exported for the server’s local command and has no HTTP route:

recover_operator_password(state, username, new_password, reason)

The command validates and hashes the new password before acquiring SQLite’s writer lock. Its single transaction enables the named existing human principal, increments its revision, revokes all browser sessions, and records a bounded reason. The principal’s ID and role remain unchanged. The transaction does not modify agent credentials or sessions, attempts, jobs, submissions, or resource holds. The old password and all prior browser sessions remain invalid.

The recovery event identifies initiator_kind as host_operator and sets authenticated_principal_id to null, because the local command has no authenticated browser or agent principal. subject_principal_id identifies the recovered account. The existing non-null events.actor_id foreign key also points to that account solely as a subject_reference, as recorded by actor_id_role; it must not be presented as the initiator of the recovery.

Backup and restore storage contract

Status: implemented storage engine. Authority invalidation, maintenance-mode reconciliation, server command wiring, timers, and the measured recovery exercise are integrated separately.

Commands and callable interface

The host-local server commands are:

agent-coordinator-server ... backup --repository BACKUP_REPOSITORY
agent-coordinator-server backup-verify --snapshot SNAPSHOT_DIRECTORY
agent-coordinator-server restore --snapshot SNAPSHOT_DIRECTORY \
  --destination ABSENT_DATA_DIRECTORY --reason REASON

The engine exposes these asynchronous Rust functions:

create_backup(state: &AppState, repository: &Path) -> anyhow::Result<Value>
verify_backup(snapshot: &Path) -> anyhow::Result<Value>
restore_backup(snapshot: &Path, destination: &Path, reason: &str)
    -> anyhow::Result<Value>

backup requires an existing configured database opened read-only by command wiring. It does not create, migrate, or write the live database. backup-verify does not open the configured live database. restore runs before normal live database opening and accepts only a destination directory that does not exist.

Successful backup output includes snapshot_id, the absolute completed snapshot_path, created_at, database_bytes, artifact_count, artifact_bytes, snapshot_bytes, and retention counts and bytes. Successful verification returns the same identity and byte counts with verified: true. Successful restore returns snapshot_id, the absolute data_directory and database_path, artifact counts and bytes, and the authority invalidation result. Paths are host-local operational output and are not written into the snapshot manifest.

Consistent snapshot and artifact hold

SQLite VACUUM main INTO ? produces the database image at a consistent point while the service remains online and includes committed data represented by the WAL. Copying a live main database file by itself is unsupported. The destination database file must be absent.

Before starting that SQLite snapshot, backup obtains a shared advisory lock on:

<database-filename>.artifacts/.gc.lock

It retains the lock through database verification and copying every finalized artifact named by the copied database. Uploads and ordinary database writes can continue. Every physical artifact deletion and orphan cleanup attempts an exclusive, nonblocking lock after any database writer transaction has ended. If backup holds the shared lock, cleanup leaves the bytes in place for a later bounded reconciliation pass. A process exit releases the operating-system lock.

The manifest selects only upload artifacts whose copied row is finalized and whose retention is live at the snapshot timestamp, or which are pinned. A file must be regular, no larger than 16 MiB, and exactly match the recorded size and lowercase SHA-256 digest. Missing or changed bytes fail the snapshot before publication. External-link artifacts remain in SQLite and require no local blob.

Repository and publication

The repository layout is:

BACKUP_REPOSITORY/
  .lock
  .staging/
    <timestamp>-<snapshot-uuid>.partial/
  snapshots/
    <timestamp>-<snapshot-uuid>/
      database.sqlite3
      blobs/
        <artifact-storage-uuid>.blob
      manifest.json
      COMPLETE

Each snapshot directory is self-contained and immutable. The first release does not deduplicate bytes across snapshots, so capacity planning must use the reported snapshot_bytes and retained_bytes. At maximum retention the union usually contains up to 53 complete copies because the newest daily snapshot also belongs to an hourly bucket.

Creation and retention take an exclusive operating-system lock on BACKUP_REPOSITORY/.lock. Verification and restore take a shared lock when the snapshot is in that repository’s snapshots or .staging directory and the lock already exists. A standalone copied snapshot is verified without creating files, directories, or changing permissions in its parent; the caller must keep such a standalone directory immutable during verification.

The engine creates directories with mode 0700 and files with mode 0600 on Unix, rejects symbolic links, creates files without overwrite, streams through a 64 KiB buffer, and syncs files and directories. It writes COMPLETE only after the database, blobs, and manifest are durable. COMPLETE binds the snapshot ID and exact manifest digest. The fully assembled and verified partial directory is renamed into snapshots and the parent is synced. Consumers count only a directory with a valid COMPLETE; partial directories are never usable backups.

Creation verifies SQLite integrity, foreign keys, the exact successful migration version/description/checksum set, and exact agreement between the manifest and all eligible artifact rows before publication. Verification repeats those checks as well as every recorded size and digest. For version-1 snapshots from schema 12 (the first backup implementation) onward, verification also accepts an exact, known contiguous prefix of the current migration set. Missing, altered, failed, unknown, future, and pre-format migration histories are rejected. Verification never migrates the snapshot; restore migrates only its private staged copy before invalidating authority. Opening a live database for backup still requires this executable’s complete current migration set. A failed assembly removes only its generated partial directory and neither publishes nor prunes. If a separately completed snapshot publishes but retention later fails, output contains warning.code: retention_failed; the new and prior usable snapshots remain available for inspection.

Fixed safety bounds

The first-release storage bounds are explicit constants:

• 8 GiB maximum SQLite image
• 1,000,000 finalized artifact entries
• 64 MiB maximum manifest before deserialization
• 16 MiB per artifact blob
• 256 MiB free-space reserve at the backup or restore destination
• 64 KiB streaming copy/hash buffers
• 45-minute operation deadline enforced by SQLite progress callbacks during snapshot and integrity work, and checked during streamed copies, hashing, and restore publication

The database size is checked from SQLite page count before VACUUM INTO and from the result afterward. Free space is checked before the database image and each artifact copy. The backup systemd unit has an additional 50-minute process limit. The engine deadline is cooperative: authority invalidation and operating-system I/O may finish after it, but an expired operation cannot publish restored data. The 45-minute engine limit supports the one-hour recovery target but does not establish that target until the documented exercise measures verification, copying, invalidation, administrator recovery, service start, and one recovered client on the release host.

Retention

After successful publication, retention sorts completed snapshots by their recorded UTC timestamp. It keeps the newest successful snapshot in each of the newest 24 represented UTC hour buckets and the newest successful snapshot in each of the newest 30 represented UTC day buckets. The retained set is the union. A corrupt manifest or completion marker aborts retention before deletion. Removal is restricted to complete snapshot directories outside that union and occurs under the exclusive repository lock.

Fresh-path restore

Restore first verifies the source completion marker, manifest, database, and all blobs without touching the configured live database. It creates a private sibling staging directory next to the requested absent destination and builds:

ABSENT_DATA_DIRECTORY/
  coordinator.sqlite3
  coordinator.sqlite3.artifacts/
    blobs/<first-two-storage-key-characters>/<storage-key>.blob
    staging/

The copied database must match the immutable snapshot manifest before authority invalidation. The engine then opens and migrates only the staged database, calls restore::invalidate_restored_state, checkpoints and closes SQLite, rechecks database integrity, foreign keys, and the current migration set, and re-hashes every restored artifact. The database checksum is expected to differ after authority invalidation. Finally it syncs the tree, atomically renames the whole staging directory to the absent destination, and syncs the parent. It never overwrites or renames the old installation.

The old installation must remain stopped and fenced. The authority layer rotates epochs, revokes restored credentials and sessions, expires attempt authority, marks in-flight jobs uncertain, retains physical and integration holds, and requires explicit reconciliation before coordination resumes. Those mutations and their audit contract are defined by the restore authority implementation.

Off-server copying

An off-server copy must contain the entire self-contained snapshot directory. Authentication and encryption belong to the operator-selected transfer system; credentials must not enter command arguments, manifests, logs, or the repository. For a repository copy, hold a shared source .lock; hold an exclusive destination .lock, copy database/blobs/manifest into a fresh .staging directory, and copy COMPLETE last. Release the destination lock, run backup-verify on that partial directory, then reacquire exclusive lock and atomically rename it into snapshots without overwrite. Record local creation separately from verified off-server copy time. Hourly local snapshots alone do not provide one-hour host loss protection.

Agent coordination service: discovery and implementation plan

Status: preserved release design and decision record. Product choices were resolved with the operator, and the body records the intended first release and the implementation sequence as it was planned. Consult implementation status for current behavior and accepted evidence; proposed names and historical milestone language below are not a claim that a particular interface is available.

Revised 2026-09-09 after reviewing SithBit, Submission, their memory, and local and global hooks. See the evidence report.

Goal

Allow autonomous agents on different workstations to coordinate work on a shared project without independently selecting the same task. Preserve task outcomes, handoffs, and useful lessons so subsequent agents can work from shared context. Remain independent of agent vendors, model providers, and Git hosting providers.

Planning and clarification are complete. The operator subsequently authorized implementation. See implementation status for working service/client behavior and remaining milestones; the full design below continues to define the release target. No production deployment has occurred.

Requirements supplied by the operator

• Agents start in a project directory associated with a project name and usually a Git repository.
• One service instance must coordinate multiple projects simultaneously, with agents working on different projects at the same time.
• Workstations connect over the public internet using HTTPS.
• Install the server as a native Linux service managed by systemd, behind an HTTPS reverse proxy.
• The service is the authoritative record for tasks, handoffs, and lessons, with Markdown import and export.
• Every authenticated person and agent has access to every project. There are no project-specific access grants in the first release.
• People use local operator accounts with passwords. Agents use separately issued, revocable API tokens.
• Agents can create and claim tasks autonomously; each project configures whether completion requires review.
• An expired task can be recovered by another agent after checking saved work and still-running jobs. Projects can instead require manual recovery.
• Code tasks count as complete and unblock dependencies only after required review, integration into the target branch, and validation of the integrated result.
• The service coordinates existing harnesses through API, CLI, and optional hooks. Local runners report jobs; remote agent launch/supervision is out of scope for the first release.
• The CLI and local job reporter support Linux and native Windows.
• The first release includes the HTTP API, web dashboard, and CLI with human-readable and JSON output. The MCP endpoint was implemented as item 6.1; TUI remains deferred.
• Each project can require independent agent review, human review, or both; the default review mode is independent agent review.
• Each project can allow agents to integrate approved, validated work automatically or require human authorization for integration.
• Each project may authorize agents to change binding project rules without human approval, in addition to publishing and correcting shared lessons.
• Store bounded log/report uploads in the service and source checkpoints in Git remotes; also support artifact links. Source-worktree bundles are not required.
• Test the first release for up to 20 projects, 50 simultaneous agent sessions, and 100,000 historical tasks across the service.
• Provide hourly backups, retaining 24 hourly and 30 daily copies, documented off-server copying, and a one-hour restore target.
• The production host is undecided. Use Ubuntu 24.04 LTS, x86_64, 2 CPU cores, and 4 GB RAM as the engineering test baseline, not a selected production host.
• Existing orientation and work records include AGENTS.md / CLAUDE.md, HANDOFF.md, BACKLOG.md, and DURABLE-RECORD.md. The request also mentions BACKOFF.md; support it as an additional configurable import filename without requiring it to mean the same thing as BACKLOG.md.
• Agents discover work, obtain ownership, report progress and health, and record completion results through the service.
• Other agents can distinguish active long-running work from potentially abandoned work and continue appropriately.
• Repository instructions should be short. An initial service connection should supply the detailed workflow and API usage instructions.
• Authentication is required. Unauthenticated clients receive safe instructions to show their human operator for configuring access.
• Operators need a web interface, CLI, and/or TUI to inspect projects, tasks, status, and outcomes.
• Preferred stack: Rust, Axum, SQLite; vanilla JavaScript and modern CSS for a web interface, with Alpine.js if it simplifies the implementation.
• Confirmed during discovery: separate Git worktrees per implementation task, with one integration step at a time into the target branch in the first release.

Workspace observations

• The project directory was empty when discovery began.
• It was not an initialized Git repository.
• No applicable AGENTS.md or CLAUDE.md was found in the directory or its parents.
• Reference-project review found an existing phased, parallel workflow in SithBit and a simpler migration/branch workflow in Submission. The service needs to represent both without requiring a particular agent harness.

Proposed foundation

These engineering choices implement the confirmed requirements. Tunable defaults are specified in the linked contracts and acceptance plan.

1. A central HTTP service owns coordination state. Workstations use an API; they never access a shared SQLite database file directly.
2. Provide a versioned JSON API with complete request/response examples and concise, versioned agent instructions. A CLI can expose the same operations in both human-readable and JSON formats. MCP support was added as backlog item 6.1; TUI remains deferred; core functionality does not require either.
3. Claiming a task atomically verifies eligibility and records ownership in one database transaction. Listing work does not reserve it. Both “claim this task” and “claim next eligible task” can use the same eligibility rules.
4. Separate the durable task from each execution attempt. Keep previous ownership, checkpoints, failures, and results when a new attempt starts.
5. An attempt has an authenticated owner, a session identity, an expiring lease, and an ownership generation. Heartbeats renew only a still-valid lease. Updates from an expired or superseded attempt cannot change current work.
6. Use service time for lease decisions. Evaluate validity on every ownership operation; correctness must not depend on a background cleanup timer.
7. Distinguish a health heartbeat from evidence of progress. A running helper process alone must not be interpreted as proof that an agent is advancing.
8. Retryable mutations use idempotency keys. An interrupted response must not cause duplicate task creation, an extra claim, or repeated completion records.
9. Save completion, handoff, linked lessons, and the task transition together. Do not lose context between independently submitted updates.
10. Record Git branch, commit, and optional review URL as evidence. The service must not claim it can stop an offline process or guarantee exactly-once Git pushes, deployments, or other actions outside its own database.
11. Keep searchable lessons with provenance and revision history. Distinguish agent observations from policy adopted by a human or an authorized agent. Relevant context should fit a bounded response instead of dumping every historical record.
12. Return useful next actions and recovery instructions with API responses, particularly for authentication failures, conflicts, and expired leases.
13. Treat implementation, verification, review, integration, and deployment as separately evidenced activities. Task completion requirements are explicit; a local test pass or a branch commit must not imply integration or deployment.
14. Bind shared records to stable project IDs and source revisions, independent of workstation paths or vendor-specific memory locations. Preserve scoped decisions and completed/rejected-task records through archival and import.

The proposed ownership and recovery contract is developed further in docs/coordination-contract.md. It deliberately defines the selected agent-driven recovery and integrated-completion rules. Lease timing and reviewer separation are specified as engineering defaults there. Password accounts, separate agent API tokens, and project-configurable agent or human review are confirmed.

The design-stage first-connection flow and short repository snippet are in docs/onboarding-contract.md. See the current CLI guide for implemented commands.

The engineering draft in docs/workflow-spec.md specifies task lifecycles, derived work statuses, activity eligibility, and completion transactions, consistent with the confirmed product decisions. The proposed wire interface is in docs/api-contract.md, including agent sessions, claim/renewal examples, and actionable error responses. Persistence constraints, operation permissions, and release checks are in docs/implementation-spec.md.

Decision log

Repository evidence informed these choices. A policy or permission recorded for a past task in a reference project is not a new permission for this service or a future task.

Multiple-project operation

Confirmed scope: a single deployment can coordinate SithBit, Submission, and other projects concurrently. All authenticated users and agents can access all projects. Projects separate work context, not audiences. Administrative actions and attempt ownership follow the operation-level permission matrix in implementation-spec.md.

• Every task, attempt, decision, evidence record, and project-specific lesson belongs to an explicit project. Claims use the repository’s project binding or an explicit project argument; there is no service-wide “current project”.
• Project configuration controls its workflow and review requirements. A credential works across all projects. Each operation checks authentication, any required role/ownership, and consistency of its project/record references.
• The operator dashboard shows an overview of all projects after login and filters into each project’s tasks, active work, decisions, and knowledge.
• Integrations into different repository targets can proceed concurrently. A blocked task or resource in one project does not block unrelated projects. If projects intentionally share a repository target or physical resource, they must share its canonical reservation identity rather than evade its concurrency limit through different project names.
• Knowledge retains its source-project association for relevance and provenance, while every authenticated caller can read across projects. A proposed common collection holds reusable lessons with source links; it is not an access boundary.
• Work and long-running checks take place outside database transactions. Coordination transactions stay short so an active project does not hold the service’s database transaction open for the duration of its work.

Revisions based on the repository review

The evidence IDs below refer to the review report.

1. Phases and dependencies belong in the first-release model (E1). A task may have child tasks and explicit dependencies. Model implementation, review, and integration as typed work when separately owned; a simple task need not use all types. One primary attempt owns each task, so independent reviewers use their own linked review tasks. Parent completion depends on required child outcomes and its own acceptance criteria. Prevent dependency cycles.
2. Task readiness needs an evidence check (E2, E3). Record the candidate’s source, observed repository revision, acceptance criteria, and whether it is still undone. After claiming, check this against the actual checkout before changing code. If work is already delivered, reconcile the record with the commit and acceptance evidence through an authorized transition. Keep stable duplicate, supersession, rejection, and closure links; do not reuse old IDs.
3. External jobs have their own records (E6). A test run outlives an observer or agent turn. Store job/run ID, host and checkout identity, producer status, last observation, start/end, exit status, and artifact references. Separate running, succeeded, failed, interrupted, and unknown. An observation timeout cannot trigger a restart. A job heartbeat does not automatically renew an implementation lease or prove useful agent progress.
4. Verification evidence describes an exact input (E1, E6, E7). Record commit/tree or dirty-snapshot digest, gate-definition version, environment, required checks, actual checks, skipped checks with reasons, terminal status, and relevant log/result digests. A subset pass, advisory formatter, or killed run cannot satisfy a full verification requirement. Changed inputs invalidate applicability, not the historical record. A path on another workstation is labeled local-only unless a separately accessible artifact is published.
5. Resource admission complements worktree isolation (E7). Support named exclusive resources and bounded local build slots with workstation, checkout, or shared project/environment scope. Client-declared dependency/conflict information determines which work interferes; file-name disjointness alone is not sufficient. Atomic acquisition prevents partial reservation deadlocks. An expired task lease does not prove an external job released a resource.
6. Decisions and blockers are structured (E8, E9). Distinguish dependency, missing capability, human decision, upstream release, verification failure, and permission blockers. Preserve rationale, affected tasks, reopening condition, observation time, and answer. During unattended work, park a blocked task and continue eligible work. An empty eligible queue is a normal stop, with a reason and a suggested next check, not a reason to invent work.
7. Separate knowledge from standing policy (E4, E5, E9). Store practical lessons, validated facts, decisions, rejected alternatives, and current checkpoints as distinct record kinds. Include scope, provenance, applicable versions, supersession, and reopening conditions. Start with full-text search and tags; optional semantic retrieval must not require one model vendor. Promotion into binding policy requires the appropriate authority. Repository instructions continue to express local rules; imported memory cannot silently override them. Track useful/corrected lessons to improve retrieval over time.
8. Migration must reconcile, not just copy Markdown (E2–E5). Provide a previewed import from configurable paths, including SAVERS.md, nested handoffs, archives, and multiple memory directories. Record unresolved links and conflicting statements. Preserve original source plus proposed structured records. Historical, struck, consumed, or rejected entries are not ready tasks. Stable import identities include source context, not just an item number. Re-import cannot reopen closed work or overwrite newer service state.
9. Hooks are optional local adapters (E6–E8). Define a vendor-neutral lifecycle: connect/resume, claim, checkpoint, observe a job, submit, release, and disconnect. Repeated startup/compaction reconnects to an existing attempt; it does not claim twice. A turn-end hook does not declare completion or kill work. Keep formatters and command guards local. Report actual adapter capabilities/version rather than assuming an installed script is active. Imported hook text is never executed by the service. Recovery instructions must respect applicable local rules and expose a policy conflict rather than repeatedly recommend an action that the caller’s harness refuses.
10. Autonomy has scope and stop conditions (E8, E9). Keep a durable record of what the operator authorized, for which task/action/environment/revision, and any limits. A service login or task claim does not confer deployment or data mutation authority. Preserve valid grants without repeated questions. Retry limits count by stable finding/defect class across attempt or phase renaming; time/work limits and escalation thresholds are configurable project policy.

Proposed minimum data model

Relational constraints, indexes, permissions, transactions, and retention defaults are specified in implementation-spec.md.

Avoid a mandatory custom workflow language, embeddings service, or agent-vendor SDK. These are conventional records exposed through one API and CLI.

Proposed agent workflow

1. Read the short repository connection instructions and applicable local rules.
2. Connect using locally configured credentials. Without authentication, receive public setup help suitable for showing the human; expose no project records.
3. Resolve the stable project binding and reconnect to an existing session or attempt if resuming. Report checkout state and capabilities; do not overwrite a dirty checkout or advance one used by an active job.
4. Receive a bounded orientation packet: active work, pending decisions, applicable policy, candidate tasks, relevant lessons, and exact next actions.
5. Atomically claim eligible work. If none is available, receive structured reasons such as missing capability, pending decision, or work owned elsewhere.
6. For implementation, prepare its separate worktree. Confirm the task is still undone on the relevant revision, acquire required resources, and record the initial checkpoint before making changes.
7. Report progress, blockers, and job observations. Renew ownership within the lease contract. If a tool observer dies, reconnect to the known job before considering a new run. If authority expires, preserve recovery information without changing the new owner’s work.
8. Submit the outcome, exact-revision verification, handoff, and new lessons. Required review is separate work with its own owner and evidence.
9. An authorized integration task reserves the target branch, validates the candidate and current target, integrates, and records validation of the resulting revision. A merge conflict or incomplete validation remains visible.
10. Update task/parent state according to the completion policy. Preserve historical records and generate optional Markdown snapshots and a run summary.

Operator interface proposal

The operator selected both a CLI and a web interface in the first release. The CLI supports human-readable and JSON output on Linux and native Windows; the web interface uses vanilla JavaScript and CSS, adding Alpine.js only where it removes complexity. MCP was implemented as backlog item 6.1; a TUI remains deferred.

The browser should make these questions easy to answer:

• Which projects and tasks are ready, active, blocked, under review, or complete?
• Who owns this attempt, where is its worktree, and when did it last report health versus actual progress?
• Is its long-running check still running, unknown, interrupted, or finished? What exact code and checks does the evidence cover?
• What decision, permission, capability, or shared resource is blocking progress?
• What was integrated, and what remains only on a branch or workstation?
• Which lessons and rejected approaches apply here, and what supports them?
• Which credentials and workstation registrations are active, and what operations may each role perform?

Decision prompts must explain the tradeoff without requiring prior session shorthand. The service should expose actionable failures and retain answers.

Implementation readiness

No blocking product questions remain. docs/release-scope.md records engineering defaults, deferred features, and readiness evidence. The production hostname, final host, off-server backup destination, and credentials are installation inputs, and are not required to build and test the service.

The planning output includes the architecture and data model, transaction and authority rules, workflow/API contracts, native-client behavior, permissions, migration, shared knowledge, artifact handling, and measurable release checks. Generated OpenAPI schemas, executable examples, migrations, and automated tests belong to the implementation milestones below; the documents do not claim they already exist.

Proposed implementation milestones

Implement against the linked contracts. Each milestone leaves an executable slice with the stated evidence and matching generated API/CLI documentation.

1. Service, persistence, and access: Rust/Axum server, SQLite migrations, principal/operation permissions, public authentication help, authenticated bootstrap, HTTPS deployment configuration, structured logs, and health/readiness. Demonstrate that unauthenticated callers cannot read project data, and that a newly authenticated workstation can access all projects.
2. Ownership and task state: typed tasks/dependencies, attempts, conditional state transitions, leases, events, idempotency, and a minimal JSON-capable CLI. Demonstrate competing claims, dependency cycles/refusals, lost responses, expiry, revocation, and the selected recovery policy.
3. Worktrees, jobs, and integration: checkout registration, named resource admission, external-job observations, verification receipts, review records, and serialized integration. Demonstrate observer loss without duplicate work and reject completion evidence for changed inputs.
4. Shared context and migration: knowledge revisions/search, decision queue, authorization scopes, provenance-aware Markdown/memory import and export, and bounded orientation/task packets. Exercise representative SithBit and Submission fixtures, including closed items and conflicting historical records.
5. Operator and agent experience: browser views, full CLI workflows, service-delivered instructions, short AGENTS.md/CLAUDE.md snippets, and optional lifecycle adapters. Verify that a small agent can complete the documented workflow without vendor-specific context or credentials in its prompt.
6. Operational release: packaging for selected operating systems, backup/restore and authority invalidation, restart/disconnection scenarios, appropriate load tests, and a two-workstation end-to-end acceptance exercise. Check real behavior against every accepted requirement before release.

Candidate acceptance scenarios

• Simultaneous eligible claims yield exactly one current owner for a task.
• A valid heartbeat extends the owning attempt; another session cannot renew it.
• A late heartbeat or completion cannot revive an expired/superseded attempt.
• Recovery preserves previous checkpoints and applies the chosen takeover policy.
• Repeating a request after a lost response returns the same logical outcome; replaying an old claim never grants fresh authority.
• A failed completion transaction leaves no partial result or state transition.
• Unauthorized clients see setup help without project or credential disclosure.
• Authentication and applicable operation/ownership checks apply to every private read/write surface; no project allow-list is required after authentication.
• SithBit and Submission can each have active claims and jobs at the same time; an integration reservation in one does not block an unrelated target in the other.
• An authenticated caller can deliberately select either project and read its tasks, lessons, events, search results, imports, and permitted artifacts.
• A task operation cannot attach another project’s attempt or evidence by accident; cross-project sharing uses an explicit authorized operation.
• An authenticated operator sees both projects in the overview, while each agent’s next-task request stays bound to its selected project.
• A new agent can follow only the repository snippet and returned instructions to claim, checkpoint, complete, and retrieve relevant lessons.
• Restarting the service preserves coordination state. Restoring an older database invalidates old live sessions/leases before accepting work.
• Stale or disconnected agents are given an explicit stop/reconcile response; the service does not imply control over their local processes.
• A repeated startup/resume/compaction event reconnects without a second claim.
• An ended observer/turn does not turn a live test run into a failed task or launch another copy. Old completion markers cannot identify a new run.
• A test of an uncommitted snapshot is not attributed to a later commit without verified equivalence. Skipped or missing checks cannot satisfy required coverage.
• Two tasks on separate worktrees can proceed independently, while integrations into the same project target branch serialize.
• Reservations distinguish workstation-local build slots from a shared test environment. Loss of a task lease does not assert that a process has stopped.
• Parent tasks and downstream tasks remain incomplete/unready until their declared acceptance requirements and dependency outcomes are satisfied.
• A closed task stays closed across archive/export/re-import; repeated numeric labels in different source contexts are not conflated.
• Import can flag a historical branch-only handoff beside later merge evidence, retaining both with their dates/revisions and no new implied merge permission.
• A renamed checkout can retrieve the same project knowledge, while broken memory links and conflicting versions are reported rather than discarded.
• Pending decisions survive agent sessions; an answer unblocks only applicable work and cannot be recorded as a human authorization by an ordinary agent.
• A repeated defect class reaches its configured escalation limit even when the agent renames or splits the task. Unrelated ready tasks remain eligible.
• Imported hooks or lessons cannot execute commands, expand credentials’ access, or automatically promote themselves to trusted policy.

Technical references checked during discovery

• SQLite transactions: SQLite supports one writer at a time; short write transactions can serialize claims.
• SQLite WAL: WAL permits readers alongside a writer and requires database access on one host, not a network filesystem.
• Axum documentation: official framework documentation for the proposed Rust HTTP implementation.

Dependency versions and security advisories must be checked when implementation begins; this discovery document does not pin dependencies.

Additional release sequencing — 2026-09-09

The operator requested an MCP server endpoint as backlog item 6.1, followed by consolidating documentation and READMEs into an mdBook project, item 6.2. Both follow Linux acceptance and precede the operator-initiated native Windows workstation acceptance at item 7. Windows CI remains part of implementation validation and does not replace that physical-workstation exercise. The main agent reviews each completed backlog item before proceeding to the next.

First-release scope and implementation readiness

Status: historical implementation-readiness record. Confirmed choices are in PLAN.md. Engineering defaults remain distinguishable from operator-selected requirements; completed validation and remaining acceptance work are recorded in implementation status.

Operating targets

No blocking product questions remain. Native Linux systemd installation, HTTPS, all-project access, authentication method, agent task creation/claiming, worktree isolation, recovery mode, and integrated completion are already settled. Linux and native Windows clients, HTTP API/web/CLI interfaces, and configurable agent/human/both review are also confirmed. Projects may allow automatic agent integration or require human authorization. Agents maintain shared lessons; projects may additionally delegate changes to binding rules without human approval. Bounded log/report uploads, Git source checkpoints, and artifact links are confirmed. The operator subsequently added an MCP endpoint as backlog item 6.1 and documentation consolidation into mdBook as item 6.2, both before final Windows workstation acceptance at item 7. TUI and source-worktree bundle storage remain deferred.

Proposed engineering defaults

These are recommendations for the implementation specification, not additional operator confirmations. They may be adjusted when the product answers require it.

Propose a one-minute reporting interval and ten-minute renewable ownership window, configurable per project within server limits. The bounded-helper contract is in coordination-contract.md. A stale-progress warning is separate from lease expiry. Validate these defaults against the long-running SithBit gates before treating them as release defaults.

For Markdown input, treat BACKOFF.md as a configurable additional filename. Support BACKLOG.md directly. This avoids making a possible filename typo block the design while still supporting a distinct file if one exists.

Explicitly deferred features

• Interactive TUI clients.
• Remote launching/supervision of agent harnesses and arbitrary server-run jobs.
• Shared editable checkouts for independent implementation tasks.
• Source-worktree/build-directory bundle storage and a hosted source repository.
• Required model APIs, embeddings, semantic deduplication, or a custom workflow DSL.
• Multiple active coordinator servers, database clustering, and project ACLs.
• Email/chat/push integrations, public account registration, and email-dependent account recovery.
• Automatic rewrites of existing project/global hooks or automatic migration of SithBit/Submission into the running service.

Readiness evidence and implementation work

During implementation, derive OpenAPI schemas and executable Linux/PowerShell examples from the shared request/response types; implement migrations and automated tests alongside each milestone. These deliverables are specified work, not unfinished product questions or claims of an existing implementation.

The final host, public hostname, TLS configuration, off-server backup destination, and initial credentials will be supplied during installation. Local acceptance uses isolated repositories, disposable databases, test credentials, and the documented baseline. Production access is not needed to implement the service.

Implementation readiness means these contracts agree and have testable outcomes. It does not require selecting every crate version, cosmetic UI detail, or table index in advance. Those are implementation choices validated during the relevant milestone.

Proposed workflow specification

Status: historical implementation design. The current service implements the workflow using the contracts and routes listed in implementation status; some design-stage names and state shapes below differ from that implementation. This chapter preserves the reasoning beneath the confirmed product choices in PLAN.md. Projects can permit automatic integration or require human authorization, and can delegate binding-rule changes to agents. The service stores bounded logs/reports; source checkpoints remain in Git remotes.

Task outcome and available work

A task represents an outcome. Its kind is code, general, objective, agent_review, human_review, or integration. Agents normally create code or general tasks. An objective groups required children. Review and integration tasks are linked workflow activities, created transactionally from a submission and the applicable project policy, with stable identities preventing duplicates.

Persist a task lifecycle of planned, open, done, canceled, or superseded. Expose a derived work status alongside it, so agents do not have to infer readiness from raw records:

The response also includes structured blockers, current attempt, submission, linked activities, and exact next actions. It never uses a status string as a substitute for these facts. These statuses describe the selected task: a code task can be waiting for review while its linked review task is in progress.

Status is derived from authoritative records, including server-time lease validity. The database must not rely on a periodic job to change in_progress to recovery_required before refusing a stale update. Transactional projections may speed dashboards but cannot grant authority or hide expired ownership. The API does not accept arbitrary writes to derived status.

An expired current attempt or unresolved external outcome takes precedence over ready. A task waiting for an active linked review/integration displays that phase and retains any blockers as separate fields. No task becomes complete merely because all owners have released their attempts.

Admission and task selection

Task admission requires a title, concrete outcome, acceptance criteria, kind, project, priority, source/provenance, and any required capabilities or resources. A code task also needs its repository/target binding. Use a checklist of explicit criteria; do not require the service to interpret arbitrary prose as executable tests. Planned tasks can hold incomplete drafts.

Default priorities are urgent, high, normal, and low. Within a project, selection filters eligibility and requested task kinds/capabilities, then orders by priority, oldest-ready time, and stable ID. Ready time resets only when work actually becomes eligible again, not on a heartbeat or cosmetic edit. There is no agent-vendor preference or required scheduling model.

Agents search existing and recently closed tasks before creating new work. An optional stable external source key is unique within its declared source namespace. Similar titles produce suggestions, not automatic task identity. The service can prevent duplicate claims and repeated source imports; it cannot guarantee that separately worded outcomes never overlap.

Normal dependency edges require the prerequisite task to be done. Cancellation and supersession produce an explicit blocker and replacement reference; they do not silently satisfy the old edge. Authorized dependency edits retain history. Objectives cannot complete until required children and their own acceptance criteria are satisfied. Reject parent and dependency cycles, including concurrent edits that would create a cycle only when combined.

Attempts and checkpoints

Each attempt belongs to one task, authenticated session, workstation, and ownership generation. Its state is active, submitted, released, blocked, expired, or canceled. Only an active, current, unexpired attempt can perform ownership-dependent mutations. Historical attempts are retained.

For code work, claim first, then prepare/register the separate worktree and checkpoint the starting repository state before editing. A preparation failure releases or blocks the attempt with a reason; it does not leave an unreported reservation. Lease renewal can continue during preparation.

A checkpoint includes a short summary, current action, progress since the last checkpoint, next step, blockers, and current branch/revision/job references when relevant. Heartbeat requests need not fabricate a new progress summary. Save health time and progress time independently.

Explicit relinquishment records a final checkpoint and an outcome. A request to cancel an active task revokes service authority but retains jobs/resource holds until their actual disposition is known. Operators see the distinction between canceled work and a still-running external job.

Submission, review, and completion

Submitting a code attempt creates an immutable candidate containing the task revision, policy revision, repository/base/candidate revisions, result summary, acceptance evidence, handoff, and linked/new lessons. Artifact references include their accessibility. The submission ends the implementation attempt and admits the required review activities atomically.

Reviews become eligible from this candidate, not from the code task’s done state. This avoids a dependency cycle. Review tasks use their own ownership, candidate binding, and attribution. A completed review task means a decision was recorded; its decision may be approval or changes requested. The code task uses the decision as a guard, not merely the review task’s terminal state.

The review modes are none, agent, human, and both, with agent as the selected default. Propose one required approval of each enabled type initially. Human decisions require human authentication; agent review uses the separation rules in the coordination contract.

A changes-requested decision makes a revision attempt eligible, subject to the project’s retry/escalation rules. It does not reopen the completed review task. A new code submission supersedes the old candidate and creates its own required activities; old approvals never transfer automatically. Pending obsolete activities are canceled, while active ones lose authority and retain history.

After the required approvals, a linked integration task becomes eligible under the selected integration-authorization policy. It reserves the canonical target, records the observed target and candidate, and prepares the intended result. Conflicts, a moved target, incomplete checks, and uncertain publication outcomes have distinct blockers and recovery instructions.

The integration result records its base, resulting commit/tree, publication observation, verification roster, check outcomes, and reporting actor. Required checks must apply to the exact integrated tree and configured environment. Pre-publication testing can count when tree equivalence is verified; checks whose semantics require a published/deployed state must actually run there.

The final transaction verifies the current candidate and required policy, applicable approvals, integration evidence, required checks, and ownership. It completes the integration activity and code task together, records the final handoff, and reevaluates downstream readiness. Publishing code without satisfying these checks leaves visible incomplete work. It does not automatically revert the target or authorize deployment.

General tasks complete from their own acceptance evidence and configured review, without artificial Git requirements. Objectives complete from their required children and criteria. They have no implementation worktree of their own.

Recovery and late information

Recovery is an attempt mode on the affected task, not permission to overwrite the old attempt. Claiming recovery is atomic and uses a new generation. In manual projects, an operator must release the recovery hold first. Inspection can read all retained evidence but cannot bypass reservations held by uncertain jobs.

Record one recovery disposition: resume accessible work, reconcile an already delivered result, restart with an explanation, or remain blocked. Reconciliation must satisfy the same completion guards as normal work. A commit found in Git does not itself establish passed review or verification.

Support append-only late notes and job observations from their authorized reporters after an attempt expires. They are clearly historical/unaccepted and cannot renew ownership, approve the current candidate, release another owner’s resources, or complete the task. A current recovery owner may explicitly adopt applicable evidence after verification.

Changes to requirements or policy during work

Descriptions, acceptance criteria, dependency sets, and project policy use revision checks. Cosmetic changes preserve acceptance applicability when marked as such; substantive changes require explicit reconciliation. An agent cannot erase or weaken acceptance criteria to make its own submission pass.

A project may explicitly authorize agents to change binding project rules without human approval. Each change checks the existing delegation, requires the expected policy revision, and records rationale and actor. Delegation and service credential administration are separate from editable rule text: changing a rule cannot grant its author a new administrative identity or fabricate human approval.

A new policy revision prevents new claims/submissions from silently following the old policy. Existing attempts receive a policy-changed next action and can checkpoint or relinquish safely; they must reconcile before taking a newly restricted action. Historical approvals remain tied to their original policy. The service records the authorized actor’s resolution instead of silently grandfathering or retroactively rewriting every in-progress task.

Proposed HTTP and CLI contract

Status: design, not an implemented API. Names and examples below are the proposed first-release interface. Implementation will publish matching OpenAPI schemas, CLI help, and service-delivered agent instructions from shared contract types.

Conventions

Use /api/v1 for the JSON API over HTTPS. IDs are opaque strings. Project IDs are explicit in project operations; there is no global current project. Dates are UTC RFC 3339 values; revision and ownership-generation fields are integers. Pagination uses opaque cursors with a proposed default of 50 and maximum of 200 items. Responses include request_id and server_time.

server_time is the UTC wall-clock observation when the response is emitted, for correlation and display. It can be behind protected coordinator time during a clock incident; never calculate lease authority as expires_at - server_time. Use server-computed lease_remaining_ms and renew_after_seconds, subtract local monotonic request elapsed time and a safety margin, and honor explicit authority validity flags. See clock safety.

Authenticate agents with Authorization: Bearer <agent-token>. Browser sessions use the cookie/CSRF contract in onboarding-contract.md. Never put authentication or session secrets in query strings. Resolve authentication before disclosing whether a requested project/record exists.

Every retryable mutation requires an Idempotency-Key generated and persisted before the first request. The key is scoped to principal, method, and operation path. Its stored request fingerprint includes relevant input, expected revisions, session identity, and a verifier for any submitted secret, never raw secrets. Reusing the key with different input is a conflict. Receipt access still checks current authentication and authorization.

Propose retaining full mutation receipts for at least 30 days, plus compact principal/operation/key/fingerprint tombstones after that. A retry after receipt expiry returns idempotency_receipt_expired and reconciliation instructions; it cannot execute the old operation as new. A principal’s tombstones can be removed when the principal is permanently retired and all credentials revoked.

Use these HTTP/error families:

An error body contains error.code, a concise error.message, structured error.details, error.next_actions, and retryable. Never recommend blind retries for expired authority, denied permission, or missing verification.

Public help, credentials, and sessions

GET /api/v1/info returns product/API/instruction versions and public setup help. GET /api/v1/help/authentication returns human-readable enrollment steps and non-secret configuration examples. Neither returns private project information. An unauthenticated private request returns the same useful help reference:

{
  "request_id": "request-example",
  "server_time": "2026-09-09T18:00:00Z",
  "error": {
    "code": "authentication_required",
    "message": "Configure this workstation's agent credential, then reconnect.",
    "details": {"help_path": "/api/v1/help/authentication"},
    "next_actions": [{"action": "show_operator_setup_help"}],
    "retryable": false
  }
}

Browser sign-in/out and password-change endpoints operate on local human accounts. Admin endpoints manage principals and issue/revoke tokens. First-admin creation and account recovery use a host-local command, never public enrollment. All issued agent tokens have agent identity; an agent token cannot record a human approval even when a human operator created it.

Credential issuance displays a newly generated token once. Its mutation receipt retains the issued credential ID and metadata, never a replayable token value. If the issuing response is lost, replay returns the original issuance identity with secret_unavailable and instructions to revoke that unused credential and issue a replacement. It cannot silently create a second token or claim the old secret can be recovered from its verifier.

For a harness session, the client generates and saves a session ID and a random session proof before POST /api/v1/sessions. Send the proof in X-Coordinator-Session-Proof; store only its verifier on the service. Return the session ID and state without echoing the proof. Identical retries can recover the session without a stored replayable secret. The session records its principal, issuing credential, workstation, harness label/version, and declared capabilities.

Subsequent ownership operations send X-Coordinator-Session and the proof header in addition to the agent token. A session belongs to its issuing credential; revocation invalidates its authority. GET /api/v1/sessions/{id} reconciles it; POST /api/v1/sessions/{id}/close closes it explicitly. Compaction and ordinary turn completion do not close it or create another session automatically.

The CLI implements this as agent-coordinator connect, returning orientation and current ownership. It reads repository binding and protected local session state. First connection creates a session; resume reconciles the saved one. The underlying HTTP workflow remains fully documented for clients without the CLI.

Orientation and claiming

GET /api/v1/projects lists all projects for every authenticated principal. GET /api/v1/projects/{project_id}/orientation returns current rules, policy and instruction versions, session work, recovery candidates, blockers/decisions, relevant lessons, candidate tasks, and next actions. Required rules may require pagination; instructions_complete: false prevents new work until acknowledged. POST /api/v1/sessions/{id}/instruction-acknowledgments records the project, policy/instruction revisions, and required section IDs the client has received and read. Claims require the current complete acknowledgment. This establishes protocol acknowledgment, not proof that a model understood the prose.

POST /api/v1/projects/{project_id}/claims accepts exactly one task ID or a next-eligible selector. For an explicit task, include its expected revision. For next-eligible work, supply permitted kinds, capabilities, and the policy/ instruction versions read. The server applies the same eligibility checks to both forms. mode is work or recovery.

{
  "task_id": "task-example",
  "expected_task_revision": 4,
  "mode": "work",
  "policy_revision": 3,
  "instruction_version": "1"
}

A successful claim returns the task, attempt ID, ownership generation, server lease deadline, remaining lease duration sampled before sending the response, renewal recommendation, and worktree/resource preparation steps. The client subtracts elapsed monotonic request time and a safety margin from the returned remaining duration. It does not assume its wall clock matches the service.

A specific competing claim returns 409 with current status and alternatives. Next-eligible selection with no available work returns 200 with claim: null, structured queue reasons, and a suggested next check. An empty queue is normal. Retrying a successful claim retrieves its original receipt and current authority status; it cannot return a different task as a substitute.

CLI equivalents are tasks list, claim --task <id> --revision <n>, and claim --next, with explicit project binding and JSON output available.

Attempt operations

All paths below are under /api/v1/projects/{project_id}. Attempts have their own IDs and ownership generation; ownership-dependent input must name the expected generation. Authorized historical reads/late notes do not require a live ownership grant. Human review uses browser authentication and CSRF with a human-owned review attempt; it does not require an agent-token session proof.

Checkpointing does not implicitly renew ownership. The CLI may explicitly perform both operations and must report each outcome. This avoids a retry of an old checkpoint appearing to grant a fresh lease.

Submitting code includes a candidate repository/base/commit/tree identity and check evidence. Review submission names the immutable reviewed submission and decision. Integration submission names the target before/after revisions, publication observation, and verification of the resulting tree. Shared domain guards reject content that is inappropriate for the task kind or obsolete input.

Jobs, artifacts, knowledge, and decisions

Job observations have a separate authorized reporter identity and monotonically increasing observation sequence. They do not require a still-active implementation lease, but do require current reporter authorization. Late running observations cannot overwrite a terminal result. Corrections to erroneous terminal reports are explicit attributed amendments, not silent history replacement.

Task submission may reference only finalized artifacts. Knowledge updates made independently use revision checks; new lessons included in a task submission are stored atomically with that submission. Import apply must detect service changes since its preview rather than overwriting them.

CLI behavior and contract checks

Provide JSON input through --input <file> or standard input and JSON output through --json. Native PowerShell and Linux-shell examples use the same payload files. Keep API tokens/session proofs in the protected client configuration or explicit process environment, not positional command-line arguments or output.

Use exit 0 for success, 2 for invalid input, 3 for authentication/setup required, 4 for denied operations, 5 for state/ownership conflicts, 6 for unsatisfied requirements, and 7 for temporary transport/service failures. JSON includes the stable API error code and remedy. The CLI never generates a new mutation key merely because a response was lost.

Reporter credentials are subordinate to the issuing principal/agent credential. They permit only the named observations and, when explicitly delegated, bounded lease renewal. They cannot create sessions or extend their own scope/deadline. Revoking the parent credential revokes reporters. Attempt expiry stops delegated renewal; appropriately authorized job observations may continue without reactivating the expired attempt. Use a separate bearer credential namespace so reporter credentials cannot be mistaken for full agent API tokens.

Acceptance includes a complete workflow performed once through the CLI and once through direct HTTP: connect, read instructions, claim, register checkout, checkpoint/renew, report a job, submit, review, integrate, retrieve lessons. Exercise lost responses, two-session ownership isolation, revocation, expiry, stale policy, no-ready-work, and a reviewer who contributed to the candidate. Every advertised next action and CLI command must match shipped schemas/help.

Implementation structure and release acceptance

Status: preserved implementation and acceptance specification. Product choices remain authoritative in PLAN.md. The runtime and acceptance suite now exist; implementation status records what shipped and where the current interface differs from this design-stage chapter.

The accepted target is 20 projects, 50 simultaneous agent sessions, and 100,000 historical tasks. The production host is undecided; use Ubuntu 24.04 LTS on x86_64 with 2 CPU cores and 4 GB RAM as the engineering test baseline. Ubuntu lists standard security maintenance for 24.04 LTS through May 2029 in its release lifecycle.

Service boundaries

flowchart LR
  Browser[Operator browser] -->|HTTPS| Proxy[Reverse proxy]
  CLI[Linux or Windows CLI] -->|HTTPS| Proxy
  Agent[Direct HTTP agent] -->|HTTPS| Proxy
  Proxy --> Server[Rust Axum service]
  Server --> DB[(Local SQLite)]
  Server --> Files[Bounded artifact storage]
  CLI --> Local[Local worktrees and job observations]
  Local --> Git[Git remote]

Use one Rust workspace with a domain/types library, server, reusable HTTP client, and CLI/local-reporter executable. Keep ownership and workflow validation in shared server-side application functions called by every HTTP surface. The CLI does not have authority to bypass those functions. Serve embedded vanilla JS/CSS assets from Axum; no Node process is needed on the installed server.

Proposed implementation libraries are Tokio/Axum, SQLx with SQLite migrations, Serde, a Rust TLS HTTP client, Clap, tracing, Argon2id, and operating-system cryptographic randomness. Verify supported versions and advisories when coding, commit Cargo.lock, and keep dependency choices within these responsibilities. No agent SDK, model API, vector database, queue service, or Git-hosting SDK is required. Git operations occur on workstations under existing Git credentials.

Relational model and invariants

Use ordinary relational records and append-only audit events, not a requirement to rebuild the whole service through event replay. Store timestamps as UTC epoch milliseconds internally; present RFC 3339 dates through the API. Use JSON only for bounded structured content, not for identities or relationships requiring foreign-key/uniqueness guarantees.

Enable foreign-key enforcement on every connection. Project-owned records use composite (project_id, id) references where needed to prevent attaching an attempt or evidence to another project’s task accidentally. Explicit common knowledge references and canonical shared resources retain their own provenance. All authenticated callers can still deliberately access every project. See SQLite foreign-key documentation.

Use a partial unique index for one attempt in stored active state per task. Time is not part of that index predicate. An expired attempt is retired before the recovery attempt is inserted, in the same transaction; every authoritative operation separately checks its deadline. Keep the task pointer and generation consistent in that transaction. See SQLite partial indexes.

Index the project/eligibility/priority/ready-time task queue, dependencies by both endpoints, attempts by task and expiry, events by project/sequence, and jobs by workstation/state. Use SQLite FTS5 for current searchable task/knowledge text, updated transactionally alongside its source record. Treat index rebuild as maintenance, without losing provenance or becoming a second authoritative store. See SQLite FTS5.

Transaction boundaries

Use WAL on local storage, bounded connection pools, bounded write-lock waits, and short explicit write transactions. Coordination mutations begin with BEGIN IMMEDIATE; check ownership time after obtaining the writer lock. On an error, roll back the whole application transaction. A failed operation must not accidentally commit earlier statements. Never hold a transaction while uploading bytes, running Git/tests, waiting for humans, or making HTTP requests. These choices follow SQLite transaction behavior.

For imports too large for one short transaction, apply explicitly identified chunks. A preview lists chunk boundaries, each committed chunk is resumable, and the result reports partial completion honestly. Do not advertise all-or-none atomicity for an arbitrarily large import.

Handle capacity limits using the writer transaction, not a preflight-only count. Use an injectable service clock for expiry tests and monitor backward time jumps. After a detected material clock anomaly, pause new authority and reconcile live deadlines rather than extending leases implicitly. Restarts use persisted server deadlines; restore uses a new authority epoch.

Operation permissions

These are engineering defaults implementing all-project access and configurable autonomy. They do not introduce project visibility grants.

Agent task edits cannot seize another attempt. Closing or superseding unowned work requires revision checks, rationale, and evidence where relevant; active work or uncertain side effects require recovery or an explicit human override. Keep human override distinct from evidence-based completion: it may cancel or resolve authority, but cannot manufacture a test pass or a review record.

Human review uses the same candidate/check/revision guards without requiring the person to impersonate an agent session. Browser authentication plus CSRF protects its claim/decision operation. Do not leave browser review unusable by requiring agent-only authentication headers on every mutation.

Packaging and bounded operation

Produce native x86_64 Linux server/CLI releases and a native x86_64 Windows CLI/reporter release as the initial engineering baseline; document source builds for others until tested release targets are added. The installation includes a systemd unit, configuration examples, proxy example, backup timer, migrations, and an operator recovery guide. It must not change existing project hooks.

Engineering starting limits: 1 MiB JSON requests, 16 MiB per uploaded artifact, 10 GiB aggregate live artifact quota, and 50/200 default/max page size. Make storage/request limits explicit configuration; reject excess before exhausting memory/disk. Stream artifact bytes. Keep a disk-free reserve and give actionable storage errors. Deployment disk sizing must include retained backups separately from the live artifact quota.

Retain task/handoff/lesson history by default; archive it for normal browsing without erasing closure/import identities. Propose a 90-day artifact retention default, with explicit pinning for selected evidence and quota accounting. Keep per-job/per-attempt first/last observation and progress; avoid an ever-growing full JSON event for every unchanged one-minute heartbeat. Periodic health rollups and terminal/checkpoint events preserve useful history. Semantic state changes retain their audit records and provenance.

Expose minimal public liveness and authenticated diagnostics. Record request IDs, operation latency, failed authentication totals, lock waits, lease/recovery state, artifact usage, backup freshness, and disk capacity without credentials or raw task prose in ordinary request logs. Use a five-second dashboard refresh default, with visible freshness and an attention queue; realtime push is unnecessary for first-release correctness.

Acceptance gates

Use deterministic clocks and synchronized real SQLite transactions for protocol tests. Add process-level integration tests where shutdowns, local producer state, or persistence matter. Test native Windows behavior on Windows, not merely with Windows-looking strings on Linux. Sanitize reference-project fixtures; do not upload production histories, run project hooks, or touch live Git/SQL targets as part of automated tests.

Benchmark on the baseline with 20 projects, 100,000 historical tasks, and 50 simultaneous agent sessions. Engineering workload target: 50 API requests/second, 40 reads and 10 writes, for 30 minutes; burst 100 simultaneous claims; include a concurrent artifact upload and backup. Target p95 under 500 ms for ordinary metadata operations, no unexpected server errors or incorrect ownership, and bounded memory under the machine’s capacity. These are acceptance targets, not claims of measured performance. Report latency and resource measurements.

The final two-workstation exercise uses isolated sample repositories: create two projects; race claims; publish and recover checkpoints; keep an observed long job alive across session loss; review an exact candidate; serialize integration; verify downstream readiness and shared lessons; restore a backup and reconcile authority. No customer repository is needed to prove the protocol.

Agent and operator onboarding contract

Status: mixed implementation and release design. Authentication, native session connection, task coordination, the repository bootstrap, service-authoritative records, project access, local human accounts, and revocable agent tokens are implemented. Linux systemd packaging, trusted-HTTPS installation, backups, and restore controls have passed the disposable release exercises. An actual production host, domain, and off-server backup destination remain operator installation choices. Automated CI validates Linux and native Windows packages; the final operator-initiated physical Windows workstation exercise remains pending acceptance evidence. Operation roles are specified in implementation-spec.md. Use the CLI guide for implemented commands.

Repository binding and workstation credentials

Keep a small, non-secret .agent-coordinator.toml in each repository. It identifies the service URL and stable project ID. It must not contain an API credential, an operator password, or a session secret. Separate worktrees inherit the same project binding; their paths and branches identify their distinct checkouts.

Credentials live in workstation configuration outside the repository, or are supplied through the process environment. The implementation must choose and document the supported secure storage mechanism for each selected client OS. The agent receives credential references and setup status, not secret values in its orientation text. Direct HTTP clients can use locally configured credentials without depending on the CLI or an agent-vendor integration.

A credential is bound locally to its trusted service origin. Editing a repository binding cannot redirect an existing credential to another host, and the HTTP client must not forward credentials across an origin-changing redirect. Project bindings help route requests; the service still performs authorization itself.

Public HTTPS deployment

The public API and browser interface use HTTPS. The selected native Linux deployment runs Axum under systemd. Terminate TLS at a reverse proxy and keep the Axum listener private. Use Caddy for the documented baseline; an existing proxy can use the same HTTP upstream contract. Trust forwarded scheme/client headers only from configured proxies. Clients validate the service certificate; the documented normal workflow must not require disabling certificate checks.

Credentials travel in authentication headers or the login request body, never URLs. The service redacts credentials from logs, limits request sizes and request rates, and validates allowed operations on the server. These choices follow OWASP’s REST security guidance.

For the selected local password accounts, use salted Argon2id password hashes with an implementation-time work-factor review, following OWASP’s password-storage guidance. For browser sessions, the proposal is server-managed sessions using Secure, HttpOnly, explicitly SameSite cookies with CSRF protection and server-side revocation. Do not store long-lived agent tokens in browser local storage. See OWASP’s session guidance.

Task and lesson content renders as text or sanitized Markdown, with raw active HTML disabled. Public setup help provides orientation, not remote execution. Service-internal management actions require the selected administrative role, even though all authenticated callers can access all projects.

Native Linux installation proposal

Ship a versioned release containing the Rust server, CLI, database migrations, embedded browser assets, a systemd unit, and example proxy configuration. The installed service should not require a source checkout or frontend build tools. Run it under a dedicated unprivileged service account; keep configuration under /etc/agent-coordinator and writable data under /var/lib/agent-coordinator. Use a local filesystem for SQLite. One active service process owns coordination; this first-release design does not include active/active server replication.

Use Caddy as the documented default proxy, while retaining a standard HTTP upstream for an existing alternative. Caddy can obtain and renew certificates for an appropriately configured public hostname; DNS, network reachability, and persistent certificate storage are deployment prerequisites. See the official automatic HTTPS documentation. The final hostname and production Linux distribution are installation inputs, not project IDs or assumptions embedded in application code. The test baseline is Ubuntu 24.04 LTS on x86_64 with 2 CPU cores and 4 GB RAM.

Provide a database-aware backup command suitable for a systemd timer, using SQLite’s supported online snapshot facilities. Do not document copying only a live database file while ignoring its WAL. The operator selected hourly backups, 24 hourly and 30 daily retained copies, documented off-server copying, and a one-hour restore target. The selected log/report uploads require the backup manifest to cover their storage and referenced digests as well as database records.

An upgrade checks schema compatibility, creates a verified backup, applies migrations under exclusive maintenance access, and verifies readiness before accepting work. Running jobs remain on workstations; service maintenance cannot declare them stopped. Document the expected coordination outage and lease recovery behavior. Do not automatically roll back the binary across an incompatible database migration.

Restoring an older backup is an explicit maintenance operation. It must invalidate pre-restore sessions and attempt authority, preserve uncertain-job resource holds, and require credential/account reconciliation before public access resumes. Restoring old authentication tables must not silently reactivate a subsequently revoked credential or account. Use the restore procedure below, including a new authority epoch and host-local account/credential recovery.

Backup and restore procedure

The systemd backup timer starts an hourly snapshot job under a single-job lock. The command creates a consistent SQLite backup and an artifact manifest listing the finalized files referenced by that snapshot. Preserve referenced immutable artifact bytes while the snapshot is being assembled: artifact deletion/garbage collection must respect the active backup hold. Do not keep a database writer transaction open while copying files.

Store a complete immutable artifact copy within each snapshot directory. The first release favors independently verifiable and transferable bundles; shared blob deduplication is deferred and storage sizing must include full retained copies. Each snapshot has its own database image, manifest, schema/service versions, timestamps, digests, and completion marker. Only publish the completion marker after database integrity and all referenced file digests are verified. A partial snapshot is never counted as a usable backup. Retain the newest snapshot from each of the most recent 24 hourly buckets plus one successful snapshot for each of the most recent 30 days. Prune only complete snapshot directories outside the retention set. Insufficient space or a failed backup leaves prior usable snapshots intact and raises a visible operator alert.

Document copying the completed backup repository to an operator-selected server or storage destination, with authentication/encryption supplied by that transfer mechanism. Include manifests, database images, and referenced blobs. Copying only the manifest is insufficient. Track local snapshot time separately from the operator’s verified off-server copy time. Hourly local backups do not establish one-hour data-loss protection against host loss unless off-server copying also meets that schedule. The destination and its credentials are installation inputs.

The restore procedure is:

1. Stop the service and keep public access in maintenance mode. Preserve the current damaged state separately; select a completed compatible snapshot.
2. Verify its database and artifact digests, then restore into a staging data directory with the service account’s permissions. Reject incomplete snapshots.
3. Generate a fresh authority epoch. In the restored database, revoke all agent tokens/reporters/browser sessions, invalidate agent sessions and attempts, and suspend restored human accounts pending reconciliation. Treat known in-flight jobs/resources as uncertain, retaining their recovery holds.
4. A host-local recovery command establishes a fresh administrator credential. That administrator explicitly reconciles people and issues fresh agent credentials. Never re-enable a restored password/token just because its historical record predates a revocation.
5. Check schema/readiness, atomically promote the staged data directory while the service is stopped, start the service, and verify new authentication and old-credential rejection before restoring public access.
6. Agents reconnect with fresh credentials, inspect saved work and jobs, and use the normal recovery workflow. Reconcile actual Git targets before admitting conflicting integration work; a restored database can lag external effects.

Measure the one-hour restore target from starting this documented procedure on an available compatible host with access to a completed backup and the operator’s host credentials, through restored service availability and one recovered client. Include data verification/copying and credential recovery in the exercise. New server procurement and recovery of unavailable off-server storage are external dependencies; record them explicitly in a real incident. Restoring task data does not promise that all formerly running workstation jobs finish within an hour.

Ordinary service restarts do not rotate the restore epoch or reset credentials. They preserve stored lease deadlines; elapsed downtime can cause normal expiry.

First connection without authentication

The service returns a consistent authentication-required response, including when an unauthenticated caller names an unknown project. It supplies:

• A stable error code and supported protocol version.
• A short human-readable explanation that authentication is missing or invalid.
• Public setup instructions and a same-service help location.
• The configuration fields needed, using placeholders rather than credentials.
• An explicit next action: show the operator the instructions, then reconnect after local credential configuration has been completed.

Example human message:

This workstation is not authenticated with Agent Coordinator. Open this service’s setup page, sign in with an operator account, and register the workstation. Configure the issued credential in the local client, then retry the connection. Keep the credential out of this conversation and repository.

Initial administrator setup uses the host-local installation command. People use local password accounts. Public help must work before login, but cannot include project names, users, tasks, permission grants, or credential values. Authentication failure is not an instruction to continue selecting work offline.

Operator setup flow

Proposed normal sequence:

1. Set up the service and its first administrator using the selected installation method. Initial administrator creation must not be an unauthenticated public operation available after setup.
2. Create a project and assign its stable ID and repository binding.
3. Configure project workflow and completion requirements.
4. Register a workstation/agent credential. It can access all projects; its operation role determines any administrative privileges separately.
5. Configure that credential locally using the supported client mechanism.
6. Generate the non-secret repository binding and short orientation snippet.
7. Verify connection and project visibility before importing or claiming work.

Credential rotation and revocation must be available without editing committed repository instructions. Failure to connect, insufficient permission for an operation, and a missing/mismatched project binding have distinct remedies.

Proposed identity and credential lifecycle

Keep a principal’s identity separate from its credentials. Revoking a token must preserve attribution on tasks, reviews, and lessons. Multiple tokens may belong to one agent principal, with workstation labels and individual revocation; a new token alone does not establish reviewer independence. A review records its actual principal and session separately from the credential used to authenticate.

An administrator creates people and agent principals. There is no public self-registration. Agent credentials cannot create administrators, issue their own replacement tokens, or represent a human decision. All principals retain access to every project; these proposed limits concern operations, not project visibility. Projects may separately delegate binding-rule changes to agents; that delegation does not confer credential-administration privileges.

Generate agent tokens with cryptographic randomness, display them only when issued, and persist a verifier rather than the original token. Keep a non-secret credential ID, issuer, created/last-used timestamps, optional expiry, and revocation timestamp for management. Never return the verifier through the API. Rotation creates a replacement credential; the operator may then revoke the old one. Revocation takes effect on subsequent authenticated operations, including retries of previously successful requests.

Distinguish the agent credential, the harness session, and the task attempt. Two sessions using one workstation credential must not accidentally share task authority. A session needs a locally protected resume credential, and ownership-dependent requests must authenticate that session as well as its principal. Compaction/resume reuses session identity; starting another harness creates a new session. The wire format is specified in api-contract.md.

Credential revocation invalidates dependent session authority. It does not prove that a local job stopped or release an uncertain shared resource. The task enters the selected recovery process, retaining its checkpoints and job observations. A replacement credential can discover recoverable work, but cannot silently revive an expired attempt.

Initial-admin creation and lost-password recovery use a documented command on the service host in the first-release proposal. This avoids requiring email delivery as infrastructure. The browser supports password changes, session revocation, and token issuance/revocation for authorized people. Password changes invalidate existing browser sessions; agent-token revocation is a separate, explicit operation.

Linux and native Windows client contract

Distribute native client binaries. Windows users must not need WSL, Bash, or a Unix process supervisor to connect, claim work, or report a job. Git operations use the locally installed Git executable and existing authentication. The coordinator never needs the workstation’s Git private keys on its server.

Give every substantive CLI operation a JSON output mode and a way to read structured input from a file or standard input. Document both PowerShell and Linux-shell examples without making correctness depend on complex shell quoting. Failures have stable exit codes and structured remedies; never turn a failed child command into a successful job report.

Keep credential and session files outside repositories. Use the selected OS credential facility or explicitly protected local storage, including permissions on Linux and ACLs on Windows. Headless Linux installations must have a supported noninteractive mechanism. Do not require desktop keyring prompts during a task.

Identify a checkout using workstation identity and resolved Git/worktree metadata. Paths are displayed in the workstation’s native form, including Windows drive letters and spaces. Path strings alone cannot prove two checkouts are separate. Keep process instance identity distinct from its numeric PID so PID reuse cannot make a new process appear to be an old running job.

The local reporter observes explicitly registered local jobs and reconnects to their durable records. It does not accept remote shell-execution requests. A platform-specific observer that cannot determine process state reports unknown, retains its last evidence, and requests reconciliation rather than inventing a terminal status.

Authenticated orientation

A successful connection returns a bounded orientation packet with:

Do not dump full archives or every project’s memory into the response. Optional history can be summarized or paginated with provenance. Required policy cannot be silently truncated: if more required instructions must be read, mark the orientation incomplete and supply continuation steps before a new claim.

Repeated connect/resume/compaction must not create duplicate attempts. Connection does not itself select work or authorize publishing, deployment, or data changes. An authenticated user lacking permission for an administrative or ownership- dependent operation receives a permission remedy, not a fresh enrollment loop. Every authenticated user can select and access any project.

Proposed short AGENTS.md / CLAUDE.md snippet

After installing the native CLI and configuring the repository binding and an origin-bound workstation credential, use this snippet in AGENTS.md or CLAUDE.md. Replace UNIQUE_HARNESS_NAME with a name unique to this harness on this workstation; keep it stable when resuming and pass it on every command.

This project coordinates work through Agent Coordinator.
Read .agent-coordinator.toml. Select a unique, stable name for this harness.
Run `agent-coordinator --session UNIQUE_HARNESS_NAME connect`; use that
same --session value on every following command. Connection reserves no task.
Follow the returned workflow alongside this repository's applicable rules.
Claim a task before changing code; use its separate worktree.
Report progress and renew ownership as instructed; submit results and lessons.
If authentication is required, show the setup message to the human operator.
If ownership expires, stop changing the task and follow the recovery steps.

The generated service help must also provide complete HTTP examples for agents without the CLI. Examples cover connect, claim, checkpoint, renew, observe a job, submit, release, retrieve lessons, and resolve a conflict. They use explicit field names and long CLI flags, require no vendor-specific tools, and interpolate credentials locally without displaying their values.

Acceptance scenarios

• Starting with only the repository snippet and a preconfigured credential, an agent can connect, claim, checkpoint, submit, and retrieve relevant lessons.
• An unconfigured workstation receives setup help without private project data.
• A configured workstation can select any project. A refused administrative or ownership-dependent operation gets a permission-specific remedy.
• Revoking or rotating a credential does not require a repository change.
• Moving a checkout to another path does not lose its project or shared memory.
• A changed service URL or redirect cannot receive another origin’s credential.
• A repeated connection restores the current attempt instead of claiming again.
• A short context response retains required policy or explicitly requires the agent to fetch the remaining instructions before taking new work.
• The same workflow succeeds through documented HTTP calls without a CLI, hook, MCP server, or agent-vendor account.
• Public deployment rejects insecure credential transport; untrusted forwarded headers cannot impersonate the trusted HTTPS proxy or bypass request limits.
• Browser content cannot execute stored task/lesson HTML, and cookie-authenticated mutations require the configured CSRF protection.

Repository and hook review

Reviewed locally on 2026-09-09 to refine the coordination-service plan. Recommendations below are design proposals, not new authorization to change either reference project or to execute its workflows.

Evidence and limits

This was a document, configuration, script-source, and local Git review. No reference-project tests or hooks were executed, and no remote fetch was needed. Historical test reports are evidence of what was reported at their recorded revision; this review does not certify either project’s current runtime behavior. The 38,684-line SithBit archive was searched and sampled, not read end to end.

Submission has no root CLAUDE.md, HANDOFF.md, BACKLOG.md, DURABLE-RECORD.md, or HANDOFF-archive.md in the inspected checkout. Its equivalent records use other paths. No Submission-specific memory directory was found among the inspected local Claude project directories. This does not establish that another machine or harness has no such memory.

The records demonstrate local parallel work and cross-workstation knowledge drift. They are not evidence that a distributed task-claim protocol already exists; the proposed service still needs to supply that missing coordination.

Findings that change the design

E1. Real work already has phases, reviewers, and integration

SithBit’s current plan has parallel groups of work, ordered phases, acceptance criteria, required capabilities, shared files, and a dependency-driven exception: the final health-listener tests wait for two other groups to finish because they build the same crates. Its overnight task ledger distinguishes building, verifying, gating, landed, parked, blocked, and done. The branch gate is followed by a separate gate after integration. Submission likewise records phase checkpoints and a separately authorized integration step.

Evidence: SithBit plan: /home/marshall/src/solana/sithbit/HANDOFF.md:161, overnight ledger: /home/marshall/src/solana/sithbit/.claude/overnight-tasks.md:1, Submission phases: /home/marshall/src/ids/submission/EmarsModern/docs/document-scripts/IMPLEMENTATION-PLAN.md:94. SithBit commit c2f593a3 introduced the worktree-aware overnight orchestration; 970258f3 and 55539f0d are recent integration commits.

Revision: support parent tasks, ordered dependencies, execution attempts, review records, and integration evidence. An implementation can finish before its parent objective is complete. Keep this a small typed model; a general user-programmable workflow engine is not needed to represent these examples.

E2. Finished work is repeatedly rediscovered as open work

SithBit explicitly preserves do-not-requeue records because copying a list of spawned candidates instead of consumed candidates resurrected completed work. An archived session found backlog item 26 already implemented three waves earlier and corrected the record without rebuilding it. Current memory requires checking Git history, symbols, and acceptance criteria before implementing a backlog item. The same numeric labels also recur in separate session contexts.

Evidence: archive closure records: /home/marshall/src/solana/sithbit/HANDOFF-archive.md:36069, item 26 correction: /home/marshall/src/solana/sithbit/HANDOFF-archive.md:36774, selection feedback: /home/marshall/.claude/projects/-home-marshall-src-solana-sithbit/memory/backlog-item-prove-undone.md, backlog warning: /home/marshall/src/solana/sithbit/BACKLOG.md:2201.

Revision: stable task identities, source-scoped import identities, durable closure/duplicate/supersession links, and a recorded pre-implementation check. Archive and export operations must never make completed work eligible again. An agent finding an already-delivered result should reconcile the task with evidence, not manufacture another implementation attempt’s code changes.

E3. Session prose can be valid historically and misleading now

Submission’s AGENTS.md says repository-level Git history is unavailable, although the current checkout has 73 reachable commits. Its migration handoff describes work remaining on a feature branch. Later merge commits d0d3cee and fb3ef17 record integration, and ancestry checks confirm 3b2dbde and 171d8d8 are ancestors of the current HEAD. context/RESUME.md still displays July 20 as its update date despite a later edit in 3ccb46b and later implementation records.

Evidence: history statement: /home/marshall/src/ids/submission/AGENTS.md:34, branch-specific handoff: /home/marshall/src/ids/submission/EmarsModern/docs/document-scripts/BUILDING-GROUP-MIGRATION-HANDOFF.md:3, resume record: /home/marshall/src/ids/submission/context/RESUME.md:3.

Revision: source revision, observation time, applicable branch/environment, and supersession status belong on imported claims. Current state is a projection of explicit events, not whichever prose paragraph appears first. Historical permission to work on a branch does not become authority for future merges.

E4. Knowledge is broader than a durable-record file

SithBit uses SAVERS.md for practical discoveries, DURABLE-RECORD.md for standing rules and decisions, HANDOFF.md for current context, and the archive for history. Its root CLAUDE.md explicitly directs a fresh session to load those selectively. Pruning first extracts reusable facts; relocating a narrative must not erase the rule it taught. The current HANDOFF.md alone is 3,587 lines.

Evidence: handoff organization: /home/marshall/src/solana/sithbit/HANDOFF.md:3, SAVERS.md: /home/marshall/src/solana/sithbit/SAVERS.md:1, durable record: /home/marshall/src/solana/sithbit/DURABLE-RECORD.md:1. Commit 28be6495 moved twelve completed records into the archive and added extracted facts to SAVERS.md.

Revision: distinguish current checkpoints, practical lessons, decisions, standing policy, rejected approaches, and historical narratives. Return a short orientation packet and task-specific knowledge, with links to supporting history. Import must accept SAVERS.md, alternative handoff paths, and memory directories.

E5. Memory continuity currently depends on fragile local paths

The repository memory README documents a per-machine symlink strategy. On this machine, the current SithBit workstation memory directory is a real directory with a different index from the repository memory directory. The older -home-marshall-src-solana-rust/memory symlink resolves to the nonexistent /home/marshall/src/solana/rust/.claude/memory. Two targets named by the repository MEMORY.md index are also absent from that directory. These observations show separate or unresolved sources, not that their content is necessarily lost everywhere.

Evidence: memory bootstrap: /home/marshall/src/solana/sithbit/.claude/memory/README.md:7, repository index: /home/marshall/src/solana/sithbit/.claude/memory/MEMORY.md, workstation index: /home/marshall/.claude/projects/-home-marshall-src-solana-sithbit/memory/MEMORY.md.

Revision: identify projects independently of absolute paths. Import multiple memory sources with provenance and unresolved-link reporting; deduplicate without silently discarding differing versions. Keep machine/harness-specific lessons scoped, and use explicit sharing for general lessons across projects.

E6. Agent liveness, job liveness, and test success are different facts

SithBit recorded three lost approximately 25-minute gate runs, killed waiters whose gates survived, inherited file-lock handles, stale completion markers, and incorrect results inferred from human log banners. Its detached runner now uses structured run records, the producer’s exit status, and process observations. An uninterrupted test leg can legitimately produce no log output for about 400 seconds. The local memory’s older launch recipe predates this runner.

Evidence: recovery guidance: /home/marshall/src/solana/sithbit/CLAUDE.md:242, runner contract: /home/marshall/src/solana/sithbit/scripts/gate-detached.sh:1, older memory: /home/marshall/.claude/projects/-home-marshall-src-solana-sithbit/memory/gate-launch-detached.md. Commit 713b1ad7 added the detached runner and failure cases.

Revision: record external jobs independently of agent sessions. Preserve job identity across observation failures and distinguish running, finished, failed, interrupted, and unknown. Agent heartbeats, runner heartbeats, progress, and lease validity must be separately visible. Never restart or reclaim merely because a waiter disconnected or logs are quiet.

E7. Shared resources extend beyond file names

Standing rule 16 documents interference through build dependencies despite disjoint edits. The archive records a shared-tree stash temporarily reverting another builder’s twelve in-flight edits; that incident recovered without loss. The overnight ledger records separate worktree builds exhausting disk space and requiring serialized checks. Worktrees isolate editable files but do not automatically isolate disk, fixture services, build outputs, or shared branches.

Evidence: dependency interference: /home/marshall/src/solana/sithbit/DURABLE-RECORD.md:104, shared-tree incident: /home/marshall/src/solana/sithbit/HANDOFF-archive.md:20946, disk exhaustion: /home/marshall/src/solana/sithbit/.claude/overnight-tasks.md:42, resource policy: /home/marshall/src/solana/sithbit/scripts/overnight.sh:90.

Revision: record checkout identity, expected edits, and named resources with scope: workstation, checkout, or shared project/environment. Prefer isolated implementation worktrees and serialized integration. Resource admission must not serialize independent computers just because both run a command named “gate”. A disconnected process may still occupy a resource after its task lease expires; resource recovery needs its own evidence.

E8. Eligibility depends on environment and decisions

SithBit’s QRESYNC item is blocked on a published dependency release, with a local fork explicitly rejected. Other work was skipped because emulators were down. Preflight requirements cover capabilities such as working toolchains, available disk, and access to an environment. Submission’s live SQL checks depend on VPN access; its mutation checks additionally depend on designated inputs and explicit authorization. A credential existing on a workstation is insufficient evidence that it is usable for the requested operation.

Evidence: upstream blocker: /home/marshall/src/solana/sithbit/BACKLOG.md:385, preflight contract: /home/marshall/src/solana/sithbit/CLAUDE.md:253, Submission live checks: /home/marshall/src/ids/submission/EmarsModern/README.md:409, mutation scope: /home/marshall/src/ids/submission/EmarsModern/README.md:672. Upstream release facts here describe the local record; no current external release claim is made by this review.

Revision: typed blockers with reopening conditions; capability requirements and expiring observations; decisions and authorization records bound to scope. Claiming a task does not authorize database mutations, publishing, or deployment. Preserve granted authority without repeatedly asking, and require a new decision only when the actual action falls outside its recorded scope.

E9. Repeated mistakes need correction and escalation, not more prose

SithBit records accepted designs and rejections with reopening conditions. It also limits repeated disputes by defect class, because renaming a phase or rescoping work previously reset the retry count and wasted further attempts. Current handoff records distinguish a candidate new rule from an adopted rule.

Evidence: dispute rule: /home/marshall/src/solana/sithbit/DURABLE-RECORD.md:248, rejections: /home/marshall/src/solana/sithbit/DURABLE-RECORD.md:948, candidate rule: /home/marshall/src/solana/sithbit/HANDOFF.md:150.

Revision: stable finding IDs/classes, linked attempts, explicit decision records, and configurable retry/escalation limits. Lessons can be observed, validated, superseded, or rejected. Promotion into standing policy is a distinct authorized action. Retrieval can show whether a lesson helped or was corrected; this is shared operational knowledge, not a claim of model training.

Hook inventory and implications

These are registrations observed in the inspected configuration, not proof of coverage in every harness invocation. Both repositories’ default Git hook directories contain samples only; neither resolves a configured core.hooksPath.

Sources: global registrations: /home/marshall/.claude/settings.json, SithBit registrations: /home/marshall/src/solana/sithbit/.claude/settings.json, global hook scripts: /home/marshall/.claude/hooks/git-sync-on-start.sh, preflight guard: /home/marshall/src/solana/sithbit/scripts/deploy-preflight-guard.sh, formatter: /home/marshall/src/solana/sithbit/scripts/fmt-edited-rust.sh. Relevant global-config commits include 4dda2cc (Git synchronization), 62446a4 (structured gate-status check), and ea017f4 (parked decisions).

One source-level inconsistency reinforces the need for contextual recovery instructions: the synchronization hook recommends “Commit/stash” for a dirty, behind checkout, while the registered tree-revert guard blocks mutating stash commands. The service should name a recovery action compatible with the caller’s reported policy, or explain the unresolved policy conflict; it should not send an agent into a retry loop against an action its harness refuses.

Resulting scope recommendation

Keep Rust/Axum, SQLite, and the proposed lightweight web technology. The changes are primarily to the domain model, client workflow, import, and operator views.

Recommend a central coordination API, a CLI usable by any agent with shell access, and a browser interface. Existing harnesses continue launching work; optional adapters translate their lifecycle events into the same API. No vendor hook names, model choices, or shell snippets become mandatory server semantics. The coordinator must not execute imported hooks or turn retrieved prose into an executable command. Local execution remains governed by the harness/operator.

Recommend supporting both simple tasks and phased objectives in the data model, including separate review/integration tasks when policy requires them. During this review the operator confirmed separate worktrees per implementation task, with one integration step at a time into the target branch for the first release. The operator subsequently selected coordination of existing harnesses through API, CLI, and optional hooks, with local runners reporting jobs. Remote agent launch/supervision and a shared-directory execution mode are not required for the first release.

After this review, the operator also confirmed public HTTPS access, service- authoritative records with Markdown import/export, and access to every project for every authenticated person and agent. Those decisions supersede the earlier proposal for project-specific access grants. People will use local password accounts and agents will use revocable API tokens. Agents create and claim tasks autonomously, with required review configured per project. The selected server installation is native Linux under systemd behind an HTTPS reverse proxy. Expired work permits agent-driven recovery after inspecting saved work and jobs, with a per-project manual alternative. Code tasks finish after required review, target-branch integration, and validation of the integrated result.

Self-hosting review: cross-workstation work and record retirement

Reviewed on 2026-09-10 against main commit f8d8a41. The workspace test suite passed locally on that commit (182 tests). This chapter answers three questions for an agent or operator deciding how to adopt the service for this repository’s own development. It is an adoption assessment, not a new contract, and it grants no authority.

The review assumes a reliable production instance built from the current code, reachable at a well-known HTTPS URL, with a human administrator, off-server backups, and a rehearsed restore. Where a claim depends on that assumption it is labeled. Where a claim rests on retained evidence, the evidence is linked.

Question 1: Can agents on different workstations complete tasks for one project?

Yes. This is demonstrated, not only designed. With a permanent instance the only structural gap from the acceptance exercise, a disposable service behind a temporary tunnel, is closed.

Operational duties remain with the workstations and the operator:

• Source never moves through the service. Every workstation needs push and fetch access to the shared Git remote, and candidate commits must be pushed before another workstation can review or integrate them.
• Hosts should keep synchronized time. A material clock rollback pauses new authority until reconciliation. See clock safety.
• Workstation facts are client attestations. The service does not inspect a remote filesystem and is not remote attestation.

Question 2: Can the service retire HANDOFF.md and BACKLOG.md?

BACKLOG.md: yes, immediately. HANDOFF.md: yes, after adopting one convention. Neither file should be deleted until this repository’s own instructions and tooling stop reading it.

Record mapping

Details are in the knowledge contract, the import contract, and the objective contract.

The wave-level narrative convention

HANDOFF.md today holds one integrated story spanning several tasks. The service stores state per task. The substitute is:

1. One knowledge record of kind checkpoint per wave, corrected in place so its revision history is the wave history.
2. One decision per parked product question, answered by a human in the dashboard.
3. Orientation as the cold-start entry point instead of reading a file.

Service-delivered agent instructions are a versioned constant compiled into the server, not per-project editable text. Project-specific guidance belongs in the project rules or in knowledge records.

Changes required in this repository before deletion

1. AGENTS.md tells agents to read both files first and to update them after each item. Replace that with: connect, read orientation, record checkpoints, submit with a handoff and lessons.
2. The book includes both root files at build time and the documentation checks require those wrappers. Replace the includes with a committed export, or remove the chapters and the wrapper check together.
3. Harness skills that read HANDOFF.md (resume, handoff, prune, wave planning, session economics) must read the CLI export or orientation output instead.
4. DURABLE-RECORD.md can migrate to shared knowledge records the same way. It is outside the question and may stay.

Residual points

• Agents cannot apply an import. Each migration is a human dashboard action.
• A generated export is deliberately not re-importable. Once the service is authoritative, committed Markdown is a mirror and never a place to edit.

Question 3: Can this project use the service to bootstrap its own development?

Yes, with one rule: the production instance must run an accepted release package, never the working tree it coordinates. The service’s own change process then flows through the service, and every release is exercised by the project’s real records before the next release is cut.

Why it holds

• The service already supports the shape of this project’s work: code tasks with a canonical repository key, a required-check roster, independent review, and compare-and-swap publication to main.
• Upgrade compatibility is exercised. The prior executable upgraded a schema-12 database to schema 16 while preserving principals, sessions, an active attempt, and checkpoints, and old snapshots restored under the new executable. See Linux acceptance evidence.
• A schema or instruction-version change made by a task is not live until a release is installed. An instruction bump then requires every session to acknowledge the new version before its next claim, which is the intended rollout gate.

Bootstrap sequence

1. Install the accepted release on the production host with the documented systemd, HTTPS, backup, and maintenance units. Verify a backup and rehearse a restore before the first task.
2. Create the project, set its canonical repository key to this repository’s Git remote, and register at least one required check. The check is a locally launched job that runs the gate (cargo fmt --check, warnings-denied Clippy, workspace tests, scripts/smoke.py). Hosted CI is not a registered producer; it remains a second, independent gate on the pull request.
3. Commit .agent-coordinator.toml naming the service URL and project. Issue one credential per workstation; each harness chooses a stable session name.
4. Import BACKLOG.md and HANDOFF.md once through a preview, apply it as a human, and record the migration as a knowledge record.
5. Make the repository changes listed in question 2 in a task coordinated by the service. From that commit on, the root files are export mirrors or removed.
6. Continue development as ordinary service tasks. Cut releases through the existing release workflow. Upgrade the production host only from a package that passed release CI, after a fresh verified backup.

Risks specific to self-hosting

What this does not establish

No production instance exists at the time of this review, and this repository is not bound to one. The sequence above is a plan for an operator to execute. Nothing in this chapter claims a deployment, a URL, a credential, or a completed migration.

Maintaining this book

Install the pinned mdBook release and build from the repository root:

cargo install mdbook --version 0.5.4 --locked
mdbook build

Open target/book/index.html in a browser. Navigation, search, styles, and fonts are bundled with the generated book. External source and reference links still need network access. The source repository is public, but no public documentation site is deployed by this build.

Edit the maintained source

Edit chapters under book/src, and add each new chapter exactly once to book/src/SUMMARY.md. The short files under the original docs directory, root PLAN.md, and deploy/README.md direct existing readers to their maintained chapters. Keep those compatibility links when moving an established document.

AGENTS.md, BACKLOG.md, HANDOFF.md, and DURABLE-RECORD.md remain authoritative at the repository root. Their book chapters include those files directly during each build, so update the root record rather than copying its prose into a second maintained document. Each included chapter links to its original source.

Use relative chapter links and retain the distinction between current behavior and historical design. The implementation status records implemented routes and acceptance evidence. Preserve the exact accepted revision and binary identities when relocating evidence. Historical workstation file references are literal paths because those files are not shipped with the book.

Verify a change

From a source checkout, run the documentation checker:

python3 scripts/check_docs.py

It builds with the pinned version and checks chapter coverage, live-record includes, generated local links, fragments, and assets. CI runs this check as well. Inspect the changed pages in a browser, including search and narrow layouts when their content or navigation changes. A successful build alone does not prove that a link’s destination or anchor exists.

Inspect an existing Linux package without installing it:

python3 scripts/linux_install_smoke.py \
  --package /absolute/path/to/agent-coordinator-VERSION-linux-x86_64.tar.gz

This checks the archive and its offline source links. It does not repeat the systemd, HTTPS, or sustained-capacity acceptance exercises.

Release archives contain the Markdown sources, book.toml, the root records, and deployment examples, allowing readers to build the book from the extracted package. Generated HTML is not committed or included in binary packages. Build and test scripts are maintained in the source checkout. Do not place credentials, application state, build trees, or unrelated files in book/src.

The authoritative maintained source is the repository root AGENTS.md. The content below is included directly during every book build.

Working on Agent Coordinator

Read README.md, HANDOFF.md, BACKLOG.md, and book/src/docs/implementation-status.md first. book/src/PLAN.md and the contract documents in book/src/docs define the intended release; implemented features are listed separately. Do not describe planned endpoints as working.

Use Rust/Axum/SQLite and embedded vanilla JavaScript/CSS. Keep credentials outside the repository and never print tokens, proofs, passwords, request headers, or SQL bind values. Do not enable remote execution by the service.

For concurrent implementation, assign disjoint files and separate Git worktrees. Use a separate Cargo target directory inside each worktree; concurrent builds from different source trees must not share compiled crate metadata. Integrate one reviewed commit at a time. Preserve other worktrees and uncommitted changes. Use smaller subagents for bounded tasks when appropriate.

Every database mutation must obtain the SQLite writer lock before checking the current clock, credential/session validity, generation, task ownership, and policy. Record the effect, idempotency receipt, and event in that same transaction. Never hold a transaction across network or process work. A retry must reuse its saved request and key; a receipt must not imply renewed ownership. Protect native client state against simultaneous processes and interrupted writes.

Run cargo fmt, workspace Clippy with warnings denied, and meaningful workspace tests. For service/client changes, build the workspace and run scripts/smoke.py. For UI changes, run node –check web/app.js and verify the running page in a browser. For documentation changes, build with the pinned mdBook version and run the documentation link/package checks documented in the book. Edit canonical chapters in book/src; root AGENTS.md, HANDOFF.md, BACKLOG.md, and DURABLE-RECORD.md remain authoritative and are included directly in the book. Native Windows claims require actual Windows CI evidence. Update the handoff and backlog with completed behavior, tests, limitations, and next steps.

This repository is not yet bound to a running coordination service. Do not invent a service URL, enrollment token, task completion record, or production deployment.

The authoritative maintained source is the repository root HANDOFF.md. The content below is included directly during every book build.

Implementation handoff — 2026-09-10

Backlog items 6.1 (MCP), 6.2 (mdBook), and 7 (native Windows workstation acceptance) are implemented, exercised, and reviewed. The numbered release backlog and hosted-CI validation follow-up are complete. Item 7 used the actual native Windows workstation and a separate Linux workstation; it did not substitute native Windows CI or package inspection for the physical exercise.

Native Windows workstation acceptance

The accepted agent-coordinator 0.1.0 Windows x86-64 CLI from release run 34459987847 ran natively on workstation MINIAIR against a disposable service on Linux workstation mxmini through a temporary Cloudflare HTTPS tunnel. The downloaded Windows archive SHA-256 was be5e96ec6f8843a63c8be319d9b4f9745f4e0870bebbbc160c2c24f69dc32a1d, matching its adjacent checksum; the exercised executable SHA-256 was 40240c59aeaf012aba6721f378e9dec57419ceb8833aec0187391c3bccb1111d.

The exercise used distinct Windows and Linux principals and sessions across two isolated projects. Windows claimed, checkpointed, and submitted project-two work; a same-principal agent review was rejected, while the independent Linux principal claimed and approved the exact submission. Project-one work remained isolated. At a published barrier, both workstations attempted the same project-one task and revision. Linux obtained generation 2 ownership and Windows received the expected claim_conflict; Linux checkpointed the result and released the task ready. In a separate recovery case, Windows checkpointed with a dedicated session, that session was closed, and Linux inspected and released the retained work without reviving the Windows session.

Sanitized cross-workstation evidence is retained in completion commit c017a69. The encrypted one-time handoff branch was removed after Windows decrypted it. The disposable public service and tunnel were shut down after acceptance. This proves the scoped native two-workstation workflow; it is not a production deployment, permanent public endpoint, hardware attestation, or off-server backup exercise.

MCP completion

The authenticated stateless /mcp endpoint exposes 56 closed, typed tools through the same guarded REST handlers. Connections, discovery, ping, and receipt replay never renew ownership. The native mcp-client launcher securely shares one saved harness session with a trusted foreground MCP client and its native CLI children. Git, local producers, and binary transfer remain native operations. The launcher requires an absolute executable path and does not supervise background clients. No service-side execution or OAuth discovery is introduced.

Candidate cd9a633 passed all 182 local workspace tests, formatting, warnings-denied Clippy, build, and both smoke exercises. Linux workspace/smoke, native Windows client/CLI/local-runner tests, and the dependency audit passed in CI run 34468585302. The integrated 677253a also passed CI run 34468821269. Official SDK tests exercise modern and legacy protocols over real TCP; wire tests cover revocation, races, policies, shared REST receipts, and generation/lease guards. See the MCP guide for configuration and compatibility limits.

Documentation completion

Canonical guides, contracts, the plan, and README content now live in book/src. The original docs/README/PLAN entry paths remain short compatibility links. Root AGENTS.md, HANDOFF.md, BACKLOG.md, and DURABLE-RECORD.md remain authoritative; their book chapters include them at build time. Edit those root files directly.

The pinned mdBook 0.5.4 build covers all 35 chapters. Main review verified retained acceptance evidence, proposal/current-status distinctions, source references, rendered internal links and fragments, search, live includes, and desktop/390-pixel phone layout without page overflow or JavaScript errors. Release packages retain bounded Markdown sources and book configuration, without generated HTML.

Package checks passed for layout, checksums, permissions, offline links, explicit member/size limits, and building the book from an extracted archive. The final local structural package used stripped copies of the current local debug binaries; it is documentation validation, not new release-binary or capacity acceptance. Local-file navigation and search also passed with networking disabled.

After the repository became public, hosted validation was retried on main commit ebf72f7. The pinned mdBook build and documentation checks passed in documentation run 34487173043. Linux formatting, warnings-denied Clippy, all workspace tests, the workspace build, both service/CLI smoke exercises, native Windows client/CLI/local-runner tests, and the locked dependency audit passed in coordination run 34487175859. These runs supersede the earlier failed-to-start billing-blocked attempts; no job steps ran in those attempts, so they remain historical scheduling failures rather than test results.

Use mdbook build or python3 scripts/check_docs.py from a source checkout; output is target/book/index.html. See book maintenance. The retained Linux acceptance evidence remains unchanged and identifies its exact accepted package and server. Schema version is 16 and instruction version is 7. No production deployment or actual off-server transfer is claimed. Host/domain/backup-destination choices remain installation inputs. Keep separate Cargo targets for concurrent worktrees.

The authoritative maintained source is the repository root BACKLOG.md. The content below is included directly during every book build.

Implementation backlog

Execution instruction: complete the remaining items sequentially. The main agent reviews each item’s code and validation evidence before starting the next item. Continue without operator review; ask only for information or access needed to proceed. The numbered release backlog through item 7 is complete. Item 7 used an actual native Windows workstation coordinated with a separate Linux workstation; CI did not replace that physical exercise.

The foundation, job/worktree evidence, and reviewed completion milestones are implemented. Use implementation status for precise limits.

Completed: native worktree preparation, stable producer identity, durable local job reports, global named resource reservations, scoped health reporting, observation-only reconnect, and recovery inspection. Uncertain producers retain physical resource holds; resolving them records evidence without rewriting results.

Completed item 2: immutable submissions, exact-source/check receipts, independent agent/human/both review, canonical repository/target integration holds, guarded native Git publication, and integrated-result validation before done/dependency release. Publication recovery preserves original evidence and transfers the hold to a fresh integration activity for fresh checks.

Completed item 3: revisioned lessons, binding-rule provenance, scoped decisions, bounded context, artifact links/uploads, and authoritative Markdown import/export. The main agent reviewed the integrated behavior; all 114 tests, the built smoke exercise, browser checks, Linux/Windows CI, and dependency audit passed. Audited SithBit/Submission fixtures preserve historical closure and leave source repositories unchanged.

Completed item 4: operator accounts and browser sessions, password recovery, agent credential rotation, full policy editing, revisioned objective grouping, complete task history pagination, inspected operator recovery controls, and named native commands. Main-agent review and all 129 local tests, Clippy, build, smoke, and browser checks passed. Linux/native-Windows CI and the dependency audit passed; evidence is linked in the handoff.

Completed item 5: verified consistent SQLite/artifact snapshots, 24 hourly and 30 daily retention buckets, documented off-server copying, fresh-directory restore, old-authority invalidation, and an explicit operator reconciliation checklist. Main-agent review, all 140 workspace tests, Clippy, build, both smoke exercises, and browser verification passed. The disposable restore took 6.9 seconds; this does not establish production-size recovery or off-server protection.

Completed item 6: clock rollback handling, bounded retention, reproducible native packages, and Linux systemd/HTTPS installation with both timers and verified backup. All 160 workspace tests and Linux/Windows CI passed. The exact packaged server completed 90,000 requests over 30 minutes at 50 requests/second with 20 projects, 50 sessions, and 100,000 historical tasks under a shared two-CPU/4-GiB/no-swap limit. Overall p95 was 21.0 ms, with no unexpected errors and exact ownership preserved. The snapshot restore stage passed in 6.511 seconds. Main-agent review is complete; see retained acceptance evidence for scope and limits.

Completed item 6.1: authenticated stateless MCP with 56 typed tools sharing the REST session, policy, ownership, and receipt checks. The native client launcher securely shares its saved harness identity with a trusted foreground MCP client. All 182 workspace tests, formatting, Clippy, build, and both smoke exercises pass. Main-agent review is complete; see MCP guidance for compatibility and local-operation limits. Linux/native Windows CI and the dependency audit passed in CI run 34468585302.

Completed item 6.2: all guides and README content are consolidated into a 35-chapter mdBook with pinned builds, source coverage and link checks, concise compatibility entry points, and live includes of root agent/handoff/backlog/lesson records. Main-agent review, the rendered link/fragment checks, browser navigation/search and phone layout, package bounds/checksums/offline links, and a book build from an extracted package passed, including local-file navigation/search with networking disabled. After the repository became public, the pinned documentation workflow passed on main in documentation run 34487173043, and the complete Linux/native-Windows coordination workflow and dependency audit passed in coordination run 34487175859. See book maintenance.

Completed item 7: the accepted Windows x86-64 CLI ran on the native MINIAIR workstation against a disposable service on Linux workstation mxmini over HTTPS. The exercise covered public help and origin-bound authentication, separate principals and sessions, two-project isolation, Windows claim/checkpoint/submission, same-principal review rejection, independent Linux review, an exactly-one-owner cross-workstation claim race, and Linux recovery of checkpointed work after the dedicated Windows session was closed. Sanitized evidence is retained in completion commit c017a69. The disposable public service and tunnel were shut down afterward. This completes the numbered release backlog, but does not claim a production deployment, permanent endpoint, hardware attestation, or off-server backup protection.

No unrestricted task-status edit, automatic force recovery, or unverified completion shortcut should be added to make these milestones appear complete.

The authoritative maintained source is the repository root DURABLE-RECORD.md. The content below is included directly during every book build.

Durable engineering record

• Claiming is a transaction. A task listing reserves nothing. Sample server time and recheck authorization after acquiring the database writer lock; a request may have waited behind another writer until after its lease expired.

• A receipt is historical evidence. Replaying a renewal never resets its countdown. Claims report current authority separately; ownership-dependent checkpoint/checkout/recovery retries also revalidate the current attempt. Release retries can acknowledge the historical release because that operation deliberately ends ownership.

• Session identity is distinct from a workstation token. Separate harnesses need distinct persisted random proofs. Another harness using the same token still cannot write to an attempt it does not own.

• The client participates in correctness. Persist the exact mutation and key before sending, bind it to the credential/origin/session, lock across processes, and publish state durably. A malformed or missing success response is uncertainty, not permission to discard the original key.

• Repository text is not a credential destination grant. Bind environment tokens to an independently configured trusted origin, reject redirects, and refuse credential-bearing repository bindings. Otherwise changing a repository URL could redirect secrets to another service.

• Operator views must use derived status. A stored active attempt may be expired or revoked. Show recovery-required when authority is lost and do not label blocked/planned work available merely because it has no active owner.

• State what is actually verified. A Windows CI definition is not a passing Windows run. A deployment example is not a deployment. A checkout attestation is not remote filesystem inspection. A lease foundation is not reviewed integration.

• A lease and a physical resource hold are different records. Losing task authority or an observer never proves a producer stopped. Keep holds until terminal producer evidence or explicit operator termination/isolation evidence. Record reconciliation separately from the producer’s reported outcome.

• Launch identity must survive uncertainty. Persist one producer identity and launch intent before spawning. Reconnect observes it; an unlocked guardian file or missing PID is not permission to start a replacement. Include boot/process creation identity so PID reuse cannot impersonate the original process.

• Local observation must survive service failure. Publish terminal results durably before uploading, preserve exact pending observation keys, and recover an interrupted journal write without dropping the result. Reading status must not wait for the lifetime lock held by a running guardian.

• Historical registration is not launch permission. Check current scoped launch authority and remaining lease time immediately before starting work. A successful replay cannot refresh an expired grant. Verify the actual checkout identity and clean source again before launch.

• Log setup is part of launch correctness. On Windows, an append-only handle cannot truncate a file. Initialize logs through a writable handle before launch intent, then append while draining. Persist pre-spawn failures as not_started with a safe local explanation so a detached guardian cannot fail silently.

• Native identity paths and tool arguments have different requirements. Keep canonical Windows paths for identity comparisons, but convert verbatim drive/UNC prefixes at the Git boundary. Git may reject the native extended path syntax.

• Publication intent is a boundary for retries. Save intent before spawning Git, require fresh service authority and a conservative local deadline, and use an exact expected target for compare-and-swap. Once launch intent exists, retry observes; seeing an unchanged remote does not prove an old publisher stopped.

• Checks verify an immutable source and definition. Resolve registered producer receipts against the exact result commit/tree, check identity/version/environment, successful exit, and unchanged inputs. A generic successful job or reconciled unknown result cannot substitute for required evidence.

• Recovery must preserve policy and target ownership together. Releasing an old integration hold and creating its replacement in separate transactions permits competing integration or policy changes to strand an already published result. Transfer the hold atomically, retain historical publication evidence, and run fresh checks under the replacement activity.

• Every claim path needs onboarding. Review and integration claims must follow the same durable instruction acknowledgment flow as implementation claims. A reviewer may have connected without ever claiming an implementation task.

• Build outputs belong to a source worktree. Concurrent builds from different worktrees must use separate Cargo target directories. Shared compiled metadata can resolve a dependency against another worktree’s version and produce misleading missing-type errors during integration.

• Decision guards include reads and retries. Evaluate the current attempt mode and current decision scope before reporting authority. A saved recovery claim must not bypass a later decision after recovery changes into ordinary work. Keep checkpoint, inspection, and release available while work is blocked.

• Append the revision before advancing its projection. Historical knowledge reimports must satisfy the same immutable-revision trigger as ordinary edits. Keep the standard scope/provenance shape and immutable record kind so imported records remain searchable and correctable through the same interface.

• Bind SQL parameters consistently. Mixing reused numbered parameters with unnamed placeholders can shift SQLx’s argument mapping. Use a consistent scheme and exercise create/detail/list/next-selection together after projection changes.

• Clock safety includes reads and host commands. Persist protected time before authenticating deadline-dependent reads, including scoped reporter reads. Check account recovery, upload preflight, and backup timestamp paths as well as ordinary mutations. A later incident must invalidate an earlier successful reconciliation response’s claim of current readiness.

• Preserve fractional monotonic time. Repeatedly truncating elapsed duration to milliseconds and resetting its anchor can discard time under frequent calls. Keep a fixed anchor, preserve concurrent progress, and use an explicit simulated clock for deterministic deadline tests. Never change the host clock for a test.

• A bounded response is not a bounded query. Apply project scope within the full-text index and limit the ranking cursor before joining task details. Retention must bound inspected candidates even when no row qualifies; an update limit alone can still scan all history while holding the writer lock.

• Permanent mutation identity outlives its response. Compacted receipt payloads need an explicit marker checked before decoding. Keep principal, operation, key, fingerprint, and restore epoch; a stale or compacted key must never create a new effect. Account-creation preflight lookups need the same protection.

• Capacity evidence needs exact assertions. Count the intended owner/session/ generation bindings, verify operation mix and actual concurrent intervals, and name the measured CPU/memory constraints. Per-process limits do not establish an aggregate host-memory baseline; short runs do not establish sustained capacity.

• Measure the binary that will be installed. A server-only build and a combined server/CLI build can enable different shared dependency features. Feed the accepted package into capacity testing and assert its recorded executable digest before setup; a matching source revision alone does not prove binary identity. Native Windows reproducibility also needs deterministic linker and archive flags, including C dependencies.

• Share clock observations only after commit. Coalesce requests already waiting when a protected-time sample was taken, publish it only after its transaction commits, and resample for later arrivals. Every request still verifies current credentials; mutations recheck time and ownership under their own writer lock. A failed clock commit cannot be published to waiting callers.

• Protocol connection is not coordination authority. Route each MCP tool through the same guarded operation as REST, and preserve its original receipt. Discovery, reconnect, and ping must not renew task ownership. Strip credentials before an SDK sees request headers and suppress SDK payload logging. A native client bridge must carry the exact saved harness proof; matching names alone cannot share ownership.

• A book needs one maintained source per record. Include live root handoff, backlog, and instruction files at build time. Check rendered links and anchors as well as chapter coverage; a successful Markdown build can still leave a broken destination. Retain compatibility links and exact acceptance evidence when moving documents.