Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,25 @@ All notable changes to this project are documented here. Format based on
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project adheres to
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- **Independent out-of-band re-execution watcher** (`prd_taskmaster/tournament/watcher.py`;
`watcher-run` / `watcher-status` CLI) — the precondition for ever enabling real
(`--enforce-slash`) forfeiture. It re-adjudicates settled tournament submissions from
**primary evidence** (the claimed commit + the CDD card) without trusting the recorded
verdict: it re-runs the oracle gate, re-derives `sha256(diff base..HEAD)` to catch diff-copy
tamper independently of the live collector, and accumulates a concordance ledger over real
*slash decisions*. A `permit_enforce_slash` gate is **fail-closed**: real slashing is permitted
only when every to-be-slashed submission is independently confirmed, there is **no discrepancy
or abstain anywhere in the job**, and the watcher's historical concordance clears a threshold
over a minimum number of prior decisions. Inability to verify (oracle could not run, no
worktree, failed hash recompute) **abstains** — it is never counted as grounds to slash.
- **Engine-enforced shadow-until-permitted** — `run_tournament` now consults the watcher before
any real `--enforce-slash` settle and **downgrades to shadow** unless the watcher permits the
job (fail-closed on any watcher error). Real AtlasCoin is never burned without an independent
positive confirmation; the shadow-slash default path is unchanged.

## [5.3.0] — 2026-06-17

The "unfakable done" release: a task ships only when its test genuinely re-ran green AND
Expand Down
94 changes: 90 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,59 @@ provable?*
What makes Atlas a product, not a prompt pack (the four moats):
1. **Cross-vendor arbitrage** — route work across any harness/API by cost and capability.
2. **Engine-enforced unfakable gates** — validate-tasks, ship-check (`SHIP_CHECK_OK`),
structural tests. If a feature could be a skill, it's not a product feature until the
engine enforces it.
the two-axis "done" (oracle + reachability, below), structural tests. If a feature could
be a skill, it's not a product feature until the engine enforces it.
3. **Persistent vendor-neutral task graph** — tasks.json survives across vendors/sessions.
4. **Cross-vendor cost ledger** — telemetry + economy-report prove the savings.

## Unfakable "done" — the two gates (5.3.0+, the trust backbone)

A task ships only when **both** axes pass — `done = oracle-PASS AND code WIRED/EXEMPT`.
This is the property the whole product (and the marketplace below) rests on; do not weaken it.

- **Gate 5 — re-execution oracle.** The binding ship gate is `skel/ship-check.py`
(oracle-backed). It re-runs the operator-held tests at the *claimed commit* in a
network-isolated, digest-pinned **podman** sandbox, so a submitter cannot pass by editing
their own logs. `prd_taskmaster/oracle_bridge.py` maps a CDD card → graded verdict via the
spine's `atlas oracle` CLI; it is **fail-closed** (any ambiguity → FAIL, never PASS). The
self-grantable `SHIP_CHECK_OVERRIDE_ADMIN` bypass was removed.
(`prd_taskmaster/shipcheck.py` is a NON-binding display heuristic only — never add an oracle
call there.)
- **Gate 6 — reachability** (`prd_taskmaster/reachability.py`, `reachability-sweep` CLI). A
green test on an orphan module (imported by nothing) is **blocked** and surfaced as
`⚠ scaffolded`. Verdicts: WIRED (a non-test file imports it) / EXEMPT (declared
`cli:|route:|tool:|hook:|plugin:|dynamic:` scheme, accepted on trust in v1) / ORPHAN (fail).
Read-only at runtime.

## Tournament marketplace (`prd_taskmaster/tournament/`)

The trustworthy two-gate signal feeds a **settled tournament**: N executors race one job →
every submission is adjudicated through both gates → the winner is paid in **AtlasCoin** →
a UCB reputation store routes the next job to the cheapest proven-capable executor.

- `tournament-run` / `tournament-status` CLI; flow is `spawn → collect → adjudicate → settle
→ reputation` (`cmd.py`), with anti-sybil slots released in a crash-safe `finally`.
- `collect.py` is the security core — **commit-reveal** (`commit_hash = sha256(diff base..HEAD)`)
defeats the diff-copy attack. `antisybil.py` enforces per-job / per-operator economic caps.
- `goose_backend.py` is the cheap-API racer (one OpenRouter model via `goose run`).
- `watcher.py` (`watcher-run` / `watcher-status`) is the **independent out-of-band re-execution
watcher** — the precondition for real slashing. It re-adjudicates settled submissions from
primary evidence (re-runs the oracle, independently re-derives `sha256(diff base..HEAD)` to
catch diff-copy), abstains when it cannot verify (never a false confirm), and **engine-gates**
real forfeiture: `run_tournament` downgrades `--enforce-slash` to shadow unless the fail-closed
`permit_enforce_slash` confirms the whole job behind a concordance track record.
- **Invariants (enforced + tested — keep them):** **shadow-slash by default**; real slashing only
behind the watcher's fail-closed permit (no AtlasCoin burned without an independent positive
confirmation); honest losers are **always refunded**; **AtlasCoin is conserved** (no mint/burn,
even under account aliasing).

## The spine (`atlas-protocol`, separate repo)

The gates shell out to the **spine CLI** (`atlas oracle` / `atlas tournament settle` / ledger),
which lives in the `atlas-protocol` repo (branch `dev`) and runs via `tsx` today. Point
`ATLAS_ORACLE_CMD` at it (e.g. `tsx apps/cli/src/index.ts`) for the real-podman e2e gates.
Packaging the spine for production is on the roadmap — see CHANGELOG `## [5.3.0]`.

## Backend model (v4.1+)

TaskMaster is one pluggable backend, not a prerequisite. *"Atlas speaks TaskMaster
Expand All @@ -40,6 +88,30 @@ natively — but doesn't need it."*
- The normative reference is the **"## Backend operations" table in SKILL.md**. When docs
and code disagree, fix the docs to match the table.

## Repository layout

The engine is **stdlib-only Python** in `prd_taskmaster/`, surfaced two ways that share the
same code — keep them in lockstep:
- `script.py` — thin shim over `prd_taskmaster/cli.py:main` (~40 subcommands; `python3 script.py -h`).
- `mcp-server/server.py` — FastMCP server exposing the same ops as MCP tools (`next_task`,
`set_task_status`, `validate_prd`, …). It needs `pip install -r mcp-server/requirements.txt`;
the CLI does not.

Key modules: `cli.py` (dispatch) · `backend.py` (5-op backend protocol) · `validation.py` +
`tasks.py` (graded PRD + task graph) · `task_state.py` (atomic next/claim/set-status) ·
`reachability.py` (Gate 6) · `oracle_bridge.py` + `shipcheck.py` (Gate 5 wiring) · `economy.py`
+ `reputation.py` (cost ledger + UCB routing) · `fleet.py` / `parallel.py` (fleet waves) ·
`tournament/` (the settled marketplace; `tournament/watcher.py` is the independent re-execution
watcher that engine-gates real slashing). The binding ship gate is `skel/ship-check.py`; the
skill/slash commands live in `skills/` (atlas, go, generate, execute-task, execute-fleet, …) and
ship inside the package. `SKILL.md` is the normative behavior spec. Tests split into
`tests/core` + `tests/plugin` (stdlib-only), `tests/mcp` (needs the mcp package), and
`tests/parity` (golden-parity harness, run as a module, not via pytest).

`AGENTS.md` is the harness-neutral twin of this file — the **same operating contract** for any
agent (Codex, Gemini, the next harness). No test enforces their agreement, so when you change an
operating rule here, mirror it in `AGENTS.md` (and vice-versa).

## Task execution workflow

1. `python3 script.py next-task` — get the next dependency-ready task
Expand All @@ -51,16 +123,30 @@ natively — but doesn't need it."*
## Commands

```bash
pytest tests/ -q # full suite — must stay green
pytest tests/ -q # full suite (877); the 4 real-podman e2e gates
# auto-skip unless the env below is set
pytest tests/core tests/plugin -q # stdlib-only path (no mcp pkg); tests/mcp needs
# pip install -r mcp-server/requirements.txt first
pytest tests/core/test_validation.py::test_name -q # run a single test (file::test or -k expr)
ATLAS_ORACLE_CMD="<spine>/node_modules/.bin/tsx <spine>/apps/cli/src/index.ts" \
pytest tests/core -q -k "e2e or dogfood" # un-skip the 4 real-podman gates (need the spine CLI + podman)
python3 script.py engine-preflight # one-call environment + backend probe
python3 script.py validate-tasks <tasks.json> # gate before any tasks.json write
python3 script.py reachability-sweep # Gate 6 — orphan-module detection
python3 script.py tournament-run # settled marketplace: spawn→…→reputation
python3 script.py watcher-run --job <dir> --card <card> --task <id> --base-ref <sha> # out-of-band re-adjudication
python3 script.py watcher-status # watcher concordance + whether real slashing is permitted yet
python3 script.py economy-report # cost ledger from .atlas-ai/telemetry.jsonl
```

## Conventions (engine code)

- **Stdlib only** in `prd_taskmaster/` and `mcp-server/server.py` — no new dependencies.
CI enforces stdlib-only imports on a bare runner.
CI (`.github/workflows/ci.yml`) AST-scans the package for non-stdlib imports on a bare runner.
- CI also enforces, and these break easily: **all version strings must agree** —
`prd_taskmaster/__init__.py`, `package.json`, `.claude-plugin/plugin.json`, `install.sh`, plus a
matching `CHANGELOG.md` entry (bump them together); and the **native-no-keys** path must still
resolve `backend=native, ai_ops=agent` with no TaskMaster binary and no API keys.
- Errors return FR-28-safe dicts; library code never `sys.exit`s.
- All `.atlas-ai/*.jsonl` writes go through the shared flock-guarded append
(`economy.append_telemetry` pattern).
Expand Down
29 changes: 28 additions & 1 deletion prd_taskmaster/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,12 @@
from prd_taskmaster import fleet, parallel, task_state
from prd_taskmaster.lib import _detect_taskmaster_method
from prd_taskmaster.reachability_cmd import cmd_reachability_sweep
from prd_taskmaster.tournament.cmd import cmd_tournament_run, cmd_tournament_status
from prd_taskmaster.tournament.cmd import (
cmd_tournament_run,
cmd_tournament_status,
cmd_watcher_run,
cmd_watcher_status,
)


def _backend_source() -> str:
Expand Down Expand Up @@ -419,6 +424,26 @@ def build_parser() -> argparse.ArgumentParser:
p.add_argument("--reputation-path", default=None, help="Path to reputation.jsonl (default: .atlas-ai/reputation.jsonl)")
p.add_argument("--operators-path", default=None, help="Path to operators.json (default: .atlas-ai/tournament/operators.json)")

# watcher-run — independent out-of-band re-adjudication of a settled job
p = sub.add_parser(
"watcher-run",
help="Re-adjudicate a settled tournament job out-of-band; append concordance; report the real-slash permit",
)
p.add_argument("--job", required=True, help="Path to the settled job dir (contains submissions.json)")
p.add_argument("--card", required=True, help="Path to the CDD card JSON")
p.add_argument("--task", required=True, help="Task id (e.g. 7 or 1.2)")
p.add_argument("--base-ref", required=True, help="Fork-point git SHA the diff range is measured from")
p.add_argument("--repo-root", default=".", help="Repo containing the racer commits (default: .)")
p.add_argument("--held-root", default=None, help="Held root for the oracle gate (default: .atlas-ai/cdd)")
p.add_argument("--ledger-path", default=None, help="Watcher ledger path (default: .atlas-ai/tournament/watcher.jsonl)")

# watcher-status — report watcher concordance + real-slash readiness
p = sub.add_parser(
"watcher-status",
help="Show the watcher's historical concordance and whether real slashing is permitted yet",
)
p.add_argument("--ledger-path", default=None, help="Watcher ledger path (default: .atlas-ai/tournament/watcher.jsonl)")

# status — render progress panels
p = sub.add_parser("status", help="Render Atlas progress panels for the current phase")
p.add_argument("--phase", default=None, help="Render a specific phase instead of the current one")
Expand Down Expand Up @@ -476,6 +501,8 @@ def cmd_status(args) -> None:
"reachability-sweep": cmd_reachability_sweep,
"tournament-run": cmd_tournament_run,
"tournament-status": cmd_tournament_status,
"watcher-run": cmd_watcher_run,
"watcher-status": cmd_watcher_status,
"economy-report": cmd_economy_report,
"context-pack": cmd_context_pack,
"feedback-add": cmd_feedback_add,
Expand Down
111 changes: 110 additions & 1 deletion prd_taskmaster/tournament/cmd.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@

import argparse
import json
import subprocess
import sys
from pathlib import Path
from typing import Any, Callable, Optional
Expand All @@ -47,9 +48,28 @@
FakeClock,
)
from prd_taskmaster.tournament.adjudicate import adjudicate_job, settle_job
from prd_taskmaster.tournament import watcher as _watcher
from prd_taskmaster.reputation import record_tournament, summarize_reputation, _winner_id


def _resolve_repo_root(cwd: "str | None" = None, default: str = ".") -> str:
"""Resolve the git repo root for the watcher's re-execution worktrees.

Falls back to *default* (never raises) when not in a git repo, so the
watcher gate degrades gracefully rather than crashing the settle path.
"""
try:
proc = subprocess.run(
["git", "rev-parse", "--show-toplevel"],
cwd=cwd, capture_output=True, text=True, timeout=10,
)
if proc.returncode == 0 and proc.stdout.strip():
return proc.stdout.strip()
except Exception: # noqa: BLE001
pass
return default


# ─── Core orchestration function ─────────────────────────────────────────────


Expand All @@ -74,6 +94,10 @@ def run_tournament(
now: str,
window_s: float = 120.0,
enforce_slash: bool = False,
repo_root: str = ".",
watcher_ledger_path: "str | Path | None" = None,
_re_adjudicate: "Optional[Callable[..., dict]]" = None,
_permit: "Optional[Callable[..., dict]]" = None,
_spawn_fn: "Callable[[Any], dict]" = default_launcher_adapter,
_inbox_read: "Callable[..., list[dict]]" = default_inbox_adapter,
_dispatch_reveal: "Callable[..., Optional[dict]]" = default_reveal_adapter,
Expand Down Expand Up @@ -290,8 +314,38 @@ def run_tournament(
job_poster=job_poster,
)

# ── Step 5b: Watcher gate on REAL slashing (engine-enforced) ─────
# Shadow-slash only until the independent out-of-band watcher PERMITS
# real forfeiture for THIS job. Fail-closed: a watcher error, a failed
# re-adjudication, or a non-permit downgrades enforce_slash to shadow
# so no AtlasCoin is ever burned without an independent confirmation.
effective_enforce = enforce_slash
if enforce_slash:
_radj = _re_adjudicate or _watcher.re_adjudicate_job
_perm = _permit or _watcher.permit_enforce_slash
w_ledger = (
Path(watcher_ledger_path) if watcher_ledger_path
else Path(".atlas-ai/tournament/watcher.jsonl")
)
try:
wrec = _radj(
job_dir=job_dir, repo_root=repo_root, card_path=card_path,
held_root=held_root, task_id=task_id, start_commit=base_ref,
base_ref=base_ref, now=now, ledger_path=w_ledger,
)
if not wrec.get("ok"):
permit = {"permitted": False, "reason": "watcher re-adjudication failed"}
else:
permit = _perm(wrec, ledger_path=w_ledger, current_job_id=job_id)
except Exception as exc: # noqa: BLE001 — fail-closed → shadow
permit = {"permitted": False, "reason": f"watcher error: {exc}"}
summary["watcher_permit"] = permit
if not permit.get("permitted"):
effective_enforce = False
summary["enforce_slash_downgraded"] = True

# ── Step 6: Settle (FAIL-CLOSED on ok:false) ─────────────────────
settle_env = _settle_fn(job_dir=job_dir, enforce_slash=enforce_slash)
settle_env = _settle_fn(job_dir=job_dir, enforce_slash=effective_enforce)

settled_ok = settle_env.get("ok") is True
summary["settled_ok"] = settled_ok
Expand Down Expand Up @@ -415,6 +469,7 @@ def cmd_tournament_run(args: argparse.Namespace) -> None:
now=now,
window_s=float(getattr(args, "window", 120.0)),
enforce_slash=bool(getattr(args, "enforce_slash", False)),
repo_root=_resolve_repo_root(),
# Real adapters must be wired by the orchestrator skill; the defaults
# raise RuntimeError with guidance if called directly.
_spawn_fn=default_launcher_adapter,
Expand All @@ -426,6 +481,60 @@ def cmd_tournament_run(args: argparse.Namespace) -> None:
_emit({"ok": False, "error": str(exc)})


def cmd_watcher_run(args: argparse.Namespace) -> None:
"""CLI handler for ``watcher-run``.

Re-adjudicates a settled job out-of-band from primary evidence, appends a
concordance row to the watcher ledger, and reports the fail-closed
real-slash permit for the job. FAIL-CLOSED: a missing/unreadable job → ok:false.
"""
job_dir = Path(args.job)
repo_root = getattr(args, "repo_root", None) or "."
card_path = Path(args.card)
held_root = Path(getattr(args, "held_root", None) or ".atlas-ai/cdd")
ledger_path = Path(getattr(args, "ledger_path", None) or ".atlas-ai/tournament/watcher.jsonl")

import datetime
now = datetime.datetime.now(datetime.timezone.utc).isoformat()

try:
rec = _watcher.re_adjudicate_job(
job_dir=job_dir,
repo_root=repo_root,
card_path=card_path,
held_root=held_root,
task_id=args.task,
start_commit=args.base_ref,
base_ref=args.base_ref,
now=now,
ledger_path=ledger_path,
)
except Exception as exc: # noqa: BLE001
_emit({"ok": False, "error": str(exc)})
return

if not rec.get("ok"):
_emit({"ok": False, **rec})
return

# Exclude the in-flight job (just appended to the ledger) from its own gate.
permit = _watcher.permit_enforce_slash(
rec, ledger_path=ledger_path, current_job_id=rec.get("job_id")
)
_emit({"ok": True, **rec, "permit": permit})


def cmd_watcher_status(args: argparse.Namespace) -> None:
"""CLI handler for ``watcher-status``.

Reports the watcher's historical concordance and whether the track record
would clear the real-slash gate (``real_slash_ready``). A view command.
"""
ledger_path = Path(getattr(args, "ledger_path", None) or ".atlas-ai/tournament/watcher.jsonl")
summ = _watcher.concordance_summary(ledger_path)
_emit({"ok": True, **summ})


def cmd_tournament_status(args: argparse.Namespace) -> None:
"""CLI handler for ``tournament-status``.

Expand Down
Loading
Loading