Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions docs/dogfood/2026-06-16-oracle-slice1-dogfood.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Oracle Slice-1 DOGFOOD — cross-repo acceptance test

**Date:** 2026-06-16
**Task:** 5.3 — cross-repo DOGFOOD acceptance test (capstone of Slice 1)
**Test:** `tests/core/test_oracle_dogfood.py` (engine worktree)
**Status:** GREEN — 3/3 passed (real podman, real CLI, real ship-check subprocess; no mocks)

## What was proven

The unfakable Atlas oracle gates a real `prd-taskmaster` ship-check end-to-end.
For each DONE task, `skel/ship-check.py` (Gate 5) shells `atlas oracle grade`,
which checks out the submitted commit into a throwaway worktree, **overlays the
operator-held tests over the submitter's tree**, re-executes the card's grading
command inside a digest-pinned podman sandbox, derives `PASS` iff exit 0, and
appends a tamper-evident ledger event.

1. **Genuine pass ships.** Operator-held `grade.sh` = `exit 0`. Ship-check emits
`SHIP_CHECK_OK`, returncode 0, and the ledger records `verdict == "PASS"`.
2. **Reward hack blocked (the acceptance criterion).** The submitter ships a
cheat — a committed `grade.sh` that always `exit 0` — while the operator-held
`grade.sh` is `exit 1`. The cheat does **NOT** ship: the oracle overlays and
re-executes the operator's copy, so the verdict is `FAIL`. `SHIP_CHECK_OK` is
absent, returncode is non-zero, stderr names the oracle FAIL for task 1, and
the ledger records `verdict == "FAIL"`. **Non-vacuous:** the only difference
between case 1 and case 2 is which `grade.sh` the oracle re-runs — the
submitter's committed copy never reaches the verdict.
3. **Ledger integrity.** `atlas ledger verify <dir>` reports
`{"ok": true, "eventCount": 1}` after the genuine pass.

## ATLAS_ORACLE_CMD used

```
ATLAS_ORACLE_CMD="<spine>/node_modules/.bin/tsx <spine>/apps/cli/src/index.ts"
```
(`<spine>` = your local atlas-protocol checkout. The test reads `ATLAS_ORACLE_CMD` from the environment and skips when it is unset.)

`ship-check.py` shlex-splits `ATLAS_ORACLE_CMD` and appends
`oracle grade --repo <root> --commit <HEAD> --card ... --held ... --evidence ... --ledger ...`.

**Why `tsx` on the CLI source and not `node dist/index.js`:** the spine
monorepo's workspace packages (`@atlas-protocol/core|cards|evidence|executor`)
declare `exports: "./src/index.ts"`, so the compiled `apps/cli/dist/index.js`
resolves its workspace deps to TypeScript sources that bare `node` cannot load
(`ERR_MODULE_NOT_FOUND: .../packages/executor/src/grade.js`). The `tsx`
executable runs the identical CLI code path the spine's own vitest suite uses,
with **no edits to the spine repo**. (Building the CLI — `pnpm --filter
@atlas-protocol/cli build`, plus `pnpm -r build` for the workspace deps — was
done as Step 0 and is required so the source resolves; the dist itself is not
the run target.)

## Observed results

### Pass case (operator-held `grade.sh` = `exit 0`)

```
$ ATLAS_ORACLE_CMD="$TSX $SRC" python3 skel/ship-check.py --cwd /tmp/ev_pass
SHIP_CHECK_OK
rc=0
```

Ledger event payload (excerpt):

```json
{
"type": "verification.completed",
"actor": { "kind": "executor", "id": "oracle" },
"lifecycleState": "verification_passed",
"payload": { "verdict": "PASS", "exitCode": 0, "overlayHash": "sha256:..." }
}
```

### Reward-hack case (submitter cheat `exit 0`, operator-held `exit 1`)

```
$ ATLAS_ORACLE_CMD="$TSX $SRC" python3 skel/ship-check.py --cwd /tmp/ev_hack
rc=1
--- stdout (empty — no SHIP_CHECK_OK) ---
--- stderr ---
FAIL: task 1: oracle verdict FAIL
```

The committed cheat (`exit 0`) was overlaid away by the operator-held `exit 1`
and re-executed in the sandbox → verdict `FAIL` → ship blocked.

## pytest

```
$ python3 -m pytest tests/core/test_oracle_dogfood.py -v
tests/core/test_oracle_dogfood.py::test_genuine_pass_ships PASSED [ 33%]
tests/core/test_oracle_dogfood.py::test_reward_hack_blocked PASSED [ 66%]
tests/core/test_oracle_dogfood.py::test_ledger_integrity PASSED [100%]
============================== 3 passed in 13.05s ==============================
```

## Slice-2 hardening note

The Graded Card's `contentHash` is a syntactically valid placeholder.
`gradeSubmission` does not re-verify `contentHash` against the card body in
Slice 1 (it is only echoed into the ledger payload). Slice 2 should recompute
and verify it before grading so a tampered card body cannot be graded under a
stale hash. (Also tracked in the executor's `SLICE-1` deferral comments:
infra-failure exit codes are currently recorded as `FAIL`, and `evidenceRef`
records a path rather than a `sha256:`-prefixed content ref.)
127 changes: 127 additions & 0 deletions prd_taskmaster/oracle_bridge.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
"""Oracle bridge: map a CDD card to a Graded Card verdict via the atlas oracle CLI.

Fail-closed: any ambiguity (missing verdict, unparseable output, CLI crash) yields
("FAIL", {...}) — never "PASS" and never an uncaught exception from the grading path.
OracleCardError is the one intended raise, for a missing/unreadable/invalid card.
"""
from __future__ import annotations

import json
import os
import subprocess
from pathlib import Path


class OracleCardError(Exception):
"""Raised when a CDD card cannot be graded (e.g. missing 'grading' block)."""


def _oracle_cmd() -> list[str]:
"""Configurable CLI invocation. Default: the 'atlas' binary on PATH.

Override with ATLAS_ORACLE_CMD (shell-split), e.g.:
ATLAS_ORACLE_CMD="node /path/to/atlas-protocol/apps/cli/dist/index.js"
"""
raw = os.environ.get("ATLAS_ORACLE_CMD")
if raw:
import shlex
return shlex.split(raw)
return ["atlas"]


def grade_card(
*,
card_path: str | Path,
repo_path: str | Path,
commit_sha: str,
held_root: str | Path,
evidence_dir: str | Path,
ledger_dir: str | Path,
oracle_cmd: list[str] | None = None,
) -> tuple[str, dict]:
"""Grade a submission via the atlas oracle CLI.

Returns (verdict, detail) where verdict is "PASS" or "FAIL".

FAIL-CLOSED: any error (missing verdict, unparseable output, CLI crash)
yields ("FAIL", {...}) — never raises after the card has been validated.

Raises:
OracleCardError: if the card file is missing, unreadable, or has no 'grading' block.
"""
card_path = Path(card_path)

try:
card = json.loads(card_path.read_text())
except (OSError, json.JSONDecodeError) as exc:
raise OracleCardError(f"cannot read card {card_path}: {exc}") from exc

if "grading" not in card:
raise OracleCardError(
f"card {card_path} has no 'grading' block; cannot grade"
)

cmd = (oracle_cmd or _oracle_cmd()) + [
"oracle", "grade",
"--repo", str(repo_path),
"--commit", commit_sha,
"--card", str(card_path),
"--held", str(held_root),
"--evidence", str(evidence_dir),
"--ledger", str(ledger_dir),
]

try:
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
except (OSError, subprocess.SubprocessError) as exc:
return ("FAIL", {"error": f"oracle CLI invocation failed: {exc}"})

try:
parsed = json.loads(proc.stdout)
except json.JSONDecodeError:
return (
"FAIL",
{
"error": "oracle CLI produced no parseable JSON verdict",
"returncode": proc.returncode,
"stdout": proc.stdout[:500],
"stderr": proc.stderr[:500],
},
)

verdict = parsed.get("verdict")
if verdict not in ("PASS", "FAIL"):
return ("FAIL", {"error": "oracle CLI verdict missing/invalid", "parsed": parsed})

return (verdict, parsed)


def grade_task(
*,
repo_root: str | Path,
task_id,
commit_sha: str,
held_root: str | Path,
evidence_dir: str | Path,
ledger_dir: str | Path,
oracle_cmd: list[str] | None = None,
) -> tuple[str, dict]:
"""Convenience: locate the CDD card for a task and grade it.

Looks for .atlas-ai/cdd/task-<id>.json under repo_root.

Raises:
OracleCardError: if the card file does not exist or fails validation.
"""
card_path = Path(repo_root) / ".atlas-ai" / "cdd" / f"task-{task_id}.json"
if not card_path.exists():
raise OracleCardError(f"no CDD card at {card_path}")
return grade_card(
card_path=card_path,
repo_path=repo_root,
commit_sha=commit_sha,
held_root=held_root,
evidence_dir=evidence_dir,
ledger_dir=ledger_dir,
oracle_cmd=oracle_cmd,
)
2 changes: 1 addition & 1 deletion prd_taskmaster/render.py
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ def shipcheck_panel(shipcheck: dict, *, ascii_mode: bool = False) -> str:
lines.append(f"{sym} Gate {i} {name:<26} {word}")
lines.append("")
if shipcheck.get("passed"):
token = "SHIP_CHECK_OK" + (" [OVERRIDE]" if shipcheck.get("override_active") else "")
token = "SHIP_CHECK_OK"
lines.append(f"{ok} {token}")
else:
lines.append(f"{blocked} not shippable — {len(shipcheck.get('failures', []))} gate(s) failed")
Expand Down
Loading
Loading