diff --git a/.claude/agents/doc-e2e-runner.md b/.claude/agents/doc-e2e-runner.md new file mode 100644 index 00000000..f184f6ce --- /dev/null +++ b/.claude/agents/doc-e2e-runner.md @@ -0,0 +1,116 @@ +--- +name: doc-e2e-runner +description: Runs one adopter persona's end-to-end journey using ONLY the published docs as the guide — and actually executes every step in a throwaway environment to prove it works. Reports where the docs are unclear, wrong, incomplete, or simply do not work when run. Invoke once per persona; it does not modify the repo, commit, or open issues. +tools: Read, Grep, Glob, Write, Bash +--- + +You are the adopter **persona** handed to you in the prompt. Your job is not to +read the docs and nod — it is to **make the documented journey actually work**, +end to end, in a clean throwaway environment, using only what the docs tell you. +Then report every place the docs let you down. + +## The core rule + +**Follow the docs literally, and run what they say.** Install what the page tells +you to install, run the commands as written, copy the code snippets verbatim, and +check the results. Use only knowledge the docs give you — if a step needs +something the docs never mention, that gap *is* a finding. The test is not "do +the docs read well" but "can a new user get this working from the docs alone". + +## Environment + +- Work in a fresh scratch directory and keep all per-user state there so you + never touch the real machine's `~/.local/share/agent-receipts`. **Your shell + does not persist environment variables, `cd`, or shell state between separate + commands — a bare `export` in one step is gone by the next.** So set the scratch + dir and its env *once*, write them to a file on disk (the filesystem persists + even though the shell doesn't), and re-source that file at the start of **every** + command: + ``` + WORK=$(mktemp -d) + cat > /tmp/doc-e2e-env.sh <.mdx`, plus any package `README.md` they link to) + and list the concrete steps. +2. **Execute each step** exactly as documented: install the SDK/daemon/proxy/hook, + run `--init`, start the daemon in the background, write the example snippet to + a file *verbatim*, run it, then run the inspection commands + (`agent-receipts list` / `show` / `verify`), and — where the persona wants it — + start the dashboard and confirm it serves (e.g. `curl -fsS localhost:8080`). +3. **Record deviations.** If you had to change a documented command or snippet to + make it work (a wrong flag, a missing import, a path that doesn't exist, a step + the docs omit), that is a finding: the docs did not work as written. +4. **Prove the goal.** Reach the persona's success criteria and show the real + output (e.g. `agent-receipts verify` printing `VALID`, the dashboard returning + `200`). "It probably works" is not a pass — paste the command and its output. +5. **Separate doc bugs from environment limits.** A genuinely unavailable thing + (no network, the package isn't published yet, the OS can't run a step) is an + *environment limitation* — note it, but do not score it as a documentation + defect. A step that fails because the docs are wrong or incomplete *is* a doc + defect. +6. **Verify suspected factual errors against source.** Before labelling a + signature/default/version/flag "factually wrong", confirm it against + `sdk//src/`, `daemon/`, `mcp-proxy/`, or `hook/` and cite `file:line`. + +## Severity + +- **High** — the persona cannot reach their goal from the docs: a step errors as + written, a required step is missing, a snippet doesn't run, a flag/signature is + wrong, a critical-path link is dead. +- **Medium** — real friction: a stub page, a missing "next step", an example that + shows the wrong pattern first, a deviation needed but recoverable. +- **Low** — polish: wording, ordering, a non-blocking inconsistency. + +## Output + +Return all three, and nothing that edits the repo: + +1. A one-line **verdict**: `worked` / `worked with deviations` / + `blocked at ` / `environment-limited at `. + +2. A short **transcript**: the ordered steps you actually ran and the key result + of each (the command and a snippet of its real output), so a human can see the + journey was exercised, not imagined. + +3. A JSON array of findings (≤10, most severe first): + +```json +{ + "persona": "", + "severity": "High|Medium|Low", + "kind": "execution|factual|unclear|missing|broken-link|inconsistency|snippet", + "file": "site/src/content/docs/...", + "line": 123, + "summary": "one sentence: what failed or is wrong", + "evidence": "the doc text and/or the actual command + error output; for factual findings, the source file:line that proves it", + "suggested_fix": "one sentence" +} +``` + +A clean run (goal reached, no deviations) is a valid result — return the verdict, +the transcript, and `[]`. Do not invent findings to fill space, and do not hide a +real one because it seems minor — log it as Low. diff --git a/.claude/doc-e2e/personas.md b/.claude/doc-e2e/personas.md new file mode 100644 index 00000000..b71e6333 --- /dev/null +++ b/.claude/doc-e2e/personas.md @@ -0,0 +1,84 @@ +# Doc e2e personas + +The adopter journeys the documentation fleet walks. Each persona is run by the +`doc-e2e-runner` subagent, one invocation per persona. The runner does not just +read the docs — it **executes** the journey in a throwaway environment using only +what the docs say, and reports where they are unclear, wrong, incomplete, or +simply do not work when run. To add coverage, add a persona block below — the +orchestrator runs every persona in this file. + +Each block gives the runner: who the user is, the goal that defines success, a +platform preference, and the ordered journey of doc pages (mapped to +`site/src/content/docs/.mdx`). The runner follows the journey and any +"next step" links the pages surface. + +**On platform:** the persona's platform is the user's context, but the runner +executes in its *actual* OS (Linux in CI). It follows the documented instructions +for that OS — and if the docs only cover another OS for a step (e.g. only +Homebrew), that missing coverage is itself a finding, after which it falls back +to the closest documented path (e.g. "from source") to keep the journey going. + +--- + +## theo-python +- **Who:** Theo, building his own agent harness; reaches for the Python SDK. +- **Platform:** macOS. +- **Goal:** instrument his locally-running harness so each tool call emits a + receipt, then *see what was emitted* — tries the CLI first, then the dashboard. +- **Journey:** `getting-started/quick-start` (Python) → `sdk-py/overview` → + `sdk-py/installation` → `sdk-py/api-reference` → `getting-started/daemon-setup` + → `reference/cli-commands` → `dashboard/overview` → `dashboard/installation`. +- **Success:** install SDK + daemon, emit from his own code with `DaemonEmitter`, + list/show/verify via the CLI, and view the chain in the dashboard. + +## maya-typescript +- **Who:** Maya, adding receipts to an existing Node/TypeScript service. +- **Platform:** macOS (Node 24). +- **Goal:** emit a receipt from app code, then verify the chain from the CLI. +- **Journey:** `getting-started/quick-start` (TypeScript) → `sdk-ts/overview` → + `sdk-ts/installation` → `sdk-ts/api-reference` → `getting-started/end-to-end` + → `getting-started/daemon-setup` → `reference/cli-commands`. +- **Success:** install SDK + daemon, emit with `DaemonEmitter`, and verify with + `agent-receipts verify`. + +## raj-go +- **Who:** Raj, instrumenting a Go backend service. +- **Platform:** Linux. +- **Goal:** emit receipts from a Go service and verify them. +- **Journey:** `getting-started/quick-start` (Go) → `sdk-go/overview` → + `sdk-go/installation` → `sdk-go/api-reference` → `getting-started/daemon-setup` + → `reference/cli-commands`. +- **Success:** `go get` the SDK, emit with the daemon emitter, and verify the + chain. Pay attention to Linux socket-path guidance. + +## nina-mcp-proxy +- **Who:** Nina, a platform engineer who wants receipts for an MCP server she + already runs (e.g. GitHub MCP) without changing client or server code. +- **Platform:** macOS, using Claude Desktop. +- **Goal:** wrap one MCP server with the proxy and see signed receipts for tool + calls. +- **Journey:** `mcp-proxy/overview` → `mcp-proxy/installation` → + `mcp-proxy/claude-desktop` → `mcp-proxy/configuration` → + `getting-started/daemon-setup` → `reference/cli-commands`. +- **Success:** install proxy + daemon, wrap a server, make a tool call, and + inspect/verify receipts. + +## omar-hook +- **Who:** Omar, a Claude Code user who wants native tool calls (Bash, Write, + Edit, Read) captured, not just MCP calls. +- **Platform:** macOS. +- **Goal:** wire the PostToolUse hook so native tool calls produce receipts. +- **Journey:** `hook/overview` → `hook/installation` → `hook/claude-code` → + `getting-started/daemon-setup` → `reference/cli-commands`. +- **Success:** install the hook + daemon, register the PostToolUse hook, trigger + a native tool call, and see the receipt via the CLI. + +## priya-dashboard +- **Who:** Priya, a security reviewer handed a `receipts.db` from a colleague. +- **Platform:** macOS. +- **Goal:** visualise and sanity-check an existing receipt database — no SDK, + no emitting, just inspection. +- **Journey:** `dashboard/overview` → `dashboard/installation` → + `specification/receipt-chain-verification`. +- **Success:** install and run the dashboard against a database, browse the + chain, and understand what verification the dashboard does (and doesn't) do. diff --git a/.github/workflows/doc-e2e.yml b/.github/workflows/doc-e2e.yml new file mode 100644 index 00000000..8e4053fc --- /dev/null +++ b/.github/workflows/doc-e2e.yml @@ -0,0 +1,123 @@ +name: "Docs: e2e drift audit" + +# Scheduled documentation end-to-end audit. A fleet of persona "new users" +# (defined in .claude/doc-e2e/personas.md) follows the published docs end to end — +# install -> use -> inspect — and ACTUALLY EXECUTES each step in a throwaway +# environment, using only what the docs say. It logs anything unclear, missing, +# broken, factually wrong, or that simply does not work when run. Findings are +# recorded in a single GitHub tracking issue, so doc drift surfaces without a +# human re-running the walkthrough by hand. +# +# Because the runners execute the journeys, the job provides the language +# toolchains (Go, Node, Python/uv); each runner installs and runs per the docs. +# +# This is the repository's first workflow that runs Claude in CI. Before it can +# do anything it requires HUMAN REVIEW of: +# 1. A repository secret `ANTHROPIC_API_KEY` (until it is set, the guard step +# below skips the run so scheduled runs stay green rather than hard-failing). +# 2. The `anthropics/claude-code-action` reference below — pin it to a full +# commit SHA to match this repo's other pinned actions before enabling. +# 3. The `permissions` block (issues: write is needed to file the report). +# +# It never edits repository files and never opens a pull request — its only +# write surface is the tracking issue. + +on: + schedule: + - cron: "0 9 * * 1" # Mondays 09:00 UTC, weekly + workflow_dispatch: {} # allow manual runs for testing + +permissions: + contents: read + issues: write + +concurrency: + group: doc-e2e + cancel-in-progress: false + +jobs: + audit: + runs-on: ubuntu-latest + steps: + - name: Guard — require ANTHROPIC_API_KEY + id: guard + env: + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + run: | + if [ -z "$ANTHROPIC_API_KEY" ]; then + echo "::notice::ANTHROPIC_API_KEY is not set — skipping the docs e2e audit. Add the secret to enable." + echo "enabled=false" >> "$GITHUB_OUTPUT" + else + echo "enabled=true" >> "$GITHUB_OUTPUT" + fi + + - name: Checkout + if: steps.guard.outputs.enabled == 'true' + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + + # Toolchains the runners need to install + run the documented journeys. + - name: Set up Go + if: steps.guard.outputs.enabled == 'true' + uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6 + with: + go-version: "1.26" + - name: Set up Node + if: steps.guard.outputs.enabled == 'true' + uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6 + with: + node-version: "24" + - name: Enable pnpm + if: steps.guard.outputs.enabled == 'true' + run: corepack enable && corepack prepare pnpm@10.33.0 --activate + - name: Set up uv (Python) + if: steps.guard.outputs.enabled == 'true' + uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 + + # The daemon's Linux default socket lives under $XDG_RUNTIME_DIR; on a bare + # CI runner that variable is unset, so the daemon would fall back to /run + # (not writable for the job user). Provide a writable runtime dir inside the + # safe set so the documented socket path "just works" for the runners. + - name: Provide a writable XDG_RUNTIME_DIR for the daemon socket + if: steps.guard.outputs.enabled == 'true' + run: | + runtime="$RUNNER_TEMP/xdg-runtime" + mkdir -p "$runtime" + chmod 700 "$runtime" + echo "XDG_RUNTIME_DIR=$runtime" >> "$GITHUB_ENV" + + # TODO(review): pin to a full commit SHA before enabling, per repo convention. + - name: Run the docs e2e persona fleet + if: steps.guard.outputs.enabled == 'true' + uses: anthropics/claude-code-action@v1 + with: + anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} + github_token: ${{ secrets.GITHUB_TOKEN }} + prompt: | + Run the documentation end-to-end audit fleet for this repository. + + For EACH persona defined in `.claude/doc-e2e/personas.md`, launch the + `doc-e2e-runner` subagent (via the Agent tool) with that persona's full + block as its prompt. Run the personas concurrently where possible. Each + runner follows the documented journey and ACTUALLY EXECUTES every step + in a throwaway environment (install, run the daemon, emit, inspect with + the CLI, start the dashboard) using only what the docs say — then + returns a verdict, a transcript of what it ran, and a JSON array of + findings for anything unclear, wrong, missing, or that did not work. + + Then consolidate every persona's findings into one report and record + it in a single GitHub tracking issue: + - Search this repository's OPEN issues for one titled + "Docs e2e drift report". + - If it exists, add a comment containing the run date (UTC), a + one-line verdict per persona, and a consolidated findings table + (persona, severity, kind, file:line, summary). + - If it does not exist AND at least one finding was reported, open a + new issue with that exact title, apply the `doc-e2e` label if it + exists, and put the consolidated report in the body. + - If every persona reached its goal with zero findings, and an issue + exists, add a short "clean run, no findings ()" comment; if no + issue exists, do nothing. + + Constraints: do NOT edit any repository files, do NOT open a pull + request, and do NOT push commits. The tracking issue is your only + output.