Skip to content

Commit 57ed132

Browse files
committed
docs(walkthrough): headless live-evidence audit runlog — Slice-2A tournament (2 blocking + 5 important fixed, Second-Checked)
1 parent fdf909e commit 57ed132

4 files changed

Lines changed: 141 additions & 0 deletions

File tree

.i-walkthroughs/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Slice-2A Walkthrough Runlog
2+
3+
Headless live-evidence audit (surfaces=screens; live test/CLI/e2e runs=evidence).
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
id: slice2a-tournament-audit
3+
title: Slice-2A settled-tournament — headless live-evidence walkthrough audit
4+
surface: tournament settlement system (TS spine + Python engine orchestrator)
5+
mode: headless
6+
first_check:
7+
method: fresh-eyes live-evidence audit (9 independent auditors, one per surface; live test/CLI/e2e runs as evidence)
8+
result: 2 surfaces SOLID (collect, goose); 7 surfaces DEFECT_FOUND — 2 blocking + 5 important + 9 minor
9+
second_check:
10+
method: per-fix fresh re-run of the exact broken scenario (before→after) + full-suite regression
11+
result: pass
12+
regression_scan: pass
13+
regression_checks:
14+
- "atlas-protocol full suite: pnpm -r run test green (122 core + 28 cli)"
15+
- "engine full suite: python3 -m pytest -q → 868 passed on the reconciled branch"
16+
- "real-podman e2e re-run green with genuine discrimination (fake oracle exitCode=1)"
17+
verdict: resolved
18+
status: shipped
19+
---
20+
21+
## What the audit found (and fixed)
22+
23+
A headless live-evidence walkthrough (`--headless` mode: deliverable surfaces are "screens"; live test/CLI/e2e runs are the authoritative evidence) caught real issues build-time review missed:
24+
25+
**Blocking (TS CLI):**
26+
- A missing `--job` dir threw to stderr with EMPTY stdout → the Python orchestrator's `json.loads` chokes. Fixed: validate job dir → stage-tagged `{ok:false,stage:'parse_input'}` envelope; top-level catch always emits JSON.
27+
- The `settled.json` marker was written AFTER the ledger mutation → a crash between them left the winner paid-but-unmarked and permanently stuck. Fixed: `settling.json` intent marker written BEFORE the state save + resumable detection (`partial_settle_needs_repair`).
28+
29+
**Important:** stale `.settle.lock` permanently blocked settle (fixed: pid/TTL reclaim); `sweep_expired` tz-naive/aware crash (fixed: normalize to UTC); roster rollback only caught `SybilLimitError` → slot leak (fixed: broaden to any exception); the **capstone e2e was non-discriminating**`fake` failed for an infra/card-hash reason (`exitCode:null`) not a genuine reward-hack, and assertion-4 (`win credited bounty`) passed without a bounty (fixed: `fake` is now a real reward-hacker with a matching FAIL-content card → oracle `exitCode=1`; assertion tightened to `>=60`; strict `ledger verify ok:true`).
30+
31+
**Minor:** integer-only AC enforced at the settlement layer; in-contract `FAIL` verdict (not out-of-union `ERROR`); finite/non-negative `settled_cost`; accurate `reputation_recorded`/`settle_envelope_stage` flags; shared `verify` validation.
32+
33+
## Second-Check evidence (resolved)
34+
Every fix was re-run fresh against the exact scenario the audit proved broken (before→after numeric assertion) and gated on the full suite. The capstone e2e now runs the real podman oracle and genuinely discriminates: `win`=PASS/exitCode0 wins & is paid (free 10→65), `fake`=FAIL/exitCode1 is shadow-logged in `wouldSlash`, AC conserved (158==158), ledger + tournament verify ok, reputation routes (explore cold-start + exploit winner). No fix was reported done until its scenario re-ran clean AND the full suite stayed green.
35+
36+
## Trap named
37+
Build-time review tests the happy path + designed error paths; it does NOT test plausible *operational* failure modes (a wrong path from a caller → stderr-only; a crash between two durable writes; an infra-FAIL masquerading as a caught cheat). The headless live-evidence audit — run each surface for real and adversarially probe the holes — is what surfaced them. Reusable rule: a green unit suite is necessary, not sufficient; a fresh independent live-evidence pass per surface is the second axis.

.i-walkthroughs/index.jsonl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"slice2a-tournament-audit","verdict":"resolved","status":"shipped","mode":"headless"}

.i-walkthroughs/schema.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# `.i-walkthroughs/` schema (v1)
2+
3+
> **Skill template.** This is the canonical copy the `interactive-walkthrough` skill carries.
4+
> On first use in a repo that has no `.i-walkthroughs/`, copy this file to
5+
> `<repo>/.i-walkthroughs/schema.md` (then create `README.md`, an empty `index.jsonl`,
6+
> `entries/`, and `evidence/`) and commit, so the Runlog gate's "per `.i-walkthroughs/schema.md`"
7+
> always resolves against a repo-local source of truth.
8+
9+
Canonical field dictionary for walkthrough runlog entries. A future repo-wide RAG ingester
10+
reads THIS file. Stable contract: glob `**/.i-walkthroughs/entries/*.md`, parse YAML
11+
frontmatter as the metadata record, treat each `##` body heading as a chunk boundary, upsert
12+
keyed on `id` (== filename stem), gate migrations on `schema_version`.
13+
14+
## Folder layout
15+
16+
```
17+
.i-walkthroughs/
18+
README.md # what this is + RAG-forward note
19+
schema.md # THIS file — single source of truth
20+
index.jsonl # append-only machine index, one JSON object per entry
21+
entries/
22+
<id>.md # one self-contained learning unit per file
23+
evidence/
24+
<id>/ # per-entry assets (id == entry id)
25+
before-<vp>.png # pre-fix VIEWPORT screenshot
26+
after-<vp>.png # post-fix (second-check) VIEWPORT screenshot
27+
dom-measure.json # {"before":{...},"after":{...}} authoritative numbers
28+
```
29+
30+
**File strategy:** one file per `(screen, issue)` learning unit (NOT an append-to-one-log),
31+
because a RAG hit must return a complete, single-issue document with correct metadata so
32+
filters like `verdict:resolved AND severity:P1` select exactly the right unit. `index.jsonl`
33+
is the fast non-vector path to enumerate/pre-filter the corpus without parsing markdown.
34+
35+
**`<id>` == filename stem == frontmatter `id` == `index.jsonl` row key.** Never reused, never changed.
36+
37+
## Slug / id rules
38+
39+
`<id>` = `<YYYY-MM-DD>-<route-slug>-<issue-slug>`
40+
- `route-slug`: route with `/``-`, leading `-` stripped; root `/``root`.
41+
- `issue-slug`: 2–4 word kebab summary of the defect.
42+
- lowercase ASCII, no spaces. Same-day collision → append `-2`, `-3`.
43+
44+
## Frontmatter fields
45+
46+
| key | type | purpose |
47+
|---|---|---|
48+
| `id` | string (== filename stem) | Primary key for RAG upsert/dedup + `index.jsonl` join + git anchor. Immutable. |
49+
| `schema_version` | int (start 1) | Migration gate for field renames. |
50+
| `repo` | string `owner/name` | Repo-wide RAG filter dimension (this repo's `owner/name`, e.g. `anombyte93/atlas-ai-website`). |
51+
| `branch` | string | Branch the fix landed on. |
52+
| `commit` | git SHA \| `pending` | Hard link to the resolving diff; `pending` until committed. |
53+
| `date` | ISO date `YYYY-MM-DD` | Day the second check passed; filename prefix + index sort key. |
54+
| `route` | string app route | Primary filter: "everything learned about `/calculator`". Leading slash; root `/`. |
55+
| `screen_purpose` | string (one sentence) | The screen soul purpose; anchors the embedding to intended behavior. |
56+
| `mode` | enum `interactive` \| `auto` \| `mobile` | Which walkthrough mode produced this. |
57+
| `viewport` | string `WxH` or `WxH@dpr` | Reproduction context (e.g. `390x844`). |
58+
| `device_class` | enum `mobile` \| `tablet` \| `desktop` | Coarse class for filtering + regression arm. |
59+
| `severity` | enum `P0` \| `P1` \| `P2` \| `P3` | Reuses `reference/auto-report-schema.md` labels verbatim. |
60+
| `category` | enum `layout` \| `copy` \| `a11y` \| `perf` \| `security` \| `interaction` \| `data` \| `trust` \| `role-boundary` | Defect taxonomy for clustering. |
61+
| `root_cause` | string (one line) | The why (NOT the symptom). High-value learn-from-mistakes field. |
62+
| `assumption_trap` | string \| null | The false belief made/nearly made (e.g. "desktop pass == mobile pass"). Drives `reusable_rule`. |
63+
| `reusable_rule` | string (one line, imperative) | Portable rule a future agent applies to avoid repeating this. Highest-signal retrieval target. |
64+
| `evidence_method` | list[enum `dom-measure` \| `viewport-screenshot` \| `console` \| `network` \| `axe` \| `lighthouse` \| `code`] | How proof was gathered, ordered by authority (`dom-measure` first). |
65+
| `files_changed` | list[string] repo-relative | Source files the fix touched. |
66+
| `first_check` | object `{result: fail\|pass, metric, evidence_ref}` | Pre-fix proof the issue was real. Required for P0/P1. |
67+
| `second_check` | object `{result: pass\|fail, metric, evidence_ref, regression_scan: pass\|fail}` | MANDATORY post-fix re-verification. `result=pass` ONLY if same metric now passes AND `regression_scan=pass`. `metric` must be the SAME numeric assertion as `first_check.metric`. |
68+
| `regression_checks` | list[string] | The specific adjacent things re-verified (other viewport class, overflow, focus, console/CSP). Backs `second_check.regression_scan`. |
69+
| `verdict` | enum `resolved` \| `partial` \| `regressed` \| `reverted` | Headline outcome. `resolved`=gone + no regression; `partial`=residual remains; `regressed`=fix broke something; `reverted`=rolled back. |
70+
| `status` | enum `open` \| `verified` \| `shipped` | Lifecycle, orthogonal to verdict. `open`=no passing second check yet; `verified`=second check passed; `shipped`=committed/merged. |
71+
| `approval` | string \| `pending` | Who approved the mutating action (owner handle + when + channel). In `--mobile` this is the Discord reply. |
72+
| `evidence_dir` | string repo-relative | `evidence/<id>/` — keeps binaries out of embedded text but linkable. |
73+
| `tags` | list[string] kebab | Free-form RAG facets beyond the fixed enums. |
74+
| `related` | list[string] entry ids | Graph edges (recurrence → prior resolved entry, supersedes, caused-by). |
75+
| `title` | string (short headline) | Display + embedding-friendly summary; also `index.jsonl` `title`. |
76+
77+
## Body sections (in order; each `##` is a chunk boundary)
78+
79+
1. **Lesson (TL;DR)** — standalone 2–4 sentences; restates `root_cause` + `reusable_rule` in prose. Lead chunk.
80+
2. **Screen & Purpose** — route, role/context, viewport, soul purpose.
81+
3. **Issue (first check)** — observable QUANTITATIVE symptom; DOM numbers first, then screenshot ref. Never an opinion.
82+
4. **Root Cause** — underlying cause + the `assumption_trap` narrative.
83+
5. **Fix** — smallest coherent change, `files_changed`, load-bearing snippet, commit.
84+
6. **Second Check (re-verification, MANDATORY)** — fresh same-screen/same-viewport evidence: (a) same-defect proof via the same metric, (b) explicit regression scan, then the verdict.
85+
7. **Reusable Rule** — the portable rule, imperative.
86+
8. **Decision Trail** — owner decision + channel, rejected alternatives, deferred items, open questions.
87+
9. **Revisions** — append-only log of in-turn updates (e.g. a regressed second check → re-fix). Keeps the full mistake arc in one unit.
88+
89+
## Re-touch rule
90+
91+
- In-turn correction (a fix regresses on its own second check this turn) → **update the same entry** (Revisions section; verdict flips).
92+
- A separate later walkthrough re-touching the same screen → **new dated entry** with `related` linking back. The corpus accumulates `mistake → fix → second-check` triples over time.
93+
94+
## `index.jsonl` row shape
95+
96+
One JSON object per line, append-only (never rewrite prior lines):
97+
98+
```json
99+
{"id":"...","repo":"owner/name","branch":"...","route":"/...","date":"YYYY-MM-DD","severity":"P0|P1|P2|P3","category":"...","verdict":"resolved|partial|regressed|reverted","status":"open|verified|shipped","title":"...","file":"entries/<id>.md"}
100+
```

0 commit comments

Comments
 (0)