Merge pull request #9 from SharedIntellect/feature/tester-critic

v0.6.0: Tester Critic (6/9) + Shipping Process Infrastructure

What shipped

Tester Critic (Milestone #10) — Critic 6 of 9

• L1 deterministic verification: loci existence, line number range, excerpt matching, file path validation
• L2 LLM claim verification: evaluates whether findings are substantiated by cited evidence
• 805 lines code, 994 lines tests, 76/76 passing

Shipping Process Infrastructure — prevents the doc staleness that triggered this work

• critic-status.yaml: single source of truth for critic/feature status (replaces hardcoded counts across 10+ files)
• tools/validate-docs.py: automated doc validation against the manifest (CI-enforced)
• SHIPPING.md: shipping checklist for humans and agents
• CLAUDE.md: agent instructions with mandatory shipping process
• .github/workflows/validate-docs.yml: CI gate — runs on every PR to main

Documentation sweep — 11 files updated from stale "4 critics" / "5 critics" to accurate "6 critics" with Cross-Consistency noted as requiring --relationships flag.

Quorum self-validation results

validate-docs.py was validated by Quorum at standard depth across two runs:

Run 1 (pre-fix): REVISE — 5 HIGH, 4 MEDIUM, 1 LOW. All 5 HIGHs were legitimate code quality issues: broad exception handling, unvalidated YAML input, missing type annotations, no file size guards. All addressed in commit 2ad0df0.

Run 2 (post-fix): REVISE — 3 HIGH, 6 MEDIUM, 2 LOW. These are SEC-07 convergence findings — new issues generated by the fixes themselves:

| #   | Severity | Finding                                                          | Disposition                                                                                                 |
| --- | -------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| 1   | HIGH     | read_file_lines() returns None without None checks at call sites | False positive. validate_docs() explicitly checks if lines is None: continue on line 218.                   |
| 2   | HIGH     | load_manifest() calls sys.exit(2) violating return type contract | By design. CLI utility, not a library. sys.exit() in validation helpers is standard Python CLI pattern.     |
| 3   | HIGH     | read_file_lines() return behavior inconsistent with docstring    | Docstring says "returns None on failure." Returning None on oversized files IS the documented failure path. |
| 4   | MEDIUM   | sys.exit() prevents testability                                  | Accepted for a CLI script. Would refactor if this became a library.                                         |
| 5   | MEDIUM   | main() mixes concerns                                            | Addressed in run 1 (extracted 3 helpers). Remaining main() is ~30 lines of orchestration.                   |
| 6   | MEDIUM   | Hardcoded yaml dependency                                        | Not actionable — yaml IS the dependency.                                                                    |
| 7   | MEDIUM   | YAML not validated before field access                           | False positive. load_manifest() validates structure — this was the H3 fix from run 1.                       |
| 8-9 | MEDIUM   | Error messages expose file paths                                 | Local developer tool. Paths in errors are diagnostic, not a security exposure.                              |
| 10  | LOW      | MAX_FILE_SIZE_BYTES is a magic number                            | False positive. It IS a named constant with a docstring.                                                    |
| 11  | LOW      | TOCTOU race on file size check                                   | Theoretical — local markdown files in a git repo, not untrusted concurrent input.                           |


This is the SEC-07 self-validation convergence pattern documented in docs/SEC07_SELF_VALIDATION_CONVERGENCE.md: each fix round changes code, giving the critic fresh context to generate new findings about the fixes. Run 1 findings were genuine; run 2 findings are the critic being conservative about a 250-line CLI utility. Tests pass (76/76). Merging per SEC-07 interim policy: advisory-only self-validation, CRITICAL-only blocking.

Author: dacervera

.github/workflows/validate-docs.yml

@@ -0,0 +1,25 @@
+name: Validate Documentation
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main]
+
+jobs:
+  validate-docs:
+    name: Check docs match critic-status.yaml
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install PyYAML
+        run: pip install pyyaml
+
+      - name: Run doc validation
+        run: python tools/validate-docs.py

CLAUDE.md

@@ -0,0 +1,46 @@
+# CLAUDE.md — Agent Instructions for Quorum
+
+This file provides context for Claude Code and other AI agents working on this repository.
+
+## Project Overview
+
+Quorum is a multi-critic quality assurance framework. The reference implementation lives in `reference-implementation/` and is published on PyPI as `quorum-validator`.
+
+## Current State
+
+- **6 of 9 critics shipped**: Correctness, Completeness, Security, Code Hygiene, Cross-Consistency (`--relationships` flag required), Tester (L1 + L2)
+- **3 planned**: Architecture, Delegation, Style
+- **Version**: See `critic-status.yaml` for the canonical version and critic status
+- **Tests**: `cd reference-implementation && pytest tests/ -v` (exclude `test_e2e_smoke.py` and `test_performance.py` for offline runs)
+
+## Shipping Checklist — MANDATORY
+
+**When shipping any feature or critic to main, follow `SHIPPING.md` in the repo root.**
+
+Key steps:
+1. Update `critic-status.yaml` (the single source of truth for critic/feature status)
+2. Bump version in `critic-status.yaml` AND `reference-implementation/pyproject.toml`
+3. Run `python tools/validate-docs.py` — fix all findings before merging
+4. Run `tools/boundary-scan.sh` (located in the workspace, not this repo) — ensure no private paths leak
+5. Update `docs/CHANGELOG.md` with a full release entry
+6. All doc files that reference critic counts or status markers must match the manifest
+
+**Why this matters:** We've shipped features with 10+ doc files still claiming old critic counts. The manifest + validation script + CI gate exists to prevent that. Don't skip it.
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `critic-status.yaml` | Single source of truth for shipped critics and features |
+| `SHIPPING.md` | Full shipping checklist |
+| `tools/validate-docs.py` | Automated doc validation (reads manifest, flags stale references) |
+| `SPEC.md` | Framework specification |
+| `reference-implementation/` | Python CLI implementation |
+| `docs/CHANGELOG.md` | Release history |
+
+## Conventions
+
+- **Evidence grounding**: Every critic finding must cite specific locus (file, line, excerpt). No hand-waving.
+- **Test coverage**: New critics need both unit tests and integration tests. Target: 20+ tests per critic.
+- **Status markers**: Use ✅ Shipped or 🔜 Planned in docs. Update `critic-status.yaml` first, then docs.
+- **Boundary scan**: No private file paths, internal names, or credentials in public files. `quorum-runs/` is gitignored.

README.md

@@ -109,10 +109,12 @@ Not every artifact needs the full treatment. Tell me how much is riding on it, a
 | Depth | Critics | Time | Cost* | When to use it |
 |-------|---------|------|-------|----------------|
 | **Quick** | 2 (correctness, completeness) | 5-10 min | ~$0.15 | "Give me a sanity check before I keep going" |
-| **Standard** | 4 (+ security, code_hygiene) | 15-30 min | ~$0.50 | Most work — solid coverage without the wait |
-| **Thorough** | 4 now; more when they ship | 30-60 min | ~$1.50+ | "This is going to production. It cannot be wrong." |
+| **Standard** | 6 (+ security, code_hygiene, cross_consistency†, tester) | 15-30 min | ~$0.50 | Most work — solid coverage without the wait |
+| **Thorough** | 6 shipped; more when they ship | 30-60 min | ~$1.50+ | "This is going to production. It cannot be wrong." |
 
-*Estimates on Claude Sonnet. Scales with model and artifact size. Pre-screen (10 deterministic checks + DevSkim + Ruff + Bandit + PSScriptAnalyzer) runs before LLM critics at every depth level — no extra cost. Today I ship with 4 critics (Correctness, Completeness, Security, Code Hygiene) + re-validation loops + learning memory. Architecture, Delegation, and Tester are coming — the full architecture supports all 9 (see [SPEC.md](SPEC.md)).
+*†Cross-Consistency requires `--relationships` flag with a relationships manifest.*
+
+*Estimates on Claude Sonnet. Scales with model and artifact size. Pre-screen (10 deterministic checks + DevSkim + Ruff + Bandit + PSScriptAnalyzer) runs before LLM critics at every depth level — no extra cost. Today I ship with 6 critics (Correctness, Completeness, Security, Code Hygiene, Cross-Consistency†, and Tester) + re-validation loops + learning memory. The full architecture supports all 9 — Architecture, Delegation, and Style are next (see [SPEC.md](SPEC.md)).*
 
 ---
 
@@ -137,8 +139,11 @@ Not every artifact needs the full treatment. Tell me how much is riding on it, a
    │  ┌──────────┐ ┌──────────┐             │
    │  │ Security │ │CodeHygine│  ← shipped  │
    │  └──────────┘ └──────────┘             │
+   │  ┌──────────┐ ┌──────────┐             │
+   │  │CrossConsist│ │ Tester │  ← shipped  │
+   │  └──────────┘ └──────────┘  (†Cross-Consistency requires --relationships)
    │  ┌──────┐ ┌──────┐ ┌────────┐         │
-   │  │ Arch │ │Deleg │ │ Tester │ ← roadmap│
+   │  │ Arch │ │Deleg │ │ Style  │ ← roadmap│
    │  └──────┘ └──────┘ └────────┘         │
    └──────────────┬───────────────────────────┘
                   │ Phase 1.5: (if max_fix_loops > 0)
@@ -189,9 +194,9 @@ I auto-detect your model on first run and configure myself accordingly. Details:
 
 I'm working. I'm real. I'm also still growing.
 
-**What I can do today** (v0.5.3):
+**What I can do today** (v0.6.0):
 - Full CLI: `quorum run --target <file> [--depth] [--rubric] [--pattern] [--relationships] [--output-dir] [--verbose] [--fix-loops N] [--no-learning] [--max-cost USD] [--audit-report] [--resume <batch-dir>] [--yes]`
-- **4 critics** — Correctness, Completeness, Security (OWASP ASVS 5.0, CWE Top 25, NIST SA-11; see [SEC-02 workflow](docs/SEC02_BUSINESS_LOGIC_VALIDATION.md) for business logic validation guidance), Code Hygiene (ISO 25010:2023, CISQ) — all with evidence grounding
+- **6 critics** — Correctness, Completeness, Security (OWASP ASVS 5.0, CWE Top 25, NIST SA-11; see [SEC-02 workflow](docs/SEC02_BUSINESS_LOGIC_VALIDATION.md) for business logic validation guidance), Code Hygiene (ISO 25010:2023, CISQ), Cross-Consistency† (`--relationships`), and Tester (L1 deterministic + L2 LLM claim verification) — all with evidence grounding
 - **Parallel execution** — critics run concurrently (ThreadPoolExecutor, max 4); batch files run concurrently (max 3)
 - **Re-validation loops** — Fixer proposes text replacements for CRITICAL/HIGH findings → applies them → re-runs only the critics that flagged the originals → reports improved/unchanged/regressed. Up to 3 loops, stops early when findings clear. `--fix-loops N` or `--depth thorough` (default 1 loop)
 - **Learning memory** — tracks recurring failure patterns across runs in `known_issues.json`. High-frequency patterns auto-promote to mandatory checks injected into critic prompts. `quorum issues list|promote|reset` CLI. Disable with `--no-learning`
@@ -210,7 +215,7 @@ I'm working. I'm real. I'm also still growing.
 
 **What's coming:**
 - Dynamic model pricing updates (`quorum costs update`)
-- More critics (Architecture, Delegation, Style, Tester)
+- More critics (Architecture, Delegation, Style)
 - Domain-specific rubric packs (compliance, security, infrastructure)
 - Confidence calibration against golden sets
 - Branch protection enforcement (require PR + passing validation to merge to main)

SHIPPING.md

@@ -0,0 +1,46 @@
+# Shipping Checklist
+
+Follow this checklist before merging any feature PR to `main`. Both humans and agents (including Claude Code) must complete all steps.
+
+---
+
+## 1. Code Quality
+- [ ] All tests pass (`pytest`)
+- [ ] No regressions in existing functionality
+
+## 2. Update Manifest
+- [ ] `critic-status.yaml` updated (add/change status for shipped feature)
+- [ ] Version string bumped in `critic-status.yaml`
+- [ ] Version string bumped in `reference-implementation/pyproject.toml`
+
+## 3. Documentation Sweep
+Run `python tools/validate-docs.py` to find stale references, then fix:
+
+- [ ] `CHANGELOG.md` — add entry for this release
+- [ ] `README.md` — "What I can do today" section accurate
+- [ ] `README.md` — critic count in depth table footnote
+- [ ] `README.md` — architecture diagram labels (shipped vs roadmap)
+- [ ] `README.md` — "What's coming" section (remove shipped items)
+- [ ] `SPEC.md` — status matrix / implementation checklist updated
+- [ ] `SPEC.md` — shipped critic counts in prose
+- [ ] `docs/FOR_BEGINNERS.md` — critic counts and cost estimates
+- [ ] `docs/CONFIG_REFERENCE.md` — critic status markers
+- [ ] `docs/TUTORIAL.md` — critic descriptions and status markers
+- [ ] `docs/IMPLEMENTATION.md` — critic status markers and depth profiles
+- [ ] `SKILL.md` — critic counts in depth descriptions
+
+**⚠️ Note on Cross-Consistency:** Cross-Consistency is a shipped critic but requires the `--relationships` flag. When updating counts, use language like: "6 shipped critics (including Cross-Consistency, activated with `--relationships`)".
+
+## 4. Validation
+- [ ] `python tools/validate-docs.py` passes (exit 0)
+- [ ] `tools/boundary-scan.sh` passes (exit 0)
+
+## 5. Commit & Ship
+- [ ] Commit message includes: what shipped, critic N of 9, known limitations
+- [ ] Tag release if version bump warrants it
+
+---
+
+## Why This Exists
+
+PR #8 (Tester critic) shipped with 10+ doc files still claiming "4 critics." The Cross-Consistency critic shipped at some point with zero doc updates. This checklist + `validate-docs.py` + CI enforcement exists to prevent that from happening again.

SKILL.md

@@ -33,8 +33,10 @@ python -m quorum.cli run --target <path-to-artifact> --rubric <rubric-name>
 ### Depth Profiles
 
 - `quick` — 2 critics (correctness, completeness) + pre-screen, ~5-10 min
-- `standard` — 4 critics (+ security, code_hygiene) + pre-screen, ~15-30 min (default)
-- `thorough` — all 4 shipped critics + pre-screen, ~30-60 min
+- `standard` — 6 critics (+ security, code_hygiene, cross_consistency†, tester) + pre-screen, ~15-30 min (default)
+- `thorough` — all 6 shipped critics + pre-screen + fix loops, ~30-60 min
+
+*†Cross-Consistency requires `--relationships` flag with a relationships manifest.*
 
 All depth profiles include the deterministic pre-screen (10 checks: credentials, PII, syntax errors, broken links, TODOs, and more) before any LLM critic runs.
 

SPEC.md

@@ -9,7 +9,7 @@
 
 ## 1. Overview
 
-Quorum is a quality assurance framework with a nine-agent target architecture. Currently, 5 critics are implemented (Correctness, Completeness, Security, Code Hygiene, and Cross-Artifact Consistency), with 4 additional critics planned. The framework is designed to rigorously evaluate multi-agent systems, configurations, research, code, and operational procedures against domain-specific rubrics. It combines:
+Quorum is a quality assurance framework with a nine-agent target architecture. Currently, 6 critics are implemented (Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency†, and Tester), with 3 additional critics planned. †Cross-Consistency is activated with the `--relationships` flag. The framework is designed to rigorously evaluate multi-agent systems, configurations, research, code, and operational procedures against domain-specific rubrics. It combines:
 
 - **Parallel specialized critics** (9 agents with distinct expertise)
 - **Grounded evidence requirement** (every critique must cite tool-verified proof)
@@ -403,8 +403,7 @@ Status as of v0.5.0 (reference implementation):
 - [x] LLM provider — LiteLLM universal provider (100+ models, any tier combination)
 - [x] File-based artifact passing (no in-memory state between agents)
 - [x] Pre-screen layer — 10 built-in checks (PS-001–PS-010) + DevSkim SAST integration
-- [x] 4 critics implemented — Correctness, Completeness, Security, Code Hygiene
-- [x] Cross-Artifact Consistency critic (Phase 2, relationships manifest)
+- [x] 6 critics implemented — Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency† (`--relationships`), Tester (L1 + L2)
 - [x] Rubric system (JSON schema + validator, 3 built-in rubrics: research-synthesis, agent-config, python-code)
 - [x] Batch/multi-file validation with `BatchVerdict`
 - [x] Aggregator synthesis logic (conflict resolution)
@@ -416,7 +415,7 @@ Status as of v0.5.0 (reference implementation):
 - [x] Fixer agent — proposal mode + re-validation loops (Phase 1.5; proposes and applies text replacements for CRITICAL/HIGH, then re-runs critics to verify)
 - [x] Python code rubric (25 criteria, PC-001–PC-025, auto-detects on .py files)
 - [x] Learning memory system (known_issues.json frequency tracking + mandatory check promotion — shipped in v0.5.3)
-- [ ] Remaining 4 critics — Architecture, Delegation, Style, Tester
+- [ ] Remaining 3 critics — Architecture, Delegation, Style
 - [ ] Trust/monitoring system (per-critic accuracy tracking)
 
 See IMPLEMENTATION.md for a reference walkthrough.
@@ -441,7 +440,7 @@ Quorum is built on these peer-reviewed papers:
 
 ### Current Limitations (v3.0)
 
-- Only **5 of 9 critics are implemented** (Architecture, Delegation, Style, Tester are specified but not built)
+- Only **6 of 9 critics are implemented** (Architecture, Delegation, Style are specified but not built)
 - Rubric panel is **static** (doesn't specialize per artifact type dynamically)
 - **No critic-to-critic debate** (relies on Aggregator to resolve conflicts)
 - Learning is **frequency-based** only (no semantic deduplication of patterns yet)
@@ -450,7 +449,7 @@ Quorum is built on these peer-reviewed papers:
 
 ### Planned
 
-- Remaining critics: Architecture, Delegation, Style, Tester
+- Remaining critics: Architecture, Delegation, Style
 - ~~Re-validation loops~~ → **Shipped in v0.5.3**
 - ~~Learning memory~~ → **Shipped in v0.5.3**
 - Dynamic critic specialization (spawn domain-specific critics on-demand)

critic-status.yaml

@@ -0,0 +1,64 @@
+# critic-status.yaml — Single source of truth for shipped features
+# Used by tools/validate-docs.py to detect stale documentation.
+# Update this file FIRST when shipping a critic or feature.
+
+version: "0.6.0"
+
+critics:
+  correctness:
+    status: shipped
+    since: "0.3.0"
+    description: "Factual accuracy, citation validity, logical consistency"
+  completeness:
+    status: shipped
+    since: "0.3.0"
+    description: "Coverage gaps, missing sections, rubric criterion coverage"
+  security:
+    status: shipped
+    since: "0.4.0"
+    description: "OWASP ASVS 5.0, CWE Top 25, NIST SA-11, business logic"
+  code_hygiene:
+    status: shipped
+    since: "0.4.0"
+    description: "ISO 25010:2023, CISQ quality measures, maintainability"
+  cross_consistency:
+    status: shipped
+    since: "0.5.0"
+    description: "Inter-file consistency via relationships manifest (requires --relationships flag)"
+    note: "Activated with --relationships flag; not part of default depth panel"
+  tester:
+    status: shipped
+    since: "0.6.0"
+    description: "Finding verification — L1 deterministic + L2 LLM claim checking"
+  architecture:
+    status: planned
+    description: "Structural design, modularity, dependency analysis"
+  delegation:
+    status: planned
+    description: "Multi-agent coordination, responsibility boundaries"
+  style:
+    status: planned
+    description: "Writing quality, tone consistency, formatting standards"
+
+features:
+  prescreen:
+    status: shipped
+    description: "10 deterministic checks + DevSkim + Ruff + Bandit + PSScriptAnalyzer"
+  fix_loops:
+    status: shipped
+    description: "Re-validation loops with fixer proposals, up to 3 iterations"
+  learning_memory:
+    status: shipped
+    description: "Recurring pattern tracking via known_issues.json"
+  batch_validation:
+    status: shipped
+    description: "Directory/pattern validation with crash-resilient progressive saves"
+  cost_tracking:
+    status: shipped
+    description: "Per-call token counts, cost estimates, budget caps"
+  audit_reports:
+    status: shipped
+    description: "CSV audit detail + summary reports with SHA-256 integrity"
+  time_estimation:
+    status: shipped
+    description: "Pre-run time + cost estimates with --estimate-time flag"