|
| 1 | +# SEC-07: Self-Validation Convergence Problem |
| 2 | + |
| 3 | +**Status:** Observed, documenting |
| 4 | +**Author:** Akkari (with Daniel Cervera) |
| 5 | +**Date:** 2026-03-10 |
| 6 | +**Scope:** Known pattern where iterative self-validation fails to converge to zero findings |
| 7 | + |
| 8 | +**Origin:** Observed during PR #8 (Tester critic). Three rounds of GRAD self-validation fixes, each resolving findings but generating new ones. Tests passed consistently; self-validation never stabilized. |
| 9 | + |
| 10 | +--- |
| 11 | + |
| 12 | +## 1. Problem Statement |
| 13 | + |
| 14 | +When Quorum validates its own code via CI (the GRAD workflow), fixing findings from one run can generate new findings in the next. The cycle does not reliably converge to zero findings, creating a potential infinite loop that blocks merging even when all tests pass. |
| 15 | + |
| 16 | +This is a product-level concern — any user who enables Quorum self-validation as a required CI check will encounter this pattern. |
| 17 | + |
| 18 | +## 2. Why It Happens |
| 19 | + |
| 20 | +### 2.1 Fresh Context Per Run |
| 21 | + |
| 22 | +Each CI run evaluates the code with no memory of prior runs. A fix that resolved a HIGH finding may introduce code patterns that trigger new MEDIUM findings. The critic doesn't know the previous code was worse — it only sees the current state. |
| 23 | + |
| 24 | +### 2.2 Fix-Induced Findings |
| 25 | + |
| 26 | +Defensive code patterns recommended by the critic can themselves be flagged: |
| 27 | +- Narrowing `except Exception` to specific types → critic flags "does this cover all failure modes?" |
| 28 | +- Adding explicit `None` checks → critic flags "redundant guard" or "information disclosure in error path" |
| 29 | +- Sanitizing error messages → critic flags "lost exception context" |
| 30 | + |
| 31 | +The recommended fix for finding A becomes the trigger for finding B. |
| 32 | + |
| 33 | +### 2.3 Severity Recalibration |
| 34 | + |
| 35 | +Fixing a CRITICAL or HIGH finding can cause the critic to reassess the surrounding code at a higher standard, surfacing MEDIUMs and LOWs that were previously shadowed by the more severe finding. |
| 36 | + |
| 37 | +### 2.4 No Delta Awareness |
| 38 | + |
| 39 | +The CI workflow evaluates all findings in changed files, not just *new* findings relative to the base branch. A file with 10 existing findings that gets a one-line fix will report all 10 + any new ones, making it impossible to distinguish progress from regression. |
| 40 | + |
| 41 | +## 3. Impact |
| 42 | + |
| 43 | +- **CI blocker:** If self-validation is a required status check, PRs with passing tests can be stuck indefinitely |
| 44 | +- **Developer frustration:** Each "fix" round takes minutes of CI time and may produce more work than it resolves |
| 45 | +- **False signal:** The pattern can erode trust in self-validation — developers learn to ignore or override it |
| 46 | +- **Cost:** Each CI validation run consumes LLM tokens ($0.50–$2.00 at standard depth) |
| 47 | + |
| 48 | +## 4. Candidate Mitigations (to explore) |
| 49 | + |
| 50 | +### 4.1 Delta-Only Mode |
| 51 | +Only report findings that are NEW relative to the base branch. Existing findings in changed files are suppressed. This is how most linters work (`--diff` mode) and prevents the "fix one, find ten" pattern. |
| 52 | + |
| 53 | +**Effort:** Medium. Requires storing baseline findings per branch or computing a diff against the base branch's last validation run. |
| 54 | + |
| 55 | +### 4.2 Convergence Detection |
| 56 | +Track finding count across iterations. If the count isn't decreasing after N rounds, warn the user and offer a graceful override rather than a hard block. |
| 57 | + |
| 58 | +**Effort:** Low. Requires a counter in the CI workflow and a conditional pass. |
| 59 | + |
| 60 | +### 4.3 Severity-Based Gating |
| 61 | +Only gate on CRITICAL findings. HIGH/MEDIUM/LOW are reported as advisory annotations on the PR but don't block merging. |
| 62 | + |
| 63 | +**Effort:** Low. Workflow change only — check exit code against a severity threshold. |
| 64 | + |
| 65 | +### 4.4 Finding Deduplication Across Runs |
| 66 | +Hash each finding's core attributes (location, description pattern, criterion) and suppress re-flagged findings that were present in the previous run. Only surface genuinely new findings. |
| 67 | + |
| 68 | +**Effort:** Medium-High. Requires persistent storage of finding hashes across CI runs (cache or artifact). |
| 69 | + |
| 70 | +### 4.5 Max Iteration Cap |
| 71 | +Allow N rounds of self-validation fixes. After N rounds, if findings remain, report them as known issues and allow merge with an annotation. |
| 72 | + |
| 73 | +**Effort:** Low. Policy decision, not a code change. |
| 74 | + |
| 75 | +## 5. Recommended Default |
| 76 | + |
| 77 | +Until a proper delta mode is implemented: |
| 78 | + |
| 79 | +- **Self-validation as advisory**, not required — report findings as PR comments but don't block merge |
| 80 | +- **Tests as the hard gate** — all unit/integration tests must pass |
| 81 | +- **CRITICAL-only blocking** — only CRITICAL self-validation findings block merge (§4.3) |
| 82 | + |
| 83 | +This preserves the value of self-validation (visibility into code quality) without creating an unresolvable blocker. |
| 84 | + |
| 85 | +## 6. References |
| 86 | + |
| 87 | +- PR #8 (SharedIntellect/quorum) — first observed instance |
| 88 | +- SEC-03 §3.3 — Assurance Level Model (gating thresholds should scale with assurance level) |
| 89 | +- GRAD workflow (`.github/workflows/quorum-validate.yml`) |
0 commit comments