Skip to content

Commit 6d9e432

Browse files
author
Akkari
committed
docs: SEC-07 self-validation convergence problem
Documents the known pattern where iterative self-validation fails to converge, observed during PR #8. Includes root cause analysis, impact assessment, and 5 candidate mitigations. Recommends advisory-only self-validation with CRITICAL-only blocking as interim default. This is a product-level concern — any user enabling GRAD as a required check will encounter this pattern.
1 parent 0c5db0e commit 6d9e432

1 file changed

Lines changed: 89 additions & 0 deletions

File tree

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# SEC-07: Self-Validation Convergence Problem
2+
3+
**Status:** Observed, documenting
4+
**Author:** Akkari (with Daniel Cervera)
5+
**Date:** 2026-03-10
6+
**Scope:** Known pattern where iterative self-validation fails to converge to zero findings
7+
8+
**Origin:** Observed during PR #8 (Tester critic). Three rounds of GRAD self-validation fixes, each resolving findings but generating new ones. Tests passed consistently; self-validation never stabilized.
9+
10+
---
11+
12+
## 1. Problem Statement
13+
14+
When Quorum validates its own code via CI (the GRAD workflow), fixing findings from one run can generate new findings in the next. The cycle does not reliably converge to zero findings, creating a potential infinite loop that blocks merging even when all tests pass.
15+
16+
This is a product-level concern — any user who enables Quorum self-validation as a required CI check will encounter this pattern.
17+
18+
## 2. Why It Happens
19+
20+
### 2.1 Fresh Context Per Run
21+
22+
Each CI run evaluates the code with no memory of prior runs. A fix that resolved a HIGH finding may introduce code patterns that trigger new MEDIUM findings. The critic doesn't know the previous code was worse — it only sees the current state.
23+
24+
### 2.2 Fix-Induced Findings
25+
26+
Defensive code patterns recommended by the critic can themselves be flagged:
27+
- Narrowing `except Exception` to specific types → critic flags "does this cover all failure modes?"
28+
- Adding explicit `None` checks → critic flags "redundant guard" or "information disclosure in error path"
29+
- Sanitizing error messages → critic flags "lost exception context"
30+
31+
The recommended fix for finding A becomes the trigger for finding B.
32+
33+
### 2.3 Severity Recalibration
34+
35+
Fixing a CRITICAL or HIGH finding can cause the critic to reassess the surrounding code at a higher standard, surfacing MEDIUMs and LOWs that were previously shadowed by the more severe finding.
36+
37+
### 2.4 No Delta Awareness
38+
39+
The CI workflow evaluates all findings in changed files, not just *new* findings relative to the base branch. A file with 10 existing findings that gets a one-line fix will report all 10 + any new ones, making it impossible to distinguish progress from regression.
40+
41+
## 3. Impact
42+
43+
- **CI blocker:** If self-validation is a required status check, PRs with passing tests can be stuck indefinitely
44+
- **Developer frustration:** Each "fix" round takes minutes of CI time and may produce more work than it resolves
45+
- **False signal:** The pattern can erode trust in self-validation — developers learn to ignore or override it
46+
- **Cost:** Each CI validation run consumes LLM tokens ($0.50–$2.00 at standard depth)
47+
48+
## 4. Candidate Mitigations (to explore)
49+
50+
### 4.1 Delta-Only Mode
51+
Only report findings that are NEW relative to the base branch. Existing findings in changed files are suppressed. This is how most linters work (`--diff` mode) and prevents the "fix one, find ten" pattern.
52+
53+
**Effort:** Medium. Requires storing baseline findings per branch or computing a diff against the base branch's last validation run.
54+
55+
### 4.2 Convergence Detection
56+
Track finding count across iterations. If the count isn't decreasing after N rounds, warn the user and offer a graceful override rather than a hard block.
57+
58+
**Effort:** Low. Requires a counter in the CI workflow and a conditional pass.
59+
60+
### 4.3 Severity-Based Gating
61+
Only gate on CRITICAL findings. HIGH/MEDIUM/LOW are reported as advisory annotations on the PR but don't block merging.
62+
63+
**Effort:** Low. Workflow change only — check exit code against a severity threshold.
64+
65+
### 4.4 Finding Deduplication Across Runs
66+
Hash each finding's core attributes (location, description pattern, criterion) and suppress re-flagged findings that were present in the previous run. Only surface genuinely new findings.
67+
68+
**Effort:** Medium-High. Requires persistent storage of finding hashes across CI runs (cache or artifact).
69+
70+
### 4.5 Max Iteration Cap
71+
Allow N rounds of self-validation fixes. After N rounds, if findings remain, report them as known issues and allow merge with an annotation.
72+
73+
**Effort:** Low. Policy decision, not a code change.
74+
75+
## 5. Recommended Default
76+
77+
Until a proper delta mode is implemented:
78+
79+
- **Self-validation as advisory**, not required — report findings as PR comments but don't block merge
80+
- **Tests as the hard gate** — all unit/integration tests must pass
81+
- **CRITICAL-only blocking** — only CRITICAL self-validation findings block merge (§4.3)
82+
83+
This preserves the value of self-validation (visibility into code quality) without creating an unresolvable blocker.
84+
85+
## 6. References
86+
87+
- PR #8 (SharedIntellect/quorum) — first observed instance
88+
- SEC-03 §3.3 — Assurance Level Model (gating thresholds should scale with assurance level)
89+
- GRAD workflow (`.github/workflows/quorum-validate.yml`)

0 commit comments

Comments
 (0)