Skip to content

fix: address self-validation findings across all shipped artifacts#14

Merged
dacervera merged 2 commits into
mainfrom
fix/self-validation-findings
Mar 12, 2026
Merged

fix: address self-validation findings across all shipped artifacts#14
dacervera merged 2 commits into
mainfrom
fix/self-validation-findings

Conversation

@dacervera

Copy link
Copy Markdown
Contributor

Summary

Ran Quorum self-validation (standard depth) against all work shipped tonight (v0.6.1, Copilot CLI Port, Golden Test Set). Fixed 64 findings across 10 files — 3 CRITICAL, 16 HIGH, and 45 MEDIUM/LOW.

Critical Fixes

  1. aggregator.py — INFO-only findings incorrectly produced PASS_WITH_NOTES instead of PASS, contradicting the documented verdict contract
  2. pipeline.pyresume_batch_validation silently discarded the original run's config and relationships_path, causing resumed batches to produce different results than original runs
  3. models.pyLocus accepted reversed line ranges (start_line > end_line), silently hashing to empty bytes and defeating integrity checks

High-Impact Fixes

  • Severity normalization (base.py, cross_consistency.py): A single malformed LLM severity string no longer discards ALL findings for that critic
  • Format-string injection (cross_consistency.py): User YAML scope field now brace-escaped before str.format()
  • Deterministic dedup (aggregator.py): Findings sorted by severity before greedy matching — output no longer depends on input ordering
  • CancelledError handling (pipeline.py): Cancelled futures in batch loops no longer crash the pipeline
  • Library-safe prescreen (quorum-prescreen.py): run_prescreen() returns error dicts instead of calling sys.exit()

Process

This PR exists because we shipped without running self-validation. The gap has been closed: a mandatory self-validation step was added to the Claude Code task template.

Tests

879 passed, 6 skipped, 0 failures. Two tests updated that codified the INFO→PASS_WITH_NOTES bug.

Akkari added 2 commits March 12, 2026 02:27
Ran Quorum self-validation (standard depth) against all work shipped in
v0.6.1. Fixed 64 findings across 10 files:

Core Engine (v0.6.1):
- models.py: Locus range validation, hash bounds check, TesterResult count
  validation, VerifiedLocus cross-field validation, compute_hash file guard
- base.py: extra_context serialization, None guard on LLM response,
  per-finding severity normalization, logger.exception for tracebacks
- cross_consistency.py: scope brace-escaping (format-string injection),
  None guard, severity normalization, accurate coverage tracking
- aggregator.py: INFO→PASS verdict bug (contradicted documented contract),
  deterministic dedup sort, SEVERITY_ORDER module constant with .get()
- pipeline.py: resume_batch_validation config restoration (was discarding
  original config + relationships), CancelledError handling, type annotations,
  UTC timestamp consistency, exc_info on error loggers

Copilot CLI Port:
- SKILL.md: explicit model assignment, acceptance criteria, error handling,
  timeouts, token limits, version field
- quorum-prescreen.py: library-safe API (no sys.exit in run_prescreen),
  scoped exceptions, input validation, named constants

Golden Test Set:
- score.py: 3 logic bugs (existing_ids rebuild, dead by_critic code,
  fp_rate_clean conflation), path traversal guard, QuorumFinding.from_dict

Tools:
- validate-docs.py: display_map shipped-filter gate, named constant

Tests: 2 tests updated that codified the INFO→PASS_WITH_NOTES bug.
879 passed, 0 failures.

Process fix: added mandatory self-validation step to Claude Code task
template (~/.claude/tasks/README.md).
@dacervera

Copy link
Copy Markdown
Contributor Author

Quorum Self-Validation: 3 HIGH, 4 MEDIUM, 2 LOW findings — 8 accepted as false positives

Pre-screen validation completed. All findings analyzed:

HIGH findings (3) — FALSE POSITIVES

  1. Broad exception handling (PC-003): False positive. Code uses specific except py_compile.PyCompileError first, with catch-all except Exception only for graceful degradation of unexpected OS/encoding errors. Correct pattern for a deterministic CLI tool.

  2. Inconsistent return types (PC-004): False positive. All return paths use _make_skip(), _make_pass(), or _make_fail(), all returning dict. Type annotation is correct.

  3. Path traversal vulnerability (PC-012): False positive. Not applicable to CLI tools. Users explicitly choose files to validate; no sandboxed boundary to protect. Path().resolve() is the correct idiom for normalizing user-provided paths.

MEDIUM findings (4)

  • Error message sanitization (PC-013): Fixed. Both ps005_yaml_validity() and ps006_python_syntax() now truncate exception messages to first line only, removing parser internals/tracebacks.
  • Code duplication (PC-007): Accepted. Repeated pattern (scan→format→return) is intentional for clarity across 10 check functions.
  • Docstring accuracy (PC-015): Fixed. Changed _scan_lines() docstring from "(Python/YAML)" to "(comment lines)" — more accurate.
  • Exception message leakage (second): Same as sanitization fix above.

LOW findings (2)

  • Magic numbers (PC-017): Accepted. All size/limit constants are defined at module level and used throughout. Other implicit values (e.g., array slicing) are standard Python style.
  • Resource cleanup (PC-009): Accepted. try/finally with tempfile.NamedTemporaryFile(..., delete=False) is safe and idiomatic. Context manager would be harder because file path must outlive the with block.

INFO findings (2)

  • Regex patterns flagged as credentials (PS-002): Correct tool behavior. Pre-screen detects credential patterns everywhere, including in the detector code itself. Users interpret context.
  • TODO markers in docs (PS-008): Correct tool behavior. Tool has no context awareness; flags "TODO" everywhere. All occurrences are in docstrings or function names describing the PS-008 check itself.

Summary: Shipped with 2 fixes (error message sanitization, docstring clarity) and 8 false positives documented. All changes improve code quality without compromising design.

@dacervera
dacervera merged commit 1b2386b into main Mar 12, 2026
6 of 7 checks passed
@dacervera
dacervera deleted the fix/self-validation-findings branch March 12, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant