Skip to content

feat: Copilot CLI port — Quorum validation as a GitHub Copilot skill#13

Merged
dacervera merged 2 commits into
mainfrom
feature/copilot-cli-port
Mar 12, 2026
Merged

feat: Copilot CLI port — Quorum validation as a GitHub Copilot skill#13
dacervera merged 2 commits into
mainfrom
feature/copilot-cli-port

Conversation

@dacervera

Copy link
Copy Markdown
Contributor

First Platform Port

Packages Quorum's multi-critic validation framework as a GitHub Copilot CLI skill. Zero infrastructure, zero pip dependencies — copy the ports/copilot-cli/ directory into any repo and point Copilot at SKILL.md.

What's Included

File Lines Purpose
SKILL.md 414 Orchestrator — classification, rubric selection, prescreen, critic dispatch, verdict, report
quorum-prescreen.py 649 Stdlib-only pre-screen (PS-001–PS-010). PyYAML optional for YAML support
critics/*.agent.md (5) 845 Verbatim critic prompts: correctness, completeness, security, code hygiene, cross-consistency
rubrics/*.json (3) 384 Byte-identical copies from reference implementation
README.md 86 Install + usage guide
ARCHITECTURE.md 123 Port design rationale and mapping to reference implementation

Design Decisions

  • Verbatim prompts: Critic prompts are extracted directly from the reference implementation source. Prompt drift between ports would undermine substantiation.
  • Stdlib-only prescreen: Copilot CLI environments may not have pip dependencies. All 10 checks run with zero imports beyond stdlib. PyYAML is optional (graceful degradation for YAML files).
  • Rubric copies over dynamic loading: Static copies for reliability — Copilot CLI file resolution is environment-dependent.
  • Multi-file pattern: Orchestrator + per-critic files matches how Copilot CLI loads context. Single monolithic file would hit context limits.

Also Changed

  • .gitignore: Removed ports/ exclusion — ports now tracked for public release.

Verification

All checks passed during build:

  • Rubrics byte-identical to reference ✅
  • Prescreen self-test (valid JSON, 10 checks, correct extension gating) ✅
  • Critic prompts verified verbatim against reference (5/5) ✅
  • validate-docs.py clean ✅
  • No boundary scan findings in port files ✅

Akkari and others added 2 commits March 12, 2026 01:01
First platform port. 12 files (10 new + 2 pre-existing), 2,378 lines.
Stdlib-only prescreen (PS-001–PS-010), 5 critic agent files with
verbatim prompts, 3 byte-identical rubric copies. Zero pip dependencies.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Quorum CI finding #8: stat() before read_text() to prevent memory
exhaustion on oversized files. The len() check after read was too late.
@dacervera

Copy link
Copy Markdown
Contributor Author

Quorum Self-Validation: 1 file scanned (quorum-prescreen.py), REVISE. 1 genuine finding (fixed), 9 false positives, 3 L1-excluded by Tester.

Genuine Finding (fixed in follow-up commit c2fed2e)

  1. [LOW] PC-012: File read before size validation — ✅ GENUINE. target_path.read_text() was called before MAX_ARTIFACT_SIZE check, meaning a multi-GB file would exhaust memory before the size limit triggered. Fixed: stat().st_size check before read_text().

False Positives

  1. [HIGH] PC-003: Broad except Exception in ps006_python_syntax — Intentional. The prescreen is a resilience-first tool — one check crashing must not kill the entire run. The except block calls _make_skip() which preserves the failure context (check ID, error message, SKIP status). This is the documented pattern for all 10 checks.

  2. [HIGH] PC-004: ps005_yaml_validity returns inconsistent types — FALSE. All three return paths (_make_skip, _make_pass, _make_fail) produce the identical dict schema: {id, name, category, status, details, evidence?, locations?}. The critic confused different status values (SKIP/PASS/FAIL) with different return types.

  3. [HIGH] PC-012: sys.argv path not validatedPath(sys.argv[1]).resolve() canonicalizes the path. This is a local developer CLI tool, not a network service. Path traversal is not in the threat model. The tool runs on the user's own filesystem with their own permissions.

  4. [MEDIUM] PC-025: ps010 mixes I/O with logic — The check's purpose is detecting empty files. Reading the file IS the logic. Separating I/O from "is this empty" would be over-abstraction.

  5. [MEDIUM] PC-015: _scan_lines docstring inaccuracy — Pedantic. The docstring describes the function's purpose; the comment about exclude_comments behavior could be more precise but doesn't affect correctness. LOW at best.

  6. [MEDIUM] PC-009: Temp file cleanup with missing_ok=TruePath.unlink(missing_ok=True) in a finally block is the stdlib idiom for safe cleanup. "Silently ignores cleanup failures" is exactly the intended behavior — the temp file may have already been cleaned up by the OS or a prior code path.

  7. [LOW] PC-017: Magic numbers on lines 25-26 — FALSE. Lines 25-26 ARE the named constants: MAX_ARTIFACT_SIZE = 500_000 and MAX_LINE_LENGTH = 10_000. The critic flagged the constant definitions themselves as magic numbers.

Bootstrap False Positives (prescreen flags itself)

  • PS-002 (credentials): Regex patterns for credential detection ((?:password|passwd|pwd)... on line 60, _RE_BASE64_SECRET on line 70) matched themselves. The tool's own detection patterns are not credentials.
  • PS-008 (TODO markers): Function names, docstrings, and regex patterns describing TODO detection functionality matched the TODO pattern. The tool's purpose includes finding TODOs — referencing "TODO" is unavoidable.

Summary: 10 findings + 2 bootstrap FPs. 1 genuine (file size check order, fixed). 9 false positives. 3 L1-excluded by Tester. Cost: $0.15.

@dacervera
dacervera merged commit c638202 into main Mar 12, 2026
6 of 7 checks passed
@dacervera
dacervera deleted the feature/copilot-cli-port branch March 12, 2026 08:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant