diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index dab1b63..004fcab 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -4,6 +4,23 @@ All notable changes to Quorum will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [Unreleased] + +### Added — Copilot CLI Port: Portable Critic Definitions (PRs #24, #25) + +- **YAML-based portable critic definitions** — Evaluation logic extracted from Python critic classes into structured YAML files consumable by any Quorum port (CLI, Claude Code, Copilot). Each YAML contains: system prompt, evaluation categories with framework references, prompt template with variable placeholders, rubric keyword filter, output schema, and pre-screen integration rules. + - `critics/correctness.yaml` — 5 categories (COR-01–05): internal contradictions, logical consistency, factual claims, reference accuracy, terminology consistency + - `critics/security.yaml` — 14 categories (SEC-01–14) in 3 evaluation tiers, grounded in OWASP ASVS 5.0, CWE Top 25 (2024), NIST SP 800-53 SA-11 + - `critics/completeness.yaml` — 5 categories (COM-01–05): missing sections, shallow treatment, edge cases, broken promises, requirement gaps + - `critics/code_hygiene.yaml` — 12 categories (CAT-01–12) + 6 agentic patterns (AP-01–06), mapped to ISO/IEC 25010:2023 and ISO/IEC 5055:2021 (CISQ) +- **`verdict-rules.yaml`** — Deterministic aggregation rules, deduplication threshold, and escalation logic as a standalone shared file +- **`rubrics/documentation.json`** — 12-criteria documentation quality rubric (DOC-001–DOC-012) +- **`learning/known_issues.json`** — Empty scaffold for pattern accumulation across validation runs +- **Rewritten SKILL.md orchestrator** — Full 4-critic orchestration with adaptive dispatch tiers (sequential default, opt-in concurrency), progress indicators, error degradation (DEGRADED/PARTIAL/QUORUM_FAILED), and delegation boundary between Code Hygiene and Security critics +- **Updated README.md** — Reflects new file layout and explains the portable critic definition architecture + +--- + ## [0.7.3] — 2026-03-15 ### Fixed — Pre-Existing Code Findings Cleanup diff --git a/ports/copilot-cli/README.md b/ports/copilot-cli/README.md index c7e4279..90070e7 100644 --- a/ports/copilot-cli/README.md +++ b/ports/copilot-cli/README.md @@ -19,26 +19,25 @@ cp -r quorum/ports/copilot-cli ~/.copilot/skills/quorum "Run the security critic on auth.py" "Check this config for completeness" -### Cross-artifact consistency +### Cross-artifact consistency (future) "Check if implementation.py matches spec.md" ## What It Checks - **Correctness** — Logic errors, contradictions, false claims - **Completeness** — Missing sections, edge cases, broken promises -- **Security** — OWASP ASVS 5.0, CWE Top 25, framework-grounded -- **Code Hygiene** — ISO 25010/5055, structural quality beyond linting -- **Cross-Consistency** — Spec/impl, docs/code, schema contracts +- **Security** — OWASP ASVS 5.0, CWE Top 25, NIST SA-11, framework-grounded +- **Code Hygiene** — ISO 25010/5055, structural quality beyond linting, agentic code patterns ## Verdict Scale -- **PASS** — No issues found +- **PASS** — No issues found (or only INFO) - **PASS_WITH_NOTES** — Minor issues only (MEDIUM/LOW) -- **REVISE** — HIGH severity issues requiring rework -- **REJECT** — CRITICAL issues found +- **REVISE** — Significant issues requiring rework (any CRITICAL, or 3+ HIGH) +- **REJECT** — Supervisor judgment: artifact fundamentally unsalvageable ## Requirements - GitHub Copilot CLI with skill support - Python 3.10+ (for pre-screen script) -- No additional dependencies +- No additional dependencies or API keys ## Skill Structure @@ -46,40 +45,73 @@ cp -r quorum/ports/copilot-cli ~/.copilot/skills/quorum ~/.copilot/skills/quorum/ ├── SKILL.md # Orchestration (Copilot reads this) ├── quorum-prescreen.py # Stdlib-only pre-screen script +├── critics/ +│ ├── correctness.yaml # Portable critic definition +│ ├── completeness.yaml # Portable critic definition +│ ├── security.yaml # Portable critic definition (14 SEC categories) +│ ├── code_hygiene.yaml # Portable critic definition (12 CAT + 6 AP categories) +│ ├── correctness.agent.md # Direct single-critic invocation +│ ├── completeness.agent.md +│ ├── security.agent.md +│ ├── code-hygiene.agent.md +│ └── cross-consistency.agent.md ├── rubrics/ │ ├── python-code.json # 25 criteria for Python code -│ ├── research-synthesis.json # Research report criteria -│ └── agent-config.json # Config/YAML criteria -└── critics/ - ├── correctness.agent.md # Direct single-critic invocation - ├── completeness.agent.md - ├── security.agent.md - ├── code-hygiene.agent.md - └── cross-consistency.agent.md +│ ├── documentation.json # 12 criteria for documentation +│ ├── agent-config.json # Config/YAML criteria +│ └── research-synthesis.json # Research report criteria +├── learning/ +│ └── known_issues.json # Pattern accumulation (grows over time) +└── verdict-rules.yaml # Deterministic aggregation logic ``` +### Portable Critic Definitions (YAML) + +The `.yaml` critic files are the core innovation of this port. Each contains: +- **System prompt** — the critic's identity and constraints +- **Evaluation categories** — structured breakdown with framework references +- **Prompt template** — with variable placeholders for orchestrator-agnostic use +- **Output schema** — structured JSON format all critics return +- **Rubric keyword filter** — criteria selection logic +- **Pre-screen integration** — how deterministic checks feed into LLM judgment + +These YAMLs are designed as the **single source of truth** consumable by any port (CLI, Claude Code, Copilot). The `.agent.md` files provide a simpler direct-invocation alternative for single-critic use. + +## Dispatch Tiers + +Sequential dispatch by default (safe for ≤16GB RAM devices). Users opt in to concurrency: + +| Tier | Flag | Strategy | +|------|------|----------| +| 💡 Lightweight | default | Sequential (1 agent at a time) | +| ⚡ Standard | `--dispatch standard` | 2 concurrent agents | +| 🚀 Performance | `--dispatch performance` | All agents concurrent | + +*Note: Adaptive dispatch is a Copilot port-specific adaptation. The reference implementation uses Python's ThreadPoolExecutor with a fixed worker count.* + ## What Changes vs Reference Implementation -- LiteLLM provider -> Copilot `task` tool for parallel dispatch -- ThreadPoolExecutor -> Copilot parallel `task` calls -- CLI arguments -> natural language invocation -- Config YAML depth profiles -> SKILL.md instructions -- Python package -> file-based skill (SKILL.md + scripts + rubrics) -- Pydantic models -> plain JSON (in prescreen script) +- LiteLLM provider → Copilot `task` tool for dispatch +- ThreadPoolExecutor → Copilot parallel `task` calls (opt-in) +- CLI arguments → natural language invocation +- Python class critic definitions → portable YAML definitions +- Config YAML depth profiles → SKILL.md instructions +- Python package → file-based skill (SKILL.md + scripts + rubrics) ## What Stays Identical - Rubric schema and criteria content -- Finding JSON schema (FINDINGS_SCHEMA) +- Finding JSON schema - Pre-screen check IDs (PS-001 through PS-010) - Verdict taxonomy (PASS / PASS_WITH_NOTES / REVISE / REJECT) - Severity levels (CRITICAL / HIGH / MEDIUM / LOW / INFO) - Evidence grounding requirement -- Critic system prompts (verbatim) +- Critic evaluation logic (extracted verbatim from Python classes) -## What Is NOT Included in This Port -- **Tester (Phase 3)** — verification requires filesystem access patterns that differ in Copilot -- **Fix loops** — Copilot's edit model differs from file-write patterns -- **Learning memory** — future enhancement +## Not Yet Ported +- **Tester (L1/L2)** — verification requires filesystem access patterns that differ in Copilot +- **Fixer** — automated remediation; Copilot's edit model differs - **Batch mode** — Copilot invocations are per-file +- **Cost tracking** — Copilot uses premium requests, not per-token billing +- **Cross-artifact relationship critic** — parameter reserved, not yet implemented ## References - [Reference Implementation](../../reference-implementation/)