Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,23 @@ All notable changes to Quorum will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

### Added — Copilot CLI Port: Portable Critic Definitions (PRs #24, #25)

- **YAML-based portable critic definitions** — Evaluation logic extracted from Python critic classes into structured YAML files consumable by any Quorum port (CLI, Claude Code, Copilot). Each YAML contains: system prompt, evaluation categories with framework references, prompt template with variable placeholders, rubric keyword filter, output schema, and pre-screen integration rules.
- `critics/correctness.yaml` — 5 categories (COR-01–05): internal contradictions, logical consistency, factual claims, reference accuracy, terminology consistency
- `critics/security.yaml` — 14 categories (SEC-01–14) in 3 evaluation tiers, grounded in OWASP ASVS 5.0, CWE Top 25 (2024), NIST SP 800-53 SA-11
- `critics/completeness.yaml` — 5 categories (COM-01–05): missing sections, shallow treatment, edge cases, broken promises, requirement gaps
- `critics/code_hygiene.yaml` — 12 categories (CAT-01–12) + 6 agentic patterns (AP-01–06), mapped to ISO/IEC 25010:2023 and ISO/IEC 5055:2021 (CISQ)
- **`verdict-rules.yaml`** — Deterministic aggregation rules, deduplication threshold, and escalation logic as a standalone shared file
- **`rubrics/documentation.json`** — 12-criteria documentation quality rubric (DOC-001–DOC-012)
- **`learning/known_issues.json`** — Empty scaffold for pattern accumulation across validation runs
- **Rewritten SKILL.md orchestrator** — Full 4-critic orchestration with adaptive dispatch tiers (sequential default, opt-in concurrency), progress indicators, error degradation (DEGRADED/PARTIAL/QUORUM_FAILED), and delegation boundary between Code Hygiene and Security critics
- **Updated README.md** — Reflects new file layout and explains the portable critic definition architecture

---

## [0.7.3] — 2026-03-15

### Fixed — Pre-Existing Code Findings Cleanup
Expand Down
88 changes: 60 additions & 28 deletions ports/copilot-cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,67 +19,99 @@ cp -r quorum/ports/copilot-cli ~/.copilot/skills/quorum
"Run the security critic on auth.py"
"Check this config for completeness"

### Cross-artifact consistency
### Cross-artifact consistency (future)
"Check if implementation.py matches spec.md"

## What It Checks
- **Correctness** — Logic errors, contradictions, false claims
- **Completeness** — Missing sections, edge cases, broken promises
- **Security** — OWASP ASVS 5.0, CWE Top 25, framework-grounded
- **Code Hygiene** — ISO 25010/5055, structural quality beyond linting
- **Cross-Consistency** — Spec/impl, docs/code, schema contracts
- **Security** — OWASP ASVS 5.0, CWE Top 25, NIST SA-11, framework-grounded
- **Code Hygiene** — ISO 25010/5055, structural quality beyond linting, agentic code patterns

## Verdict Scale
- **PASS** — No issues found
- **PASS** — No issues found (or only INFO)
- **PASS_WITH_NOTES** — Minor issues only (MEDIUM/LOW)
- **REVISE** — HIGH severity issues requiring rework
- **REJECT** — CRITICAL issues found
- **REVISE** — Significant issues requiring rework (any CRITICAL, or 3+ HIGH)
- **REJECT** — Supervisor judgment: artifact fundamentally unsalvageable

## Requirements
- GitHub Copilot CLI with skill support
- Python 3.10+ (for pre-screen script)
- No additional dependencies
- No additional dependencies or API keys

## Skill Structure

```
~/.copilot/skills/quorum/
├── SKILL.md # Orchestration (Copilot reads this)
├── quorum-prescreen.py # Stdlib-only pre-screen script
├── critics/
│ ├── correctness.yaml # Portable critic definition
│ ├── completeness.yaml # Portable critic definition
│ ├── security.yaml # Portable critic definition (14 SEC categories)
│ ├── code_hygiene.yaml # Portable critic definition (12 CAT + 6 AP categories)
│ ├── correctness.agent.md # Direct single-critic invocation
│ ├── completeness.agent.md
│ ├── security.agent.md
│ ├── code-hygiene.agent.md
│ └── cross-consistency.agent.md
├── rubrics/
│ ├── python-code.json # 25 criteria for Python code
│ ├── research-synthesis.json # Research report criteria
│ └── agent-config.json # Config/YAML criteria
└── critics/
├── correctness.agent.md # Direct single-critic invocation
├── completeness.agent.md
├── security.agent.md
├── code-hygiene.agent.md
└── cross-consistency.agent.md
│ ├── documentation.json # 12 criteria for documentation
│ ├── agent-config.json # Config/YAML criteria
│ └── research-synthesis.json # Research report criteria
├── learning/
│ └── known_issues.json # Pattern accumulation (grows over time)
└── verdict-rules.yaml # Deterministic aggregation logic
```

### Portable Critic Definitions (YAML)

The `.yaml` critic files are the core innovation of this port. Each contains:
- **System prompt** — the critic's identity and constraints
- **Evaluation categories** — structured breakdown with framework references
- **Prompt template** — with variable placeholders for orchestrator-agnostic use
- **Output schema** — structured JSON format all critics return
- **Rubric keyword filter** — criteria selection logic
- **Pre-screen integration** — how deterministic checks feed into LLM judgment

These YAMLs are designed as the **single source of truth** consumable by any port (CLI, Claude Code, Copilot). The `.agent.md` files provide a simpler direct-invocation alternative for single-critic use.

## Dispatch Tiers

Sequential dispatch by default (safe for ≤16GB RAM devices). Users opt in to concurrency:

| Tier | Flag | Strategy |
|------|------|----------|
| 💡 Lightweight | default | Sequential (1 agent at a time) |
| ⚡ Standard | `--dispatch standard` | 2 concurrent agents |
| 🚀 Performance | `--dispatch performance` | All agents concurrent |

*Note: Adaptive dispatch is a Copilot port-specific adaptation. The reference implementation uses Python's ThreadPoolExecutor with a fixed worker count.*

## What Changes vs Reference Implementation
- LiteLLM provider -> Copilot `task` tool for parallel dispatch
- ThreadPoolExecutor -> Copilot parallel `task` calls
- CLI arguments -> natural language invocation
- Config YAML depth profiles -> SKILL.md instructions
- Python package -> file-based skill (SKILL.md + scripts + rubrics)
- Pydantic models -> plain JSON (in prescreen script)
- LiteLLM provider Copilot `task` tool for dispatch
- ThreadPoolExecutor Copilot parallel `task` calls (opt-in)
- CLI arguments natural language invocation
- Python class critic definitions → portable YAML definitions
- Config YAML depth profiles → SKILL.md instructions
- Python package → file-based skill (SKILL.md + scripts + rubrics)

## What Stays Identical
- Rubric schema and criteria content
- Finding JSON schema (FINDINGS_SCHEMA)
- Finding JSON schema
- Pre-screen check IDs (PS-001 through PS-010)
- Verdict taxonomy (PASS / PASS_WITH_NOTES / REVISE / REJECT)
- Severity levels (CRITICAL / HIGH / MEDIUM / LOW / INFO)
- Evidence grounding requirement
- Critic system prompts (verbatim)
- Critic evaluation logic (extracted verbatim from Python classes)

## What Is NOT Included in This Port
- **Tester (Phase 3)** — verification requires filesystem access patterns that differ in Copilot
- **Fix loops** — Copilot's edit model differs from file-write patterns
- **Learning memory** — future enhancement
## Not Yet Ported
- **Tester (L1/L2)** — verification requires filesystem access patterns that differ in Copilot
- **Fixer** — automated remediation; Copilot's edit model differs
- **Batch mode** — Copilot invocations are per-file
- **Cost tracking** — Copilot uses premium requests, not per-token billing
- **Cross-artifact relationship critic** — parameter reserved, not yet implemented

## References
- [Reference Implementation](../../reference-implementation/)
Expand Down
Loading