Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
docs/external-reviews/
ports/
__pycache__/
.pytest_cache/
.hypothesis/
Expand Down
14 changes: 14 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,20 @@ All notable changes to Quorum will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [0.7.0-port.1] — 2026-03-12

### Added
- **Copilot CLI port** at `ports/copilot-cli/` — Quorum's multi-critic validation framework packaged as a GitHub Copilot CLI skill. Zero infrastructure, zero pip dependencies.
- `SKILL.md` (414 lines): Orchestrator skill — artifact classification, rubric selection, prescreen gating, critic dispatch, verdict aggregation, report generation.
- `quorum-prescreen.py` (649 lines): Stdlib-only pre-screen implementing PS-001 through PS-010. PyYAML optional for YAML artifact support.
- 5 critic `.agent.md` files: Verbatim prompts for correctness, completeness, security, code hygiene, and cross-consistency critics.
- 3 rubric JSON files: Byte-identical copies from reference implementation (python-code, research-synthesis, agent-config).
- `README.md`: Install and usage guide.
- `ARCHITECTURE.md`: Port design rationale and mapping to reference implementation.

### Changed
- `.gitignore`: Removed `ports/` exclusion — ports are now tracked for public release.

## [0.7.0] — 2026-03-12

### Added
Expand Down
123 changes: 123 additions & 0 deletions ports/copilot-cli/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Copilot CLI Port — Architecture Mapping

How Quorum's reference implementation components map to Copilot CLI primitives.

---

## Component-Level Mapping

### Supervisor → SKILL.md Orchestration

The reference implementation's `supervisor.py` selects critics based on depth profile and artifact type. In the Copilot port, this logic moves into SKILL.md as natural language instructions that Copilot follows to orchestrate the validation flow.

```
Reference: supervisor.py → selects critics → ThreadPoolExecutor → collects results
Copilot: SKILL.md instructions → spawns task agents → collects responses
```

### Critics → Task Agents

Each critic's system prompt + rubric injection becomes a `task` agent call:

```python
# Reference implementation (simplified)
critic_result = litellm.completion(
model=config.tier2_model,
messages=[system_prompt + rubric + artifact]
)

# Copilot CLI equivalent (conceptual)
critic_result = task(
agent_type="explore", # or "general-purpose" for security
model="sonnet",
prompt=system_prompt + rubric + artifact
)
```

### Pre-Screen → Script Execution

The pre-screen layer runs identically — it's already a Python script that shells out to external tools:

```
Reference: quorum run → pre_screen.py → [DevSkim, Ruff, Bandit, PSSA]
Copilot: SKILL.md → python quorum-prescreen.py → [DevSkim, Ruff, Bandit, PSSA]
```

**Key difference:** Pre-screen must be stdlib-only for the port. The reference implementation uses pyyaml for config parsing — make it optional with JSON fallback.

### Aggregator → SKILL.md Logic or Dedicated Agent

The aggregator merges findings, resolves conflicts, and assigns verdicts. Two options:

1. **Inline in SKILL.md** — aggregation logic as natural language instructions. Simpler, fewer premium requests.
2. **Dedicated agent** — `general-purpose` agent with aggregator prompt. More robust for complex conflict resolution.

Recommendation: Start with option 1, move to option 2 if quality suffers.

### Fixer → General-Purpose Agent

The fixer needs write access to propose changes:

```
Reference: fixer.py → reads findings + artifact → proposes text replacements → writes fix-proposals.json
Copilot: general-purpose task agent → reads findings + artifact → proposes changes → writes to disk
```

### Learning Memory → File Persistence

`known_issues.json` is read at session start and written after verdict. Files persist across Copilot sessions — no changes needed to the data format.

### Cost Tracking → Not Applicable

LiteLLM cost tracking doesn't apply. Copilot uses premium requests, not per-token billing. The port should track premium request count instead.

### Batch Validation → Sequential Task Dispatch

The reference implementation's batch mode processes multiple files with crash-resilient progressive saves. In Copilot:

```
Reference: BatchVerdict → ThreadPoolExecutor(max_workers=3) → progressive manifest
Copilot: SKILL.md loops over files → sequential or parallel task dispatch → progressive file writes
```

---

## Prompt Extraction Guide

When extracting critic prompts from Python classes, preserve:

1. **System prompt** — the critic's identity, evaluation criteria, and output format requirements
2. **Rubric injection** — how rubric criteria are formatted and injected into the prompt
3. **Evidence grounding instruction** — the requirement to cite specific locus (file, line, excerpt) for every finding
4. **Output schema** — the JSON structure critics must return (findings array with severity, location, criterion, evidence)

Do NOT extract:
- LiteLLM-specific code (model routing, retry logic, cost tracking)
- ThreadPoolExecutor orchestration (replaced by `task` agents)
- File I/O wrappers (Copilot agents handle file operations natively)

---

## Testing Strategy

### Equivalence Testing

For each critic, run the same artifact through both:
1. Reference implementation (`quorum run --target <file> --depth standard`)
2. Copilot CLI port (natural language invocation)

Compare: finding count, severity distribution, evidence quality, verdict. They should converge but won't be identical (different model access patterns, prompt formatting).

### Regression Artifacts

Use the reference implementation's existing test artifacts:
- `examples/sample-research.md` — planted flaws for research validation
- `examples/bad-config.yaml` — planted security + completeness issues
- Golden set artifacts (if graduated to public)

### Platform-Specific Testing

- Windows native PowerShell (primary)
- WSL (secondary)
- macOS Terminal (tertiary)
- Linux (tertiary)
86 changes: 86 additions & 0 deletions ports/copilot-cli/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Quorum — Copilot CLI Skill

Multi-critic quality validation for code, configs, and documentation.

## Install

```bash
git clone https://github.com/SharedIntellect/quorum
cp -r quorum/ports/copilot-cli ~/.copilot/skills/quorum
```

## Usage

### Full validation (all critics)
"Run Quorum validation on this file"
"Validate api_handler.py with Quorum"

### Single critic
"Run the security critic on auth.py"
"Check this config for completeness"

### Cross-artifact consistency
"Check if implementation.py matches spec.md"

## What It Checks
- **Correctness** — Logic errors, contradictions, false claims
- **Completeness** — Missing sections, edge cases, broken promises
- **Security** — OWASP ASVS 5.0, CWE Top 25, framework-grounded
- **Code Hygiene** — ISO 25010/5055, structural quality beyond linting
- **Cross-Consistency** — Spec/impl, docs/code, schema contracts

## Verdict Scale
- **PASS** — No issues found
- **PASS_WITH_NOTES** — Minor issues only (MEDIUM/LOW)
- **REVISE** — HIGH severity issues requiring rework
- **REJECT** — CRITICAL issues found

## Requirements
- GitHub Copilot CLI with skill support
- Python 3.10+ (for pre-screen script)
- No additional dependencies

## Skill Structure

```
~/.copilot/skills/quorum/
├── SKILL.md # Orchestration (Copilot reads this)
├── quorum-prescreen.py # Stdlib-only pre-screen script
├── rubrics/
│ ├── python-code.json # 25 criteria for Python code
│ ├── research-synthesis.json # Research report criteria
│ └── agent-config.json # Config/YAML criteria
└── critics/
├── correctness.agent.md # Direct single-critic invocation
├── completeness.agent.md
├── security.agent.md
├── code-hygiene.agent.md
└── cross-consistency.agent.md
```

## What Changes vs Reference Implementation
- LiteLLM provider -> Copilot `task` tool for parallel dispatch
- ThreadPoolExecutor -> Copilot parallel `task` calls
- CLI arguments -> natural language invocation
- Config YAML depth profiles -> SKILL.md instructions
- Python package -> file-based skill (SKILL.md + scripts + rubrics)
- Pydantic models -> plain JSON (in prescreen script)

## What Stays Identical
- Rubric schema and criteria content
- Finding JSON schema (FINDINGS_SCHEMA)
- Pre-screen check IDs (PS-001 through PS-010)
- Verdict taxonomy (PASS / PASS_WITH_NOTES / REVISE / REJECT)
- Severity levels (CRITICAL / HIGH / MEDIUM / LOW / INFO)
- Evidence grounding requirement
- Critic system prompts (verbatim)

## What Is NOT Included in This Port
- **Tester (Phase 3)** — verification requires filesystem access patterns that differ in Copilot
- **Fix loops** — Copilot's edit model differs from file-write patterns
- **Learning memory** — future enhancement
- **Batch mode** — Copilot invocations are per-file

## References
- [Reference Implementation](../../reference-implementation/)
- [Architecture Mapping](ARCHITECTURE.md)
Loading
Loading