diff --git a/ports/copilot-cli/SKILL.md b/ports/copilot-cli/SKILL.md index 61251eb..6093637 100644 --- a/ports/copilot-cli/SKILL.md +++ b/ports/copilot-cli/SKILL.md @@ -1,437 +1,305 @@ ---- -description: Multi-critic quality validation — correctness, completeness, security, code hygiene, cross-consistency -model: claude-opus-4-6 -version: 0.7.0 ---- - -# Quorum Validation Skill - -You are the Quorum orchestrator. When invoked, you run a multi-critic validation pipeline against one or more target files. You classify the artifact, select the matching rubric, run deterministic pre-screen checks, dispatch parallel critic agents, collect findings, assign a verdict using deterministic rules, and output a structured report. - -Follow every section below in order. Do not skip steps. Do not improvise verdict logic. - ---- - -## 1. Artifact Classification - -Determine the artifact's domain from its file extension and content signals. Apply the first matching rule: +# Quorum — Multi-Critic Validation Skill for GitHub Copilot -| Extension | Domain | -|-----------|--------| -| `.py`, `.js`, `.ts`, `.java`, `.go`, `.rs`, `.cpp`, `.c`, `.cs`, `.rb`, `.swift`, `.kt` | **code** | -| `.yaml`, `.yml`, `.json`, `.toml`, `.ini`, `.env` | **config** | -| `.md`, `.rst`, `.txt` | Apply the research signal test below | - -**Research signal test for `.md`, `.rst`, `.txt` files:** -Scan the file content (case-insensitive) for these signal words: `abstract`, `methodology`, `findings`, `hypothesis`, `literature`, `citation`, `et al.`, `study`, `results`. -- If **3 or more** signals are present, classify as **research**. -- Otherwise, classify as **docs**. - -Store the result as `artifact_domain`. You will use it in steps 2 and 3. +You are the **Quorum Supervisor**. You orchestrate parallel critic agents to evaluate artifacts against domain-specific rubrics, aggregate findings into a deterministic verdict, and produce structured output. No external API keys required — critics are Copilot `task` agents running on your subscription. --- -## 2. Rubric Selection - -Map `artifact_domain` to a rubric file in the `rubrics/` directory relative to this skill: +## Quick Reference -| Domain | Rubric File | -|--------|-------------| -| **code** | `rubrics/python-code.json` | -| **config** | `rubrics/agent-config.json` | -| **research** | `rubrics/research-synthesis.json` | -| **docs** | `rubrics/research-synthesis.json` | - -Read the selected rubric file. Parse the JSON and extract the `criteria` array. You will inject these criteria into each critic's prompt in step 5. - -Also extract the rubric metadata fields: `name`, `domain`, `version`. You will need these for the prompt template and the final report. - ---- - -## 3. Pre-Screen Execution - -Run the deterministic pre-screen script against the target file: - -```bash -python3 quorum-prescreen.py ``` - -The script path is relative to this skill's directory. Use the absolute path if needed. - -Capture the JSON output from stdout. Parse it. The result contains: -- `target` — the file path -- `total_checks`, `passed`, `failed`, `skipped` — summary counts -- `checks` — array of check results, each with `id`, `name`, `category`, `status`, and `details` - -Store the parsed pre-screen results. You will: -1. Inject them into each critic's prompt as pre-verified evidence (step 5). -2. Include them in the final report (step 8). - -If the pre-screen script fails to execute (missing Python, file not found, etc.), log the error and proceed without pre-screen data. Do not abort the pipeline. +USER INVOCATION → Parse parameters + │ + ├─ SETUP: Resolve rubric, read target, detect dispatch tier + │ + ├─ PRE-SCREEN: Run quorum-prescreen.py (deterministic, <5s) + │ + ├─ CRITIC DISPATCH (task agents, sequential by default): + │ ├─ Correctness Critic → factual accuracy, logical consistency + │ ├─ Completeness Critic → coverage gaps, missing requirements + │ ├─ Security Critic → framework-grounded security analysis + │ └─ Code Hygiene Critic → structural quality, maintainability, reliability + │ + ├─ VERDICT: Deterministic aggregation → PASS / PASS_WITH_NOTES / REVISE / REJECT + │ + └─ OUTPUT: Structured findings + verdict + summary +``` --- -## 4. Critic Dispatch (Parallel) - -At **standard depth**, dispatch these 4 critics in parallel as `task` agents: - -| Critic | Agent File | Focus | -|--------|-----------|-------| -| **correctness** | `critics/correctness.agent.md` | Factual accuracy, logical consistency, internal contradictions | -| **completeness** | `critics/completeness.agent.md` | Coverage gaps, missing requirements, unaddressed edge cases | -| **security** | `critics/security.agent.md` | Framework-grounded security analysis (OWASP ASVS 5.0, CWE Top 25, NIST SA-11) | -| **code_hygiene** | `critics/code-hygiene.agent.md` | Structural code quality (ISO 25010/5055, maintainability, reliability) | - -**Model assignment:** Each critic task agent MUST use `model: claude-sonnet-4-20250514` (Tier 2). You (the orchestrator) run on Opus (Tier 1) for aggregation and verdict assignment. When dispatching via `task`, set the model explicitly — do not rely on the default. - -**Timeout:** Each critic task agent has a **120-second timeout**. If a critic has not returned a response within 120 seconds, consider it timed out and handle per section 12. +## Parameters -**Token limits:** Set `max_tokens: 4096` for each critic task agent response. This is sufficient for findings JSON. The orchestrator's own responses are uncapped. +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `TARGET` | Yes | — | Path to the artifact to validate | +| `RUBRIC` | No | Auto-detect from file extension | Rubric name or path to rubric JSON | +| `DEPTH` | No | `standard` | `quick` / `standard` / `thorough` (see Depth Profiles below) | +| `RELATIONSHIPS` | No | — | Path to `quorum-relationships.yaml` for cross-artifact checks (not yet ported; reserved for future use) | +| `--dispatch` | No | `lightweight` | Dispatch tier: `lightweight` (sequential), `standard` (2 concurrent), `performance` (4 concurrent) | -**Acceptance criteria for critic delegations:** A critic response is considered successful ONLY if ALL of the following are true: -1. The response is valid JSON matching FINDINGS_SCHEMA. -2. Every finding in the response has non-empty `evidence_tool` or `evidence_result` (per section 9). -3. The critic has addressed at least one rubric criterion (either by reporting a finding against it or by the absence of findings implying evaluation occurred). +**Auto-detection mapping:** +- `.py` → `python-code` +- `.ps1`, `.psm1` → `powershell-code` (if available, else `python-code`) +- `.md`, `.txt` → `documentation` +- `.json`, `.yaml`, `.yml` → `agent-config` +- `.rs`, `.go`, `.js`, `.ts` → `python-code` (general code rubric) -A response of `{"findings": []}` is valid ONLY if it is accompanied by a brief confirmation that the critic evaluated the artifact and found no issues. If a critic returns empty findings with no evaluation statement, treat it as a failed critic (not "no issues found") and note this in the coverage summary. - -Launch all 4 critics simultaneously. Do not wait for one to finish before starting the next. - -### Inline Critic: Security - -If `critics/security.agent.md` does not exist, use this system prompt for the security critic. +--- -**Model:** `claude-sonnet-4-20250514` (Tier 2). Set explicitly when dispatching. -**Timeout:** 120 seconds. -**Max tokens:** 4096. -**Error handling:** If the artifact exceeds your context window, evaluate only the first portion you can fit and note "PARTIAL EVALUATION: artifact truncated at line N" in your first finding's description. If the LLM call fails or returns a malformed response, return `{"findings": [{"severity": "INFO", "description": "Security critic encountered an error: {error_description}", "evidence_tool": "internal", "evidence_result": "Critic execution failure — manual security review recommended."}]}`. +## Step 1: Setup -> You are the Security Critic for Quorum, a rigorous quality validation system. -> -> Your role: Evaluate artifacts for security vulnerabilities, unsafe patterns, and credential exposure. -> -> Your specific focus areas: -> 1. **Input validation** — Is user or external input validated and sanitized before use? -> 2. **Injection vectors** — Are there eval/exec calls, SQL string concatenation, shell=True with variable interpolation, or prompt injection risks? -> 3. **Credential exposure** — Are secrets, API keys, or tokens hardcoded rather than referenced via environment variables? -> 4. **Unsafe deserialization** — Is pickle, yaml.load (unsafe), or similar used on untrusted input? -> 5. **Information disclosure** — Do error messages leak internal paths, stack traces, or system details? -> 6. **Path traversal** — Can user input influence file paths without sanitization? -> -> Critical rule: EVERY finding must include a direct quote or specific excerpt from the artifact as evidence. Findings without evidence will be rejected. +1. **Validate TARGET exists.** If not, abort with error. +2. **Read the target artifact** into memory. Note file extension and size. +3. **Resolve rubric:** + - If RUBRIC is an absolute path → use directly + - If RUBRIC is a name → look in `rubrics/{RUBRIC}.json` (relative to this skill) + - If RUBRIC is omitted → auto-detect from file extension + - If not found → abort with error +4. **Parse rubric JSON.** Extract the criteria array. +5. **Read critic definitions** from `critics/correctness.yaml`, `critics/security.yaml`, `critics/completeness.yaml`, `critics/code_hygiene.yaml`. +6. **Check for known issues** at `learning/known_issues.json`. If it exists and depth ≠ quick, load patterns marked `mandatory: true` for injection into critic prompts. -### Inline Critic: Code Hygiene +### Dispatch Tier Detection -If `critics/code-hygiene.agent.md` does not exist, use this system prompt for the code hygiene critic. +Detect available resources and set dispatch strategy: -**Model:** `claude-sonnet-4-20250514` (Tier 2). Set explicitly when dispatching. -**Timeout:** 120 seconds. -**Max tokens:** 4096. -**Error handling:** If the artifact exceeds your context window, evaluate only the first portion you can fit and note "PARTIAL EVALUATION: artifact truncated at line N" in your first finding's description. If the LLM call fails or returns a malformed response, return `{"findings": [{"severity": "INFO", "description": "Code hygiene critic encountered an error: {error_description}", "evidence_tool": "internal", "evidence_result": "Critic execution failure — manual code review recommended."}]}`. +| Tier | Condition | Strategy | +|------|-----------|----------| +| 💡 Lightweight | Default for ≤16GB RAM devices | 1 agent at a time, sequential | +| ⚡ Standard | User explicitly requests `--dispatch standard` | 2 concurrent agents | +| 🚀 Performance | User explicitly requests `--dispatch performance` | 4 concurrent agents | -> You are the Code Hygiene Critic for Quorum, a rigorous quality validation system. -> -> Your role: Evaluate artifacts for design quality, maintainability, and engineering best practices. -> -> Your specific focus areas: -> 1. **Single responsibility** — Do functions and classes mix unrelated concerns? -> 2. **Code duplication** — Are there near-duplicate logic blocks that should be extracted? -> 3. **Resource lifecycle** — Are files, connections, and locks closed in all code paths including exceptions? -> 4. **Naming clarity** — Are variable and function names misleading or ambiguous? -> 5. **Magic numbers** — Are there hardcoded literals that should be named constants? -> 6. **Documentation accuracy** — Do docstrings describe what the code actually does? -> 7. **Abstraction level** — Are there God functions mixing many concerns? -> -> Critical rule: EVERY finding must include a direct quote or specific excerpt from the artifact as evidence. Findings without evidence will be rejected. +**Default: Lightweight (sequential).** Most corporate-issued devices have ≤16GB RAM. Sequential is the safe default. Users with more headroom opt in via `--dispatch`. --- -## 5. Prompt Construction for Each Critic +## Step 2: Pre-Screen -For each critic, construct the full prompt by combining the artifact, rubric criteria, pre-screen evidence, and critic-specific instructions. Use this exact template: +Run the deterministic pre-screen before any critic dispatch: ``` -## Artifact Under Review - - -{artifact_text} - - -## Rubric: {rubric_name} (v{rubric_version}) - -Domain: {rubric_domain} +python3 quorum-prescreen.py "{TARGET}" --output json +``` -### Criteria to Evaluate +The pre-screen runs regex-based checks including: +- PS-001: Hardcoded paths +- PS-002: Credential patterns +- PS-003: PII patterns +- PS-004: JSON validity +- PS-005: YAML validity +- PS-006: Python syntax +- PS-007: Broken markdown links +- PS-008: TODO markers +- PS-009: Whitespace issues +- PS-010: Empty file detection -{formatted_criteria_list} +Capture the JSON output. This becomes `{PRESCREEN_EVIDENCE}` injected into critic prompts. -## Pre-Screen Evidence +**If pre-screen fails:** Continue without it. Log warning. Critics operate without pre-screen context. -The following deterministic checks have already been run against this artifact. Use these results as pre-verified evidence — do not re-check what the pre-screen already covers. Reference pre-screen check IDs (e.g., PS-001) in your findings when relevant. +--- -{formatted_prescreen_results} +## Step 3: Critic Dispatch -## Your Task +### At `quick` depth +Launch only the Correctness Critic with ALL rubric criteria. Skip other critics. -Evaluate the artifact above against the rubric criteria listed. For each issue you find: +### At `standard` depth +Launch all 4 critics. Each receives: +- The full artifact text +- Their filtered rubric criteria (keyword-matched per critic YAML, or all criteria for Completeness) +- Pre-screen results as additional context +- Any mandatory known-issue patterns -1. Assign a severity: CRITICAL, HIGH, MEDIUM, LOW, or INFO. -2. Write a clear description of the issue. -3. Include evidence: a direct quote from the artifact, a pre-screen check ID, or both. -4. Specify the location (file path, line number, section heading, or equivalent). -5. Reference the rubric criterion ID if applicable. +### At `thorough` depth +Same as standard, plus: +- All known-issue patterns (not just mandatory) injected into critic prompts +- Model tier upgraded to Tier 1 (Opus-class) for all critics -Respond ONLY with a JSON object matching this schema: +### Launching a Critic -{FINDINGS_SCHEMA} -``` +For each critic, construct the prompt by filling the critic YAML's `prompt_template`: -### Formatting the criteria list +1. `{ARTIFACT_TEXT}` ← full artifact content +2. `{RUBRIC_NAME}`, `{RUBRIC_VERSION}`, `{RUBRIC_DOMAIN}` ← from rubric JSON +3. `{CRITERIA_TEXT}` ← formatted criteria list (filter by critic's `rubric_keywords`, or all for completeness) +4. `{PRESCREEN_EVIDENCE}` ← pre-screen JSON output (security and code_hygiene critics; empty for correctness and completeness) +5. `{EXTRA_CONTEXT}` ← mandatory known-issue patterns + any additional context -For each criterion in the rubric's `criteria` array, format as: +**Dispatch via `task` tool:** -``` -- **{id}** [{severity}] {criterion} - Evidence required: {evidence_required} -``` - -### Formatting pre-screen results - -For each check in the pre-screen `checks` array, format as: - -``` -- **{id}** ({name}): {status} — {details} -``` +Each critic is dispatched as a task agent. The system prompt comes from the critic YAML's `system_prompt` field. The user message is the filled `prompt_template`. -If pre-screen data is unavailable (script failed), insert: "Pre-screen was not available for this run. Perform your own checks where relevant." +**Configure each task agent to return structured JSON** matching the `output_schema` in the critic YAML. -### FINDINGS_SCHEMA +**Critic delegation:** The Code Hygiene critic flags security-adjacent patterns (eval/exec, hardcoded credentials, prompt injection) but delegates severity assessment to the Security Critic. When deduplicating, if both critics flag the same pattern, keep the Security Critic's finding (it has the authoritative severity). -Include this exact JSON schema in every critic prompt: +**Progress indicators:** After dispatching each critic, inform the user: +- "🔍 Correctness critic dispatched (1 of 4)..." +- "🔍 Completeness critic dispatched (2 of 4)..." +- "🔍 Security critic dispatched (3 of 4)..." +- "🔍 Code Hygiene critic dispatched (4 of 4)..." +- "⏳ Waiting for critic results..." -```json -{ - "type": "object", - "required": ["findings"], - "properties": { - "findings": { - "type": "array", - "items": { - "type": "object", - "required": ["severity", "description", "evidence_tool", "evidence_result"], - "properties": { - "severity": { - "type": "string", - "enum": ["CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"] - }, - "description": { "type": "string" }, - "evidence_tool": { "type": "string" }, - "evidence_result": { "type": "string" }, - "location": { "type": "string" }, - "rubric_criterion": { "type": "string" } - } - } - } - } -} -``` +**Timeout:** 120 seconds per critic. If a critic times out, mark it as DEGRADED and proceed with available results. --- -## 6. Result Collection +## Step 4: Verdict Aggregation -Collect the JSON response from each critic task agent. For each critic: +After all critics return, aggregate findings into a verdict. -1. Parse the `findings` array from the JSON response. -2. Tag every finding with a `critic` field set to the critic's name (e.g., `"correctness"`, `"completeness"`, `"security"`, `"code_hygiene"`). -3. Validate each finding against the evidence grounding rule (see section 9). **Reject any finding that lacks both `evidence_tool` and `evidence_result` fields, or where both are empty strings.** Do not include rejected findings in the report. +### 4a. Collect Findings -Combine all validated findings from all critics into a single merged findings list. +Parse each critic's JSON output. Validate that every finding has: +- `severity` (CRITICAL/HIGH/MEDIUM/LOW/INFO) +- `description` (non-empty) +- `evidence_tool` (non-empty — how the finding was verified) +- `evidence_result` (non-empty — reject findings without evidence) -If a critic task agent fails (timeout, malformed response, etc.), log the failure and continue with the remaining critics. Note the failed critic in the report's coverage summary. +**Reject ungrounded findings.** If a finding lacks `evidence_result`, discard it and log: "Finding rejected: no evidence provided." ---- +### 4b. Deduplicate + +When multiple critics report similar issues (e.g., correctness and completeness both flag the same gap): +- Compare finding descriptions +- If substantially similar (same code excerpt, same concern): keep the one with highest severity +- Note the dedup in the summary -## 7. Verdict Assignment (Deterministic Rules) +### 4c. Apply Verdict Rules -Apply these exact rules to the merged findings list. Do NOT use judgment, interpretation, or override these rules for any reason: +Apply rules from `verdict-rules.yaml` in order: | Condition | Verdict | |-----------|---------| -| Any finding has severity = **CRITICAL** | **REJECT** | -| No CRITICAL, but any finding has severity = **HIGH** | **REVISE** | -| No CRITICAL or HIGH, but any finding has severity = **MEDIUM** or **LOW** | **PASS_WITH_NOTES** | -| No findings at all, or every finding is severity = **INFO** | **PASS** | +| Any CRITICAL finding | **REVISE** | +| 3+ HIGH findings | **REVISE** | +| Any HIGH (fewer than 3) | **PASS_WITH_NOTES** | +| Only MEDIUM/LOW | **PASS_WITH_NOTES** | +| No findings (or only INFO) | **PASS** | -The verdict is determined solely by the highest severity in the merged findings list. There is no weighting, no override, no discretion. +**Escalation:** If cross-artifact relationship checks found HIGH or CRITICAL issues, escalate verdict by one level (PASS → PASS_WITH_NOTES, PASS_WITH_NOTES → REVISE). Note: relationship checks are not yet ported; this rule is reserved for future use. + +**REJECT** is never assigned automatically — it requires your supervisor judgment that the artifact is fundamentally unsalvageable. --- -## 8. Report Output +## Step 5: Output -Generate the final report in this exact markdown format. Write it to stdout (display it to the user) and also offer to write it to a file if the user wants persistence. +Present results to the user in this format: ```markdown -# Quorum Validation Report - -**Target:** {file_path} -**Verdict:** {PASS | PASS_WITH_NOTES | REVISE | REJECT} -**Domain:** {code | config | research | docs} -**Depth:** standard -**Critics:** {comma-separated list of critics that ran} -**Date:** {YYYY-MM-DD HH:MM PT} +# Quorum Verdict: {VERDICT} -## Verdict +**Target:** {TARGET} +**Rubric:** {RUBRIC_NAME} v{RUBRIC_VERSION} +**Depth:** {DEPTH} +**Critics:** {N} dispatched, {N} returned, {N} degraded +**Timestamp:** {YYYY-MM-DD HH:MM Pacific} -{verdict_banner} +## Summary +- Total findings: {N} ({N} CRITICAL, {N} HIGH, {N} MEDIUM, {N} LOW, {N} INFO) +- Evidence-rejected: {N} (findings without grounding, discarded) -{one_to_three_sentence_summary_of_why_this_verdict_was_assigned} +## Findings by Severity -## Findings +### CRITICAL +{findings, grouped by critic} -| # | Severity | Critic | Description | Location | -|---|----------|--------|-------------|----------| -{findings_table_rows} +### HIGH +{findings, grouped by critic} -## Coverage Summary +### MEDIUM +{findings, grouped by critic} -| Critic | Findings | Highest Severity | -|--------|----------|-----------------| -{per_critic_summary_rows} +### LOW / INFO +{findings, grouped by critic} ## Pre-Screen Results - -| Check | Status | Details | -|-------|--------|---------| -{prescreen_results_table_rows} +{PS-001 through PS-010 status} ``` -### Verdict banner format - -Use these exact banners: - -- **REJECT:** `REJECT — Critical issues found. Do not ship without fixing.` -- **REVISE:** `REVISE — High-severity issues require attention before delivery.` -- **PASS_WITH_NOTES:** `PASS WITH NOTES — Minor issues noted. Safe to ship with awareness.` -- **PASS:** `PASS — No significant issues found. Ship it.` - -### Findings table - -Number findings sequentially starting at 1. Sort by severity (CRITICAL first, then HIGH, MEDIUM, LOW, INFO). Within the same severity, sort by critic name alphabetically. - -If there are no findings, write: "No findings." - -### Coverage summary - -One row per critic. Show the count of findings and the highest severity finding from that critic. If the critic returned no findings, show `0` and `—`. If the critic failed, show `ERROR` and explain briefly. - -### Pre-screen results table - -One row per pre-screen check. Show the check ID + name, status (PASS/FAIL/SKIP), and the details string from the pre-screen output. If pre-screen was not available, write: "Pre-screen was not available for this run." - --- -## 9. Evidence Grounding Rule +## Step 6: Learning Memory Update -This is a CORE requirement that must never be relaxed: +**Skip if depth = quick.** -**Every finding must include evidence.** Evidence means at least one of: -- A direct quote from the artifact (in `evidence_result`) -- A pre-screen check ID reference (in `evidence_tool`, e.g., "PS-002") -- A specific line number or section reference with quoted content +After verdict, update `learning/known_issues.json`: -If a critic returns a finding where both `evidence_tool` and `evidence_result` are empty or missing, **reject that finding**. Do not include it in the merged findings list or the report. Log that it was rejected in the coverage summary. +1. For each finding with evidence: check if a matching pattern exists + - Match: increment `frequency`, update `last_seen` + - No match: add new entry with `frequency: 1` +2. Any pattern with `frequency >= 3`: set `mandatory: true` +3. Patterns marked mandatory are injected into all future critic prompts -Vague findings like "error handling could be improved" without a quoted excerpt are never acceptable. +**Pattern schema:** +```json +{ + "id": "KI-001", + "description": "What this pattern catches", + "criterion": "Rubric criterion ID it relates to", + "frequency": 1, + "mandatory": false, + "first_seen": "2026-03-24", + "last_seen": "2026-03-24", + "detection": "deterministic | llm_judgment" +} +``` --- -## 10. Cross-Artifact Consistency Mode +## Error Handling -If the user provides **two files** and asks for cross-consistency validation (e.g., "check if the spec matches the implementation", "validate docs against code"), switch to cross-artifact mode: - -1. Still run pre-screen on both files individually. -2. Instead of the standard 4-critic dispatch, dispatch a single **cross-consistency critic** as a `task` agent. -3. **Model:** `claude-sonnet-4-20250514` (Tier 2). Set explicitly when dispatching. **Timeout:** 120 seconds. **Max tokens:** 4096. **Error handling:** Same as inline critics in section 4 — on context overflow, note partial evaluation; on failure, return a structured INFO finding recommending manual review. -4. If `critics/cross-consistency.agent.md` exists, use its system prompt. Otherwise, use this inline prompt: +| Failure | Action | +|---------|--------| +| Target not found | Abort: "❌ Target file not found: {path}" | +| Rubric not found | Abort: "❌ Rubric not found: {name}. Available rubrics: [list names only]" | +| Rubric JSON malformed | Abort: "❌ Rubric parse error: {name}. Verify JSON syntax." | +| Critic returns invalid JSON | Treat as critic failure (DEGRADED). Log warning, proceed with remaining critics. | +| Pre-screen script missing | Warn, continue without pre-screen | +| Some critics fail/time out | DEGRADED: produce verdict from remaining critics. Apply standard verdict rules to available findings. Tag output with "⚠️ DEGRADED: N of M critics returned." | +| All but one critic fail | PARTIAL: produce verdict from sole remaining critic. Tag output with "⚠️ PARTIAL." Verdict reflects only that critic's coverage. | +| All critics fail | Abort: "❌ QUORUM_FAILED: All critics failed" | +| Finding lacks evidence | Reject finding silently, count in summary | +| Known issues file corrupted | Warn, continue without learning memory | -> You are the Cross-Consistency Critic for Quorum, a rigorous quality validation system. -> -> Your role: Evaluate the relationship between two artifacts for consistency, completeness of implementation, and specification adherence. -> -> Your specific focus areas: -> 1. **Spec-to-implementation gaps** — Features specified but not implemented, or implemented but not specified. -> 2. **Behavioral contradictions** — The implementation behaves differently from what the spec/docs describe. -> 3. **Interface mismatches** — Function signatures, parameter names, return types, or API contracts that differ between artifacts. -> 4. **Terminology drift** — The same concept named differently across artifacts without explicit mapping. -> 5. **Version skew** — One artifact references features or behaviors that belong to a different version of the other. -> -> Critical rule: EVERY finding must include direct quotes from BOTH artifacts showing the inconsistency. Findings with evidence from only one artifact will be rejected. +--- -5. The prompt template for cross-consistency includes both artifacts: +## File Layout ``` -## Artifact A (Primary) - - -{artifact_a_text} - - -## Artifact B (Secondary) - - -{artifact_b_text} - - -## Pre-Screen Evidence (Artifact A) - -{prescreen_a_results} - -## Pre-Screen Evidence (Artifact B) - -{prescreen_b_results} - -## Your Task - -Evaluate the consistency between Artifact A and Artifact B. For each inconsistency: +~/.copilot/skills/quorum/ +├── SKILL.md ← This file (orchestration) +├── quorum-prescreen.py ← Deterministic pre-screen (stdlib Python) +├── critics/ +│ ├── correctness.yaml ← Correctness critic definition +│ ├── completeness.yaml ← Completeness critic definition +│ ├── security.yaml ← Security critic definition +│ └── code_hygiene.yaml ← Code hygiene critic definition +├── rubrics/ +│ ├── python-code.json ← Python code quality rubric +│ ├── documentation.json ← Documentation quality rubric +│ ├── agent-config.json ← Agent config rubric +│ └── research-synthesis.json ← Research synthesis rubric +├── learning/ +│ └── known_issues.json ← Accumulated patterns (grows over time) +└── verdict-rules.yaml ← Deterministic verdict logic +``` -1. Assign a severity. -2. Describe the inconsistency. -3. Quote the relevant passage from BOTH artifacts. -4. Specify the location in each artifact. +### Installation -Respond ONLY with JSON matching the FINDINGS_SCHEMA. +```bash +git clone https://github.com/SharedIntellect/quorum-copilot-skill +cp -r quorum-copilot-skill ~/.copilot/skills/quorum +# Done. No API keys. No SDK. No approval process. ``` -6. Apply the same verdict rules (section 7) and report format (section 8) to the cross-consistency findings. - --- -## 11. Single-Critic Mode - -If the user asks to run a specific critic only (e.g., "run the security critic on this file", "just check correctness"): +## What This Is (and Isn't) -1. Classify the artifact (section 1). -2. Select the rubric (section 2). -3. Run pre-screen (section 3). -4. Dispatch ONLY the requested critic (section 4), using its `.agent.md` file or inline prompt. -5. Collect findings, apply verdict rules, and generate the report as normal. +**This is** a multi-critic validation skill that catches real issues in code, documentation, and configurations. It enforces evidence grounding — every claim must be backed by a direct quote from the artifact. -The report should list only the single critic in the Coverage Summary. All other sections remain the same. +**This is not** the full Quorum reference implementation. The CLI version has additional capabilities (batch processing, fix loops, cost tracking, tester verification). This skill covers the highest-value portion: deterministic pre-screen → parallel critic dispatch → evidence-grounded findings → deterministic verdict. ---- - -## 12. Error Handling - -Handle these failure modes gracefully: - -| Failure | Action | -|---------|--------| -| Target file not found | Report the error immediately. Do not proceed. | -| Target file is binary | Report: "Binary files are not supported by Quorum." Do not proceed. | -| Pre-screen script not found | Warn the user. Proceed without pre-screen data. | -| Pre-screen script crashes | Log stderr. Proceed without pre-screen data. | -| Rubric file not found | Warn the user. Proceed with no rubric criteria (critics will still evaluate based on their system prompt). | -| Critic task agent times out (>120s) | Log the timeout. Include "ERROR: timeout after 120s" in that critic's coverage summary row. Continue with remaining critics. | -| Critic returns invalid JSON | Attempt to extract findings from the response text. If that fails, log the error and mark the critic as failed in coverage summary. | -| All critics fail | Report verdict as **PASS** with a prominent warning: "All critics failed. This PASS verdict reflects no evaluation, not confirmed quality. Re-run recommended." | +**Capability coverage:** ~75% of reference implementation. Includes all four core evaluation critics. Not yet ported: L1/L2 verification, automated remediation, batch mode, cost tracking, structured output artifacts. These are planned for future releases. diff --git a/ports/copilot-cli/critics/code_hygiene.yaml b/ports/copilot-cli/critics/code_hygiene.yaml new file mode 100644 index 0000000..76baa8e --- /dev/null +++ b/ports/copilot-cli/critics/code_hygiene.yaml @@ -0,0 +1,444 @@ +# Quorum Code Hygiene Critic — Portable Definition +# Source: reference-implementation/quorum/critics/code_hygiene.py +# Version: 1.0.0 +# Consumers: CLI (Python class), Claude Code (SKILL.md), Copilot (task agent) + +name: code_hygiene +description: > + Structural quality, maintainability, and reliability analysis. The LLM judgment + layer on top of deterministic pre-screen checks (PS-001–PS-010). Evaluates issues + that static analysis cannot reliably assess from syntax alone. Primary quality + models: ISO/IEC 25010:2023 (Maintainability + Reliability) and ISO/IEC 5055:2021 + (CISQ ASCMM/ASCRM CWE mappings). + +model_tier: 2 # Sonnet-class for standard depth; Opus for thorough + +# ─── DELEGATION BOUNDARY ───────────────────────────────────────────────────── +# This critic flags security-adjacent patterns for traceability and DELEGATES +# security severity assessment to the Security Critic. It does NOT assign +# HIGH/CRITICAL to these patterns — that is Security Critic's domain. + +delegation: + delegates_to: + - critic: security + patterns: + - "eval()/exec()/__import__() with external input" + - "Invoke-Expression in PowerShell" + - "Hardcoded credential literals" + - "Unsanitized user input in prompt construction (AP-01)" + finding_template: > + Code hygiene concern: {pattern} is difficult to audit and test. + Security severity and exploitability assessment delegated to Security Critic. + max_severity_from_this_critic: MEDIUM + +# ─── SYSTEM PROMPT ─────────────────────────────────────────────────────────── + +system_prompt: | + You are the Code Hygiene Critic for Quorum, a rigorous multi-critic quality + validation system. + + Your role: Evaluate source code for structural quality issues that require LLM + semantic judgment — things that static analysis tools CANNOT reliably catch + from syntax alone. + + You are the LLM layer on top of the deterministic pre-screen (PS-001–PS-010). + Do NOT re-run pattern matching the pre-screen already performed. Your job is + semantic and design-quality judgment. + + ━━━ PRIMARY QUALITY MODELS ━━━ + + ISO/IEC 25010:2023 — Maintainability and Reliability: + Maintainability: Modularity, Reusability, Analysability, Modifiability, Testability + Reliability: Faultlessness, Fault Tolerance, Availability, Recoverability + + ISO/IEC 5055:2021 (CISQ) — CWE mappings: + Reliability (ASCRM): CWE-252, CWE-390, CWE-391, CWE-703, CWE-404, CWE-662, CWE-833 + Maintainability (ASCMM): CWE-1041, CWE-1047, CWE-1048, CWE-1052, CWE-1064, + CWE-1080, CWE-1083, CWE-1085, CWE-1121 + + ━━━ DELEGATION BOUNDARY ━━━ + + When you detect eval()/exec(), Invoke-Expression, hardcoded credentials, or + unsanitized user input in prompts: flag the code hygiene concern (hard to audit, + hard to test) and explicitly state security assessment is delegated to the + Security Critic. Assign LOW or MEDIUM severity only from this critic. + + ━━━ EVIDENCE RULES ━━━ + + EVERY finding must include ONE of: + - A direct code excerpt from the artifact showing the problematic code + - A pre-screen check ID (e.g. [PS-007]) that pre-verified the issue + + If you cannot quote the artifact or cite a pre-screen check, do not report it. + + ━━━ SEVERITY GUIDE ━━━ + + CRITICAL: Bug causing runtime failure, data corruption, or system hang in common usage + HIGH: Logic error, exception swallowing, resource leak, race condition with high impact + MEDIUM: Incomplete handling, maintainability-reducing complexity, missing error context + LOW: Naming clarity, near-duplicate code, minor testability concerns + INFO: Style organization, optional documentation improvements + + ━━━ DO NOT ━━━ + - Re-flag PASSED pre-screen checks + - Report auto-fixable formatting issues (indentation, line length, whitespace) + - Assign security severity to credential/injection patterns — delegate to Security Critic + - Invent code quotes — only cite text that appears verbatim in the artifact + - Flag the same issue in multiple findings + +# ─── EVALUATION CATEGORIES ─────────────────────────────────────────────────── + +categories: + # Core quality categories (ISO 25010 + CISQ mapped) + - id: CAT-01 + name: Code Correctness + iso_mapping: "Reliability → Faultlessness" + cisq_cwes: ["CWE-252", "CWE-480", "CWE-682"] + description: > + Issues SAST cannot catch: off-by-one errors in range/index/boundary logic, + wrong algorithm producing wrong results, unreachable branches that are + logically impossible but syntactically valid, incorrect comparison semantics, + closure variable capture in loop bodies. + + - id: CAT-02 + name: Error Handling + iso_mapping: "Reliability → Fault Tolerance" + cisq_cwes: ["CWE-390", "CWE-391", "CWE-703"] + description: > + Exception handling completeness — does code handle ALL meaningful failure modes? + Appropriate exception granularity, error propagation correctness, recovery logic + adequacy. PowerShell: $? fragile flag vs structured try/catch, missing + -ErrorAction Stop on cmdlets inside try blocks. + + - id: CAT-03 + name: Resource Management + iso_mapping: "Performance Efficiency → Resource Utilization" + cisq_cwes: ["CWE-401", "CWE-404", "CWE-772"] + description: > + Context manager usage across ALL code paths including exception paths, + connection lifecycle completeness, timeout adequacy, unbounded result + accumulation in loops, generator vs list opportunity. + + - id: CAT-04 + name: Complexity & Modularity + iso_mapping: "Maintainability → Analysability, Modularity" + cisq_cwes: ["CWE-1047", "CWE-1064", "CWE-1083", "CWE-1121"] + description: > + Cohesion (one thing or many?), inappropriate abstraction, God class/function + patterns, semantic layer violations. PowerShell: Export-ModuleMember usage, + function decomposition. + + - id: CAT-05 + name: Code Duplication + iso_mapping: "Maintainability → Reusability" + cisq_cwes: ["CWE-1041"] + description: > + Near-duplicate logic with parameter variation (semantic clones, not just + exact copies), missed abstraction opportunities, justified vs unjustified + duplication. + + - id: CAT-06 + name: Naming & Documentation + iso_mapping: "Maintainability → Analysability" + cisq_cwes: ["CWE-1052", "CWE-1085"] + description: > + Docstring accuracy (not just presence — is it correct?), naming clarity and + semantic precision, magic numbers/strings needing named constants, comment + quality (WHY not WHAT). PowerShell: ProvideCommentHelp quality beyond presence. + + - id: CAT-07 + name: Type Safety & Data Integrity + iso_mapping: "Reliability → Faultlessness" + cisq_cwes: ["CWE-681", "CWE-704"] + description: > + Type annotation correctness (not just presence), None/Optional propagation + gaps, optional chaining adequacy on dicts. PowerShell: type coercion surprises, + array vs scalar pipeline behavior. + + - id: CAT-08 + name: Async & Concurrency + iso_mapping: "Reliability → Fault Tolerance" + cisq_cwes: ["CWE-366", "CWE-662", "CWE-833"] + description: > + Async/sync boundary consistency, asyncio.CancelledError propagation (MUST NOT + be swallowed by except Exception:), race conditions in shared mutable state, + deadlock potential. PowerShell: ForEach-Object -Parallel thread safety. + + - id: CAT-09 + name: Import & Dependency Hygiene + iso_mapping: "Maintainability → Modularity" + cisq_cwes: ["CWE-1047", "CWE-1048"] + description: > + Conditional import justification, dependency necessity, version pinning + adequacy. PowerShell: RequiredModules declared with version constraints. + + - id: CAT-10 + name: Style & Formatting + iso_mapping: "Maintainability → Analysability" + cisq_cwes: ["CWE-1080"] + description: > + Logical organization (related functions grouped?), consistent abstraction + level within functions. Do NOT flag auto-fixable formatting — only semantic + organization issues. + + - id: CAT-11 + name: Portability & Compatibility + iso_mapping: "Flexibility → Adaptability" + cisq_cwes: ["CWE-758", "CWE-1051"] + description: > + Subtle OS assumptions (hardcoded paths, platform-specific APIs without guards), + PowerShell Windows vs Linux parity (PS 5.1 vs PS 7 behavior differences), + hardcoded configuration (IP addresses, ports, connection strings). + + - id: CAT-12 + name: Testability + iso_mapping: "Maintainability → Testability" + cisq_cwes: [] + description: > + Function purity and side effects, dependency injection design, complex logic + branches appearing untested, test isolation (hidden deps on global state, + filesystem, network), mock adequacy. + + # Agentic code patterns (no ISO mapping — Quorum-specific) + - id: AP-01 + name: Prompt Construction + tier: agentic + description: > + Unsanitized user input in system/user messages, role boundary enforcement, + prompt template exfiltration risk, token budget management, structured output + schema validation. + + - id: AP-02 + name: LLM API Call Patterns + tier: agentic + description: > + Retry logic (429, 503 with exponential backoff + jitter), model version pinning, + response structure validation before field access, max_tokens set, streaming + response cleanup on error. + + - id: AP-03 + name: Agent Pipeline Errors + tier: agentic + description: > + Partial success handling, structured error returns vs exceptions consistency, + fallback behavior, error context preservation (input, tool name, pipeline stage), + asyncio.CancelledError propagation. + + - id: AP-04 + name: Credential Management + tier: agentic + description: > + Environment variable usage (no plaintext fallback defaults), secrets in log + output at any level, secrets in exception messages, .env files excluded from VCS. + NOTE: Flag pattern only — delegate security assessment to Security Critic. + + - id: AP-05 + name: Timeout & Retry Logic + tier: agentic + description: > + Exponential backoff calculation, max retry count, jitter for thundering herd, + retry on correct exception types (not logic errors), circuit breaker for + high-volume agents, timeout scope completeness. + + - id: AP-06 + name: Logging & Observability + tier: agentic + description: > + Structured logging for agent decisions, log level appropriateness, correlation + IDs in multi-step pipelines, sensitive data in logs, log coverage at failure + points. + +# ─── PRE-SCREEN INTEGRATION ───────────────────────────────────────────────── + +prescreen_integration: + description: > + The pre-screen (PS-001–PS-010) runs before this critic. FAIL checks are + pre-verified — add LLM judgment on semantic impact. PASS checks are confirmed + clean — do NOT re-flag. SKIP checks are not applicable. + check_mapping: + PS-001: { relates_to: [CAT-11, AP-04], action: "Assess semantic impact of hardcoded paths" } + PS-002: { relates_to: [AP-04], action: "Flag hygiene concern, delegate security to Security Critic" } + PS-003: { relates_to: [AP-04], action: "Flag hygiene concern, delegate security to Security Critic" } + PS-006: { relates_to: [CAT-01], action: "Assess semantic impact of syntax issue" } + PS-007: { relates_to: [CAT-06], action: "Assess documentation impact of broken links" } + PS-008: { relates_to: [CAT-06], action: "Assess whether TODOs indicate incomplete implementation" } + +# ─── PROMPT TEMPLATE ───────────────────────────────────────────────────────── + +prompt_template: | + ## Artifact Under Review + + ``` + {ARTIFACT_TEXT} + ``` + + ## Rubric: {RUBRIC_NAME} (v{RUBRIC_VERSION}) + + Domain: {RUBRIC_DOMAIN} + + ### Code Hygiene Criteria to Evaluate + {CRITERIA_TEXT} + + ## Pre-Screen Evidence + + {PRESCREEN_EVIDENCE} + + {EXTRA_CONTEXT} + + ## Your Task: Semantic Code Hygiene Assessment + + Evaluate the artifact for code quality issues that static analysis CANNOT catch. + + Categories to assess: + - CAT-01 (Correctness): Off-by-one, wrong algorithms, unreachable branches + - CAT-02 (Error Handling): Exception completeness, granularity, recovery logic + - CAT-03 (Resource Management): Context managers across exception paths, timeouts + - CAT-04 (Complexity): Cohesion, God class/function, layer violations + - CAT-05 (Duplication): Semantic clones, missed abstraction opportunities + - CAT-06 (Naming & Docs): Docstring accuracy, naming clarity, magic numbers + - CAT-07 (Type Safety): Annotation correctness, None/Optional propagation + - CAT-08 (Async): Async/sync boundaries, CancelledError propagation, races + - CAT-09 (Import Hygiene): Conditional imports, dependency necessity + - CAT-10 (Style): Logical organization, consistent abstraction level + - CAT-11 (Portability): OS assumptions, platform-specific APIs, hardcoded config + - CAT-12 (Testability): Purity, side effects, injection design + + If agentic code: also assess AP-01 through AP-06 (prompt construction, LLM API + hygiene, pipeline errors, credentials, retry logic, observability). + + Delegation: For eval()/exec(), Invoke-Expression, hardcoded credentials, or + unsanitized prompt input — flag the hygiene concern and state that security + assessment is delegated to Security Critic. Assign LOW/MEDIUM only. + + For each finding: + 1. Quote the specific code excerpt OR cite pre-screen check ID + 2. Explain the quality concern — what can go wrong and why it matters + 3. Reference the applicable category (e.g. CAT-02, AP-03) + 4. Identify which rubric criterion this finding addresses (if any) + 5. Assign severity: CRITICAL / HIGH / MEDIUM / LOW / INFO + + Return empty findings list if no hygiene issues found. + Do not report auto-fixable formatting issues or re-flag pre-screen PASSED checks. + +# ─── RUBRIC KEYWORD FILTER ─────────────────────────────────────────────────── + +rubric_keywords: + # Core correctness / reliability + - correct + - correctness + - logic + - error + - exception + - handling + - fault + - tolerance + - faultless + - reliability + - robust + # Resource / async + - resource + - async + - concurrency + - concurrent + - thread + - lock + - timeout + - retry + - context manager + - leak + - blocking + # Structure / maintainability + - complex + - complexity + - modular + - modularity + - coupling + - cohesion + - duplication + - duplicate + - reuse + - reusability + # Naming / docs + - naming + - name + - documentation + - docstring + - comment + - readability + - analysability + - self-descriptive + # Type / data + - type + - typing + - annotation + - type safety + - none + - optional + # Import / dependency + - import + - dependency + - circular + # Style + - style + - formatting + - format + # Portability + - portable + - portability + - compatibility + - platform + # Testability + - test + - testability + - testable + - mock + - fixture + # Agentic + - prompt + - llm + - agent + - pipeline + - logging + - observability + - credential + - secret + - retry + - backoff + # Catch-all + - hygiene + - quality + - maintainability + - maintainable + +# ─── OUTPUT SCHEMA ─────────────────────────────────────────────────────────── + +output_schema: + type: object + required: [findings] + properties: + findings: + type: array + items: + type: object + required: [severity, description, evidence_tool, evidence_result] + properties: + severity: + type: string + enum: [CRITICAL, HIGH, MEDIUM, LOW, INFO] + description: + type: string + evidence_tool: + type: string + description: "How verified: grep, schema, read, analysis, pre-screen" + evidence_result: + type: string + description: "Actual excerpt or output proving this finding" + location: + type: string + description: "Where in the artifact (section, line, key path)" + rubric_criterion: + type: string + description: "Rubric criterion ID this finding addresses" + category: + type: string + description: "Evaluation category (CAT-01 through CAT-12, AP-01 through AP-06)" diff --git a/ports/copilot-cli/critics/completeness.yaml b/ports/copilot-cli/critics/completeness.yaml new file mode 100644 index 0000000..aa0c324 --- /dev/null +++ b/ports/copilot-cli/critics/completeness.yaml @@ -0,0 +1,165 @@ +# Quorum Completeness Critic — Portable Definition +# Source: reference-implementation/quorum/critics/completeness.py +# Version: 1.0.0 +# Consumers: CLI (Python class), Claude Code (SKILL.md), Copilot (task agent) + +name: completeness +description: > + Evaluates coverage gaps, missing requirements, and unaddressed edge cases. + Completeness is about what's NOT there — requirements the artifact fails to + address, sections that are skeletal, edge cases that should be handled. + +model_tier: 2 # Sonnet-class for standard depth; Opus for thorough + +# ─── SYSTEM PROMPT ─────────────────────────────────────────────────────────── + +system_prompt: | + You are the Completeness Critic for Quorum, a rigorous quality validation system. + + Your role: Evaluate artifacts for coverage gaps, missing requirements, and + unaddressed edge cases. + + Focus areas: + 1. Required sections missing — rubric requires content the artifact doesn't provide + 2. Shallow treatment — topics mentioned but not meaningfully addressed (stubs, "TBD") + 3. Edge cases — scenarios the artifact should address but doesn't + 4. Broken promises — content other parts of the artifact imply exists but doesn't + 5. Requirement gaps — explicit rubric criteria the artifact fails to satisfy + + EVIDENCE RULES: + - EVERY finding must include evidence: + • A direct quote showing the gap (empty section, stub, placeholder) + • A rubric criterion ID that requires missing content + • A quote from the artifact implying required content is missing + - Do not flag things as "missing" without grounding + - "Error handling is missing" is NOT acceptable + - "Section 3 mentions error handling in Appendix B, but Appendix B does not exist" IS acceptable + + SEVERITY GUIDE: + - CRITICAL: Core requirement absent — artifact cannot fulfill its stated purpose + - HIGH: Important gap that significantly reduces artifact quality or completeness + - MEDIUM: Notable omission that affects specific use cases or edge cases + - LOW: Minor gap — nice-to-have content not addressed + - INFO: Observations about potential additions worth considering + +# ─── EVALUATION CATEGORIES ─────────────────────────────────────────────────── + +categories: + - id: COM-01 + name: Required Sections Missing + description: > + Rubric criteria that require specific content which the artifact does not + provide at all. Complete absence, not shallow treatment. + examples: + - "Rubric PC-015 requires error handling documentation; no such section exists" + - "Rubric RS-003 requires methodology description; artifact jumps from intro to results" + + - id: COM-02 + name: Shallow Treatment + description: > + Topics mentioned or section headers present, but content is skeletal, uses + placeholders, or lacks substantive detail. The intent is there; the execution isn't. + examples: + - "Section 'Error Handling' contains only 'TODO: document error handling'" + - "Security considerations section is a single sentence: 'Standard security practices apply'" + + - id: COM-03 + name: Unaddressed Edge Cases + description: > + Scenarios the artifact's own framing implies should be addressed but aren't. + Error conditions, boundary cases, failure modes, concurrent access, empty input, + resource exhaustion. + examples: + - "Function handles valid input but has no path for malformed input" + - "Pipeline describes success flow but never addresses partial failure" + + - id: COM-04 + name: Broken Promises + description: > + Content that other parts of the artifact explicitly reference or imply exists + but doesn't. Cross-references to nonexistent sections, promised appendices, + forward references that are never fulfilled. + examples: + - "'See Appendix A for deployment instructions' — Appendix A does not exist" + - "README says 'configuration options documented in CONFIG.md' — file missing" + + - id: COM-05 + name: Requirement Gaps + description: > + Explicit rubric criteria that the artifact fails to satisfy. The criterion + defines a concrete requirement; the artifact does not meet it. + examples: + - "Criterion PC-012 requires type annotations on public functions; 7/15 lack them" + - "Criterion RS-008 requires statistical significance reporting; no p-values present" + +# ─── PROMPT TEMPLATE ───────────────────────────────────────────────────────── + +prompt_template: | + ## Artifact Under Review + + ``` + {ARTIFACT_TEXT} + ``` + + ## Rubric: {RUBRIC_NAME} (v{RUBRIC_VERSION}) + + Domain: {RUBRIC_DOMAIN} + + ### All Criteria to Check for Completeness + {CRITERIA_TEXT} + + {EXTRA_CONTEXT} + + ## Your Task + + For each rubric criterion above, determine: does the artifact adequately address it? + + For any criterion NOT adequately addressed: + 1. Quote the relevant (absent or skeletal) section of the artifact + 2. Reference the rubric criterion ID that requires this content + 3. Explain specifically what is missing or underdeveloped + 4. Classify by category: COM-01 (missing), COM-02 (shallow), COM-03 (edge case), + COM-04 (broken promise), COM-05 (requirement gap) + 5. Assign severity: CRITICAL / HIGH / MEDIUM / LOW / INFO + + Also flag edge cases, failure modes, or boundary conditions the artifact's own + framing implies should be addressed but aren't (COM-03). + + If a criterion IS adequately addressed, do not report it — only report gaps. + Return an empty findings list if the artifact is complete. + +# ─── RUBRIC HANDLING ───────────────────────────────────────────────────────── +# Unlike other critics, Completeness evaluates ALL rubric criteria — it checks +# whether each one is addressed. No keyword filtering. + +rubric_handling: evaluate_all_criteria + +# ─── OUTPUT SCHEMA ─────────────────────────────────────────────────────────── + +output_schema: + type: object + required: [findings] + properties: + findings: + type: array + items: + type: object + required: [severity, description, evidence_tool, evidence_result] + properties: + severity: + type: string + enum: [CRITICAL, HIGH, MEDIUM, LOW, INFO] + description: + type: string + evidence_tool: + type: string + description: "How verified: grep, schema, read, analysis" + evidence_result: + type: string + description: "Actual excerpt or output proving this finding" + location: + type: string + description: "Where in the artifact (section, line, key path)" + rubric_criterion: + type: string + description: "Rubric criterion ID this finding addresses" diff --git a/ports/copilot-cli/critics/correctness.yaml b/ports/copilot-cli/critics/correctness.yaml new file mode 100644 index 0000000..34a74ee --- /dev/null +++ b/ports/copilot-cli/critics/correctness.yaml @@ -0,0 +1,192 @@ +# Quorum Correctness Critic — Portable Definition +# Source: reference-implementation/quorum/critics/correctness.py +# Version: 1.0.0 +# Consumers: CLI (Python class), Claude Code (SKILL.md), Copilot (task agent) + +name: correctness +description: > + Evaluates artifacts for factual accuracy, logical consistency, and internal + contradictions. Catches claims that conflict with each other, unsupported + reasoning leaps, and reference inaccuracies. + +model_tier: 2 # Sonnet-class for standard depth; Opus for thorough + +# ─── SYSTEM PROMPT ─────────────────────────────────────────────────────────── +# The critic's identity and operating constraints. Injected as the system +# message (or equivalent preamble) before the evaluation prompt. + +system_prompt: | + You are the Correctness Critic for Quorum, a rigorous quality validation system. + + Your role: Evaluate artifacts for factual accuracy, logical consistency, and + internal contradictions. + + Focus areas: + 1. Internal contradictions — statements in one section incompatible with another + 2. Logical consistency — conclusions follow from premises; no non-sequiturs + 3. Factual claims — stated facts are plausible and internally coherent + 4. Reference accuracy — citations, names, and figures are internally consistent + 5. Terminology consistency — same concept not described by conflicting terms + + EVIDENCE RULES: + - EVERY finding must include a direct quote or specific excerpt from the artifact + - If you cannot quote the artifact to support a finding, do not report it + - Vague claims like "this section is unclear" without evidence will be rejected + - Do not invent quotes — only cite text that appears verbatim in the artifact + + SEVERITY GUIDE: + - CRITICAL: Fatal factual error or logical contradiction that invalidates core claims + - HIGH: Significant inconsistency or unsupported claim affecting key conclusions + - MEDIUM: Notable inconsistency or questionable reasoning in non-critical sections + - LOW: Minor terminology drift or cosmetic inconsistency + - INFO: Observations worth noting but not blocking + +# ─── EVALUATION CATEGORIES ─────────────────────────────────────────────────── +# Structured breakdown of what this critic evaluates. Ports use these to +# construct evaluation prompts appropriate to their platform. + +categories: + - id: COR-01 + name: Internal Contradictions + description: > + Statements in one section that are incompatible with statements in another. + Includes numerical inconsistencies, conflicting claims about the same entity, + and self-contradicting logic. + examples: + - "Section 2 states the timeout is 30 seconds; Section 5 configures it as 60" + - "Abstract claims 99% accuracy; Results section reports 94%" + + - id: COR-02 + name: Logical Consistency + description: > + Conclusions must follow from premises. Flag non-sequiturs, unsupported leaps, + circular reasoning, and arguments where the conclusion doesn't follow from + the evidence presented. + examples: + - "Claims system is production-ready based solely on unit test pass rate" + - "Concludes X is impossible but earlier demonstrates X working" + + - id: COR-03 + name: Factual Claims + description: > + Stated facts that are internally incoherent or implausible given the artifact's + own context. This critic does NOT verify external facts — it checks whether + claims are consistent with each other and with the artifact's own evidence. + examples: + - "Claims to support 10 file types but only lists 7 in the implementation" + - "States 'all tests pass' but test output shows 3 failures" + + - id: COR-04 + name: Reference Accuracy + description: > + Citations, cross-references, figure numbers, section references, and quoted + values must be internally consistent. Includes: references to nonexistent + sections, incorrect figure numbers, misquoted values from other parts of + the artifact. + examples: + - "References 'Section 4.3' which does not exist" + - "Cites 'Table 2' but the table shows different data than described" + + - id: COR-05 + name: Terminology Consistency + description: > + The same concept should be described with consistent terminology throughout. + Flag cases where different terms are used interchangeably without explicit + acknowledgment, creating ambiguity about whether they refer to the same thing. + examples: + - "Uses 'finding', 'issue', and 'defect' interchangeably without definition" + - "Section 1 calls it 'evidence grounding'; Section 4 calls it 'citation verification'" + +# ─── PROMPT TEMPLATE ───────────────────────────────────────────────────────── +# The evaluation prompt injected as the user message. Variables are denoted +# with {VARIABLE_NAME} and must be resolved by the port's orchestrator. +# +# Required variables: +# {ARTIFACT_TEXT} — full text of the artifact under review +# {RUBRIC_NAME} — human-readable rubric name +# {RUBRIC_VERSION} — rubric semver +# {RUBRIC_DOMAIN} — rubric domain tag +# {CRITERIA_TEXT} — formatted rubric criteria (port-specific formatting) +# {EXTRA_CONTEXT} — optional additional context (known issues, pre-screen results) + +prompt_template: | + ## Artifact Under Review + + ``` + {ARTIFACT_TEXT} + ``` + + ## Rubric: {RUBRIC_NAME} (v{RUBRIC_VERSION}) + + Domain: {RUBRIC_DOMAIN} + + ### Criteria to Evaluate + {CRITERIA_TEXT} + + {EXTRA_CONTEXT} + + ## Your Task + + Review the artifact for correctness issues across these categories: + - COR-01: Internal contradictions + - COR-02: Logical consistency + - COR-03: Factual claims (internal coherence) + - COR-04: Reference accuracy + - COR-05: Terminology consistency + + For each finding: + 1. Quote the specific text from the artifact that is problematic + 2. Explain the correctness violation clearly + 3. Identify which rubric criterion it violates (if any) + 4. Assign severity: CRITICAL / HIGH / MEDIUM / LOW / INFO + + If the artifact is entirely correct, return an empty findings list. + Only report findings you can back up with direct quotes from the artifact. + +# ─── RUBRIC KEYWORD FILTER ─────────────────────────────────────────────────── +# When a rubric has more criteria than this critic needs, filter by these +# keywords (matched against criterion text, case-insensitive). If no criteria +# match, fall back to evaluating all criteria. + +rubric_keywords: + - accurate + - correct + - consistent + - factual + - logical + - contradict + - valid + - truth + - claim + - support + +# ─── OUTPUT SCHEMA ─────────────────────────────────────────────────────────── +# The structured output format critics must return. All ports enforce this. + +output_schema: + type: object + required: [findings] + properties: + findings: + type: array + items: + type: object + required: [severity, description, evidence_tool, evidence_result] + properties: + severity: + type: string + enum: [CRITICAL, HIGH, MEDIUM, LOW, INFO] + description: + type: string + evidence_tool: + type: string + description: "How verified: grep, schema, read, analysis" + evidence_result: + type: string + description: "Actual excerpt or output proving this finding" + location: + type: string + description: "Where in the artifact (section, line, key path)" + rubric_criterion: + type: string + description: "Rubric criterion ID this finding addresses" diff --git a/ports/copilot-cli/critics/security.yaml b/ports/copilot-cli/critics/security.yaml new file mode 100644 index 0000000..5275592 --- /dev/null +++ b/ports/copilot-cli/critics/security.yaml @@ -0,0 +1,380 @@ +# Quorum Security Critic — Portable Definition +# Source: reference-implementation/quorum/critics/security.py +# Version: 1.0.0 +# Consumers: CLI (Python class), Claude Code (SKILL.md), Copilot (task agent) + +name: security +description: > + Framework-grounded security and sensitivity analysis. Applies LLM judgment + on top of deterministic pre-screen results. Grounded in OWASP ASVS 5.0, + CWE Top 25 (2024), NIST SP 800-53 SA-11, ISO/IEC 25010:2023, and the + Quorum Security Critic Framework (SEC-01 through SEC-14). + +model_tier: 2 # Sonnet-class for standard depth; Opus for thorough + +# ─── SYSTEM PROMPT ─────────────────────────────────────────────────────────── + +system_prompt: | + You are the Security Critic for Quorum, a rigorous quality validation system. + + Your role: Apply framework-grounded security judgment to artifacts — determine + what is genuinely risky vs. what is benign. + + ━━━ FRAMEWORK GROUNDING ━━━ + + OWASP ASVS 5.0.0 (V1–V17): Primary requirement-level security framework. + CWE Top 25 (2024): Prevalence-weighted vulnerability prioritization. + NIST SP 800-53 SA-11: Compliance vocabulary (SA-11(1) static, SA-11(3) independent, + SA-11(4) manual review, SA-11(6) attack surface). + ISO/IEC 25010:2023: Confidentiality, Integrity, Non-repudiation, Accountability, + Authenticity, Resistance. + Quorum Security Critic Framework v1.0: SEC-01 through SEC-14. + + ━━━ DETECTION CAPABILITY ━━━ + + You are the LLM layer on top of the deterministic pre-screen (PS-001–PS-010). + Focus deepest analysis where SAST is BLIND: + + 🔴 LLM-ONLY (no SAST coverage): + - Authorization logic, IDOR, privilege escalation (SEC-04) + - Authentication bypass paths (SEC-03) + - SSRF via user-controlled URLs (SEC-10) + - JWT alg=none, signature bypass (SEC-05) + - Path traversal in file operations (SEC-08) + - LDAP/XPath injection (SEC-01) + - Regex catastrophic backtracking / ReDoS (SEC-14) + - PowerShell AMSI bypass, download cradles (SEC-01) + + 🟡 LLM-HEAVY (SAST weak, LLM adds significant value): + - Input validation completeness (SEC-02) + - Secrets in log output / exception messages (SEC-07) + - Cryptographic context correctness (SEC-06) + - Error messages leaking system details (SEC-11) + + 🟢 BOTH (SAST strong, LLM adds chained-pattern detection): + - Injection — SQL, OS command, code (SEC-01) + - Hardcoded credentials (SEC-07) + - Deserialization (SEC-09) + - SSL/TLS verification bypass (SEC-03/SEC-06) + + ━━━ EVIDENCE RULES ━━━ + + EVERY finding must include ONE of: + - A direct quote from the artifact showing the specific problematic text + - A pre-screen check ID (e.g. [PS-001]) confirming the finding + + If you cannot quote the artifact or cite a pre-screen check, do not report it. + + ━━━ SEVERITY GUIDE ━━━ + + CRITICAL: Live credentials, exploitable injection, active private keys; + CWE Top 25 top-10 with direct exploitability + HIGH: Auth/authz bypass, IDOR, SSRF, JWT alg=none; CWE Top 25 #11–25; + confirmed sensitive data exposure + MEDIUM: Indirect vulnerability requiring conditions; info disclosure; + ASVS L2 requirement failure + LOW: Defense-in-depth; config hardening; unpinned deps; logging adequacy + INFO: Observations worth noting but not blocking + + ━━━ DO NOT ━━━ + - Re-run pattern matching the pre-screen already did + - Flag things as sensitive without grounding in artifact text + - Invent quotes — only cite verbatim text + - Flag CWE memory-safety issues (C/C++ domain; Python/PowerShell are GC'd) + +# ─── EVALUATION CATEGORIES ─────────────────────────────────────────────────── + +categories: + # Tier 1 — Must Evaluate (every artifact) + - id: SEC-04 + name: Authorization & Access Control + tier: 1 + description: > + Sensitive operations without identity checks, IDOR patterns, horizontal/vertical + privilege escalation. SAST-blind — highest LLM value. + frameworks: ["CWE-862", "CWE-863", "ASVS V8", "SA-11(4)"] + + - id: SEC-03 + name: Authentication + tier: 1 + description: > + Credential handling, bypass paths (conditional auth skipping, debug-mode disable), + hardcoded credentials. + frameworks: ["CWE-287", "CWE-306", "CWE-798", "ASVS V6", "SA-11(4)"] + + - id: SEC-01 + name: Injection + tier: 1 + description: > + SQL, OS command, code, LDAP, XPath injection. Direct injection AND indirect — + taint across function boundaries, multi-step query construction, PowerShell + pipeline injection. + frameworks: ["CWE-89", "CWE-78", "CWE-94", "ASVS V1–V2", "SA-11(4)"] + + - id: SEC-07 + name: Secrets & Credential Handling + tier: 1 + description: > + Beyond hardcoded (pre-screen catches that): secrets in log output, exception + messages, URL query parameters, env var fallback defaults. + frameworks: ["CWE-798", "CWE-200", "ASVS V13", "SA-11(4)"] + + - id: SEC-11 + name: Error Handling & Information Disclosure + tier: 1 + description: > + Stack traces returned to callers, user-distinguishing error messages, debug + mode enabled, verbose errors leaking system details. + frameworks: ["CWE-200", "CWE-203", "CWE-209", "ASVS V16.1.*", "SA-11(4)"] + + # Tier 2 — Should Evaluate (standard depth) + - id: SEC-06 + name: Cryptography + tier: 2 + description: > + Algorithm correctness for context, IV/nonce reuse, key derivation functions, + CSPRNG usage. + frameworks: ["CWE-327", "CWE-328", "CWE-330", "ASVS V11"] + + - id: SEC-09 + name: Deserialization + tier: 2 + description: > + pickle/yaml from untrusted sources, jsonpickle, PowerShell Import-CliXml + from network sources. + frameworks: ["CWE-502", "ASVS V2", "ASVS V15"] + + - id: SEC-10 + name: SSRF + tier: 2 + description: > + User-controlled URLs in HTTP client calls, missing allowlisting, cloud + metadata endpoint access not blocked. + frameworks: ["CWE-918", "ASVS V4.2.*", "SA-11(4)"] + + - id: SEC-08 + name: Path Traversal & File Handling + tier: 2 + description: > + User input in file paths without canonicalization, zipfile.extractall() + without member validation, file uploads without extension allowlisting. + frameworks: ["CWE-22", "CWE-434", "ASVS V5"] + + - id: SEC-05 + name: Session & Token Management + tier: 2 + description: > + JWT alg=none, signature bypass, missing expiry validation, weak HMAC secret, + CSRF protection, cookie security attributes. + frameworks: ["CWE-347", "ASVS V9.2.*", "SA-11(4)"] + + - id: SEC-02 + name: Input Validation & Sanitization + tier: 2 + description: > + Whitelist vs. blacklist, type coercion without validation, business logic + bounds, encoding/decoding correctness. + frameworks: ["CWE-20", "ASVS V2"] + + - id: SEC-13 + name: Dependency & Supply Chain + tier: 2 + description: > + Unpinned dependencies, dangerous libraries (pycrypto, yaml.load without + SafeLoader), dynamic code loading, non-standard registries. + frameworks: ["ASVS V15.5.*", "SA-11(2)"] + + # Tier 3 — Deep Analysis (thorough depth) + - id: SEC-12 + name: Security Logging & Audit + tier: 3 + description: > + Auth events logged, authorization failures logged, log injection prevention, + sensitive data in logs, log integrity. + frameworks: ["CWE-117", "CWE-778", "CWE-532", "ASVS V16"] + + - id: SEC-14 + name: Resource Consumption & DoS + tier: 3 + description: > + ReDoS patterns, unbounded data loading, missing rate limits, missing timeouts, + zip/decompression bomb detection. + frameworks: ["CWE-400", "CWE-1088", "ASVS V4.4.*"] + +# ─── PRE-SCREEN INTEGRATION ───────────────────────────────────────────────── +# How this critic interacts with deterministic pre-screen results. + +prescreen_integration: + description: > + The pre-screen (PS-001–PS-010) runs before this critic. Its results are + injected as Additional Context. This critic adds judgment — upgrading + confirmed risks and downgrading false positives. + check_mapping: + PS-001: { relates_to: [SEC-08, SEC-11], action: "Determine if flagged paths are genuine risks or documentation examples" } + PS-002: { relates_to: [SEC-07], action: "Determine if flagged credentials are live or placeholders" } + PS-003: { relates_to: [SEC-07, SEC-11], action: "Determine if flagged PII is real or synthetic test data" } + rules: + - "FAIL checks are pre-verified — add judgment on genuine risk vs false positive" + - "PASS checks are confirmed clean — do NOT re-flag" + - "SKIP checks are not applicable — ignore" + +# ─── PROMPT TEMPLATE ───────────────────────────────────────────────────────── + +prompt_template: | + ## Artifact Under Review + + ``` + {ARTIFACT_TEXT} + ``` + + ## Rubric: {RUBRIC_NAME} (v{RUBRIC_VERSION}) + + Domain: {RUBRIC_DOMAIN} + + ### Security-Relevant Criteria + {CRITERIA_TEXT} + + ## Pre-Screen Evidence + + {PRESCREEN_EVIDENCE} + + {EXTRA_CONTEXT} + + ## Your Task + + Evaluate the artifact for security issues using the tiered model: + + **Tier 1 (always evaluate):** Authorization (SEC-04), Authentication (SEC-03), + Injection (SEC-01), Secrets (SEC-07), Info Disclosure (SEC-11), Context-aware + sensitivity assessment of pre-screen findings. + + **Tier 2 (standard+ depth):** Cryptography (SEC-06), Deserialization (SEC-09), + SSRF (SEC-10), Path Traversal (SEC-08), JWT/Tokens (SEC-05), Input Validation + (SEC-02), Supply Chain (SEC-13), Proprietary Content, Prompt Injection. + + **Tier 3 (thorough depth):** Security Logging (SEC-12), Resource/DoS (SEC-14), + PowerShell-specific deep patterns. + + For each finding: + 1. Quote the specific artifact text OR cite the pre-screen check ID + 2. Explain the security concern, citing the relevant SEC category + 3. State why this is genuine risk (or why a pre-screen finding is benign) + 4. Identify the rubric criterion it relates to (if any) + 5. Assign severity: CRITICAL / HIGH / MEDIUM / LOW / INFO + 6. Include framework references: + - ASVS 5.0.0 §[section]: [Requirement] + - CWE-[ID]: [Name] + - NIST SP 800-53 SA-11([1|3|4|6]) + - Detection Method: [SAST pre-screen | LLM semantic analysis | Both] + + Return empty findings list if artifact has no security issues. + +# ─── RUBRIC KEYWORD FILTER ─────────────────────────────────────────────────── + +rubric_keywords: + - security + - sensitive + - credential + - secret + - private + - internal + - disclosure + - injection + - sanitiz + - auth + - token + - key + - password + - pii + - personal + - boundary + - external + - public + - dependency + - supply + - chain + - sql + - command + - traversal + - path + - deserializ + - encrypt + - crypto + - cipher + - hash + - algorithm + - authori + - privilege + - access control + - permission + - session + - cookie + - csrf + - jwt + - ssrf + - idor + - rate limit + - resource + - logging + - audit + - error + - exception + - stack trace + - debug + +# ─── DELEGATION BOUNDARY ───────────────────────────────────────────────────── +# Other critics that interact with this one. + +delegation: + receives_from: + - critic: code_hygiene + patterns: ["eval/exec usage", "Invoke-Expression", "hardcoded credentials", "prompt injection vectors"] + note: > + Code Hygiene flags these patterns for traceability and delegates + security severity assessment to this critic. This critic assigns + the authoritative severity and remediation. + +# ─── OUTPUT SCHEMA ─────────────────────────────────────────────────────────── + +output_schema: + type: object + required: [findings] + properties: + findings: + type: array + items: + type: object + required: [severity, description, evidence_tool, evidence_result] + properties: + severity: + type: string + enum: [CRITICAL, HIGH, MEDIUM, LOW, INFO] + description: + type: string + evidence_tool: + type: string + description: "How verified: grep, schema, read, analysis, pre-screen" + evidence_result: + type: string + description: "Actual excerpt or output proving this finding" + location: + type: string + description: "Where in the artifact (section, line, key path)" + rubric_criterion: + type: string + description: "Rubric criterion ID this finding addresses" + framework_references: + type: object + properties: + asvs: + type: string + description: "ASVS 5.0.0 section reference" + cwe: + type: string + description: "CWE ID and name" + nist: + type: string + description: "NIST SP 800-53 SA-11 sub-control" + detection_method: + type: string + enum: [sast_prescreen, llm_semantic, both] diff --git a/ports/copilot-cli/learning/known_issues.json b/ports/copilot-cli/learning/known_issues.json new file mode 100644 index 0000000..fe51488 --- /dev/null +++ b/ports/copilot-cli/learning/known_issues.json @@ -0,0 +1 @@ +[] diff --git a/ports/copilot-cli/rubrics/documentation.json b/ports/copilot-cli/rubrics/documentation.json new file mode 100644 index 0000000..e807b6d --- /dev/null +++ b/ports/copilot-cli/rubrics/documentation.json @@ -0,0 +1,123 @@ +{ + "name": "Documentation Quality Rubric", + "domain": "documentation", + "version": "1.0", + "description": "Evaluates project documentation for claim discipline, internal consistency, navigation coherence, and implementation honesty. Designed for READMEs, specs, changelogs, tutorials, and user-facing guides. Complements mechanical checks (validate-docs.py) with semantic and narrative evaluation.", + "companion_tools": [ + { + "tool": "tools/validate-docs.py", + "relationship": "complementary", + "note": "validate-docs.py handles deterministic mechanical checks: version consistency across files, critic counts vs critic-status.yaml, DEC-019 denominator fractions, and file existence. This rubric handles semantic evaluation: claim accuracy, narrative coherence, navigation structure, and boundary discipline. Run validate-docs.py BEFORE this rubric — its output can serve as evidence for DOC-002, DOC-003, and DOC-005." + } + ], + "criteria": [ + { + "id": "DOC-001", + "criterion": "Every capability claim matches the actual shipped state — no features described as working that are specified-only or planned", + "severity": "CRITICAL", + "evidence_required": "Quote the claim, then show evidence it is not implemented (e.g., 'learning memory' described as active when code shows it is not wired up)", + "why": "Overclaiming is the #1 credibility killer for open source projects. One false claim undermines all true ones.", + "category": "claim_discipline", + "deterministic": false + }, + { + "id": "DOC-002", + "criterion": "Numeric claims are accurate — critic counts, test counts, check counts, coverage percentages match reality", + "severity": "HIGH", + "evidence_required": "Quote the numeric claim and show the actual count from source code, test output, or canonical status files (e.g., critic-status.yaml) that contradicts it", + "why": "Stale numbers (e.g., '10 checks' when 12 exist, '9 critics' when 6 are shipped) signal unmaintained docs.", + "category": "claim_discipline", + "deterministic": false + }, + { + "id": "DOC-003", + "criterion": "Version numbers are consistent across all documents — no file says v0.3.0 while another says v0.5.1", + "severity": "HIGH", + "evidence_required": "Quote the version references from two or more files that disagree", + "why": "Inconsistent versions confuse users about what release they are reading about.", + "category": "consistency", + "deterministic": false + }, + { + "id": "DOC-004", + "criterion": "Implementation status markers are present and accurate for all major components — shipped features are not marked as planned, and vice versa", + "severity": "HIGH", + "evidence_required": "Quote a component described without status markers, or with incorrect markers (e.g., marked 'not yet built' but present in codebase, or marked 'shipped' but not callable)", + "why": "Users and contributors need to know what works today vs what is planned. Stale status markers are a special case of overclaiming.", + "category": "claim_discipline", + "deterministic": false + }, + { + "id": "DOC-005", + "criterion": "All internal cross-references resolve — links to other docs, sections, or files point to targets that exist at the referenced path", + "severity": "HIGH", + "evidence_required": "Quote the link or reference and show the target does not exist, has been renamed, or has moved to a different path", + "why": "Broken links are the most common defect after file restructuring. Every broken link is a dead end for a user trying to learn the system.", + "category": "consistency", + "deterministic": true + }, + { + "id": "DOC-006", + "criterion": "No internal or proprietary references leak into public-facing docs — no workspace paths, internal project names, private repo references, or team member names", + "severity": "CRITICAL", + "evidence_required": "Quote the text containing the internal reference (e.g., absolute paths, internal tool names, private repo URLs, team names)", + "why": "Leaking internal structure undermines professional credibility and may expose sensitive organizational information.", + "category": "boundary", + "deterministic": false + }, + { + "id": "DOC-007", + "criterion": "Examples and code snippets are runnable — CLI commands use correct syntax, import paths match the actual package structure, output samples reflect current tool behavior", + "severity": "HIGH", + "evidence_required": "Quote the example and identify the specific error (wrong flag name, nonexistent module, incorrect syntax, outdated output format)", + "why": "Broken examples in docs are the fastest way to lose a new user during onboarding.", + "category": "accuracy", + "deterministic": false + }, + { + "id": "DOC-008", + "criterion": "Navigation hubs (README, docs index, table of contents) link to all documentation files in their scope — no orphaned docs that are unreachable from any hub", + "severity": "MEDIUM", + "evidence_required": "List the documentation files that exist on disk but are not linked from any navigation hub or table of contents", + "why": "Orphaned docs are invisible docs. If a user can't discover a file through navigation, it effectively doesn't exist.", + "category": "navigation", + "deterministic": true + }, + { + "id": "DOC-009", + "criterion": "Onboarding progression is coherent — entry points (README, QUICK_START) lead logically to deeper docs without circular references or missing steps", + "severity": "MEDIUM", + "evidence_required": "Trace the onboarding path from README and identify where the reader hits a dead end, a circular loop, or a prerequisite that was never introduced", + "why": "Documentation is a directed graph. If the happy path has gaps, new users fall off at the first missing link.", + "category": "navigation", + "deterministic": false + }, + { + "id": "DOC-010", + "criterion": "Tone and voice are consistent within each document — no unexplained shifts between casual first-person and formal third-person", + "severity": "LOW", + "evidence_required": "Quote two passages from the same document that use conflicting voice or register", + "why": "Inconsistent voice makes a project feel unpolished and confuses the reader about who is speaking.", + "category": "quality", + "deterministic": false + }, + { + "id": "DOC-011", + "criterion": "Changelogs and release notes accurately reflect what was actually changed — no omitted breaking changes, no listed changes that weren't included in the release", + "severity": "HIGH", + "evidence_required": "Compare the changelog entry against the actual git diff or release contents and identify discrepancies (missing entries, phantom entries, misattributed versions)", + "why": "Changelogs are the trust contract between maintainers and users. Inaccurate changelogs erode update confidence.", + "category": "accuracy", + "deterministic": false + }, + { + "id": "DOC-012", + "criterion": "No stale 'coming soon', 'TODO', 'not yet implemented', or placeholder text remains for features that have since shipped", + "severity": "MEDIUM", + "evidence_required": "Quote the placeholder text and show that the referenced feature is now implemented in the codebase", + "why": "Stale placeholders make shipped features look unfinished and confuse contributors about what work remains.", + "category": "claim_discipline", + "deterministic": false + } + ] +} \ No newline at end of file diff --git a/ports/copilot-cli/verdict-rules.yaml b/ports/copilot-cli/verdict-rules.yaml new file mode 100644 index 0000000..183500d --- /dev/null +++ b/ports/copilot-cli/verdict-rules.yaml @@ -0,0 +1,60 @@ +# Quorum Verdict Rules — Shared Aggregation Logic +# Version: 1.0.0 +# These rules are deterministic. Every port applies them identically. + +# ─── VERDICT TAXONOMY ──────────────────────────────────────────────────────── + +verdicts: + PASS: + description: "No FAIL findings. Artifact meets all evaluated criteria." + action: "Ship it." + + PASS_WITH_NOTES: + description: "Minor or moderate findings that don't block acceptance." + action: "Review notes. Fix at discretion. Safe to proceed." + + REVISE: + description: "Significant findings that must be addressed before acceptance." + action: "Fix findings. Re-run validation. Do not ship until resolved." + + REJECT: + description: "Artifact is fundamentally unsalvageable in current form." + action: "Requires rewrite or major restructuring. Supervisor judgment call." + +# ─── AGGREGATION RULES ─────────────────────────────────────────────────────── +# Applied in order. First matching rule determines the verdict. + +rules: + - condition: "Any CRITICAL finding" + verdict: REVISE + + - condition: "3 or more HIGH findings" + verdict: REVISE + + - condition: "Any HIGH finding (fewer than 3)" + verdict: PASS_WITH_NOTES + + - condition: "Only MEDIUM and/or LOW findings" + verdict: PASS_WITH_NOTES + + - condition: "No findings, or only INFO-severity findings" + verdict: PASS + +# ─── ESCALATION ────────────────────────────────────────────────────────────── +# Cross-artifact relationship findings can escalate the verdict by one level. + +escalation: + trigger: "Cross-artifact relationship critic finds HIGH or CRITICAL" + effect: "Escalate verdict by one level: PASS → PASS_WITH_NOTES, PASS_WITH_NOTES → REVISE" + note: "REVISE does not escalate to REJECT. REJECT is a supervisor judgment call." + +# ─── DEDUPLICATION ─────────────────────────────────────────────────────────── +# When multiple critics report similar findings, deduplicate before aggregation. + +deduplication: + method: "Similarity threshold on description text" + threshold: 0.72 + conflict_resolution: "Keep highest severity when duplicates conflict" + note: > + Ports without programmatic dedup should instruct the aggregation prompt + to identify and merge duplicate findings, keeping the highest severity. diff --git a/reference-implementation/pyproject.toml b/reference-implementation/pyproject.toml index adc4ea8..ec6edd6 100644 --- a/reference-implementation/pyproject.toml +++ b/reference-implementation/pyproject.toml @@ -14,7 +14,7 @@ readme = "README.md" license = {text = "MIT"} requires-python = ">=3.10" dependencies = [ - "litellm>=1.40.0", + "litellm==1.82.0", # PINNED: 1.82.8 confirmed compromised (2026-03-24), "pydantic>=2.0.0", "pyyaml>=6.0.0", "click>=8.0.0", diff --git a/reference-implementation/requirements.txt b/reference-implementation/requirements.txt index 7766326..6b31413 100644 --- a/reference-implementation/requirements.txt +++ b/reference-implementation/requirements.txt @@ -1,5 +1,5 @@ # Core dependencies -litellm>=1.40.0 +litellm==1.82.0 # PINNED: 1.82.8 confirmed compromised (2026-03-24) pydantic>=2.0.0 pyyaml>=6.0.0 click>=8.0.0