Skip to content

Commit 1cda7e4

Browse files
author
Akkari
committed
ports/copilot-cli: YAML-based portable critic definitions + 4-critic SKILL.md
Extracts critic evaluation logic from Python classes into portable YAML definitions consumable by any Quorum port (CLI, Claude Code, Copilot). New files: - critics/correctness.yaml (5 categories: COR-01–05) - critics/security.yaml (14 categories: SEC-01–14, 3 tiers, OWASP/CWE/NIST) - critics/completeness.yaml (5 categories: COM-01–05) - critics/code_hygiene.yaml (12 categories: CAT-01–12, 6 agentic: AP-01–06, ISO 25010 + CISQ mapped) - verdict-rules.yaml (deterministic aggregation, dedup, escalation) - learning/known_issues.json (empty scaffold) - rubrics/documentation.json (12-criteria DOC rubric) Updated: - SKILL.md: Rewritten as 4-critic orchestrator with dispatch tiers, progress indicators, error degradation, delegation boundary between code_hygiene and security critics. Each YAML contains: system prompt, evaluation categories, prompt template, rubric keyword filter, output schema, and pre-screen integration rules. Designed as single source of truth for multi-port architecture. Smoke-tested against golden test set artifact gt-py-001-sql-injection: 3-critic Sonnet run produced ~33 findings vs CLI thorough 36 findings, with comparable severity distribution and all critical issues caught.
1 parent d1711c1 commit 1cda7e4

8 files changed

Lines changed: 1567 additions & 342 deletions

File tree

‎ports/copilot-cli/SKILL.md‎

Lines changed: 202 additions & 342 deletions
Large diffs are not rendered by default.

‎ports/copilot-cli/critics/code_hygiene.yaml‎

Lines changed: 444 additions & 0 deletions
Large diffs are not rendered by default.

‎ports/copilot-cli/critics/completeness.yaml‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Quorum Completeness Critic — Portable Definition
2+
# Source: reference-implementation/quorum/critics/completeness.py
3+
# Version: 1.0.0
4+
# Consumers: CLI (Python class), Claude Code (SKILL.md), Copilot (task agent)
5+
6+
name: completeness
7+
description: >
8+
Evaluates coverage gaps, missing requirements, and unaddressed edge cases.
9+
Completeness is about what's NOT there — requirements the artifact fails to
10+
address, sections that are skeletal, edge cases that should be handled.
11+
12+
model_tier: 2 # Sonnet-class for standard depth; Opus for thorough
13+
14+
# ─── SYSTEM PROMPT ───────────────────────────────────────────────────────────
15+
16+
system_prompt: |
17+
You are the Completeness Critic for Quorum, a rigorous quality validation system.
18+
19+
Your role: Evaluate artifacts for coverage gaps, missing requirements, and
20+
unaddressed edge cases.
21+
22+
Focus areas:
23+
1. Required sections missing — rubric requires content the artifact doesn't provide
24+
2. Shallow treatment — topics mentioned but not meaningfully addressed (stubs, "TBD")
25+
3. Edge cases — scenarios the artifact should address but doesn't
26+
4. Broken promises — content other parts of the artifact imply exists but doesn't
27+
5. Requirement gaps — explicit rubric criteria the artifact fails to satisfy
28+
29+
EVIDENCE RULES:
30+
- EVERY finding must include evidence:
31+
• A direct quote showing the gap (empty section, stub, placeholder)
32+
• A rubric criterion ID that requires missing content
33+
• A quote from the artifact implying required content is missing
34+
- Do not flag things as "missing" without grounding
35+
- "Error handling is missing" is NOT acceptable
36+
- "Section 3 mentions error handling in Appendix B, but Appendix B does not exist" IS acceptable
37+
38+
SEVERITY GUIDE:
39+
- CRITICAL: Core requirement absent — artifact cannot fulfill its stated purpose
40+
- HIGH: Important gap that significantly reduces artifact quality or completeness
41+
- MEDIUM: Notable omission that affects specific use cases or edge cases
42+
- LOW: Minor gap — nice-to-have content not addressed
43+
- INFO: Observations about potential additions worth considering
44+
45+
# ─── EVALUATION CATEGORIES ───────────────────────────────────────────────────
46+
47+
categories:
48+
- id: COM-01
49+
name: Required Sections Missing
50+
description: >
51+
Rubric criteria that require specific content which the artifact does not
52+
provide at all. Complete absence, not shallow treatment.
53+
examples:
54+
- "Rubric PC-015 requires error handling documentation; no such section exists"
55+
- "Rubric RS-003 requires methodology description; artifact jumps from intro to results"
56+
57+
- id: COM-02
58+
name: Shallow Treatment
59+
description: >
60+
Topics mentioned or section headers present, but content is skeletal, uses
61+
placeholders, or lacks substantive detail. The intent is there; the execution isn't.
62+
examples:
63+
- "Section 'Error Handling' contains only 'TODO: document error handling'"
64+
- "Security considerations section is a single sentence: 'Standard security practices apply'"
65+
66+
- id: COM-03
67+
name: Unaddressed Edge Cases
68+
description: >
69+
Scenarios the artifact's own framing implies should be addressed but aren't.
70+
Error conditions, boundary cases, failure modes, concurrent access, empty input,
71+
resource exhaustion.
72+
examples:
73+
- "Function handles valid input but has no path for malformed input"
74+
- "Pipeline describes success flow but never addresses partial failure"
75+
76+
- id: COM-04
77+
name: Broken Promises
78+
description: >
79+
Content that other parts of the artifact explicitly reference or imply exists
80+
but doesn't. Cross-references to nonexistent sections, promised appendices,
81+
forward references that are never fulfilled.
82+
examples:
83+
- "'See Appendix A for deployment instructions' — Appendix A does not exist"
84+
- "README says 'configuration options documented in CONFIG.md' — file missing"
85+
86+
- id: COM-05
87+
name: Requirement Gaps
88+
description: >
89+
Explicit rubric criteria that the artifact fails to satisfy. The criterion
90+
defines a concrete requirement; the artifact does not meet it.
91+
examples:
92+
- "Criterion PC-012 requires type annotations on public functions; 7/15 lack them"
93+
- "Criterion RS-008 requires statistical significance reporting; no p-values present"
94+
95+
# ─── PROMPT TEMPLATE ─────────────────────────────────────────────────────────
96+
97+
prompt_template: |
98+
## Artifact Under Review
99+
100+
```
101+
{ARTIFACT_TEXT}
102+
```
103+
104+
## Rubric: {RUBRIC_NAME} (v{RUBRIC_VERSION})
105+
106+
Domain: {RUBRIC_DOMAIN}
107+
108+
### All Criteria to Check for Completeness
109+
{CRITERIA_TEXT}
110+
111+
{EXTRA_CONTEXT}
112+
113+
## Your Task
114+
115+
For each rubric criterion above, determine: does the artifact adequately address it?
116+
117+
For any criterion NOT adequately addressed:
118+
1. Quote the relevant (absent or skeletal) section of the artifact
119+
2. Reference the rubric criterion ID that requires this content
120+
3. Explain specifically what is missing or underdeveloped
121+
4. Classify by category: COM-01 (missing), COM-02 (shallow), COM-03 (edge case),
122+
COM-04 (broken promise), COM-05 (requirement gap)
123+
5. Assign severity: CRITICAL / HIGH / MEDIUM / LOW / INFO
124+
125+
Also flag edge cases, failure modes, or boundary conditions the artifact's own
126+
framing implies should be addressed but aren't (COM-03).
127+
128+
If a criterion IS adequately addressed, do not report it — only report gaps.
129+
Return an empty findings list if the artifact is complete.
130+
131+
# ─── RUBRIC HANDLING ─────────────────────────────────────────────────────────
132+
# Unlike other critics, Completeness evaluates ALL rubric criteria — it checks
133+
# whether each one is addressed. No keyword filtering.
134+
135+
rubric_handling: evaluate_all_criteria
136+
137+
# ─── OUTPUT SCHEMA ───────────────────────────────────────────────────────────
138+
139+
output_schema:
140+
type: object
141+
required: [findings]
142+
properties:
143+
findings:
144+
type: array
145+
items:
146+
type: object
147+
required: [severity, description, evidence_tool, evidence_result]
148+
properties:
149+
severity:
150+
type: string
151+
enum: [CRITICAL, HIGH, MEDIUM, LOW, INFO]
152+
description:
153+
type: string
154+
evidence_tool:
155+
type: string
156+
description: "How verified: grep, schema, read, analysis"
157+
evidence_result:
158+
type: string
159+
description: "Actual excerpt or output proving this finding"
160+
location:
161+
type: string
162+
description: "Where in the artifact (section, line, key path)"
163+
rubric_criterion:
164+
type: string
165+
description: "Rubric criterion ID this finding addresses"

‎ports/copilot-cli/critics/correctness.yaml‎

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# Quorum Correctness Critic — Portable Definition
2+
# Source: reference-implementation/quorum/critics/correctness.py
3+
# Version: 1.0.0
4+
# Consumers: CLI (Python class), Claude Code (SKILL.md), Copilot (task agent)
5+
6+
name: correctness
7+
description: >
8+
Evaluates artifacts for factual accuracy, logical consistency, and internal
9+
contradictions. Catches claims that conflict with each other, unsupported
10+
reasoning leaps, and reference inaccuracies.
11+
12+
model_tier: 2 # Sonnet-class for standard depth; Opus for thorough
13+
14+
# ─── SYSTEM PROMPT ───────────────────────────────────────────────────────────
15+
# The critic's identity and operating constraints. Injected as the system
16+
# message (or equivalent preamble) before the evaluation prompt.
17+
18+
system_prompt: |
19+
You are the Correctness Critic for Quorum, a rigorous quality validation system.
20+
21+
Your role: Evaluate artifacts for factual accuracy, logical consistency, and
22+
internal contradictions.
23+
24+
Focus areas:
25+
1. Internal contradictions — statements in one section incompatible with another
26+
2. Logical consistency — conclusions follow from premises; no non-sequiturs
27+
3. Factual claims — stated facts are plausible and internally coherent
28+
4. Reference accuracy — citations, names, and figures are internally consistent
29+
5. Terminology consistency — same concept not described by conflicting terms
30+
31+
EVIDENCE RULES:
32+
- EVERY finding must include a direct quote or specific excerpt from the artifact
33+
- If you cannot quote the artifact to support a finding, do not report it
34+
- Vague claims like "this section is unclear" without evidence will be rejected
35+
- Do not invent quotes — only cite text that appears verbatim in the artifact
36+
37+
SEVERITY GUIDE:
38+
- CRITICAL: Fatal factual error or logical contradiction that invalidates core claims
39+
- HIGH: Significant inconsistency or unsupported claim affecting key conclusions
40+
- MEDIUM: Notable inconsistency or questionable reasoning in non-critical sections
41+
- LOW: Minor terminology drift or cosmetic inconsistency
42+
- INFO: Observations worth noting but not blocking
43+
44+
# ─── EVALUATION CATEGORIES ───────────────────────────────────────────────────
45+
# Structured breakdown of what this critic evaluates. Ports use these to
46+
# construct evaluation prompts appropriate to their platform.
47+
48+
categories:
49+
- id: COR-01
50+
name: Internal Contradictions
51+
description: >
52+
Statements in one section that are incompatible with statements in another.
53+
Includes numerical inconsistencies, conflicting claims about the same entity,
54+
and self-contradicting logic.
55+
examples:
56+
- "Section 2 states the timeout is 30 seconds; Section 5 configures it as 60"
57+
- "Abstract claims 99% accuracy; Results section reports 94%"
58+
59+
- id: COR-02
60+
name: Logical Consistency
61+
description: >
62+
Conclusions must follow from premises. Flag non-sequiturs, unsupported leaps,
63+
circular reasoning, and arguments where the conclusion doesn't follow from
64+
the evidence presented.
65+
examples:
66+
- "Claims system is production-ready based solely on unit test pass rate"
67+
- "Concludes X is impossible but earlier demonstrates X working"
68+
69+
- id: COR-03
70+
name: Factual Claims
71+
description: >
72+
Stated facts that are internally incoherent or implausible given the artifact's
73+
own context. This critic does NOT verify external facts — it checks whether
74+
claims are consistent with each other and with the artifact's own evidence.
75+
examples:
76+
- "Claims to support 10 file types but only lists 7 in the implementation"
77+
- "States 'all tests pass' but test output shows 3 failures"
78+
79+
- id: COR-04
80+
name: Reference Accuracy
81+
description: >
82+
Citations, cross-references, figure numbers, section references, and quoted
83+
values must be internally consistent. Includes: references to nonexistent
84+
sections, incorrect figure numbers, misquoted values from other parts of
85+
the artifact.
86+
examples:
87+
- "References 'Section 4.3' which does not exist"
88+
- "Cites 'Table 2' but the table shows different data than described"
89+
90+
- id: COR-05
91+
name: Terminology Consistency
92+
description: >
93+
The same concept should be described with consistent terminology throughout.
94+
Flag cases where different terms are used interchangeably without explicit
95+
acknowledgment, creating ambiguity about whether they refer to the same thing.
96+
examples:
97+
- "Uses 'finding', 'issue', and 'defect' interchangeably without definition"
98+
- "Section 1 calls it 'evidence grounding'; Section 4 calls it 'citation verification'"
99+
100+
# ─── PROMPT TEMPLATE ─────────────────────────────────────────────────────────
101+
# The evaluation prompt injected as the user message. Variables are denoted
102+
# with {VARIABLE_NAME} and must be resolved by the port's orchestrator.
103+
#
104+
# Required variables:
105+
# {ARTIFACT_TEXT} — full text of the artifact under review
106+
# {RUBRIC_NAME} — human-readable rubric name
107+
# {RUBRIC_VERSION} — rubric semver
108+
# {RUBRIC_DOMAIN} — rubric domain tag
109+
# {CRITERIA_TEXT} — formatted rubric criteria (port-specific formatting)
110+
# {EXTRA_CONTEXT} — optional additional context (known issues, pre-screen results)
111+
112+
prompt_template: |
113+
## Artifact Under Review
114+
115+
```
116+
{ARTIFACT_TEXT}
117+
```
118+
119+
## Rubric: {RUBRIC_NAME} (v{RUBRIC_VERSION})
120+
121+
Domain: {RUBRIC_DOMAIN}
122+
123+
### Criteria to Evaluate
124+
{CRITERIA_TEXT}
125+
126+
{EXTRA_CONTEXT}
127+
128+
## Your Task
129+
130+
Review the artifact for correctness issues across these categories:
131+
- COR-01: Internal contradictions
132+
- COR-02: Logical consistency
133+
- COR-03: Factual claims (internal coherence)
134+
- COR-04: Reference accuracy
135+
- COR-05: Terminology consistency
136+
137+
For each finding:
138+
1. Quote the specific text from the artifact that is problematic
139+
2. Explain the correctness violation clearly
140+
3. Identify which rubric criterion it violates (if any)
141+
4. Assign severity: CRITICAL / HIGH / MEDIUM / LOW / INFO
142+
143+
If the artifact is entirely correct, return an empty findings list.
144+
Only report findings you can back up with direct quotes from the artifact.
145+
146+
# ─── RUBRIC KEYWORD FILTER ───────────────────────────────────────────────────
147+
# When a rubric has more criteria than this critic needs, filter by these
148+
# keywords (matched against criterion text, case-insensitive). If no criteria
149+
# match, fall back to evaluating all criteria.
150+
151+
rubric_keywords:
152+
- accurate
153+
- correct
154+
- consistent
155+
- factual
156+
- logical
157+
- contradict
158+
- valid
159+
- truth
160+
- claim
161+
- support
162+
163+
# ─── OUTPUT SCHEMA ───────────────────────────────────────────────────────────
164+
# The structured output format critics must return. All ports enforce this.
165+
166+
output_schema:
167+
type: object
168+
required: [findings]
169+
properties:
170+
findings:
171+
type: array
172+
items:
173+
type: object
174+
required: [severity, description, evidence_tool, evidence_result]
175+
properties:
176+
severity:
177+
type: string
178+
enum: [CRITICAL, HIGH, MEDIUM, LOW, INFO]
179+
description:
180+
type: string
181+
evidence_tool:
182+
type: string
183+
description: "How verified: grep, schema, read, analysis"
184+
evidence_result:
185+
type: string
186+
description: "Actual excerpt or output proving this finding"
187+
location:
188+
type: string
189+
description: "Where in the artifact (section, line, key path)"
190+
rubric_criterion:
191+
type: string
192+
description: "Rubric criterion ID this finding addresses"

0 commit comments

Comments
 (0)