Copilot CLI Port: YAML-based portable critic definitions (4 critics)#24
Merged
Conversation
added 3 commits
March 24, 2026 16:09
litellm_init.pth in 1.82.8 contains base64-encoded payload that exfiltrates credentials and self-replicates. Pin to last known good version until BerriAI confirms remediation. Refs: requirements.txt, pyproject.toml
…SKILL.md Extracts critic evaluation logic from Python classes into portable YAML definitions consumable by any Quorum port (CLI, Claude Code, Copilot). New files: - critics/correctness.yaml (5 categories: COR-01–05) - critics/security.yaml (14 categories: SEC-01–14, 3 tiers, OWASP/CWE/NIST) - critics/completeness.yaml (5 categories: COM-01–05) - critics/code_hygiene.yaml (12 categories: CAT-01–12, 6 agentic: AP-01–06, ISO 25010 + CISQ mapped) - verdict-rules.yaml (deterministic aggregation, dedup, escalation) - learning/known_issues.json (empty scaffold) - rubrics/documentation.json (12-criteria DOC rubric) Updated: - SKILL.md: Rewritten as 4-critic orchestrator with dispatch tiers, progress indicators, error degradation, delegation boundary between code_hygiene and security critics. Each YAML contains: system prompt, evaluation categories, prompt template, rubric keyword filter, output schema, and pre-screen integration rules. Designed as single source of truth for multi-port architecture. Smoke-tested against golden test set artifact gt-py-001-sql-injection: 3-critic Sonnet run produced ~33 findings vs CLI thorough 36 findings, with comparable severity distribution and all critical issues caught.
CRITICAL fix:
- {PRESCREEN_EVIDENCE} now injected for security AND code_hygiene critics
(was incorrectly scoped to security only, breaking code_hygiene's
pre-screen integration design)
HIGH fixes:
- Document all 10 pre-screen checks (PS-001–PS-010) including PS-004/005/009/010
- Add --dispatch parameter to Parameters table
- Add error handlers: corrupt rubric JSON, invalid critic JSON response
- Fix verdict PASS condition: 'No findings or only INFO' (was undefined 'FAIL')
- Add DEGRADED/PARTIAL verdict behavior documentation
- Add evidence_tool to Step 4a validation requirements
MEDIUM fixes:
- Fix Quick Reference: 'task agents, sequential by default' (was 'parallel')
- Align critic dispatch order between diagram and progress messages
- Normalize 'Security Critic' naming in code_hygiene.yaml (was mixed CamelCase)
- Mark RELATIONSHIPS parameter and escalation rule as 'not yet ported'
- Sanitize error message templates (rubric names only, not full path listing)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extracts critic evaluation logic from Python classes into portable YAML definitions consumable by any Quorum port (CLI, Claude Code, Copilot).
What's New
4 Critic YAML Definitions
Each YAML contains: system prompt, evaluation categories with framework references, prompt template with variable placeholders, rubric keyword filter, output schema, and pre-screen integration rules.
Supporting Files
verdict-rules.yaml— Deterministic aggregation rules, deduplication, escalationrubrics/documentation.json— 12-criteria documentation quality rubriclearning/known_issues.json— Empty scaffold for pattern accumulationUpdated SKILL.md
Rewritten as 4-critic orchestrator with:
Design: Portable Critic Definitions
The YAML schema is designed as the single source of truth for multi-port architecture. Any port (CLI Python class, Claude Code skill, Copilot task agent) reads the same YAML and constructs platform-appropriate evaluation calls.
Smoke Test
Tested against golden test set artifact
gt-py-001-sql-injection.py: