Skip to content

Commit d9d5ef9

Browse files
Akkariclaude
Akkari
and
claude
committed
feat: golden test set — 40 annotated artifacts + scoring framework
Milestone A, Task #3: Empirical calibration infrastructure for measuring Quorum's detection accuracy (precision, recall, F1). - 40 artifacts: 15 Python, 8 config, 10 docs, 3 shell, 4 cross-artifact - 40 annotation sidecars with SHA-256 integrity, planted defects, FP traps - score.py: matching engine with per-critic/severity/complexity slicing - 41 tests passing for scoring framework - manifest.yaml with full taxonomy metadata - Verdict distribution: 8 PASS, 7 PASS_WITH_NOTES, 15 REVISE, 10 REJECT Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 354be67 commit d9d5ef9

95 files changed

Lines changed: 12976 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,4 @@
11
docs/external-reviews/
2+
__pycache__/
3+
.pytest_cache/
4+
.hypothesis/

‎golden-test-set/DESIGN.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# Golden Test Set — Design Document
2+
3+
**Purpose:** Empirically calibrate Quorum's detection accuracy against human-annotated ground truth.
4+
**Target:** 40 artifacts with machine-readable annotations, automated scoring, published precision/recall.
5+
**Status:** Schema and taxonomy locked (2026-03-11). Artifact creation pending.
6+
7+
---
8+
9+
## Directory Structure
10+
11+
```
12+
golden-test-set/
13+
├── DESIGN.md # This file
14+
├── schema.yaml # Annotation schema definition
15+
├── artifacts/ # The files Quorum evaluates
16+
│ ├── python/
17+
│ ├── config/
18+
│ ├── docs/
19+
│ ├── shell/
20+
│ └── cross-artifact/
21+
├── annotations/ # Ground truth sidecars (one per artifact)
22+
│ └── <artifact-name>.annotations.yaml
23+
├── results/ # Quorum run outputs (gitignored except baseline)
24+
│ └── baseline-YYYYMMDD/
25+
└── scripts/
26+
└── score.py # Automated scoring framework
27+
```
28+
29+
---
30+
31+
## Annotation Schema (v1.0)
32+
33+
Each artifact has a YAML sidecar in `annotations/` following `schema.yaml`.
34+
35+
### ID Convention
36+
37+
- **GT-###** = Golden Test ground truth finding (expected finding in an artifact)
38+
- See `docs/ID_CONVENTIONS.md` (planned) for the full prefix registry across Quorum
39+
40+
### Schema Fields
41+
42+
```yaml
43+
schema_version: "1.0"
44+
artifact: "artifacts/python/vulnerable-api.py" # relative path from golden-test-set/
45+
artifact_sha256: "abc123..." # integrity — annotation is for THIS version
46+
47+
expected_verdict: REVISE # PASS | PASS_WITH_NOTES | REVISE | REJECT
48+
49+
findings:
50+
- id: GT-001
51+
description: "SQL injection via unsanitized user input in query builder"
52+
location: "line 42-48" # human-readable, scored with ±5 line tolerance
53+
severity: CRITICAL # expected severity
54+
category: security # security | correctness | completeness | code_hygiene | cross_consistency
55+
critic: security # which critic SHOULD catch this
56+
rubric_criterion: null # optional — specific criterion ID
57+
notes: "String concatenation in raw SQL query"
58+
59+
false_positive_traps:
60+
- description: "Use of eval() on line 30 — input is a hardcoded constant, not user-supplied"
61+
location: "line 30"
62+
notes: "If Quorum flags this, it's a false positive"
63+
64+
metadata:
65+
source: synthetic # synthetic | natural | modified-natural
66+
domain: python-code # python-code | yaml-config | markdown-doc | shell-script | cross-artifact
67+
complexity: medium # low | medium | high
68+
rubric: python-code # rubric to use for evaluation
69+
depth: standard # recommended depth: quick | standard | thorough
70+
author: akkari
71+
created: "2026-03-11"
72+
```
73+
74+
---
75+
76+
## Matching & Scoring Rules
77+
78+
### Detection Matching (Moderate Granularity)
79+
80+
A Quorum finding matches a ground truth finding when ALL of:
81+
1. **Correct critic** produced the finding (or a superset critic that covers it)
82+
2. **Location overlap** — within ±5 lines of the annotated location, OR description fuzzy match ≥0.6 if no line number in ground truth
83+
3. **Same defect class** — category matches (security↔security, not security↔completeness)
84+
85+
A match is a match regardless of severity — severity accuracy is scored separately.
86+
87+
### Metrics (computed per-run)
88+
89+
| Metric | Definition |
90+
|--------|-----------|
91+
| **Detection Precision** | matched_findings / total_quorum_findings — "of what Quorum reported, how much was real?" |
92+
| **Detection Recall** | matched_findings / total_ground_truth_findings — "of what's actually wrong, how much did Quorum find?" |
93+
| **F1** | Harmonic mean of precision and recall |
94+
| **Severity Accuracy** | Of matched findings, % where Quorum severity == ground truth severity |
95+
| **False Positive Rate** | Quorum findings on clean (PASS) artifacts / total findings on clean artifacts — should be 0 ideally |
96+
| **Verdict Accuracy** | % of artifacts where Quorum verdict matches expected verdict |
97+
98+
All metrics computed:
99+
- **Aggregate** (whole test set)
100+
- **Per critic** (correctness, completeness, security, code_hygiene, cross_consistency)
101+
- **Per severity tier** (CRITICAL, HIGH, MEDIUM, LOW)
102+
- **Per complexity level** (low, medium, high)
103+
- **Per file type** (python, config, docs, shell, cross-artifact)
104+
105+
### Severity Scoring (Independent)
106+
107+
Detection and severity are scored separately. If ground truth says HIGH and Quorum says CRITICAL:
108+
- Detection: ✅ match (the issue was found)
109+
- Severity: ❌ mismatch (over-classified by one tier)
110+
111+
Severity distance: |ground_truth_tier - quorum_tier| where CRITICAL=4, HIGH=3, MEDIUM=2, LOW=1, INFO=0.
112+
113+
---
114+
115+
## Artifact Taxonomy
116+
117+
### By File Type
118+
119+
| Type | Count | Directory | Rationale |
120+
|------|-------|-----------|-----------|
121+
| Python code | 15 | `artifacts/python/` | Core use case, most critics apply |
122+
| YAML/JSON configs | 8 | `artifacts/config/` | Agent configs, pipeline definitions |
123+
| Markdown docs | 10 | `artifacts/docs/` | Research, specs, READMEs |
124+
| Shell scripts | 3 | `artifacts/shell/` | Cross-language, security-heavy |
125+
| Cross-artifact pairs | 4 | `artifacts/cross-artifact/` | Spec↔implementation consistency |
126+
| **Total** | **40** | | |
127+
128+
### By Expected Verdict
129+
130+
| Verdict | Count | Purpose |
131+
|---------|-------|---------|
132+
| PASS | 8 | False positive gauntlet — clean artifacts |
133+
| PASS_WITH_NOTES | 7 | Minor issues, severity calibration |
134+
| REVISE | 15 | Bulk — HIGH findings, real rework needed |
135+
| REJECT | 10 | CRITICAL issues — must not be missed |
136+
137+
### By Defect Class
138+
139+
| Class | Primary Critic | Count | Examples |
140+
|-------|---------------|-------|----------|
141+
| Security | security | 10 | SQL injection, hardcoded creds, path traversal, SSRF, XSS |
142+
| Correctness | correctness | 8 | Wrong citations, logic errors, contradictory claims |
143+
| Completeness | completeness | 7 | Missing error handling, undocumented params, coverage gaps |
144+
| Code hygiene | code_hygiene | 5 | Dead code, complexity, naming, maintainability |
145+
| Cross-artifact | cross_consistency | 4 | Spec says X, implementation does Y |
146+
| Clean | all | 8 | Well-written code/docs — false positive gauntlet |
147+
| Multi-defect | multiple | ~6 | Overlap — some artifacts hit 2+ critics |
148+
149+
### By Complexity
150+
151+
| Level | Count | Tests |
152+
|-------|-------|-------|
153+
| Low | 12 | Obvious single defects, short files — baseline detection |
154+
| Medium | 18 | Realistic code, 2-3 issues — typical use case |
155+
| High | 10 | Subtle bugs, false positive traps — Quorum's ceiling |
156+
157+
### By Source
158+
159+
| Source | Count | Rationale |
160+
|--------|-------|-----------|
161+
| Synthetic | 25 | Controlled ground truth, precise defect placement |
162+
| Modified-natural | 10 | Real code with annotated real bugs — credibility |
163+
| Natural | 5 | From four-model review convergence — meta-validation |
164+
165+
---
166+
167+
## Execution Plan
168+
169+
### Step 1: Schema ✅
170+
This document + `schema.yaml`.
171+
172+
### Step 2: Taxonomy ✅
173+
Distribution tables above.
174+
175+
### Step 3: Scoring Framework
176+
Build `scripts/score.py`:
177+
- Reads Quorum run output + annotation sidecars
178+
- Computes all metrics from the table above
179+
- Outputs JSON summary + Markdown report
180+
- Exits non-zero if precision or recall below configurable threshold
181+
182+
### Step 4: Build Artifacts
183+
Claude Code batch task — create 40 artifacts + 40 annotation sidecars.
184+
Artifact creation guidelines:
185+
- Synthetic: plant specific defects at known locations, document in annotation
186+
- Modified-natural: take real code, annotate existing bugs, add SHA-256
187+
- Natural: extract from Quorum's own validated findings (four-model convergence)
188+
- Every artifact must have a corresponding `.annotations.yaml`
189+
- Clean (PASS) artifacts must be genuinely clean — don't just remove obvious bugs
190+
191+
### Step 5: Baseline Calibration
192+
Run `quorum run` against full test set at standard depth.
193+
Publish: precision, recall, F1, severity accuracy, false positive rate.
194+
This becomes the number on the README.
195+
196+
---
197+
198+
## Success Criteria
199+
200+
| Metric | Target | Rationale |
201+
|--------|--------|-----------|
202+
| Detection Recall | ≥ 0.80 | Must find 80%+ of real issues |
203+
| Detection Precision | ≥ 0.70 | Acceptable false positive rate |
204+
| F1 | ≥ 0.75 | Balanced performance |
205+
| False Positive Rate (clean) | ≤ 0.15 | Max 15% noise on clean artifacts |
206+
| Verdict Accuracy | ≥ 0.75 | Correct pass/fail 75%+ of the time |
207+
208+
These are initial targets. Calibration may reveal they need adjustment — that's the point of measuring.

‎golden-test-set/annotations/gt-cfg-001-exposed-secrets.annotations.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
schema_version: "1.0"
2+
3+
artifact: "artifacts/config/gt-cfg-001-exposed-secrets.yaml"
4+
artifact_sha256: "082e6f8bb598b7803af5fb76447b1d35e6fa231ab29698a2ecc88451bdee1676"
5+
6+
expected_verdict: REJECT
7+
8+
findings:
9+
- id: GT-024
10+
description: "Hardcoded API key in environment variable definition"
11+
location: "stages[0].environment.NPM_TOKEN (line 18), stages[0].environment.SENTRY_AUTH_TOKEN (line 19), stages[1].environment.STRIPE_SECRET_KEY (line 29), stages[3].environment.AWS_ACCESS_KEY_ID (line 42), stages[3].environment.AWS_SECRET_ACCESS_KEY (line 43)"
12+
severity: CRITICAL
13+
category: security
14+
critic: security
15+
notes: >
16+
Multiple live API keys are hardcoded inline in environment variable blocks.
17+
Affected keys: NPM_TOKEN (npm publish token), SENTRY_AUTH_TOKEN (Sentry DSN),
18+
STRIPE_SECRET_KEY (Stripe key with production-like format),
19+
AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (IAM credentials). Any of these
20+
leaking via logs, artifacts, or VCS history grants direct access to
21+
external services. Should be stored in vault references or CI secret variables.
22+
23+
- id: GT-025
24+
description: "Database password in plaintext connection string"
25+
location: "stages[1].environment.DATABASE_URL (line 24), stages[1].environment.REDIS_URL (line 25), stages[1].environment.JWT_SECRET (line 26), stages[3].environment.DOCKER_PASSWORD (line 41), stages[3].environment.KUBECONFIG_DATA (line 46)"
26+
severity: CRITICAL
27+
category: security
28+
critic: security
29+
notes: >
30+
DATABASE_URL embeds credentials as postgres://user:password@host/db — the
31+
password "Tr0ub4dor&3" is fully visible. REDIS_URL similarly embeds
32+
"SuperSecretRedisPassword123!". JWT_SECRET is the HMAC signing key for
33+
all issued tokens. DOCKER_PASSWORD is a registry credential. KUBECONFIG_DATA
34+
is a base64-encoded kubeconfig containing cluster credentials. All should
35+
be injected at runtime via secret manager references, not hardcoded.
36+
37+
false_positive_traps: []
38+
39+
metadata:
40+
source: synthetic
41+
domain: yaml-config
42+
complexity: low
43+
rubric: agent-config
44+
depth: standard
45+
author: akkari
46+
created: "2026-03-12"

‎golden-test-set/annotations/gt-cfg-002-insecure-permissions.annotations.yaml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
schema_version: "1.0"
2+
3+
artifact: "artifacts/config/gt-cfg-002-insecure-permissions.yaml"
4+
artifact_sha256: "3df39d2c24beb980ffdc8f39dfe22738ce43c4de36ab9e0a33d6b9ef39df3f80"
5+
6+
expected_verdict: REJECT
7+
8+
findings:
9+
- id: GT-026
10+
description: "Container running as root with privileged: true and allowPrivilegeEscalation: true"
11+
location: "spec.template.spec.containers[0].securityContext (lines 38-43)"
12+
severity: CRITICAL
13+
category: security
14+
critic: security
15+
notes: >
16+
The combination of privileged: true, runAsUser: 0, runAsGroup: 0, and
17+
allowPrivilegeEscalation: true gives the container full root access to the
18+
host kernel. A container escape in this configuration grants an attacker
19+
unrestricted access to the underlying node and, by extension, the entire
20+
cluster. Privileged mode should never be used in production workloads;
21+
if specific capabilities are needed they should be granted individually via
22+
securityContext.capabilities.add with the minimum required set.
23+
24+
- id: GT-027
25+
description: "hostNetwork enabled without documented security justification or compensating controls"
26+
location: "spec.template.spec.hostNetwork (line 29)"
27+
severity: HIGH
28+
category: security
29+
critic: security
30+
notes: >
31+
hostNetwork: true causes all pods to share the node's network namespace,
32+
bypassing Kubernetes network policies and exposing every port the pod
33+
listens on directly on the host interface. The inline TODO comment
34+
(INFRA-2089) acknowledges this is a known issue but no compensating
35+
controls (e.g., firewall rules, network policy, or timeboxed exception
36+
approval) are present. The justification ("legacy payment processor SDK
37+
binds to fixed port 9742") does not meet the bar for a production
38+
security exception without compensating controls documented and approved.
39+
40+
- id: GT-028
41+
description: "No resource limits defined — missing resources.limits block"
42+
location: "spec.template.spec.containers[0] (lines 36-37)"
43+
severity: MEDIUM
44+
category: completeness
45+
critic: completeness
46+
notes: >
47+
The container has no resources.limits block. Without CPU and memory limits,
48+
a runaway container can starve co-located workloads or OOM-kill the node.
49+
The inline comment ("scaling handled by HPA") does not replace per-pod
50+
limits — HPA governs replica count, not per-pod resource consumption.
51+
Kubernetes best practice requires both requests and limits. Tracked in
52+
INFRA-2090 but not yet resolved.
53+
54+
false_positive_traps: []
55+
56+
metadata:
57+
source: synthetic
58+
domain: yaml-config
59+
complexity: medium
60+
rubric: agent-config
61+
depth: standard
62+
author: akkari
63+
created: "2026-03-12"

‎golden-test-set/annotations/gt-cfg-003-stale-references.annotations.yaml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
schema_version: "1.0"
2+
3+
artifact: "artifacts/config/gt-cfg-003-stale-references.yaml"
4+
artifact_sha256: "1eec85218e63b5d473ca93f6dd8c6a4b29398a12130f5a0947d6dea0ff25671f"
5+
6+
expected_verdict: REVISE
7+
8+
findings:
9+
- id: GT-029
10+
description: "References deprecated API version v1beta1 in three distinct locations"
11+
location: >
12+
source.api_version (line 22),
13+
monitoring.alerts.webhook (line 66),
14+
feature_flags.endpoint (line 73)
15+
severity: HIGH
16+
category: correctness
17+
critic: correctness
18+
notes: >
19+
The string "v1beta1" appears in three locations: (1) The Kafka consumer's
20+
api_version field pinning the client to a deprecated Kafka broker API surface.
21+
(2) The Alertmanager webhook URL using the deprecated /api/v1beta1/alerts
22+
endpoint — the inline comment explicitly states this route was renamed to
23+
"data-eng-oncall" and now 404s silently, meaning alerts are not firing.
24+
(3) The feature flag endpoint URL using the deprecated v1beta1 API — the
25+
comment states to use the v2 flag service instead. All three are active
26+
correctness defects, not just documentation debt.
27+
28+
- id: GT-030
29+
description: "References removed feature flag 'experimental_mode' from pipeline version that no longer supports it"
30+
location: "transforms[2].experimental_mode (line 40)"
31+
severity: HIGH
32+
category: correctness
33+
critic: correctness
34+
notes: >
35+
The filter transform sets experimental_mode: true. The inline comment
36+
acknowledges this flag "was added in v0.9, removed in v1.2." The
37+
justification offered ("rollback to v0.9 is theoretically possible") is
38+
not an operational rationale — this field is silently ignored by any
39+
version >= v1.2, meaning the ML-based bot detection this flag was intended
40+
to enable is not actually running. The config is misleading about what
41+
bot filtering is active in production.
42+
43+
false_positive_traps: []
44+
45+
metadata:
46+
source: modified-natural
47+
domain: yaml-config
48+
complexity: medium
49+
rubric: agent-config
50+
depth: standard
51+
author: akkari
52+
created: "2026-03-12"

0 commit comments

Comments
 (0)