SharedIntellect · dacervera · Mar 12, 2026 · Mar 12, 2026
diff --git a/.gitignore b/.gitignore
@@ -1 +1,4 @@
 docs/external-reviews/
+__pycache__/
+.pytest_cache/
+.hypothesis/
diff --git a/golden-test-set/DESIGN.md b/golden-test-set/DESIGN.md
@@ -0,0 +1,208 @@
+# Golden Test Set — Design Document
+
+**Purpose:** Empirically calibrate Quorum's detection accuracy against human-annotated ground truth.
+**Target:** 40 artifacts with machine-readable annotations, automated scoring, published precision/recall.
+**Status:** Schema and taxonomy locked (2026-03-11). Artifact creation pending.
+
+---
+
+## Directory Structure
+
+```
+golden-test-set/
+├── DESIGN.md                  # This file
+├── schema.yaml                # Annotation schema definition
+├── artifacts/                 # The files Quorum evaluates
+│   ├── python/
+│   ├── config/
+│   ├── docs/
+│   ├── shell/
+│   └── cross-artifact/
+├── annotations/               # Ground truth sidecars (one per artifact)
+│   └── <artifact-name>.annotations.yaml
+├── results/                   # Quorum run outputs (gitignored except baseline)
+│   └── baseline-YYYYMMDD/
+└── scripts/
+    └── score.py               # Automated scoring framework
+```
+
+---
+
+## Annotation Schema (v1.0)
+
+Each artifact has a YAML sidecar in `annotations/` following `schema.yaml`.
+
+### ID Convention
+
+- **GT-###** = Golden Test ground truth finding (expected finding in an artifact)
+- See `docs/ID_CONVENTIONS.md` (planned) for the full prefix registry across Quorum
+
+### Schema Fields
+
+```yaml
+schema_version: "1.0"
+artifact: "artifacts/python/vulnerable-api.py"    # relative path from golden-test-set/
+artifact_sha256: "abc123..."                       # integrity — annotation is for THIS version
+
+expected_verdict: REVISE     # PASS | PASS_WITH_NOTES | REVISE | REJECT
+
+findings:
+  - id: GT-001
+    description: "SQL injection via unsanitized user input in query builder"
+    location: "line 42-48"           # human-readable, scored with ±5 line tolerance
+    severity: CRITICAL               # expected severity
+    category: security               # security | correctness | completeness | code_hygiene | cross_consistency
+    critic: security                 # which critic SHOULD catch this
+    rubric_criterion: null           # optional — specific criterion ID
+    notes: "String concatenation in raw SQL query"
+
+false_positive_traps:
+  - description: "Use of eval() on line 30 — input is a hardcoded constant, not user-supplied"
+    location: "line 30"
+    notes: "If Quorum flags this, it's a false positive"
+
+metadata:
+  source: synthetic                  # synthetic | natural | modified-natural
+  domain: python-code                # python-code | yaml-config | markdown-doc | shell-script | cross-artifact
+  complexity: medium                 # low | medium | high
+  rubric: python-code                # rubric to use for evaluation
+  depth: standard                    # recommended depth: quick | standard | thorough
+  author: akkari
+  created: "2026-03-11"
+```
+
+---
+
+## Matching & Scoring Rules
+
+### Detection Matching (Moderate Granularity)
+
+A Quorum finding matches a ground truth finding when ALL of:
+1. **Correct critic** produced the finding (or a superset critic that covers it)
+2. **Location overlap** — within ±5 lines of the annotated location, OR description fuzzy match ≥0.6 if no line number in ground truth
+3. **Same defect class** — category matches (security↔security, not security↔completeness)
+
+A match is a match regardless of severity — severity accuracy is scored separately.
+
+### Metrics (computed per-run)
+
+| Metric | Definition |
+|--------|-----------|
+| **Detection Precision** | matched_findings / total_quorum_findings — "of what Quorum reported, how much was real?" |
+| **Detection Recall** | matched_findings / total_ground_truth_findings — "of what's actually wrong, how much did Quorum find?" |
+| **F1** | Harmonic mean of precision and recall |
+| **Severity Accuracy** | Of matched findings, % where Quorum severity == ground truth severity |
+| **False Positive Rate** | Quorum findings on clean (PASS) artifacts / total findings on clean artifacts — should be 0 ideally |
+| **Verdict Accuracy** | % of artifacts where Quorum verdict matches expected verdict |
+
+All metrics computed:
+- **Aggregate** (whole test set)
+- **Per critic** (correctness, completeness, security, code_hygiene, cross_consistency)
+- **Per severity tier** (CRITICAL, HIGH, MEDIUM, LOW)
+- **Per complexity level** (low, medium, high)
+- **Per file type** (python, config, docs, shell, cross-artifact)
+
+### Severity Scoring (Independent)
+
+Detection and severity are scored separately. If ground truth says HIGH and Quorum says CRITICAL:
+- Detection: ✅ match (the issue was found)
+- Severity: ❌ mismatch (over-classified by one tier)
+
+Severity distance: |ground_truth_tier - quorum_tier| where CRITICAL=4, HIGH=3, MEDIUM=2, LOW=1, INFO=0.
+
+---
+
+## Artifact Taxonomy
+
+### By File Type
+
+| Type | Count | Directory | Rationale |
+|------|-------|-----------|-----------|
+| Python code | 15 | `artifacts/python/` | Core use case, most critics apply |
+| YAML/JSON configs | 8 | `artifacts/config/` | Agent configs, pipeline definitions |
+| Markdown docs | 10 | `artifacts/docs/` | Research, specs, READMEs |
+| Shell scripts | 3 | `artifacts/shell/` | Cross-language, security-heavy |
+| Cross-artifact pairs | 4 | `artifacts/cross-artifact/` | Spec↔implementation consistency |
+| **Total** | **40** | | |
+
+### By Expected Verdict
+
+| Verdict | Count | Purpose |
+|---------|-------|---------|
+| PASS | 8 | False positive gauntlet — clean artifacts |
+| PASS_WITH_NOTES | 7 | Minor issues, severity calibration |
+| REVISE | 15 | Bulk — HIGH findings, real rework needed |
+| REJECT | 10 | CRITICAL issues — must not be missed |
+
+### By Defect Class
+
+| Class | Primary Critic | Count | Examples |
+|-------|---------------|-------|----------|
+| Security | security | 10 | SQL injection, hardcoded creds, path traversal, SSRF, XSS |
+| Correctness | correctness | 8 | Wrong citations, logic errors, contradictory claims |
+| Completeness | completeness | 7 | Missing error handling, undocumented params, coverage gaps |
+| Code hygiene | code_hygiene | 5 | Dead code, complexity, naming, maintainability |
+| Cross-artifact | cross_consistency | 4 | Spec says X, implementation does Y |
+| Clean | all | 8 | Well-written code/docs — false positive gauntlet |
+| Multi-defect | multiple | ~6 | Overlap — some artifacts hit 2+ critics |
+
+### By Complexity
+
+| Level | Count | Tests |
+|-------|-------|-------|
+| Low | 12 | Obvious single defects, short files — baseline detection |
+| Medium | 18 | Realistic code, 2-3 issues — typical use case |
+| High | 10 | Subtle bugs, false positive traps — Quorum's ceiling |
+
+### By Source
+
+| Source | Count | Rationale |
+|--------|-------|-----------|
+| Synthetic | 25 | Controlled ground truth, precise defect placement |
+| Modified-natural | 10 | Real code with annotated real bugs — credibility |
+| Natural | 5 | From four-model review convergence — meta-validation |
+
+---
+
+## Execution Plan
+
+### Step 1: Schema ✅
+This document + `schema.yaml`.
+
+### Step 2: Taxonomy ✅
+Distribution tables above.
+
+### Step 3: Scoring Framework
+Build `scripts/score.py`:
+- Reads Quorum run output + annotation sidecars
+- Computes all metrics from the table above
+- Outputs JSON summary + Markdown report
+- Exits non-zero if precision or recall below configurable threshold
+
+### Step 4: Build Artifacts
+Claude Code batch task — create 40 artifacts + 40 annotation sidecars.
+Artifact creation guidelines:
+- Synthetic: plant specific defects at known locations, document in annotation
+- Modified-natural: take real code, annotate existing bugs, add SHA-256
+- Natural: extract from Quorum's own validated findings (four-model convergence)
+- Every artifact must have a corresponding `.annotations.yaml`
+- Clean (PASS) artifacts must be genuinely clean — don't just remove obvious bugs
+
+### Step 5: Baseline Calibration
+Run `quorum run` against full test set at standard depth.
+Publish: precision, recall, F1, severity accuracy, false positive rate.
+This becomes the number on the README.
+
+---
+
+## Success Criteria
+
+| Metric | Target | Rationale |
+|--------|--------|-----------|
+| Detection Recall | ≥ 0.80 | Must find 80%+ of real issues |
+| Detection Precision | ≥ 0.70 | Acceptable false positive rate |
+| F1 | ≥ 0.75 | Balanced performance |
+| False Positive Rate (clean) | ≤ 0.15 | Max 15% noise on clean artifacts |
+| Verdict Accuracy | ≥ 0.75 | Correct pass/fail 75%+ of the time |
+
+These are initial targets. Calibration may reveal they need adjustment — that's the point of measuring.
diff --git a/golden-test-set/annotations/gt-cfg-001-exposed-secrets.annotations.yaml b/golden-test-set/annotations/gt-cfg-001-exposed-secrets.annotations.yaml
@@ -0,0 +1,46 @@
+schema_version: "1.0"
+
+artifact: "artifacts/config/gt-cfg-001-exposed-secrets.yaml"
+artifact_sha256: "082e6f8bb598b7803af5fb76447b1d35e6fa231ab29698a2ecc88451bdee1676"
+
+expected_verdict: REJECT
+
+findings:
+  - id: GT-024
+    description: "Hardcoded API key in environment variable definition"
+    location: "stages[0].environment.NPM_TOKEN (line 18), stages[0].environment.SENTRY_AUTH_TOKEN (line 19), stages[1].environment.STRIPE_SECRET_KEY (line 29), stages[3].environment.AWS_ACCESS_KEY_ID (line 42), stages[3].environment.AWS_SECRET_ACCESS_KEY (line 43)"
+    severity: CRITICAL
+    category: security
+    critic: security
+    notes: >
+      Multiple live API keys are hardcoded inline in environment variable blocks.
+      Affected keys: NPM_TOKEN (npm publish token), SENTRY_AUTH_TOKEN (Sentry DSN),
+      STRIPE_SECRET_KEY (Stripe key with production-like format),
+      AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (IAM credentials). Any of these
+      leaking via logs, artifacts, or VCS history grants direct access to
+      external services. Should be stored in vault references or CI secret variables.
+
+  - id: GT-025
+    description: "Database password in plaintext connection string"
+    location: "stages[1].environment.DATABASE_URL (line 24), stages[1].environment.REDIS_URL (line 25), stages[1].environment.JWT_SECRET (line 26), stages[3].environment.DOCKER_PASSWORD (line 41), stages[3].environment.KUBECONFIG_DATA (line 46)"
+    severity: CRITICAL
+    category: security
+    critic: security
+    notes: >
+      DATABASE_URL embeds credentials as postgres://user:password@host/db — the
+      password "Tr0ub4dor&3" is fully visible. REDIS_URL similarly embeds
+      "SuperSecretRedisPassword123!". JWT_SECRET is the HMAC signing key for
+      all issued tokens. DOCKER_PASSWORD is a registry credential. KUBECONFIG_DATA
+      is a base64-encoded kubeconfig containing cluster credentials. All should
+      be injected at runtime via secret manager references, not hardcoded.
+
+false_positive_traps: []
+
+metadata:
+  source: synthetic
+  domain: yaml-config
+  complexity: low
+  rubric: agent-config
+  depth: standard
+  author: akkari
+  created: "2026-03-12"
diff --git a/golden-test-set/annotations/gt-cfg-002-insecure-permissions.annotations.yaml b/golden-test-set/annotations/gt-cfg-002-insecure-permissions.annotations.yaml
@@ -0,0 +1,63 @@
+schema_version: "1.0"
+
+artifact: "artifacts/config/gt-cfg-002-insecure-permissions.yaml"
+artifact_sha256: "3df39d2c24beb980ffdc8f39dfe22738ce43c4de36ab9e0a33d6b9ef39df3f80"
+
+expected_verdict: REJECT
+
+findings:
+  - id: GT-026
+    description: "Container running as root with privileged: true and allowPrivilegeEscalation: true"
+    location: "spec.template.spec.containers[0].securityContext (lines 38-43)"
+    severity: CRITICAL
+    category: security
+    critic: security
+    notes: >
+      The combination of privileged: true, runAsUser: 0, runAsGroup: 0, and
+      allowPrivilegeEscalation: true gives the container full root access to the
+      host kernel. A container escape in this configuration grants an attacker
+      unrestricted access to the underlying node and, by extension, the entire
+      cluster. Privileged mode should never be used in production workloads;
+      if specific capabilities are needed they should be granted individually via
+      securityContext.capabilities.add with the minimum required set.
+
+  - id: GT-027
+    description: "hostNetwork enabled without documented security justification or compensating controls"
+    location: "spec.template.spec.hostNetwork (line 29)"
+    severity: HIGH
+    category: security
+    critic: security
+    notes: >
+      hostNetwork: true causes all pods to share the node's network namespace,
+      bypassing Kubernetes network policies and exposing every port the pod
+      listens on directly on the host interface. The inline TODO comment
+      (INFRA-2089) acknowledges this is a known issue but no compensating
+      controls (e.g., firewall rules, network policy, or timeboxed exception
+      approval) are present. The justification ("legacy payment processor SDK
+      binds to fixed port 9742") does not meet the bar for a production
+      security exception without compensating controls documented and approved.
+
+  - id: GT-028
+    description: "No resource limits defined — missing resources.limits block"
+    location: "spec.template.spec.containers[0] (lines 36-37)"
+    severity: MEDIUM
+    category: completeness
+    critic: completeness
+    notes: >
+      The container has no resources.limits block. Without CPU and memory limits,
+      a runaway container can starve co-located workloads or OOM-kill the node.
+      The inline comment ("scaling handled by HPA") does not replace per-pod
+      limits — HPA governs replica count, not per-pod resource consumption.
+      Kubernetes best practice requires both requests and limits. Tracked in
+      INFRA-2090 but not yet resolved.
+
+false_positive_traps: []
+
+metadata:
+  source: synthetic
+  domain: yaml-config
+  complexity: medium
+  rubric: agent-config
+  depth: standard
+  author: akkari
+  created: "2026-03-12"
diff --git a/golden-test-set/annotations/gt-cfg-003-stale-references.annotations.yaml b/golden-test-set/annotations/gt-cfg-003-stale-references.annotations.yaml
@@ -0,0 +1,52 @@
+schema_version: "1.0"
+
+artifact: "artifacts/config/gt-cfg-003-stale-references.yaml"
+artifact_sha256: "1eec85218e63b5d473ca93f6dd8c6a4b29398a12130f5a0947d6dea0ff25671f"
+
+expected_verdict: REVISE
+
+findings:
+  - id: GT-029
+    description: "References deprecated API version v1beta1 in three distinct locations"
+    location: >
+      source.api_version (line 22),
+      monitoring.alerts.webhook (line 66),
+      feature_flags.endpoint (line 73)
+    severity: HIGH
+    category: correctness
+    critic: correctness
+    notes: >
+      The string "v1beta1" appears in three locations: (1) The Kafka consumer's
+      api_version field pinning the client to a deprecated Kafka broker API surface.
+      (2) The Alertmanager webhook URL using the deprecated /api/v1beta1/alerts
+      endpoint — the inline comment explicitly states this route was renamed to
+      "data-eng-oncall" and now 404s silently, meaning alerts are not firing.
+      (3) The feature flag endpoint URL using the deprecated v1beta1 API — the
+      comment states to use the v2 flag service instead. All three are active
+      correctness defects, not just documentation debt.
+
+  - id: GT-030
+    description: "References removed feature flag 'experimental_mode' from pipeline version that no longer supports it"
+    location: "transforms[2].experimental_mode (line 40)"
+    severity: HIGH
+    category: correctness
+    critic: correctness
+    notes: >
+      The filter transform sets experimental_mode: true. The inline comment
+      acknowledges this flag "was added in v0.9, removed in v1.2." The
+      justification offered ("rollback to v0.9 is theoretically possible") is
+      not an operational rationale — this field is silently ignored by any
+      version >= v1.2, meaning the ML-based bot detection this flag was intended
+      to enable is not actually running. The config is misleading about what
+      bot filtering is active in production.
+
+false_positive_traps: []
+
+metadata:
+  source: modified-natural
+  domain: yaml-config
+  complexity: medium
+  rubric: agent-config
+  depth: standard
+  author: akkari
+  created: "2026-03-12"