Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .clawhubignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
reference-implementation/venv/
reference-implementation/quorum.egg-info/
golden-test-set/artifacts/
golden-test-set/annotations/
golden-test-set/scripts/
.hypothesis/
.pytest_cache/
__pycache__/
4 changes: 2 additions & 2 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ python -m quorum.cli run --target <path-to-artifact> --rubric <rubric-name>
### Depth Profiles

- `quick` — 2 critics (correctness, completeness) + pre-screen, ~5-10 min
- `standard` — 6 critics (+ security, code_hygiene, cross_consistency†, tester) + pre-screen, ~15-30 min (default)
- `thorough` — all 6 shipped critics + pre-screen + fix loops, ~30-60 min
- `standard` — 4 active (correctness, completeness, security + tester) + pre-screen, ~15-30 min (default)
- `thorough` — 5 active (+ code_hygiene) + pre-screen + fix loops, ~30-60 min

*†Cross-Consistency requires `--relationships` flag with a relationships manifest.*

Expand Down
14 changes: 7 additions & 7 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ Three execution profiles balance rigor, speed, and cost:
| Depth | Critics | Fix Loops | Runtime | Use Case |
|-------|---------|-----------|---------|----------|
| **quick** | Correctness, Completeness | 0 | 5-10 min | Fast feedback; low stakes |
| **standard** | + Security, Code Hygiene | 0 | 15-30 min | Most work; default |
| **standard** | + Security | 0 | 15-30 min | Most work; default |
| **thorough** | All shipped critics | 1 (apply + re-verify) | 30-60 min | Critical decisions; production |

Pre-screen (10 built-in checks + optional DevSkim SAST, §3.1) runs before LLM critics at all depth levels. Fix loops are implemented (Phase 1.5): the Fixer proposes text replacements for CRITICAL/HIGH findings, applies them, and re-runs critics to verify resolution. Full 9-critic panels are the roadmap target for thorough depth.
Expand Down Expand Up @@ -159,7 +159,7 @@ Supervisor (Orchestrator)
├─ Architecture Critic (Tier 2) [SPECIFIED, not yet built]
├─ Delegation Critic (Tier 1) [SPECIFIED, not yet built]
├─ Style Critic (Tier 2) [SPECIFIED, not yet built]
├─ Tester (Tier 2, tools: grep/web/exec) [SPECIFIED, not yet built]
├─ Tester (Tier 2, tools: grep/web/exec) [IMPLEMENTED] — L1 deterministic + L2 LLM claim verification; see docs/critics/TESTER_CRITIC_BRIEF.md
├─ Fixer (Tier 1, optional) [IMPLEMENTED — proposal mode + re-validation loops]
├─ Aggregator (Tier 1) [IMPLEMENTED]
└─ Supervisor (Tier 1, final) [IMPLEMENTED]
Expand Down Expand Up @@ -386,7 +386,7 @@ This follows Tomasev's "trust as runtime primitive" principle.
| Component | Cost | Amortization |
|-----------|------|--------------|
| Per-run setup (Supervisor intake) | $0.02 | 1 run |
| 4 shipped critics (parallel, max 30min; 9 at full build-out) | $0.15-0.45 | 1 run |
| 6 shipped critics (parallel, max 30min; 9 at full build-out) | $0.15-0.45 | 1 run |
| Aggregator synthesis | $0.01 | 1 run |
| Tester tools (grep, git, web, exec) | $0.00 | amortized |
| Learning update (`known_issues.json`) | $0.00 | amortized |
Expand All @@ -398,7 +398,7 @@ Additional runs on related artifacts reuse critic prompts and tools, amortizing

## 8. Implementation Checklist

Status as of v0.5.0 (reference implementation):
Status as of v0.7.2 (reference implementation):

- [x] LLM provider — LiteLLM universal provider (100+ models, any tier combination)
- [x] File-based artifact passing (no in-memory state between agents)
Expand All @@ -411,7 +411,7 @@ Status as of v0.5.0 (reference implementation):
- [x] Depth preset system (quick/standard/thorough YAML configs)
- [x] Path traversal security (boundary enforcement)
- [x] Exit codes (0/1/2)
- [x] Parallel critic dispatch (ThreadPoolExecutor, max 4 critics; batch files max 3)
- [x] Parallel critic dispatch (ThreadPoolExecutor, max 6 critics; batch files max 3)
- [x] Fixer agent — proposal mode + re-validation loops (Phase 1.5; proposes and applies text replacements for CRITICAL/HIGH, then re-runs critics to verify)
- [x] Python code rubric (25 criteria, PC-001–PC-025, auto-detects on .py files)
- [x] Learning memory system (known_issues.json frequency tracking + mandatory check promotion — shipped in v0.5.3)
Expand Down Expand Up @@ -444,7 +444,7 @@ Quorum is built on these peer-reviewed papers:
- Rubric panel is **static** (doesn't specialize per artifact type dynamically)
- **No critic-to-critic debate** (relies on Aggregator to resolve conflicts)
- Learning is **frequency-based** only (no semantic deduplication of patterns yet)
- **Confidence calibration** is not yet implemented
- **Confidence calibration** replaced by criteria coverage counts in v0.6.1
- **Trust/monitoring system** is not yet implemented

### Planned
Expand All @@ -470,7 +470,7 @@ Quorum is built on these peer-reviewed papers:
---

**Quorum is a production-oriented, early-stage validation framework.**
*Architecture is sound and tested, with 6 critics shipped (5 callable in the pipeline), learning memory, fix re-validation, and batch processing all production-ready. Trust and monitoring systems are specified but not yet wired.*
*Architecture is sound and tested, with 6 critics shipped (all callable), learning memory, fix re-validation, batch processing, and the Tester verification layer all production-ready. Trust and monitoring systems are specified but not yet wired.*


---
Expand Down
15 changes: 14 additions & 1 deletion docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [0.7.2] — 2026-03-12

### Documentation — Stale Content Cleanup (PR #18)

- **External reviews removed**: 3 frontier model review files (GPT-5.4, Opus 4.6, Grok 4.20) and ideas-backlog.md deleted — reviews were run against v0.5.3 and no longer reflect current capabilities. `EXTERNAL_REVIEWS.md` replaced with community invitation template.
- **13 stale content findings resolved** across SPEC.md, IMPLEMENTATION.md, CONFIG_REFERENCE.md, TUTORIAL.md, SKILL.md, RUBRIC_BUILDING_GUIDE.md, SECURITY_CRITIC_FRAMEWORK.md, CODE_HYGIENE_FRAMEWORK.md, TESTER_CRITIC_BRIEF.md, and INSTALLATION.md:
- Tester marked as shipped (was "SPECIFIED, not yet built")
- Critic counts updated to 6 shipped across all docs
- Learning memory marked as shipped (v0.5.3) across all docs
- Confidence scores replaced with coverage language (v0.6.1)
- Version headers updated to v0.7.2
- Depth profile critic counts aligned with critic-status.yaml
- Broken cross-reference paths fixed (TESTER_CRITIC_BRIEF, INSTALLATION)
- Security and Code Hygiene framework status checklists updated

### Documentation — Docs Restructure & README Rewrite (PR #16)

- **README.md** rewritten: ~400 lines → ~120 lines. Focused on what Quorum is, how to run it, and where to go next. First-person voice preserved.
Expand All @@ -14,7 +27,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- `architecture/` — IMPLEMENTATION, THE_NINE
- `critics/` — all critic-specific docs and frameworks
- `guides/` — RUBRIC_BUILDING_GUIDE, CROSS_ARTIFACT_DESIGN
- `reviews/` — external review snapshots
- `reviews/` — community and practitioner evaluations
- `configuration/` — CONFIG_REFERENCE
- **New files:** `docs/getting-started/QUICK_START.md` (zero to running in <5 min), `docs/README.md` (navigation hub)
- All cross-references updated across 30+ files
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Everything you need to use, configure, and extend Quorum.

| Doc | Description |
|-----|-------------|
| [External Reviews](reviews/EXTERNAL_REVIEWS.md) | Independent evaluations of Quorum by other models |
| [External Reviews](reviews/EXTERNAL_REVIEWS.md) | Community and practitioner evaluations |
| [Changelog](CHANGELOG.md) | Version history and release notes |
| [Code of Conduct](CODE_OF_CONDUCT.md) | Community guidelines |

Expand Down
18 changes: 10 additions & 8 deletions docs/architecture/IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This guide walks through building Quorum from the architectural spec. It's structured as a reference walkthrough, not a copy-paste tutorial. The goal is understanding the patterns so you can adapt them to your stack.

> **Implementation Status (v0.6.0):** This guide describes the full target architecture. The reference implementation ships 6 critics (5 callable in the pipeline), the fixer (proposal mode), parallel execution, batch processing, and pre-screen integration. Sections marked 🔜 describe planned components. See `critic-status.yaml` for the authoritative status matrix.
> **Implementation Status (v0.7.2):** This guide describes the full target architecture. All 6 critics are shipped and callable: Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, and Tester. Also shipped: the fixer (proposal mode), parallel execution, batch processing, and pre-screen integration. See `critic-status.yaml` for the authoritative status matrix. Sections below marked 🔜 describe components not yet built (Architecture, Delegation, Style).

---

Expand Down Expand Up @@ -127,7 +127,7 @@ Your job is assignment, coordination, and synthesis — not evaluation.

### 2.2 The Critics

> ✅ **Shipped:** Correctness, Security, Completeness, Code Hygiene (+ Cross-Artifact Consistency in batch mode)
> ✅ **Shipped:** Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, Tester
> 🔜 **Planned:** Architecture, Delegation & Coordination, Style

Each critic receives:
Expand Down Expand Up @@ -196,9 +196,9 @@ The hardest agent to get right. Its job:

---

## Phase 3: The Learning System 🔜
## Phase 3: The Learning System

> **Status:** Specified in architecture, not yet implemented. The mechanism below describes the target design.
> **Status:** Shipped in v0.5.3. The `known_issues.json` system tracks failure patterns across runs with frequency-based promotion to mandatory checks.

### 3.1 Extracting Lessons

Expand Down Expand Up @@ -247,8 +247,8 @@ This is how past failures automatically improve future validation.
### 4.1 Quick (5-10 min)

```yaml
critics: [correctness, security, completeness]
tester: enabled
critics: [correctness, completeness]
tester: disabled
fixer: disabled
aggregator: simplified (no conflict resolution)
fix_loops: 0
Expand All @@ -260,7 +260,7 @@ Use for: iterative development, fast feedback loops, low-stakes work.
### 4.2 Standard (15-30 min)

```yaml
critics: [all 5 critics + tester]
critics: [correctness, completeness, security + tester]
fixer: enabled for CRITICAL only
aggregator: full
fix_loops: 1
Expand All @@ -272,7 +272,7 @@ Use for: most production work, configuration reviews, research validation.
### 4.3 Thorough (45-90 min)

```yaml
critics: [all 5 critics + tester]
critics: [correctness, completeness, security, code_hygiene + tester]
fixer: enabled for CRITICAL + HIGH
aggregator: full + external validator
fix_loops: 2
Expand All @@ -282,6 +282,8 @@ cost_ceiling: $1.50

Use for: critical decisions, pre-launch reviews, irreversible actions.

**Note:** Cross-Artifact Consistency is a separate mode activated with the `--relationships` flag — it's additive at any depth, not part of the base critic panels above.

---

## Phase 5: Integration Patterns
Expand Down
14 changes: 6 additions & 8 deletions docs/configuration/CONFIG_REFERENCE.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Configuration Reference

> **Implementation Status (v0.5.2):** This document describes both shipped and planned features. Items marked 🔜 are specified but not yet implemented.
> **Implementation Status (v0.7.2):** This document describes both shipped and planned features. Items marked 🔜 are specified but not yet implemented.

## Depth Profiles

| Profile | Critics | Fix Rounds | Time | Use Case | Status |
|---------|---------|------------|------|----------|--------|
| **quick** | Shipped critics (currently 4) | 0 | 5–15 min | Iteration, drafts | ✅ Shipped |
| **quick** | Shipped critics (currently 2: correctness, completeness) | 0 | 5–15 min | Iteration, drafts | ✅ Shipped |
| **standard** | Shipped critics + fixer | 1 on CRITICAL | 15–30 min | Pre-merge, reviews | ✅ Shipped |
| **thorough** | All shipped critics + fixer | ≤2 on CRITICAL/HIGH | 30–60 min | Critical decisions, production | ✅ Shipped |
| *standard (full)* | *6 (+ Architecture, Delegation, Style)* | *1 on CRITICAL* | *15–30 min* | *Pre-merge, reviews* | 🔜 Planned |
Expand Down Expand Up @@ -68,16 +68,14 @@ Rubrics are JSON files with the following structure:

| Verdict | Meaning |
|---------|---------|
| **PASS** | No CRITICAL or HIGH issues; confidence0.8 |
| **PASS_WITH_NOTES** | No CRITICAL issues; ≤ 2 HIGH issues with mitigations; confidence0.6 |
| **PASS** | No CRITICAL or HIGH issues; coverage80% of criteria evaluated |
| **PASS_WITH_NOTES** | No CRITICAL issues; ≤ 2 HIGH issues with mitigations; coverage60% of criteria evaluated |
| **REVISE** | CRITICAL issues found but fixable; or > 2 HIGH issues |
| **REJECT** | Fundamental architectural or security issues; requires redesign |

## Known Issues / Learning Memory 🔜
## Known Issues / Learning Memory

> **Status:** Specified in architecture, not yet wired into production runs.

The `known_issues.json` file will accumulate failure patterns across validation runs:
> **Status:** Shipped in v0.5.3. The `known_issues.json` file accumulates failure patterns across validation runs:

```json
{
Expand Down
12 changes: 6 additions & 6 deletions docs/critics/CODE_HYGIENE_FRAMEWORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,19 @@ The boundary is clean: when code hygiene detects a pattern that has security imp

## Status

**v0.5.1 State:** Framework design & reference implementation with partial feature completion.
**v0.7.2 State:** Framework design complete. All features shipped.

- [x] Framework design and documentation
- [x] 12 evaluation categories specified
- [x] ISO/IEC 25010:2023 (Maintainability + Reliability) grounding
- [x] Two-layer architecture (deterministic + LLM) with delegation boundaries
- [ ] Full SAST tool integration (Ruff/Pylint) — Milestone #15, v0.5.2
- [ ] Business logic validation workflow (SEC-02) — v0.5.3 planned
- [ ] Learning memory wiring — v0.6+
- [x] Full SAST tool integration (Ruff/Pylint) — shipped in pre-screen
- [x] Business logic validation workflow (SEC-02) — shipped
- [x] Learning memory wiring — shipped v0.5.3

**Known Limitations:**
- Pre-screen layer runs 10 regex checks; deterministic Python analyzer not fully integrated
- Business logic checks require specification/requirements context (not yet automated)
- Ruff and Bandit run in pre-screen; full deterministic coverage continues to expand
- Business logic checks require specification/requirements context

---

Expand Down
13 changes: 6 additions & 7 deletions docs/critics/SECURITY_CRITIC_FRAMEWORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,19 @@

## Status

**v0.5.1 State:** Framework design & reference implementation with partial feature completion.
**v0.7.2 State:** Framework design complete. All features shipped.

- [x] Framework design and documentation
- [x] 14 evaluation categories (SEC-01–SEC-14) specified
- [x] OWASP ASVS 5.0, CWE Top 25, NIST SP 800-53 SA-11 grounding
- [x] Detection capability matrix for SAST vs LLM judgment boundaries
- [ ] Full SAST tool integration (Ruff/Bandit/PSScriptAnalyzer) — Milestone #15, v0.5.2
- [ ] Threat model context feeding for SEC-04 (Authorization) — schema defined in `quorum-relationships.example.yaml` (`threat_context` type); runtime integration v0.5.3 planned
- [ ] Learning memory wiring (issue tracking) — v0.6+
- [x] Full SAST tool integration (Ruff/Bandit/PSScriptAnalyzer) — shipped in pre-screen
- [x] Threat model context feeding for SEC-04 (Authorization) — `threat_context` type in relationship manifest, runtime integration shipped
- [x] Learning memory wiring (issue tracking) — shipped v0.5.3

**Known Limitations:**
- Pre-screen layer runs 10 regex checks; 80+ referenced SAST rules not yet integrated
- Authorization review (SEC-04) is speculative without threat model context — users can now define `threat_context` in the relationship manifest (roles, trust boundaries, sensitive operations); runtime feeding to the critic is planned for v0.5.3
- PowerShell coverage ~70% vs 85%+ for Python (tooling ecosystem gap)
- SAST rule coverage varies by language: Python ~85%+, PowerShell ~70% (tooling ecosystem gap)
- Authorization review (SEC-04) benefits from threat model context — users can define `threat_context` in the relationship manifest (roles, trust boundaries, sensitive operations)

---

Expand Down
13 changes: 6 additions & 7 deletions docs/critics/TESTER_CRITIC_BRIEF.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Tester Critic — Design Brief

**Status:** Pre-spec. This document describes what to build, not how. A full SPEC section should be written from this brief before any code.
**Status:** Shipped (v0.6.0). L1 deterministic + L2 LLM verification. Wired into pipeline as Phase 3 (v0.7.0). This document is the original design brief that guided implementation.

**Why this matters:** Three independent external reviews (GPT-5.4, Grok 4.20 Heavy, Opus 4.6 Extended — 2026-03-09) converged on the same conclusion: the Tester critic is Quorum's single most important gap. Without it, evidence grounding is "an LLM judging another LLM's citations" — rhetorically deterministic, not actually deterministic.
**Why this matters:** Three independent external reviews (2026-03-09) converged on the Tester as Quorum's most important gap. The Tester shipped in v0.6.0.

**Design principle reminder:** Quorum is the gate, not the road.

Expand Down Expand Up @@ -30,19 +30,19 @@ It answers: "The Correctness critic says line 247 of cli.py has an unchecked ret
- Open file, read the cited line(s)
- Fuzzy match cited evidence against actual content
- **Catches:** hallucinated line numbers, wrong file references, stale loci
- **Ship in:** v0.6.0
- **Shipped:** v0.6.0

### Level 2: Claim Verification (hybrid — deterministic + light LLM)
- Read the code/text surrounding the locus
- One focused LLM call: "Given this excerpt and this claim, is the claim accurate?"
- **Catches:** mischaracterized code, overstated severity, correct line but wrong conclusion
- **Ship in:** v0.6.0
- **Shipped:** v0.6.0

### Level 3: Execution Verification (deterministic, sandboxed)
- For executable claims (regex patterns, import checks, syntax validation): actually run them
- For coverage claims: run coverage analysis
- Requires sandboxing — significant security surface
- **Ship in:** v0.7.0 (deferred, gated behind `--tester-execute`)
- **Status:** Deferred, gated behind `--tester-execute`

## Output: Verification Tags

Expand Down Expand Up @@ -101,7 +101,6 @@ Before building the Tester itself:
- ~3 focused sessions, not one marathon

## References
- Opus 4.6 review: `docs/external-reviews/2026-03-09-opus-extended-review.md`
- GPT-5.4 review: `docs/external-reviews/2026-03-09-gpt54-review.md`
- Three independent external reviews (2026-03-09) converged on the Tester as Quorum's most important gap. The Tester shipped in v0.6.0.
- Current critic architecture: `SPEC.md` §Critics, §Aggregator
- Finding/Evidence/Locus models: `quorum/models.py`
6 changes: 3 additions & 3 deletions docs/getting-started/TUTORIAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ quorum run \

1. **Supervisor** reads `bad-config.yaml`, loads `swarm-config-rubric.json`, writes `run-manifest.json`

2. **Five critics** run in parallel:
2. **Three critics + Tester** run at standard depth:
- **Correctness Critic:** No factual errors, but notes the version is 1.0 with no changelog
- **Security Critic:** Finds `$USER_QUERY` in spawn pattern — CRITICAL injection vector
- **Completeness Critic:** Finds missing `model` on researcher — CRITICAL gap
Expand All @@ -97,7 +97,7 @@ quorum run \
```json
{
"verdict": "REVISE",
"confidence": 0.95,
"coverage": "12/14 criteria evaluated",
"summary": "2 CRITICAL, 3 HIGH, 1 MEDIUM findings. Shell injection and missing model assignment must be resolved before production use.",
"issues": [
{
Expand Down Expand Up @@ -202,7 +202,7 @@ That's Quorum working as designed.
- Try the research synthesis rubric on a LLM-generated report
- Write a custom rubric for your domain
- Run at `--depth thorough` for your next production deployment
- Check `known_issues.json` after a few runs — patterns will start emerging *(learning memory is specified but not yet wired)*
- Check `known_issues.json` after a few runs — learning memory tracks recurring patterns across runs (shipped v0.5.3)

---

Expand Down
Loading
Loading