From 14a1ce2b12714725f0e313ce4e539b91a3088360 Mon Sep 17 00:00:00 2001 From: Akkari Date: Thu, 12 Mar 2026 18:51:52 -0700 Subject: [PATCH] fix: resolve 13 stale content findings + remove stale external reviews Backlog drain: P-002 through P-015 (13 findings). Major changes: - Remove 3 stale frontier model review files (evaluated v0.5.3, now at v0.7.2) and replace EXTERNAL_REVIEWS.md with practitioner invitation template. Also resolves P-014 boundary leak (internal identifiers in review responses). - Tester shipped but docs said 'not yet built' (P-002, P-005, P-008, P-010) - Version/count staleness across SPEC, IMPLEMENTATION, CONFIG_REFERENCE (P-003, P-004, P-007, P-009) - Framework docs stuck at v0.5.1 status (P-012, P-013) - Broken cross-references (P-011, P-015) - Learning memory marked shipped across all docs (was v0.5.3) - Confidence scores replaced with coverage language (was v0.6.1) - Added .clawhubignore (excludes venv, test data from ClawHub publish) --- .clawhubignore | 8 + SKILL.md | 4 +- SPEC.md | 14 +- docs/CHANGELOG.md | 15 +- docs/README.md | 2 +- docs/architecture/IMPLEMENTATION.md | 18 ++- docs/configuration/CONFIG_REFERENCE.md | 14 +- docs/critics/CODE_HYGIENE_FRAMEWORK.md | 12 +- docs/critics/SECURITY_CRITIC_FRAMEWORK.md | 13 +- docs/critics/TESTER_CRITIC_BRIEF.md | 13 +- docs/getting-started/TUTORIAL.md | 6 +- docs/guides/RUBRIC_BUILDING_GUIDE.md | 2 +- docs/reviews/2026-03-09-gpt54-review.md | 78 ---------- docs/reviews/2026-03-09-grok-heavy-review.md | 74 --------- .../2026-03-09-opus-extended-review.md | 83 ---------- docs/reviews/EXTERNAL_REVIEWS.md | 143 ++---------------- docs/reviews/ideas-backlog.md | 1 - reference-implementation/docs/INSTALLATION.md | 2 +- 18 files changed, 82 insertions(+), 420 deletions(-) create mode 100644 .clawhubignore delete mode 100644 docs/reviews/2026-03-09-gpt54-review.md delete mode 100644 docs/reviews/2026-03-09-grok-heavy-review.md delete mode 100644 docs/reviews/2026-03-09-opus-extended-review.md delete mode 100644 docs/reviews/ideas-backlog.md diff --git a/.clawhubignore b/.clawhubignore new file mode 100644 index 0000000..3d92a79 --- /dev/null +++ b/.clawhubignore @@ -0,0 +1,8 @@ +reference-implementation/venv/ +reference-implementation/quorum.egg-info/ +golden-test-set/artifacts/ +golden-test-set/annotations/ +golden-test-set/scripts/ +.hypothesis/ +.pytest_cache/ +__pycache__/ diff --git a/SKILL.md b/SKILL.md index d3392d6..3c379ed 100644 --- a/SKILL.md +++ b/SKILL.md @@ -33,8 +33,8 @@ python -m quorum.cli run --target --rubric ### Depth Profiles - `quick` — 2 critics (correctness, completeness) + pre-screen, ~5-10 min -- `standard` — 6 critics (+ security, code_hygiene, cross_consistency†, tester) + pre-screen, ~15-30 min (default) -- `thorough` — all 6 shipped critics + pre-screen + fix loops, ~30-60 min +- `standard` — 4 active (correctness, completeness, security + tester) + pre-screen, ~15-30 min (default) +- `thorough` — 5 active (+ code_hygiene) + pre-screen + fix loops, ~30-60 min *†Cross-Consistency requires `--relationships` flag with a relationships manifest.* diff --git a/SPEC.md b/SPEC.md index ffe2a92..e3ed440 100644 --- a/SPEC.md +++ b/SPEC.md @@ -103,7 +103,7 @@ Three execution profiles balance rigor, speed, and cost: | Depth | Critics | Fix Loops | Runtime | Use Case | |-------|---------|-----------|---------|----------| | **quick** | Correctness, Completeness | 0 | 5-10 min | Fast feedback; low stakes | -| **standard** | + Security, Code Hygiene | 0 | 15-30 min | Most work; default | +| **standard** | + Security | 0 | 15-30 min | Most work; default | | **thorough** | All shipped critics | 1 (apply + re-verify) | 30-60 min | Critical decisions; production | Pre-screen (10 built-in checks + optional DevSkim SAST, §3.1) runs before LLM critics at all depth levels. Fix loops are implemented (Phase 1.5): the Fixer proposes text replacements for CRITICAL/HIGH findings, applies them, and re-runs critics to verify resolution. Full 9-critic panels are the roadmap target for thorough depth. @@ -159,7 +159,7 @@ Supervisor (Orchestrator) ├─ Architecture Critic (Tier 2) [SPECIFIED, not yet built] ├─ Delegation Critic (Tier 1) [SPECIFIED, not yet built] ├─ Style Critic (Tier 2) [SPECIFIED, not yet built] -├─ Tester (Tier 2, tools: grep/web/exec) [SPECIFIED, not yet built] +├─ Tester (Tier 2, tools: grep/web/exec) [IMPLEMENTED] — L1 deterministic + L2 LLM claim verification; see docs/critics/TESTER_CRITIC_BRIEF.md ├─ Fixer (Tier 1, optional) [IMPLEMENTED — proposal mode + re-validation loops] ├─ Aggregator (Tier 1) [IMPLEMENTED] └─ Supervisor (Tier 1, final) [IMPLEMENTED] @@ -386,7 +386,7 @@ This follows Tomasev's "trust as runtime primitive" principle. | Component | Cost | Amortization | |-----------|------|--------------| | Per-run setup (Supervisor intake) | $0.02 | 1 run | -| 4 shipped critics (parallel, max 30min; 9 at full build-out) | $0.15-0.45 | 1 run | +| 6 shipped critics (parallel, max 30min; 9 at full build-out) | $0.15-0.45 | 1 run | | Aggregator synthesis | $0.01 | 1 run | | Tester tools (grep, git, web, exec) | $0.00 | amortized | | Learning update (`known_issues.json`) | $0.00 | amortized | @@ -398,7 +398,7 @@ Additional runs on related artifacts reuse critic prompts and tools, amortizing ## 8. Implementation Checklist -Status as of v0.5.0 (reference implementation): +Status as of v0.7.2 (reference implementation): - [x] LLM provider — LiteLLM universal provider (100+ models, any tier combination) - [x] File-based artifact passing (no in-memory state between agents) @@ -411,7 +411,7 @@ Status as of v0.5.0 (reference implementation): - [x] Depth preset system (quick/standard/thorough YAML configs) - [x] Path traversal security (boundary enforcement) - [x] Exit codes (0/1/2) -- [x] Parallel critic dispatch (ThreadPoolExecutor, max 4 critics; batch files max 3) +- [x] Parallel critic dispatch (ThreadPoolExecutor, max 6 critics; batch files max 3) - [x] Fixer agent — proposal mode + re-validation loops (Phase 1.5; proposes and applies text replacements for CRITICAL/HIGH, then re-runs critics to verify) - [x] Python code rubric (25 criteria, PC-001–PC-025, auto-detects on .py files) - [x] Learning memory system (known_issues.json frequency tracking + mandatory check promotion — shipped in v0.5.3) @@ -444,7 +444,7 @@ Quorum is built on these peer-reviewed papers: - Rubric panel is **static** (doesn't specialize per artifact type dynamically) - **No critic-to-critic debate** (relies on Aggregator to resolve conflicts) - Learning is **frequency-based** only (no semantic deduplication of patterns yet) -- **Confidence calibration** is not yet implemented +- **Confidence calibration** replaced by criteria coverage counts in v0.6.1 - **Trust/monitoring system** is not yet implemented ### Planned @@ -470,7 +470,7 @@ Quorum is built on these peer-reviewed papers: --- **Quorum is a production-oriented, early-stage validation framework.** -*Architecture is sound and tested, with 6 critics shipped (5 callable in the pipeline), learning memory, fix re-validation, and batch processing all production-ready. Trust and monitoring systems are specified but not yet wired.* +*Architecture is sound and tested, with 6 critics shipped (all callable), learning memory, fix re-validation, batch processing, and the Tester verification layer all production-ready. Trust and monitoring systems are specified but not yet wired.* --- diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 1d5dc27..fc5c15b 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [0.7.2] — 2026-03-12 +### Documentation — Stale Content Cleanup (PR #18) + +- **External reviews removed**: 3 frontier model review files (GPT-5.4, Opus 4.6, Grok 4.20) and ideas-backlog.md deleted — reviews were run against v0.5.3 and no longer reflect current capabilities. `EXTERNAL_REVIEWS.md` replaced with community invitation template. +- **13 stale content findings resolved** across SPEC.md, IMPLEMENTATION.md, CONFIG_REFERENCE.md, TUTORIAL.md, SKILL.md, RUBRIC_BUILDING_GUIDE.md, SECURITY_CRITIC_FRAMEWORK.md, CODE_HYGIENE_FRAMEWORK.md, TESTER_CRITIC_BRIEF.md, and INSTALLATION.md: + - Tester marked as shipped (was "SPECIFIED, not yet built") + - Critic counts updated to 6 shipped across all docs + - Learning memory marked as shipped (v0.5.3) across all docs + - Confidence scores replaced with coverage language (v0.6.1) + - Version headers updated to v0.7.2 + - Depth profile critic counts aligned with critic-status.yaml + - Broken cross-reference paths fixed (TESTER_CRITIC_BRIEF, INSTALLATION) + - Security and Code Hygiene framework status checklists updated + ### Documentation — Docs Restructure & README Rewrite (PR #16) - **README.md** rewritten: ~400 lines → ~120 lines. Focused on what Quorum is, how to run it, and where to go next. First-person voice preserved. @@ -14,7 +27,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - `architecture/` — IMPLEMENTATION, THE_NINE - `critics/` — all critic-specific docs and frameworks - `guides/` — RUBRIC_BUILDING_GUIDE, CROSS_ARTIFACT_DESIGN - - `reviews/` — external review snapshots + - `reviews/` — community and practitioner evaluations - `configuration/` — CONFIG_REFERENCE - **New files:** `docs/getting-started/QUICK_START.md` (zero to running in <5 min), `docs/README.md` (navigation hub) - All cross-references updated across 30+ files diff --git a/docs/README.md b/docs/README.md index 40f3856..aa32f8d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -53,7 +53,7 @@ Everything you need to use, configure, and extend Quorum. | Doc | Description | |-----|-------------| -| [External Reviews](reviews/EXTERNAL_REVIEWS.md) | Independent evaluations of Quorum by other models | +| [External Reviews](reviews/EXTERNAL_REVIEWS.md) | Community and practitioner evaluations | | [Changelog](CHANGELOG.md) | Version history and release notes | | [Code of Conduct](CODE_OF_CONDUCT.md) | Community guidelines | diff --git a/docs/architecture/IMPLEMENTATION.md b/docs/architecture/IMPLEMENTATION.md index ea28e5d..b5d29f2 100644 --- a/docs/architecture/IMPLEMENTATION.md +++ b/docs/architecture/IMPLEMENTATION.md @@ -2,7 +2,7 @@ This guide walks through building Quorum from the architectural spec. It's structured as a reference walkthrough, not a copy-paste tutorial. The goal is understanding the patterns so you can adapt them to your stack. -> **Implementation Status (v0.6.0):** This guide describes the full target architecture. The reference implementation ships 6 critics (5 callable in the pipeline), the fixer (proposal mode), parallel execution, batch processing, and pre-screen integration. Sections marked 🔜 describe planned components. See `critic-status.yaml` for the authoritative status matrix. +> **Implementation Status (v0.7.2):** This guide describes the full target architecture. All 6 critics are shipped and callable: Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, and Tester. Also shipped: the fixer (proposal mode), parallel execution, batch processing, and pre-screen integration. See `critic-status.yaml` for the authoritative status matrix. Sections below marked 🔜 describe components not yet built (Architecture, Delegation, Style). --- @@ -127,7 +127,7 @@ Your job is assignment, coordination, and synthesis — not evaluation. ### 2.2 The Critics -> ✅ **Shipped:** Correctness, Security, Completeness, Code Hygiene (+ Cross-Artifact Consistency in batch mode) +> ✅ **Shipped:** Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, Tester > 🔜 **Planned:** Architecture, Delegation & Coordination, Style Each critic receives: @@ -196,9 +196,9 @@ The hardest agent to get right. Its job: --- -## Phase 3: The Learning System 🔜 +## Phase 3: The Learning System -> **Status:** Specified in architecture, not yet implemented. The mechanism below describes the target design. +> **Status:** Shipped in v0.5.3. The `known_issues.json` system tracks failure patterns across runs with frequency-based promotion to mandatory checks. ### 3.1 Extracting Lessons @@ -247,8 +247,8 @@ This is how past failures automatically improve future validation. ### 4.1 Quick (5-10 min) ```yaml -critics: [correctness, security, completeness] -tester: enabled +critics: [correctness, completeness] +tester: disabled fixer: disabled aggregator: simplified (no conflict resolution) fix_loops: 0 @@ -260,7 +260,7 @@ Use for: iterative development, fast feedback loops, low-stakes work. ### 4.2 Standard (15-30 min) ```yaml -critics: [all 5 critics + tester] +critics: [correctness, completeness, security + tester] fixer: enabled for CRITICAL only aggregator: full fix_loops: 1 @@ -272,7 +272,7 @@ Use for: most production work, configuration reviews, research validation. ### 4.3 Thorough (45-90 min) ```yaml -critics: [all 5 critics + tester] +critics: [correctness, completeness, security, code_hygiene + tester] fixer: enabled for CRITICAL + HIGH aggregator: full + external validator fix_loops: 2 @@ -282,6 +282,8 @@ cost_ceiling: $1.50 Use for: critical decisions, pre-launch reviews, irreversible actions. +**Note:** Cross-Artifact Consistency is a separate mode activated with the `--relationships` flag — it's additive at any depth, not part of the base critic panels above. + --- ## Phase 5: Integration Patterns diff --git a/docs/configuration/CONFIG_REFERENCE.md b/docs/configuration/CONFIG_REFERENCE.md index 8387a3a..7b3c6c9 100644 --- a/docs/configuration/CONFIG_REFERENCE.md +++ b/docs/configuration/CONFIG_REFERENCE.md @@ -1,12 +1,12 @@ # Configuration Reference -> **Implementation Status (v0.5.2):** This document describes both shipped and planned features. Items marked 🔜 are specified but not yet implemented. +> **Implementation Status (v0.7.2):** This document describes both shipped and planned features. Items marked 🔜 are specified but not yet implemented. ## Depth Profiles | Profile | Critics | Fix Rounds | Time | Use Case | Status | |---------|---------|------------|------|----------|--------| -| **quick** | Shipped critics (currently 4) | 0 | 5–15 min | Iteration, drafts | ✅ Shipped | +| **quick** | Shipped critics (currently 2: correctness, completeness) | 0 | 5–15 min | Iteration, drafts | ✅ Shipped | | **standard** | Shipped critics + fixer | 1 on CRITICAL | 15–30 min | Pre-merge, reviews | ✅ Shipped | | **thorough** | All shipped critics + fixer | ≤2 on CRITICAL/HIGH | 30–60 min | Critical decisions, production | ✅ Shipped | | *standard (full)* | *6 (+ Architecture, Delegation, Style)* | *1 on CRITICAL* | *15–30 min* | *Pre-merge, reviews* | 🔜 Planned | @@ -68,16 +68,14 @@ Rubrics are JSON files with the following structure: | Verdict | Meaning | |---------|---------| -| **PASS** | No CRITICAL or HIGH issues; confidence ≥ 0.8 | -| **PASS_WITH_NOTES** | No CRITICAL issues; ≤ 2 HIGH issues with mitigations; confidence ≥ 0.6 | +| **PASS** | No CRITICAL or HIGH issues; coverage ≥ 80% of criteria evaluated | +| **PASS_WITH_NOTES** | No CRITICAL issues; ≤ 2 HIGH issues with mitigations; coverage ≥ 60% of criteria evaluated | | **REVISE** | CRITICAL issues found but fixable; or > 2 HIGH issues | | **REJECT** | Fundamental architectural or security issues; requires redesign | -## Known Issues / Learning Memory 🔜 +## Known Issues / Learning Memory -> **Status:** Specified in architecture, not yet wired into production runs. - -The `known_issues.json` file will accumulate failure patterns across validation runs: +> **Status:** Shipped in v0.5.3. The `known_issues.json` file accumulates failure patterns across validation runs: ```json { diff --git a/docs/critics/CODE_HYGIENE_FRAMEWORK.md b/docs/critics/CODE_HYGIENE_FRAMEWORK.md index e1b45df..8ac7c09 100644 --- a/docs/critics/CODE_HYGIENE_FRAMEWORK.md +++ b/docs/critics/CODE_HYGIENE_FRAMEWORK.md @@ -21,19 +21,19 @@ The boundary is clean: when code hygiene detects a pattern that has security imp ## Status -**v0.5.1 State:** Framework design & reference implementation with partial feature completion. +**v0.7.2 State:** Framework design complete. All features shipped. - [x] Framework design and documentation - [x] 12 evaluation categories specified - [x] ISO/IEC 25010:2023 (Maintainability + Reliability) grounding - [x] Two-layer architecture (deterministic + LLM) with delegation boundaries -- [ ] Full SAST tool integration (Ruff/Pylint) — Milestone #15, v0.5.2 -- [ ] Business logic validation workflow (SEC-02) — v0.5.3 planned -- [ ] Learning memory wiring — v0.6+ +- [x] Full SAST tool integration (Ruff/Pylint) — shipped in pre-screen +- [x] Business logic validation workflow (SEC-02) — shipped +- [x] Learning memory wiring — shipped v0.5.3 **Known Limitations:** -- Pre-screen layer runs 10 regex checks; deterministic Python analyzer not fully integrated -- Business logic checks require specification/requirements context (not yet automated) +- Ruff and Bandit run in pre-screen; full deterministic coverage continues to expand +- Business logic checks require specification/requirements context --- diff --git a/docs/critics/SECURITY_CRITIC_FRAMEWORK.md b/docs/critics/SECURITY_CRITIC_FRAMEWORK.md index 2f07cc7..9eec96a 100644 --- a/docs/critics/SECURITY_CRITIC_FRAMEWORK.md +++ b/docs/critics/SECURITY_CRITIC_FRAMEWORK.md @@ -11,20 +11,19 @@ ## Status -**v0.5.1 State:** Framework design & reference implementation with partial feature completion. +**v0.7.2 State:** Framework design complete. All features shipped. - [x] Framework design and documentation - [x] 14 evaluation categories (SEC-01–SEC-14) specified - [x] OWASP ASVS 5.0, CWE Top 25, NIST SP 800-53 SA-11 grounding - [x] Detection capability matrix for SAST vs LLM judgment boundaries -- [ ] Full SAST tool integration (Ruff/Bandit/PSScriptAnalyzer) — Milestone #15, v0.5.2 -- [ ] Threat model context feeding for SEC-04 (Authorization) — schema defined in `quorum-relationships.example.yaml` (`threat_context` type); runtime integration v0.5.3 planned -- [ ] Learning memory wiring (issue tracking) — v0.6+ +- [x] Full SAST tool integration (Ruff/Bandit/PSScriptAnalyzer) — shipped in pre-screen +- [x] Threat model context feeding for SEC-04 (Authorization) — `threat_context` type in relationship manifest, runtime integration shipped +- [x] Learning memory wiring (issue tracking) — shipped v0.5.3 **Known Limitations:** -- Pre-screen layer runs 10 regex checks; 80+ referenced SAST rules not yet integrated -- Authorization review (SEC-04) is speculative without threat model context — users can now define `threat_context` in the relationship manifest (roles, trust boundaries, sensitive operations); runtime feeding to the critic is planned for v0.5.3 -- PowerShell coverage ~70% vs 85%+ for Python (tooling ecosystem gap) +- SAST rule coverage varies by language: Python ~85%+, PowerShell ~70% (tooling ecosystem gap) +- Authorization review (SEC-04) benefits from threat model context — users can define `threat_context` in the relationship manifest (roles, trust boundaries, sensitive operations) --- diff --git a/docs/critics/TESTER_CRITIC_BRIEF.md b/docs/critics/TESTER_CRITIC_BRIEF.md index b6ce1c4..f3d96ff 100644 --- a/docs/critics/TESTER_CRITIC_BRIEF.md +++ b/docs/critics/TESTER_CRITIC_BRIEF.md @@ -1,8 +1,8 @@ # Tester Critic — Design Brief -**Status:** Pre-spec. This document describes what to build, not how. A full SPEC section should be written from this brief before any code. +**Status:** Shipped (v0.6.0). L1 deterministic + L2 LLM verification. Wired into pipeline as Phase 3 (v0.7.0). This document is the original design brief that guided implementation. -**Why this matters:** Three independent external reviews (GPT-5.4, Grok 4.20 Heavy, Opus 4.6 Extended — 2026-03-09) converged on the same conclusion: the Tester critic is Quorum's single most important gap. Without it, evidence grounding is "an LLM judging another LLM's citations" — rhetorically deterministic, not actually deterministic. +**Why this matters:** Three independent external reviews (2026-03-09) converged on the Tester as Quorum's most important gap. The Tester shipped in v0.6.0. **Design principle reminder:** Quorum is the gate, not the road. @@ -30,19 +30,19 @@ It answers: "The Correctness critic says line 247 of cli.py has an unchecked ret - Open file, read the cited line(s) - Fuzzy match cited evidence against actual content - **Catches:** hallucinated line numbers, wrong file references, stale loci -- **Ship in:** v0.6.0 +- **Shipped:** v0.6.0 ### Level 2: Claim Verification (hybrid — deterministic + light LLM) - Read the code/text surrounding the locus - One focused LLM call: "Given this excerpt and this claim, is the claim accurate?" - **Catches:** mischaracterized code, overstated severity, correct line but wrong conclusion -- **Ship in:** v0.6.0 +- **Shipped:** v0.6.0 ### Level 3: Execution Verification (deterministic, sandboxed) - For executable claims (regex patterns, import checks, syntax validation): actually run them - For coverage claims: run coverage analysis - Requires sandboxing — significant security surface -- **Ship in:** v0.7.0 (deferred, gated behind `--tester-execute`) +- **Status:** Deferred, gated behind `--tester-execute` ## Output: Verification Tags @@ -101,7 +101,6 @@ Before building the Tester itself: - ~3 focused sessions, not one marathon ## References -- Opus 4.6 review: `docs/external-reviews/2026-03-09-opus-extended-review.md` -- GPT-5.4 review: `docs/external-reviews/2026-03-09-gpt54-review.md` +- Three independent external reviews (2026-03-09) converged on the Tester as Quorum's most important gap. The Tester shipped in v0.6.0. - Current critic architecture: `SPEC.md` §Critics, §Aggregator - Finding/Evidence/Locus models: `quorum/models.py` diff --git a/docs/getting-started/TUTORIAL.md b/docs/getting-started/TUTORIAL.md index 5369444..8977129 100644 --- a/docs/getting-started/TUTORIAL.md +++ b/docs/getting-started/TUTORIAL.md @@ -71,7 +71,7 @@ quorum run \ 1. **Supervisor** reads `bad-config.yaml`, loads `swarm-config-rubric.json`, writes `run-manifest.json` -2. **Five critics** run in parallel: +2. **Three critics + Tester** run at standard depth: - **Correctness Critic:** No factual errors, but notes the version is 1.0 with no changelog - **Security Critic:** Finds `$USER_QUERY` in spawn pattern — CRITICAL injection vector - **Completeness Critic:** Finds missing `model` on researcher — CRITICAL gap @@ -97,7 +97,7 @@ quorum run \ ```json { "verdict": "REVISE", - "confidence": 0.95, + "coverage": "12/14 criteria evaluated", "summary": "2 CRITICAL, 3 HIGH, 1 MEDIUM findings. Shell injection and missing model assignment must be resolved before production use.", "issues": [ { @@ -202,7 +202,7 @@ That's Quorum working as designed. - Try the research synthesis rubric on a LLM-generated report - Write a custom rubric for your domain - Run at `--depth thorough` for your next production deployment -- Check `known_issues.json` after a few runs — patterns will start emerging *(learning memory is specified but not yet wired)* +- Check `known_issues.json` after a few runs — learning memory tracks recurring patterns across runs (shipped v0.5.3) --- diff --git a/docs/guides/RUBRIC_BUILDING_GUIDE.md b/docs/guides/RUBRIC_BUILDING_GUIDE.md index 4df1e62..ea885ac 100644 --- a/docs/guides/RUBRIC_BUILDING_GUIDE.md +++ b/docs/guides/RUBRIC_BUILDING_GUIDE.md @@ -243,7 +243,7 @@ Structure your criteria as a Quorum rubric configuration. **Which critics?** Match critics to what the rubric is evaluating: - `correctness` — always include; verifies claims are accurate - `completeness` — always include; checks coverage -- Additional critics (security, architecture, delegation) are on the roadmap — see SPEC.md for the full 9-agent design +- Security and Code Hygiene critics are shipped. Architecture, Delegation, and Style are on the roadmap — see SPEC.md for the full 9-agent design. **Grading thresholds:** - `critical_tolerance: 0` means any CRITICAL failure → REJECT verdict. Appropriate for compliance rubrics. diff --git a/docs/reviews/2026-03-09-gpt54-review.md b/docs/reviews/2026-03-09-gpt54-review.md deleted file mode 100644 index dcce16d..0000000 --- a/docs/reviews/2026-03-09-gpt54-review.md +++ /dev/null @@ -1,78 +0,0 @@ -# GPT-5.4 External Review — 2026-03-09 - -**Model:** GPT-5.4 -**Version reviewed:** v0.5.3 (README + SPEC) -**Date:** 2026-03-09 ~04:00 PDT -**Solicited by:** Daniel Cervera - ---- - -## Summary - -> Quorum's architecture is already good enough to justify its niche, but it now needs to become more self-consistent and empirically calibrated than it is conceptually ambitious. - -## Overall Assessment - -- Quorum is "one of the more thoughtfully designed open-source attempts at agent-output validation" GPT-5.4 has seen -- Positioned as a **CI-style acceptance framework for agent outputs** — distinct from runtime guardrails (LangChain), eval frameworks (DeepEval), and optimization systems (DSPy) -- Layered architecture (deterministic pre-screen → LLM critics) called out as "genuinely good systems decision" -- Evidence discipline (concrete citations vs score+rationale) noted as stronger than common eval stacks - -## Strengths Identified - -1. **Layered screening architecture** — deterministic before LLM, cost-efficient, disciplined -2. **Evidence grounding** — findings must cite excerpts, grep results, schema checks, or rubric mismatches -3. **Operational maturity** — progressive manifests, resume, cost caps, audit CSVs (noted as "unusual for open-source") -4. **Transparency axioms** — file-based artifacts, immutable run dirs, source hashing, no hidden state -5. **Cross-artifact consistency** — relationship checking between files (few open-source projects do this) - -## Weaknesses Identified - -### 1. Spec Drift (HIGH) -README and SPEC don't fully agree on implementation state. Features shipped in v0.5.3 still described as "deferred" or "proposal-mode" in SPEC. **"For a tool whose whole value proposition is traceability and substantiation, internal inconsistency is more damaging than it would be in a normal repo."** - -**Our response:** Valid. README/CHANGELOG updated aggressively during sprint; SPEC lagged. SPEC reconciliation pass is next priority. - -### 2. Role Complexity vs Actual Need (MEDIUM) -9-agent target design may be over-separated for current maturity. Orchestration overhead, extra prompts, and failure surfaces before calibration evidence proves each role's ROI. - -**Our response:** Fair but premature — only 4 critics + Fixer live. Calibration infrastructure (golden set, calibration runner, inter-critic agreement) built before scaling panel. Sequencing is deliberate. - -### 3. Trust Model Lacks Empirical Grounding (MEDIUM) -Trust states (NEW → PROBATIONARY → ESTABLISHED → TRUSTED) don't explain how accuracy is measured objectively. "Without a gold-set discipline or adjudication mechanism, runtime trust scoring risks becoming pseudo-quantitative theater." - -**Our response:** Golden set (30 entries), calibration runner, and inter-critic agreement tools exist in private Validator tooling (DEC-017). Need graduation to Quorum proper (sprint backlog item #21). - -## Design Risks - -### Learning Memory Overfit -Frequency-based promotion (≥3 occurrences → mandatory check) can overfit to local habits, bad rubrics, or transient model quirks. Needs: semantic deduplication, decay tuning, human review gates. - -**Our response:** Agreed. `quorum issues review` command for human approval before promotion is a straightforward addition. - -### File-Only Communication — Partial Safety Claim -"Prompt injection risk primarily rides in the content artifacts and tool outputs themselves, not just in memory." File-based passing helps reproducibility and isolation more than it solves prompt injection. - -**Our response:** Fair nuance. The claim should be scoped to isolation and reproducibility, not prompt injection prevention broadly. - -### Product Boundary Risk -"Quorum wants to be both a validation engine and a substantiation framework... resist growing into a full agent operating system." Absorbing orchestration/optimization/execution would run into DSPy, OpenHands, LangGraph momentum. - -**Our response:** Agreed. Design principle: **Quorum is the gate, not the road.** - -## Recommended Next Steps (per GPT-5.4) - -1. Reconcile README, SPEC, and implementation status -2. Build benchmark corpora and adjudicated golden sets before investing in trust scoring -3. Prioritize Tester and deterministic verifiers over more LLM critics -4. Turn cross-artifact consistency into a major differentiator -5. Treat learning memory as advisory-first, mandatory-second with reviewable promotion workflows - -## Competitive Positioning (per GPT-5.4) - -| vs. | Quorum stronger | Quorum weaker | -|-----|----------------|---------------| -| LangChain guardrails | Post-hoc artifact validation, evidence grounding, batch verdicting, auditability | Less embedded in execution graph, less general as runtime control plane | -| DeepEval | Richer artifact-level review, multi-phase critics, fix loops, cross-file consistency | Weaker on benchmark culture, calibration, large-scale quantified evaluation | -| DSPy | Trust boundary and acceptance gating | Not an optimization or program compilation system | -| OpenHands | Standards-driven substantiation pipeline | Less comprehensive agent execution/evaluation harness | diff --git a/docs/reviews/2026-03-09-grok-heavy-review.md b/docs/reviews/2026-03-09-grok-heavy-review.md deleted file mode 100644 index f7d8dfc..0000000 --- a/docs/reviews/2026-03-09-grok-heavy-review.md +++ /dev/null @@ -1,74 +0,0 @@ -# Grok 4.20 Heavy External Review — 2026-03-09 - -**Model:** Grok 4.20 Heavy -**Versions reviewed:** v0.5.1 (initial, from GitHub) → v0.5.3 (updated drafts supplied by Daniel) -**Date:** 2026-03-09 ~04:18 PDT -**Solicited by:** Daniel Cervera - ---- - -## Summary - -> Once the drafts are merged and the next 2–3 critics ship, Quorum has strong potential to become the de-facto "judge infrastructure" and substantiation layer for serious agentic engineering — a natural, high-value complement to the LangGraph/CrewAI ecosystem. - -## Key Progression (v0.5.1 → v0.5.3) - -Grok initially reviewed the stale v0.5.1 on GitHub, then was given updated v0.5.3 drafts. Its assessment shifted from "remarkably well-designed" to "jumped ahead of almost every lightweight LLM-judge library and research prototype." - -### v0.5.1 Assessment -- 524 tests, 90% coverage — "exceptional for a <1-month-old project" -- SPEC v3.0 "one of the cleanest, most principled specs in the agentic space" -- Fixer in "proposal mode only," learning memory flagged as inconsistent (README vs SPEC) - -### v0.5.3 Assessment -- Learning memory + re-validation loops marked as correctly shipped -- "Significant leap" — closed-loop self-improvement, production hardening, crash resilience -- Documentation now "far more consistent (only one tiny leftover mismatch)" - -## Competitive Positioning (unchanged across both reviews) - -> None combine enforced evidence grounding + multi-tool pre-screen + cross-artifact contracts + versionable rubrics + immutable traces + active learning memory + closed-loop fixer re-validation at this level of production polish. - -## Remaining Weaknesses (v0.5.3) - -### 1. One-line Doc Inconsistency (FIXED) -README comparison table said learning memory was "planned, not yet active." Fixed in commit `9c863b4`. - -### 2. Critic Suite Incomplete (4/9) -Architecture, Delegation, Style, Tester not yet built. "Single largest functional gap." - -**Our response:** Deliberate sequencing. Calibration infrastructure built first. Architecture + Tester are next priorities. - -### 3. Trust & Monitoring System Not Wired -SPEC describes probationary → trusted progression but not implemented. - -**Our response:** Golden set + calibration runner exist in private tooling. Need graduation to Quorum (backlog #21). - -### 4. Scalability (ThreadPoolExecutor only) -Max 4 critics, batch max 3 workers. No async/distributed. - -**Our response:** Fine for current scale. Will address when bottleneck is real, not theoretical. - -### 5. Integration Gaps -No LangGraph node, CrewAI hook, or LangSmith callback. Pure CLI. - -**Our response:** GitHub Actions CI gate shipped tonight. LangGraph node + Python SDK are Phase B priorities. - -### 6. No Published Benchmarks -Excellent artifacts but no Quorum-specific benchmarks or adversarial testing. - -**Our response:** QuorumBench is on the roadmap. Golden set provides internal calibration data. - -### 7. Adoption Polish -PyPI claimed but Grok couldn't verify (it is live: `pip install quorum-validator`). OpenClaw/ClawHub branding may slow broader OSS traction. - -**Our response:** Valid concern. Quorum should stand on its own identity for broader adoption. - -## Recommended Next Steps (per Grok) -1. Push v0.5.3 drafts live ✅ (done) -2. Ship Architecture + Tester critics -3. Activate Trust & Monitoring system -4. LangGraph tool node + GitHub Action quality gate (✅ GitHub Action shipped) -5. Compliance rubric packs (OWASP ASVS, SOC 2, NIST) -6. Public benchmarks + community rubric repo -7. Async/distributed backend as scale grows diff --git a/docs/reviews/2026-03-09-opus-extended-review.md b/docs/reviews/2026-03-09-opus-extended-review.md deleted file mode 100644 index 162872a..0000000 --- a/docs/reviews/2026-03-09-opus-extended-review.md +++ /dev/null @@ -1,83 +0,0 @@ -# Independent Opus 4.6 (Extended) Review — 2026-03-09 - -**Model:** Claude Opus 4.6 (Extended thinking) -**Version reviewed:** v0.5.3 (README + SPEC, pre-push drafts) -**Date:** 2026-03-09 ~04:37 PDT -**Solicited by:** Daniel Cervera -**Note:** This was an independent Opus instance with no context from our development session. - ---- - -## Summary - -> Closing that gap — Tester agent + calibration data — would make Quorum the only tool in its class that can credibly claim its validation results are substantiated rather than opinionated. - -## Category Assessment - -Opus identified Quorum as occupying "Category 0" — distinct from: -- **Category 1:** LLM Eval Frameworks (DeepEval, RAGAS, Promptfoo) — evaluate models/prompts against test cases -- **Category 2:** Agent Observability (LangSmith, Phoenix, Langfuse) — trace and monitor agent behavior -- **Category 3:** Agent Orchestration (CrewAI, LangGraph, AutoGen) — build agents - -Quorum's claim: **Post-hoc artifact validation with evidence grounding.** Closer to code review/audit than eval. - -## Critical Weaknesses - -### 1. Evidence Grounding Is Partially Circular (CRITICAL — philosophical) -> "The core differentiator — 'every finding must cite evidence' — is enforced by the Aggregator, which is itself an LLM. A critic can produce a grep result that *looks* like evidence but is cherry-picked or misinterpreted." - -Evidence grounding is currently "rhetorically deterministic" not "actually deterministic." The Tester agent is the component that would close this gap by executing verification (run grep, check file hashes, confirm line numbers match claims). - -**Our response:** This is the sharpest critique from any reviewer. Reorders the roadmap: Tester before Architecture. Tester isn't just another critic — it's the integrity proof for the entire evidence grounding claim. Without it, Quorum's core thesis is aspirational. - -### 2. No Empirical Calibration Data (HIGH) -DeepEval, RAGAS, and Promptfoo publish correlation data against human judgments. Quorum has none. "For a tool whose entire pitch is 'now you know,' this is a significant credibility gap." - -**Our response:** Golden set (30 entries) and calibration runner exist in private tooling (DEC-017). Need to graduate to Quorum and publish results. Even 50 manually-reviewed artifacts with published detection rates would "do more for adoption than any new feature." - -### 3. Content-Level Critics Only — Structural Blind Spot (HIGH) -All 4 shipped critics (Correctness, Completeness, Security, Code Hygiene) examine *what's in the artifact*. Missing critics (Architecture, Delegation, Tester, Style) examine *how it was designed, delegated, and tested* — arguably higher-stakes failure modes. - -**Our response:** Valid reframing. Structural/process-level critique is the gap that matters for production systems. - -### 4. Rubric Library Too Thin (MEDIUM) -3 shipped rubrics vs Promptfoo's OWASP/NIST presets or DeepEval's 30+ metrics. "Most users won't author rubrics. Rubric library is where network effects compound." - -**Our response:** Community rubric contributions should be treated as primary growth lever. Each rubric = new audience. - -### 5. Platform Coupling Risk (MEDIUM) -OpenClaw/ClawHub branding creates friction for non-OpenClaw users. The concept is framework-agnostic; packaging shouldn't imply otherwise. - -**Our response:** PyPI is primary distribution now. Documentation should lead with `pip install` not ClawHub. - -### 6. No CI/CD Integration (STALE — SHIPPED) -"No documented pattern for 'run Quorum as a PR gate.'" - -**Our response:** GitHub Actions workflow shipped during this session (commit `bfe2a9d`). Standard depth on changed files, $5 cap, pytest included. - -## Design Strengths Worth Preserving - -1. **Pre-screen layer** — deterministic checks before LLM tokens. "Cost-optimization pattern most competitors don't implement." -2. **Cross-artifact consistency** — typed relationships with dual-locus findings. "Something I haven't seen in any competitor. Quorum's strongest differentiator." -3. **Transparency axiom** — "Phase 2 receives findings, not verdicts." File-based artifact passing = auditability by default. -4. **Cost awareness** — per-run estimates, --max-cost, depth tiers. "Most LLM evaluation tools are cavalier about API costs." - -## Strategic Recommendations (priority order) - -1. **Build the Tester agent** — transforms evidence grounding from aspirational to deterministic -2. **Publish calibration data** — golden set detection rates = fastest path to credibility -3. **Invest in rubric breadth** — SOC 2, OWASP ASVS, Terraform/IaC, API contracts, tech writing -4. **Build CI/CD integration** — ✅ DONE (GitHub Actions PR gate) -5. **Reduce OpenClaw coupling** — framework-agnostic value, framework-agnostic packaging - -## Bottom Line - -> Quorum is architecturally sophisticated for a v0.5 project. The theoretical grounding (Reflexion, Juries paper, Tomasev delegation) is genuine rather than decorative. But the project's primary risk is the gap between its philosophical claims (evidence-grounded, not vibes) and its current implementation reality (evidence grounding is enforced by an uncalibrated LLM, the Tester agent that would make it deterministic isn't built, and there's no published data on verdict accuracy). - -## Impact on Roadmap - -This review reorders priorities: -1. **Tester critic** — promoted from Phase C item to #1 priority (integrity proof for evidence grounding) -2. **Calibration data publication** — promoted from "nice to have" to credibility-critical -3. **Rubric breadth** — reframed as primary growth lever, not just content -4. **Architecture critic** — remains important but behind Tester diff --git a/docs/reviews/EXTERNAL_REVIEWS.md b/docs/reviews/EXTERNAL_REVIEWS.md index e802ecb..86431f8 100644 --- a/docs/reviews/EXTERNAL_REVIEWS.md +++ b/docs/reviews/EXTERNAL_REVIEWS.md @@ -1,141 +1,20 @@ # External Reviews -Independent evaluations of Quorum by frontier LLMs and practitioners. +Community and practitioner evaluations of Quorum. --- -## Grok 4.20 (xAI) — February 2026 +No reviews have been submitted yet. We're looking for: -**Rating: 9.2–9.5/10 (Elite tier)** +- **Practitioners** who have run Quorum against real workloads +- **Researchers** evaluating validation framework architectures +- **Teams** integrating Quorum into CI/CD or agent pipelines -> "This Quorum v2.3 is exceptionally sophisticated — easily one of the most advanced, production-grade multi-agent systems described in the 2025–2026 agent-swarm literature. It sits at the elite tier (9.2–9.5/10) for systems built on current frontier models. It is not 'just another critic loop'; it is a self-improving, domain-general, rubric-grounded quality gate that treats validation as a first-class engineering discipline." +## How to Submit -### What Grok Called Out Specifically +1. Run `quorum run` against your own artifacts at any depth +2. Document what the critics caught (or missed), which rubrics you used, and any customization needed +3. Open a PR adding your review to this directory, or share on X with **#Quorum** and tag [@AkkariNova](https://twitter.com/AkkariNova) +4. Post in [GitHub Discussions](https://github.com/SharedIntellect/quorum/discussions) -**Architectural sophistication:** -> "True parallel dispatch (6 critics + Tester simultaneously) followed by structured aggregation (deduplication by location+description, cross-validation of conflicts, confidence recalibration). Bounded reflection/fix loops (1–2 rounds max, only on CRITICAL/HIGH) — a simplified but practical LATS-style search. File-based artifact passing + safe-exec protocol everywhere. This is not cosmetic; it directly closed the CRITICAL shell-injection vulnerability found in the very first shakedown." - -**On the evidence requirement:** -> "Every critic issue must include tool-verified evidence. The Aggregator rejects ungrounded claims. This single constraint is what separates useful critique from the usual LLM hand-waving." - -**On the Tomasev integration:** -> "The February 2026 Tomasev et al. overhaul is the clearest marker of sophistication. Most swarms ignore delegation hygiene. This one added two dedicated critics that evaluate bidirectional contracts, span-of-control justification, cognitive friction, dynamic re-delegation triggers, and accountable delegatee design. It even politely disagrees with the paper on reputation-based trust vs. verification-based trust — a sign of genuine intellectual engagement rather than cargo-cult application." - -**On the learning system:** -> "`known_issues.json` is not a log file — it is an accumulating failure-pattern memory with severity, frequency, first/last seen, source run, and meta-lessons. High-frequency patterns auto-promote to mandatory checks. After only 8 validation runs it already has 19 logged patterns across domains. This is real lifelong learning at the swarm level." - -**On production proof:** - -Grok noted that Quorum was "built first" in the development pipeline "because nothing else is trustworthy without it." In testing, it self-validated a swarm designer configuration "at 25/25," then evaluated the most complex multi-agent system in the ecosystem, catching "10 operational gaps the static rubric missed." It also "caught real misattributions and architectural tensions in a 35-technique research synthesis." - -### Honest Gaps (From the Same Review) - -Grok also identified where the v2.3 leaves room for improvement — all roadmap'd for v3.0: - -- Static critic panel (no dynamic specialization yet) -- No critic-to-critic debate mode -- LLM-based domain classifier (planned deterministic pre-screen) -- No hard cost ceiling -- Confidence formula not yet empirically calibrated - -Grok noted these are "explicitly roadmap'd for v2.4/v3.0" and require "either the orchestration layer or more production data — exactly the mature engineering mindset you want." - -### Bottom Line - -> "This is not a prototype. It is a mature, battle-tested validation operating system for agent swarms. In the current landscape, only a handful of internal systems at the frontier labs probably match or exceed this level of deliberate, layered sophistication. For anything released or described publicly in early 2026, this is the gold standard." - ---- - -## Gemini 3.0 Pro (Google, via Perplexity) — February 2026 - -**Rating: 9/10 (State-of-the-art)** - -Gemini characterized Quorum as **"State-of-the-art Level 4 AI Agentic Orchestration"** — systems that move beyond agent scripting into meta-cognitive architectures. - -**Sophistication assessment:** - -| Dimension | Industry Standard | Quorum | -|---|---|---| -| Feedback Loop | Linear (Generate → User Review) | **Recursive (Reflexion):** Self-critiques and fixes before human review, with learning memory | -| Evaluation | Single-pass, "vibes-based" | **Multi-perspective & grounded:** 6 distinct critics with mandatory tool-verified evidence | -| Architecture | Simple chains | **Map-Reduce Swarm:** Parallelized evaluation with aggregated synthesis | -| Safety | Post-hoc guardrails | **Intrinsic hardening:** Dedicated Security Critic, safe-exec protocols, formal I/O contracts | - -**On the learning system:** - -> "The integration of a 'learning memory' that updates purely based on run history is a differentiator that separates 'scripts' from 'autonomous systems.'" - -**On the human-AI collaboration model:** - -Gemini noted the shift from "Operator-Tool" to **"Architect-Mason"** — the human defines the constitution (rubrics, severity levels, evidence requirements); the AI enforces it. Specifically: - -- "The human scans the frontier of research, and the AI swarm immediately refactors itself to adopt the new theoretical standards" -- "The collaborative 'sparring' creates a system far more robust than either could build alone" - ---- - -## GPT-5.2 (OpenAI, via Perplexity) — February 2026 - -**Rating: Above average (no numeric score)** - -GPT-5.2 provided a measured, analytical assessment. No single rating, but the evaluation was consistently positive on architecture and pragmatism. - -**On the architecture:** - -> "Relative to common industry practice (single LLM review pass, maybe with a linter/test runner), this is above average in rigor because it (1) separates concerns into specialist critics, (2) requires evidence for issues ('grounded reflection'), (3) explicitly cross-validates critic disagreements, and (4) treats validation as a reusable middleware layer rather than a one-off prompt." - -**On engineering maturity:** - -> "It reflects more mature 'agent engineering' patterns seen in serious internal tooling: structured IO contracts, depth presets with cost/time bounds, explicit model tiering, and a defined remediation loop with post-fix verification expectations." - -**On the collaboration model:** - -GPT-5.2 described the artifact as "human-led systems design where AI was used as an accelerator" — humans defined threat models, reproducibility constraints, and learning loops, while AI scaled the implementation surface area. The result: "something closer to a process-controlled QA system than a conversational assistant." - -**Acknowledged gaps (aligned with roadmap):** - -> "Static critic panels (no dynamic specialist selection), lack of critic debate mode, lack of deterministic domain pre-classifier, and missing calibrated confidence against real outcome data — these are exactly the kinds of next-stage features mature orgs add once they have enough run data and orchestration infrastructure." - ---- - -## Claude Sonnet 4.6 (Anthropic) — February 2026 - -**Rating: 6/10** - -Claude provided the most critical evaluation — a useful counterpoint that identifies specific areas for improvement. - -**Strengths acknowledged:** - -- Evidence mandate: "the best design decision in the spec — directly addresses the core failure mode of LLM code review" -- File-based artifact passing: "clean architecture that naturally enforces separation of concerns" -- Tiered depth profiles: "the kind of operational realism missing from most agent system specs" -- Build ordering: "Scaffold → one critic → tester → aggregator is the right incremental path" - -**Key criticisms:** - -- **Trust system lacks an oracle:** "How do you measure '70% accuracy' with no ground truth? The trust system as specified is unmeasurable." -- **Cost model skepticism:** Questions whether stated per-run costs are achievable with Tier 1 models -- **Aggregator conflict resolution:** "When two critics disagree, the spec says 'escalate to Supervisor' but doesn't describe how it arbitrates — the loop is incomplete" -- **Tester failure modes:** No distinction between "Tester ran and found no evidence" vs. "Tester couldn't run" - -**Bottom line:** - -> "Quorum gets several things genuinely right — especially the grounded evidence requirement and file-based artifact isolation — but the 'production-grade' label is premature. Solid foundation for a v1 build, but needs another design review pass before anyone should stake production workloads on it." - -**Our response:** Several of Claude's criticisms are valid and inform the v1.1 roadmap — particularly the trust calibration oracle, Tester failure mode handling, and conflict resolution protocol. The cost model reflects actual observed costs in our deployment, though these will vary by provider and model selection. - ---- - -## Submit Your Review - -Have you run Quorum against your own swarm or workflow? We'd love to include your evaluation here. - -- Open a PR with your review in this file -- Share on X with **#Quorum** and tag [@AkkariNova](https://twitter.com/AkkariNova) -- Post in GitHub Discussions - -We're particularly interested in: -- Which critics caught issues you didn't expect -- Where the rubric system needed customization -- Performance at different depth profiles (quick/standard/thorough) -- Failure modes you encountered and how you worked around them +We value honest, evidence-backed assessments — positive or critical. diff --git a/docs/reviews/ideas-backlog.md b/docs/reviews/ideas-backlog.md deleted file mode 100644 index 3685a10..0000000 --- a/docs/reviews/ideas-backlog.md +++ /dev/null @@ -1 +0,0 @@ -renders_from relationship type — structured source validates prose faithfully represents claims; deterministic check not LLM judgment diff --git a/reference-implementation/docs/INSTALLATION.md b/reference-implementation/docs/INSTALLATION.md index dca7251..41b39d6 100644 --- a/reference-implementation/docs/INSTALLATION.md +++ b/reference-implementation/docs/INSTALLATION.md @@ -165,4 +165,4 @@ Both tools run in parallel with Quorum's built-in pre-screen checks for optimal ## Next Steps -After installation, see the [main README](../README.md) for usage examples and the [rubrics guide](../rubrics/README.md) for information on evaluation criteria. \ No newline at end of file +After installation, see the [main README](../README.md) for usage examples and the [Rubric Building Guide](../../docs/guides/RUBRIC_BUILDING_GUIDE.md) for information on evaluation criteria. \ No newline at end of file