diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b68963e..3e453b5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -54,7 +54,7 @@ Improvements to docs, tutorials, and examples are always welcome. If something w ### π¬ External Reviews -Run Quorum against your own system and share the results. See `docs/EXTERNAL_REVIEWS.md` for how to submit. +Run Quorum against your own system and share the results. See `docs/reviews/EXTERNAL_REVIEWS.md` for how to submit. ## Pull Request Process diff --git a/README.md b/README.md index 913dc35..680961a 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@
@@ -15,27 +15,30 @@ You built something with your AI agent. A research report. A config. A codebase. Maybe a whole swarm produced it β five agents researching, synthesizing, writing β and now you're staring at the output wondering: -*"How do I _know_ this is actually right?"* +*"How do I know this is actually right?"* You could read every line yourself β but that defeats the point of having agents. You could ask the swarm to review its own work β but you already know that's just grading your own exam. **That's where I come in.** -I read what your agent produced. I bring in independent critics β each one focused on a different dimension β and they go through it carefully. Not vibes. Not "looks good to me." Every finding has to point to something specific in your work. If a critic can't show me the evidence, I throw out the finding. - -When I'm done, you get a clear answer. +I bring in independent critics β each one focused on a different dimension β and they go through your work carefully. Not vibes. Not "looks good to me." Every finding has to cite specific evidence from your artifact. If a critic can't show me the proof, I throw out the finding. ``` -You: "Run a quorum check on my-research-report.md" +$ quorum run --target my-report.md --depth standard + +Running Quorum (standard depth, critics: correctness, completeness, security) ... + + β QUORUM VERDICT: PASS_WITH_NOTES + ββββββββββββββββββββββββββββββββββ + Issues: 0 HIGH Β· 2 MEDIUM Β· 1 LOW/INFO (3 total) -Me: Spawning critics (correctness, completeness, security, code_hygiene)... - Evaluating against research-quality rubric... + 1. [MEDIUM] Β§2.4 claims "all implementations use AES-256" but Β§5.2 + references a ChaCha20 fallback β contradictory coverage claim + 2. [MEDIUM] Rubric criterion RC-007 (threat model) has no matching + content in any section + 3. [INFO] Minor: bibliography entry [14] not cited in text - Verdict: PASS_WITH_NOTES - ββ 3 claims need stronger citations [evidence: Β§2.4, Β§3.1, Β§5.2] - ββ Missing coverage of edge case X [evidence: rubric item 7, no match] - ββ Security: clean - ββ Architecture: well-structured, minor reordering suggestion + Cost: $0.42 (12,840 prompt + 3,201 completion tokens) ``` Now you know. Not because you hoped. Because it was checked. @@ -44,225 +47,89 @@ Now you know. Not because you hoped. Because it was checked. ## What Makes Me Different -You've got options. You could ask your agent to self-review. You could eyeball it. Here's what I do that they don't: - | The usual approach | What I do instead | |---|---| -| One model reviews its own output | I bring in **separate critics** that never saw the original prompt | -| "This looks great!" β it wrote it, of course it thinks so | My critics come in cold. **No bias from the creation process** | -| Vague suggestions you can't act on | **Every finding cites evidence** β an excerpt, a grep result, a schema check | -| LLM spends tokens on obvious problems | **Multi-layer pre-screen catches issues first** β 10 deterministic checks + DevSkim + Ruff + Bandit + PSScriptAnalyzer β before LLM runs | -| Reviews only one file at a time | **Batch validation** β run across a whole directory, or by `--pattern "*.md"`. One command, one verdict per file | -| Each file judged in isolation | **Cross-artifact consistency** β I check whether your files actually agree with each other via a relationships manifest | -| Same effort whether it's a quick sanity check or a full audit | I scale: **quick** ($0.15), **standard** ($0.50), **thorough** ($1.50+) | -| Each review starts from zero | I **learn patterns over time** β recurring findings auto-promote to mandatory checks via `known_issues.json` | - -Single agent, multi-agent swarm, hundred-step pipeline β doesn't matter how it was built. If it produced an output, I can tell you whether it holds up. +| One model reviews its own output | **Separate critics** that never saw the original prompt | +| "Looks great!" β it wrote it, of course it thinks so | Critics come in cold. **No bias from creation** | +| Vague suggestions you can't act on | **Every finding cites evidence** β excerpts, line numbers, schema checks | +| Same effort for a draft vs. production | **Three depth levels**: quick ($0.15), standard ($0.50), thorough ($1.50+) | +| Reviews one file at a time | **Batch + cross-artifact** β validate a directory with consistency checks between files | +| LLM wastes tokens on obvious problems | **Multi-layer pre-screen** catches issues first β 10 deterministic checks + DevSkim + Ruff + Bandit before LLM critics run | +| Each review starts from zero | **Learning memory** β recurring patterns auto-promote to mandatory checks | You wouldn't ship code without tests. I'm here so you don't ship AI outputs without validation either. --- -## The Real Question: Can You Prove It? +## The Deeper Question -Validation is the beginning, not the end. The deeper question β the one that matters in compliance, audits, and anything with stakes β is **substantiation**: +Validation is the beginning. The question that matters in compliance, audits, and anything with stakes is **substantiation**: *Can you prove the output meets a specific standard?* -*Can you prove that output meets a specific standard?* +That's what rubrics are for. A rubric encodes a standard β OWASP ASVS, SOC 2 controls, your internal style guide β as testable, machine-readable criteria. I evaluate evidence against those criteria. You get findings with citations, not feelings. -Not "it looks right." Not "the agent said so." Traced, cited, documented proof. - -That's where rubrics come in. A rubric isn't just a checklist. It's a machine-readable encoding of a standard β OWASP ASVS, SOC 2 controls, your internal style guide β that turns vague compliance questions into testable, evidence-grounded verdicts. I run those rubrics. Critics evaluate evidence against criteria. You get a finding with a citation, not a feeling. - -This is what I'm growing into: not just agent output validation, but a **substantiation framework** for anything a standard can express. +I ship with three rubrics (research-synthesis, agent-config, python-code). Need one for your domain? β [Rubric Building Guide](docs/guides/RUBRIC_BUILDING_GUIDE.md) --- -## Let's Get Started +## Install -**From PyPI:** +**PyPI** (recommended): ```bash pip install quorum-validator -export ANTHROPIC_API_KEY=your-key # or OPENAI_API_KEY, etc. quorum run --target your-file.py --depth standard ``` -**Or from source:** +**From source:** ```bash git clone https://github.com/SharedIntellect/quorum.git cd quorum/reference-implementation pip install -e . -quorum run --target examples/sample-research.md --depth quick ``` -Also available on ClawHub: `openclaw skills add dacervera/quorum` - -First time? I'll walk you through two quick setup questions β which model you have and how thorough you want me to be by default. I'll save your preferences so we only do this once. - -**Completely new to AI agent tooling?** No problem. β [FOR_BEGINNERS.md](docs/FOR_BEGINNERS.md) β I'll start from the very beginning. - ---- - -## You Decide How Deep I Go - -Not every artifact needs the full treatment. Tell me how much is riding on it, and I'll match my effort to the stakes. - -| Depth | Critics | Time | Cost* | When to use it | -|-------|---------|------|-------|----------------| -| **Quick** | 2 (correctness, completeness) | 5-10 min | ~$0.15 | "Give me a sanity check before I keep going" | -| **Standard** | 6 (+ security, code_hygiene, cross_consistencyβ , tester) | 15-30 min | ~$0.50 | Most work β solid coverage without the wait | -| **Thorough** | 6 shipped; more when they ship | 30-60 min | ~$1.50+ | "This is going to production. It cannot be wrong." | - -*β Cross-Consistency requires `--relationships` flag with a relationships manifest.* - -*Estimates on Claude Sonnet. Scales with model and artifact size. Pre-screen (10 deterministic checks + DevSkim + Ruff + Bandit + PSScriptAnalyzer) runs before LLM critics at every depth level β no extra cost. Today I ship with 6 critics (Correctness, Completeness, Security, Code Hygiene, Cross-Consistencyβ , and Tester) + re-validation loops + learning memory. The full architecture supports all 9 β Architecture, Delegation, and Style are next (see [SPEC.md](SPEC.md)).* - ---- - -## How I Work Under the Hood - -``` - You: "Validate this" - β - ββββββββββ΄βββββββββ - β Pre-Screen β 10 deterministic + DevSkim + Ruff + Bandit + PSSA β instant, no LLM - βββββββββ¬ββββββββββ (credentials, PII, syntax, broken links, TODOs, security patterns) - β prescreen.json - βββββββββ΄ββββββββββ - β Supervisor β I pick the right critics for the job - βββββββββ¬ββββββββββ - β Phase 1: spawns critics (parallel, ThreadPoolExecutor max 4) - ββββββββββββββββΌβββββββββββββββββββββββββββ - β Critics (working independently) β - β ββββββββββββ ββββββββββββ β - β βCorrectnessβ βCompletns β β shipped β - β ββββββββββββ ββββββββββββ β - β ββββββββββββ ββββββββββββ β - β β Security β βCodeHygineβ β shipped β - β ββββββββββββ ββββββββββββ β - β ββββββββββββ ββββββββββββ β - β βCrossConsistβ β Tester β β shipped β - β ββββββββββββ ββββββββββββ (β Cross-Consistency requires --relationships) - β ββββββββ ββββββββ ββββββββββ β - β β Arch β βDeleg β β Style β β roadmapβ - β ββββββββ ββββββββ ββββββββββ β - ββββββββββββββββ¬ββββββββββββββββββββββββββββ - β Phase 1.5: (if max_fix_loops > 0) - βββββββββ΄ββββββββββ - β Fixer Agent β proposes text replacements for CRITICAL/HIGH findings - βββββββββ¬ββββββββββ - β Phase 2: (if --relationships provided) - βββββββββ΄βββββββββββββββ - β Cross-Artifact β checks consistency between your files - β Consistency Critic β receives Phase 1 findings as context - βββββββββ¬βββββββββββββββ - β evidence-grounded findings (all phases) - βββββββββ΄ββββββββββ - β Aggregator β I merge findings, resolve conflicts, remove noise - βββββββββ¬ββββββββββ - β - βββββββββ΄ββββββββββ - β Verdict β PASS / PASS_WITH_NOTES / REVISE / REJECT - βββββββββββββββββββ +**OpenClaw / ClawHub:** +```bash +openclaw skills add dacervera/quorum ``` -You tell me what "good" looks like by giving me a rubric β a JSON file with your evaluation criteria. I ship with three built-in rubrics (research-synthesis, agent-config, python-code). Need one for your domain? β [RUBRIC_BUILDING_GUIDE.md](docs/RUBRIC_BUILDING_GUIDE.md) walks you through the process step by step. - -The research I'm built on: [Reflexion](https://arxiv.org/abs/2303.11366), [Replacing Judges with Juries](https://arxiv.org/abs/2404.18796), Intelligent Delegation (Tomasev et al., 2026), [LATM](https://arxiv.org/abs/2305.17126). Full architecture: [SPEC.md](SPEC.md). - ---- - -## What I Need From You - -Just a model that can reason well. I'll figure out the rest. +### Platform Ports -| Tier | Models | What to expect | -|------|--------|---------------| -| **Recommended** | Claude Opus 4.6+, GPT-5.2+ | Full capability β judgment-heavy work (thorough depth, cross-artifact consistency) benefits substantially from frontier reasoning | -| **Great** | Claude Sonnet 4.6+, Gemini 2.0+ | Excellent for quick and standard depth β most validation work lives here | -| **Functional** | Claude Haiku 4.5+, GPT-4o | I'll still help, but with less depth on nuanced findings | -| **Untested** | Llama 70B, most open models (early 2026) | Not yet evaluated in Quorum; may lack the reasoning depth for judgment-heavy criteria | +Quorum also runs natively inside other agent platforms β no API key required: -**Model routing tip:** For `--depth thorough` or `--relationships` runs, set `model_tier1` to a frontier model (Opus, GPT-5.2) in your config. Quick and standard runs work great with Sonnet-class models on both tiers. See depth configs in `quorum/configs/` for defaults. +| Port | What it does | Install | +|------|-------------|---------| +| **[Copilot CLI](ports/copilot-cli/)** | Full validation as a GitHub Copilot skill β stdlib-only pre-screen + 5 LLM critics | Copy `ports/copilot-cli/` to `~/.copilot/skills/quorum/` | +| **[Claude Code](ports/claude-code/)** | Quorum validation via Claude Code subscription tokens | Copy `ports/claude-code/` to your project | -I auto-detect your model on first run and configure myself accordingly. Details: [MODEL_REQUIREMENTS.md](docs/MODEL_REQUIREMENTS.md) +### Configuration -**Note on model assessment:** These recommendations are based on architectural requirements (reasoning depth, token budget for multi-stage evaluation). Assessments are not backed by empirical Quorum benchmarks yet. Users are encouraged to test with their preferred models. +You'll need a model that can reason well. I auto-detect your provider on first run. ---- +| Tier | Models | Best for | +|------|--------|----------| +| **Recommended** | Claude Opus, GPT-5+ | Thorough depth, cross-artifact consistency | +| **Great** | Claude Sonnet, Gemini 2.0+ | Quick and standard β where most work lives | +| **Functional** | Claude Haiku, GPT-4o | Lighter validation, less nuance on edge cases | -## Where I Am Right Now - -I'm working. I'm real. I'm also still growing. - -**What I can do today** (v0.6.0): -- Full CLI: `quorum run --targetMIT License Β· SharedIntellect Β· 2026
- - ---- - -> βοΈ **LICENSE** β Not part of the operational specification above. -> This file is part of [Quorum](https://github.com/SharedIntellect/quorum). -> Copyright 2026 SharedIntellect. MIT License. -> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms. diff --git a/SHIPPING.md b/SHIPPING.md index 92129df..b37031e 100644 --- a/SHIPPING.md +++ b/SHIPPING.md @@ -24,10 +24,10 @@ Run `python tools/validate-docs.py` to find stale references, then fix: - [ ] `README.md` β "What's coming" section (remove shipped items) - [ ] `SPEC.md` β status matrix / implementation checklist updated - [ ] `SPEC.md` β shipped critic counts in prose -- [ ] `docs/FOR_BEGINNERS.md` β critic counts and cost estimates -- [ ] `docs/CONFIG_REFERENCE.md` β critic status markers -- [ ] `docs/TUTORIAL.md` β critic descriptions and status markers -- [ ] `docs/IMPLEMENTATION.md` β critic status markers and depth profiles +- [ ] `docs/getting-started/FOR_BEGINNERS.md` β critic counts and cost estimates +- [ ] `docs/configuration/CONFIG_REFERENCE.md` β critic status markers +- [ ] `docs/getting-started/TUTORIAL.md` β critic descriptions and status markers +- [ ] `docs/architecture/IMPLEMENTATION.md` β critic status markers and depth profiles - [ ] `SKILL.md` β critic counts in depth descriptions **β οΈ Note on Cross-Consistency:** Cross-Consistency is a shipped critic but requires the `--relationships` flag. When updating counts, use language like: "6 shipped critics (including Cross-Consistency, activated with `--relationships`)". diff --git a/SKILL.md b/SKILL.md index 34300fa..d3392d6 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,6 +1,6 @@ --- name: quorum -description: Multi-agent validation framework β 4 independent AI critics evaluate artifacts against rubrics with evidence-grounded findings. +description: Multi-agent validation framework β 6 independent AI critics evaluate artifacts against rubrics with evidence-grounded findings. metadata: {"openclaw":{"requires":{"bins":["python3","pip"],"env":["ANTHROPIC_API_KEY","OPENAI_API_KEY"]},"install":[{"id":"clone-repo","kind":"shell","command":"git clone https://github.com/SharedIntellect/quorum.git /tmp/quorum-install && cd /tmp/quorum-install/reference-implementation && pip install -r requirements.txt","label":"Clone Quorum repo and install Python dependencies"}],"source":"https://github.com/SharedIntellect/quorum"}} --- @@ -100,9 +100,9 @@ Each finding includes: severity (CRITICAL/HIGH/MEDIUM/LOW), evidence citations p ## More Information - [SPEC.md](https://github.com/SharedIntellect/quorum/blob/main/SPEC.md) β Full architectural specification -- [MODEL_REQUIREMENTS.md](https://github.com/SharedIntellect/quorum/blob/main/docs/MODEL_REQUIREMENTS.md) β Supported models and tiers -- [CONFIG_REFERENCE.md](https://github.com/SharedIntellect/quorum/blob/main/docs/CONFIG_REFERENCE.md) β All configuration options -- [FOR_BEGINNERS.md](https://github.com/SharedIntellect/quorum/blob/main/docs/FOR_BEGINNERS.md) β New to agent validation? Start here +- [MODEL_REQUIREMENTS.md](https://github.com/SharedIntellect/quorum/blob/main/docs/getting-started/MODEL_REQUIREMENTS.md) β Supported models and tiers +- [CONFIG_REFERENCE.md](https://github.com/SharedIntellect/quorum/blob/main/docs/configuration/CONFIG_REFERENCE.md) β All configuration options +- [FOR_BEGINNERS.md](https://github.com/SharedIntellect/quorum/blob/main/docs/getting-started/FOR_BEGINNERS.md) β New to agent validation? Start here --- diff --git a/SPEC.md b/SPEC.md index 906d525..ffe2a92 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3,7 +3,7 @@ **Version:** 3.0 **Last Updated:** March 2026 **Status:** Documented and partially implemented β see Β§3 for implementation status per component -**Platform:** Designed for [OpenClaw](https://openclaw.ai) agent systems. Cross-platform compatibility with other agent frameworks is under active exploration β see [MODEL_REQUIREMENTS.md](docs/MODEL_REQUIREMENTS.md) for supported models and platforms. +**Platform:** Designed for [OpenClaw](https://openclaw.ai) agent systems. Cross-platform compatibility with other agent frameworks is under active exploration β see [MODEL_REQUIREMENTS.md](docs/getting-started/MODEL_REQUIREMENTS.md) for supported models and platforms. --- @@ -153,7 +153,7 @@ Pre-screen results are written to `prescreen.json` in the run directory. If any Supervisor (Orchestrator) ββ Correctness Critic (Tier 2) [IMPLEMENTED] ββ Completeness Critic (Tier 2) [IMPLEMENTED] -ββ Security Critic (Tier 1) [IMPLEMENTED] β grounded: OWASP ASVS 5.0, CWE Top 25, NIST SA-11; see docs/SEC02_BUSINESS_LOGIC_VALIDATION.md for business logic workflow +ββ Security Critic (Tier 1) [IMPLEMENTED] β grounded: OWASP ASVS 5.0, CWE Top 25, NIST SA-11; see docs/critics/SEC02_BUSINESS_LOGIC_VALIDATION.md for business logic workflow ββ Code Hygiene Critic (Tier 2) [IMPLEMENTED] β grounded: ISO 25010:2023, CISQ ββ Cross-Artifact Consistency [IMPLEMENTED] β Phase 2, separate from BaseCritic ββ Architecture Critic (Tier 2) [SPECIFIED, not yet built] @@ -462,9 +462,9 @@ Quorum is built on these peer-reviewed papers: ## 11. Getting Started -1. Read [IMPLEMENTATION.md](docs/IMPLEMENTATION.md) for a reference walkthrough +1. Read [IMPLEMENTATION.md](docs/architecture/IMPLEMENTATION.md) for a reference walkthrough 2. Review [examples/](examples/) for your use case -3. Adapt built-in rubrics or build custom ones (see [RUBRIC_BUILDING_GUIDE.md](docs/RUBRIC_BUILDING_GUIDE.md)) +3. Adapt built-in rubrics or build custom ones (see [RUBRIC_BUILDING_GUIDE.md](docs/guides/RUBRIC_BUILDING_GUIDE.md)) 4. Run the tutorial: `quorum run --target examples/sample-research.md --depth quick` --- diff --git a/critic-status.yaml b/critic-status.yaml index 003d80d..c7abcac 100644 --- a/critic-status.yaml +++ b/critic-status.yaml @@ -2,7 +2,7 @@ # Used by tools/validate-docs.py to detect stale documentation. # Update this file FIRST when shipping a critic or feature. -version: "0.7.0" +version: "0.7.2" spec_version: "3.0" critics: diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a282abe..1d5dc27 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -4,6 +4,29 @@ All notable changes to Quorum will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [0.7.2] β 2026-03-12 + +### Documentation β Docs Restructure & README Rewrite (PR #16) + +- **README.md** rewritten: ~400 lines β ~120 lines. Focused on what Quorum is, how to run it, and where to go next. First-person voice preserved. +- **23 docs reorganized** into 6 categorized subdirectories under `docs/`: + - `getting-started/` β QUICK_START, INSTALLATION, TUTORIAL, FOR_BEGINNERS, MODEL_REQUIREMENTS + - `architecture/` β IMPLEMENTATION, THE_NINE + - `critics/` β all critic-specific docs and frameworks + - `guides/` β RUBRIC_BUILDING_GUIDE, CROSS_ARTIFACT_DESIGN + - `reviews/` β external review snapshots + - `configuration/` β CONFIG_REFERENCE +- **New files:** `docs/getting-started/QUICK_START.md` (zero to running in <5 min), `docs/README.md` (navigation hub) +- All cross-references updated across 30+ files +- `validate-docs.py` exclusion updated to match new `docs/reviews/` path + +### Fixed +- 7 broken relative links caused by docs moving one directory deeper (CONFIG_REFERENCE, SPEC, CONTRIBUTING, reference-implementation paths) +- `SKILL.md` frontmatter: "4 independent AI critics" β "6" (matched shipped state) +- `README.md` version badge: v0.7.0 β v0.7.2 + +--- + ## [0.7.1] β 2026-03-12 ### Fixed β Self-Validation Findings (PR #14) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index 8e20629..16d45b8 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -25,7 +25,7 @@ Every source file gets a metadata block at the top, using `@KEY: value` pairs in ```python # @module: quorum.critics.code_hygiene # @purpose: Code quality evaluation grounded in ISO 25010 + CISQ -# @grounding: CODE_HYGIENE_FRAMEWORK.md +# @grounding: critics/CODE_HYGIENE_FRAMEWORK.md # @owner: code-hygiene-critic # @version: 0.3.0 # @tier: tier-2 (default model assignment) @@ -34,7 +34,7 @@ Every source file gets a metadata block at the top, using `@KEY: value` pairs in **Markdown:** ```markdown - + @@ -47,7 +47,7 @@ Every source file gets a metadata block at the top, using `@KEY: value` pairs in # @config: quorum-relationships.yaml # @purpose: Declared file relationships for cross-artifact consistency # @schema-version: 1.0 -# @see: CROSS_ARTIFACT_DESIGN.md +# @see: guides/CROSS_ARTIFACT_DESIGN.md ``` **Shell:** @@ -199,9 +199,9 @@ This enables a future pre-screen check (`PS-011: Header validation`) that mechan - [ ] `prescreen.py` ### Phase 3: Framework & Design Docs -- [ ] `CODE_HYGIENE_FRAMEWORK.md` -- [ ] `SECURITY_CRITIC_FRAMEWORK.md` -- [ ] `CROSS_ARTIFACT_DESIGN.md` +- [ ] `critics/CODE_HYGIENE_FRAMEWORK.md` +- [ ] `critics/SECURITY_CRITIC_FRAMEWORK.md` +- [ ] `guides/CROSS_ARTIFACT_DESIGN.md` - [ ] `SPEC.md` - [ ] `CHANGELOG.md` @@ -219,7 +219,7 @@ This enables a future pre-screen check (`PS-011: Header validation`) that mechan | Pipeline & CLI | `docs/PIPELINE_REFERENCE.md` | High β users need this | | Pre-Screen System | `docs/PRESCREEN_REFERENCE.md` | High β 10 checks need documentation | | Critic Framework | `docs/CRITIC_ARCHITECTURE.md` | Medium β developers extending Quorum | -| Rubric System | `docs/RUBRIC_BUILDING_GUIDE.md` | **Already exists** β | +| Rubric System | `docs/guides/RUBRIC_BUILDING_GUIDE.md` | **Already exists** β | | Cross-Artifact | `docs/CROSS_ARTIFACT_DESIGN.md` | **Already exists** β | --- diff --git a/docs/README.md b/docs/README.md index b4ebf91..40f3856 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,28 +1,73 @@ -# Quorum Documentation Index +# Quorum Documentation + +Everything you need to use, configure, and extend Quorum. + +--- ## Getting Started -- **[INSTALLATION.md](INSTALLATION.md)** β Setup, dependencies, and first run -- **[TUTORIAL.md](TUTORIAL.md)** β Step-by-step walkthrough -- **[FOR_BEGINNERS.md](FOR_BEGINNERS.md)** β What Quorum does and why, for newcomers -- **[CONFIG_REFERENCE.md](CONFIG_REFERENCE.md)** β Depth profiles, model tiers, and all configuration options - -## Architecture & Design -- **[THE_NINE.md](THE_NINE.md)** β The nine-agent target architecture -- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** β Current implementation status and technical details -- **[MODEL_REQUIREMENTS.md](MODEL_REQUIREMENTS.md)** β Why Quorum needs capable models and what "Tier 1/Tier 2" means -- **[CROSS_ARTIFACT_DESIGN.md](CROSS_ARTIFACT_DESIGN.md)** β Phase 2 cross-artifact consistency validation - -## Security & Code Quality Frameworks -- **[SECURITY_CRITIC_FRAMEWORK.md](SECURITY_CRITIC_FRAMEWORK.md)** β SEC-01 through SEC-14: the full security critic rule set -- **[SEC02_BUSINESS_LOGIC_VALIDATION.md](SEC02_BUSINESS_LOGIC_VALIDATION.md)** β Business logic validation workflow (requirements β critic path, threat_context integration) -- **[CODE_HYGIENE_FRAMEWORK.md](CODE_HYGIENE_FRAMEWORK.md)** β Code quality rules grounded in ISO 25010:2023 and CISQ -- **[POWERSHELL_COVERAGE.md](POWERSHELL_COVERAGE.md)** β Honest assessment of PowerShell SAST coverage (~70%) and landscape gaps - -## Building Rubrics -- **[RUBRIC_BUILDING_GUIDE.md](RUBRIC_BUILDING_GUIDE.md)** β How to create domain-specific rubrics for Quorum - -## Project -- **[CHANGELOG.md](CHANGELOG.md)** β Release history -- **[EXTERNAL_REVIEWS.md](EXTERNAL_REVIEWS.md)** β Independent evaluations of Quorum's architecture -- **[DOCUMENTATION_STANDARDS.md](DOCUMENTATION_STANDARDS.md)** β Standards for Quorum's own documentation -- **[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)** β Community guidelines + +| Doc | Description | +|-----|-------------| +| [Quick Start](getting-started/QUICK_START.md) | Your first validation in 60 seconds | +| [Installation](getting-started/INSTALLATION.md) | Detailed setup, dependencies, and troubleshooting | +| [For Beginners](getting-started/FOR_BEGINNERS.md) | New to AI agent tooling? Start here | +| [Tutorial](getting-started/TUTORIAL.md) | Walkthrough of a full validation workflow | +| [Model Requirements](getting-started/MODEL_REQUIREMENTS.md) | Which models work with Quorum and why | + +## Configuration + +| Doc | Description | +|-----|-------------| +| [Config Reference](configuration/CONFIG_REFERENCE.md) | Every config option, rubric format, and CLI flag | + +## Guides + +| Doc | Description | +|-----|-------------| +| [Rubric Building Guide](guides/RUBRIC_BUILDING_GUIDE.md) | How to build rubrics for new domains step by step | +| [Cross-Artifact Design](guides/CROSS_ARTIFACT_DESIGN.md) | Checking consistency between related files | + +## Architecture + +| Doc | Description | +|-----|-------------| +| [SPEC.md](../SPEC.md) | Full architectural specification | +| [The Nine Critics](architecture/THE_NINE.md) | The critic architecture β what each one does and why | +| [Implementation Notes](architecture/IMPLEMENTATION.md) | Technical decisions and internals | + +## Critics β Deep Dives + +| Doc | Description | +|-----|-------------| +| [Security Critic Framework](critics/SECURITY_CRITIC_FRAMEWORK.md) | OWASP ASVS 5.0, CWE Top 25, NIST SA-11 coverage | +| [Code Hygiene Framework](critics/CODE_HYGIENE_FRAMEWORK.md) | ISO 25010:2023, CISQ quality model | +| [Tester Critic Brief](critics/TESTER_CRITIC_BRIEF.md) | L1 deterministic + L2 LLM claim verification | +| [Business Logic Validation](critics/SEC02_BUSINESS_LOGIC_VALIDATION.md) | SEC-02 workflow for business logic review | +| [Evidence Integrity](critics/SEC03_EVIDENCE_INTEGRITY.md) | SEC-03 evidence grounding and integrity | +| [Pipeline Resilience](critics/SEC05_PIPELINE_RESILIENCE.md) | SEC-05 crash recovery and fault tolerance | +| [Rubric Framework](critics/SEC06_RUBRIC_FRAMEWORK.md) | SEC-06 rubric architecture and extensibility | +| [Self-Validation Convergence](critics/SEC07_SELF_VALIDATION_CONVERGENCE.md) | SEC-07 Quorum validating itself | +| [PowerShell Coverage](critics/POWERSHELL_COVERAGE.md) | PSScriptAnalyzer integration details | + +## Reviews & Research + +| Doc | Description | +|-----|-------------| +| [External Reviews](reviews/EXTERNAL_REVIEWS.md) | Independent evaluations of Quorum by other models | +| [Changelog](CHANGELOG.md) | Version history and release notes | +| [Code of Conduct](CODE_OF_CONDUCT.md) | Community guidelines | + +## Platform Ports + +Quorum runs natively in other agent platforms: + +| Port | Description | +|------|-------------| +| [Copilot CLI](../ports/copilot-cli/) | GitHub Copilot skill β stdlib-only pre-screen + 5 LLM critics | +| [Claude Code](../ports/claude-code/) | Claude Code subscription β zero API cost validation | + +## Contributing + +β [CONTRIBUTING.md](../CONTRIBUTING.md) + +We especially welcome rubric contributions for new domains. diff --git a/docs/IMPLEMENTATION.md b/docs/architecture/IMPLEMENTATION.md similarity index 100% rename from docs/IMPLEMENTATION.md rename to docs/architecture/IMPLEMENTATION.md diff --git a/docs/THE_NINE.md b/docs/architecture/THE_NINE.md similarity index 98% rename from docs/THE_NINE.md rename to docs/architecture/THE_NINE.md index 280bf42..b617b50 100644 --- a/docs/THE_NINE.md +++ b/docs/architecture/THE_NINE.md @@ -180,7 +180,7 @@ That's why there are nine. --- -*The characters in this story correspond to the nine agents defined in the [Quorum Specification](../SPEC.md). Their behaviors, evidence requirements, and interaction patterns are real. The drama is only slightly exaggerated.* +*The characters in this story correspond to the nine agents defined in the [Quorum Specification](../../SPEC.md). Their behaviors, evidence requirements, and interaction patterns are real. The drama is only slightly exaggerated.* --- diff --git a/docs/CONFIG_REFERENCE.md b/docs/configuration/CONFIG_REFERENCE.md similarity index 100% rename from docs/CONFIG_REFERENCE.md rename to docs/configuration/CONFIG_REFERENCE.md diff --git a/docs/CODE_HYGIENE_FRAMEWORK.md b/docs/critics/CODE_HYGIENE_FRAMEWORK.md similarity index 100% rename from docs/CODE_HYGIENE_FRAMEWORK.md rename to docs/critics/CODE_HYGIENE_FRAMEWORK.md diff --git a/docs/POWERSHELL_COVERAGE.md b/docs/critics/POWERSHELL_COVERAGE.md similarity index 100% rename from docs/POWERSHELL_COVERAGE.md rename to docs/critics/POWERSHELL_COVERAGE.md diff --git a/docs/SEC02_BUSINESS_LOGIC_VALIDATION.md b/docs/critics/SEC02_BUSINESS_LOGIC_VALIDATION.md similarity index 100% rename from docs/SEC02_BUSINESS_LOGIC_VALIDATION.md rename to docs/critics/SEC02_BUSINESS_LOGIC_VALIDATION.md diff --git a/docs/SEC03_EVIDENCE_INTEGRITY.md b/docs/critics/SEC03_EVIDENCE_INTEGRITY.md similarity index 100% rename from docs/SEC03_EVIDENCE_INTEGRITY.md rename to docs/critics/SEC03_EVIDENCE_INTEGRITY.md diff --git a/docs/SEC05_PIPELINE_RESILIENCE.md b/docs/critics/SEC05_PIPELINE_RESILIENCE.md similarity index 100% rename from docs/SEC05_PIPELINE_RESILIENCE.md rename to docs/critics/SEC05_PIPELINE_RESILIENCE.md diff --git a/docs/SEC06_RUBRIC_FRAMEWORK.md b/docs/critics/SEC06_RUBRIC_FRAMEWORK.md similarity index 100% rename from docs/SEC06_RUBRIC_FRAMEWORK.md rename to docs/critics/SEC06_RUBRIC_FRAMEWORK.md diff --git a/docs/SEC07_SELF_VALIDATION_CONVERGENCE.md b/docs/critics/SEC07_SELF_VALIDATION_CONVERGENCE.md similarity index 100% rename from docs/SEC07_SELF_VALIDATION_CONVERGENCE.md rename to docs/critics/SEC07_SELF_VALIDATION_CONVERGENCE.md diff --git a/docs/SECURITY_CRITIC_FRAMEWORK.md b/docs/critics/SECURITY_CRITIC_FRAMEWORK.md similarity index 100% rename from docs/SECURITY_CRITIC_FRAMEWORK.md rename to docs/critics/SECURITY_CRITIC_FRAMEWORK.md diff --git a/docs/TESTER_CRITIC_BRIEF.md b/docs/critics/TESTER_CRITIC_BRIEF.md similarity index 100% rename from docs/TESTER_CRITIC_BRIEF.md rename to docs/critics/TESTER_CRITIC_BRIEF.md diff --git a/docs/FOR_BEGINNERS.md b/docs/getting-started/FOR_BEGINNERS.md similarity index 99% rename from docs/FOR_BEGINNERS.md rename to docs/getting-started/FOR_BEGINNERS.md index 9ada84a..57d05d2 100644 --- a/docs/FOR_BEGINNERS.md +++ b/docs/getting-started/FOR_BEGINNERS.md @@ -66,7 +66,7 @@ That's it. No build step. No configuration files to wrestle with. No Docker cont ## "But where's the code?" -Right here: [`reference-implementation/`](../reference-implementation/) +Right here: [`reference-implementation/`](../../reference-implementation/) ```bash cd reference-implementation diff --git a/docs/INSTALLATION.md b/docs/getting-started/INSTALLATION.md similarity index 92% rename from docs/INSTALLATION.md rename to docs/getting-started/INSTALLATION.md index 0f4edef..a860a5e 100644 --- a/docs/INSTALLATION.md +++ b/docs/getting-started/INSTALLATION.md @@ -55,7 +55,7 @@ models: depth: standard # quick | standard | thorough ``` -See [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md) for full options and [MODEL_REQUIREMENTS.md](MODEL_REQUIREMENTS.md) for supported models. +See [CONFIG_REFERENCE.md](../configuration/CONFIG_REFERENCE.md) for full options and [MODEL_REQUIREMENTS.md](MODEL_REQUIREMENTS.md) for supported models. ## Requirements @@ -66,7 +66,7 @@ See [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md) for full options and [MODEL_REQUI ## Troubleshooting -**"No model configured"** β Run `quorum run` once and follow the first-run setup prompts, or manually create `quorum-config.yaml` (see [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md)). +**"No model configured"** β Run `quorum run` once and follow the first-run setup prompts, or manually create `quorum-config.yaml` (see [CONFIG_REFERENCE.md](../configuration/CONFIG_REFERENCE.md)). **"API key not found"** β Set your provider's API key as an environment variable: ```bash diff --git a/docs/MODEL_REQUIREMENTS.md b/docs/getting-started/MODEL_REQUIREMENTS.md similarity index 95% rename from docs/MODEL_REQUIREMENTS.md rename to docs/getting-started/MODEL_REQUIREMENTS.md index 017da95..b7ae01d 100644 --- a/docs/MODEL_REQUIREMENTS.md +++ b/docs/getting-started/MODEL_REQUIREMENTS.md @@ -4,7 +4,7 @@ Quorum's critics need models that can reason about complex artifacts, follow str ## Tier System -Quorum uses a two-tier architecture. **Tier 1** handles judgment-heavy roles (supervisor, aggregator, security critic). **Tier 2** handles pattern-matching roles (correctness, completeness, architecture critics). See [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md) for which roles map to which tier. +Quorum uses a two-tier architecture. **Tier 1** handles judgment-heavy roles (supervisor, aggregator, security critic). **Tier 2** handles pattern-matching roles (correctness, completeness, architecture critics). See [CONFIG_REFERENCE.md](../configuration/CONFIG_REFERENCE.md) for which roles map to which tier. ### Tier 1 β Recommended (Judgment Roles) @@ -49,7 +49,7 @@ models: tier_2: anthropic/claude-sonnet-4-6 # Evaluation roles (correctness, completeness, architecture) ``` -For per-critic overrides, see [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md). +For per-critic overrides, see [CONFIG_REFERENCE.md](../configuration/CONFIG_REFERENCE.md). **First-run behavior:** If no config exists, Quorum auto-detects your available model, asks two setup questions (preferred tier 1 and tier 2 models), writes the config, and proceeds immediately. Zero-config users get sensible defaults. diff --git a/docs/getting-started/QUICK_START.md b/docs/getting-started/QUICK_START.md new file mode 100644 index 0000000..883d685 --- /dev/null +++ b/docs/getting-started/QUICK_START.md @@ -0,0 +1,69 @@ +# Quick Start + +Get your first Quorum validation running in 60 seconds. + +## 1. Install + +```bash +pip install quorum-validator +``` + +## 2. Set your model provider + +```bash +export ANTHROPIC_API_KEY=your-key +# or: export OPENAI_API_KEY=your-key +# Quorum uses LiteLLM β 100+ models supported +``` + +## 3. Run your first validation + +```bash +quorum run --target your-file.py --depth quick +``` + +That's it. Quorum auto-detects the file type, picks the right rubric, spawns critics, and gives you a verdict. + +## What You'll See + +``` +Running Quorum (quick depth, critics: correctness, completeness) ... + + β QUORUM VERDICT: PASS_WITH_NOTES + ββββββββββββββββββββββββββββββββββ + Issues: 0 HIGH Β· 1 MEDIUM Β· 2 LOW/INFO (3 total) + + 1. [MEDIUM] Function `process_data` has no error handling for + empty input β raises unhandled IndexError at line 47 + 2. [INFO] Docstring for `validate` mentions param `strict` + which doesn't exist in the function signature + 3. [INFO] TODO marker at line 12 β "TODO: add retry logic" + + Cost: $0.14 (8,200 prompt + 1,840 completion tokens) +``` + +## Depth Levels + +| Depth | Critics | Cost* | Use when | +|-------|---------|-------|----------| +| `quick` | 2 (correctness, completeness) | ~$0.15 | Sanity check before continuing | +| `standard` | 6 (+ security, code hygiene, cross-consistency, tester) | ~$0.50 | Default for most work | +| `thorough` | 6 + fix loops + full audit trail | ~$1.50+ | Production-bound, can't be wrong | + +*Estimates on Claude Sonnet. Scales with model and artifact size.* + +## Next Steps + +- **Validate a directory:** `quorum run --target ./src/ --depth standard` +- **Use a custom rubric:** `quorum run --target report.md --rubric my-rubric.json` +- **Check cross-artifact consistency:** `quorum run --target ./docs/ --relationships quorum-relationships.yaml` +- **Build your own rubric:** [Rubric Building Guide](../guides/RUBRIC_BUILDING_GUIDE.md) +- **Understand the critics:** [The Nine](../architecture/THE_NINE.md) + +## Troubleshooting + +First run will prompt you to configure your model. If you hit issues: + +- [Full Installation Guide](INSTALLATION.md) +- [Model Requirements](MODEL_REQUIREMENTS.md) +- [Config Reference](../configuration/CONFIG_REFERENCE.md) diff --git a/docs/TUTORIAL.md b/docs/getting-started/TUTORIAL.md similarity index 100% rename from docs/TUTORIAL.md rename to docs/getting-started/TUTORIAL.md diff --git a/docs/CROSS_ARTIFACT_DESIGN.md b/docs/guides/CROSS_ARTIFACT_DESIGN.md similarity index 100% rename from docs/CROSS_ARTIFACT_DESIGN.md rename to docs/guides/CROSS_ARTIFACT_DESIGN.md diff --git a/docs/RUBRIC_BUILDING_GUIDE.md b/docs/guides/RUBRIC_BUILDING_GUIDE.md similarity index 98% rename from docs/RUBRIC_BUILDING_GUIDE.md rename to docs/guides/RUBRIC_BUILDING_GUIDE.md index 9a3b14f..4df1e62 100644 --- a/docs/RUBRIC_BUILDING_GUIDE.md +++ b/docs/guides/RUBRIC_BUILDING_GUIDE.md @@ -23,7 +23,7 @@ Before you start, you need: 1. **The source document in parseable format.** PDFs with embedded text work fine; scanned PDFs need OCR. See Step 1. 2. **Domain knowledge.** This is non-negotiable. Standards are dense, and normative decomposition without domain knowledge produces rubrics that are technically correct and practically useless. The criteria generation step (Step 3) is where expertise matters most. 3. **A clear scope boundary.** Full standards are large. Decide upfront which sections or requirement types you're encoding. Scope creep kills rubric projects. -4. **The Quorum rubric schema.** See [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md) for the full spec. A rubric is a JSON file; critics consume it at evaluation time. +4. **The Quorum rubric schema.** See [CONFIG_REFERENCE.md](../configuration/CONFIG_REFERENCE.md) for the full spec. A rubric is a JSON file; critics consume it at evaluation time. --- @@ -353,4 +353,4 @@ These assume focused work, good source quality, and pre-existing domain expertis --- -*Questions or rubric contributions β [CONTRIBUTING.md](../CONTRIBUTING.md)* +*Questions or rubric contributions β [CONTRIBUTING.md](../../CONTRIBUTING.md)* diff --git a/docs/reviews/2026-03-09-gpt54-review.md b/docs/reviews/2026-03-09-gpt54-review.md new file mode 100644 index 0000000..dcce16d --- /dev/null +++ b/docs/reviews/2026-03-09-gpt54-review.md @@ -0,0 +1,78 @@ +# GPT-5.4 External Review β 2026-03-09 + +**Model:** GPT-5.4 +**Version reviewed:** v0.5.3 (README + SPEC) +**Date:** 2026-03-09 ~04:00 PDT +**Solicited by:** Daniel Cervera + +--- + +## Summary + +> Quorum's architecture is already good enough to justify its niche, but it now needs to become more self-consistent and empirically calibrated than it is conceptually ambitious. + +## Overall Assessment + +- Quorum is "one of the more thoughtfully designed open-source attempts at agent-output validation" GPT-5.4 has seen +- Positioned as a **CI-style acceptance framework for agent outputs** β distinct from runtime guardrails (LangChain), eval frameworks (DeepEval), and optimization systems (DSPy) +- Layered architecture (deterministic pre-screen β LLM critics) called out as "genuinely good systems decision" +- Evidence discipline (concrete citations vs score+rationale) noted as stronger than common eval stacks + +## Strengths Identified + +1. **Layered screening architecture** β deterministic before LLM, cost-efficient, disciplined +2. **Evidence grounding** β findings must cite excerpts, grep results, schema checks, or rubric mismatches +3. **Operational maturity** β progressive manifests, resume, cost caps, audit CSVs (noted as "unusual for open-source") +4. **Transparency axioms** β file-based artifacts, immutable run dirs, source hashing, no hidden state +5. **Cross-artifact consistency** β relationship checking between files (few open-source projects do this) + +## Weaknesses Identified + +### 1. Spec Drift (HIGH) +README and SPEC don't fully agree on implementation state. Features shipped in v0.5.3 still described as "deferred" or "proposal-mode" in SPEC. **"For a tool whose whole value proposition is traceability and substantiation, internal inconsistency is more damaging than it would be in a normal repo."** + +**Our response:** Valid. README/CHANGELOG updated aggressively during sprint; SPEC lagged. SPEC reconciliation pass is next priority. + +### 2. Role Complexity vs Actual Need (MEDIUM) +9-agent target design may be over-separated for current maturity. Orchestration overhead, extra prompts, and failure surfaces before calibration evidence proves each role's ROI. + +**Our response:** Fair but premature β only 4 critics + Fixer live. Calibration infrastructure (golden set, calibration runner, inter-critic agreement) built before scaling panel. Sequencing is deliberate. + +### 3. Trust Model Lacks Empirical Grounding (MEDIUM) +Trust states (NEW β PROBATIONARY β ESTABLISHED β TRUSTED) don't explain how accuracy is measured objectively. "Without a gold-set discipline or adjudication mechanism, runtime trust scoring risks becoming pseudo-quantitative theater." + +**Our response:** Golden set (30 entries), calibration runner, and inter-critic agreement tools exist in private Validator tooling (DEC-017). Need graduation to Quorum proper (sprint backlog item #21). + +## Design Risks + +### Learning Memory Overfit +Frequency-based promotion (β₯3 occurrences β mandatory check) can overfit to local habits, bad rubrics, or transient model quirks. Needs: semantic deduplication, decay tuning, human review gates. + +**Our response:** Agreed. `quorum issues review` command for human approval before promotion is a straightforward addition. + +### File-Only Communication β Partial Safety Claim +"Prompt injection risk primarily rides in the content artifacts and tool outputs themselves, not just in memory." File-based passing helps reproducibility and isolation more than it solves prompt injection. + +**Our response:** Fair nuance. The claim should be scoped to isolation and reproducibility, not prompt injection prevention broadly. + +### Product Boundary Risk +"Quorum wants to be both a validation engine and a substantiation framework... resist growing into a full agent operating system." Absorbing orchestration/optimization/execution would run into DSPy, OpenHands, LangGraph momentum. + +**Our response:** Agreed. Design principle: **Quorum is the gate, not the road.** + +## Recommended Next Steps (per GPT-5.4) + +1. Reconcile README, SPEC, and implementation status +2. Build benchmark corpora and adjudicated golden sets before investing in trust scoring +3. Prioritize Tester and deterministic verifiers over more LLM critics +4. Turn cross-artifact consistency into a major differentiator +5. Treat learning memory as advisory-first, mandatory-second with reviewable promotion workflows + +## Competitive Positioning (per GPT-5.4) + +| vs. | Quorum stronger | Quorum weaker | +|-----|----------------|---------------| +| LangChain guardrails | Post-hoc artifact validation, evidence grounding, batch verdicting, auditability | Less embedded in execution graph, less general as runtime control plane | +| DeepEval | Richer artifact-level review, multi-phase critics, fix loops, cross-file consistency | Weaker on benchmark culture, calibration, large-scale quantified evaluation | +| DSPy | Trust boundary and acceptance gating | Not an optimization or program compilation system | +| OpenHands | Standards-driven substantiation pipeline | Less comprehensive agent execution/evaluation harness | diff --git a/docs/reviews/2026-03-09-grok-heavy-review.md b/docs/reviews/2026-03-09-grok-heavy-review.md new file mode 100644 index 0000000..f7d8dfc --- /dev/null +++ b/docs/reviews/2026-03-09-grok-heavy-review.md @@ -0,0 +1,74 @@ +# Grok 4.20 Heavy External Review β 2026-03-09 + +**Model:** Grok 4.20 Heavy +**Versions reviewed:** v0.5.1 (initial, from GitHub) β v0.5.3 (updated drafts supplied by Daniel) +**Date:** 2026-03-09 ~04:18 PDT +**Solicited by:** Daniel Cervera + +--- + +## Summary + +> Once the drafts are merged and the next 2β3 critics ship, Quorum has strong potential to become the de-facto "judge infrastructure" and substantiation layer for serious agentic engineering β a natural, high-value complement to the LangGraph/CrewAI ecosystem. + +## Key Progression (v0.5.1 β v0.5.3) + +Grok initially reviewed the stale v0.5.1 on GitHub, then was given updated v0.5.3 drafts. Its assessment shifted from "remarkably well-designed" to "jumped ahead of almost every lightweight LLM-judge library and research prototype." + +### v0.5.1 Assessment +- 524 tests, 90% coverage β "exceptional for a <1-month-old project" +- SPEC v3.0 "one of the cleanest, most principled specs in the agentic space" +- Fixer in "proposal mode only," learning memory flagged as inconsistent (README vs SPEC) + +### v0.5.3 Assessment +- Learning memory + re-validation loops marked as correctly shipped +- "Significant leap" β closed-loop self-improvement, production hardening, crash resilience +- Documentation now "far more consistent (only one tiny leftover mismatch)" + +## Competitive Positioning (unchanged across both reviews) + +> None combine enforced evidence grounding + multi-tool pre-screen + cross-artifact contracts + versionable rubrics + immutable traces + active learning memory + closed-loop fixer re-validation at this level of production polish. + +## Remaining Weaknesses (v0.5.3) + +### 1. One-line Doc Inconsistency (FIXED) +README comparison table said learning memory was "planned, not yet active." Fixed in commit `9c863b4`. + +### 2. Critic Suite Incomplete (4/9) +Architecture, Delegation, Style, Tester not yet built. "Single largest functional gap." + +**Our response:** Deliberate sequencing. Calibration infrastructure built first. Architecture + Tester are next priorities. + +### 3. Trust & Monitoring System Not Wired +SPEC describes probationary β trusted progression but not implemented. + +**Our response:** Golden set + calibration runner exist in private tooling. Need graduation to Quorum (backlog #21). + +### 4. Scalability (ThreadPoolExecutor only) +Max 4 critics, batch max 3 workers. No async/distributed. + +**Our response:** Fine for current scale. Will address when bottleneck is real, not theoretical. + +### 5. Integration Gaps +No LangGraph node, CrewAI hook, or LangSmith callback. Pure CLI. + +**Our response:** GitHub Actions CI gate shipped tonight. LangGraph node + Python SDK are Phase B priorities. + +### 6. No Published Benchmarks +Excellent artifacts but no Quorum-specific benchmarks or adversarial testing. + +**Our response:** QuorumBench is on the roadmap. Golden set provides internal calibration data. + +### 7. Adoption Polish +PyPI claimed but Grok couldn't verify (it is live: `pip install quorum-validator`). OpenClaw/ClawHub branding may slow broader OSS traction. + +**Our response:** Valid concern. Quorum should stand on its own identity for broader adoption. + +## Recommended Next Steps (per Grok) +1. Push v0.5.3 drafts live β (done) +2. Ship Architecture + Tester critics +3. Activate Trust & Monitoring system +4. LangGraph tool node + GitHub Action quality gate (β GitHub Action shipped) +5. Compliance rubric packs (OWASP ASVS, SOC 2, NIST) +6. Public benchmarks + community rubric repo +7. Async/distributed backend as scale grows diff --git a/docs/reviews/2026-03-09-opus-extended-review.md b/docs/reviews/2026-03-09-opus-extended-review.md new file mode 100644 index 0000000..162872a --- /dev/null +++ b/docs/reviews/2026-03-09-opus-extended-review.md @@ -0,0 +1,83 @@ +# Independent Opus 4.6 (Extended) Review β 2026-03-09 + +**Model:** Claude Opus 4.6 (Extended thinking) +**Version reviewed:** v0.5.3 (README + SPEC, pre-push drafts) +**Date:** 2026-03-09 ~04:37 PDT +**Solicited by:** Daniel Cervera +**Note:** This was an independent Opus instance with no context from our development session. + +--- + +## Summary + +> Closing that gap β Tester agent + calibration data β would make Quorum the only tool in its class that can credibly claim its validation results are substantiated rather than opinionated. + +## Category Assessment + +Opus identified Quorum as occupying "Category 0" β distinct from: +- **Category 1:** LLM Eval Frameworks (DeepEval, RAGAS, Promptfoo) β evaluate models/prompts against test cases +- **Category 2:** Agent Observability (LangSmith, Phoenix, Langfuse) β trace and monitor agent behavior +- **Category 3:** Agent Orchestration (CrewAI, LangGraph, AutoGen) β build agents + +Quorum's claim: **Post-hoc artifact validation with evidence grounding.** Closer to code review/audit than eval. + +## Critical Weaknesses + +### 1. Evidence Grounding Is Partially Circular (CRITICAL β philosophical) +> "The core differentiator β 'every finding must cite evidence' β is enforced by the Aggregator, which is itself an LLM. A critic can produce a grep result that *looks* like evidence but is cherry-picked or misinterpreted." + +Evidence grounding is currently "rhetorically deterministic" not "actually deterministic." The Tester agent is the component that would close this gap by executing verification (run grep, check file hashes, confirm line numbers match claims). + +**Our response:** This is the sharpest critique from any reviewer. Reorders the roadmap: Tester before Architecture. Tester isn't just another critic β it's the integrity proof for the entire evidence grounding claim. Without it, Quorum's core thesis is aspirational. + +### 2. No Empirical Calibration Data (HIGH) +DeepEval, RAGAS, and Promptfoo publish correlation data against human judgments. Quorum has none. "For a tool whose entire pitch is 'now you know,' this is a significant credibility gap." + +**Our response:** Golden set (30 entries) and calibration runner exist in private tooling (DEC-017). Need to graduate to Quorum and publish results. Even 50 manually-reviewed artifacts with published detection rates would "do more for adoption than any new feature." + +### 3. Content-Level Critics Only β Structural Blind Spot (HIGH) +All 4 shipped critics (Correctness, Completeness, Security, Code Hygiene) examine *what's in the artifact*. Missing critics (Architecture, Delegation, Tester, Style) examine *how it was designed, delegated, and tested* β arguably higher-stakes failure modes. + +**Our response:** Valid reframing. Structural/process-level critique is the gap that matters for production systems. + +### 4. Rubric Library Too Thin (MEDIUM) +3 shipped rubrics vs Promptfoo's OWASP/NIST presets or DeepEval's 30+ metrics. "Most users won't author rubrics. Rubric library is where network effects compound." + +**Our response:** Community rubric contributions should be treated as primary growth lever. Each rubric = new audience. + +### 5. Platform Coupling Risk (MEDIUM) +OpenClaw/ClawHub branding creates friction for non-OpenClaw users. The concept is framework-agnostic; packaging shouldn't imply otherwise. + +**Our response:** PyPI is primary distribution now. Documentation should lead with `pip install` not ClawHub. + +### 6. No CI/CD Integration (STALE β SHIPPED) +"No documented pattern for 'run Quorum as a PR gate.'" + +**Our response:** GitHub Actions workflow shipped during this session (commit `bfe2a9d`). Standard depth on changed files, $5 cap, pytest included. + +## Design Strengths Worth Preserving + +1. **Pre-screen layer** β deterministic checks before LLM tokens. "Cost-optimization pattern most competitors don't implement." +2. **Cross-artifact consistency** β typed relationships with dual-locus findings. "Something I haven't seen in any competitor. Quorum's strongest differentiator." +3. **Transparency axiom** β "Phase 2 receives findings, not verdicts." File-based artifact passing = auditability by default. +4. **Cost awareness** β per-run estimates, --max-cost, depth tiers. "Most LLM evaluation tools are cavalier about API costs." + +## Strategic Recommendations (priority order) + +1. **Build the Tester agent** β transforms evidence grounding from aspirational to deterministic +2. **Publish calibration data** β golden set detection rates = fastest path to credibility +3. **Invest in rubric breadth** β SOC 2, OWASP ASVS, Terraform/IaC, API contracts, tech writing +4. **Build CI/CD integration** β β DONE (GitHub Actions PR gate) +5. **Reduce OpenClaw coupling** β framework-agnostic value, framework-agnostic packaging + +## Bottom Line + +> Quorum is architecturally sophisticated for a v0.5 project. The theoretical grounding (Reflexion, Juries paper, Tomasev delegation) is genuine rather than decorative. But the project's primary risk is the gap between its philosophical claims (evidence-grounded, not vibes) and its current implementation reality (evidence grounding is enforced by an uncalibrated LLM, the Tester agent that would make it deterministic isn't built, and there's no published data on verdict accuracy). + +## Impact on Roadmap + +This review reorders priorities: +1. **Tester critic** β promoted from Phase C item to #1 priority (integrity proof for evidence grounding) +2. **Calibration data publication** β promoted from "nice to have" to credibility-critical +3. **Rubric breadth** β reframed as primary growth lever, not just content +4. **Architecture critic** β remains important but behind Tester diff --git a/docs/EXTERNAL_REVIEWS.md b/docs/reviews/EXTERNAL_REVIEWS.md similarity index 100% rename from docs/EXTERNAL_REVIEWS.md rename to docs/reviews/EXTERNAL_REVIEWS.md diff --git a/docs/reviews/ideas-backlog.md b/docs/reviews/ideas-backlog.md new file mode 100644 index 0000000..3685a10 --- /dev/null +++ b/docs/reviews/ideas-backlog.md @@ -0,0 +1 @@ +renders_from relationship type β structured source validates prose faithfully represents claims; deterministic check not LLM judgment diff --git a/reference-implementation/pyproject.toml b/reference-implementation/pyproject.toml index 61859a7..93011c0 100644 --- a/reference-implementation/pyproject.toml +++ b/reference-implementation/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "quorum-validator" -version = "0.7.0" +version = "0.7.2" description = "Multi-critic quality validation for agents, research, and configurations" authors = [ {name = "Daniel Cervera", email = "daniel@sharedintellect.com"}, diff --git a/tools/validate-docs.py b/tools/validate-docs.py index 69dd91f..5ee943d 100755 --- a/tools/validate-docs.py +++ b/tools/validate-docs.py @@ -78,7 +78,7 @@ def find_md_files(repo_root: Path) -> list[Path]: """Find all .md files in the repo, excluding dirs that shouldn't be validated.""" exclude_dirs = { "venv", "node_modules", ".git", "quorum-runs", "__pycache__", "dist", - ".hypothesis", "external-reviews", # external reviews are point-in-time snapshots + ".hypothesis", "external-reviews", "reviews", # external reviews are point-in-time snapshots } exclude_files = { "SHIPPING.md", # contains historical context about the problem