Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 21 additions & 10 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,28 @@ A single model reviewing a long prompt generates:
- Hand-waving without evidence (LLMs can justify anything)
- ~~No learning across validations~~ → **Solved in v0.5.3:** Learning memory tracks recurring patterns and promotes high-frequency findings to mandatory checks

Quorum uses **nine specialized critics in parallel**, each with deep expertise in:
Quorum's target architecture has **nine specialized agents** — six critics are currently shipped, with three more planned:

**Shipped critics (6):**

1. **Correctness Critic** — Factual accuracy, logical consistency, claim support
2. **Security Critic** — Vulnerability patterns, permission issues, injection risks
3. **Completeness Critic** — Coverage gaps, missing requirements, unaddressed edge cases
4. **Architecture Critic** — Design coherence, pattern consistency, scalability concerns
5. **Delegation & Coordination Critic** — Span of control, reversibility, bidirectional contracts
6. **Tester Agent** — Executes validation (schema checks, git queries, web searches, shell execution)
7. **Fixer Agent** — Generates fixes for CRITICAL/HIGH issues (optional, 1-2 loops max)
8. **Aggregator Critic** — Merges findings, resolves conflicts, recalibrates confidence
9. **Supervisor Agent** — Manages workflow, checkpoints, final verdict
2. **Completeness Critic** — Coverage gaps, missing requirements, unaddressed edge cases
3. **Security Critic** — Vulnerability patterns, permission issues, injection risks
4. **Code Hygiene Critic** — ISO 25010 maintainability, naming, complexity metrics
5. **Cross-Artifact Consistency Critic**† — Inter-file consistency (activated with `--relationships`)
6. **Tester Agent** — Finding verification: deterministic checks + LLM claim validation

**Shipped infrastructure agents:**

7. **Fixer Agent** — Proposes and applies fixes for CRITICAL/HIGH findings (optional, 1-2 loops max)
8. **Aggregator** — Merges findings, resolves conflicts, recalibrates confidence
9. **Supervisor** — Manages workflow, checkpoints, final verdict

**Planned critics (not yet built):**

10. **Architecture Critic** — Design coherence, pattern consistency, scalability concerns
11. **Delegation & Coordination Critic** — Span of control, reversibility, bidirectional contracts
12. **Style Critic** — Writing quality, tone consistency, formatting standards

Critics don't vote. The Aggregator synthesizes their findings into a final verdict with explicit confidence levels.

Expand Down Expand Up @@ -398,7 +409,7 @@ Additional runs on related artifacts reuse critic prompts and tools, amortizing

## 8. Implementation Checklist

Status as of v0.7.2 (reference implementation):
Status as of v0.7.3 (reference implementation):

- [x] LLM provider — LiteLLM universal provider (100+ models, any tier combination)
- [x] File-based artifact passing (no in-memory state between agents)
Expand Down
2 changes: 1 addition & 1 deletion critic-status.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Used by tools/validate-docs.py to detect stale documentation.
# Update this file FIRST when shipping a critic or feature.

version: "0.7.2"
version: "0.7.3"
spec_version: "3.0"

critics:
Expand Down
14 changes: 14 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,20 @@ All notable changes to Quorum will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [0.7.3] — 2026-03-15

### Fixed — Pre-Existing Code Findings Cleanup

- **`__version__` corrected** — `reference-implementation/quorum/__init__.py` now reads `0.7.3` (was stuck at `0.5.3` since initial release; PyPI showed `0.7.2` but runtime `__version__` did not match)
- **Broad exception handling narrowed** (prescreen.py) — PS-006 `except Exception` narrowed to `(OSError, UnicodeDecodeError)`; external tool integrations (Ruff, DevSkim, Bandit, PSSA) now log with `exc_info=True` for traceback capture on unexpected failures
- **PSScriptAnalyzer security documentation** (prescreen.py) — Added comprehensive `# SECURITY:` comment explaining the PowerShell single-quote injection trust model for `_run_pssa`
- **SPEC.md §2.1 overclaiming fixed** — Numbered agent list now separates shipped critics (6) from planned critics (3) and infrastructure agents; Code Hygiene and Cross-Artifact Consistency added (were missing from list despite being shipped)
- **validate-docs.py hardcoded range removed** — `check_hardcoded_counts` `range(2, 9)` replaced with parameterized bounds derived from manifest total critic count
- **validate-docs.py SRP cleanup** — Extracted `_extract_pyproject_version()` and `_extract_badge_versions()` as pure functions; `main()` refactored to delegate to `validate_docs()` (eliminated duplicated validation loop)
- **Test exception handling** (test_batch_resilience.py) — Broad `except Exception` narrowed to `(json.JSONDecodeError, OSError)` with explanatory comment

---

## [0.7.2] — 2026-03-12

### Documentation — Stale Content Cleanup (PR #18)
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This guide walks through building Quorum from the architectural spec. It's structured as a reference walkthrough, not a copy-paste tutorial. The goal is understanding the patterns so you can adapt them to your stack.

> **Implementation Status (v0.7.2):** This guide describes the full target architecture. All 6 critics are shipped and callable: Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, and Tester. Also shipped: the fixer (proposal mode), parallel execution, batch processing, and pre-screen integration. See `critic-status.yaml` for the authoritative status matrix. Sections below marked 🔜 describe components not yet built (Architecture, Delegation, Style).
> **Implementation Status (v0.7.3):** This guide describes the full target architecture. All 6 critics are shipped and callable: Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, and Tester. Also shipped: the fixer (proposal mode), parallel execution, batch processing, and pre-screen integration. See `critic-status.yaml` for the authoritative status matrix. Sections below marked 🔜 describe components not yet built (Architecture, Delegation, Style).

---

Expand Down
2 changes: 1 addition & 1 deletion docs/critics/CODE_HYGIENE_FRAMEWORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The boundary is clean: when code hygiene detects a pattern that has security imp

## Status

**v0.7.2 State:** Framework design complete. All features shipped.
**v0.7.3 State:** Framework design complete. All features shipped.

- [x] Framework design and documentation
- [x] 12 evaluation categories specified
Expand Down
2 changes: 1 addition & 1 deletion docs/critics/SECURITY_CRITIC_FRAMEWORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

## Status

**v0.7.2 State:** Framework design complete. All features shipped.
**v0.7.3 State:** Framework design complete. All features shipped.

- [x] Framework design and documentation
- [x] 14 evaluation categories (SEC-01–SEC-14) specified
Expand Down
2 changes: 1 addition & 1 deletion reference-implementation/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "quorum-validator"
version = "0.7.2"
version = "0.7.3"
description = "Multi-critic quality validation for agents, research, and configurations"
authors = [
{name = "Daniel Cervera", email = "daniel@sharedintellect.com"},
Expand Down
2 changes: 1 addition & 1 deletion reference-implementation/quorum/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@
specialized critics, each required to provide grounded evidence.
"""

__version__ = "0.5.3"
__version__ = "0.7.3"
__all__ = ["__version__"]
19 changes: 10 additions & 9 deletions reference-implementation/quorum/prescreen.py
Original file line number Diff line number Diff line change
Expand Up @@ -240,8 +240,7 @@ def run(self, artifact_path: Path, artifact_text: str) -> PreScreenResult:
ruff_checks = self._findings_to_checks(ruff_findings, "ruff")
checks.extend(ruff_checks)
except Exception as e:
logger.warning("Ruff integration failed: %s", e)
# Graceful degradation: treat as "no issues found" when tool fails
logger.warning("Ruff integration failed: %s", e, exc_info=True)
checks.append(_pass("EXT-RUFF", "ruff_analysis", "external_tools",
Severity.INFO, "No issues found by ruff (tool unavailable)"))

Expand All @@ -251,8 +250,7 @@ def run(self, artifact_path: Path, artifact_text: str) -> PreScreenResult:
devskim_checks = self._findings_to_checks(devskim_findings, "devskim")
checks.extend(devskim_checks)
except Exception as e:
logger.warning("DevSkim integration failed: %s", e)
# Graceful degradation: treat as "no issues found" when tool fails
logger.warning("DevSkim integration failed: %s", e, exc_info=True)
checks.append(_pass("EXT-DEVSKIM", "devskim_analysis", "external_tools",
Severity.INFO, "No issues found by devskim (tool unavailable)"))

Expand All @@ -264,7 +262,7 @@ def run(self, artifact_path: Path, artifact_text: str) -> PreScreenResult:
bandit_checks = self._findings_to_checks(bandit_findings, "bandit")
checks.extend(bandit_checks)
except Exception as e:
logger.warning("Bandit integration failed: %s", e)
logger.warning("Bandit integration failed: %s", e, exc_info=True)
checks.append(_pass("EXT-BANDIT", "bandit_analysis", "external_tools",
Severity.INFO, "No issues found by bandit (tool unavailable)"))

Expand All @@ -275,7 +273,7 @@ def run(self, artifact_path: Path, artifact_text: str) -> PreScreenResult:
pssa_checks = self._findings_to_checks(pssa_findings, "pssa")
checks.extend(pssa_checks)
except Exception as e:
logger.warning("PSScriptAnalyzer integration failed: %s", e)
logger.warning("PSScriptAnalyzer integration failed: %s", e, exc_info=True)
checks.append(_pass("EXT-PSSA", "pssa_analysis", "external_tools",
Severity.INFO, "No issues found by pssa (tool unavailable)"))

Expand Down Expand Up @@ -447,9 +445,9 @@ def _ps006_python_syntax(
f"Python syntax error at {loc}",
msg, [loc] if loc_match else [])

except Exception as exc:
except (OSError, UnicodeDecodeError) as exc:
return _skip("PS-006", "python_syntax", "syntax", Severity.HIGH,
"Python syntax check", f"Could not compile: {exc}")
"Python syntax check", f"Could not compile ({type(exc).__name__}): {exc}")

finally:
if tmp_path:
Expand Down Expand Up @@ -855,7 +853,10 @@ def _run_pssa(self, artifact_path: Path, artifact_text: str) -> list[Finding]:
}

try:
# Escape single quotes in path to prevent PowerShell command injection
# SECURITY: PowerShell command injection mitigated. Single-quoted strings
# in PowerShell don't expand variables/backticks/subexpressions; only
# internal single quotes need doubling (' → ''), which we do below.
# artifact_path originates from filesystem resolution, not user input.
escaped_path = str(artifact_path).replace("'", "''")
ps_command = (
f"Invoke-ScriptAnalyzer -Path '{escaped_path}' "
Expand Down
4 changes: 2 additions & 2 deletions reference-implementation/tests/test_batch_resilience.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,8 @@ def spy_validate(*args, **kwargs):
for p in (tmp_path / "runs").glob("batch-*/batch-manifest.json"):
try:
manifests_seen.append(json.loads(p.read_text()))
except Exception:
pass
except (json.JSONDecodeError, OSError):
pass # manifest may be partially written during race
return original_validate(*args, **kwargs)

# Create 3 files
Expand Down
2 changes: 1 addition & 1 deletion reference-implementation/tests/test_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def test_help(self, runner):
def test_version(self, runner):
result = runner.invoke(cli, ["--version"])
assert result.exit_code == 0
assert "0.5.3" in result.output
assert "0.7.3" in result.output

def test_no_command_shows_usage(self, runner):
result = runner.invoke(cli, [])
Expand Down
59 changes: 26 additions & 33 deletions tools/validate-docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,9 @@ def read_file_lines(file_path: Path) -> list[str] | None:
return None


def check_hardcoded_counts(lines: list[str], shipped_count: int, file_path: Path) -> list[str]:
def check_hardcoded_counts(
lines: list[str], shipped_count: int, total_count: int, file_path: Path,
) -> list[str]:
"""Flag lines with hardcoded critic counts that don't match shipped count."""
findings: list[str] = []
# Match patterns like "4 critics", "ships with 4", "currently 5 critics"
Expand Down Expand Up @@ -165,7 +167,7 @@ def check_hardcoded_counts(lines: list[str], shipped_count: int, file_path: Path
n = int(match.group(1))
# Only flag if it looks like a shipped count that's wrong
# Skip 9 (total architecture), 3 (planned), and the correct count
if n != shipped_count and n in range(2, 9) and n != 9:
if n != shipped_count and n != total_count and 2 <= n <= total_count:
# Skip lines about specific depth profiles with intentionally fewer critics
# (e.g., "quick" depth legitimately runs 2 critics)
if "quick" in line_lower and n == 2:
Expand Down Expand Up @@ -263,33 +265,40 @@ def check_roadmap_shipped(
return findings


def _extract_pyproject_version(content: str) -> str | None:
"""Extract the version string from pyproject.toml content (pure)."""
match = re.search(r'^version\s*=\s*"([^"]+)"', content, re.MULTILINE)
return match.group(1) if match else None


def _extract_badge_versions(content: str) -> list[str]:
"""Extract version strings from badge markup in README content (pure)."""
badge_pattern = re.compile(r'badge[/](?:version-)?v?(\d+\.\d+\.\d+)')
return [m.group(1) for m in badge_pattern.finditer(content)]


def check_version_consistency(repo_root: Path, manifest_version: str) -> list[str]:
"""Check that version strings across the repo match the manifest version."""
findings: list[str] = []

# Check pyproject.toml
pyproject = repo_root / "reference-implementation" / "pyproject.toml"
if pyproject.exists():
try:
content = pyproject.read_text(encoding="utf-8")
version_match = re.search(r'^version\s*=\s*"([^"]+)"', content, re.MULTILINE)
if version_match and version_match.group(1) != manifest_version:
pyproject_ver = _extract_pyproject_version(content)
if pyproject_ver and pyproject_ver != manifest_version:
findings.append(
f" pyproject.toml: version=\"{version_match.group(1)}\" "
f" pyproject.toml: version=\"{pyproject_ver}\" "
f"(manifest={manifest_version})"
)
except OSError:
pass

# Check README for badge version mismatches
readme = repo_root / "README.md"
if readme.exists():
try:
content = readme.read_text(encoding="utf-8")
# Match badge patterns like: badge/version-v0.5.3 or badge/v0.5.3
badge_pattern = re.compile(r'badge[/](?:version-)?v?(\d+\.\d+\.\d+)')
for match in badge_pattern.finditer(content):
badge_ver = match.group(1)
for badge_ver in _extract_badge_versions(content):
if badge_ver != manifest_version:
findings.append(
f" README.md: badge version \"{badge_ver}\" "
Expand Down Expand Up @@ -396,6 +405,7 @@ def validate_docs(repo_root: Path) -> list[str]:
manifest = load_manifest(repo_root)
shipped = get_critics_by_status(manifest, "shipped")
shipped_count = len(shipped)
total_count = len(manifest.get("critics", {}))
manifest_version = manifest.get("version", "unknown")

md_files = find_md_files(repo_root)
Expand All @@ -414,7 +424,7 @@ def validate_docs(repo_root: Path) -> list[str]:
continue

rel_path = md_file.relative_to(repo_root)
all_findings.extend(check_hardcoded_counts(lines, shipped_count, rel_path))
all_findings.extend(check_hardcoded_counts(lines, shipped_count, total_count, rel_path))
all_findings.extend(check_stale_status_markers(lines, shipped, rel_path))
all_findings.extend(check_roadmap_shipped(lines, shipped, rel_path))
all_findings.extend(check_depth_config_claims(lines, manifest, rel_path))
Expand All @@ -429,10 +439,10 @@ def main() -> int:
script_dir = Path(__file__).resolve().parent
repo_root = script_dir.parent

# Print manifest summary for human readers
manifest = load_manifest(repo_root)
shipped = get_critics_by_status(manifest, "shipped")
planned = get_critics_by_status(manifest, "planned")
shipped_count = len(shipped)
manifest_version = manifest.get("version", "unknown")

callable_critics = {
Expand All @@ -442,7 +452,7 @@ def main() -> int:
depth_configs = manifest.get("depth_configs", {})

print(f"Manifest version: {manifest_version}")
print(f"Shipped critics ({shipped_count}): {', '.join(shipped.keys())}")
print(f"Shipped critics ({len(shipped)}): {', '.join(shipped.keys())}")
print(f"Callable critics ({len(callable_critics)}): {', '.join(sorted(callable_critics))}")
print(f"Planned critics ({len(planned)}): {', '.join(planned.keys())}")
if depth_configs:
Expand All @@ -453,25 +463,8 @@ def main() -> int:
md_files = find_md_files(repo_root)
print(f"Scanning {len(md_files)} markdown files...\n")

all_findings: list[str] = []

# Version consistency check
version_findings = check_version_consistency(repo_root, manifest_version)
if version_findings:
all_findings.append("VERSION MISMATCH:")
all_findings.extend(version_findings)

for md_file in md_files:
lines = read_file_lines(md_file)
if lines is None:
continue

rel_path = md_file.relative_to(repo_root)
all_findings.extend(check_hardcoded_counts(lines, shipped_count, rel_path))
all_findings.extend(check_stale_status_markers(lines, shipped, rel_path))
all_findings.extend(check_roadmap_shipped(lines, shipped, rel_path))
all_findings.extend(check_depth_config_claims(lines, manifest, rel_path))
all_findings.extend(check_framework_version_strings(lines, manifest_version, rel_path))
# Delegate validation logic (no duplication with validate_docs())
all_findings = validate_docs(repo_root)

if all_findings:
print(f"FINDINGS ({len(all_findings)}):\n")
Expand Down
Loading