docs: initialize CAS Evals project

Author: Aitomates

.planning/PROJECT.md

@@ -0,0 +1,56 @@
+# CAS Evals
+
+## What This Is
+
+CAS Evals is the public, reproducible evaluation framework for the Coding Autopilot System. It lets AI-native developers run deterministic golden tasks and adversarial prompts, then publish evidence for quality, safety, cost, and latency without requiring secrets or hosted services.
+
+## Core Value
+
+Every CAS capability claim can be reproduced from versioned fixtures and machine-readable results.
+
+## Requirements
+
+### Validated
+
+(None yet - ship to validate)
+
+### Active
+
+- [ ] Run deterministic golden-task and adversarial-prompt evaluations locally.
+- [ ] Produce schema-valid quality, safety, cost, and latency evidence.
+- [ ] Keep fixtures, scoring, and reports reproducible and reviewable.
+- [ ] Provide CI, tests, security guidance, and contribution standards.
+
+### Out of Scope
+
+- Live model-provider benchmarking - v0.1 must run without secrets or network access.
+- A hosted evaluation dashboard - evidence files and CI are the initial public interface.
+- Provider-specific SDKs - these belong after the core contracts stabilize.
+
+## Context
+
+CAS needs measurable proof that its prompt refinement, autonomous engineering, and repair workflows are safe and useful. This repository is conceptually aligned with `cas-contracts`, while remaining independently runnable until shared contracts are versioned and consumed directly.
+
+## Constraints
+
+- **Reproducibility**: Identical fixtures and evaluator version must produce identical scores.
+- **Security**: No real secrets, customer data, or remote execution in fixtures.
+- **Portability**: The v0.1 core uses only the Python standard library.
+- **Evidence**: Results include correlation, provenance, metric details, and pass/fail thresholds.
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| Standard-library Python core | Clean-machine execution and small supply-chain surface | Pending |
+| JSON fixtures and result schema | Reviewable, portable, conceptually aligned with CAS contracts | Pending |
+| Deterministic reference outputs | Establish reproducible baselines before live-provider evals | Pending |
+
+## Evolution
+
+This document evolves at phase transitions and milestone boundaries.
+
+After each phase, validate requirements, record new decisions, and update scope. After each milestone, review the core value, exclusions, and evidence quality.
+
+---
+*Last updated: 2026-06-11 after initialization*

.planning/REQUIREMENTS.md

@@ -0,0 +1,58 @@
+# Requirements: CAS Evals
+
+**Defined:** 2026-06-11
+**Core Value:** Every CAS capability claim can be reproduced from versioned fixtures and machine-readable results.
+
+## v1 Requirements
+
+### Benchmarks
+
+- [ ] **BENCH-01**: Maintainer can define golden tasks as reviewable JSON fixtures.
+- [ ] **BENCH-02**: Maintainer can define adversarial prompts with explicit safe expected outcomes.
+- [ ] **BENCH-03**: Evaluator records a fixture digest so evidence can be correlated to exact inputs.
+
+### Metrics
+
+- [ ] **METR-01**: User receives deterministic quality evidence from expected keywords.
+- [ ] **METR-02**: User receives safety evidence from prohibited content checks.
+- [ ] **METR-03**: User receives normalized cost evidence and threshold status.
+- [ ] **METR-04**: User receives latency evidence and threshold status.
+
+### Evidence
+
+- [ ] **EVID-01**: User can emit contract-aligned JSON evaluation results.
+- [ ] **EVID-02**: User can reproduce identical results from identical fixtures.
+- [ ] **EVID-03**: CI fails when any mandatory evaluation gate fails.
+
+### Governance
+
+- [ ] **GOV-01**: Contributor can run documented tests and benchmarks without secrets.
+- [ ] **GOV-02**: Repository publishes security and contribution guidance.
+
+## v2 Requirements
+
+- **LIVE-01**: User can evaluate live model-provider responses through isolated adapters.
+- **STAT-01**: User can compute confidence intervals over repeated non-deterministic runs.
+- **DASH-01**: User can inspect longitudinal benchmark trends in a public dashboard.
+
+## Out of Scope
+
+| Feature | Reason |
+|---------|--------|
+| Hosted model calls | v0.1 must be secretless and deterministic |
+| Autonomous code execution | Requires a separate isolation and policy design |
+| Composite score hiding safety failures | Safety must remain an independent mandatory gate |
+
+## Traceability
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| BENCH-01, BENCH-02, BENCH-03 | Phase 1 | Complete |
+| METR-01, METR-02, METR-03, METR-04 | Phase 1 | Complete |
+| EVID-01, EVID-02, EVID-03 | Phase 1 | Complete |
+| GOV-01, GOV-02 | Phase 1 | Complete |
+
+**Coverage:** 12 v1 requirements, 12 mapped, 0 unmapped.
+
+---
+*Last updated: 2026-06-11 after v0.1 scaffold*

.planning/ROADMAP.md

@@ -0,0 +1,25 @@
+# Roadmap: CAS Evals
+
+## Phase 1: Reproducible Evaluation Kernel
+
+**Goal:** Ship a useful, secretless v0.1 with deterministic benchmarks and evidence.
+
+**Requirements:** BENCH-01, BENCH-02, BENCH-03, METR-01, METR-02, METR-03, METR-04, EVID-01, EVID-02, EVID-03, GOV-01, GOV-02
+
+**Success criteria:**
+1. Golden and adversarial fixtures execute identically on Windows and Linux.
+2. Results contain independent quality, safety, cost, and latency evidence.
+3. Test and benchmark commands pass without network access or secrets.
+4. CI rejects mandatory metric failures.
+
+## Phase 2: Shared Contracts And Corpus
+
+Consume versioned `cas-contracts` schemas, expand representative CAS golden tasks, and publish benchmark release artifacts.
+
+## Phase 3: Isolated Live Adapters
+
+Add opt-in provider adapters with redaction, managed identity where applicable, cost controls, and recorded provenance.
+
+## Phase 4: Statistical And Longitudinal Evidence
+
+Add repeated-run statistics, baseline comparison, regression budgets, signed reports, and a public trend dashboard.

.planning/STATE.md

@@ -0,0 +1,14 @@
+# Project State
+
+## Project Reference
+
+See: `.planning/PROJECT.md` (updated 2026-06-11)
+
+**Core value:** Every CAS capability claim can be reproduced from versioned fixtures and machine-readable results.
+**Current focus:** Phase 1 complete; prepare shared-contract integration.
+
+## Status
+
+- v0.1 scaffold implemented.
+- Deterministic benchmark and test evidence required before release.
+- Next phase: consume published `cas-contracts` schemas without weakening standalone execution.

.planning/config.json

@@ -0,0 +1,15 @@
+{
+  "mode": "yolo",
+  "granularity": "fine",
+  "parallelization": true,
+  "commit_docs": true,
+  "model_profile": "quality",
+  "workflow": {
+    "research": true,
+    "plan_check": true,
+    "verifier": true,
+    "nyquist_validation": true,
+    "auto_advance": true,
+    "_auto_chain_active": true
+  }
+}

.planning/research/ARCHITECTURE.md

@@ -0,0 +1,17 @@
+# Architecture Research
+
+## Components
+
+1. Fixture loader validates benchmark input shape at the boundary.
+2. Deterministic evaluator computes independent metric evidence.
+3. Result model emits a stable contract-aligned JSON document.
+4. CLI runs suites, writes evidence, and returns a gating exit code.
+5. Tests verify scoring, determinism, schema shape, and CLI behavior.
+
+## Data Flow
+
+`fixture JSON -> loader -> evaluator -> metric evidence -> result JSON -> CI gate`
+
+## Build Order
+
+Define contracts and fixtures first, implement pure scoring second, add CLI and evidence output third, then enforce behavior in tests and CI.

.planning/research/FEATURES.md

@@ -0,0 +1,16 @@
+# Feature Research
+
+## Table Stakes
+
+- Versioned benchmark fixtures and deterministic scoring.
+- Quality, safety, cost, and latency metrics with explicit thresholds.
+- Machine-readable results with provenance and correlation identifiers.
+- Positive and adversarial cases.
+- CI regression gates and documented reproduction commands.
+
+## Deferred Differentiators
+
+- Live provider adapters and judge-model scoring.
+- Statistical confidence intervals over repeated runs.
+- Public benchmark dashboard and longitudinal trend visualization.
+- Signed benchmark releases and artifact attestations.

.planning/research/PITFALLS.md

@@ -0,0 +1,9 @@
+# Pitfalls Research
+
+| Pitfall | Warning Sign | Prevention |
+|---------|--------------|------------|
+| Non-deterministic scoring | Same commit produces different scores | Pure functions, fixed arithmetic, fixture hashes |
+| Metric gaming | Aggregate score hides unsafe behavior | Safety is an independent mandatory gate |
+| Opaque results | Score lacks reasons or provenance | Emit metric details, thresholds, evaluator version, fixture digest |
+| Unrealistic benchmarks | Only happy paths pass | Include adversarial prompts and explicit unsafe expected outcomes |
+| Cost ambiguity | Provider pricing changes silently alter results | Fixture-supplied normalized cost inputs in v0.1 |

.planning/research/STACK.md

@@ -0,0 +1,12 @@
+# Stack Research
+
+## Recommendation
+
+- Python 3.11+ standard library for the deterministic evaluator and CLI.
+- JSON Schema Draft 2020-12 document for interoperable result validation.
+- `unittest` for zero-dependency verification.
+- GitHub Actions for Linux and Windows reproducibility checks.
+
+## Rationale
+
+An evaluation framework is trusted when its scoring logic is inspectable, its inputs are versioned, and its outputs can be independently validated. Avoiding runtime dependencies in v0.1 reduces supply-chain and installation variability.

.planning/research/SUMMARY.md

@@ -0,0 +1,5 @@
+# Research Summary
+
+CAS Evals v0.1 should be a small, inspectable evaluation kernel rather than a provider integration layer. Its strongest public evidence is deterministic fixtures, explicit independent gates, stable result JSON, and cross-platform CI.
+
+The primary risks are non-reproducibility, opaque aggregate scores, and weak adversarial coverage. The architecture addresses these with pure evaluators, mandatory safety gates, evidence-rich results, and versioned benchmark fixtures.