Skip to content

Commit 5a57631

Browse files
AitomatesAitomates
committed
docs(phase-2): complete shared contracts and corpus
1 parent 83f91cf commit 5a57631

5 files changed

Lines changed: 79 additions & 12 deletions

File tree

‎.planning/PROJECT.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,11 @@ Every CAS capability claim can be reproduced from versioned fixtures and machine
1212

1313
### Validated
1414

15-
(None yet - ship to validate)
15+
- [x] Shared evaluation results consume provenance-pinned published `cas-contracts` schemas offline.
16+
- [x] The public corpus represents core CAS engineering workflows and independent safety risks.
17+
- [x] Benchmark release artifacts regenerate byte-identically with machine-readable provenance.
18+
19+
Validated in Phase 2: Shared Contracts And Corpus.
1620

1721
### Active
1822

@@ -50,7 +54,9 @@ CAS needs measurable proof that its prompt refinement, autonomous engineering, a
5054

5155
This document evolves at phase transitions and milestone boundaries.
5256

57+
Phase 2 is complete. The repository now consumes shared contracts offline, runs a representative v0.2 corpus, and publishes reproducible v0.2.0 release evidence. Phase 3 adds isolated opt-in live adapters.
58+
5359
After each phase, validate requirements, record new decisions, and update scope. After each milestone, review the core value, exclusions, and evidence quality.
5460

5561
---
56-
*Last updated: 2026-06-11 after initialization*
62+
*Last updated: 2026-06-11 after Phase 2 completion*

‎.planning/REQUIREMENTS.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,9 @@
3131

3232
### Shared Contracts And Corpus
3333

34-
- [ ] **SHRD-01**: Maintainer can validate emitted evaluation results against a provenance-pinned published `cas-contracts` schema without network access.
35-
- [ ] **CORP-01**: User can run a representative golden-task corpus covering core CAS engineering workflows.
36-
- [ ] **REL-01**: Maintainer can deterministically generate and publish reviewable benchmark release artifacts.
34+
- [x] **SHRD-01**: Maintainer can validate emitted evaluation results against a provenance-pinned published `cas-contracts` schema without network access.
35+
- [x] **CORP-01**: User can run a representative golden-task corpus covering core CAS engineering workflows.
36+
- [x] **REL-01**: Maintainer can deterministically generate and publish reviewable benchmark release artifacts.
3737

3838
## v2 Requirements
3939

@@ -57,7 +57,7 @@
5757
| METR-01, METR-02, METR-03, METR-04 | Phase 1 | Complete |
5858
| EVID-01, EVID-02, EVID-03 | Phase 1 | Complete |
5959
| GOV-01, GOV-02 | Phase 1 | Complete |
60-
| SHRD-01, CORP-01, REL-01 | Phase 2 | Pending |
60+
| SHRD-01, CORP-01, REL-01 | Phase 2 | Complete |
6161

6262
**Coverage:** 15 v1 requirements, 15 mapped, 0 unmapped.
6363

‎.planning/ROADMAP.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,21 @@
1616

1717
Consume versioned `cas-contracts` schemas, expand representative CAS golden tasks, and publish benchmark release artifacts.
1818

19+
**Status:** Complete (2026-06-11)
20+
1921
## Phase 3: Isolated Live Adapters
2022

2123
Add opt-in provider adapters with redaction, managed identity where applicable, cost controls, and recorded provenance.
2224

2325
## Phase 4: Statistical And Longitudinal Evidence
2426

2527
Add repeated-run statistics, baseline comparison, regression budgets, signed reports, and a public trend dashboard.
28+
29+
## Progress
30+
31+
| Phase | Status | Completed |
32+
|-------|--------|-----------|
33+
| 1. Reproducible Evaluation Kernel | Complete | 2026-06-11 |
34+
| 2. Shared Contracts And Corpus | Complete | 2026-06-11 |
35+
| 3. Isolated Live Adapters | Pending | - |
36+
| 4. Statistical And Longitudinal Evidence | Pending | - |

‎.planning/STATE.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,15 @@
22
gsd_state_version: 1.0
33
milestone: v0.1
44
milestone_name: milestone
5-
status: unknown
6-
last_updated: "2026-06-11T11:03:22.404Z"
5+
status: ready_to_plan
6+
last_updated: 2026-06-11T11:09:44.109Z
77
progress:
88
total_phases: 4
9-
completed_phases: 0
9+
completed_phases: 2
1010
total_plans: 3
11-
completed_plans: 0
12-
percent: 0
11+
completed_plans: 3
12+
percent: 50
13+
stopped_at: Phase 2 complete (3/3) — ready to discuss Phase 3
1314
---
1415

1516
# Project State
@@ -19,7 +20,7 @@ progress:
1920
See: `.planning/PROJECT.md` (updated 2026-06-11)
2021

2122
**Core value:** Every CAS capability claim can be reproduced from versioned fixtures and machine-readable results.
22-
**Current focus:** Phase 2 — Shared Contracts And Corpus
23+
**Current focus:** Phase 3 — isolated live adapters
2324

2425
## Status
2526

‎.planning/phases/02-shared-contracts-and-corpus/02-VERIFICATION.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
phase: 02-shared-contracts-and-corpus
3+
status: passed
4+
verified: 2026-06-11
5+
score: 9/9
6+
requirements: [SHRD-01, CORP-01, REL-01]
7+
---
8+
9+
# Phase 2 Verification
10+
11+
## Goal
12+
13+
Consume versioned `cas-contracts` schemas, expand representative CAS golden tasks, and publish benchmark release artifacts without weakening standalone execution.
14+
15+
## Must-Have Verification
16+
17+
| Must-have | Status | Evidence |
18+
|-----------|--------|----------|
19+
| Published shared schemas are consumed | Passed | Exact `cas-contracts` v0.1.0 schemas vendored with tag, source, blob SHA, and SHA-256 provenance. |
20+
| Shared validation is offline and standalone | Passed | `src/cas_evals/contracts.py` uses only the Python standard library and no network calls. |
21+
| Per-case results conform to shared contract | Passed | Contract tests validate emitted and released results and reject malformed or extended objects. |
22+
| Safety remains an independent mandatory gate | Passed | Evaluator and corpus tests prove safety failure independently fails a case. |
23+
| Golden corpus is representative | Passed | Eight explicit capability categories are enforced by corpus tests. |
24+
| Adversarial corpus is representative | Passed | Six independent safety-risk categories are enforced by corpus tests. |
25+
| Release artifacts are deterministic | Passed | `python -m cas_evals.release --check` proves byte-for-byte regeneration. |
26+
| Release artifacts contain provenance | Passed | Manifest records shared contract, fixture, and artifact digests. |
27+
| Cross-platform verification is enforced | Passed | CI covers Windows/Linux and JSON LF policy preserves digest stability. |
28+
29+
## Requirement Traceability
30+
31+
| Requirement | Status | Evidence |
32+
|-------------|--------|----------|
33+
| SHRD-01 | Passed | Vendored schemas, provenance verification, shared result validation tests. |
34+
| CORP-01 | Passed | v0.2 golden/adversarial fixtures and capability coverage tests. |
35+
| REL-01 | Passed | v0.2.0 release artifacts, manifest, generator, reproducibility tests. |
36+
37+
## Automated Verification
38+
39+
- `powershell -ExecutionPolicy Bypass -File scripts/verify.ps1` - passed.
40+
- `python -m unittest discover -s tests -v` - 21 tests passed.
41+
- Golden v0.2 benchmark - 8/8 passed.
42+
- Adversarial v0.2 benchmark - 6/6 passed.
43+
- `python -m cas_evals.release --check` - passed.
44+
- `python -m compileall -q src tests` - passed.
45+
- `git diff --check` - passed.
46+
47+
## Result
48+
49+
Phase goal achieved. No human verification or gap-closure plans are required.

0 commit comments

Comments
 (0)