Skip to content

Commit 8b98613

Browse files
OgeonX-AiOgeonX-Ai
authored
Merge pull request #1 from Coding-Autopilot-System/gsd/phase-2-shared-contracts-corpus
feat: complete Phase 2 shared contracts and corpus
2 parents cec9519 + 5a57631 commit 8b98613

40 files changed

Lines changed: 2287 additions & 64 deletions

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.json text eol=lf

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,6 @@ jobs:
2222
python-version: ${{ matrix.python-version }}
2323
- run: python -m pip install -e .
2424
- run: python -m unittest discover -s tests -v
25-
- run: python -m cas_evals.cli benchmarks/v0.1/golden.json
26-
- run: python -m cas_evals.cli benchmarks/v0.1/adversarial.json
25+
- run: python -m cas_evals.cli benchmarks/v0.2/golden.json
26+
- run: python -m cas_evals.cli benchmarks/v0.2/adversarial.json
27+
- run: python -m cas_evals.release --check

‎.planning/PROJECT.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,11 @@ Every CAS capability claim can be reproduced from versioned fixtures and machine
1212

1313
### Validated
1414

15-
(None yet - ship to validate)
15+
- [x] Shared evaluation results consume provenance-pinned published `cas-contracts` schemas offline.
16+
- [x] The public corpus represents core CAS engineering workflows and independent safety risks.
17+
- [x] Benchmark release artifacts regenerate byte-identically with machine-readable provenance.
18+
19+
Validated in Phase 2: Shared Contracts And Corpus.
1620

1721
### Active
1822

@@ -50,7 +54,9 @@ CAS needs measurable proof that its prompt refinement, autonomous engineering, a
5054

5155
This document evolves at phase transitions and milestone boundaries.
5256

57+
Phase 2 is complete. The repository now consumes shared contracts offline, runs a representative v0.2 corpus, and publishes reproducible v0.2.0 release evidence. Phase 3 adds isolated opt-in live adapters.
58+
5359
After each phase, validate requirements, record new decisions, and update scope. After each milestone, review the core value, exclusions, and evidence quality.
5460

5561
---
56-
*Last updated: 2026-06-11 after initialization*
62+
*Last updated: 2026-06-11 after Phase 2 completion*

‎.planning/REQUIREMENTS.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,12 @@
2929
- [ ] **GOV-01**: Contributor can run documented tests and benchmarks without secrets.
3030
- [ ] **GOV-02**: Repository publishes security and contribution guidance.
3131

32+
### Shared Contracts And Corpus
33+
34+
- [x] **SHRD-01**: Maintainer can validate emitted evaluation results against a provenance-pinned published `cas-contracts` schema without network access.
35+
- [x] **CORP-01**: User can run a representative golden-task corpus covering core CAS engineering workflows.
36+
- [x] **REL-01**: Maintainer can deterministically generate and publish reviewable benchmark release artifacts.
37+
3238
## v2 Requirements
3339

3440
- **LIVE-01**: User can evaluate live model-provider responses through isolated adapters.
@@ -51,8 +57,9 @@
5157
| METR-01, METR-02, METR-03, METR-04 | Phase 1 | Complete |
5258
| EVID-01, EVID-02, EVID-03 | Phase 1 | Complete |
5359
| GOV-01, GOV-02 | Phase 1 | Complete |
60+
| SHRD-01, CORP-01, REL-01 | Phase 2 | Complete |
5461

55-
**Coverage:** 12 v1 requirements, 12 mapped, 0 unmapped.
62+
**Coverage:** 15 v1 requirements, 15 mapped, 0 unmapped.
5663

5764
---
5865
*Last updated: 2026-06-11 after v0.1 scaffold*

‎.planning/ROADMAP.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,21 @@
1616

1717
Consume versioned `cas-contracts` schemas, expand representative CAS golden tasks, and publish benchmark release artifacts.
1818

19+
**Status:** Complete (2026-06-11)
20+
1921
## Phase 3: Isolated Live Adapters
2022

2123
Add opt-in provider adapters with redaction, managed identity where applicable, cost controls, and recorded provenance.
2224

2325
## Phase 4: Statistical And Longitudinal Evidence
2426

2527
Add repeated-run statistics, baseline comparison, regression budgets, signed reports, and a public trend dashboard.
28+
29+
## Progress
30+
31+
| Phase | Status | Completed |
32+
|-------|--------|-----------|
33+
| 1. Reproducible Evaluation Kernel | Complete | 2026-06-11 |
34+
| 2. Shared Contracts And Corpus | Complete | 2026-06-11 |
35+
| 3. Isolated Live Adapters | Pending | - |
36+
| 4. Statistical And Longitudinal Evidence | Pending | - |

‎.planning/STATE.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,26 @@
1+
---
2+
gsd_state_version: 1.0
3+
milestone: v0.1
4+
milestone_name: milestone
5+
status: ready_to_plan
6+
last_updated: 2026-06-11T11:09:44.109Z
7+
progress:
8+
total_phases: 4
9+
completed_phases: 2
10+
total_plans: 3
11+
completed_plans: 3
12+
percent: 50
13+
stopped_at: Phase 2 complete (3/3) — ready to discuss Phase 3
14+
---
15+
116
# Project State
217

318
## Project Reference
419

520
See: `.planning/PROJECT.md` (updated 2026-06-11)
621

722
**Core value:** Every CAS capability claim can be reproduced from versioned fixtures and machine-readable results.
8-
**Current focus:** Phase 1 complete; prepare shared-contract integration.
23+
**Current focus:** Phase 3 — isolated live adapters
924

1025
## Status
1126

‎.planning/config.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,6 @@
1010
"verifier": true,
1111
"nyquist_validation": true,
1212
"auto_advance": true,
13-
"_auto_chain_active": true
13+
"_auto_chain_active": false
1414
}
1515
}

‎.planning/phases/02-shared-contracts-and-corpus/02-01-PLAN.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
phase: 02-shared-contracts-and-corpus
3+
plan: "01"
4+
type: execute
5+
wave: 1
6+
depends_on: []
7+
files_modified:
8+
- vendor/cas-contracts/v0.1.0/common.schema.json
9+
- vendor/cas-contracts/v0.1.0/evaluation-result.schema.json
10+
- vendor/cas-contracts/v0.1.0/provenance.json
11+
- src/cas_evals/contracts.py
12+
- src/cas_evals/evaluator.py
13+
- tests/test_contracts.py
14+
- tests/test_evaluator.py
15+
requirements: [SHRD-01]
16+
autonomous: true
17+
must_haves:
18+
truths:
19+
- "Per-case results validate against the published shared EvaluationResult contract."
20+
- "Contract validation requires no network, secrets, or third-party runtime packages."
21+
- "Vendored schema provenance is verified."
22+
---
23+
24+
<objective>
25+
Consume the published shared evaluation contract and align deterministic evaluator output.
26+
</objective>
27+
28+
<tasks>
29+
<task type="auto">
30+
<name>Vendor and verify published shared schemas</name>
31+
<read_first>AGENTS.md, .planning/phases/02-shared-contracts-and-corpus/02-CONTEXT.md</read_first>
32+
<action>Vendor exact v0.1.0 common and evaluation-result schemas with immutable provenance. Add a standard-library validator that checks provenance and the current schema constraint surface.</action>
33+
<acceptance_criteria>Contract tests pass offline and reject malformed results.</acceptance_criteria>
34+
</task>
35+
<task type="auto">
36+
<name>Align evaluator output to shared contract</name>
37+
<read_first>src/cas_evals/evaluator.py, tests/test_evaluator.py</read_first>
38+
<action>Emit exact shared EvaluationResult objects and preserve detailed mandatory-gate evidence in the suite envelope.</action>
39+
<acceptance_criteria>Existing and new evaluator tests pass; safety remains independently mandatory.</acceptance_criteria>
40+
</task>
41+
</tasks>
42+
43+
<verification>
44+
python -m unittest discover -s tests -v
45+
</verification>
46+
47+
<success_criteria>
48+
Shared contract consumption is pinned, offline, tested, and used by evaluator output.
49+
</success_criteria>

‎.planning/phases/02-shared-contracts-and-corpus/02-01-SUMMARY.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
phase: 02-shared-contracts-and-corpus
3+
plan: "01"
4+
status: complete
5+
completed: 2026-06-11
6+
requirements: [SHRD-01]
7+
---
8+
9+
# Plan 02-01 Summary
10+
11+
Vendored the immutable `cas-contracts` v0.1.0 common and evaluation-result schemas with source, blob SHA, and SHA-256 provenance. Added a standard-library offline validator and aligned every emitted per-case result to the published shared contract.
12+
13+
Detailed thresholds, fixture digests, and mandatory gate decisions remain in the suite evidence envelope so shared results reject local extensions while safety remains independently mandatory.
14+
15+
## Verification
16+
17+
- `python -m unittest discover -s tests -v` - 12 tests passed.
18+
- `python -m cas_evals.cli benchmarks/v0.1/golden.json --output artifacts/golden.json` - passed.
19+
- `python -m cas_evals.cli benchmarks/v0.1/adversarial.json --output artifacts/adversarial.json` - passed.
20+
- `git diff --check` - passed.
21+
22+
## Deviations from Plan
23+
24+
None - plan executed exactly as written.
25+
26+
## Self-Check: PASSED

‎.planning/phases/02-shared-contracts-and-corpus/02-02-PLAN.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
phase: 02-shared-contracts-and-corpus
3+
plan: "02"
4+
type: execute
5+
wave: 2
6+
depends_on: ["02-01"]
7+
files_modified:
8+
- benchmarks/v0.2/golden.json
9+
- benchmarks/v0.2/adversarial.json
10+
- tests/test_corpus.py
11+
requirements: [CORP-01]
12+
autonomous: true
13+
must_haves:
14+
truths:
15+
- "The corpus represents core CAS engineering workflows."
16+
- "All fixtures remain deterministic, reviewable, secretless, and safe."
17+
---
18+
19+
<objective>
20+
Expand the representative golden and adversarial benchmark corpus.
21+
</objective>
22+
23+
<tasks>
24+
<task type="auto">
25+
<name>Author representative v0.2 corpus</name>
26+
<read_first>benchmarks/v0.1/golden.json, benchmarks/v0.1/adversarial.json, .planning/phases/02-shared-contracts-and-corpus/02-CONTEXT.md</read_first>
27+
<action>Add representative golden and adversarial cases with fixed release metadata and deterministic observations.</action>
28+
<acceptance_criteria>Corpus tests prove unique IDs, required workflow coverage, safe fixtures, and passing suites.</acceptance_criteria>
29+
</task>
30+
</tasks>
31+
32+
<verification>
33+
python -m unittest discover -s tests -v
34+
python -m cas_evals.cli benchmarks/v0.2/golden.json
35+
python -m cas_evals.cli benchmarks/v0.2/adversarial.json
36+
</verification>
37+
38+
<success_criteria>
39+
The v0.2 corpus gives representative, deterministic CAS workflow coverage.
40+
</success_criteria>

0 commit comments

Comments
 (0)