Skip to content

Commit 0358ed5

Browse files
committed
docs: record reference product integration verification
1 parent aaeed60 commit 0358ed5

4 files changed

Lines changed: 98 additions & 0 deletions

File tree

.planning/STATE.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,11 @@ See: `.planning/PROJECT.md` (updated 2026-06-11)
2727
- v0.1 scaffold implemented.
2828
- Deterministic benchmark and test evidence required before release.
2929
- Next phase: consume published `cas-contracts` schemas without weakening standalone execution.
30+
31+
### Quick Tasks Completed
32+
33+
| # | Description | Date | Commit | Status | Directory |
34+
|---|-------------|------|--------|--------|-----------|
35+
| 260612-sob | Deterministic cas-reference-product golden path | 2026-06-12 | `aaeed60` | Verified | [260612-sob-implement-deterministic-cas-reference-pr](./quick/260612-sob-implement-deterministic-cas-reference-pr/) |
36+
37+
Last activity: 2026-06-12 - Completed quick task 260612-sob: deterministic cas-reference-product golden path
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
status: complete
3+
task: deterministic cas-reference-product golden path
4+
---
5+
6+
# Quick Task 260612-sob Plan
7+
8+
## Goal
9+
10+
Add an opt-in, executable `cas-reference-product` HTTP evaluation path without weakening the existing offline evaluator.
11+
12+
## Must Haves
13+
14+
- Score the actual `POST /api/v1/workflows` returned `output`.
15+
- Preserve and verify `correlationId`, `promptId`, `runId`, and trace context in deterministic evidence.
16+
- Support golden and adversarial fixture suites.
17+
- Keep persisted timing normalized and byte-stable.
18+
- Keep the existing offline CLI and release path unchanged.
19+
- Add focused tests, reference-product corpus fixtures, documentation, and CI validation.
20+
21+
## Tasks
22+
23+
1. Add a standard-library reference-product adapter and CLI opt-in.
24+
2. Add deterministic reference-product golden/adversarial corpora and regression tests.
25+
3. Update documentation, CI, GSD state, and run all verification gates.
26+
27+
## Verification
28+
29+
- `python -m unittest discover -s tests -v`
30+
- `python -m cas_evals.cli benchmarks/v0.2/golden.json`
31+
- `python -m cas_evals.cli benchmarks/v0.2/adversarial.json`
32+
- `python -m cas_evals.release --check`
33+
- `git diff --check`
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
status: complete
3+
completed: 2026-06-12
4+
---
5+
6+
# Quick Task 260612-sob Summary
7+
8+
Implemented an opt-in deterministic HTTP adapter for the local
9+
`cas-reference-product` workflow endpoint while preserving the existing offline
10+
evaluation and release paths.
11+
12+
## Delivered
13+
14+
- Actual returned workflow output is scored for quality and safety.
15+
- Lifecycle metadata and trace context are generated deterministically, verified
16+
against returned events, and preserved in evaluation evidence.
17+
- Persisted live evidence excludes server timestamps and endpoint addresses and
18+
uses normalized fixture timing.
19+
- Golden and adversarial reference-product corpora pass against the actual local
20+
sibling service.
21+
- HTTP, CLI, metadata-drift, actual-output, determinism, and failure-path tests
22+
run in CI.
23+
- User documentation describes the local executable golden path.
24+
25+
## Commits
26+
27+
- `2a89a9a` - deterministic reference-product evaluation and tests
28+
- `aaeed60` - integration documentation and CI coverage
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
status: passed
3+
verified: 2026-06-12
4+
---
5+
6+
# Quick Task 260612-sob Verification
7+
8+
## Result
9+
10+
Passed. All must-haves in the plan are implemented and directly verified.
11+
12+
## Evidence
13+
14+
- `powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\verify.ps1`
15+
- 30/30 unit tests passed.
16+
- Offline golden corpus passed 8/8.
17+
- Offline adversarial corpus passed 6/6.
18+
- Checked-in v0.2.0 release artifacts regenerated byte-identically.
19+
- `python -m unittest discover -s tests -p test_reference_product.py -v`
20+
- 9/9 reference-product integration tests passed.
21+
- Actual local `cas-reference-product` service:
22+
- reference-product golden corpus passed 1/1.
23+
- reference-product adversarial corpus passed 1/1.
24+
- returned output was scored and lifecycle metadata was preserved.
25+
- `git diff --check` passed.
26+
27+
## Scope
28+
29+
Only `C:\PersonalRepo\portfolio\cas-evals` was modified.

0 commit comments

Comments
 (0)