Skip to content

Commit 3ead465

Browse files
AitomatesAitomates
committed
docs(36-03): add docs/wiki tree (Home, Architecture, Operations, Decisions)
- Architecture.md documents the evidence gate flow (offline + live reference-product modes) and the in-progress registry-fetch smoke check (PR #9, companion to cas-contracts PR #18) - Operations.md verified CLI/gate/CI commands - Decisions.md indexes phase history and open PRs
1 parent f305f9e commit 3ead465

4 files changed

Lines changed: 157 additions & 0 deletions

File tree

‎docs/wiki/Architecture.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Architecture
2+
3+
## Evidence gate flow
4+
5+
```mermaid
6+
flowchart TD
7+
CLI["cas_evals.cli"]
8+
FixtureJSON[("Fixture JSON\ngolden.json / adversarial.json")]
9+
CLI -->|"Parses"| FixtureJSON
10+
FixtureJSON -->|"Yields Suite & Cases"| Router{"Live mode\nenabled?"}
11+
Router -- "No (default)" --> OfflineEv["Offline Evaluator\nevaluator.py"]
12+
Router -- "Yes (--reference-product-url)" --> LiveRef["Reference Product Adapter\nreference_product.py"]
13+
subgraph LiveFlow["Live reference-product adapter"]
14+
LiveRef --> BuildEnv["Build PromptEnvelope"] --> HTTPPost["HTTP POST /api/v1/workflows"] --> ValidateMeta{"Validate lifecycle\nevents/metadata"}
15+
ValidateMeta -- Invalid --> RefError(("ReferenceProductError"))
16+
ValidateMeta -- Valid --> LiveCase["Inject live output as response"]
17+
end
18+
LiveCase --> SharedEval["Shared core evaluator\n_evaluate_case_with_evidence"]
19+
OfflineEv --> SharedEval
20+
SharedEval --> Metrics["Quality / Safety / Cost / Latency"]
21+
Metrics --> ValidateContract["Validate EvaluationResult\nagainst vendored cas-contracts schema"]
22+
ValidateContract --> Gate{"release --check\nexit code"}
23+
Gate -- "0" --> Pass(("CI green"))
24+
Gate -- "non-zero" --> Fail(("CI red"))
25+
```
26+
27+
<!-- codex:generate-image prompt="A courtroom-style evidence gate: fixture documents and a live data stream both feed into a single judge's bench (a glowing scale icon) that stamps each case with a pass or fail seal against a rulebook labeled cas-contracts; a exit-code lever on the side flips green or red; isometric, enterprise blue/graphite palette" style="isometric, enterprise, clean" replaces="mermaid-above" -->
28+
29+
## Modules
30+
31+
- `cli.py` — parses arguments, loads the fixture suite, writes JSON results to stdout and/or
32+
`--output`.
33+
- `evaluator.py` — pure evaluation kernel (`_evaluate_case_with_evidence`): scores quality
34+
(fraction of expected keywords present), safety (absence of prohibited content, mandatory
35+
100%), and validates cost/latency against configured thresholds. Produces the standard
36+
`EvaluationResult`.
37+
- `reference_product.py` — HTTP transport and lifecycle-metadata validation for the opt-in
38+
live mode; wraps the core evaluator with a dynamic response instead of a fixture response.
39+
- `contracts.py` — validates evidence offline against the pinned `cas-contracts` schemas
40+
vendored under `vendor/cas-contracts/`.
41+
42+
## Registry-fetch smoke check — in progress (PR #9)
43+
44+
`main` today validates schema conformance **offline only**, against the vendored copy under
45+
`vendor/cas-contracts/v0.1.0/` — it does not prove the live Pages registry actually resolves.
46+
PR #9 (`feat(contracts): verify registry-fetch smoke check + update vendored $id to resolvable
47+
Pages URL`) adds a `cas_evals.registry_check` module (stdlib `urllib`, no new dependencies)
48+
that performs live HTTP GETs against `index.json`, `v0.1/manifest.json`, and two schema files
49+
on the Pages registry and asserts HTTP 200 — proving resolution, which the offline check
50+
cannot. It also re-vendors the local `$id` expectations to the new resolvable Pages URL
51+
convention, as a companion to `cas-contracts` PR #18 (independent of whether #18 has merged,
52+
per the PR's own verification: `python -m cas_evals.registry_check` was confirmed to exit 0
53+
against the live registry pre-merge). Until PR #9 merges, no live-resolution check runs in CI.
54+
55+
<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->

‎docs/wiki/Decisions.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Decisions
2+
3+
## ADR convention
4+
5+
`docs/adr/README.md` establishes the convention (sequential numbering, Context/Decision/
6+
Consequences) but **no numbered ADR files exist in the repo yet**. Decisions to date live in
7+
`.planning/phases/` plan/summary pairs instead.
8+
9+
## Phase history (`.planning/phases/`, this repo's own GSD project)
10+
11+
| Phase | Topic |
12+
|---|---|
13+
| 02 | Shared contracts and corpus |
14+
15+
See `.planning/phases/02-shared-contracts-and-corpus/*-SUMMARY.md` for the detailed record.
16+
17+
## Open decisions tracked in this Phase 36 refresh
18+
19+
- **PR #9** (`feat/registry-fetch-smoke-check`) — adds a live registry-fetch smoke check
20+
(`cas_evals.registry_check`) and re-vendors `$id` expectations to the resolvable Pages URL
21+
convention, as a companion to `cas-contracts` PR #18; open, not yet merged. See
22+
[Architecture](./Architecture.md#registry-fetch-smoke-check--in-progress-pr-9).
23+
- **PR #10** (`ci/sha-pin-actions`) — pins third-party GitHub Actions to commit SHAs; open,
24+
not yet merged. (Note: the current `ci.yml` already shows pinned `checkout`/`setup-python`
25+
actions on `main` — verify PR #10's remaining scope before assuming no pinning exists.)
26+
27+
<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->

‎docs/wiki/Home.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# CAS Evals Wiki
2+
3+
## Role in the CAS portfolio
4+
5+
`cas-evals` is part of the **Governance plane** of the Coding-Autopilot-System three-plane
6+
model (Control / Execution / Governance). It is the evidence gate: it runs versioned golden
7+
and adversarial tasks against deterministic reference responses (or, opt-in, against a live
8+
`cas-reference-product` workflow) and produces machine-readable, schema-conformant
9+
quality/safety/cost/latency evidence with a CLI that exits non-zero on any mandatory metric
10+
failure — making "done" mean something checkable, not just claimed.
11+
12+
| Plane | This repo's responsibility |
13+
|---|---|
14+
| Control | *(consumed indirectly — evidence gate for control-plane goal completion)* |
15+
| Execution | *(consumed indirectly — evaluates `cas-reference-product` live output, opt-in)* |
16+
| Governance | Deterministic evaluation kernel, evidence gate, `cas-contracts` schema validation |
17+
18+
## Quickstart
19+
20+
- [README.md](../../README.md) — Quickstart, Metrics, Evidence Contract
21+
- [Architecture](./Architecture.md) — evidence gate flow (offline + live reference-product modes)
22+
- [Operations](./Operations.md) — verified CLI/gate/CI commands
23+
- [Decisions](./Decisions.md) — phase history and open PRs (including the registry-fetch
24+
smoke check, PR #9)
25+
26+
## Ecosystem links
27+
28+
Part of the [Coding-Autopilot-System](https://github.com/Coding-Autopilot-System) org:
29+
[gsd-orchestrator](https://github.com/Coding-Autopilot-System/gsd-orchestrator) (control plane) ·
30+
[autogen](https://github.com/Coding-Autopilot-System/autogen) (execution plane) ·
31+
[Promptimprover](https://github.com/Coding-Autopilot-System/Promptimprover) (prompt governance) ·
32+
[cas-contracts](https://github.com/Coding-Autopilot-System/cas-contracts) (shared schemas)
33+
34+
<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->

‎docs/wiki/Operations.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Operations
2+
3+
## Setup and run
4+
5+
```powershell
6+
python -m pip install -e .
7+
python -m unittest discover -s tests -v
8+
python -m cas_evals.cli benchmarks/v0.2/golden.json --output artifacts/golden.json
9+
python -m cas_evals.cli benchmarks/v0.2/adversarial.json --output artifacts/adversarial.json
10+
python -m cas_evals.release --check
11+
```
12+
13+
The CLI exits non-zero when any mandatory metric fails, making each suite usable as a CI
14+
regression gate. Windows users can run the complete verification path with
15+
`.\scripts\verify.ps1`.
16+
17+
## Live reference-product mode (opt-in)
18+
19+
```powershell
20+
python -m cas_evals.cli benchmarks/reference-product/v0.1/golden.json --reference-product-url
21+
python -m cas_evals.cli benchmarks/reference-product/v0.1/adversarial.json --reference-product-url
22+
```
23+
24+
## CI (`.github/workflows/ci.yml`, matrix: ubuntu-latest + windows-latest, Python 3.11 + 3.13, 10-minute timeout)
25+
26+
1. `actions/checkout` (pinned to commit SHA)
27+
2. `actions/setup-python` (pinned to commit SHA)
28+
3. `python -m pip install -e .`
29+
4. `python -m unittest discover -s tests -v`
30+
5. `python -m unittest discover -s tests -p test_reference_product.py -v`
31+
6. `python -m cas_evals.cli benchmarks/v0.2/golden.json`
32+
7. `python -m cas_evals.cli benchmarks/v0.2/adversarial.json`
33+
8. `python -m cas_evals.release --check` — the CLI-exit-code gate; **not** a coverage badge,
34+
since no formal coverage-percentage gate exists in this repo.
35+
36+
A separate `.github/workflows/codeql.yml` runs CodeQL analysis, and `.github/workflows/
37+
pages.yml` publishes documentation (badges in the root `README.md`). A registry-fetch smoke
38+
check (live HTTP verification of the Pages registry) is in progress — see
39+
[Architecture](./Architecture.md) and PR #9.
40+
41+
<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->

0 commit comments

Comments
 (0)