|
| 1 | +# Architecture |
| 2 | + |
| 3 | +## Evidence gate flow |
| 4 | + |
| 5 | +```mermaid |
| 6 | +flowchart TD |
| 7 | + CLI["cas_evals.cli"] |
| 8 | + FixtureJSON[("Fixture JSON\ngolden.json / adversarial.json")] |
| 9 | + CLI -->|"Parses"| FixtureJSON |
| 10 | + FixtureJSON -->|"Yields Suite & Cases"| Router{"Live mode\nenabled?"} |
| 11 | + Router -- "No (default)" --> OfflineEv["Offline Evaluator\nevaluator.py"] |
| 12 | + Router -- "Yes (--reference-product-url)" --> LiveRef["Reference Product Adapter\nreference_product.py"] |
| 13 | + subgraph LiveFlow["Live reference-product adapter"] |
| 14 | + LiveRef --> BuildEnv["Build PromptEnvelope"] --> HTTPPost["HTTP POST /api/v1/workflows"] --> ValidateMeta{"Validate lifecycle\nevents/metadata"} |
| 15 | + ValidateMeta -- Invalid --> RefError(("ReferenceProductError")) |
| 16 | + ValidateMeta -- Valid --> LiveCase["Inject live output as response"] |
| 17 | + end |
| 18 | + LiveCase --> SharedEval["Shared core evaluator\n_evaluate_case_with_evidence"] |
| 19 | + OfflineEv --> SharedEval |
| 20 | + SharedEval --> Metrics["Quality / Safety / Cost / Latency"] |
| 21 | + Metrics --> ValidateContract["Validate EvaluationResult\nagainst vendored cas-contracts schema"] |
| 22 | + ValidateContract --> Gate{"release --check\nexit code"} |
| 23 | + Gate -- "0" --> Pass(("CI green")) |
| 24 | + Gate -- "non-zero" --> Fail(("CI red")) |
| 25 | +``` |
| 26 | + |
| 27 | +<!-- codex:generate-image prompt="A courtroom-style evidence gate: fixture documents and a live data stream both feed into a single judge's bench (a glowing scale icon) that stamps each case with a pass or fail seal against a rulebook labeled cas-contracts; a exit-code lever on the side flips green or red; isometric, enterprise blue/graphite palette" style="isometric, enterprise, clean" replaces="mermaid-above" --> |
| 28 | + |
| 29 | +## Modules |
| 30 | + |
| 31 | +- `cli.py` — parses arguments, loads the fixture suite, writes JSON results to stdout and/or |
| 32 | + `--output`. |
| 33 | +- `evaluator.py` — pure evaluation kernel (`_evaluate_case_with_evidence`): scores quality |
| 34 | + (fraction of expected keywords present), safety (absence of prohibited content, mandatory |
| 35 | + 100%), and validates cost/latency against configured thresholds. Produces the standard |
| 36 | + `EvaluationResult`. |
| 37 | +- `reference_product.py` — HTTP transport and lifecycle-metadata validation for the opt-in |
| 38 | + live mode; wraps the core evaluator with a dynamic response instead of a fixture response. |
| 39 | +- `contracts.py` — validates evidence offline against the pinned `cas-contracts` schemas |
| 40 | + vendored under `vendor/cas-contracts/`. |
| 41 | + |
| 42 | +## Registry-fetch smoke check — in progress (PR #9) |
| 43 | + |
| 44 | +`main` today validates schema conformance **offline only**, against the vendored copy under |
| 45 | +`vendor/cas-contracts/v0.1.0/` — it does not prove the live Pages registry actually resolves. |
| 46 | +PR #9 (`feat(contracts): verify registry-fetch smoke check + update vendored $id to resolvable |
| 47 | +Pages URL`) adds a `cas_evals.registry_check` module (stdlib `urllib`, no new dependencies) |
| 48 | +that performs live HTTP GETs against `index.json`, `v0.1/manifest.json`, and two schema files |
| 49 | +on the Pages registry and asserts HTTP 200 — proving resolution, which the offline check |
| 50 | +cannot. It also re-vendors the local `$id` expectations to the new resolvable Pages URL |
| 51 | +convention, as a companion to `cas-contracts` PR #18 (independent of whether #18 has merged, |
| 52 | +per the PR's own verification: `python -m cas_evals.registry_check` was confirmed to exit 0 |
| 53 | +against the live registry pre-merge). Until PR #9 merges, no live-resolution check runs in CI. |
| 54 | + |
| 55 | +<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 --> |
0 commit comments