Skip to content

Commit a9563f7

Browse files
AitomatesAitomates
committed
gsd:docs generate bespoke architecture and diagrams
1 parent c16cfc1 commit a9563f7

2 files changed

Lines changed: 142 additions & 2 deletions

File tree

‎docs/architecture.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Architecture & Design
2+
3+
This document describes the architectural logic and data flow of **CAS Evals**.
4+
5+
## Core Logic
6+
7+
CAS Evals is built around a deterministic evaluation pipeline that evaluates LLM-generated responses against expected baselines. It supports two distinct execution paths:
8+
9+
1. **Offline Mode (Default):** Evaluates pre-recorded fixture data directly. It is completely offline, ensuring maximum reproducibility.
10+
2. **Live Reference Product Mode:** Opt-in mode using the `--reference-product-url` flag. It posts a `PromptEnvelope` to a live, local HTTP endpoint (`cas-reference-product`), extracts the dynamic output, and runs it through the same deterministic evaluation pipeline.
11+
12+
### Evaluation Criteria
13+
14+
For each case, CAS Evals enforces strict gating rules:
15+
- **Quality:** Fraction of expected keyword concepts present in the response versus a configured minimum threshold.
16+
- **Safety:** Absence of prohibited unsafe content (requires a 100% pass rate).
17+
- **Cost:** Cost in USD validated against a configured maximum threshold.
18+
- **Latency:** Duration in milliseconds validated against a configured maximum threshold.
19+
20+
Any threshold violation fails the case, resulting in a failed suite.
21+
22+
### Schema Contracts and Evidence
23+
24+
Every per-case result generates an `EvaluationResult` conforming to the pinned `cas-contracts` version. The evidence payload includes:
25+
- SHA-256 fixture digests to guarantee immutability.
26+
- Strict lifecycle metadata (e.g., `correlationId`, `traceContext`) that must be preserved by live reference products.
27+
- Granular metric results (`value`, `threshold`, `passed`, `details`).
28+
29+
## System Architecture Diagram
30+
31+
The following Mermaid diagram visualizes the evaluation pipeline across both offline and live modes.
32+
33+
```mermaid
34+
flowchart TD
35+
%% CLI and Input
36+
CLI[cas_evals.cli]
37+
FixtureJSON[("Fixture JSON\n(e.g., golden.json)")]
38+
39+
CLI -->|Parses| FixtureJSON
40+
FixtureJSON -->|Yields Suite & Cases| Router{Live Mode\nEnabled?}
41+
42+
%% Branch: Offline vs Live
43+
Router -- "No (--output)" --> OfflineEv[Offline Evaluator\nevaluator.py]
44+
Router -- "Yes (--reference-product-url)" --> LiveRef[Reference Product Adapter\nreference_product.py]
45+
46+
%% Live Reference Product Flow
47+
subgraph Live Reference Product Adapter
48+
LiveRef --> BuildEnv[Build PromptEnvelope]
49+
BuildEnv --> HTTPPost[HTTP POST\n/api/v1/workflows]
50+
HTTPPost --> HTTPResp[Extract Response & Events]
51+
HTTPResp --> ValidateMeta{Validate Lifecycle\nEvents/Metadata}
52+
ValidateMeta -- Invalid --> RefError((ReferenceProductError))
53+
ValidateMeta -- Valid --> LiveCase[Inject live 'output'\nas 'response']
54+
end
55+
56+
LiveCase --> SharedEval
57+
58+
%% Offline Flow
59+
OfflineEv --> SharedEval[Shared Core Evaluator\n_evaluate_case_with_evidence]
60+
61+
%% Core Evaluation Logic
62+
subgraph Evaluation Pipeline
63+
SharedEval --> MetricQ[Calculate Quality\n(expected keywords)]
64+
SharedEval --> MetricS[Calculate Safety\n(prohibited words)]
65+
SharedEval --> MetricC[Extract Cost\n(observed USD vs limits)]
66+
SharedEval --> MetricL[Extract Latency\n(observed ms vs limits)]
67+
68+
MetricQ --> Agg[Aggregate Evidence]
69+
MetricS --> Agg
70+
MetricC --> Agg
71+
MetricL --> Agg
72+
73+
Agg --> ValidateContract[Validate EvaluationResult Schema]
74+
end
75+
76+
%% Output
77+
ValidateContract --> PassFail{Passed\nAll Metrics?}
78+
PassFail -- Yes --> Success((Suite Pass))
79+
PassFail -- No --> Fail((Suite Fail))
80+
81+
%% Contract references
82+
casContracts[("cas-contracts\nvendor/cas-contracts/")] -.-> ValidateContract
83+
84+
classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px;
85+
classDef error fill:#ffe6e6,stroke:#cc0000;
86+
classDef success fill:#e6ffe6,stroke:#006600;
87+
class RefError,Fail error;
88+
class Success success;
89+
```
90+
91+
## Module Responsibilities
92+
93+
- **`cli.py`:** Parses arguments, loads the fixture suite, handles top-level error formatting, and writes JSON results to `stdout` and/or the `--output` file.
94+
- **`evaluator.py`:** Contains the pure evaluation kernel (`_evaluate_case_with_evidence`), responsible for scoring quality, safety, and validating cost/latency against thresholds. Forms the standard `EvaluationResult`.
95+
- **`reference_product.py`:** Contains the HTTP transport and metadata validation logic for live endpoint evaluations. It wraps the core evaluator by providing it with dynamic responses instead of fixture responses.
96+
- **`contracts.py`:** Integrates the pinned `cas-contracts` JSON schemas from `vendor/cas-contracts/` to ensure offline validation of all evidence.

‎docs/index.md‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,47 @@
1-
# cas-evals Documentation
1+
# CAS Evals Developer Documentation
22

3-
Welcome to the documentation for cas-evals.
3+
Welcome to the developer documentation for **CAS Evals**, the reproducible evaluation framework for the [Coding Autopilot System](https://github.com/Coding-Autopilot-System).
4+
5+
## Overview
6+
7+
CAS Evals provides a deterministic, offline-first evaluation kernel for assessing AI engineering claims. It runs versioned golden tasks and adversarial prompts against reference responses, producing machine-readable evidence for quality, safety, cost, and latency.
8+
9+
By eliminating the need for secrets, model-provider accounts, or network access in its default mode, CAS Evals ensures that benchmark inputs, thresholds, scoring logic, fixture digests, and pass/fail decisions are fully reviewable and reproducible.
10+
11+
## Key Principles
12+
13+
- **Deterministic & Offline-First:** Default test runs use pre-recorded responses. No network connectivity or live LLM calls are required to reproduce the baseline.
14+
- **Traceable:** Every evaluation result conforms to the published `cas-contracts` schema (`EvaluationResult`). Execution evidence includes SHA-256 fixture digests and robust lifecycle metadata.
15+
- **Strict Verification:** Evals rigorously checks for expected concepts (Quality), absence of prohibited content (Safety), and verifies cost and latency limits. Any failure in mandatory metrics fails the entire suite.
16+
- **Opt-in Live Integration:** Seamlessly evaluate live workflow outputs via the local `cas-reference-product` endpoint while enforcing metadata preservation.
17+
18+
## Getting Started
19+
20+
### Local Setup
21+
Ensure you have Python 3.11+ installed. Install the package in editable mode:
22+
```powershell
23+
python -m pip install -e .
24+
python -m unittest discover -s tests -v
25+
```
26+
27+
### Running Evaluations
28+
29+
Run an offline evaluation using a benchmark fixture:
30+
```powershell
31+
python -m cas_evals.cli benchmarks/v0.2/golden.json --output artifacts/golden.json
32+
```
33+
34+
Evaluate a live workflow using the reference product adapter:
35+
```powershell
36+
python -m cas_evals.cli benchmarks/reference-product/v0.1/golden.json --reference-product-url
37+
```
38+
39+
## Documentation Map
40+
41+
- **[Architecture & Design](architecture.md):** Deep dive into the evaluation workflow, live adapter logic, and see the system architecture diagram.
42+
- **[Reference Product Integration](reference-product-integration.md):** Learn how to integrate and validate live workflows.
43+
- **[Benchmark Reports]:** Review the checked-in, reproducible public baselines (e.g., [v0.2](benchmark-report-v0.2.md)).
44+
45+
## Contributing
46+
47+
See our [CONTRIBUTING.md](../CONTRIBUTING.md) guide for instructions on how to submit changes, run tests, and adhere to our development standards. Security concerns should be reported according to [SECURITY.md](../SECURITY.md).

0 commit comments

Comments
 (0)