Commit aa07c88
authored
test: phase 0 conformance harness — typed parser for all 68 fixtures (#7)
Phase 0 of the v0.4 → v0.8 implementation plan: the conformance test
harness loads every YAML fixture under
openarmature-spec/spec/<capability>/conformance/ into a typed
pydantic config. Phases 1+ add runtime interpretation under
harness/runtime/ without re-touching parsing.
Bumps the spec submodule to v0.8.0 (8 accepted proposals: foundation,
explicit subgraph mapping, observer hooks, llm-provider, middleware,
fan-out, OTel, checkpointing).
Harness package layout (tests/conformance/harness/):
- fixtures.py Three top-level discriminated variants
(LlmProviderFixture | CasesFixture |
GraphFixture). Discriminator picks by presence
of `mock_provider` / `cases:` keys.
- directives.py Node directive sub-models with mutual-exclusion
validator on the 13 primary directives.
Middleware discriminated union over the 5 types
in fixtures.
- expectations.py Per-capability expected-block discriminator
(graph_engine | llm_provider |
pipeline_utilities | observability) chosen by
which assertion keys appear.
- loader.py Auto-discovers NNN-*.yaml across all four
capability directories.
- skip.py Structured SkipReason with capability +
directive list + phase mapping.
- runtime/ Empty stub package + README locking in the
Phase 0 boundary.
Strictness contract: strict (extra="forbid") at the structural
skeleton (top-level fixture, StateSchema, NodeSpec primary directive
set, ObserverSpec, MiddlewareConfig). Permissive (extra="allow") for
payload-shape models (LLM mock responses, middleware params, flaky
config, case shapes) — those evolve frequently in the spec without
restructuring the directive surface.
Tests added (test_fixture_parsing.py): every fixture parses (68)
AND round-trips (parse → model_dump → re-parse → equal). 137
assertions total. Phase 0 exit criterion.
test_conformance.py: existing graph-engine runtime tests still pass
for fixtures the legacy adapter can translate. Fixtures using the
v0.6 pair model or new node directives (fan_out, flaky variants,
calls_llm, update_pure*, emits_log, etc.) skip with phase-tagged
reasons.
Result: 204 pass, 7 skip, 0 fail.1 parent aedaf28 commit aa07c88
11 files changed
Lines changed: 1413 additions & 2 deletions
File tree
- tests/conformance
- harness
- runtime
Submodule openarmature-spec updated 122 files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
0 commit comments