Skip to content

Commit aa07c88

Browse files
test: phase 0 conformance harness — typed parser for all 68 fixtures (#7)
Phase 0 of the v0.4 → v0.8 implementation plan: the conformance test harness loads every YAML fixture under openarmature-spec/spec/<capability>/conformance/ into a typed pydantic config. Phases 1+ add runtime interpretation under harness/runtime/ without re-touching parsing. Bumps the spec submodule to v0.8.0 (8 accepted proposals: foundation, explicit subgraph mapping, observer hooks, llm-provider, middleware, fan-out, OTel, checkpointing). Harness package layout (tests/conformance/harness/): - fixtures.py Three top-level discriminated variants (LlmProviderFixture | CasesFixture | GraphFixture). Discriminator picks by presence of `mock_provider` / `cases:` keys. - directives.py Node directive sub-models with mutual-exclusion validator on the 13 primary directives. Middleware discriminated union over the 5 types in fixtures. - expectations.py Per-capability expected-block discriminator (graph_engine | llm_provider | pipeline_utilities | observability) chosen by which assertion keys appear. - loader.py Auto-discovers NNN-*.yaml across all four capability directories. - skip.py Structured SkipReason with capability + directive list + phase mapping. - runtime/ Empty stub package + README locking in the Phase 0 boundary. Strictness contract: strict (extra="forbid") at the structural skeleton (top-level fixture, StateSchema, NodeSpec primary directive set, ObserverSpec, MiddlewareConfig). Permissive (extra="allow") for payload-shape models (LLM mock responses, middleware params, flaky config, case shapes) — those evolve frequently in the spec without restructuring the directive surface. Tests added (test_fixture_parsing.py): every fixture parses (68) AND round-trips (parse → model_dump → re-parse → equal). 137 assertions total. Phase 0 exit criterion. test_conformance.py: existing graph-engine runtime tests still pass for fixtures the legacy adapter can translate. Fixtures using the v0.6 pair model or new node directives (fan_out, flaky variants, calls_llm, update_pure*, emits_log, etc.) skip with phase-tagged reasons. Result: 204 pass, 7 skip, 0 fail.
1 parent aedaf28 commit aa07c88

11 files changed

Lines changed: 1413 additions & 2 deletions

File tree

openarmature-spec

Submodule openarmature-spec updated 122 files
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
"""Conformance fixture harness — typed parsing for the four spec capabilities.
2+
3+
Phase 0 (per the implementation plan): every fixture under
4+
``openarmature-spec/spec/<capability>/conformance/`` lands as a typed pydantic
5+
config. Phases 1–6 add runtime interpretation under ``harness/runtime/``;
6+
they never re-touch parsing.
7+
8+
Public surface:
9+
10+
- :func:`loader.load_fixture` — parse one YAML path into a typed fixture.
11+
- :func:`loader.discover_fixtures` — auto-discover fixture paths across the
12+
four capability directories on the spec submodule.
13+
- :class:`fixtures.Fixture` — the root discriminated union
14+
(``LlmProviderFixture | CasesFixture | GraphFixture``).
15+
- :class:`skip.SkipReason` — structured "fixture needs directives X, current
16+
phase doesn't support them" used by the test runner to skip cleanly.
17+
"""
18+
19+
from .fixtures import (
20+
CasesFixture,
21+
Fixture,
22+
GraphFixture,
23+
LlmProviderFixture,
24+
)
25+
from .loader import discover_fixtures, load_fixture
26+
from .skip import SkipReason
27+
28+
__all__ = [
29+
"CasesFixture",
30+
"Fixture",
31+
"GraphFixture",
32+
"LlmProviderFixture",
33+
"SkipReason",
34+
"discover_fixtures",
35+
"load_fixture",
36+
]

0 commit comments

Comments
 (0)