Skip to content

Commit e8906da

Browse files
Activate proposal 0049 conformance fixtures 050-056 (#139)
* Activate proposal 0049 conformance fixtures 050-056 Adds harness support for the typed_event_collector directive introduced by proposal 0049's seven conformance fixtures (050-056). The directive declares one or more typed event collectors with optional filter_event_type and include_caller_metadata flags; the parser aggregates the latter into the provider's populate_caller_metadata kwarg. Adds eight assertion shape handlers consumed by the fixtures: contains_event, contains_exactly_one_event_of_type, contains_event_of_type, contains_exactly_n_events_of_type, does_not_contain_event_of_type, captured_event_field_values_cover, every_captured_event_has, relative_order_of_events_matching. The captured_event_field_values_cover shape uses set equality (matching the spec text's "cover" semantic) rather than sorted-by-str (which silently masks None-vs-primitive ordering ambiguity). An unknown-key guard rejects fixture typos before any assertion runs. Three graph builders cover the fixture topologies: a simple entry-ask-END shape (050, 051, 052, 053, 056), a fan-out builder (054), and a parallel-branches builder (055). The failure-path runner (053) sources the calling-node NodeEvent from an unfiltered named collector when one exists; otherwise attaches an AllEventsCollector specifically for that assertion. The cause-chain walk handles the NodeException wrapper the engine layers over the underlying ProviderUnavailable. Adds tests/conformance/test_typed_event_harness.py with 31 unit tests covering the harness helpers' edge cases: typo detection in assertion keys, set-equality semantics, namespace tuple-vs-list normalization, usage Mapping-vs-Usage comparison, cause-chain walking, populate_caller_metadata aggregation, and the mock_model resolution. These would otherwise only surface when a future fixture trips them. Flips conformance.toml's [proposals."0049"] from not-yet to implemented since 0.13.0. The leading-comment block notes that the OTel + Langfuse observers continue driving their span / generation surface off the sentinel NodeEvent pair during the dual-emit transition window; type-discrimination migration of those observers waits on the request-side-fields extension currently being drafted. * Address PR review feedback on typed-event harness Two distinct issues across five review threads: 1. Three sites used a truthy check on caller_metadata (if metadata:) that collapsed an explicit empty mapping (caller_metadata: {}) to the "no metadata key" case and dropped the metadata= kwarg from graph.invoke(). Switched all three sites (_invoke_typed_fixture, _run_typed_event_fanout_case, _run_typed_event_branches_case) to is-not-None checks so empty mappings flow through to the engine. 2. Two harness helpers (_event_fields_match and the every_captured_event_has branch in _assert_observer_expectations) used getattr(event, name, None), conflating a missing attribute with an attribute present and set to None. A fixture-side field-name typo (node_nam vs node_name) would silently match against any None expectation. Added hasattr() preambles that raise AssertionError naming the missing field, since upstream type filtering guarantees attribute presence on the captured set. Two new harness unit tests pin the missing-attribute contract: a typo'd field on _event_fields_match raises a clear error, and the every_captured_event_has branch does the same.
1 parent 5e98f4e commit e8906da

3 files changed

Lines changed: 1321 additions & 3 deletions

File tree

conformance.toml

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -295,10 +295,19 @@ status = "implemented"
295295
since = "0.12.0"
296296

297297
# Spec v0.41.0 (proposal 0049). Typed LLM Completion Event — first
298-
# typed event variant on the observer event union. Queued for
299-
# v0.13.0 alongside 0047.
298+
# typed event variant on the observer event union. Shipped in
299+
# v0.13.0: provider dual-emits the typed event alongside the sentinel
300+
# NodeEvent pair (success-only per spec scope); LlmCompletionEvent
301+
# carries identity/scoping/outcome fields per the spec field table.
302+
# Conformance fixtures 050-056 activated by the typed_event_collector
303+
# harness directive. The OTel + Langfuse observers continue to drive
304+
# their §5.5 / §8.4.4 surface off the sentinel NodeEvent pair during
305+
# the dual-emit transition window; type-discrimination migration
306+
# lands once the follow-on request-side-fields extension (proposal
307+
# 0057) ships.
300308
[proposals."0049"]
301-
status = "not-yet"
309+
status = "implemented"
310+
since = "0.13.0"
302311

303312
# Spec v0.42.0 (proposal 0050). Retry & degradation primitives —
304313
# failure-isolation middleware + call-level retry. Queued for

0 commit comments

Comments
 (0)