Add 0083 token-budget WARNING surfaces#237
Merged
Merged
Conversation
Add the two vendor-neutral over-budget WARNING surfaces from proposal 0083, on top of the completion-path signal. - observability: lift the shared _token_budget_evaluations helper to llm_event.py for both observers. - langfuse: on a budget exceedance the success Generation ends at level WARNING with a statusMessage naming the breached bound(s), and metadata.token_budget carries the declared bounds. A hard ERROR still takes precedence on the failure path. - otel: emit one section 7 WARNING log per over-budget attempt on the openarmature.observability logger, dispatched independently of the span (disable_llm_spans) and metric (enable_metrics) gates. The record correlates via correlation_id (no active span at emit time, so no trace_id/span_id). - conformance: un-defer fixture 130 (Langfuse WARNING). The structured-output-failure path (131) and the implemented flip follow in PR C.
There was a problem hiding this comment.
Pull request overview
This PR extends OpenArmature’s proposal 0083 token-budget observability by adding two vendor-neutral “over budget” WARNING surfaces (Langfuse Generation level/statusMessage and a spec §7 warning log), and centralizes shared per-bound evaluation logic so observers cannot drift.
Changes:
- Add §7 token-budget WARNING logging on
openarmature.observability, emitted once per over-budget attempt and independent of span/metric gates. - Add Langfuse advisory WARNING mapping for successful completions that exceed declared token budget bounds, including
statusMessageformatting andmetadata.token_budget. - Move the shared token-budget per-bound evaluation helper into
observability/llm_event.pyand enable conformance fixture 130.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_observability_otel.py | Adds unit coverage for the §7 warning log surface (cardinality, formatting, gating independence, prompt identity, failure/no-usage cases). |
| tests/unit/test_observability_langfuse.py | Adds unit coverage for Langfuse WARNING level/statusMessage and token budget metadata behavior on success and failure. |
| tests/conformance/test_observability_langfuse.py | Wires token_budget fixture data into the conformance harness prompt objects and enables fixture 130. |
| tests/conformance/test_fixture_parsing.py | Removes the “not implemented” deferral for fixture 130 while keeping 131 deferred. |
| src/openarmature/observability/otel/observer.py | Implements the §7 warning log surface and switches to the shared token-budget evaluation helper. |
| src/openarmature/observability/llm_event.py | Introduces the shared _token_budget_evaluations helper for consistent per-bound evaluation across observers. |
| src/openarmature/observability/langfuse/observer.py | Implements Langfuse Generation WARNING level/statusMessage on budget breach and maps metadata.token_budget. |
| conformance.toml | Updates proposal 0083 notes to reflect fixture 130 now running while 131 remains deferred. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of three PRs for proposal 0083 (per-prompt token-budget observability). Adds the two vendor-neutral over-budget WARNING surfaces on top of PR A's completion-path signal. The structured-output-failure path (fixture 131) and the
implementedflip follow in PR C.What it adds
level = "WARNING"with astatusMessagenaming each breached bound (e.g.token budget exceeded: input 20 > 10), andmetadata.token_budgetcarries the declared bounds. A hardERRORstill takes precedence on the failure path (the advisory WARNING only applies to a successful call).openarmature.observabilitylogger, naming the prompt and the breached bound(s). It is its own surface, dispatched independently of the span (disable_llm_spans) and metric (enable_metrics) gates. The record correlates viacorrelation_id; it does not carry OTeltrace_id/span_id, since OA's observer never activates its spans and the record is emitted outside any span context.observability/llm_event.pyso both observers share one implementation.Conformance
Un-defers fixture 130 (the Langfuse WARNING). The §7 log is prose-pinned (no fixture) and covered by unit tests.
conformance.tomlstaysnot-yetuntil PR C lands the failure path.Tests
Fixture 130 plus unit coverage for: the Langfuse WARNING (over/under budget, both-bounds statusMessage,
metadata.token_budgeton success and on a failed Generation) and the §7 log (prompt identity, multi-bound formatting, per-attempt cardinality, the structured-output-failure and no-usage cases, and independence from both the span and metric gates). Full suite green; ruff and pyright clean.