aaron/custom llm providers#3953
Draft
aaronvg wants to merge 122 commits into
Draft
Conversation
Adds baml.errors.UnknownError (the universal escape-hatch wrapper: original
thrown value + breadcrumb) and the baml.errors.CallError classifier interface,
the foundation the new provider/capability interfaces throw on. Foreign errors
normalize at the catch site via an interface-match arm.
Note: the design's generic UnknownError.from<T>/with_message<T> helpers are
omitted — they need a runtime "is this value a monomorphized T?" test, which the
language does not support today (`v is T` on a generic param is always false;
`match { let t: T => }` is an irrefutable catch-all).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduces the marker + capability model:
- Provider — the bare composable marker (no interaction method)
- HttpProvider requires Provider — owns build_request/send/parse/meta_of, with
an associated `type Body` and generic call_with<T,V,E2>/derived call<T>
- CallResult<T,V> — the value+sidecar carrier (replaces the (T,V) tuple,
which is not yet a language feature)
- ResponseMeta — lazily-normalized out-of-band metadata
call_with threads the projection's error channel E2 (the Iterator.map pattern).
`prompt` is a plain string for now; PromptAst wiring is a later step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first concrete HttpProvider, written entirely in BAML (request/response logic in BAML via baml.json/baml.http; only leaf primitives stay host functions). Chat Completions endpoint, options as plain fields, base_url override for testing. Tests (crates/baml_tests/tests/ai_provider.rs): - openai_implements_capabilities — existential Provider matches HttpProvider - capability_error_normalization — known CallError passes; foreign boxes - openai_call_via_mock — full pipeline vs a wiremock endpoint - openai_live_call — real gpt-5.4-mini, gated on OPENAI_API_KEY Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Public wrapper over the internal non-streaming SAP entrypoint (plan P5: "expose it, don't rebuild it"), so a BAML-authored provider can decode an LLM's raw text into a typed T the same way the built-in orchestrator does — robust to markdown fences and minor JSON deviations, and a passthrough for T = string. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build_request<T> now renders T's output schema (baml.llm.render_output_format) and appends it to the prompt (empty for T = string, so text calls are unchanged); parse<T> decodes the response content with baml.sap.parse<T> instead of strict from_json, so `call<SomeClass>` yields a typed value. Tests: openai_structured_via_mock (fenced-JSON content decoded by SAP) and openai_structured_live_call (real gpt-5.4-mini returning a typed Person, gated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A real user-declared `client<llm> { provider openai }` + LLM `function` now
executes through the BAML-authored `baml.ai.OpenAi` provider, via orchestrator
delegation in `execute_once_oneshot` (openai only; other providers keep the
legacy Rust path). Bridge: a new `baml.llm.prompt_to_text(PromptAst) -> string`
host helper flattens the rendered prompt for the string-taking provider (roles
flattened for v1).
Tests: e2e_client_function_via_new_provider_{mock,live}. Full baml_src suite
(1987) + existing LLM tests stay green.
Deviations from the plan tracked in _plan/deviations.md (orchestrator delegation
vs client-as-sugar rewrite; prompt role flattening; two language findings:
`match` on a generic `T` / `unknown` is an irrefutable catch-all).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`baml.ai.Fallback` and `baml.ai.Retry` are plain classes implementing HttpProvider that override the primitive `call_with` (so `call` derives) and forward by a runtime `match` over their members. Fluent factories `with_retry`/`fallback_to` on the `Provider` marker (inherited by every provider). Also: `OpenAi.send` now errors on non-2xx responses (so retry has something to retry). Tests: fallback_routes_to_first_working_member, retry_recovers_after_transient_ failures, retry_exhausts_and_throws. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- `baml.ai.Streaming requires Provider` with `stream<TStream,TFinal>(prompt) -> baml.llm.Stream<TStream,TFinal>`; `baml.errors.StreamError`. - `OpenAi implements Streaming`: builds the streaming request in BAML + fetch_sse, reuses the existing SSE accumulator (leaf) via a shorthand-built PrimitiveClient. - `Fallback` routes .stream to the first working streaming member; `Retry` forwards once (streams aren't transparently retryable). Tests: openai_stream_via_mock (SSE deltas -> "pong"), openai_stream_live (real gpt-5.4-mini). All 13 ai_provider tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion) execute_once_stream now routes OpenAI through baml.ai.OpenAi.stream (mirrors the oneshot delegation); a user LLM function's `Foo$stream(...)` companion streams through the new provider. Test: e2e_function_stream_via_new_provider_mock. All 14 ai_provider + baml_src (1987) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/34)
- `baml.ai.Usage { input_tokens, output_tokens }`; `ResponseMeta.usage()` and
`finish_reason()` are now best-effort (`throws never`).
- `OpenAi`: `type Body = string` (read body once in `send` — `Response.text()` is
not idempotent, which double-read of parse+meta_of exposed); `OpenAiResponseMeta`
reads usage/finish_reason from that text.
- Metering rides `call_with(prompt, m => m.usage()) -> (value, Usage)`.
Test: call_with_projects_usage. All 15 ai_provider green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o endpoints Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- baml.ai.Tools requires Provider: opaque `type Transcript`, begin/step/submit, and a default `run_tools<T>(prompt, tools, dispatch)` loop (step returns `T | ToolCalls`, mirroring Iterator.next -> Item | Done). `baml.errors.ToolError`. - Types: Tool / ToolCall / ToolResult / ToolCalls. - `OpenAi implements Tools` — real OpenAI function-calling wire (tools in request, tool_calls parsed via json nav since `function` is a keyword; assistant turn preserved in `step`, tool results appended in `submit`; final via SAP). Deviation: `Tool.parameters` is an app-provided JSON-Schema string until a `type -> JSON Schema` host fn lands (see _plan/deviations.md). Tests: tools_loop_via_mock (2-turn), tools_loop_live (real gpt-5.4-mini weather agent). All 18 ai_provider green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cascade_escalates_on_low_confidence: cheap provider answers or signals ESCALATE, falling up to the expensive provider (cascade = Fallback-shaped; routing = a function returning a Provider). - baml.ai.RoundRobin: load-balances across members via a mutable cursor (round_robin_alternates_members). All 20 ai_provider green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…channels - baml.ai.Constrained requires Provider (no default: a token-level guarantee can't be faked). OpenAi does NOT implement it, so negotiation throws Unsupported at runtime — the honest B1 "capability is a runtime promise" story. - baml.errors.Unsupported now implements CallError/StreamError/ToolError so it is legal on any capability channel (per the design). Test: constrained_capability_absent_is_runtime_promise. All 21 ai_provider green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(scenarios 07/08) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Realtime/Channel/LiveControl interfaces + OpenAiRealtime (config-only client, pass-in channel; duplex transport stubbed pending P8) + RealtimeError. Capability negotiation tested (realtime provider matches Realtime/LiveControl; HTTP does not). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…,27,31,44) Conversational/Session, Compaction/Window, Branching, Chain/ChainHandle, MemoryStore, Background/Job, ManagedCache/CacheHandle, Suspendable/Suspend — interfaces + handles + negotiated usage examples. Persistence host surface stubbed (Family A gap); methods on the UnknownError channel for now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New stdlib capabilities (ns_ai): Realtime/Channel/LiveControl (+RealtimeError), stateful (Conversational/Session, Compaction, Branching, Chain, MemoryStore, Background/Job, ManagedCache, Suspendable), Capabilities/Support lattice (36), harness providers (ClaudeCode/PiAgent). Enriched ResponseMeta dims (reasoning/logprobs/citations) on OpenAi. Usage examples (crates/baml_tests/baml_src/ns_ai_examples/): tools family (10-16), workflows (43-47), cross-cutting (05,06,33,35), voice/security/searchable/obs (25,16,13,46). All compile as part of baml_src (bytecode-snapshotted). Coverage: every scenario 01-47 now has real, compiling BAML — full+live for the HTTP path (01-04,09,28-30,32,34), capability+negotiation shapes elsewhere (host transport/persistence stubbed pending P8/P6). Flags a MAJOR compiler limitation in _plan/deviations.md: cross-package `requires`-satisfaction is broken, so user-package classes can't yet implement the capability interfaces (providers must live in stdlib until fixed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ing code Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…media
Audit of the provider work found three real regressions on the e2e delegation
path, each now fixed with a request-capture regression test:
1. Double schema injection: OpenAi.build_request appended T's schema even when
the rendered template already carried ${ctx.output_format}. New
`append_output_schema: bool?` field (default true for standalone use); the
orchestrator delegation sets false, restoring legacy schema-iff-referenced
semantics. (e2e_structured_schema_injected_once)
2. Dropped client options: temperature/max_tokens (request_body) and user
headers never reached the wire. New `extra_body` / `extra_headers` fields on
OpenAi, forwarded by the delegation (_openai_from); query_params and
finish-reason lists fall back to the legacy builder (_openai_delegation_ok).
(e2e_options_and_headers_forwarded)
3. Dropped media (pre-existing upstream bug): the backtick prompt assembler
returned "" for media values, so ${img} never entered the PromptAst on ANY
path. assemble_prompt_ast_impl now emits PromptAstSimple::Media parts
(text-only prompts keep byte-identical shapes); new prompt_has_media host fn
routes media-bearing prompts to the media-aware legacy builder.
(e2e_media_prompt_reaches_wire)
Also: findings logged in _plan/deviations.md (quoted config keys silently
dropped; to_string<unknown> vs to_json<unknown>; json-alias match arms don't
bind on unknown), REALIZED.md live/tested labels corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…BAML The provider exchange type is now baml.ai.ChatMessage[] (role + interleaved text/media MessageParts); the whole chat-completions wire is BAML: - HttpProvider: call_messages_with/call_messages are THE primitives; call_with/call are single-user-message string sugar. build_request<T> takes messages. Streaming: stream_messages + stream sugar. Fallback/Retry/RoundRobin route the message primitives. - One leaf bridge: baml.ai.prompt_to_messages(PromptAst) -> ChatMessage[] (roles preserved, media interleaved). The e2e delegation uses it — roles are no longer flattened (e2e_roles_preserved_on_wire). - Media rides natively: BAML _encode_part emits image_url (url or data-URI), input_audio, and file(pdf) parts; the media fallback-to-legacy is removed. LIVE-proven: e2e_multimodal_live (gpt-5.4-mini reads an inline base64 image through the native pipeline). - MessagePart is a product type (unions don't cross host construction); ChatMessage.user/system/assistant + MessagePart.from_* constructors. New language findings in _plan/deviations.md: a `_` arm after a media-typed binding breaks the media runtime type-test (null-elimination works); cross-ns $rust_io_function returns need the classes' own namespace for owned codegen; media at the host boundary is the baml.media.* instance form. All 29 ai_provider tests + prompt-tag suites + baml_src (1987) green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
App-held transcript threaded through call_messages: mock asserts the full 4-message history reaches the wire; live test proves the model remembers a fact from an earlier turn (gpt-5.4-mini). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…projection Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ario 20)
Native BAML implementation of OpenAI's /v1/responses API:
- baml.ai.OpenAiResponses implements Provider + HttpProvider (input-shaped body,
output[].content[].output_text parse via json navigation + SAP) and
root.ai.Chain (extend<T> threads previous_response_id). start_chain mints
ChainHandle { id, owner: "openai-responses" }.
- Tests (tests/ai_responses.rs): mock call, mock chain (asserts turn 2 carries
previous_response_id), and responses_live_chain — the real model recalls a
fact across extend with only the handle re-sent. Scenario 20 LIVE.
New language finding (deviations): a class field named `type` crashes the
builtins codegen at Rust-generation time (like `function`, but later/obscurer);
workaround is json navigation for type-tagged envelopes.
Built by an opus-4.8 subagent; verified independently (3/3 ai_responses incl.
live, 32/32 ai_provider, baml_src 1987 green; ai_examples bytecode snapshot
churn is the new Provider implementor in is_type dispatch chains).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract$stream on a class-returning function: partials arrive and final() parses to the typed Person through the native message pipeline against gpt-5.4-mini. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ass rewrite Two ergonomics improvements driven by the ugliest code in the tree (the nested json.field walk in openai_responses.baml): 1. baml.json.path<T>(j, selector) — jq-style typed JSON accessor: .usage.input_tokens, .choices[0].message.content, ["quoted.key"] map access, [0] on a root array, "." identity. Throws JsonPathError (missing field, shape mismatch, OOB index, malformed selector, uncoercible leaf) instead of null. Pure BAML over json.field + json.to<T>. 17 unit tests (happy + every error path) in baml_src/ns_json_path/. 2. openai_responses.baml rewritten with typed wire classes using `kind string @alias("type")` decoded via baml.sap.parse — which honors @alias (baml.json.from_json does NOT: a finding). Kills the triple-nested match navigation; verified live (server-stored chain still recalls in prod). Also adds _plan/baml_gotchas.md — the consolidated field-notes reference (match/ runtime-type-test quirks, reserved names incl. `env` param shadowing, json/ serialization asymmetries, host-boundary rules, throws strictness). All green: 2004 baml_src tests (1987 + 17 new), 3/3 ai_responses (live chain), 33/33 ai_provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dlib classes BAML itself always allowed `type` as a field name (decl, construction, `.type` access all work in user code); only the builtins codegen crashed generating Rust identifiers from stdlib field names. Both generators now escape keyword-named fields as raw identifiers (`r#type`) — quote-based codegen_io.rs via a shared field_ident() helper (7 sites), string-based codegen.rs via rust_field_name() (view accessors, copy-struct decls, copy_field_to_value) — with a trailing- underscore fallback for the non-rawable set (self/Self/super/crate/_). String positions (panic messages, VM field lookups) keep the original name. Proof: the OpenAiResponses wire classes now use `type: string` directly (the `kind @alias("type")` workaround removed) — exact wire fidelity for the most common JSON key in existence. Verified live (server-stored chain still recalls in prod); 23/23 codegen unit tests, 3/3 ai_responses, 33/33 ai_provider, 2004 baml_src tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ments Background
The Responses API's background mode maps directly onto the Background capability:
submit<T> POSTs with "background": true (+ caller-owned Idempotency-Key header,
design D2) and returns Job<T>{id, owner}; poll<T> GETs /responses/{id}, yields
null while queued/in_progress, and SAP-parses the typed value once completed.
Tests: responses_background_via_mock (queued -> pending -> value lifecycle) and
responses_background_live — a real server-side job submitted, polled, and
completed against gpt-5.4-mini. Scenario 27 LIVE (9 scenarios now live).
Also refreshes the stale Deferred section in _plan/implementation-checklist.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rror, backoff - baml.errors.Failure: the cross-capability decision axis (is_retryable/ is_effectful/is_policy_refusal/is_resumable/is_unsupported). OPT-IN for now: the plan's `CallError requires Failure` is blocked by the E0125 cross-package requires bug (sharper repro found: a user class implementing BOTH interfaces in-body still fails the check), which would break all user-defined errors. - Provider.is_effectful (default false); OpenAiResponses overrides true (server-stored state + billed jobs). - Retry: refuses effectful providers with typed baml.errors.CannotRetry (refuse-then-error, never silent double-drive); consults Failure.is_retryable on typed errors (400 -> no retry; 429/5xx -> retry); exponential backoff; unknown errors keep legacy assume-transient behavior. - OpenAi.send throws typed OpenAiHttpError (CallError+StreamError+Failure; is_rate_limit on 429) instead of UnknownError — interface catch arms are now reachable. Tests: retry_skips_non_retryable_errors (single request on 400, typed catch), retry_refuses_effectful_provider (CannotRetry). 35/35 ai_provider, 5/5 ai_responses green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d by scenario tests DCP §1.6 (BAML-native preference): the pure-VM baml_test! tests in ai_provider.rs whose coverage now lives in the scenario suite are deleted, each with a named replacement: - capability_error_normalization -> 01/03/04 typed-Unsupported tests - retry_exhausts_and_throws -> NEW 29 'Retry exhausts its budget…' (migrated) - constrained_capability_absent_is_runtime_promise -> 03 - realtime_capability_negotiation -> 22/23 - stateful_capabilities_negotiation -> 17/19/21/27/44 - retry_refuses_effectful_provider -> 10 'retry refuses…' openai_implements_capabilities stays (pins the native class's full capability set — distinct value). Wiremock/request-capture and live tests stay in Rust by design. ai_provider 38 green; 29's suite 3 green; lib 1768 + bytecode regen green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n Retry/Fallback - Retry.base_delay_ms (default 100), doubling per attempt — restores legacy exponential backoff with a single knob. - Retry and Fallback now drive members with an identity projection and apply the caller's project exactly once to the winning result: a throwing E2 no longer re-drives (re-bills) the call or triggers failover. - Call-counting tests in ns_ai_scenarios/29_reliability pin both behaviors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LLM function bodies now compile to the same shape every companion has: baml.ai.drive_call<T>(client, Foo$render_prompt(args…, client = client)) and $stream to drive_stream<Partial<T>, T>(…) via make_drive_companion (explicit type args — closes the stream-path inferred-type-args hole). The backtick prompt closure lives only in the $render_prompt/$build_request* companions; prompt diagnostics ride the first companion build. - Strategy configs lower at declaration: fallback → baml.ai.Fallback, round-robin → baml.ai.RoundRobin (start → counter), retry_policy → baml.ai.Retry wrap (max_retries → max, initial_delay_ms → base_delay_ms; RetryPolicy delay knobs now optional). - The legacy Client bridge owns routing: call_messages_with override routes ported primitives (openai/anthropic/google-ai) onto native classes; new `implements Streaming for Client` (native route → legacy SSE codec) closes Phase B's open bridge-streaming item. Retry gained stream-initiation retry with backoff; RoundRobin gained Streaming. - llm.render_prompt renders native providers UNSPECIALIZED (the Anthropic system-hoist 400 fix now covers every native-override path). - call_llm_function/stream_llm_function are 1-line drive delegators; the orchestrator (execute_*, build_plan/build_attempt, PlannerState, OrchestrationStep, ExecutionContext) and the ignored orchestration.rs are deleted. - Host: render_output_format returns "" for top-level BuiltinUnknown (inference-only T at direct call_llm_function call sites). Gate: lib 1768 / baml_src 2107 green; all ai_* wiremock + live suites (OpenAI, Anthropic incl. streaming, Gemini, Responses, strict), live integ-test* testsets 3/3, bex_engine render/request suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…describe) Every LLM function body/stream companion changed shape, and the stdlib lost the orchestrator — bytecode listings, __baml_std__ HIR/TIR/MIR/codegen, per-project compiles snapshots, TIR file snapshots, and the baml-cli describe listings all churn accordingly. Accepted after the full functional + live gate passed (user rule: snapshots last). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… combinator
- baml.ai.AggregateMeta { parts }: a ResponseMeta over several member calls —
usage() sums parts, finish_reason() is the last part's, per-part dimensions
stay readable on .parts.
- baml.ai.UsageMeter (reference-shared accumulator: record/calls/total/
aggregate/wrap) + Traced combinator (transparent recording forward; the
caller's projection still sees the original winner meta) + fluent
Provider.traced(meter).
- 4 offline tests in ns_ai_scenarios/34_cost_tokens: aggregate summing,
traced transparency, meter.wrap over fallback members (only the winner is
observable), one meter across round-robin members.
Residue (deviations.md): failed attempts throw without meta (billed-but-
failed calls invisible); Tools exposes no per-turn meta so ToolLoop can't sum
turns; Retry/Fallback keep projecting the winner's meta unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ments Streaming - sys_llm stream accumulator accepts google-ai: parses the chunked GenerateContentResponse SSE shape (candidates[0].content.parts[*].text concat, finishReason ends the stream — no [DONE] sentinel, cumulative usageMetadata, modelVersion). - baml.ai.Gemini implements Streaming via :streamGenerateContent?alt=sse — identical body to the oneshot path, only the URL suffix differs; Stream wraps the host accumulator via a google-ai shorthand primitive. - Declared `provider google-ai` clients now stream natively through the bridge (the C.3 native-route arm no longer falls through to legacy). - Tests: accumulator unit tests (google-ai supported + gemini accumulation), gemini_stream_via_mock (wiremock SSE), gemini_stream_live (proven against the real API). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…se_url
baml.ai.OpenAiCompatible { model, base_url (required), auth, … }: the Chat
Completions wire shape against proxies, gateways, and local runtimes (vLLM,
llama.cpp server, LM Studio, Ollama /v1). The request/parse pipeline is
DELEGATED to an internal OpenAi value so the two providers cannot drift;
what differs is the required base_url and the TYPED auth field —
BearerAuth { token } | HeaderAuth { name, value } | NoAuth {}.
- implements Streaming ("stream": true) over the openai-generic host
accumulator (same delta shape as OpenAI).
- native_provider_for maps legacy `provider openai-generic` primitives
(with base_url + model) onto the native class: api_key -> Bearer,
keyless -> NoAuth; configs without base_url/model keep the legacy builder.
- 6 wiremock tests (ai_openai_compatible.rs): bearer/header/no-auth request
capture, streaming, typed 429 (OpenAiHttpError), and declared-client
native routing e2e.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, sap.parse_type - baml.ai.invoke_tool<A>(call, handler): the CANONICAL inbound dispatch step — SAP-coerce ToolCall.args into the handler's declared type; a mismatch returns an error ToolResult (the model self-corrects next turn) instead of aborting the loop. - Tool.params_type: type? (set by from_type, null for hand-built wire schemas) + Tool.validate_args(call): the optional integrity check against the STORED source type, so the advertised schema and dispatch coercion cannot silently drift. - baml.sap.parse_type(t: type, raw) -> unknown: SAP against a RUNTIME type value — pure BAML over StreamCache.new(t, t) (no new host surface; the cache already carries runtime types). - Finding (gotchas): SAP's single-string-field passthrough swallows nearly any JSON, so schema validation only discriminates on multi-field shapes — tests use a two-field args class. - 4 offline tests in ns_ai_scenarios/09_tool_calling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r helpers
baml.json.path_or(j, selector, default) (T infers from the default): a
missing field, shape mismatch, OOB index, malformed selector, or uncoercible
leaf yields the default instead of throwing — the "field with default" idiom
(Python d.get(k, default), serde .unwrap_or, jq // default) as path's
throws-never sibling; path stays strict for callers who want misses surfaced.
- The OpenAI tools decode and realtime event reads use it directly
(".function.name" nested selectors replace the two-step field chains);
the private _openai_json_str / _realtime_json_str coalescers are deleted.
- Type-arg inference from the default argument is verified end-to-end at
runtime (probe + dedicated test) — call sites need no explicit <T>.
- 6 unit tests in ns_json_path (hit, miss, mismatch, bad selector, nested,
inferred).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"meta_of" read as a getter; the method is the metadata half of the response codec — `parse` decodes the value, `parse_meta` decodes the ResponseMeta sidecar from the same Body. Mechanical rename across the interface, all providers, the legacy bridge, combinators, ToolLoop, and test fakes (28 sites); snapshots regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tools.run_tools, drive_run_tools, and the generated Foo$run_tools now take stop_when: ((LoopInfo) -> bool throws never)? = null and return T | Budget<T>: a budget hit is a first-class match arm, not a control signal masquerading as a ToolError (plan D5 — "run_to_budget becomes the default, not a bolt-on"). Refusals and transport failures stay on throws; the plain-call path (ToolLoop.call_messages_with) still throws LoopBudgetExceeded because an HttpProvider call must produce T. Compiler: DriveCompanionSpec.null_defaulted_extras — companion extras with a null default allocated into the cloned FunctionDefaults arena (the append_default_client_param pattern) and passed by name (E0005). $run_tools grows stop_when and the union return. Migration (intended breaking change): 8 corpus call sites now match the sum; new budget-arm tests in scenarios 09/10 (interface + companion); live tools agents (loop, typed, multi-tool, handoff) re-proven against the real API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… arm
A tool marked handoff: true is a TRANSFER, not a dispatch: the loop ends and
returns HandoffRequest { to, args, steps_taken } — routing is the caller's
decision (plan D5: no more handoff smuggled through dispatch/submit).
- Tool.handoff: bool?; run_tools / drive_run_tools / Foo$run_tools /
ToolLoop.run_to_budget return T | Budget<T> | HandoffRequest.
- The plain-call path (LLM function with a ToolLoop client) must produce T,
so an unresolved handoff there is the typed HandoffUnresolved error
(resumable=true, effectful=true — mirrors LoopBudgetExceeded structurally).
- Tests in scenario 14: HandoffRequest arm via run_to_budget and the
interface loop; typed throw via the plain LLM-function path (design-true
spelling per DCP §1.6). Exhaustive matches in 09/10 grew the third arm.
- Live tools agents re-proven against the real API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-query' into aaron/custom-llm-providers # Conflicts: # baml_language/crates/baml_compiler2_mir/src/lower.rs # baml_language/crates/baml_compiler2_tir/src/normalize.rs
…data
The pkg-alias-query perf merge rebuilt package_lowering_data from
dependencies + self only, dropping the open-world implementor union — an
interface match compiled inside a dependency package (stdlib drive_call's
capability negotiation) could no longer see user-package implementors, so
EVERY LLM function called with a user-authored provider threw
Unsupported ("client's provider supports neither HttpProvider nor
Streaming"); 54 corpus tests failed once new user providers entered the
session.
Restores the original block: implementor relations union across all session
packages (sorted, push-unique); field/enum schema maps stay package-scoped
via scratch outputs. Adds stdlib_match_sees_user_package_implementors as a
minimal fast regression test so the next refactor cannot drop this silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-merge drop) The pkg-alias-query perf merge also deleted the implements_for-loop branch that registers OUT-OF-BODY `implements Iface for ConcreteClass` blocks into interface_implementors — so any class whose implements block lives in a DIFFERENT file than its class declaration was invisible to interface match/catch arms (the in-file class-def loop masked the same-file case). Concretely: `Unsupported implements StreamError` (errors.baml vs capability.baml) vanished from StreamError's implementor set, breaking the documented catch-by-channel-interface triage idiom and any interface-as- union-member match over cross-file implementors. Restores the registration (register_class_for_interface_closure, push- unique) for class targets in both the package pass and the session-wide union pass. Three regression tests in interfaces.rs pin the family: interface-as-union-member over a cross-file implementor, catch-by-channel- interface over a thrown concrete error, and the in-body sibling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r the original corpus Each ns_ai_scenarios/NN_* rewritten (or created: 05, 06, 08) to match llm-provider/ideas/scenarios/NN-* as closely as the real surface allows, per-scenario reports collected for the ALIGNMENT.md comparison: - 01-03: implementation.baml added (design's custom providers/shapes). - 05 multimodal-input, 06 non-text-output: built fresh (media pipeline shapes, SaveTo sidecar-forwarding combinator, typed Refused; media persistence BLOCKED P8). - 07 reasoning: reasoning-content kinds, external-impl meta dimensions, Continuity capability + cross-provider guard; 11 offline tests. - 08 enriched-outputs: Cited/Grounded/Annotated dimensions, choice scoring with typed LowConfidence, Fallback P1-win; 10 offline tests. - 09 tool-calling: design's declare-once/typed-args/approval-gate/resilient -tools story on ToolLoop + $run_tools; D5/D6 behaviors preserved; 15 tests. Corpus: 2204/2204 offline green on the fixed compiler (50s — the perf merge's speedup holds now that its two dropped registrations are restored). Bug queue from the sweep logged in the checklist (companion user-class extras crash remains open). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lel tools, 12 taxonomy
- 10: Bounded -> real ToolLoop mapping; plain-LLM-function-first spellings
(ResearchIt + ToolLoop client); shared fixtures (ScriptedTools/
echo_dispatch/lookup_tools) moved to implementation.baml byte-stable;
9 tests; D7 per_turn_tools gap recorded.
- 11: the design's 5 Dispatcher strategies as closure factories over real
BEP-034 concurrency (spawn + future.all + TaskGroup rate cap); order
preservation, partial-failure survival, effect-aware fan/sequence; 9 tests.
- 12: execution-location taxonomy as three routes — client-executed
(dispatch), handoff-shaped (Tool{handoff:true} -> HandoffRequest /
HandoffUnresolved), provider-executed (BLOCKED P8, observe-only); 8 tests
incl. a mixed-kinds loop.
Corpus: 2218/2218 offline green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ulti-agent, 15 guardrails - 13: the meta-tool pattern — tool_search as a plain client tool, catalog- backed dispatch pages definitions in as ToolResults; hosted deferral/MCP federation BLOCKED (no wire seam / D7 set_tools / P8); 6 tests. - 14: both handoff stories — delegation (specialist-in-dispatch) AND the D5-honest HandoffRequest arm with a caller-side Router + roster, depth cap (s14_HandoffDepthExceeded) and unknown target (s14_NoSuchHandoffTarget); 8 tests, 3 preserved D5 tests verbatim. - 15: guards as wrapping providers (Traced pattern), typed tripwires on the Failure axis, ModelGuard judge + CodeGuard marker; 11 tests. New bugs queued from the sweep: spawn-in-closure+await VM deadlock (repro in 15/implementation.baml §6); opaque associated-type forwarding gap. New gotchas: defer reserved; meta interfaces need requires ResponseMeta. Corpus: 2235/2235 offline green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every test body now races a deadline inside the stdlib runner (testing/registry.baml run_test): a hung test — stalled network read, deadlocked await — becomes a FAIL with "test timed out after Nms" instead of hanging the whole run. baml.future.race cancels the loser on settle (BEP-034), so a timed-out body is actively cancelled and a finished test's timer doesn't linger. Default 300000ms; override with BAML_TEST_TIMEOUT_MS. Verified: a 60s-sleep test fails at a 2s override while the suite exits promptly; full offline corpus unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… 18 compaction + live-tier fixes - 16: Tainted<T> provenance (viral map/combine), Quarantined capability- subtraction wrapper, allowlist + tainted-input gating with typed refusals, the caller-obligation residue proven as tests; 14 tests. Finding: user interfaces can't runtime-negotiate off the Provider existential without the full capability-registry route (confirms design OQ5). - 17: both memory models — client-held ChatMessage[] threading (incl. the forgot-the-assistant-turn bug as a driver) + Conversational sessions (persistence BLOCKED P8); 7 tests. - 18: summarize-and-replace compaction where the summarizer IS an LLM function; Compaction capability shape+negotiation (P8); 4 tests. - Live-tier fixes from the first full keyed sweep (22->25/25): 01 asserts routing not model obedience (+log.info), 02 GEMINI_API_KEY -> GOOGLE_API_KEY, 06 drops response_format (gpt-image-1 rejects the DALL-E-only param) + log.warn/info observability. Gates: offline corpus 2255/2255; live integ sweep 25/25 under the 60s per-test timeout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6fb3b3c. Configure here.
| throws root.errors.RealtimeError | root.errors.UnknownError { | ||
| let text = root.llm.prompt_to_text(prompt); | ||
| match (client) { | ||
| let r: Realtime => r.run(text, io), |
There was a problem hiding this comment.
Live driver flattens prompts
Medium Severity
drive_live converts the rendered PromptAst with root.llm.prompt_to_text before Realtime.run, while companion drivers such as drive_call and drive_stream use prompt_to_messages. That drops role structure and multimodal parts, so Foo$live can send a flattened string unlike the native chat path used elsewhere.
Reviewed by Cursor Bugbot for commit 6fb3b3c. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
High Risk
Large compiler lowering/MIR and LLM execution-path changes (desugar, bridge, orchestrator removal) affect every LLM call; plus auth/network (rustls WebSocket) and broad stdlib API shifts with intentional breaking companion/
run_toolsshapes.Overview
Replaces the closed Rust provider enum and orchestrator special-cases with a marker + capability model in stdlib:
Provider,HttpProvider(nativeChatMessage[]wire),Streaming,Tools, combinators, and multiple BAML-native providers (OpenAI, Anthropic, Gemini, Responses, Realtime, OpenAI-compatible), plus a legacyClientbridge for unported configs.Compiler / runtime wiring: Parses
//baml:llm_capabilityand//baml:llm_companion(<suffix>), builds an HIR capability registry, validates E0150–E0152, and desugars LLM functions todrive_call/ generated$stream,$with,$run_tools,$livecompanions (user-package drivers included). Injected LLM param isclient: baml.ai.Provider(overrides, combinators, user provider classes). Strategy clients lower toFallback/Retry/RoundRobin; the old planner/orchestrator path is removed in favor of capabilitymatch+ bridge routing. MIR fixes restore session-wide interface implementors and out-of-bodyimplementsfor negotiation.Surface additions: Public
baml.sap.parse,baml.json.path/path_or,baml.schema.json_schema,baml.ws(rustls-backed WebSocket), D2/D8 Failure axis, D5T | Budget<T>/ handoff sums, D6invoke_tool/sap.parse_type, D4 UsageMeter/Traced, Gemini streaming,HttpProvider.parse_metarename. Tests move tons_ai_scenarios/+ gatedinteg-test*; large_plan/docs and checklist track DCP completion. Tooling deps add rayon; compile/check perf work is included.Reviewed by Cursor Bugbot for commit 6fb3b3c. Bugbot is set up for automated code reviews on this repo. Configure here.