Skip to content

[bot] Anthropic tracer does not capture output_config request parameter (structured outputs, effort) #124

@braintrust-bot

Description

@braintrust-bot

Summary

The Anthropic Messages tracer does not capture the output_config request parameter in span metadata. This parameter controls two key generation behaviors: structured output format (JSON schema constraints) and output effort level. The equivalent OpenAI parameters (response_format and reasoning_effort) are both captured by the OpenAI Chat Completions tracer.

What is missing

1. output_config absent from request metadata fields

In trace/contrib/anthropic/messages.go (lines 51–66), the metadata fields list is:

metadataFields := []string{
    "model",
    "max_tokens",
    "temperature",
    "top_p",
    "top_k",
    "stop_sequences",
    "stream",
    "tools",
    "tool_choice",
    "metadata",
    "container",
    "mcp_servers",
    "service_tier",
    "thinking",
}

output_config is not included. When a user sends a request with structured output constraints or effort configuration, these parameters are silently dropped from the trace metadata.

2. output_config has two subfields that affect generation

From the Anthropic Messages API docs:

"output_config": {
  "effort": "high",
  "format": {
    "type": "json_schema",
    "schema": { ... }
  }
}
  • effort — controls output generation effort level ("low", "medium", "high", "max"), analogous to OpenAI's reasoning_effort
  • format — constrains output to a JSON schema, analogous to OpenAI's response_format

Both are generation execution parameters that directly affect model output quality and format.

3. OpenAI captures both equivalent parameters

The OpenAI Chat Completions tracer (trace/contrib/openai/chatcompletions.go, lines 51–76) captures both:

  • response_format — structured output configuration
  • reasoning_effort — effort level for reasoning models

This creates a parity gap: users tracing OpenAI calls can see structured output config in their traces, but users tracing Anthropic calls cannot.

Braintrust docs status

Braintrust's Anthropic integration docs mention prompt caching and tool calls but do not mention structured outputs specifically. Status: not_found for Anthropic structured output tracing.

Upstream sources

Braintrust docs sources

Local repo files inspected

  • trace/contrib/anthropic/messages.go — request metadata fields (lines 51–66): output_config absent
  • trace/contrib/anthropic/traceanthropic.go — router and middleware setup
  • trace/contrib/openai/chatcompletions.go — reference: captures response_format (line 61) and reasoning_effort (line 60)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions