AWS Bedrock/Strands Instrumentation

[JWinermaSplunk]

Comparison: Upstream Bedrock Instrumentation vs. Strands Instrumentation

Architecture & Approach

Upstream Bedrock Instrumentation (opentelemetry-instrumentation-botocore):

• Instruments AWS SDK (botocore) at the HTTP/API call level
• Uses monkey-patching on boto3 client methods
• Creates CLIENT spans for all Bedrock API operations (InvokeModel, Converse, InvokeAgent, etc.)
• Framework-agnostic — works with any boto3 usage

Strands Instrumentation (splunk-otel-instrumentation-strands):

• Instruments Strands Agents SDK and Bedrock AgentCore at the framework/application level
• Uses a hybrid approach:
  • Strands hooks (BeforeModelCallEvent, AfterModelCallEvent, BeforeToolCallEvent, AfterToolCallEvent) for LLM and tool call telemetry
  • Wrapt monkey-patching for agent lifecycle (Agent.__init__, Agent.invoke_async) and AgentCore component methods
• Creates semantic GenAI spans (AgentInvocation, LLMInvocation, ToolCall, RetrievalInvocation, Workflow)
• Framework-specific — requires Strands SDK or Bedrock AgentCore components

What Each Captures

Upstream Bedrock:

• AWS API calls (bedrock-runtime.InvokeModel, bedrock-runtime.Converse, etc.)
• HTTP request/response metadata (region, endpoint, status codes)
• Raw API payloads (if body capture enabled)
• AWS-level errors (throttling, quotas, service errors)

Strands Instrumentation:

• Agent orchestration: name, model, tools list, system prompt, input/output messages
• LLM interactions: input messages (multi-modal content blocks supported), output messages, stop reason, token usage
• Tool executions: tool name, toolUseId, structured JSON arguments, result
• Memory operations:
  • retrieve_memories → RetrievalInvocation (query, top_k, documents retrieved count)
  • create_event, create_blob_event, list_events → ToolCall spans
• Code Interpreter sessions: start/stop, code execution, package installation, file uploads
• Browser automation sessions: start/stop, take/release control, session status
• BedrockAgentCoreApp entrypoint: root Workflow span per request (sync and async)

Not captured (not available in current SDK):

• Gateway and ServerTools — no callable instrumentation surface in bedrock-agentcore package
• Strands built-in tracer spans (suppressed by default via OTEL_INSTRUMENTATION_STRANDS_SUPPRESS_BUILTIN_TRACER=true)

Span Hierarchy

BedrockAgentCoreApp.entrypoint  (Workflow)
└── Agent: <name>               (AgentInvocation)
    ├── chat <model-id>         (LLMInvocation)  ← via Strands BeforeModelCallEvent
    └── <tool-name>             (ToolCall)        ← via Strands BeforeToolCallEvent

MemoryClient.retrieve_memories  (RetrievalInvocation)
MemoryClient.create_event       (ToolCall)
CodeInterpreter.execute_code    (ToolCall)
BrowserClient.take_control      (ToolCall)

Key Differences

Dimension	Upstream Bedrock	Strands Instrumentation
Abstraction level	AWS API calls	Agent/framework semantics
Span type	CLIENT	AgentInvocation, LLMInvocation, ToolCall, RetrievalInvocation, Workflow
LLM system attribute	aws.bedrock	aws.bedrock (LLM), strands (agent/tool)
Hook mechanism	botocore event hooks	Strands SDK HookProvider + wrapt
AgentCore components	Raw API calls only	Structured ToolCall/RetrievalInvocation spans
Double-tracing risk	None	Strands has its own built-in OTel tracer; suppressed by default
Dependency	botocore / boto3	strands-agents >= 1.0.0; bedrock-agentcore optional

splunk-otel-util-genai Types in Use

Type	Used for	Handler methods
Workflow	BedrockAgentCoreApp.entrypoint — top-level request boundary	start_workflow, stop_workflow, fail_workflow
AgentInvocation	Agent.invoke_async — full agent execution including input/output messages, model, tools	start_agent, stop_agent, fail_agent
LLMInvocation	Per-model-call span — model ID, provider, input messages, output message, stop reason, token usage	start_llm, stop_llm, fail_llm
ToolCall	Per-tool-call span (Strands tools, Memory write ops, CodeInterpreter, Browser) — name, arguments, result	start_tool_call, stop_tool_call, fail_tool_call
RetrievalInvocation	MemoryClient.retrieve_memories — query, retriever type, top_k, documents retrieved	start_retrieval, stop_retrieval, fail_retrieval
Error	Attached to any failed span — type (exception class name) and message	passed to fail_* methods
InputMessage	Structured user/system message for AgentInvocation.input_messages and LLMInvocation	—
OutputMessage	Structured assistant response for AgentInvocation.output_messages and LLMInvocation	—
Text	Content part inside InputMessage / OutputMessage	—

Example Telemetry

https://app.us1.signalfx.com/#/apm/traces/8a25fbfade5f5ced3404e4dac4eac2dc