Subagent in a separate process joining a trace via trace_context takes over the trace-level input/output

[royassis]

This issue was investigated and drafted by an AI coding agent (Claude) working with me as operator. The agent read the SDK source and queried our Langfuse instance via the public API; I ran the deploys and confirmed the symptom in the UI. Measurements are from real runs; interpretations are the agent's.

Summary

When a subagent running in a separate process joins a parent's trace via CallbackHandler(trace_context={"trace_id": ..., "parent_span_id": ...}), its observations nest correctly, but it also takes over the trace-level input, output, release and resourceAttributes. The parent's values are silently replaced, so the trace header shows the subagent's prompt instead of the end user's message.

Architecture

Two LangGraph agents, each in its own OS process, on separate AWS Bedrock AgentCore runtimes:

user -> agent (parent, process A)
          |- tool "ask_subagent"
               |- bedrock invoke_agent_runtime  (NOT http - no header propagation)
                    |- subagent (process B)
                         |- search_tool x N

Both write to the same Langfuse project, and both use langfuse.langchain.CallbackHandler over LangGraph (agent.stream(..., config={"callbacks": [handler]})).

• Parent (process A) pins a deterministic trace id so a separate backend can attach feedback scores later:

  CallbackHandler(trace_context={"trace_id": Langfuse.create_trace_id(seed=turn_id)})

• Subagent (process B) joins that trace, receiving both ids on the invoke payload:

  CallbackHandler(trace_context={"trace_id": caller_trace_id,
                                 "parent_span_id": caller_observation_id})

Each process builds its own Langfuse client with an isolated TracerProvider (both runtimes also run under opentelemetry-instrument, and we don't want the ambient spans exported to Langfuse).

Evidence

One trace, fetched via GET /api/public/traces/{id}:

field	value	should be
trace.input	the subagent's expanded prompt	the user's original message
trace.release	the subagent's git sha	the parent's git sha
trace.metadata.resourceAttributes.service.name	the subagent's service	the parent's service
trace.tags / sessionId / userId	parent's (correct)	-

The observation tree is correct, and the parent's own root observation holds the right input:

39bdbbc09c9d6ab9  CHAIN 'LangGraph'          <- parent, input = the USER's message
  dfc06b309bbf9bf0  CHAIN 'tools'
    00802bc1e5a3a330  TOOL 'ask_subagent'
      d8bb0693c7c7fb61  CHAIN 'LangGraph'    <- subagent, nests correctly
        ... 10x 'search_tool'

So only the trace-level copy is wrong; no observation data is lost.

Root cause (agent's reading of the source)

langfuse/_client/span_processor.py:245:

mark_app_root = (
    expected_exported
    and is_app_root_eligible(cast(ReadableSpan, span))
    and not parent_expected_exported
    and not suppressed_by_parent_claim
)

parent_expected_exported is resolved from self._span_export_expectation_by_id (span_processor.py:238) - a dict owned by that process's span processor. A subagent in a different process can never have seen the parent's span, so the lookup always misses and the subagent marks itself langfuse.internal.is_app_root too. Both processes claim it.

What we tried

Three fixes, deployed and measured. All failed:

1. Suppress is_langchain_root. CallbackHandler stamps it whenever LangChain reports parent_run_id is None (langchain/CallbackHandler.py:545), which is true in each process independently. Passing CallbackManager(handlers=[h], inheritable_handlers=[h], parent_run_id=uuid4()) instead of callbacks=[h] does suppress it - verified, only the parent stayed marked - and the trace input was still the subagent's. So that marker is not the driver.

2. Set the baggage claim. Attached baggage.set_baggage("langfuse_trace_id", trace_id.lower()) around the subagent's invocation, to trigger suppressed_by_parent_claim (propagation.py:691, LANGFUSE_TRACE_ID_BAGGAGE_KEY). Verified that _get_langfuse_trace_id_from_baggage() reads back exactly what we set, and that start_observation uses otel_trace_api.use_span() so the baggage survives into the span's parent_context. Deployed - no change, trace.release still the subagent's.

3. Considered patching the attribute off at export via mask_otel_spans / OtelSpanPatch, and dropping trace_context in favour of an explicit child span. Neither attempted, since both are guesses about backend behaviour.

Both 1 and 2 have been reverted; we are back to the plain form.

Expected behavior

A span that joins an existing trace via trace_context - especially with an explicit parent_span_id - should not become the trace's app root. By construction it is not the root of that trace.

Actual behavior

It claims app root, and the later-ingested claim supplies the trace-level fields.

Environment

• langfuse 4.14.1
• langchain-core 1.5.0, langchain 1.3.14, langgraph 1.2.9
• opentelemetry-sdk 1.42.1, Python 3.13
• Two processes on AWS Bedrock AgentCore; transport between them is invoke_agent_runtime, so HTTP header propagation / propagate_attributes(as_baggage=True) is not available to us

Questions / request

1. Is there a supported way for a subagent in a separate process to join a trace without claiming the trace-level fields? If baggage is the intended mechanism, what are we missing in attempt 2?
2. Would it be reasonable for trace_context with an explicit parent_span_id to imply "not the app root"?
3. Secondary, possibly a separate bug: when trace_context has a trace_id and no parent_span_id, _create_remote_parent_span (client.py:1749) substitutes RandomIdGenerator().generate_span_id(), so the span is parented to an id that does not exist. Measured across 25 consecutive traces of ours: every one has an orphaned root and zero observations with no parent. Is that intended? It means a trace that pins its id can never have a true root observation.

What's verified vs. estimated

Verified / measured

• Trace-level fields come from the subagent - measured via the public API on a real trace
• Observation nesting is correct - same API response
• Source citations checked against main: span_processor.py:229-253, propagation.py:691, client.py:1749, langchain/CallbackHandler.py:545
• Attempt 1 suppressed is_langchain_root without changing the outcome - measured, only one root-marked observation in the trace afterwards
• Attempt 2's baggage round-trips through langfuse's own getter - verified locally
• 25 of 25 traces have an orphaned root - measured via the API

Not verified (agent's inference)

• That is_app_root is what the backend uses to populate trace-level fields - inferred from the SDK source; the backend was not inspected
• Why attempt 2 failed - unknown; either the baggage is absent from the context at the moment the span starts, or is_app_root is not the deciding signal