LangSmith Hyper-Granular Agent Trace Querying

How granularly can LangSmith agent traces be queried today, especially when traces contain nested agents, subagents, tool calls, and deep parent/child run trees?

Short answer: very granularly, with one important boundary. LangSmith exposes server-side filters for root runs, child runs, tree membership, metadata, run type, selected run fields, and exact run depth through the system metadata key ls_run_depth. Hydrated trace-tree analysis remains useful for max/min-depth summaries, path extraction, and arbitrary returned-field predicates that are not indexed as first-class filters.

This repo is an evidence-backed investigation, not a general agent demo. It is written for people building or evaluating agent observability workflows, including the teams and ecosystems around LangChain and Blitzy. The useful part is the boundary line: what can be queried server-side today, what can be proved from hydrated traces, and what would need first-class product support to become native search.

What This Proves

• Direct child-run lookup via parentRunId.
• Root-run filtering via traceFilter.
• Any-run-in-tree filtering via treeFilter.
• Run selection by run type, including llm, chain, and tool.
• Metadata filtering for scenario IDs and run IDs.
• Exact run-depth filtering through ls_run_depth, including trace-level selection with treeFilter.
• Full trace hydration with child runs loaded.
• Local min_depth / max_depth summaries over hydrated trace trees.
• Tool-call evidence detection in LLM outputs and tool runs.
• DeepAgents traces with meaningful nested parent/child structure.

Evidence Snapshot

Live proof was recorded against official OpenAI and real LangSmith tracing:

run_id: redacted-live-suite-run-id
project: langsmith-hyper-granular-agent-trace-querying
project_id: redacted-langsmith-project-id
workspace_id: redacted-langsmith-workspace-id
target: https://api.openai.com/v1
model: gpt-5.5
status: passed

Scenario coverage:

Scenario	Result	Evidence
OpenAI SDK Chat Completions tool call	passed	LLM run present; tool-call evidence found
OpenAI SDK Chat Completions streaming tool call	passed	Incremental stream timing; tool-call evidence found
OpenAI SDK Responses structured output	passed	LLM run present; structured output parsed
OpenAI SDK Responses streaming text	passed	Incremental stream timing; completion event observed
LangChain ChatOpenAI tool trace	passed	max_depth=1, LLM child selected, tool-call evidence found
DeepAgents nested trace	passed	max_depth=5, llm_run_count=4, tool_run_count=3

Repo Map

• EVIDENCE.md: capability matrix and live trace evidence.
• docs/scope-and-limits.md: exact server-side versus hydrated-local boundary.
• docs/reproducibility.md: how to rerun the examples and regenerate reports.
• examples/: minimal OpenAI SDK, LangChain, DeepAgents, and LangSmith-query scripts.
• reports/: redacted live reports, exact-depth filter checks, and official OpenAI golden fixtures.
• upstream/: optional PR body drafts and patch artifacts for LangSmith CLI/docs follow-up work.

Quick Start

cp .env.example .env
npm install
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

Add OPENAI_API_KEY and LANGSMITH_API_KEY to .env, then run the examples:

npm run example:openai
npm run example:langchain
npm run example:query
python examples/deepagents_nested_trace.py

Scope

This repo does not claim that every hydrated field is natively indexed in hosted LangSmith search. It demonstrates the maximum practical query granularity available from public LangSmith/LangChain surfaces today:

• Server-side filtering where LangSmith exposes query arguments.
• Local analysis where the data is available after trace hydration.
• Clear product follow-up candidates for arbitrary hydrated JSON-path search and richer compound predicates over returned trace-tree payloads.