Skip to content

Commit 593dcae

Browse files
committed
v0.53.0: vstack-import gains LangSmith + Phoenix formats
- from_langsmith_runs + --format langsmith: LangSmith run tree / list -> AgentTrace (run_type -> step type; inputs/outputs/error -> content; root-run goal/outcome). - --format phoenix + extended from_otel_spans: read OpenInference attrs (openinference.span.kind, input.value/output.value, llm.*_messages) so Arize Phoenix exports import cleanly. vstack-import now covers the dominant LLM-observability exports. 3,249 tests.
1 parent 75b5364 commit 593dcae

7 files changed

Lines changed: 238 additions & 33 deletions

File tree

CHANGELOG.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,28 @@ project adheres to [Semantic Versioning](https://semver.org/) from
66
`1.0.0` onward. During the `0.x` series, minor bumps may include
77
breaking changes (see API stability promise in `vstack/__init__.py`).
88

9+
## [0.53.0] — 2026-06-23
10+
11+
`vstack-import` now reads the two dominant LLM-observability exports.
12+
13+
### Added
14+
15+
- **`vstack.ingest.from_langsmith_runs`** + **`vstack-import --format
16+
langsmith`** — convert a LangSmith run (with nested `child_runs` — a run
17+
tree) or a flat list of runs into an `AgentTrace`. Maps `run_type`
18+
(llm→message, tool→tool_call, chain→thought, retriever→observation, …);
19+
uses `inputs`/`outputs`/`error` as step content; infers goal/outcome from
20+
the root run. The CLI accepts a single run, a list, or `{"runs": [...]}`.
21+
- **`vstack-import --format phoenix`** + extended `from_otel_spans` — reads
22+
OpenInference attributes (`openinference.span.kind`, `input.value`/
23+
`output.value`, `llm.input_messages`/`llm.output_messages`), so Arize
24+
Phoenix span exports import cleanly (the generic `otel` format now covers
25+
them too).
26+
27+
### Compatibility
28+
29+
- All tests pass. Additive only; no breaking changes.
30+
931
## [0.52.0] — 2026-06-23
1032

1133
The Action's CI UX is now complete: gate · annotate · comment.

README.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -130,11 +130,13 @@ vstack-recipes # browse named bundles (stuck_i
130130
vstack-diagnose --trace trace.json --recipe stuck_in_loop --client anthropic
131131
```
132132

133-
Don't have a vstack trace yet? Import the logs you already have — `vstack-import` converts OpenAI/Anthropic chat-message logs or OpenTelemetry spans into a trace, ready to pipe straight in:
133+
Don't have a vstack trace yet? Import the logs you already have — `vstack-import` converts OpenAI/Anthropic chat-message logs, OpenTelemetry spans, **Arize Phoenix** (OpenInference) spans, or **LangSmith** runs into a trace, ready to pipe straight in:
134134

135135
```bash
136-
vstack-import --format messages chat.json | vstack-diagnose --trace - --client anthropic
137-
vstack-import --format otel spans.json --goal "ship auth" | vstack-diagnose --trace -
136+
vstack-import --format messages chat.json | vstack-diagnose --trace - --client anthropic
137+
vstack-import --format otel spans.json | vstack-diagnose --trace -
138+
vstack-import --format phoenix spans.json | vstack-diagnose --trace -
139+
vstack-import --format langsmith run.json | vstack-diagnose --trace -
138140
```
139141

140142
## Install

_ingest/lib/__init__.py

Lines changed: 119 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -9,18 +9,23 @@
99
Public API:
1010
1111
* :func:`from_chat_messages` — a list of ``{role, content, tool_calls?}`` dicts.
12-
* :func:`from_otel_spans` — a list of OpenTelemetry span dicts (best-effort,
13-
reads ``gen_ai.*`` attributes).
12+
* :func:`from_otel_spans` — a list of OpenTelemetry / OpenInference span dicts
13+
(best-effort; reads ``gen_ai.*`` and OpenInference ``llm.*`` attributes —
14+
also covers Arize Phoenix exports).
15+
* :func:`from_langsmith_runs` — a LangSmith run (tree) or list of runs.
1416
"""
1517

1618
from __future__ import annotations
1719

20+
import json
1821
from datetime import datetime, timedelta, timezone
19-
from typing import Any
22+
from typing import Any, Literal
2023

2124
from vstack.aar import AgentTrace, TraceStep
2225

23-
__all__ = ["from_chat_messages", "from_otel_spans"]
26+
__all__ = ["from_chat_messages", "from_langsmith_runs", "from_otel_spans"]
27+
28+
StepType = Literal["tool_call", "message", "decision", "observation", "thought"]
2429

2530
_BASE_TS = datetime(2026, 1, 1, tzinfo=timezone.utc)
2631

@@ -133,25 +138,37 @@ def from_otel_spans(
133138
"""Build an ``AgentTrace`` from OpenTelemetry spans (best-effort).
134139
135140
Spans are ordered by start time. Each span becomes a step: GenAI/LLM spans
136-
(a ``gen_ai.*`` attribute or an ``llm``/``chat`` name) → ``tool_call``;
137-
others → ``observation``. Reads ``attributes`` either as a dict or as a
138-
list of ``{key, value}`` (OTLP-JSON form).
141+
(a ``gen_ai.*`` attribute, an OpenInference ``llm.*`` attribute or
142+
``openinference.span.kind == LLM``, or an ``llm``/``chat`` name) →
143+
``tool_call``; others → ``observation``. Reads ``attributes`` either as a
144+
dict or as a list of ``{key, value}`` (OTLP-JSON form). Also handles
145+
OpenInference spans (the format Arize Phoenix exports).
139146
"""
140147
ordered = sorted(spans, key=_span_start)
141148
steps: list[TraceStep] = []
142149

143150
for i, span in enumerate(ordered):
144151
attrs = _otel_attrs(span)
145152
name = str(span.get("name", "span"))
146-
is_genai = name.lower().startswith(("llm", "chat", "gen_ai")) or any(
147-
k.startswith("gen_ai") for k in attrs
153+
kind = str(attrs.get("openinference.span.kind", "")).upper()
154+
is_genai = (
155+
name.lower().startswith(("llm", "chat", "gen_ai"))
156+
or kind in ("LLM", "CHAIN", "AGENT")
157+
or any(k.startswith(("gen_ai", "llm.")) for k in attrs)
148158
)
149159
completion = (
150160
attrs.get("gen_ai.completion")
151161
or attrs.get("gen_ai.response.content")
152162
or attrs.get("llm.output")
163+
or attrs.get("llm.output_messages")
164+
or attrs.get("output.value") # OpenInference / Phoenix
165+
)
166+
prompt = (
167+
attrs.get("gen_ai.prompt")
168+
or attrs.get("llm.input")
169+
or attrs.get("llm.input_messages")
170+
or attrs.get("input.value") # OpenInference / Phoenix
153171
)
154-
prompt = attrs.get("gen_ai.prompt") or attrs.get("llm.input")
155172
detail = str(completion or prompt or "")
156173
content = f"{name}: {detail}" if detail else name
157174
steps.append(
@@ -187,3 +204,95 @@ def _otel_attrs(span: dict[str, Any]) -> dict[str, Any]:
187204
val = next(iter(val.values()), val)
188205
out[str(item["key"])] = val
189206
return out
207+
208+
209+
# LangSmith run_type → trace step type.
210+
_LANGSMITH_STEP_TYPE: dict[str, StepType] = {
211+
"llm": "message",
212+
"chat": "message",
213+
"tool": "tool_call",
214+
"retriever": "observation",
215+
"chain": "thought",
216+
"agent": "decision",
217+
"prompt": "thought",
218+
"parser": "observation",
219+
}
220+
221+
222+
def _short(obj: Any, limit: int = 300) -> str:
223+
"""Compact one-line rendering of a run's inputs/outputs."""
224+
if obj is None:
225+
return ""
226+
text = obj if isinstance(obj, str) else json.dumps(obj, default=str, ensure_ascii=False)
227+
text = " ".join(text.split())
228+
return text if len(text) <= limit else text[: limit - 1] + "…"
229+
230+
231+
def _flatten_runs(runs: list[dict[str, Any]]) -> list[dict[str, Any]]:
232+
"""Depth-first flatten of LangSmith run trees, then order by start time."""
233+
out: list[dict[str, Any]] = []
234+
235+
def visit(run: dict[str, Any]) -> None:
236+
out.append(run)
237+
for child in run.get("child_runs") or []:
238+
if isinstance(child, dict):
239+
visit(child)
240+
241+
for run in runs:
242+
visit(run)
243+
return sorted(out, key=lambda r: str(r.get("start_time") or ""))
244+
245+
246+
def _ls_content(run: dict[str, Any]) -> str:
247+
name = str(run.get("name", "run"))
248+
parts: list[str] = []
249+
if run.get("inputs"):
250+
parts.append(f"in: {_short(run.get('inputs'))}")
251+
if run.get("outputs"):
252+
parts.append(f"out: {_short(run.get('outputs'))}")
253+
if run.get("error"):
254+
parts.append(f"ERROR: {_short(run.get('error'))}")
255+
body = " | ".join(parts)
256+
return f"{name}{body}" if body else name
257+
258+
259+
def from_langsmith_runs(
260+
runs: list[dict[str, Any]] | dict[str, Any],
261+
*,
262+
goal: str = "",
263+
outcome: str = "",
264+
success: bool = False,
265+
agent_id: str | None = None,
266+
agent_framework: str = "langsmith",
267+
metadata: dict[str, Any] | None = None,
268+
) -> AgentTrace:
269+
"""Build an ``AgentTrace`` from LangSmith runs.
270+
271+
Accepts a single run (with nested ``child_runs`` — a run tree) or a flat
272+
list of runs. ``run_type`` maps to the step type (``llm``→message,
273+
``tool``→tool_call, ``chain``→thought, ``retriever``→observation, …);
274+
``inputs``/``outputs``/``error`` become the step content. ``goal`` defaults
275+
to the root run's inputs and ``outcome`` to its outputs.
276+
"""
277+
run_list = [runs] if isinstance(runs, dict) else list(runs)
278+
flat = _flatten_runs(run_list)
279+
steps = [
280+
TraceStep(
281+
timestamp=_ts(i),
282+
type=_LANGSMITH_STEP_TYPE.get(str(run.get("run_type", "")).lower(), "observation"),
283+
content=_ls_content(run),
284+
metadata={"run_type": str(run.get("run_type", "")), "name": str(run.get("name", ""))},
285+
)
286+
for i, run in enumerate(flat)
287+
]
288+
289+
root = run_list[0] if run_list else {}
290+
return AgentTrace(
291+
agent_id=agent_id,
292+
agent_framework=agent_framework,
293+
goal=goal or _short(root.get("inputs")) or "(goal not provided)",
294+
steps=steps,
295+
outcome=outcome or _short(root.get("outputs")) or "(outcome not provided)",
296+
success=success,
297+
metadata=metadata or {},
298+
)

_ingest/lib/_cli.py

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
from pathlib import Path
1616
from typing import Any, Sequence
1717

18-
from . import from_chat_messages, from_otel_spans
18+
from . import from_chat_messages, from_langsmith_runs, from_otel_spans
1919

2020

2121
def _load(path: str | None) -> Any:
@@ -53,9 +53,12 @@ def main(argv: Sequence[str] | None = None) -> int:
5353
parser.add_argument(
5454
"--format",
5555
"-f",
56-
choices=("messages", "otel"),
56+
choices=("messages", "otel", "phoenix", "langsmith"),
5757
required=True,
58-
help="messages = OpenAI/Anthropic chat log; otel = OpenTelemetry spans.",
58+
help=(
59+
"messages = OpenAI/Anthropic chat log; otel = OpenTelemetry spans; "
60+
"phoenix = Arize Phoenix / OpenInference spans; langsmith = LangSmith run(s)."
61+
),
5962
)
6063
parser.add_argument("--goal", default="", help="The agent's goal (else inferred).")
6164
parser.add_argument("--outcome", default="", help="What happened (else inferred).")
@@ -68,24 +71,24 @@ def main(argv: Sequence[str] | None = None) -> int:
6871
parser.add_argument("--out", "-o", default=None, help="Write to a file (default: stdout).")
6972
args = parser.parse_args(argv)
7073

74+
common = {
75+
"goal": args.goal,
76+
"outcome": args.outcome,
77+
"success": args.success,
78+
"agent_id": args.agent_id,
79+
}
7180
try:
7281
payload = _load(args.input)
7382
if args.format == "messages":
74-
trace = from_chat_messages(
75-
_items(payload, "messages"),
76-
goal=args.goal,
77-
outcome=args.outcome,
78-
success=args.success,
79-
agent_id=args.agent_id,
80-
)
81-
else:
82-
trace = from_otel_spans(
83-
_items(payload, "spans"),
84-
goal=args.goal,
85-
outcome=args.outcome,
86-
success=args.success,
87-
agent_id=args.agent_id,
88-
)
83+
trace = from_chat_messages(_items(payload, "messages"), **common)
84+
elif args.format in ("otel", "phoenix"):
85+
trace = from_otel_spans(_items(payload, "spans"), **common)
86+
else: # langsmith — accept a single run (tree), a list, or {"runs": [...]}
87+
if isinstance(payload, dict) and not isinstance(payload.get("runs"), list):
88+
runs: Any = payload # a single run / run tree
89+
else:
90+
runs = _items(payload, "runs")
91+
trace = from_langsmith_runs(runs, **common)
8992
except (OSError, ValueError, json.JSONDecodeError) as e:
9093
print(f"vstack-import: {e}", file=sys.stderr)
9194
return 2

_ingest/tests/test_ingest.py

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,3 +125,72 @@ def test_cli_bad_shape_returns_2() -> None:
125125
code, _out, err = _run_cli(["--format", "messages"], stdin=json.dumps({"nope": 1}))
126126
assert code == 2
127127
assert "vstack-import" in err
128+
129+
130+
# --- LangSmith ------------------------------------------------------------
131+
132+
133+
def test_from_langsmith_run_tree() -> None:
134+
from vstack.ingest import from_langsmith_runs
135+
136+
tree = {
137+
"name": "My Chat Bot",
138+
"run_type": "chain",
139+
"start_time": "2026-01-01T00:00:00",
140+
"inputs": {"text": "summarize meetings"},
141+
"outputs": {"output": "done"},
142+
"child_runs": [
143+
{
144+
"name": "My LLM",
145+
"run_type": "llm",
146+
"start_time": "2026-01-01T00:00:01",
147+
"inputs": {"prompts": ["..."]},
148+
"outputs": {"generations": ["use the tool"]},
149+
},
150+
{
151+
"name": "loader",
152+
"run_type": "tool",
153+
"start_time": "2026-01-01T00:00:02",
154+
"inputs": {"date": "x"},
155+
"error": "boom",
156+
},
157+
],
158+
}
159+
trace = from_langsmith_runs(tree)
160+
types = [s.type for s in trace.steps]
161+
assert types == ["thought", "message", "tool_call"] # chain, llm, tool
162+
assert trace.goal.startswith("") and "summarize meetings" in trace.goal
163+
assert "ERROR: boom" in trace.steps[2].content
164+
165+
166+
def test_from_langsmith_flat_list_orders_by_start() -> None:
167+
from vstack.ingest import from_langsmith_runs
168+
169+
runs = [
170+
{"name": "b", "run_type": "tool", "start_time": "2026-01-01T00:00:05"},
171+
{"name": "a", "run_type": "llm", "start_time": "2026-01-01T00:00:01"},
172+
]
173+
trace = from_langsmith_runs(runs)
174+
assert [s.metadata["name"] for s in trace.steps] == ["a", "b"]
175+
176+
177+
def test_cli_langsmith_single_run() -> None:
178+
run = {"name": "root", "run_type": "chain", "inputs": {"q": "hi"}, "outputs": {"a": "ok"}}
179+
code, out, _ = _run_cli(["--format", "langsmith"], stdin=json.dumps(run))
180+
assert code == 0
181+
assert json.loads(out)["agent_framework"] == "langsmith"
182+
183+
184+
def test_cli_phoenix_openinference_span() -> None:
185+
spans = [
186+
{
187+
"name": "llm",
188+
"start_time": 1,
189+
"attributes": {"openinference.span.kind": "LLM", "output.value": "the answer"},
190+
}
191+
]
192+
code, out, _ = _run_cli(["--format", "phoenix"], stdin=json.dumps(spans))
193+
assert code == 0
194+
trace = json.loads(out)
195+
assert trace["steps"][0]["type"] == "tool_call"
196+
assert "the answer" in trace["steps"][0]["content"]

_packaging/vstack/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@
7373

7474
from __future__ import annotations
7575

76-
__version__ = "0.52.0"
76+
__version__ = "0.53.0"
7777

7878
# The diagnose() function and PATTERNS registry are lazy-imported below
7979
# so that ``import vstack`` itself stays cheap. Pattern sub-packages

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "valanistack"
3-
version = "0.52.0"
3+
version = "0.53.0"
44
description = "Organizational behavior, practiced on AI agents."
55
readme = "README.md"
66
requires-python = ">=3.11"

0 commit comments

Comments
 (0)