Skip to content

Commit d6ec352

Browse files
rohita5lclaude
andauthored
Add MLflow tracing for Claude, OpenCode, and Codex via `ucode configu… (#133)
* Add MLflow tracing for Claude, OpenCode, and Codex via `ucode configure tracing` Route coding-session traces to per-user Databricks MLflow experiments (`/Users/<email>/ucode-<agent>-traces`) using the workspace profile auth ucode already configures. Adds a `tracing.py` module, a `configure tracing` subcommand (workspace selector + `--disable`), per-agent config/env wiring, get-or-create experiment helpers, and an optional `tracing` extra for the Claude Python plugin runtime (auto-installed via `uv tool`). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Address PR review: preserve current_workspace, clear stale tracing env, warn on Codex notify clobber - `configure tracing` no longer silently flips `current_workspace` when operating on a non-current workspace (snapshot + restore via new `state.set_current_workspace`). - Codex `_apply_tracing_notify` warns before replacing a pre-existing user-defined `notify` (the backup file holds the prior value). - New `tracing.apply_tracing_env` actively pops the MLflow env keys when tracing is off, so a stale outer-shell value can't leak into Codex / OpenCode subprocesses. - `--disable` picks among workspaces that actually have tracing enabled and auto-selects when there's only one match (same skip-prompt also applied to the enable path). - Document the Claude plugin "already" stderr match as best-effort. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * CI: run tracing e2e in e2e job; raise opencode launch timeout to 180s - `tests/test_e2e_tracing.py` was getting silently skipped on the unit job (no UCODE_TEST_WORKSPACE) and not collected at all on the e2e job (which restricts to `tests/test_e2e.py`). Add a dedicated step that runs it with `--extra tracing` so `import mlflow` resolves, and ignore it explicitly in the unit job to avoid the noisy collect-then-skip. - `test_launch_opencode_per_model` exceeds 90s on databricks-gemini-3-1 -flash-lite in CI; raise to 180s to match the codex/claude per-model launch budget. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test_e2e: log per-model timing and capture timeout stderr; run tracing e2e even when prior step fails - `test_launch_opencode_per_model` now prints `[opencode-per-model] provider/model OK|FAIL|TIMEOUT (Xs)` per iteration and, on TimeoutExpired, captures the subprocess's partial stderr instead of bubbling the bare TimeoutExpired. The loop continues across models so the final assert lists every failure — when CI fails we can see which model(s) were slow vs. cold-start vs. hung. - The tracing e2e step now runs with `if: !cancelled()` so a failure in the prior `tests/test_e2e.py` step doesn't skip it. The two suites are independent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Skip databricks-gemini-3-1-flash-lite in opencode per-model test Per-model timing in the previous CI run showed every other configured opencode model returns in ~3s; only gemini-3-1-flash-lite hangs past 180s with no stderr beyond `> build · <model>`. Backend-side latency we can't influence from this repo, so skip it rather than keep blocking CI. Per-model logging stays in place to surface the next flake immediately. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Format test_e2e.py with ruff Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ci: retrigger after transient AI Gateway probe timeout Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 26cb898 commit d6ec352

13 files changed

Lines changed: 3955 additions & 16 deletions

File tree

.github/workflows/ci.yml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ jobs:
1515
steps:
1616
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
1717
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
18-
- run: uv run pytest --ignore=tests/test_e2e.py
18+
- run: uv run pytest --ignore=tests/test_e2e.py --ignore=tests/test_e2e_tracing.py
1919

2020
e2e:
2121
if: vars.E2E_ENABLED == 'true'
@@ -54,3 +54,10 @@ jobs:
5454
# set, the auth code path doesn't shell out at all — this is a safety
5555
# net for any code path we may have missed.
5656
- run: uv run pytest tests/test_e2e.py -v < /dev/null
57+
# MLflow tracing e2e lives in its own file and needs the `tracing`
58+
# extra so `import mlflow` resolves (otherwise the test importorskips
59+
# and silently passes as skipped). `!cancelled()` lets this run even
60+
# when the previous pytest step failed — the two suites are
61+
# independent and one shouldn't mask the other.
62+
- if: ${{ !cancelled() }}
63+
run: uv run --extra tracing pytest tests/test_e2e_tracing.py -v < /dev/null

pyproject.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,12 @@ dependencies = [
1515
"typer>=0.12.0",
1616
]
1717

18+
[project.optional-dependencies]
19+
# Only the Claude Code tracing path needs the Python MLflow runtime; OpenCode
20+
# and Codex trace via JS plugins. Kept out of the core deps so a plain `ucode`
21+
# install stays lean. Enable with `uv tool install "ucode[tracing]"`.
22+
tracing = ["mlflow[databricks]>=3.4"]
23+
1824
[project.scripts]
1925
ucode = "ucode.cli:main"
2026

src/ucode/agents/claude.py

Lines changed: 134 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import os
66
import re
77
import shutil
8+
import subprocess
89
from pathlib import Path
910

1011
from ucode.agent_updates import available_npm_package_update
@@ -23,7 +24,8 @@
2324
)
2425
from ucode.state import mark_tool_managed, save_state
2526
from ucode.telemetry import agent_version, ucode_version
26-
from ucode.ui import print_warning
27+
from ucode.tracing import tracing_env
28+
from ucode.ui import print_note, print_success, print_warning
2729

2830
CLAUDE_CONFIG_DIR = Path.home() / ".claude"
2931
CLAUDE_SETTINGS_PATH = CLAUDE_CONFIG_DIR / "ucode-settings.json"
@@ -61,6 +63,20 @@ def _resolve_web_search_model(state: dict) -> str | None:
6163
WEB_SEARCH_MCP_NAME = "web_search"
6264
_CLAUDE_MODEL_RE = re.compile(r"^databricks-claude-(opus|sonnet)-(\d+)-(\d+)(.*)$")
6365

66+
# Env keys consumed by the MLflow Claude tracing plugin. Written into the
67+
# settings `env` block; the plugin runtime (installed separately) reads them.
68+
CLAUDE_TRACING_ENV_KEYS = (
69+
"MLFLOW_CLAUDE_TRACING_ENABLED",
70+
"MLFLOW_TRACKING_URI",
71+
"MLFLOW_EXPERIMENT_ID",
72+
)
73+
CLAUDE_TRACING_MARKETPLACE = "mlflow/mlflow"
74+
CLAUDE_TRACING_PLUGIN = "mlflow-tracing@mlflow-plugins"
75+
# The plugin runtime shells out to the `mlflow` CLI, so it must be on PATH at
76+
# this minimum version. ucode installs/upgrades it via `uv tool`.
77+
MLFLOW_CLI_SPEC = "mlflow[databricks]>=3.4"
78+
MINIMUM_MLFLOW_VERSION = (3, 4)
79+
6480

6581
def _web_search_mcp_entry(workspace: str, search_model: str, profile: str | None = None) -> dict:
6682
"""Stdio MCP server entry pointing at `ucode mcp web-search`. Resolves
@@ -198,8 +214,19 @@ def write_tool_config(state: dict, model: str) -> dict:
198214
disable_web_search=web_search_model is not None,
199215
profile=state.get("profile"),
200216
)
217+
tracing_env_vars = tracing_env(state, "claude")
218+
if tracing_env_vars:
219+
overlay["env"]["MLFLOW_CLAUDE_TRACING_ENABLED"] = "true"
220+
overlay["env"].update(tracing_env_vars)
221+
managed_keys = managed_keys + [["env", key] for key in CLAUDE_TRACING_ENV_KEYS]
222+
201223
existing = read_json_safe(CLAUDE_SETTINGS_PATH)
202224
merged = deep_merge_dict(existing, overlay)
225+
if not tracing_env_vars:
226+
env_block = merged.get("env")
227+
if isinstance(env_block, dict):
228+
for key in CLAUDE_TRACING_ENV_KEYS:
229+
env_block.pop(key, None)
203230
write_json_file(CLAUDE_SETTINGS_PATH, merged)
204231

205232
if web_search_model:
@@ -210,6 +237,112 @@ def write_tool_config(state: dict, model: str) -> dict:
210237
return state
211238

212239

240+
def ensure_tracing_runtime() -> bool:
241+
"""Ensure Claude's MLflow tracing runtime is ready: an `mlflow` CLI >= 3.4 on
242+
PATH (the plugin shells out to it) and the MLflow Claude plugin installed.
243+
244+
Best-effort — warns and returns False if a piece can't be set up, so
245+
`ucode configure tracing` can still finish for other agents."""
246+
if not _ensure_mlflow_cli():
247+
return False
248+
return _install_claude_tracing_plugin()
249+
250+
251+
def _parse_mlflow_version(text: str) -> tuple[int, int] | None:
252+
match = re.search(r"(\d+)\.(\d+)", text)
253+
if not match:
254+
return None
255+
return int(match.group(1)), int(match.group(2))
256+
257+
258+
def _installed_mlflow_version() -> tuple[int, int] | None:
259+
"""The (major, minor) of the `mlflow` CLI on PATH, or None if absent."""
260+
if not shutil.which("mlflow"):
261+
return None
262+
try:
263+
result = subprocess.run(
264+
["mlflow", "--version"], check=False, capture_output=True, text=True, timeout=30
265+
)
266+
except (OSError, subprocess.TimeoutExpired):
267+
return None
268+
return _parse_mlflow_version(result.stdout or result.stderr or "")
269+
270+
271+
def _ensure_mlflow_cli() -> bool:
272+
"""Ensure an `mlflow` CLI >= 3.4 is on PATH, installing or upgrading it via
273+
`uv tool` when needed."""
274+
current = _installed_mlflow_version()
275+
if current and current >= MINIMUM_MLFLOW_VERSION:
276+
return True
277+
278+
if not shutil.which("uv"):
279+
verb = "upgrade" if current else "install"
280+
print_warning(
281+
f"Claude tracing needs the `mlflow` CLI >= 3.4 on PATH, but `uv` is not "
282+
f'available to {verb} it. Run `uv tool install "{MLFLOW_CLI_SPEC}"` '
283+
f'(or `pip install "{MLFLOW_CLI_SPEC}"`), then re-run `ucode configure tracing`.'
284+
)
285+
return False
286+
287+
print_note(f"{'Upgrading' if current else 'Installing'} the mlflow CLI ({MLFLOW_CLI_SPEC})...")
288+
# --force replaces an existing (older) uv-managed mlflow tool in place.
289+
cmd = ["uv", "tool", "install", MLFLOW_CLI_SPEC]
290+
if current:
291+
cmd.append("--force")
292+
try:
293+
subprocess.run(cmd, check=True, timeout=600)
294+
except (OSError, subprocess.CalledProcessError, subprocess.TimeoutExpired) as exc:
295+
print_warning(f"Could not install the mlflow CLI automatically: {exc}")
296+
return False
297+
298+
if not shutil.which("mlflow"):
299+
print_warning(
300+
"Installed mlflow, but `mlflow` is still not on PATH. Ensure your uv tool "
301+
"bin directory (e.g. ~/.local/bin) is on PATH, then re-run `ucode configure tracing`."
302+
)
303+
return False
304+
print_success("mlflow CLI ready")
305+
return True
306+
307+
308+
def _install_claude_tracing_plugin() -> bool:
309+
binary = SPEC["binary"]
310+
if not shutil.which(binary):
311+
print_warning("`claude` is not installed; skipping MLflow tracing plugin install.")
312+
return False
313+
commands = [
314+
[
315+
binary,
316+
"plugin",
317+
"marketplace",
318+
"add",
319+
CLAUDE_TRACING_MARKETPLACE,
320+
"--sparse",
321+
".claude-plugin",
322+
],
323+
[binary, "plugin", "install", CLAUDE_TRACING_PLUGIN],
324+
]
325+
for cmd in commands:
326+
try:
327+
result = subprocess.run(cmd, check=False, capture_output=True, text=True, timeout=120)
328+
except (OSError, subprocess.TimeoutExpired) as exc:
329+
print_warning(f"Could not install the Claude MLflow plugin: {exc}")
330+
return False
331+
if result.returncode != 0:
332+
output = (result.stderr or result.stdout or "").strip()
333+
last = output.splitlines()[-1] if output else f"exit {result.returncode}"
334+
# `marketplace add` / `install` are idempotent; treat "already
335+
# added/installed" as success and keep going. Best-effort match
336+
# against stderr — an upstream wording change would degrade this
337+
# to a noisy warning on re-runs, but never corrupts state.
338+
if "already" in last.lower():
339+
continue
340+
print_warning(f"Claude MLflow plugin step failed: {last}")
341+
return False
342+
print_success("Claude MLflow tracing plugin installed")
343+
return True
344+
345+
213346
def default_model(state: dict) -> str | None:
214347
claude_models = state.get("claude_models") or {}
215348
return claude_models.get("opus") or claude_models.get("sonnet") or claude_models.get("haiku")

src/ucode/agents/codex.py

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
import os
66
import re
7+
import subprocess
78
from pathlib import Path
89

910
from ucode.agent_updates import available_npm_package_update
@@ -22,6 +23,8 @@
2223
)
2324
from ucode.state import mark_tool_managed, save_state
2425
from ucode.telemetry import agent_version, ucode_version
26+
from ucode.tracing import agent_tracing, apply_tracing_env
27+
from ucode.ui import print_success, print_warning
2528

2629
CODEX_CONFIG_DIR = Path.home() / ".codex"
2730
CODEX_PROFILE_NAME = "ucode"
@@ -32,6 +35,8 @@
3235
CODEX_MODEL_PROVIDER_NAME = "ucode-databricks"
3336
MINIMUM_CODEX_VERSION = (0, 134, 0)
3437
MINIMUM_CODEX_VERSION_TEXT = "0.134.0"
38+
CODEX_TRACING_NOTIFY = ["mlflow-codex", "notify-hook"]
39+
CODEX_TRACING_PACKAGE = "@mlflow/codex"
3540

3641

3742
SPEC: ToolSpec = {
@@ -235,6 +240,7 @@ def write_tool_config(state: dict, model: str | None = None) -> dict:
235240
overlay = render_legacy_overlay(workspace, chosen_model, databricks_profile)
236241
doc = read_toml_safe(LEGACY_CODEX_CONFIG_PATH)
237242
deep_merge_dict(doc, overlay)
243+
_apply_tracing_notify(doc, state)
238244
write_toml_file(LEGACY_CODEX_CONFIG_PATH, doc)
239245
state = mark_tool_managed(state, "codex", LEGACY_MANAGED_KEYS)
240246
save_state(state)
@@ -245,12 +251,55 @@ def write_tool_config(state: dict, model: str | None = None) -> dict:
245251
overlay = render_overlay(workspace, chosen_model, databricks_profile)
246252
doc = read_toml_safe(CODEX_CONFIG_PATH)
247253
deep_merge_dict(doc, overlay)
254+
_apply_tracing_notify(doc, state)
248255
write_toml_file(CODEX_CONFIG_PATH, doc)
249256
state = mark_tool_managed(state, "codex", MANAGED_KEYS)
250257
save_state(state)
251258
return state
252259

253260

261+
def _apply_tracing_notify(doc: dict, state: dict) -> None:
262+
"""Set/clear the Codex ``notify`` hook that streams session traces to MLflow.
263+
264+
Only ucode's own notify value is removed on disable, so a user-defined
265+
``notify`` is left intact. When enabling on top of a pre-existing user
266+
``notify``, warn before replacing — the prior value is in the backup file
267+
but the user has to restore it manually."""
268+
if agent_tracing(state, "codex") is not None:
269+
existing = doc.get("notify")
270+
if existing is not None and list(existing) != CODEX_TRACING_NOTIFY:
271+
print_warning(
272+
f"Codex `notify` is already set to {existing!r}; replacing it with the "
273+
"MLflow tracing hook. The previous value is preserved in the Codex "
274+
"config backup — restore it manually if you need both."
275+
)
276+
doc["notify"] = list(CODEX_TRACING_NOTIFY)
277+
elif list(doc.get("notify") or []) == CODEX_TRACING_NOTIFY:
278+
doc.pop("notify", None)
279+
280+
281+
def ensure_tracing_dependency() -> bool:
282+
"""Install the `@mlflow/codex` npm package that provides the `mlflow-codex`
283+
notify-hook binary. Best-effort: warns and returns False on failure."""
284+
import shutil
285+
286+
if shutil.which("mlflow-codex"):
287+
return True
288+
if not shutil.which("npm"):
289+
print_warning(
290+
f"`npm` is not available to install {CODEX_TRACING_PACKAGE}; "
291+
"Codex tracing will be inactive until it is installed."
292+
)
293+
return False
294+
try:
295+
subprocess.run(["npm", "install", "-g", CODEX_TRACING_PACKAGE], check=True, timeout=300)
296+
except (subprocess.CalledProcessError, subprocess.TimeoutExpired):
297+
print_warning(f"Could not install {CODEX_TRACING_PACKAGE}; Codex tracing will be inactive.")
298+
return False
299+
print_success("Codex MLflow tracing hook installed")
300+
return True
301+
302+
254303
def default_model(state: dict) -> str | None:
255304
"""Pick the newest GPT model when multiple are available.
256305
@@ -279,6 +328,10 @@ def launch(state: dict, tool_args: list[str]) -> None:
279328
workspace = state.get("workspace")
280329
if workspace:
281330
os.environ["OAUTH_TOKEN"] = get_databricks_token(workspace, state.get("profile"))
331+
# The notify hook subprocess Codex spawns inherits this env, so MLflow
332+
# routing flows through to it without writing a separate tracing config.
333+
# When tracing is off this also clears any stale outer-shell value.
334+
apply_tracing_env(os.environ, state, "codex")
282335
os.execvp(binary, [binary, "--profile", CODEX_PROFILE_NAME, *tool_args])
283336

284337

src/ucode/agents/opencode.py

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,14 @@
2323
)
2424
from ucode.state import mark_tool_managed, save_state
2525
from ucode.telemetry import agent_version, ucode_version
26+
from ucode.tracing import agent_tracing, apply_tracing_env
2627

2728
OPENCODE_XDG_CONFIG_HOME = APP_DIR / "opencode-xdg"
2829
OPENCODE_CONFIG_DIR = OPENCODE_XDG_CONFIG_HOME / "opencode"
2930
OPENCODE_CONFIG_PATH = OPENCODE_CONFIG_DIR / "opencode.json"
3031
OPENCODE_BACKUP_PATH = APP_DIR / "opencode-config.backup.json"
3132
OPENCODE_MCP_AUTH_HEADER_VALUE = "Bearer {env:OAUTH_TOKEN}"
33+
OPENCODE_TRACING_PLUGIN = "@mlflow/opencode"
3234

3335
SPEC: ToolSpec = {
3436
"binary": "opencode",
@@ -150,12 +152,29 @@ def write_tool_config(
150152
for stale in ("databricks-anthropic", "databricks-google", "databricks-openai"):
151153
providers.pop(stale, None)
152154
merged = deep_merge_dict(existing, overlay)
155+
_apply_tracing_plugin(merged, state)
153156
write_json_file(OPENCODE_CONFIG_PATH, merged)
154157
state = mark_tool_managed(state, "opencode", managed_keys)
155158
save_state(state)
156159
return state, token
157160

158161

162+
def _apply_tracing_plugin(config: dict, state: dict) -> None:
163+
"""Add/remove the MLflow plugin in opencode.json's top-level ``plugin`` list
164+
to match the current tracing state, leaving any user plugins untouched.
165+
OpenCode auto-installs listed npm plugins at startup."""
166+
plugins = config.get("plugin")
167+
plugins = (
168+
[p for p in plugins if p != OPENCODE_TRACING_PLUGIN] if isinstance(plugins, list) else []
169+
)
170+
if agent_tracing(state, "opencode") is not None:
171+
plugins.append(OPENCODE_TRACING_PLUGIN)
172+
if plugins:
173+
config["plugin"] = plugins
174+
else:
175+
config.pop("plugin", None)
176+
177+
159178
def build_mcp_server_entry(url: str) -> dict:
160179
return {
161180
"type": "remote",
@@ -216,17 +235,21 @@ def _refresh_forever(state: dict, stop_event: threading.Event) -> None:
216235
continue
217236

218237

219-
def build_runtime_env(token: str) -> dict[str, str]:
238+
def build_runtime_env(token: str, state: dict | None = None) -> dict[str, str]:
220239
env = os.environ.copy()
221240
env["OAUTH_TOKEN"] = token
222241
env["XDG_CONFIG_HOME"] = str(OPENCODE_XDG_CONFIG_HOME)
242+
if state is not None:
243+
# apply_tracing_env clears the MLflow vars when tracing is off, so a
244+
# stale outer-shell value can't leak through.
245+
apply_tracing_env(env, state, "opencode")
223246
return env
224247

225248

226249
def launch(state: dict, tool_args: list[str]) -> None:
227250
"""Launch opencode with background token refresh (same pattern as Gemini)."""
228251
token = _refresh_token_once(state)
229-
env = build_runtime_env(token)
252+
env = build_runtime_env(token, state)
230253

231254
stop_event = threading.Event()
232255
refresher = threading.Thread(
@@ -257,4 +280,4 @@ def validate_env(state: dict) -> dict[str, str]:
257280
workspace = state.get("workspace")
258281
if not workspace:
259282
raise RuntimeError("No workspace configured.")
260-
return build_runtime_env(get_databricks_token(workspace, state.get("profile")))
283+
return build_runtime_env(get_databricks_token(workspace, state.get("profile")), state)

0 commit comments

Comments
 (0)