feat: add langsmith setup to trace Claude Code and Codex to LangSmith#156
Draft
Paarth Ahuja (paarth-a) wants to merge 2 commits into
Draft
feat: add langsmith setup to trace Claude Code and Codex to LangSmith#156Paarth Ahuja (paarth-a) wants to merge 2 commits into
langsmith setup to trace Claude Code and Codex to LangSmith#156Paarth Ahuja (paarth-a) wants to merge 2 commits into
Conversation
Adds `langsmith setup {claude,codex,all}` to persistently configure the
LangSmith tracing plugins for coding agents. Writes the agent's local config
at 0600 (Claude Code settings.json marketplace/enabledPlugins/env; Codex
config.toml + langsmith.json) and best-effort installs the plugin marketplace,
so every future agent session traces to a LangSmith project. Requires an API
key (OAuth profiles are not supported by these plugins).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- accept the API key as a positional argument on setup claude/codex/all (langsmith setup claude <api-key> is the whole quick start); a conflicting --api-key errors out - stop writing a project name unless --project or $LANGSMITH_PROJECT is set: the CC_LANGSMITH_PROJECT / codex project keys are deleted when unset so the plugins' own defaults (claude-code / codex) stay the single source of truth, including on re-runs that drop a previously configured project - move Claude Code to the CC_LANGSMITH_* env contract only; self-hosted endpoints are expressed as a CC_LANGSMITH_RUNS_ENDPOINTS replica (omitting projectName when no project is configured) - drop the --no-install marketplace prefetch shell-out; Claude Code installs the declared plugin on next launch - document the default API URL (https://api.smith.langchain.com) in --api-url and setup help; refresh README and rewrite setup tests for the new behavior Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
langsmith setupcommand family that configures coding agents to send full-content traces to LangSmith, with a copy-paste quick start:--api-key,$LANGSMITH_API_KEY, or an API-key profile; passing a conflicting--api-keyalongside the argument errors). Written to the agent config at0600.https://api.smith.langchain.com(now documented in--api-urland setup help). Self-hosted endpoints (--api-url/$LANGSMITH_ENDPOINT) are expressed for Claude Code as aCC_LANGSMITH_RUNS_ENDPOINTSreplica.--project(or$LANGSMITH_PROJECT/$LANGSMITH_AGENT_PROJECT) is set. When unset the key is deleted — including stale values on re-runs — so each plugin's own default (claude-code/codex) stays the single source of truth. Both plugins treat an empty string as an explicit value, hence delete-not-write.CC_LANGSMITH_*namespace exclusively; declares the marketplace + enables the plugin (installs on next launch, no shell-out). Codex enablesplugin_hooksand the tracing plugin inconfig.toml; one manualcodex plugin marketplace addis printed.Test plan
go test ./internal/cmd -run TestSetup— 10 tests: settings/credential merge + permissions, positional-key defaults (no project key written), stale-project removal on re-run, key-conflict error, self-hosted replica JSON with/withoutprojectName, missing-key error, idempotency,setup allsmoke.setup claude <key>writes onlyTRACE_TO_LANGSMITH+CC_LANGSMITH_API_KEY(plugin defaults apply),--project/--api-urlvariants write the expected keys.main):TestTraceMessages_Paginationcan deadlock via thecaptureStdoutpipe helper; fix tracked separately.