AO-427: Add monte-carlo-instrument-agent skill

[jonavila]

Summary

Adds the monte-carlo-instrument-agent skill (AO-427) — a Setup-bucket skill that walks an MC Agent Observability customer through instrumenting a new AI agent in their Python codebase end-to-end. The skill detects AI libraries (LangChain/LangGraph, OpenAI, Anthropic, CrewAI, Bedrock, SageMaker, Vertex AI, plus the long tail via live PyPI fetch), classifies the runtime as serverless or long-running, and proposes mc.setup() plus @trace_with_workflow / @trace_with_task decorator diffs — always asking for explicit per-file approval before any edit.

The skill produces traces; monitoring-advisor consumes them. Sequential, not overlapping — bidirectional routing contract locked in code via symmetric should-not eval cases on both skills.

Plugin version bumped to 1.11.0 in lockstep across all five editor plugins.

What this PR enables

• /instrument-agent slash command in Claude Code — kicks off the workflow against the current Python codebase. Ships via the same SKILL.md routing on Cursor, Codex, Copilot CLI, and OpenCode (each via the editor's standard skill discovery; no slash-command surface in those four).
• Live + fallback library detection. The skill calls scripts/fetch_sdk_docs.py to pull the current supported-instrumentor list from the SDK README on GitHub or from PyPI's info.description (fail-closed to a snapshotted instrumentor_map.json with stale-data warnings if both fail). Live success requires all 8 PRD core libraries present — partial parses fall back rather than ship incomplete coverage.
• Serverless detection. scripts/detect_libraries.py flags serverless.yml / template.yaml / vercel.json / wrangler.toml / Lambda handler patterns / mangum/chalice/zappa/aws-lambda-powertools deps. When detected, the skill proposes the SimpleSpanProcessor variant of mc.setup() — the default BatchSpanProcessor silently drops traces on Lambda, which is documented as the Restructure #1 failure mode in references/troubleshooting.md.
• Idempotent OTLP endpoint normalization. If the user-supplied URL already ends in /v1/traces, use as-is; otherwise append. Never double-append. Resolved final URL is rendered back to the user before code-gen.
• Three V1 redaction pathways — manual mc.create_llm_span with redacted prompts_to_record, env-var disable of prompt/completion capture (the default in setup-template.md), and selective hybrid. Redaction is a gating step before mc.setup() is generated, not a post-hoc consideration.
• Before/after verification via get_agent_metadata — snapshot existing agents before changes; confirm the new agent appears with a fresh MCON after the user runs the instrumented agent. Distinguishes dev/prod twins via MCON, not display name.
• No-silent-edit guardrail covering dependency files, source code, and env files. Mechanically enforced in live evals via must_not_call: [Edit, Write, NotebookEdit] on three guardrail cases — not just judge-rubric prose.
• Bidirectional disambiguation — instrument-agent/trigger-evals.json lifts 3 monitoring-advisor should-trigger prompts as should-not cases; monitoring-advisor/trigger-evals.json adds 3 should-not cases for mc.setup() and SDK-install requests.
• Agent instrumentation Conversation Signal in context-detection (Step 0 fast-path, Step 1 categorize-intent, Step 4 routing) so ambiguous prompts route correctly and clear ones bypass the signal catalog entirely.

Key Decisions

See PRD: instrument-agent Skill for full design context.

• Skill scope vs PRD Bump to 1.0.11: fix skill script paths #4. PRD requirement Bump to 1.0.11: fix skill script paths #4 originally listed three decorators (@trace_with_tags, @trace_with_workflow, @trace_with_task). Per Bayard's DM, @trace_with_tags was explicitly excluded — the skill must never propose it. Tasks-nested-in-workflows already provides the filtering surface MC's evaluation pipeline needs. Strict no-mention gate enforces this (grep -rin "trace_with_tags" skills/instrument-agent/ returns no matches): naming the forbidden token in NEVER callouts puts it in the LLM's context, which combined with the SDK README's Travel Assistant example would have made the negative callout an attractor rather than a deterrent.
• Hybrid version-pin strategy. PyPI live-fetch is the primary source of truth (per Bayard's "don't hardcode versions"); the instrumentor_map.json snapshot ships last-known-compatible pins with a snapshot_date and STALE warning when used; fail-closed if both live-fetch fails AND the snapshot is older than ~6 months. Reconciles Bayard's intent with the reviewer concern that unpinned-latest installs silently break compatibility.
• span_processor kwarg is undocumented in the SDK README. Discovered from monte-carlo-data/saas-serverless/ai-recommendations/common/tracing.py. The skill bakes that knowledge into setup-template.md until the upstream README catches up. Both production examples (ai-agent for long-running, saas-serverless for serverless) are cited inline by URL+SHA.
• Live verification deferred to QA. Phase 3 ran structural-only — programmatic detect_libraries.py smoke against realistic LangGraph fixtures. The live verification path (testConnection + get_agent_metadata against a real MC AO tenant) needs MCP credentials we don't have locally and shouldn't pollute external tenants from a PR. Captured under "Open Questions" in the plan; QA item before/after merge.
• Substring matching is a foot-gun for credential routing. First-round fix for the GitHub-token leak used "github.com" in url; reviewer caught that https://example.com/github.com/path would still leak the token. Fixed by parsing with urllib.parse.urlparse and comparing hostname against an exact allowlist {github.com, raw.githubusercontent.com, api.github.com}.

Test plan

• python3 skills/instrument-agent/scripts/test_detect_libraries.py — 57/57 pass against 11 fixtures (long-running, serverless, mixed manifests, boto3 ambiguity, existing setup, no-deps, etc.).
• python3 plugins/claude-code/evals/run_evals.py --skill instrument-agent --dry-run — 21 trigger-eval cases load.
• python3 plugins/claude-code/evals/run_evals.py --skill monitoring-advisor --dry-run — 36 cases load (3 new should-not for bidirectional routing).
• YAML parse for instrument-agent/live-evals-dev.yaml — 6 behavior cases (silent-edit guardrails, endpoint normalization, serverless detection, before/after verification).
• find plugins -type l -name instrument-agent | wc -l returns 5.
• All 5 plugin manifests at version 1.11.0; matching CHANGELOG entries (claude-code keeps /instrument-agent slash-command claim; the other four use editor-standard discovery wording).
• grep -rin "trace_with_tags" skills/instrument-agent/ returns no matches.
• Regression check: run_evals.py --dry-run against monitoring-advisor / push-ingestion / prevent; YAML parse against monitoring-advisor / prevent / incident-response / context-detection live-evals — all clean.
• QA / pre-merge: live testConnection + get_agent_metadata BEFORE/AFTER cycle against a real MC AO tenant; LLM walks SKILL.md → references → diff-approval interactively against sample_agent and sample_serverless_agent fixtures.
• Smoke test the new /instrument-agent slash command in Claude Code by installing the plugin from this branch.

🤖 Generated with Claude Code

• Ran /code-review

[jonavila]

End-to-end testing summary

Walked the skill end-to-end against two real Python codebases and verified trace
emission with a local OTLP receiver. This surfaced 5 issues, all fixed in 7b0a5ca,
and a clean re-test confirms the fixes work. Recording the test plan + results
here so reviewers don't have to re-derive them.

Setup

• Receiver: otel-desktop-viewer in Docker (ghcr.io/ctrlspice/otel-desktop-viewer:latest-arm64),
  binding ports 4317/4318/8000.
• Long-running fixture: Fresh LangGraph + LangChain + Anthropic project, two-node
  graph (call_llm → summarize), no MC instrumentation up front.
• Serverless fixture: Same shape but as a Lambda handler (def lambda_handler(...)
  • serverless.yml) to trip the skill's serverless detection.
• Pre-flight + verify steps (0, 4, 10) skipped — those are MC-MCP-gated and
  separately validated against real customer engagements.

Resulting trace trees

Long-running (9 spans):

run_agent                         [@trace_with_workflow → workflow=qa-with-summary]
└── invoke_agent LangGraph
    └── LangGraph.workflow
        ├── execute_task call_llm
        │   ├── call_llm          [@trace_with_task → task=answer-question]
        │   └── ChatAnthropic.chat
        └── execute_task summarize
            ├── summarize_answer  [@trace_with_task → task=summarize]
            └── ChatAnthropic.chat

Serverless (5 spans, no LangGraph layer):

lambda_handler           [@trace_with_workflow → workflow=qa-with-summary-lambda]
├── answer_question      [@trace_with_task → task=answer-question]
│   └── ChatAnthropic.chat
└── summarize            [@trace_with_task → task=summarize]
    └── ChatAnthropic.chat

Both confirm the workflow span as root, task spans nested correctly,
langchain auto-instrumentor producing LLM spans inside the task,
montecarlo.workflow / montecarlo.task propagating to all descendants,
and service.name matching mc.setup(agent_name=…).

Findings (all fixed in 7b0a5ca)

#	Issue	Fix
1	detect_libraries.py dropped version_constraint from the snapshot when emitting suggested_instrumentors	Pass through version_constraint; workflow step 6 + library-detection.md updated to require pinning
2	OpenLLMetry instrumentors at <=0.53.4 pass module= to wrap_function_wrapper, which wrapt 2.x renamed to target= — without a wrapt<2 pin, mc.setup() raises TypeError at import	New additional_pins: ["wrapt<2"] field in instrumentor_map.json, flowed through suggested_instrumentors
3	fetch_sdk_docs.py GitHub README 404 (private repo) printed "Live fetch failed" — misleading since PyPI fallback worked	Reword warning + hint at GITHUB_TOKEN for private-repo access
4	existing_setup detection false-positive on files that import montecarlo_opentelemetry only to use decorators (handlers, not setup files)	Tighten _detect_existing_setup to require both an SDK import AND an actual mc.setup(...) call
5	No documented local verification path for customers iterating in dev	Document otel-desktop-viewer (or any local OTLP receiver) as an optional pre-MC sanity check in verify-traces.md

Re-test from clean state

Rolled both fixtures back to uninstrumented (deleted tracing.py, removed
decorators, restored unpinned requirements.txt, recreated venvs) and
re-walked the skill workflow end-to-end. This time the skill's
suggested_instrumentors output drove the pinning automatically — no
manual intervention.

Check	Pre-fix (manual debug)	Post-fix (skill-driven)
pip install -r requirements.txt	Failed twice (wrapt 2.x → 0.60 instrumentor → final pinned set)	Clean first try
Long-running trace	9 spans, correct hierarchy	9 spans, identical hierarchy
Lambda trace	5 spans, correct hierarchy	5 spans, identical hierarchy
existing_setup.files (after instrumentation)	["handler.py", "tracing.py"] (handler false-positive)	["tracing.py"] only

What this doesn't cover

• MC-MCP-gated steps (0/4/10) — separately validated.
• Real Lambda freeze behavior — local python handler.py flushes via process
  exit, not via Lambda runtime suspend. The SimpleSpanProcessor choice is
  structurally correct (vs. BatchSpanProcessor losing spans at freeze), but
  freeze-tolerance has to be tested in a real Lambda environment.

[jonavila]

6th finding (after-the-fact follow-up)

While walking through the trace UI to take a closer look at the spans, I noticed:

gen_ai.prompt.0.content: What is the capital of France?
gen_ai.completion.0.content: The capital of France is Paris.

Despite the long-running template's comment claiming "By default this template DOES NOT capture prompt/completion content. To capture prompts ... set OTEL_INSTRUMENTATION_LANGCHAIN_TRACE_PROMPTS=true", the spans had full prompt and completion text.

Root cause: the env var name OTEL_INSTRUMENTATION_LANGCHAIN_TRACE_PROMPTS does not exist in the OpenLLMetry instrumentors at <=0.53.4 (verified by grepping the installed source for langchain, openai, and anthropic instrumentors). They read a single env var, TRACELOOP_TRACE_CONTENT, defaulted to "true". So the template's comment was both naming a non-existent env var and wouldn't have flipped the default even if the var did exist.

Fix (in 62e94d5):

• Templates now set os.environ.setdefault("TRACELOOP_TRACE_CONTENT", "false") at module top, before importing any instrumentor. Privacy default-off is a code guarantee, not just a comment.
• Operators who want capture override by exporting TRACELOOP_TRACE_CONTENT=true.
• setup-template.md Section 3 rewritten to explain the actual mechanism with a CRITICAL callout: a comment alone is not the privacy default.
• redaction.md drops the four non-existent per-instrumentor env vars; documents TRACELOOP_TRACE_CONTENT as the single source of truth for the OpenLLMetry instrumentor family.

Empirical verification: re-running both fixtures with the corrected template, ChatAnthropic.chat spans have:

	Before fix	After fix
gen_ai.prompt.0.content	"What is the capital of France?"	(absent)
gen_ai.completion.0.content	"The capital of France is Paris."	(absent)
gen_ai.usage.input_tokens	14	14
gen_ai.usage.output_tokens	10	10
gen_ai.request.model	claude-haiku-4-5-...	claude-haiku-4-5-...

Privacy property restored. Tests still green (107 + 42).

This is the kind of bug only e2e testing catches — the unit tests assert on script output, not on the instrumented spans' actual content.

[jonavila]

Review feedback addressed in 709107a

Thanks for the thorough pass, @bayardneville — all 44 threads are answered inline. Summary of the themes:

Factual corrections

• Data residency: trace content lives in the customer's environment; the MC-hosted collector is a write-back pass-through. Reflected in redaction.md, setup-template.md, workflow.md, SKILL.md.
• Capture-on is the canonical default (PRD value proposition). Redaction is opt-in for customers with stricter privacy requirements. Workflow Step 3 question + SKILL.md routing + default template all reflect this.
• Redaction model rewritten: under auto-instrumentation, TRACELOOP_TRACE_CONTENT=false is a mandatory prerequisite for any redaction (else duplicate spans). Manual mc.create_llm_span with placeholder-substitution is an optional additional layer on top. Pathway/Strategy C is removed entirely. Hash + length is replaced with placeholder-substitution.
• No "PRD core list" — library-detection.md is single-tier (whatever PyPI shows). The eight previously-called-core libraries are presented as non-exhaustive examples that defer to PyPI.
• Decorator/manual-span availability decoupled from auto-instrumentor presence (they're independent).

Structural changes

• setup-template.md Section 1: four self-contained templates (long-running, serverless+MC-hosted+MCD_DEFAULT, serverless+MC-hosted+OTEL_HEADERS, serverless+self-hosted). No more conditional-callouts variant block.
• fetch_sdk_docs.py refactored to PyPI-only: -292 lines. Dropped GitHub fetch (private repo always 404s), snapshot fallback (per your + samkcrespo's confirmation), PRD_CORE_LIBRARIES. Fails closed on PyPI errors. instrumentor_map.json reframed as the static detection map for detect_libraries.py only (no version-pin fallback role).
• Pre-flight degradation: MC MCP availability is no longer required. If test_connection fails, the skill proceeds and asks the customer to verify in the MC UI manually.

Polish

• All private-repo links (saas-serverless, ai-agent, montecarlo-opentelemetry) removed. Public docs source is now https://pypi.org/project/montecarlo-opentelemetry/.
• verify-traces.md timing bumped to 10–15 minutes for first-time / low-traffic / dev agents.
• Decorator-placement: softened task-name rule (function name OK when meaningful); "LLM-calling or tool-calling" wording.
• Plugin CHANGELOGs slimmed across all five (claude-code, codex, copilot, cursor, opencode). Dropped "### Fixed" sections (this is the first release). README.md + skills/README.md skill rows slimmed.
• Added .github/workflows/instrument-agent-tests.yml — runs test_detect_libraries.py (107 cases) + test_fetch_sdk_docs.py (33 cases) on every PR/push touching skills/instrument-agent/**. Replaced test_fixtures/no-deps/.gitkeep with an explanatory README.

All 140 tests pass locally. CI should pick them up on this push.

[bayardneville]

From Claude:

Verification re-review of 65b3fd6 (delta 62e94d5..65b3fd6). All 7 prior BLOCKERs and 11 prior ISSUEs are addressed. Three new ISSUEs and four lower-priority items from the rewrite, posted as inline comments. Full review doc at .work/AO-427/reviews/ao-427-build-v1-instrument-agent-skill.review.2026-05-12-65b3fd6.md.

[jonavila]

Round 2 review fixes — bayardneville (2026-05-12)

Pushed fa23099 addressing today's six comments.

#	Verdict	Where
F1 [ISSUE] Template 2 partial-config crash	Fixed	setup-template.md — os.environ.get + skip-and-log block
F2 [ISSUE] Serverless TRACELOOP splice point unclear	Fixed	setup-template.md — CRITICAL callout + concrete patched Template 2 snippet showing the module-scope splice
F3 [ISSUE] Recommend deleting instrumentor_map.json	Fixed	Map deleted. detect_libraries.py rewritten as a thin discovery surface (dependencies[] + runtime/serverless_signals/existing_setup). LLM now does AI-library disambiguation against fetch_sdk_docs.py's live PyPI list. wrapt<2 guidance moved to troubleshooting.md as a symptom-driven fix. library-detection.md, workflow.md, and SKILL.md updated to the new contract.
F7 [NIT] MC_OTEL_ENDPOINT vs OTEL_ENDPOINT inconsistency	Fixed	setup-template.md — standardized on OTEL_ENDPOINT across all four templates
F9 [SUGGESTION] Move tests + fixtures to sibling tests/	Done	skills/instrument-agent/tests/{test_*.py, fixtures/}. CI workflow paths updated; both suites still pass (53 + 33 checks).
redaction.md:27 "what default is this referring to?"	Fixed	Rewrote to clarify the SDK has no built-in default; the customer always supplies it (templates resolve via OTEL_ENDPOINT env var).

Diff stats: 33 files changed, 535 insertions(+), 862 deletions(-).

Local verification: skills/instrument-agent/tests/test_detect_libraries.py (53 passed), skills/instrument-agent/tests/test_fetch_sdk_docs.py (33 passed). CI should pick up the same.