feat(uipath-platform): add traces_fetch smoke test + traces_e2e full round-trip test

[saksharthakkar]

Motivation

Adds two tests for `uip traces spans get` — the CLI command for fetching LLM trace spans from agent jobs.

Summary

`traces_fetch.yaml` — smoke (`skill-platform-traces-fetch`)

• Tags: `[uipath-platform, smoke, lifecycle:discover]`
• Goal-only prompt: agent fetches spans for a placeholder GUID; 404 is acceptable — tests command knowledge, not data
• 2 success criteria: correct command + `--job-key` flag, `--output json`

`traces_e2e.yaml` + `check_traces_e2e.py` — full round-trip (`skill-platform-traces-e2e`)

• Tags: `[uipath-platform, e2e, lifecycle:discover]`
• Goal-only prompt: "Verify that the published agent produces LLM trace spans" — skill teaches start→wait→fetch
• First test in the repo that proves the CLI returns real spans from a real job
• Process key supplied via `TRACES_SMOKE_PROCESS_KEY` GitHub secret (no hardcoded GUIDs)
• 5 success criteria including `run_command` via `check_traces_e2e.py` (weight 5.0 — the real gate)

Workflow change: `smoke-skills.yml` now injects `TRACES_SMOKE_PROCESS_KEY` into `$GITHUB_ENV` so the agent sandbox picks it up at runtime.

Prompt style: Both prompts follow goal-statement style matching the repo pattern — no CLI commands, no procedural steps. Skill teaches the workflow.

Test Results

traces_e2e full round-trip — score 1.000 ✅

Verified 2026-05-05 locally against `codereval/DefaultTenant`:

status=SUCCESS  score=1.000  duration=73.1s  iterations=1  5/5 criteria passed

Job: traces-smoke-agent → State: Successful (8s)
Job key: 216e601d-59cd-42d3-8fce-f93262b1d307
Spans: 1 — "LLM call" (gpt-4.1-mini-2025-04-14, 35 tokens)
check_traces_e2e.py → OK: 1 span(s) returned

Agent discovered the process via `uip or processes list`, started the job, fetched spans — all from the skill, no procedure in the prompt.

traces_fetch smoke — score 1.000 ✅

Verified locally — 2/2 criteria passed, ~20s.

Test plan

• `traces_fetch` smoke passes locally (score 1.000)
• `traces_e2e` full round-trip passes locally (score 1.000, 5/5 criteria)
• `check_traces_e2e.py` exits 0 with real data
• No hardcoded GUIDs — process key via `TRACES_SMOKE_PROCESS_KEY` secret
• Goal-only prompts — no CLI commands, no procedural steps (per review feedback)
• Login criterion removed — CI always authenticates via env vars before tests run
• Both pass in CI on merge

🤖 Generated with Claude Code

[github-actions]

Reviewing PR #480

• Gather context (review criteria, project conventions, existing comments)
• Read the full changed file
• Analyze diff against origin/main
• Check test task conventions and structure
• Post review findings

View job run

[uipreliga]

I approved, but please can you fix the issue below? The prompt is way too detailed.

[github-actions]

Claude finished @saksharthakkar's task in 2m 39s —— View job

---

Coder-eval task lint (advisory)

2 task YAMLs changed; verdict counts: 0 Critical, 0 High, 0 Medium, 1 Low, 1 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

• ✅ Author confirms passing run: "traces_e2e full round-trip — score 1.000 ✅" / "Verified 2026-05-05 locally against codereval/DefaultTenant" and "traces_fetch smoke — score 1.000 ✅" / "Verified via previous run — 2/2 criteria passed, ~20s."

Per-task lint

tests/tasks/uipath-platform/traces_fetch.yaml — verdict: Low

Issues:

• [Low] Meaningful coverage (lines 20–35): both criteria are command_executed only — no output validation. Acceptable for a smoke test with a placeholder GUID (404 expected), but thinner than sibling integration-service tests which pair command_executed with run_command for structural validation. The e2e companion provides the real correctness gate.

Suggested fixes:

• Consider adding a lightweight run_command criterion that asserts the CLI produced JSON output (even a 404 response is parseable JSON with Result and Message keys), e.g. a one-liner python3 -c "import json,sys; ..." that checks the command's stderr/stdout was valid JSON. This avoids self-reporting while adding a minimal correctness signal.

---

tests/tasks/uipath-platform/traces_e2e.yaml — verdict: OK

Prompt is now goal-only (good — procedural steps removed in 0be95c5). Five criteria with a strong mix: command_executed for CLI usage verification + file_exists + run_command via check_traces_e2e.py (weight 5.0) that validates real JSON structure and span count. Not gameable without actually running the commands against a real tenant.

Within-PR duplicates

• No duplicate clusters detected. traces_fetch (smoke — command knowledge with placeholder GUID) and traces_e2e (e2e — full round-trip with real job and span validation) are a complementary smoke/e2e pair testing materially different operations.

Conclusion

• ⚠ 1 task has issues, max severity Low. Advisory only — not blocking merge. The Low finding on traces_fetch.yaml (meaningful coverage) is expected for a smoke test and mitigated by the e2e companion.