Commit 6f8d0f7
feat: capture agent statistics via OTel and surface in safe outputs (#219)
* feat: add OTel JSONL parser for agent statistics
Create src/agent_stats.rs with AgentStats struct that parses Copilot
CLI OpenTelemetry file exporter output. Extracts model, token usage,
duration, tool calls, and turns from the last invoke_agent span and
execute_tool span counts.
Reuses ndjson::read_ndjson_file for JSONL parsing. Includes
to_markdown() renderer with collapsible details block.
9 unit tests including real copilot-otel.jsonl fixture.
Closes: part of #168
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: plumb agent_name and agent_stats through ExecutionContext
- Add agent_name: Option<String> and agent_stats: Option<AgentStats>
to ExecutionContext
- Plumb front_matter.name to ctx.agent_name in main.rs execute handler
- Load otel.jsonl from safe_output_dir, parse into ctx.agent_stats
(non-fatal if missing — just logs debug/warn)
- Add OTel env vars to base.yml AWF step (always-on: COPILOT_OTEL_ENABLED,
COPILOT_OTEL_FILE_EXPORTER_PATH, COPILOT_OTEL_EXPORTER_TYPE)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: append agent stats to 6 safe output write actions
Append a collapsible markdown stats block to safe outputs that produce
human-readable content. Each tool reads include_stats from its typed
config struct (deserialized via ctx.get_tool_config), matching the
existing config pattern used for all other tool options.
Safe outputs with stats:
- create-pull-request (PR description)
- create-work-item (work item description)
- comment-on-work-item (comment body)
- add-pr-comment (PR comment body)
- create-wiki-page (wiki page content)
- update-wiki-page (wiki page content)
Per-tool opt-out via front matter:
safe-outputs:
create-pull-request:
include-stats: false
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: sanitize OTel values, filter internal tools, remove redundant agent_name
Security:
- Sanitize model and agent_name before embedding in markdown stats
block. Strips control chars, neutralizes ##vso[ pipeline commands,
escapes pipe chars that break markdown tables. The OTel file is
writable by the agent inside AWF, so these values are untrusted.
Bug fix:
- Filter out Copilot CLI internal tool spans (report_intent,
permission) from tool_calls count. Only user-visible tool
invocations are counted.
Cleanup:
- Remove redundant agent_name from ExecutionContext — AgentStats
already stores it. Single source of truth.
Docs:
- Document include-stats option for all 6 safe outputs in AGENTS.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: replace <details> tags with plain markdown for ADO compatibility
Azure DevOps does not render HTML <details>/<summary> as collapsible
sections — they display as raw text. Switch to a horizontal rule +
italic heading + table, which renders correctly across ADO PRs, work
items, and wiki pages.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* refactor: compact single-line stats format, drop turns
Replace table with a single line using middle-dot separators:
🤖 _Agent Name_ · model · 45,230 in / 12,450 out · 23 tool calls · 4m 32s
Remove turns from output (low value to operators). Remove unused
total_tokens() method.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: Default impl bugs, dead internal tool filter, sanitization, PR footer order
Bugs fixed:
- CommentOnWorkItemConfig, CreateWikiPageConfig, UpdateWikiPageConfig:
replace #[derive(Default)] with manual Default impls so include_stats
defaults to true (not false from bool default)
- Remove dead "execute_tool permission" from INTERNAL_TOOL_NAMES — the
real OTel span is "permission" (no prefix), already excluded by the
starts_with("execute_tool") predicate
Improvements:
- Strip newlines in sanitize_for_markdown (single-line format)
- Reorder PR description: stats before provenance footer (footer is
the final unambiguous security marker)
- Add trailing newlines to add_pr_comment.rs and create_work_item.rs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: consolidate default_include_stats, plain agent name, doc comments, tests
- Remove italic underscores from agent name in stats line (plain text)
- Fix orphaned doc comment on compute_duration
- Consolidate 5 duplicate default_include_stats() fns into single
pub(crate) fn in agent_stats.rs
- Add 3 unit tests for append_stats_to_body (opt-out, no-stats, with-stats)
- Fix trailing newlines
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent ed93aff commit 6f8d0f7
14 files changed
Lines changed: 592 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1022 | 1022 | | |
1023 | 1023 | | |
1024 | 1024 | | |
| 1025 | + | |
1025 | 1026 | | |
1026 | 1027 | | |
1027 | 1028 | | |
| |||
1053 | 1054 | | |
1054 | 1055 | | |
1055 | 1056 | | |
| 1057 | + | |
1056 | 1058 | | |
1057 | 1059 | | |
1058 | 1060 | | |
| |||
1151 | 1153 | | |
1152 | 1154 | | |
1153 | 1155 | | |
| 1156 | + | |
1154 | 1157 | | |
1155 | 1158 | | |
1156 | 1159 | | |
| |||
1204 | 1207 | | |
1205 | 1208 | | |
1206 | 1209 | | |
| 1210 | + | |
1207 | 1211 | | |
1208 | 1212 | | |
1209 | 1213 | | |
| |||
1417 | 1421 | | |
1418 | 1422 | | |
1419 | 1423 | | |
| 1424 | + | |
1420 | 1425 | | |
1421 | 1426 | | |
1422 | 1427 | | |
| |||
1442 | 1447 | | |
1443 | 1448 | | |
1444 | 1449 | | |
| 1450 | + | |
1445 | 1451 | | |
1446 | 1452 | | |
1447 | 1453 | | |
| |||
0 commit comments