Skip to content

Commit 6f8d0f7

Browse files
feat: capture agent statistics via OTel and surface in safe outputs (#219)
* feat: add OTel JSONL parser for agent statistics Create src/agent_stats.rs with AgentStats struct that parses Copilot CLI OpenTelemetry file exporter output. Extracts model, token usage, duration, tool calls, and turns from the last invoke_agent span and execute_tool span counts. Reuses ndjson::read_ndjson_file for JSONL parsing. Includes to_markdown() renderer with collapsible details block. 9 unit tests including real copilot-otel.jsonl fixture. Closes: part of #168 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: plumb agent_name and agent_stats through ExecutionContext - Add agent_name: Option<String> and agent_stats: Option<AgentStats> to ExecutionContext - Plumb front_matter.name to ctx.agent_name in main.rs execute handler - Load otel.jsonl from safe_output_dir, parse into ctx.agent_stats (non-fatal if missing — just logs debug/warn) - Add OTel env vars to base.yml AWF step (always-on: COPILOT_OTEL_ENABLED, COPILOT_OTEL_FILE_EXPORTER_PATH, COPILOT_OTEL_EXPORTER_TYPE) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: append agent stats to 6 safe output write actions Append a collapsible markdown stats block to safe outputs that produce human-readable content. Each tool reads include_stats from its typed config struct (deserialized via ctx.get_tool_config), matching the existing config pattern used for all other tool options. Safe outputs with stats: - create-pull-request (PR description) - create-work-item (work item description) - comment-on-work-item (comment body) - add-pr-comment (PR comment body) - create-wiki-page (wiki page content) - update-wiki-page (wiki page content) Per-tool opt-out via front matter: safe-outputs: create-pull-request: include-stats: false Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: sanitize OTel values, filter internal tools, remove redundant agent_name Security: - Sanitize model and agent_name before embedding in markdown stats block. Strips control chars, neutralizes ##vso[ pipeline commands, escapes pipe chars that break markdown tables. The OTel file is writable by the agent inside AWF, so these values are untrusted. Bug fix: - Filter out Copilot CLI internal tool spans (report_intent, permission) from tool_calls count. Only user-visible tool invocations are counted. Cleanup: - Remove redundant agent_name from ExecutionContext — AgentStats already stores it. Single source of truth. Docs: - Document include-stats option for all 6 safe outputs in AGENTS.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: replace <details> tags with plain markdown for ADO compatibility Azure DevOps does not render HTML <details>/<summary> as collapsible sections — they display as raw text. Switch to a horizontal rule + italic heading + table, which renders correctly across ADO PRs, work items, and wiki pages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: compact single-line stats format, drop turns Replace table with a single line using middle-dot separators: 🤖 _Agent Name_ · model · 45,230 in / 12,450 out · 23 tool calls · 4m 32s Remove turns from output (low value to operators). Remove unused total_tokens() method. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Default impl bugs, dead internal tool filter, sanitization, PR footer order Bugs fixed: - CommentOnWorkItemConfig, CreateWikiPageConfig, UpdateWikiPageConfig: replace #[derive(Default)] with manual Default impls so include_stats defaults to true (not false from bool default) - Remove dead "execute_tool permission" from INTERNAL_TOOL_NAMES — the real OTel span is "permission" (no prefix), already excluded by the starts_with("execute_tool") predicate Improvements: - Strip newlines in sanitize_for_markdown (single-line format) - Reorder PR description: stats before provenance footer (footer is the final unambiguous security marker) - Add trailing newlines to add_pr_comment.rs and create_work_item.rs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: consolidate default_include_stats, plain agent name, doc comments, tests - Remove italic underscores from agent name in stats line (plain text) - Fix orphaned doc comment on compute_duration - Consolidate 5 duplicate default_include_stats() fns into single pub(crate) fn in agent_stats.rs - Add 3 unit tests for append_stats_to_body (opt-out, no-stats, with-stats) - Fix trailing newlines Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent ed93aff commit 6f8d0f7

14 files changed

Lines changed: 592 additions & 11 deletions

AGENTS.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1022,6 +1022,7 @@ Adds a comment to an existing Azure DevOps work item. This is the ADO equivalent
10221022

10231023
**Configuration options (front matter):**
10241024
- `max` - Maximum number of comments per run (default: 1)
1025+
- `include-stats` - Whether to append agent execution stats to the comment body (default: true)
10251026
- `target` - **Required** — scoping policy for which work items can be commented on:
10261027
- `"*"` - Any work item in the project (unrestricted, must be explicit)
10271028
- `12345` - A specific work item ID
@@ -1053,6 +1054,7 @@ Creates an Azure DevOps work item.
10531054
- `tags` - List of tags to apply
10541055
- `custom-fields` - Map of custom field reference names to values (e.g., `Custom.MyField: "value"`)
10551056
- `max` - Maximum number of create-work-item outputs allowed per run (default: 1)
1057+
- `include-stats` - Whether to append agent execution stats to the work item description (default: true)
10561058
- `artifact-link` - Configuration for GitHub Copilot artifact linking:
10571059
- `enabled` - Whether to add an artifact link (default: false)
10581060
- `repository` - Repository name override (defaults to BUILD_REPOSITORY_NAME)
@@ -1151,6 +1153,7 @@ Note: The source branch name is auto-generated from a sanitized version of the P
11511153
- `labels` - List of labels to apply
11521154
- `work-items` - List of work item IDs to link
11531155
- `max` - Maximum number of create-pull-request outputs allowed per run (default: 1)
1156+
- `include-stats` - Whether to append agent execution stats (token usage, duration, model) to the PR description (default: true)
11541157
11551158
**Multi-repository support:**
11561159
When `workspace: root` and multiple repositories are checked out, agents can create PRs for any allowed repository:
@@ -1204,6 +1207,7 @@ safe-outputs:
12041207
comment-prefix: "[Agent Review] " # Optional — prepended to all comments
12051208
allowed-repositories: [] # Optional — restrict which repos can be commented on
12061209
max: 1 # Maximum per run (default: 1)
1210+
include-stats: true # Append agent stats to comment (default: true)
12071211
```
12081212
12091213
#### reply-to-pr-comment
@@ -1417,6 +1421,7 @@ safe-outputs:
14171421
title-prefix: "[Agent] " # Optional — prepended to the last path segment (the page title)
14181422
comment: "Created by agent" # Optional — default commit comment when agent omits one
14191423
max: 1 # Maximum number of create-wiki-page outputs allowed per run (default: 1)
1424+
include-stats: true # Append agent stats to wiki page content (default: true)
14201425
```
14211426

14221427
Note: `wiki-name` is required. If it is not set, execution fails with an explicit error message.
@@ -1442,6 +1447,7 @@ safe-outputs:
14421447
title-prefix: "[Agent] " # Optional — prepended to the last path segment (the page title)
14431448
comment: "Updated by agent" # Optional — default commit comment when agent omits one
14441449
max: 1 # Maximum number of update-wiki-page outputs allowed per run (default: 1)
1450+
include-stats: true # Append agent stats to wiki page content (default: true)
14451451
```
14461452

14471453
Note: `wiki-name` is required. If it is not set, execution fails with an explicit error message.

0 commit comments

Comments
 (0)