Skip to content

Commit d33f825

Browse files
jaroslawgajewskiCopilot
jaroslawgajewski
and
Copilot
committed
feat(gh-copilot): close API gaps for per-user metrics, teams, CLI, code review, and PR fields
Add missing GitHub Copilot Metrics API fields to achieve full API parity: Enterprise/Org metrics: - CLI active user counts and CLI breakdown (sessions, requests, tokens) - Code review user counts (daily/weekly/monthly × active/passive) - Chat panel mode breakdown (agent/ask/custom/edit/plan/unknown) - Expanded PR metrics (merged, merge time, suggestions, Copilot impact) Per-user metrics: - used_cli, used_copilot_code_review_active/passive boolean flags - CLI breakdown per user (sessions, requests, tokens) User-team mapping (new): - New collector/extractor for user-teams-1-day endpoint - Enables team-level metrics via JOIN with per-user tables Seat assignments: - Team assignment fields (assigning_team id/name/slug) - User detail fields (name, email) Includes migration script 20260527 and comprehensive docs in COPILOT_METRICS_GAPS.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 9e659ab commit d33f825

17 files changed

Lines changed: 860 additions & 10 deletions

‎COPILOT_METRICS_GAPS.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
# GitHub Copilot Metrics Gaps — Changes Documentation
2+
3+
## Overview
4+
5+
This document describes the gaps identified between the DevLake `gh-copilot` plugin and the
6+
current GitHub Copilot Metrics API, and the changes made to close those gaps. All changes
7+
target per-user metrics exposure, correlation data, and full API field parity.
8+
9+
---
10+
11+
## Gap Analysis Summary
12+
13+
The GitHub Copilot Metrics API (new report-download API, replacing the deprecated `/copilot/metrics`
14+
inline endpoint sunset on April 2, 2026) exposes significantly more data than the DevLake plugin
15+
was capturing. The gaps fell into five categories:
16+
17+
| Category | Gap Description | Impact |
18+
|---|---|---|
19+
| **Enterprise/Org Daily Metrics** | Missing CLI user counts, code review user counts, chat panel modes, expanded PR metrics, CLI breakdown | Cannot track CLI adoption, code review usage, chat mode distribution, or detailed PR impact |
20+
| **Per-User Daily Metrics** | Missing `used_cli`, `used_copilot_code_review_active/passive` flags and CLI breakdown | Cannot identify which users use CLI or code review features |
21+
| **User-Team Mapping** | Entirely missing — no collector or model for `user-teams-1-day` endpoint | Cannot correlate user metrics to teams for team-level dashboards |
22+
| **Seat Assignments** | Missing team assignment fields and user name/email | Cannot identify which team assigned a seat or show user display names |
23+
| **Pull Request Metrics** | Only 4 of 14 PR fields captured | Cannot track merged PRs, merge time, Copilot review suggestions, or copilot-authored merge rates |
24+
25+
---
26+
27+
## Changes Made
28+
29+
### 1. Enterprise/Org Daily Metrics Model (`enterprise_metrics.go`)
30+
31+
**New embedded struct: `CopilotCliMetrics`**
32+
- `CliSessionCount` — Number of CLI sessions
33+
- `CliRequestCount` — Number of CLI requests
34+
- `CliPromptCount` — Number of CLI prompts
35+
- `CliOutputTokenSum` — Total output tokens from CLI
36+
- `CliPromptTokenSum` — Total prompt tokens from CLI
37+
38+
**New fields on `GhCopilotEnterpriseDailyMetrics`:**
39+
40+
| Field | Type | API Source | Description |
41+
|---|---|---|---|
42+
| `DailyActiveCliUsers` | int | `daily_active_cli_users` | Daily active CLI users |
43+
| `DailyActiveCopilotCodeReviewUsers` | int | `daily_active_copilot_code_review_users` | Users actively using code review |
44+
| `DailyPassiveCopilotCodeReviewUsers` | int | `daily_passive_copilot_code_review_users` | Users passively receiving code review |
45+
| `WeeklyActiveCopilotCodeReviewUsers` | int | `weekly_active_copilot_code_review_users` | 7-day trailing active code review users |
46+
| `WeeklyPassiveCopilotCodeReviewUsers` | int | `weekly_passive_copilot_code_review_users` | 7-day trailing passive code review users |
47+
| `MonthlyActiveCopilotCodeReviewUsers` | int | `monthly_active_copilot_code_review_users` | 28-day trailing active code review users |
48+
| `MonthlyPassiveCopilotCodeReviewUsers` | int | `monthly_passive_copilot_code_review_users` | 28-day trailing passive code review users |
49+
| `ChatPanelAgentMode` | int | `chat_panel_agent_mode` | Chat panel interactions in agent mode |
50+
| `ChatPanelAskMode` | int | `chat_panel_ask_mode` | Chat panel interactions in ask mode |
51+
| `ChatPanelCustomMode` | int | `chat_panel_custom_mode` | Chat panel interactions in custom mode |
52+
| `ChatPanelEditMode` | int | `chat_panel_edit_mode` | Chat panel interactions in edit mode |
53+
| `ChatPanelPlanMode` | int | `chat_panel_plan_mode` | Chat panel interactions in plan mode |
54+
| `ChatPanelUnknownMode` | int | `chat_panel_unknown_mode` | Chat panel interactions in unknown mode |
55+
| `PRTotalMerged` | int | `pull_requests.total_merged` | Total PRs merged |
56+
| `PRMedianMinutesToMerge` | float64 | `pull_requests.median_minutes_to_merge` | Median minutes to merge PRs |
57+
| `PRTotalSuggestions` | int | `pull_requests.total_suggestions` | Total PR review suggestions |
58+
| `PRTotalAppliedSuggestions` | int | `pull_requests.total_applied_suggestions` | Total applied PR suggestions |
59+
| `PRTotalMergedCreatedByCopilot` | int | `pull_requests.total_merged_created_by_copilot` | Merged PRs created by Copilot |
60+
| `PRTotalMergedReviewedByCopilot` | int | `pull_requests.total_merged_reviewed_by_copilot` | Merged PRs reviewed by Copilot |
61+
| `PRMedianMinToMergeCopilotAuthored` | float64 | `pull_requests.median_minutes_to_merge_copilot_authored` | Median merge time for Copilot-authored PRs |
62+
| `PRMedianMinToMergeCopilotReviewed` | float64 | `pull_requests.median_minutes_to_merge_copilot_reviewed` | Median merge time for Copilot-reviewed PRs |
63+
| `PRTotalCopilotSuggestions` | int | `pull_requests.total_copilot_suggestions` | Total Copilot review suggestions |
64+
| `PRTotalCopilotAppliedSuggestions` | int | `pull_requests.total_copilot_applied_suggestions` | Total Copilot applied suggestions |
65+
| `CopilotCliMetrics` (embedded) | struct | `totals_by_cli.*` | CLI session/request/prompt/token metrics |
66+
67+
### 2. Per-User Daily Metrics Model (`user_metrics.go`)
68+
69+
**New fields on `GhCopilotUserDailyMetrics`:**
70+
71+
| Field | Type | API Source | Description |
72+
|---|---|---|---|
73+
| `UsedCli` | bool | `used_cli` | Whether user used Copilot CLI on this day |
74+
| `UsedCopilotCodeReviewActive` | bool | `used_copilot_code_review_active` | Whether user actively used code review |
75+
| `UsedCopilotCodeReviewPassive` | bool | `used_copilot_code_review_passive` | Whether user passively used code review |
76+
| `CopilotCliMetrics` (embedded) | struct | `totals_by_cli.*` | Per-user CLI session/request/prompt/token metrics |
77+
78+
### 3. User-Team Mapping (NEW — `user_team.go`)
79+
80+
**New model: `GhCopilotUserTeam`** (table: `_tool_copilot_user_teams`)
81+
82+
This is an entirely new data source. The GitHub API `user-teams-1-day` endpoint returns which
83+
teams each user belongs to per day. This enables **team-level metrics aggregation** by joining
84+
with the per-user daily metrics tables.
85+
86+
| Field | Type | API Source | Description |
87+
|---|---|---|---|
88+
| `ConnectionId` | uint64 | — | DevLake connection (PK) |
89+
| `ScopeId` | string | — | DevLake scope (PK) |
90+
| `Day` | time.Time | `day` | Date of the mapping (PK) |
91+
| `UserId` | int64 | `user_id` | GitHub user ID (PK) |
92+
| `TeamId` | int64 | `team_id` | GitHub team ID (PK) |
93+
| `UserLogin` | string | `user_login` | GitHub username |
94+
| `OrganizationId` | string | `organization_id` | Organization ID |
95+
| `EnterpriseId` | string | `enterprise_id` | Enterprise ID |
96+
| `TeamSlug` | string | `slug` | Team slug for display |
97+
98+
**New collector:** `CollectUserTeams` — fetches from `user-teams-1-day` endpoint (JSONL format)
99+
**New extractor:** `ExtractUserTeams` — parses JSONL records into the model
100+
101+
### 4. Seat Assignments (`seat.go`)
102+
103+
**New fields on `GhCopilotSeat`:**
104+
105+
| Field | Type | API Source | Description |
106+
|---|---|---|---|
107+
| `UserName` | string | `assignee.name` | User's display name |
108+
| `UserEmail` | string | `assignee.email` | User's email address |
109+
| `AssigningTeamId` | int64 | `assigning_team.id` | Team that assigned the Copilot seat |
110+
| `AssigningTeamName` | string | `assigning_team.name` | Team display name |
111+
| `AssigningTeamSlug` | string | `assigning_team.slug` | Team URL slug |
112+
113+
### 5. Extractor Updates
114+
115+
All extractors were updated to populate the new fields:
116+
117+
- **`enterprise_metrics_extractor.go`** — Updated `enterpriseDayTotal` struct with new API fields;
118+
updated `pullRequestStats` from 4 to 14 fields; added `totalsByCli` struct; expanded extraction
119+
logic for all new enterprise-level fields.
120+
- **`metrics_extractor.go`** (org metrics) — Same expansion as enterprise extractor since org reports
121+
use the identical format. Updated seat response structs to include `copilotTeam` for team assignment.
122+
- **`user_metrics_extractor.go`** — Updated `userDailyReport` struct with 3 new boolean flags and
123+
CLI breakdown; expanded extraction logic.
124+
- **`seat_extractor.go`** — Updated to extract `UserName`, `UserEmail`, and `AssigningTeam` fields.
125+
126+
### 6. Migration Script
127+
128+
**New file:** `20260527_add_copilot_metrics_gaps.go`
129+
130+
- Adds new columns to `_tool_copilot_enterprise_daily_metrics` (28 columns)
131+
- Adds new columns to `_tool_copilot_user_daily_metrics` (8 columns)
132+
- Adds new columns to `_tool_copilot_seats` (5 columns)
133+
- Creates new table `_tool_copilot_user_teams`
134+
135+
### 7. Registration Updates
136+
137+
- `models.go` — Added `GhCopilotUserTeam` to `GetTablesInfo()`
138+
- `register.go` — Added `CollectUserTeamsMeta` and `ExtractUserTeamsMeta` to subtask list
139+
- `subtasks.go` — Added `CollectUserTeamsMeta` and `ExtractUserTeamsMeta` definitions with dependencies
140+
- `models_test.go` — Updated `TestGetTablesInfo` to include the new table
141+
- `migrationscripts/register.go` — Added `addCopilotMetricsGaps` migration
142+
143+
---
144+
145+
## Files Changed
146+
147+
| File | Change Type | Description |
148+
|---|---|---|
149+
| `models/enterprise_metrics.go` | Modified | Added `CopilotCliMetrics` embedded struct; 28 new fields on enterprise model |
150+
| `models/user_metrics.go` | Modified | Added 3 boolean flags + CLI metrics embed to per-user model |
151+
| `models/seat.go` | Modified | Added 5 team/user detail fields |
152+
| `models/user_team.go` | **Created** | New user-team mapping model |
153+
| `models/models.go` | Modified | Registered `GhCopilotUserTeam` in `GetTablesInfo()` |
154+
| `models/models_test.go` | Modified | Updated table count assertion |
155+
| `models/migrationscripts/20260527_add_copilot_metrics_gaps.go` | **Created** | Migration for all schema changes |
156+
| `models/migrationscripts/register.go` | Modified | Registered new migration |
157+
| `tasks/enterprise_metrics_extractor.go` | Modified | Expanded structs and extraction for 28 new fields |
158+
| `tasks/metrics_extractor.go` | Modified | Expanded org extraction + seat response structs |
159+
| `tasks/user_metrics_extractor.go` | Modified | Added 3 flags + CLI breakdown extraction |
160+
| `tasks/seat_extractor.go` | Modified | Added team and user detail extraction |
161+
| `tasks/user_teams_collector.go` | **Created** | New collector for user-teams-1-day endpoint |
162+
| `tasks/user_teams_extractor.go` | **Created** | New extractor for user-team JSONL records |
163+
| `tasks/subtasks.go` | Modified | Added 2 new subtask metas |
164+
| `tasks/register.go` | Modified | Added 2 new subtasks to execution order |
165+
166+
---
167+
168+
## Correlation Capabilities Enabled
169+
170+
With these changes, the following per-user and cross-dimensional analyses become possible:
171+
172+
1. **User → Team correlation**: Join `_tool_copilot_user_teams` with `_tool_copilot_user_daily_metrics`
173+
on `(day, user_id)` to compute team-level usage aggregations
174+
2. **CLI adoption tracking**: Filter users/days where `used_cli = true` or analyze CLI token usage
175+
3. **Code review adoption**: Track active vs passive code review usage per user per day
176+
4. **Chat mode analysis**: Understand distribution of agent/ask/edit/plan/custom modes
177+
5. **PR impact analysis**: Track Copilot's impact on merge velocity (`median_minutes_to_merge_copilot_authored`
178+
vs `median_minutes_to_merge`) and suggestion acceptance rates
179+
6. **Seat utilization by team**: Join seats with `assigning_team` data for team-level licensing analysis
180+
7. **User identity enrichment**: `user_name` and `user_email` on seats enable richer user profiles
181+
182+
---
183+
184+
## API Endpoints Used
185+
186+
| Endpoint | Existing/New | Purpose |
187+
|---|---|---|
188+
| `enterprises/{e}/copilot/metrics/reports/enterprise-1-day` | Existing | Enterprise daily aggregate metrics |
189+
| `orgs/{o}/copilot/metrics/reports/organization-1-day` | Existing | Org daily aggregate metrics |
190+
| `enterprises/{e}/copilot/metrics/reports/users-1-day` | Existing | Per-user daily metrics |
191+
| `orgs/{o}/copilot/metrics/reports/users-1-day` | Existing | Per-user daily metrics (org scope) |
192+
| `enterprises/{e}/copilot/metrics/reports/user-teams-1-day` | **New** | User-team mapping |
193+
| `orgs/{o}/copilot/metrics/reports/user-teams-1-day` | **New** | User-team mapping (org scope) |
194+
| `orgs/{o}/copilot/billing/seats` | Existing | Seat assignments (expanded fields) |
195+
| `enterprises/{e}/copilot/billing/seats` | Existing | Seat assignments (expanded fields) |
196+
197+
---
198+
199+
## Testing
200+
201+
- All existing unit tests pass (`go test ./plugins/gh-copilot/... -count=1`)
202+
- `TestGetTablesInfo` updated and passing with 19 tables
203+
- `go vet` clean
204+
- Build successful (`go build ./plugins/gh-copilot/...`)

‎backend/plugins/gh-copilot/models/enterprise_metrics.go‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,15 @@ type CopilotCodeMetrics struct {
4444
LocDeletedSum int `json:"locDeletedSum"`
4545
}
4646

47+
// CopilotCliMetrics contains CLI usage breakdown metrics.
48+
type CopilotCliMetrics struct {
49+
CliSessionCount int `json:"cliSessionCount" gorm:"comment:Number of CLI sessions"`
50+
CliRequestCount int `json:"cliRequestCount" gorm:"comment:Number of CLI requests"`
51+
CliPromptCount int `json:"cliPromptCount" gorm:"comment:Number of CLI prompts"`
52+
CliOutputTokenSum int `json:"cliOutputTokenSum" gorm:"comment:Total output tokens from CLI"`
53+
CliPromptTokenSum int `json:"cliPromptTokenSum" gorm:"comment:Total prompt tokens from CLI"`
54+
}
55+
4756
// GhCopilotEnterpriseDailyMetrics captures daily enterprise-level aggregate Copilot metrics.
4857
type GhCopilotEnterpriseDailyMetrics struct {
4958
ConnectionId uint64 `gorm:"primaryKey" json:"connectionId"`
@@ -57,12 +66,43 @@ type GhCopilotEnterpriseDailyMetrics struct {
5766
MonthlyActiveChatUsers int `json:"monthlyActiveChatUsers"`
5867
MonthlyActiveAgentUsers int `json:"monthlyActiveAgentUsers"`
5968

60-
PRTotalReviewed int `json:"prTotalReviewed" gorm:"comment:Total PRs reviewed"`
61-
PRTotalCreated int `json:"prTotalCreated" gorm:"comment:Total PRs created"`
62-
PRTotalCreatedByCopilot int `json:"prTotalCreatedByCopilot" gorm:"comment:PRs created by Copilot"`
63-
PRTotalReviewedByCopilot int `json:"prTotalReviewedByCopilot" gorm:"comment:PRs reviewed by Copilot"`
69+
// CLI active users
70+
DailyActiveCliUsers int `json:"dailyActiveCliUsers" gorm:"comment:Daily active CLI users"`
71+
72+
// Code review user counts
73+
DailyActiveCopilotCodeReviewUsers int `json:"dailyActiveCopilotCodeReviewUsers"`
74+
DailyPassiveCopilotCodeReviewUsers int `json:"dailyPassiveCopilotCodeReviewUsers"`
75+
WeeklyActiveCopilotCodeReviewUsers int `json:"weeklyActiveCopilotCodeReviewUsers"`
76+
WeeklyPassiveCopilotCodeReviewUsers int `json:"weeklyPassiveCopilotCodeReviewUsers"`
77+
MonthlyActiveCopilotCodeReviewUsers int `json:"monthlyActiveCopilotCodeReviewUsers"`
78+
MonthlyPassiveCopilotCodeReviewUsers int `json:"monthlyPassiveCopilotCodeReviewUsers"`
79+
80+
// Chat panel mode breakdown
81+
ChatPanelAgentMode int `json:"chatPanelAgentMode" gorm:"comment:Chat panel agent mode interactions"`
82+
ChatPanelAskMode int `json:"chatPanelAskMode" gorm:"comment:Chat panel ask mode interactions"`
83+
ChatPanelCustomMode int `json:"chatPanelCustomMode" gorm:"comment:Chat panel custom mode interactions"`
84+
ChatPanelEditMode int `json:"chatPanelEditMode" gorm:"comment:Chat panel edit mode interactions"`
85+
ChatPanelPlanMode int `json:"chatPanelPlanMode" gorm:"comment:Chat panel plan mode interactions"`
86+
ChatPanelUnknownMode int `json:"chatPanelUnknownMode" gorm:"comment:Chat panel unknown mode interactions"`
87+
88+
// Pull request metrics (expanded)
89+
PRTotalReviewed int `json:"prTotalReviewed" gorm:"comment:Total PRs reviewed"`
90+
PRTotalCreated int `json:"prTotalCreated" gorm:"comment:Total PRs created"`
91+
PRTotalMerged int `json:"prTotalMerged" gorm:"comment:Total PRs merged"`
92+
PRMedianMinutesToMerge float64 `json:"prMedianMinutesToMerge" gorm:"comment:Median minutes to merge PRs"`
93+
PRTotalSuggestions int `json:"prTotalSuggestions" gorm:"comment:Total PR review suggestions"`
94+
PRTotalAppliedSuggestions int `json:"prTotalAppliedSuggestions" gorm:"comment:Total applied PR suggestions"`
95+
PRTotalCreatedByCopilot int `json:"prTotalCreatedByCopilot" gorm:"comment:PRs created by Copilot"`
96+
PRTotalReviewedByCopilot int `json:"prTotalReviewedByCopilot" gorm:"comment:PRs reviewed by Copilot"`
97+
PRTotalMergedCreatedByCopilot int `json:"prTotalMergedCreatedByCopilot" gorm:"comment:Merged PRs created by Copilot"`
98+
PRTotalMergedReviewedByCopilot int `json:"prTotalMergedReviewedByCopilot" gorm:"comment:Merged PRs reviewed by Copilot"`
99+
PRMedianMinToMergeCopilotAuthored float64 `json:"prMedianMinToMergeCopilotAuthored" gorm:"comment:Median min to merge Copilot-authored PRs"`
100+
PRMedianMinToMergeCopilotReviewed float64 `json:"prMedianMinToMergeCopilotReviewed" gorm:"comment:Median min to merge Copilot-reviewed PRs"`
101+
PRTotalCopilotSuggestions int `json:"prTotalCopilotSuggestions" gorm:"comment:Total Copilot review suggestions"`
102+
PRTotalCopilotAppliedSuggestions int `json:"prTotalCopilotAppliedSuggestions" gorm:"comment:Total Copilot applied suggestions"`
64103

65104
CopilotActivityMetrics `mapstructure:",squash"`
105+
CopilotCliMetrics `mapstructure:",squash"`
66106
common.NoPKModel
67107
}
68108

0 commit comments

Comments
 (0)