Skip to content

Commit a9650e4

Browse files
jaroslawgajewskiCopilot
jaroslawgajewski
and
Copilot
authored
feat(gh-copilot): close Copilot metrics parity gaps (#8889)
* feat(gh-copilot): close API gaps for per-user metrics, teams, CLI, code review, and PR fields Add missing GitHub Copilot Metrics API fields to achieve full API parity: Enterprise/Org metrics: - CLI active user counts and CLI breakdown (sessions, requests, tokens) - Code review user counts (daily/weekly/monthly × active/passive) - Chat panel mode breakdown (agent/ask/custom/edit/plan/unknown) - Expanded PR metrics (merged, merge time, suggestions, Copilot impact) Per-user metrics: - used_cli, used_copilot_code_review_active/passive boolean flags - CLI breakdown per user (sessions, requests, tokens) User-team mapping (new): - New collector/extractor for user-teams-1-day endpoint - Enables team-level metrics via JOIN with per-user tables Seat assignments: - Team assignment fields (assigning_team id/name/slug) - User detail fields (name, email) Includes migration script 20260527 and comprehensive docs in COPILOT_METRICS_GAPS.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(copilot-metrics): remove outdated metrics gaps documentation and implement new user-team mapping and metrics enhancements Signed-off-by: Jarek <jaroslaw.gajewski@atos.net> --------- Signed-off-by: Jarek <jaroslaw.gajewski@atos.net> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 9e659ab commit a9650e4

19 files changed

Lines changed: 684 additions & 37 deletions

‎backend/plugins/gh-copilot/e2e/metrics/snapshot_tables/_tool_copilot_enterprise_daily_metrics.csv‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
connection_id,scope_id,day,enterprise_id,daily_active_users,weekly_active_users,monthly_active_users,monthly_active_chat_users,monthly_active_agent_users,pr_total_reviewed,pr_total_created,pr_total_created_by_copilot,pr_total_reviewed_by_copilot,user_initiated_interaction_count,code_generation_activity_count,code_acceptance_activity_count,loc_suggested_to_add_sum,loc_suggested_to_delete_sum,loc_added_sum,loc_deleted_sum
2-
1,octodemo,2025-09-01T00:00:00.000+00:00,,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
3-
1,octodemo,2025-09-02T00:00:00.000+00:00,,12,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
1+
connection_id,scope_id,day,enterprise_id,daily_active_users,weekly_active_users,monthly_active_users,monthly_active_chat_users,monthly_active_agent_users,daily_active_cli_users,daily_active_copilot_code_review_users,daily_passive_copilot_code_review_users,weekly_active_copilot_code_review_users,weekly_passive_copilot_code_review_users,monthly_active_copilot_code_review_users,monthly_passive_copilot_code_review_users,chat_panel_agent_mode,chat_panel_ask_mode,chat_panel_custom_mode,chat_panel_edit_mode,chat_panel_plan_mode,chat_panel_unknown_mode,pr_total_reviewed,pr_total_created,pr_total_merged,pr_median_minutes_to_merge,pr_total_suggestions,pr_total_applied_suggestions,pr_total_created_by_copilot,pr_total_reviewed_by_copilot,pr_total_merged_created_by_copilot,pr_total_merged_reviewed_by_copilot,pr_median_min_to_merge_copilot_authored,pr_median_min_to_merge_copilot_reviewed,pr_total_copilot_suggestions,pr_total_copilot_applied_suggestions,user_initiated_interaction_count,code_generation_activity_count,code_acceptance_activity_count,loc_suggested_to_add_sum,loc_suggested_to_delete_sum,loc_added_sum,loc_deleted_sum,cli_session_count,cli_request_count,cli_prompt_count,cli_output_token_sum,cli_prompt_token_sum
2+
1,octodemo,2025-09-01T00:00:00.000+00:00,,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
3+
1,octodemo,2025-09-02T00:00:00.000+00:00,,12,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

‎backend/plugins/gh-copilot/e2e/metrics/snapshot_tables/_tool_copilot_seats.csv‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
connection_id,organization,user_login,user_id,plan_type,created_at,last_activity_at,last_activity_editor,last_authenticated_at,pending_cancellation_date,updated_at
2-
1,octodemo,nathos,4215,enterprise,2023-08-28T23:50:42.000+00:00,2025-11-06T16:12:15.000+00:00,copilot_pr_review,2025-12-04T15:53:22.000+00:00,,2024-02-01T00:00:00.000+00:00
3-
1,octodemo,octocat,1,enterprise,2024-01-10T10:11:12.000+00:00,,vscode/1.0.0/copilot-chat/0.1.0,,,2024-02-02T00:00:00.000+00:00
1+
connection_id,organization,user_login,user_id,user_name,user_email,plan_type,assigning_team_id,assigning_team_name,assigning_team_slug,created_at,last_activity_at,last_activity_editor,last_authenticated_at,pending_cancellation_date,updated_at
2+
1,octodemo,nathos,4215,,,enterprise,0,,,2023-08-28T23:50:42.000+00:00,2025-11-06T16:12:15.000+00:00,copilot_pr_review,2025-12-04T15:53:22.000+00:00,,2024-02-01T00:00:00.000+00:00
3+
1,octodemo,octocat,1,,,enterprise,0,,,2024-01-10T10:11:12.000+00:00,,vscode/1.0.0/copilot-chat/0.1.0,,,2024-02-02T00:00:00.000+00:00

‎backend/plugins/gh-copilot/models/enterprise_metrics.go‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,15 @@ type CopilotCodeMetrics struct {
4444
LocDeletedSum int `json:"locDeletedSum"`
4545
}
4646

47+
// CopilotCliMetrics contains CLI usage breakdown metrics.
48+
type CopilotCliMetrics struct {
49+
CliSessionCount int `json:"cliSessionCount" gorm:"comment:Number of CLI sessions"`
50+
CliRequestCount int `json:"cliRequestCount" gorm:"comment:Number of CLI requests"`
51+
CliPromptCount int `json:"cliPromptCount" gorm:"comment:Number of CLI prompts"`
52+
CliOutputTokenSum int `json:"cliOutputTokenSum" gorm:"comment:Total output tokens from CLI"`
53+
CliPromptTokenSum int `json:"cliPromptTokenSum" gorm:"comment:Total prompt tokens from CLI"`
54+
}
55+
4756
// GhCopilotEnterpriseDailyMetrics captures daily enterprise-level aggregate Copilot metrics.
4857
type GhCopilotEnterpriseDailyMetrics struct {
4958
ConnectionId uint64 `gorm:"primaryKey" json:"connectionId"`
@@ -57,12 +66,43 @@ type GhCopilotEnterpriseDailyMetrics struct {
5766
MonthlyActiveChatUsers int `json:"monthlyActiveChatUsers"`
5867
MonthlyActiveAgentUsers int `json:"monthlyActiveAgentUsers"`
5968

60-
PRTotalReviewed int `json:"prTotalReviewed" gorm:"comment:Total PRs reviewed"`
61-
PRTotalCreated int `json:"prTotalCreated" gorm:"comment:Total PRs created"`
62-
PRTotalCreatedByCopilot int `json:"prTotalCreatedByCopilot" gorm:"comment:PRs created by Copilot"`
63-
PRTotalReviewedByCopilot int `json:"prTotalReviewedByCopilot" gorm:"comment:PRs reviewed by Copilot"`
69+
// CLI active users
70+
DailyActiveCliUsers int `json:"dailyActiveCliUsers" gorm:"comment:Daily active CLI users"`
71+
72+
// Code review user counts
73+
DailyActiveCopilotCodeReviewUsers int `json:"dailyActiveCopilotCodeReviewUsers"`
74+
DailyPassiveCopilotCodeReviewUsers int `json:"dailyPassiveCopilotCodeReviewUsers"`
75+
WeeklyActiveCopilotCodeReviewUsers int `json:"weeklyActiveCopilotCodeReviewUsers"`
76+
WeeklyPassiveCopilotCodeReviewUsers int `json:"weeklyPassiveCopilotCodeReviewUsers"`
77+
MonthlyActiveCopilotCodeReviewUsers int `json:"monthlyActiveCopilotCodeReviewUsers"`
78+
MonthlyPassiveCopilotCodeReviewUsers int `json:"monthlyPassiveCopilotCodeReviewUsers"`
79+
80+
// Chat panel mode breakdown
81+
ChatPanelAgentMode int `json:"chatPanelAgentMode" gorm:"comment:Chat panel agent mode interactions"`
82+
ChatPanelAskMode int `json:"chatPanelAskMode" gorm:"comment:Chat panel ask mode interactions"`
83+
ChatPanelCustomMode int `json:"chatPanelCustomMode" gorm:"comment:Chat panel custom mode interactions"`
84+
ChatPanelEditMode int `json:"chatPanelEditMode" gorm:"comment:Chat panel edit mode interactions"`
85+
ChatPanelPlanMode int `json:"chatPanelPlanMode" gorm:"comment:Chat panel plan mode interactions"`
86+
ChatPanelUnknownMode int `json:"chatPanelUnknownMode" gorm:"comment:Chat panel unknown mode interactions"`
87+
88+
// Pull request metrics (expanded)
89+
PRTotalReviewed int `json:"prTotalReviewed" gorm:"comment:Total PRs reviewed"`
90+
PRTotalCreated int `json:"prTotalCreated" gorm:"comment:Total PRs created"`
91+
PRTotalMerged int `json:"prTotalMerged" gorm:"comment:Total PRs merged"`
92+
PRMedianMinutesToMerge float64 `json:"prMedianMinutesToMerge" gorm:"comment:Median minutes to merge PRs"`
93+
PRTotalSuggestions int `json:"prTotalSuggestions" gorm:"comment:Total PR review suggestions"`
94+
PRTotalAppliedSuggestions int `json:"prTotalAppliedSuggestions" gorm:"comment:Total applied PR suggestions"`
95+
PRTotalCreatedByCopilot int `json:"prTotalCreatedByCopilot" gorm:"comment:PRs created by Copilot"`
96+
PRTotalReviewedByCopilot int `json:"prTotalReviewedByCopilot" gorm:"comment:PRs reviewed by Copilot"`
97+
PRTotalMergedCreatedByCopilot int `json:"prTotalMergedCreatedByCopilot" gorm:"comment:Merged PRs created by Copilot"`
98+
PRTotalMergedReviewedByCopilot int `json:"prTotalMergedReviewedByCopilot" gorm:"comment:Merged PRs reviewed by Copilot"`
99+
PRMedianMinToMergeCopilotAuthored float64 `json:"prMedianMinToMergeCopilotAuthored" gorm:"comment:Median min to merge Copilot-authored PRs"`
100+
PRMedianMinToMergeCopilotReviewed float64 `json:"prMedianMinToMergeCopilotReviewed" gorm:"comment:Median min to merge Copilot-reviewed PRs"`
101+
PRTotalCopilotSuggestions int `json:"prTotalCopilotSuggestions" gorm:"comment:Total Copilot review suggestions"`
102+
PRTotalCopilotAppliedSuggestions int `json:"prTotalCopilotAppliedSuggestions" gorm:"comment:Total Copilot applied suggestions"`
64103

65104
CopilotActivityMetrics `mapstructure:",squash"`
105+
CopilotCliMetrics `mapstructure:",squash"`
66106
common.NoPKModel
67107
}
68108

‎backend/plugins/gh-copilot/models/migrationscripts/20260527_add_copilot_metrics_gaps.go‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package migrationscripts
19+
20+
import (
21+
"time"
22+
23+
"github.com/apache/incubator-devlake/core/context"
24+
"github.com/apache/incubator-devlake/core/errors"
25+
"github.com/apache/incubator-devlake/core/models/migrationscripts/archived"
26+
"github.com/apache/incubator-devlake/helpers/migrationhelper"
27+
)
28+
29+
type addCopilotMetricsGaps struct{}
30+
31+
// --- Enterprise daily metrics: new columns ---
32+
33+
type enterpriseDailyMetrics20260527 struct {
34+
// CLI
35+
DailyActiveCliUsers int
36+
37+
// Code review user counts
38+
DailyActiveCopilotCodeReviewUsers int
39+
DailyPassiveCopilotCodeReviewUsers int
40+
WeeklyActiveCopilotCodeReviewUsers int
41+
WeeklyPassiveCopilotCodeReviewUsers int
42+
MonthlyActiveCopilotCodeReviewUsers int
43+
MonthlyPassiveCopilotCodeReviewUsers int
44+
45+
// Chat panel mode breakdown
46+
ChatPanelAgentMode int
47+
ChatPanelAskMode int
48+
ChatPanelCustomMode int
49+
ChatPanelEditMode int
50+
ChatPanelPlanMode int
51+
ChatPanelUnknownMode int
52+
53+
// Expanded PR metrics
54+
PRTotalMerged int
55+
PRMedianMinutesToMerge float64
56+
PRTotalSuggestions int
57+
PRTotalAppliedSuggestions int
58+
PRTotalMergedCreatedByCopilot int
59+
PRTotalMergedReviewedByCopilot int
60+
PRMedianMinToMergeCopilotAuthored float64
61+
PRMedianMinToMergeCopilotReviewed float64
62+
PRTotalCopilotSuggestions int
63+
PRTotalCopilotAppliedSuggestions int
64+
65+
// CLI breakdown
66+
CliSessionCount int
67+
CliRequestCount int
68+
CliPromptCount int
69+
CliOutputTokenSum int
70+
CliPromptTokenSum int
71+
}
72+
73+
func (enterpriseDailyMetrics20260527) TableName() string {
74+
return "_tool_copilot_enterprise_daily_metrics"
75+
}
76+
77+
// --- User daily metrics: new columns ---
78+
79+
type userDailyMetrics20260527 struct {
80+
UsedCli bool
81+
UsedCopilotCodeReviewActive bool
82+
UsedCopilotCodeReviewPassive bool
83+
84+
// CLI breakdown
85+
CliSessionCount int
86+
CliRequestCount int
87+
CliPromptCount int
88+
CliOutputTokenSum int
89+
CliPromptTokenSum int
90+
}
91+
92+
func (userDailyMetrics20260527) TableName() string {
93+
return "_tool_copilot_user_daily_metrics"
94+
}
95+
96+
// --- Seat: new columns ---
97+
98+
type seat20260527 struct {
99+
UserName string `gorm:"type:varchar(255)"`
100+
UserEmail string `gorm:"type:varchar(255)"`
101+
AssigningTeamId int64
102+
AssigningTeamName string `gorm:"type:varchar(255)"`
103+
AssigningTeamSlug string `gorm:"type:varchar(255)"`
104+
}
105+
106+
func (seat20260527) TableName() string {
107+
return "_tool_copilot_seats"
108+
}
109+
110+
// --- User-teams: new table ---
111+
112+
type userTeam20260527 struct {
113+
ConnectionId uint64 `gorm:"primaryKey"`
114+
ScopeId string `gorm:"primaryKey;type:varchar(255)"`
115+
Day time.Time `gorm:"primaryKey;type:date"`
116+
UserId int64 `gorm:"primaryKey"`
117+
TeamId int64 `gorm:"primaryKey"`
118+
119+
UserLogin string `gorm:"type:varchar(255);index"`
120+
OrganizationId string `gorm:"type:varchar(100)"`
121+
EnterpriseId string `gorm:"type:varchar(100)"`
122+
TeamSlug string `gorm:"type:varchar(255)"`
123+
124+
archived.NoPKModel
125+
}
126+
127+
func (userTeam20260527) TableName() string {
128+
return "_tool_copilot_user_teams"
129+
}
130+
131+
func (script *addCopilotMetricsGaps) Up(basicRes context.BasicRes) errors.Error {
132+
// Add new columns to existing tables
133+
if err := migrationhelper.AutoMigrateTables(basicRes,
134+
&enterpriseDailyMetrics20260527{},
135+
&userDailyMetrics20260527{},
136+
&seat20260527{},
137+
); err != nil {
138+
return err
139+
}
140+
141+
// Create new user-teams table
142+
return migrationhelper.AutoMigrateTables(basicRes,
143+
&userTeam20260527{},
144+
)
145+
}
146+
147+
func (*addCopilotMetricsGaps) Version() uint64 {
148+
return 20260527000000
149+
}
150+
151+
func (*addCopilotMetricsGaps) Name() string {
152+
return "Add Copilot metrics gaps: CLI, code review, chat modes, PR expansion, user-teams"
153+
}

‎backend/plugins/gh-copilot/models/migrationscripts/register.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,5 +30,6 @@ func All() []plugin.MigrationScript {
3030
new(migrateToUsageMetricsV2),
3131
new(addPRFieldsToEnterpriseMetrics),
3232
new(addOrganizationIdToUserMetrics),
33+
new(addCopilotMetricsGaps),
3334
}
3435
}

‎backend/plugins/gh-copilot/models/models.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,5 +45,7 @@ func GetTablesInfo() []dal.Tabler {
4545
&GhCopilotUserMetricsByModelFeature{},
4646
// Seat assignments
4747
&GhCopilotSeat{},
48+
// User-team mappings
49+
&GhCopilotUserTeam{},
4850
}
4951
}

‎backend/plugins/gh-copilot/models/models_test.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ func TestGetTablesInfo(t *testing.T) {
4040
(&GhCopilotUserMetricsByLanguageModel{}).TableName(): false,
4141
(&GhCopilotUserMetricsByModelFeature{}).TableName(): false,
4242
(&GhCopilotSeat{}).TableName(): false,
43+
(&GhCopilotUserTeam{}).TableName(): false,
4344
}
4445

4546
if len(tables) != len(expected) {

‎backend/plugins/gh-copilot/models/seat.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,12 @@ type GhCopilotSeat struct {
2929
Organization string `gorm:"primaryKey;type:varchar(255)"`
3030
UserLogin string `gorm:"primaryKey;type:varchar(255)"`
3131
UserId int64 `gorm:"index"`
32+
UserName string `gorm:"type:varchar(255)" json:"userName"`
33+
UserEmail string `gorm:"type:varchar(255)" json:"userEmail"`
3234
PlanType string `gorm:"type:varchar(32)"`
35+
AssigningTeamId int64 `json:"assigningTeamId" gorm:"comment:Team that assigned the seat"`
36+
AssigningTeamName string `json:"assigningTeamName" gorm:"type:varchar(255)"`
37+
AssigningTeamSlug string `json:"assigningTeamSlug" gorm:"type:varchar(255)"`
3338
CreatedAt time.Time
3439
LastActivityAt *time.Time
3540
LastActivityEditor string

‎backend/plugins/gh-copilot/models/user_metrics.go‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,13 +30,17 @@ type GhCopilotUserDailyMetrics struct {
3030
Day time.Time `gorm:"primaryKey;type:date" json:"day"`
3131
UserId int64 `gorm:"primaryKey" json:"userId"`
3232

33-
OrganizationId string `json:"organizationId" gorm:"type:varchar(100)"`
34-
EnterpriseId string `json:"enterpriseId" gorm:"type:varchar(100)"`
35-
UserLogin string `json:"userLogin" gorm:"type:varchar(255);index"`
36-
UsedAgent bool `json:"usedAgent"`
37-
UsedChat bool `json:"usedChat"`
33+
OrganizationId string `json:"organizationId" gorm:"type:varchar(100)"`
34+
EnterpriseId string `json:"enterpriseId" gorm:"type:varchar(100)"`
35+
UserLogin string `json:"userLogin" gorm:"type:varchar(255);index"`
36+
UsedAgent bool `json:"usedAgent"`
37+
UsedChat bool `json:"usedChat"`
38+
UsedCli bool `json:"usedCli" gorm:"comment:Whether user used Copilot CLI"`
39+
UsedCopilotCodeReviewActive bool `json:"usedCopilotCodeReviewActive" gorm:"comment:Whether user actively used code review"`
40+
UsedCopilotCodeReviewPassive bool `json:"usedCopilotCodeReviewPassive" gorm:"comment:Whether user passively used code review"`
3841

3942
CopilotActivityMetrics `mapstructure:",squash"`
43+
CopilotCliMetrics `mapstructure:",squash"`
4044
common.NoPKModel
4145
}
4246

‎backend/plugins/gh-copilot/models/user_team.go‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package models
19+
20+
import (
21+
"time"
22+
23+
"github.com/apache/incubator-devlake/core/models/common"
24+
)
25+
26+
// GhCopilotUserTeam maps users to teams per day from the user-teams-1-day report.
27+
// This enables team-level metrics aggregation by joining with per-user daily metrics.
28+
type GhCopilotUserTeam struct {
29+
ConnectionId uint64 `gorm:"primaryKey" json:"connectionId"`
30+
ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"`
31+
Day time.Time `gorm:"primaryKey;type:date" json:"day"`
32+
UserId int64 `gorm:"primaryKey" json:"userId"`
33+
TeamId int64 `gorm:"primaryKey" json:"teamId"`
34+
35+
UserLogin string `json:"userLogin" gorm:"type:varchar(255);index"`
36+
OrganizationId string `json:"organizationId" gorm:"type:varchar(100)"`
37+
EnterpriseId string `json:"enterpriseId" gorm:"type:varchar(100)"`
38+
TeamSlug string `json:"teamSlug" gorm:"type:varchar(255)"`
39+
40+
common.NoPKModel
41+
}
42+
43+
func (GhCopilotUserTeam) TableName() string {
44+
return "_tool_copilot_user_teams"
45+
}

0 commit comments

Comments
 (0)