Copilot Metrics Adoption Dashboard [GA] (#62261)

Co-authored-by: Joe Clark <31087804+jc-clark@users.noreply.github.com>
Co-authored-by: Melanie Yarbrough <11952755+myarb@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72df5fa0-bb85-4fc7-a3e1-881c7060f94a

Author: 4 people

content/copilot/concepts/copilot-usage-metrics/copilot-metrics.md

@@ -31,6 +31,7 @@ Metrics are available through:
 * The {% data variables.product.prodname_copilot_short %} usage metrics APIs, which provide detailed, exportable data at the enterprise, organization, repository, and user levels.
 * The {% data variables.product.prodname_copilot_short %} usage metrics dashboard, which visualizes 28-day usage trends across your enterprise and organizations.
 * The code generation dashboard, which breaks down how code is being generated by users and agents across your enterprise and organizations.
+* The {% data variables.product.prodname_copilot_short %} impact dashboard, which groups users into adoption cohorts and connects that adoption to pull request output.
 * The {% data variables.product.prodname_copilot_short %} usage metrics NDJSON export, which offers raw data for custom BI tools or long-term storage.
 
 ## Which usage is included?
@@ -94,15 +95,15 @@ To be included in the {% data variables.product.prodname_copilot_short %} usage
 
 ## Data freshness
 
-The data in the {% data variables.product.prodname_copilot_short %} usage metrics dashboard and API reports is updated on a regular schedule.
+The data in the dashboards and API reports is updated on a regular schedule.
 
 You can expect data to be available within **two full days**. This means that data for a given day is processed and made available within two full UTC days after that day closes.
 
 ## What does the data measure?
 
 {% data variables.product.prodname_copilot_short %} usage metrics can be grouped into a few main categories: Adoption, engagement, acceptance rate, Lines of Code (LoC), and pull request lifecycle metrics.
 
-**Adoption** measures how many licensed developers are actively using {% data variables.product.prodname_copilot_short %}. For example, daily active users (DAU) tells you how many unique users interacted with {% data variables.product.prodname_copilot_short %} on a given day. Ideally, you'll see a consistent upward trend in these metrics during rollout. {% data variables.copilot.copilot_code-review_short %} adoption is tracked separately, with distinct active and passive user counts. Active users manually requested a review or applied a suggestion; passive users had {% data variables.copilot.copilot_code-review_short %} automatically assigned to review their pull request. When a user has both signals in the same period, they are counted as active only.
+**Adoption** measures how many licensed developers are actively using {% data variables.product.prodname_copilot_short %}. For example, daily active users (DAU) tells you how many unique users interacted with {% data variables.product.prodname_copilot_short %} on a given day. Ideally, you'll see a consistent upward trend in these metrics during rollout. {% data variables.copilot.copilot_code-review_short %} adoption is tracked separately, with distinct active and passive user counts. Active users manually requested a review or applied a suggestion; passive users had {% data variables.copilot.copilot_code-review_short %} automatically assigned to review their pull request. When a user has both signals in the same period, they are counted as active only. For a deeper view of adoption depth, users are also grouped into adoption cohorts. See [Understanding adoption cohorts](#understanding-adoption-cohorts).
 
 **Engagement** measures describe how deeply developers use {% data variables.product.prodname_copilot_short %} once they’ve adopted it. Key engagement metrics show not only frequency of use but also breadth across features. For example, average chat requests per active user measures how often users open and interact with {% data variables.copilot.copilot_chat_short %}. You'd want to see regular and increasing chat use across languages and IDEs.
 
@@ -112,6 +113,35 @@ You can expect data to be available within **two full days**. This means that da
 
 **Pull request lifecycle metrics** measure how {% data variables.product.prodname_copilot_short %} activity relates to pull request outcomes and delivery flow. These metrics include pull request creation and merge counts, median time to merge, and review suggestion activity. By comparing overall pull request activity with pull requests created by {% data variables.product.prodname_copilot_short %}, you can evaluate how AI-assisted workflows influence throughput and cycle time at the organization or enterprise level.
 
+### Understanding adoption cohorts
+
+Rather than measuring adoption as a single active-user count, {% data variables.product.prodname_copilot_short %} groups users into **adoption cohorts** based on how they engage with {% data variables.product.prodname_copilot_short %}, not just whether they engage at all. A flat active-user count treats someone who occasionally accepts a code completion the same as someone who orchestrates multiple agent-driven workflows throughout the day. Cohorting separates these two users, so you can see whether your organization's usage is deepening over time rather than plateauing at initial trial.
+
+Users are grouped into the following phases:
+
+| Phase | What it represents |
+|:--|:--|
+| Passive users | The user has not met the engagement threshold for a phase during the period. In the underlying data, this phase is labeled `No Cohort`. |
+| Phase 1: Code first | The user engaged with code completions and/or IDE agent mode. |
+| Phase 2: Agent first | The user engaged with a single {% data variables.product.github %}-based agent surface, such as {% data variables.copilot.copilot_cloud_agent %}, {% data variables.copilot.copilot_code-review_short %}, or {% data variables.copilot.copilot_cli_short %}. Only active engagement with {% data variables.copilot.copilot_code-review_short %} counts here; having it automatically assigned to review a pull request without further engagement does not. |
+| Phase 3: Multi-agent | The user engaged with two or more {% data variables.product.github %}-based agent surfaces, or with the {% data variables.copilot.github_copilot_app %}. |
+
+To be grouped into a phase, a user must meet an engagement threshold of at least two active days out of the trailing 28-day window, using the surfaces associated with that phase. A user only needs two qualifying days on a higher phase's surfaces to progress to that phase. Agent usage alone can qualify a user, without separate days of plain completions.
+
+Phase assignment is recalculated daily from the trailing 28-day window, so a user's phase can shift from one day to the next as their activity within that window changes. This is expected behavior, not a data error.
+
+Users who haven't met any phase's threshold are grouped into **Passive users**, which is not a measure of inactivity, but a signal that a user's engagement hasn't yet reached the level needed to reliably classify their adoption depth. For example, a user with fewer than two active days in the trailing 28-day window, or one who only lightly uses a surface like {% data variables.copilot.copilot_chat_short %} on {% data variables.product.prodname_dotcom_the_website %}, shows as a passive user.
+
+Because phase assignment is based on **which surfaces a user engages with**, not simply how many actions they take, a user with high completion volume but no agent usage stays in Phase 1, while a user with lighter but broader usage across agent surfaces progresses to Phase 2 or Phase 3.
+
+For the underlying schema fields, see [AI adoption phase fields](/copilot/reference/copilot-usage-metrics/copilot-usage-metrics#ai-adoption-phase-fields).
+
+The impact dashboard's **adoption multiplier** divides the average amount of pull requests merged for engaged users, which are ones in Phase 1, 2, or 3, by the amount of pull requests merged for Passive users.
+
+This shows the relative impact of deeper adoption, independent of how many users fall into each phase. This is a measurement to understand the potential lift that becoming a more engaged user can provide.
+
+For example, if users in the engaged cohorts averaged 20 pull requests/user/month and the passive cohort averaged 10, then the equation is 20 / 10 = 2x multiplier.
+
 ### Interpreting pull request lifecycle metrics across scopes
 
 Pull request lifecycle metrics are available at both the organization and enterprise level. When comparing reports, keep the following in mind:
@@ -132,6 +162,8 @@ These metrics can be used together to answer key questions about your teams' usa
 | Are enablement efforts working? | Growth in adoption and engagement after training or communication campaigns |
 | Is {% data variables.product.prodname_copilot_short %} influencing delivery speed or pull request throughput? | Pull request merge counts and median time to merge |
 | How is {% data variables.copilot.copilot_code-review_short %} being adopted? | Active versus passive code review user counts |
+| Is my organization's adoption deepening over time? | Adoption cohort distribution and engagement trends |
+| How do I act on the insights from the dashboard? | See [AUTOTITLE](/copilot/reference/copilot-usage-metrics/interpret-copilot-metrics#reviewing-adoption-cohorts) |
 
 Look for patterns across these signals rather than focusing on any single number. For example, a steady DAU paired with a rising acceptance rate indicates growing trust and value.
 
@@ -141,5 +173,6 @@ Now that you understand what each {% data variables.product.prodname_copilot_sho
 
 * To view adoption and usage trends across your enterprise and organizations, see [AUTOTITLE](/copilot/how-tos/administer-copilot/view-usage-and-adoption).
 * To analyze how code is being generated by users and agents, see [AUTOTITLE](/copilot/how-tos/administer-copilot/view-code-generation).
+* To view adoption cohorts and depth of adoption, see [AUTOTITLE](/copilot/how-tos/administer-copilot/view-impact-dashboard).
 * To access {% data variables.product.prodname_copilot_short %} usage metrics programmatically, see [AUTOTITLE](/rest/copilot/copilot-usage-metrics).
 * To construct team-level metrics by aggregating per-user records, see [AUTOTITLE](/copilot/reference/copilot-usage-metrics/team-level-metrics).

content/copilot/concepts/copilot-usage-metrics/index.md

@@ -13,6 +13,7 @@ children:
   - /copilot-metrics
   - /content/copilot/how-tos/administer-copilot/view-usage-and-adoption
   - /content/copilot/how-tos/administer-copilot/view-code-generation
+  - /content/copilot/how-tos/administer-copilot/view-impact-dashboard
   - /content/copilot/reference/copilot-usage-metrics/copilot-usage-metrics
   - /content/copilot/reference/copilot-usage-metrics/interpret-copilot-metrics
   - /content/copilot/reference/copilot-usage-metrics/reconciling-usage-metrics

content/copilot/how-tos/administer-copilot/index.md

@@ -11,6 +11,7 @@ children:
   - /download-activity-report
   - /view-usage-and-adoption
   - /view-code-generation
+  - /view-impact-dashboard
 redirect_from:
   - /copilot/how-tos/administer
 contentType: how-tos

content/copilot/how-tos/administer-copilot/view-impact-dashboard.md

@@ -0,0 +1,32 @@
+---
+title: Viewing the Copilot impact dashboard
+shortTitle: View impact dashboard
+intro: 'The impact dashboard shows how deeply your organization has adopted {% data variables.product.prodname_copilot_short %}, and how that adoption connects to pull request output.'
+permissions: '{% data reusables.copilot.usage-metrics-permissions %}'
+versions:
+  feature: copilot
+contentType: how-tos
+allowTitleToDifferFromFilename: true
+category:
+  - Copilot usage metrics
+  - View metrics
+  - Track Copilot usage
+---
+
+Instead of a flat active-user count, the impact dashboard groups users into adoption cohorts based on how they engage with {% data variables.product.prodname_copilot_short %}, and connects that engagement to pull request throughput. This gives you a more meaningful signal of adoption depth than daily or weekly active user counts alone.
+
+For a detailed explanation of what the dashboard shows, including adoption cohorts, engagement trends, and recommendations, see [AUTOTITLE](/copilot/concepts/copilot-usage-metrics/copilot-metrics).
+
+## Prerequisite
+
+{% data reusables.copilot.copilot-metrics-dashboard-prereq %}
+
+## Accessing the dashboard
+
+{% data reusables.copilot.access-copilot-metrics-dashboard %}
+1. In the left sidebar, click **{% data variables.product.prodname_copilot_short %} impact**.
+
+## Next steps
+
+* To learn how to interpret cohort shifts and what action to take, see [AUTOTITLE](/copilot/reference/copilot-usage-metrics/interpret-copilot-metrics#reviewing-adoption-cohorts).
+* For raw per-team NDJSON data, such as for custom BI reporting, see [AUTOTITLE](/copilot/reference/copilot-usage-metrics/team-level-metrics).

content/copilot/reference/copilot-usage-metrics/copilot-usage-metrics.md

@@ -45,6 +45,17 @@ These metrics appear directly in the {% data variables.product.prodname_copilot_
 | Most used chat model | The most frequently used chat model in the last 28 days. |
 | Requests per chat mode | Number of chat requests by mode (ask, edit, plan, agent). |
 
+## Impact dashboard metrics
+
+These metrics appear in the impact dashboard, which groups users into adoption cohorts and connects that adoption to pull request output. For guidance on interpreting adoption cohorts, see [AUTOTITLE](/copilot/concepts/copilot-usage-metrics/copilot-metrics#understanding-adoption-cohorts).
+
+| Metric | Description |
+|:--|:--|
+| Engagement trends | How the organization's adoption cohort mix and pull request throughput (total pull requests merged per month) have changed over the last six months. |
+| Adoption cohort distribution | Share of licensed users grouped into each adoption phase (Passive users, Phase 1, Phase 2, Phase 3) for the period. Passive users correspond to the `No Cohort` value in the API. |
+| Adoption multiplier | Compares engaged users (Phase 1, 2, or 3) against passive users on code shipped (pull requests merged per user per month) and time to merge pull requests, to show the relative impact of deeper {% data variables.product.prodname_copilot_short %} adoption. |
+| Recommendations | Suggested actions, such as configuring {% data variables.copilot.copilot_cloud_agent %} or enabling {% data variables.copilot.copilot_code-review_short %}, based on the organization's current cohort distribution. |
+
 ## Code generation dashboard metrics
 
 These metrics appear in the code generation dashboard and provide a breakdown of how code is being generated across user-initiated and agent-initiated activity. All values are derived from lines of code (LoC) added or deleted in the IDE.
@@ -278,6 +289,21 @@ The `pull_requests` object appears in aggregated enterprise and organization rep
 
 {% data variables.product.prodname_copilot_short %} groups users into AI adoption phases based on their activity. Phase information appears in two places: the per-user `ai_adoption_phase` object, and the aggregated `totals_by_ai_adoption_phase` array. For the phase values, see [Breakdown dimension values](#breakdown-dimension-values).
 
+#### Classification logic
+
+A user is classified into a phase based on which `used_*` per-user fields (see [Per-user report fields](#per-user-report-fields)) are `true` on at least **two active days within the trailing 28-day window**:
+
+| Phase | Surface criteria |
+|:--|:--|
+| No Cohort (displayed as "Passive users" in the impact dashboard) | The user has not met the two-day engagement threshold for any phase in the window. |
+| Phase 1: Code first | At least two active days where `used_chat` is `true`, `code_acceptance_activity_count` is greater than `0`, or `used_agent` is `true`. |
+| Phase 2: Agent first | At least two active days where a single GitHub-based agent surface was used: `used_cli` is `true`, `used_copilot_cloud_agent` is `true`, or `used_copilot_code_review_active` is `true`. Passive {% data variables.copilot.copilot_code-review_short %} activity (`used_copilot_code_review_passive`) doesn't count toward phase classification. |
+| Phase 3: Multi-agent | At least two active days where two or more of the GitHub-based agent surfaces listed under Phase 2 were used. |
+
+A user only needs two qualifying days on a phase's own criteria to reach that phase. A user doesn't need to independently meet Phase 1 criteria to reach Phase 2 or Phase 3, although in practice most agent surface usage co-occurs with completions or chat activity.
+
+Phase assignment is recalculated each day using the trailing 28-day window, so a user's phase can change from one day to the next as their activity within the window shifts. This is expected behavior and does not indicate a data error.
+
 The per-user `ai_adoption_phase` object contains:
 
 | Field | Type | Nullable | Description |