feat: annotate automatic tools and exclude from fluency scoring (#529)

rajbos · Copilot · web-flow · commit 4d37f78b9fdc · 2026-03-30T11:26:08.000Z
* feat: annotate automatic tools and exclude from fluency scoring

Add automaticTools.json with a curated list of tool IDs that Copilot calls
autonomously (file reads, codebase searches, error checks, confirmations, etc.).
These tools do not require user configuration and should not inflate fluency scores.

Changes:
- New vscode-extension/src/automaticTools.json: list of automatic tool IDs
- maturityScoring.ts: compute nonAutoToolCount and use it for tuStage/agStage
  thresholds; evidence text now distinguishes intentional vs automatic tools
- webview/usage/main.ts: show 'auto' badge on automatic tools in tool tables
- cli/esbuild.js: include automaticTools.json in the bundled data files
- docs/FLUENCY-LEVELS.md: document the automatic vs intentional tool distinction

Co-authored-by: Copilot &lt;223556219+Copilot@users.noreply.github.com&gt;

* fix: update fluency viewer thresholds and agent docs for automatic tools

- maturityScoring.ts getFluencyLevelData(): update threshold text in Agentic
  and Tool Usage levels to say 'intentional tools' instead of 'unique tools',
  and explain that automatic tools are excluded (Stage 1 description, Stage 2
  now lists examples of intentional tools)
- .github/agents/tool-names.agent.md: add automaticTools.json as a key file,
  add Automatic vs. Intentional Tools guidance section, update checklist
- sync-toolnames prompts (both copies): update constraints to also modify
  automaticTools.json and classify new tools as automatic or intentional

Co-authored-by: Copilot &lt;223556219+Copilot@users.noreply.github.com&gt;

---------

Co-authored-by: Copilot &lt;223556219+Copilot@users.noreply.github.com&gt;
diff --git a/.github/agents/tool-names.agent.md b/.github/agents/tool-names.agent.md
@@ -15,10 +15,12 @@ Trigger this agent when:
 - New MCP tools appear in user session logs without display labels
 - The sync-toolnames workflow detects new upstream tools from `microsoft/vscode-copilot-chat`
 
-## Key File
+## Key Files
 
 **`src/toolNames.json`** — The single mapping file from raw tool identifiers to human-readable display names. Every tool the extension encounters gets looked up here; missing entries show as "Unknown" in the UI.
 
+**`src/automaticTools.json`** — An array of tool IDs that Copilot calls *automatically* on its own (file reads, directory listings, searches, error checks, confirmations, memory, etc.). These tools are excluded from fluency scoring because they don't reflect intentional user configuration. When adding new tool entries to `toolNames.json`, you **must also decide** whether each tool is automatic or intentional and add it to `automaticTools.json` if automatic.
+
 ## MCP Tool Name Conventions
 
 MCP tools follow predictable naming patterns. The raw tool identifier encodes the MCP server origin and the action:
@@ -95,7 +97,34 @@ Use these repos to look up tool definitions when needed:
 | Context7 | [upstash/context7](https://github.com/upstash/context7) | TypeScript | Library documentation retrieval |
 | Chrome DevTools MCP | [ChromeDevTools/chrome-devtools-mcp](https://github.com/ChromeDevTools/chrome-devtools-mcp) | TypeScript | Browser debugging tools |
 
-## Editing `src/toolNames.json`
+## Automatic vs. Intentional Tools
+
+When adding a new tool to `toolNames.json`, also determine if it belongs in `automaticTools.json`.
+
+**Add to `automaticTools.json` (automatic)** — tools the agent calls by itself without any user configuration:
+- File system reads: `read_file`, `list_dir`, `view`, `glob`, `grep`, file search variants
+- Codebase search: `semantic_search`, `code_search`, `search_workspace_symbols`
+- Project info: `get_errors`, `get_changed_files`, `read_project_structure`, `get_vscode_api`
+- Terminal reads (not execution): `terminal_selection`, `terminal_last_command`, `get_terminal_output`
+- Internal/session: `memory`, `detect_memories`, `tool_replay`, `vscode_get_confirmation*`, `ask_questions`, `switch_agent`, `bash`
+
+**Do NOT add to `automaticTools.json` (intentional)** — tools that require explicit user setup or represent deliberate action:
+- Terminal execution: `run_in_terminal`, `run_build`, `run_task`
+- File writing/editing: `edit_files`, `write_file`, `create_file`, `apply_patch`
+- Tests & runs: `runTests`, `run_notebook_cell`, `run_vscode_command`
+- External integrations: `fetch_webpage`, `websearch`, `webfetch`
+- MCP tools (all — user must configure the server)
+- GitHub integrations: `github_pull_request`, `github_repo`
+
+**Rule of thumb:** If the user must explicitly enable, configure, or consciously invoke the tool, it's intentional. If the agent just uses it as background context gathering, it's automatic.
+
+## Editing `src/automaticTools.json`
+
+- The file is a plain JSON array of tool ID strings
+- Add new entries at the end of the array (before the closing `]`)
+- Keep related tool variants together (e.g., all variants of `read_file`)
+
+
 
 ### Style Rules
 
@@ -124,5 +153,6 @@ The `sync-toolnames` workflow (`.github/workflows/sync-toolnames.yml`) automatic
 - [ ] For MCP tools, match the prefix to a known server or research the source
 - [ ] Generate friendly names following the conventions above
 - [ ] Add entries to `src/toolNames.json` in the correct location
+- [ ] For each new tool, decide if it is **automatic** or **intentional** — add automatic tools to `src/automaticTools.json`
 - [ ] Run `npm run compile` to validate
 - [ ] Run `npm run test:node` to confirm tests pass
diff --git a/.github/prompts/sync-toolnames.prompt.md b/.github/prompts/sync-toolnames.prompt.md
@@ -40,7 +40,8 @@ Scan `microsoft/vscode-copilot-chat` repo for model-facing tool identifiers, com
 8. Also print (as plain text, after the delta or NO_DELTA) the upstream commit SHA used for the scan and the exact file path scanned in upstream, for traceability.
 
 ## Constraints
-- Only modify our toolNames.json file.
+- Only modify `toolNames.json` and `automaticTools.json`.
 - Do not open a PR.
 - Do not include tools in the list that are not model-facing (only those defined in upstream `ToolName` / `ContributedToolName` string values).
 - Be resilient to minor refactors (enum order changes, added comments, etc.).
+- For each new tool added to `toolNames.json`, also determine if it is **automatic** (agent calls it on its own: file reads, searches, error checks, confirmations) or **intentional** (user configures it: terminal execution, file editing, websearch, MCP tools). Add automatic tools to `automaticTools.json`.
diff --git a/.github/workflows/prompts/sync-toolnames-prompt.md b/.github/workflows/prompts/sync-toolnames-prompt.md
@@ -34,7 +34,8 @@ Scan `microsoft/vscode-copilot-chat` repo for model-facing tool identifiers, com
 8. Also print (as plain text, after the delta or NO_DELTA) the upstream commit SHA used for the scan and the exact file path scanned in upstream, for traceability.
 
 ## Constraints
-- Only modify our toolNames.json file.
+- Only modify `toolNames.json` and `automaticTools.json`.
 - Do not open a PR.
 - Do not include tools in the list that are not model-facing (only those defined in upstream `ToolName` / `ContributedToolName` string values).
 - Be resilient to minor refactors (enum order changes, added comments, etc.).
+- For each new tool added to `toolNames.json`, also determine if it is **automatic** (agent calls it on its own: file reads, searches, error checks, confirmations) or **intentional** (user configures it: terminal execution, file editing, websearch, MCP tools). Add automatic tools to `automaticTools.json`.
diff --git a/cli/esbuild.js b/cli/esbuild.js
@@ -10,6 +10,7 @@ async function main() {
     "tokenEstimators.json",
     "modelPricing.json",
     "toolNames.json",
+    "automaticTools.json",
   ];
 
   for (const file of dataFiles) {
diff --git a/docs/FLUENCY-LEVELS.md b/docs/FLUENCY-LEVELS.md
@@ -64,14 +64,16 @@ Measures adoption of autonomous, multi-step agent mode workflows.
 |-------|----------|
 | 1 | No agent-mode interactions |
 | 2 | At least 1 agent-mode interaction |
-| 3 | 10+ agent-mode interactions **and** 3+ unique tools used |
-| 4 | 50+ agent-mode interactions **and** 5+ unique tools used |
+| 3 | 10+ agent-mode interactions **and** 3+ unique intentional tools used |
+| 4 | 50+ agent-mode interactions **and** 5+ unique intentional tools used |
 
 **Boosters:**
 - Multi-file edit sessions detected → at least Stage 2
 - Average 3+ files per edit session → at least Stage 3
 - 20+ multi-file edits with average 3+ files per session → Stage 4
 
+> **Note:** Only *intentional* tools count toward the unique tool thresholds — tools that Copilot calls automatically (file reads, searches, error lookups, confirmations, memory, etc.) are excluded. See [Automatic vs. Intentional Tools](#automatic-vs-intentional-tools) below.
+
 ---
 
 ### 4. 🔧 Tool Usage
@@ -80,15 +82,39 @@ Measures breadth and depth of tool integration, including MCP servers.
 
 | Stage | Criteria |
 |-------|----------|
-| 1 | No tools used |
-| 2 | At least 1 unique tool used |
+| 1 | No intentional tools used |
+| 2 | At least 1 intentional tool used |
 | 3 | 2+ advanced tools used, **or** `@workspace` agent sessions detected, **or** any MCP server usage |
 | 4 | 2+ MCP servers used |
 
 **Recognised advanced tools:** GitHub Pull Request, GitHub Repository, Run In Terminal, Edit Files, List Files
 
+> **Note:** Only *intentional* tools count toward Stage 2. Automatic tools are still shown in the tool-usage table with an `auto` badge, but are not counted for scoring. See [Automatic vs. Intentional Tools](#automatic-vs-intentional-tools) below.
+
 ---
 
+### Automatic vs. Intentional Tools
+
+Copilot calls many tools on its own during agentic sessions to gather context — reading files, searching the codebase, checking errors, etc. These are called **automatic tools** and do **not** count toward fluency scoring because they do not reflect deliberate configuration choices by the user.
+
+**Automatic tools** (excluded from fluency scoring):
+- File operations: `read_file`, `list_dir`, `ls`, `view`, `find_files`, `glob`, `grep`, `grep_search`, `file_search`, `file_glob_search`
+- Codebase search: `semantic_search`, `code_search`, `search_workspace_symbols`, `get_symbols_by_name`
+- Project info: `get_errors`, `get_changed_files`, `read_project_structure`, `get_project_setup_info`, `get_vscode_api`, `get_doc_info`
+- Terminal reads: `terminal_selection`, `terminal_last_command`, `get_terminal_output`, `await_terminal`
+- Internal/session: `memory`, `detect_memories`, `tool_replay`, `vscode_get_confirmation*`, `ask_questions`, `switch_agent`, `bash`
+
+**Intentional tools** (count toward fluency scoring) include:
+- Terminal execution: `run_in_terminal`, `run_build`, `run_task`
+- File writing/editing: `edit_files`, `write_file`, `create_file`, `apply_patch`, `insert_edit_into_file`, `replace_string_in_file`
+- Tests and runs: `runTests`, `run_notebook_cell`, `run_vscode_command`, `create_and_run_task`
+- External integrations: `fetch_webpage`, `webfetch`, `websearch`, MCP tools (all)
+- GitHub: `github_pull_request`, `github_repo`
+- Browser: `open_integrated_browser`, `renderMermaidDiagram`
+- Extensions and packages: `install_extension`, `install_python_packages`
+
+The full list of automatic tool IDs is maintained in `vscode-extension/src/automaticTools.json`.
+
 ### 5. ⚙️ Customization
 
 Measures how you tailor Copilot to your projects (custom instructions, model selection).
diff --git a/vscode-extension/src/automaticTools.json b/vscode-extension/src/automaticTools.json
@@ -0,0 +1,99 @@
+[
+  "read_file",
+  "copilot_readFile",
+  "read",
+  "view",
+  "view_image",
+  "copilot_viewImage",
+
+  "list_dir",
+  "ls",
+  "copilot_listDirectory",
+
+  "find_files",
+  "file_search",
+  "file_glob_search",
+  "copilot_findFiles",
+  "opilot_findFiles",
+  "copilot_findTextInFiles",
+  "copilot_findTestFiles",
+  "test_search",
+
+  "grep_search",
+  "grep",
+  "glob",
+
+  "semantic_search",
+  "copilot_searchCodebase",
+  "code_search",
+  "copilot_getSearchResults",
+  "get_search_view_results",
+  "search_workspace_symbols",
+  "copilot_searchWorkspaceSymbols",
+  "vscode_listCodeUsages",
+  "get_symbols_by_name",
+
+  "get_errors",
+  "copilot_getErrors",
+
+  "get_changed_files",
+  "copilot_getChangedFiles",
+
+  "read_project_structure",
+  "copilot_readProjectStructure",
+
+  "get_doc_info",
+  "copilot_getDocInfo",
+
+  "get_vscode_api",
+  "copilot_getVSCodeAPI",
+
+  "get_project_setup_info",
+  "copilot_getProjectSetupInfo",
+
+  "get_projects_in_solution",
+  "get_python_executable_details",
+
+  "get_currentfile",
+  "get_file",
+
+  "vscode_get_confirmation",
+  "vscode_get_confirmation_with_options",
+  "vscode_get_terminal_confirmation",
+  "vscode_get_modified_files_confirmation",
+
+  "memory",
+  "copilot_memory",
+  "detect_memories",
+
+  "tool_replay",
+  "copilot_toolReplay",
+  "tool_search",
+
+  "get_task_output",
+  "job_output",
+  "get_terminal_output",
+  "await_terminal",
+
+  "terminal_selection",
+  "terminal_last_command",
+
+  "read_notebook_cell_output",
+  "copilot_readNotebookCellOutput",
+  "copilot_getNotebookSummary",
+
+  "test_failure",
+  "copilot_testFailure",
+
+  "ask_questions",
+  "copilot_askQuestions",
+
+  "switch_agent",
+  "copilot_switchAgent",
+
+  "bash",
+
+  "vscode_editFile_internal",
+  "vscode_fetchWebPage_internal",
+  "vscode_searchExtensions_internal"
+]
diff --git a/vscode-extension/src/maturityScoring.ts b/vscode-extension/src/maturityScoring.ts
@@ -8,6 +8,11 @@ import type {
 	WorkspaceCustomizationMatrix,
 	UsageAnalysisPeriod,
 } from './types';
+import automaticToolIds from './automaticTools.json';
+
+/** Set of tool IDs that Copilot uses autonomously (reading files, searching, etc.).
+ *  These are excluded from fluency scoring since the user doesn't configure them. */
+const AUTOMATIC_TOOL_SET = new Set<string>(automaticToolIds);
 
 /** Format a number with thousand separators for display. */
 function fmt(n: number): string {
@@ -195,7 +200,7 @@ export function getFluencyLevelData(isDebugMode: boolean): {
               label: "Stage 3: AI Collaborator",
               description: "Regular use of agent mode with diverse tools",
               thresholds: [
-                "At least 10 agent-mode interactions AND 3+ unique tools used OR",
+                "At least 10 agent-mode interactions AND 3+ intentional tools used OR",
                 "Average 3+ files per edit session OR",
                 "Using edits agent for focused editing tasks",
               ],
@@ -209,7 +214,7 @@ export function getFluencyLevelData(isDebugMode: boolean): {
               label: "Stage 4: AI Strategist",
               description: "Heavy, strategic use of autonomous features",
               thresholds: [
-                "At least 50 agent-mode interactions AND 5+ tool types used OR",
+                "At least 50 agent-mode interactions AND 5+ intentional tools used OR",
                 "At least 20 multi-file edits with 3+ files per session average",
                 "Demonstrates mastery of agent orchestration",
               ],
@@ -229,7 +234,7 @@ export function getFluencyLevelData(isDebugMode: boolean): {
               label: "Stage 1: AI Skeptic",
               description: "Not using tools beyond basic chat",
               thresholds: [
-                "Zero unique tools used",
+                "Zero intentional tools used (automatic tools like file reads and searches are excluded)",
                 "No MCP servers configured",
                 "No workspace agent sessions",
               ],
@@ -243,8 +248,8 @@ export function getFluencyLevelData(isDebugMode: boolean): {
               label: "Stage 2: AI Explorer",
               description: "Beginning to use basic tools",
               thresholds: [
-                "At least 1 unique tool used",
-                "Using basic agent mode tools",
+                "At least 1 intentional tool used (e.g. run_in_terminal, editFiles, websearch, MCP tools)",
+                "Automatic tools (file reads, searches, error checks) do not count",
               ],
               tips: [
                 "Set up [MCP servers](https://code.visualstudio.com/docs/copilot/customization/mcp-servers) to connect Copilot to external tools (databases, APIs, cloud services) — [▶ MCP with Azure and GitHub](https://tech.hub.ms/github-copilot/videos/mcp-with-azure-and-github)",
@@ -440,6 +445,7 @@ export function calculateFluencyScoreForTeamMember(fd: {
     const hasModelSwitching = fd.mixedTierSessions > 0 || switchingFrequency > 0;
     const hasAgentMode = fd.agentModeCount > 0;
     const toolCount = Object.keys(fd.toolCallsByTool).length;
+    const nonAutoToolCount = Object.keys(fd.toolCallsByTool).filter(t => !AUTOMATIC_TOOL_SET.has(t)).length;
     const avgFilesPerSession = fd.filesPerEditCount > 0 ? fd.filesPerEditSum / fd.filesPerEditCount : 0;
     const avgApplyRate = fd.applyRateCount > 0 ? fd.applyRateSum / fd.applyRateCount : 0;
     const totalContextRefs = fd.ctxFile + fd.ctxSelection + fd.ctxSymbol + fd.ctxCodebase + fd.ctxWorkspace;
@@ -528,8 +534,8 @@ export function calculateFluencyScoreForTeamMember(fd: {
     if (fd.multiFileEdits > 0) { agStage = Math.max(agStage, 2); }
     if (avgFilesPerSession >= 3) { agStage = Math.max(agStage, 3); }
     if (fd.editsAgentCount > 0) { agStage = Math.max(agStage, 2); }
-    if (fd.agentModeCount >= 10 && toolCount >= 3) { agStage = Math.max(agStage, 3); }
-    if (fd.agentModeCount >= 50 && toolCount >= 5) { agStage = 4; }
+    if (fd.agentModeCount >= 10 && nonAutoToolCount >= 3) { agStage = Math.max(agStage, 3); }
+    if (fd.agentModeCount >= 50 && nonAutoToolCount >= 5) { agStage = 4; }
     if (fd.multiFileEdits >= 20 && avgFilesPerSession >= 3) { agStage = Math.max(agStage, 4); }
     const agTips: string[] = [];
     if (agStage < 2) { agTips.push("Try agent mode — it can run terminal commands, edit files, and explore codebases autonomously"); }
@@ -538,7 +544,7 @@ export function calculateFluencyScoreForTeamMember(fd: {
 
     // 4. Tool Usage
     let tuStage = 1;
-    if (toolCount > 0) { tuStage = 2; }
+    if (nonAutoToolCount > 0) { tuStage = 2; }
     if (fd.workspaceAgentCount > 0) { tuStage = Math.max(tuStage, 3); }
     const advancedToolIds = ["github_pull_request", "github_repo", "run_in_terminal", "editFiles", "listFiles"];
     const usedAdvancedCount = advancedToolIds.filter(t => (fd.toolCallsByTool[t] ?? 0) > 0).length;
@@ -855,12 +861,13 @@ export async function calculateMaturityScores(lastCustomizationMatrix: Workspace
 
 	// Diverse tool usage in agent mode
 	const toolCount = Object.keys(p.toolCalls.byTool).length;
-	if (p.modeUsage.agent >= 10 && toolCount >= 3) {
+	const nonAutoToolCount = Object.keys(p.toolCalls.byTool).filter(t => !AUTOMATIC_TOOL_SET.has(t)).length;
+	if (p.modeUsage.agent >= 10 && nonAutoToolCount >= 3) {
 		agStage = 3;
 	}
 
 	// Heavy agentic use with many tool types or high multi-file edit rate
-	if (p.modeUsage.agent >= 50 && toolCount >= 5) {
+	if (p.modeUsage.agent >= 50 && nonAutoToolCount >= 5) {
 		agStage = 4;
 	}
 	if (p.editScope && p.editScope.multiFileEdits >= 20 && p.editScope.avgFilesPerSession >= 3) {
@@ -876,10 +883,14 @@ export async function calculateMaturityScores(lastCustomizationMatrix: Workspace
 	const tuTips: string[] = [];
 	let tuStage = 1;
 
-	// Basic tool usage (primarily from agent mode)
-	if (toolCount > 0) {
-		tuEvidence.push(`${fmt(toolCount)} unique tools used`);
+	// Basic tool usage — only count tools the user intentionally enables/configures (not automatic agent tools)
+	if (nonAutoToolCount > 0) {
+		const autoCount = toolCount - nonAutoToolCount;
+		const autoNote = autoCount > 0 ? ` (+ ${fmt(autoCount)} automatic)` : '';
+		tuEvidence.push(`${fmt(nonAutoToolCount)} intentional tools used${autoNote}`);
 		tuStage = 2;
+	} else if (toolCount > 0) {
+		tuEvidence.push(`${fmt(toolCount)} tools used (all automatic — agent reads/searches)`);
 	}
 
 	// Agent type distribution (workspace agent shows advanced tool usage)
diff --git a/vscode-extension/src/webview/usage/main.ts b/vscode-extension/src/webview/usage/main.ts
@@ -125,8 +125,10 @@ function escapeHtml(text: string): string {
 }
 
 import toolNames from '../../toolNames.json';
+import automaticToolIds from '../../automaticTools.json';
 
 let TOOL_NAME_MAP: { [key: string]: string } | null = toolNames || null;
+const AUTOMATIC_TOOL_SET_WV = new Set<string>(automaticToolIds as string[]);
 
 function lookupToolName(id: string): string {
 	if (!TOOL_NAME_MAP) {
@@ -256,10 +258,13 @@ function renderToolsTable(byTool: { [key: string]: number }, limit = 10, nameRes
 	    const rows = sortedTools.map(([tool, count], idx) => {
 		const friendly = escapeHtml(nameResolver(tool));
 		const idEscaped = escapeHtml(tool);
+		const autoBadge = AUTOMATIC_TOOL_SET_WV.has(tool)
+			? `<span title="Automatic tool — Copilot uses this internally and it does not count toward fluency scoring" style="margin-left:6px; padding:1px 5px; font-size:10px; border-radius:3px; background:var(--bg-secondary); color:var(--text-muted); border:1px solid var(--border-subtle); vertical-align:middle;">auto</span>`
+			: '';
 		return `
 		    <tr>
 			    <td style="padding:8px 12px; border-bottom:1px solid var(--border-subtle); width:40px; max-width:40px; text-align:center;">${idx + 1}</td>
-			    <td style="padding:8px 12px; border-bottom:1px solid var(--border-subtle); word-break:break-word; overflow-wrap:break-word; max-width:0;"> <strong title="${idEscaped}">${friendly}</strong></td>
+			    <td style="padding:8px 12px; border-bottom:1px solid var(--border-subtle); word-break:break-word; overflow-wrap:break-word; max-width:0;"> <strong title="${idEscaped}">${friendly}</strong>${autoBadge}</td>
 			    <td style="padding:8px 12px; border-bottom:1px solid var(--border-subtle); text-align:right; width:90px; white-space:nowrap;">${formatNumber(count)}</td>
 		    </tr>`;
 	    }).join('');
