More file updates

Author: rajbos

.github/copilot-instructions.md

@@ -0,0 +1,61 @@
+---
+applyTo: "cli/**"
+---
+
+# CLI — Architecture & Integration Guide
+
+The CLI (`cli/`) is a standalone command-line tool that **shares the session discovery and data access classes** from `vscode-extension/src/` but has its own aggregation pipeline. It is built with TypeScript and bundled via `cli/esbuild.js` into `cli/dist/cli.js`.
+
+## Key Files
+
+- **`cli/src/helpers.ts`**: Shared helper functions — session discovery, file processing, stats aggregation. Imports all data access classes from `vscode-extension/src/`.
+- **`cli/src/commands/`**: One file per sub-command (`stats`, `usage`, `environmental`, `fluency`, `diagnostics`).
+- **`cli/esbuild.js`**: Build script. Copies JSON data files from `vscode-extension/src/` to a temp location before bundling, then removes them.
+- **`cli/tsconfig.json`**: `paths` alias points to `../vscode-extension/src/*`.
+
+## Developer Workflow
+
+```bash
+cd cli
+npm install
+npm run build            # development build
+npm run build:production # minified release build
+```
+
+Or from the repo root:
+```powershell
+./build.ps1 -Project cli
+```
+
+## Adding a New Editor / Data Source
+
+When adding support for a new editor or data source, wire it into **both** `vscode-extension/src/` (see `.github/instructions/vscode-extension.instructions.md`) **and** this CLI.
+
+### CLI Files to Update
+
+| File | What to add |
+|---|---|
+| `cli/src/helpers.ts` | Import, factory function, singleton, detection, stat routing, `processSessionFile()` branch, `calculateUsageAnalysisStats()` deps |
+| `cli/src/commands/stats.ts` | Add entry to `getEditorDisplayName()` |
+| `cli/src/commands/usage.ts` | No change needed — uses shared helpers |
+| `cli/README.md` | Add the new editor to the "Data Sources" section |
+
+### Integration Points in `cli/src/helpers.ts`
+
+1. **Import** — `import { NewEditorDataAccess } from '../../vscode-extension/src/neweditor';`
+2. **Factory function** — `function createNewEditor(): NewEditorDataAccess { return new NewEditorDataAccess(); }`
+3. **Singleton** — `const _newEditorInstance = createNewEditor();`
+4. **`createSessionDiscovery()`** — pass `newEditor: _newEditorInstance` in the deps object
+5. **`statSessionFile()`** — add guard routing virtual paths to the real DB file (before the generic `fs.promises.stat()` fallthrough)
+6. **`getEditorSourceFromPath()`** — add a path pattern check *before* the generic `'/code/'` or `'vscode'` fallthrough, returning a stable lowercase identifier (e.g. `'neweditor'`)
+7. **`processSessionFile()`** — add a guard block calling `getTokens()`, `countInteractions()`, `getModelUsage()` from the data access class and returning a `SessionData` object
+8. **`calculateUsageAnalysisStats()` deps** — pass `newEditor: _newEditorInstance` so `analyzeSessionUsage()` can route to it
+
+### Checklist
+
+- [ ] `cli/src/helpers.ts` — import, factory, singleton, detection, stat routing, processSessionFile block, usageAnalysis deps
+- [ ] `cli/src/commands/stats.ts` — `getEditorDisplayName()` entry
+- [ ] `cli/README.md` — "Data Sources" section updated
+- [ ] `npm run build` passes (from `cli/`)
+- [ ] CLI `stats` command shows the new editor in the session list
+- [ ] Token counts are non-zero and plausible

.github/instructions/cli.instructions.md

@@ -0,0 +1,39 @@
+---
+applyTo: "visualstudio-extension/**"
+---
+
+# Visual Studio Extension — Architecture & Development Guide
+
+The `visualstudio-extension/` folder contains a C# Visual Studio extension (`.vsix`) that replicates the core views of the VS Code extension for users of Visual Studio IDE.
+
+## Planned Approach (Option C — WebView2 + existing webviews)
+
+The extension hosts a **WebView2** control inside a Visual Studio Tool Window (C# `AsyncPackage`). The existing compiled webview bundles from `vscode-extension/dist/webview/` run inside WebView2 — avoiding a full UI rewrite. A thin C# bridge handles file I/O and passes JSON to the webview via `PostWebMessageAsJson`.
+
+### Key design decisions
+
+- **Data layer**: The `VisualStudioDataAccess` class in `vscode-extension/src/visualstudio.ts` already handles all session discovery and parsing for Visual Studio Copilot Chat binary session files (MessagePack format). The C# host reads session metadata and delegates token estimation to this layer by shelling out to the CLI, or by re-implementing only the discovery logic in C#.
+- **UI**: WebView2 renders the existing `details`, `chart`, `usage`, and `diagnostics` webview bundles. The VS Code CSS theme tokens (`--vscode-*`) will need a compatibility shim mapping to Visual Studio theme tokens.
+- **No cloud storage**: Azure Storage integration is out of scope for the initial version.
+
+## Developer Workflow
+
+```bash
+# Prerequisites: Visual Studio 2022, .NET SDK, VSIX workload
+cd visualstudio-extension
+dotnet build --configuration Release
+```
+
+Or from the repo root:
+```powershell
+./build.ps1 -Project visualstudio
+```
+
+## Session File Discovery
+
+Visual Studio Copilot Chat stores sessions as MessagePack-encoded binary files:
+```
+<project>\.vs\<solution>.<ext>\copilot-chat\<hash>\sessions\<uuid>
+```
+
+Discovery is already implemented — see `vscode-extension/src/visualstudio.ts` for `VisualStudioDataAccess.discoverSessions()`.

.github/instructions/visualstudio-extension.instructions.md

@@ -0,0 +1,148 @@
+---
+applyTo: "vscode-extension/**"
+---
+
+# VS Code Extension — Architecture & Development Guide
+
+## Architecture Overview
+
+The entire extension's logic is contained within the `CopilotTokenTracker` class in `vscode-extension/src/extension.ts`. Its primary function is to track GitHub Copilot token usage by reading and parsing session log files.
+
+- **Activation**: The extension activates on VS Code startup via the `onStartupFinished` event in `vscode-extension/package.json`. The `activate` function in `vscode-extension/src/extension.ts` instantiates `CopilotTokenTracker`.
+
+- **Data Source**: The core data comes from `.json` log files generated by the GitHub Copilot Chat extension. The `getCopilotSessionFiles` method locates these files in the user's OS-specific AppData directory for VS Code (e.g., `.../Code/User/workspaceStorage/.../chatSessions` and `.../Code/User/globalStorage/emptyWindowChatSessions`).
+
+- **Data Flow**:
+  1. A timer in the `constructor` calls `updateTokenStats()` every 5 minutes.
+  2. `updateTokenStats()` calls `calculateDetailedStats()` to aggregate data.
+  3. `calculateDetailedStats()` reads all session files found by `getCopilotSessionFiles()`.
+  4. For each file, it calculates token counts (`estimateTokensFromSession`), interactions (`countInteractionsInSession`), and per-model usage (`getModelUsageFromSession`).
+  5. Token counts are *estimated* using character-to-token ratios defined in the `tokenEstimators` class property. This is a key convention.
+
+- **Thinking Token Tracking**: Models that use extended thinking (e.g., Claude) produce `kind: "thinking"` response items in session logs. These are tracked separately:
+  - `estimateTokensFromSession` and `estimateTokensFromJsonlSession` return `{ tokens, thinkingTokens }`. The `tokens` field is the **grand total** (input + output + thinking). The `thinkingTokens` field is a **breakdown** showing what portion of the total was thinking — it is NOT subtracted from `tokens`.
+  - `SessionFileCache` stores `thinkingTokens` alongside `tokens`.
+  - `extractResponseData` separates `responseText` and `thinkingText` so the logviewer can display them independently per turn.
+  - `getModelUsageFromSession` does NOT separate thinking — thinking tokens are counted as `outputTokens` in the per-model breakdown. This is intentional since it keeps cost estimation simple.
+  - **Critical invariant**: When computing token totals for display (per-turn, per-session, per-period), always include thinking tokens. In the logviewer, `totalTokens = input + output + thinking`. In stats aggregation, `sessionData.tokens` already includes thinking. Never subtract thinking from a total.
+  - `CACHE_VERSION` must be bumped when changing how tokens are counted so stale caches are invalidated.
+
+- **OpenCode Support**: The extension also reads session data from [OpenCode](https://opencode.ai), a terminal-based coding agent. OpenCode stores its data in `~/.local/share/opencode/` (Linux/macOS) or the equivalent XDG data directory.
+
+  - **Dual storage backends**: OpenCode originally stored sessions as individual JSON files under `storage/session/`, `storage/message/`, and `storage/part/`. It later migrated to a SQLite database (`opencode.db`) in the same data directory. The extension supports **both** backends — it tries the SQLite DB first and falls back to JSON files.
+  - **SQLite reading**: The extension uses `sql.js` (a pure JS/WASM SQLite reader, no native dependencies) to read `opencode.db`. The WASM binary (`sql-wasm.wasm`) is copied to `dist/` during the esbuild step. The sql.js module is lazy-loaded and cached in `_sqlJsModule`.
+  - **DB schema**: The `opencode.db` database has tables `session` (id, slug, title, directory, time_created, time_updated), `message` (id, session_id, time_created, data JSON), and `part` (id, message_id, session_id, time_created, data JSON). The `data` column in `message` contains JSON with `{role, model: {providerID, modelID}, tokens: {total, input, output, reasoning, cache: {read, write}}}`.
+  - **Virtual path scheme**: Since the existing architecture is file-path-based, DB-stored sessions use virtual paths of the form `<opencode_dir>/opencode.db#ses_<id>`. Detection helpers: `isOpenCodeDbSession()` checks for the `opencode.db#ses_` pattern, `getOpenCodeSessionId()` parses both virtual paths and `ses_<id>.json` filenames.
+  - **Async methods**: `getOpenCodeMessagesForSession(filePath)` and `getOpenCodePartsForMessage(messageId)` are the primary data access methods — they try DB first, fall back to JSON. The OpenCode token/interaction/model methods (`getTokensFromOpenCodeSession`, `countOpenCodeInteractions`, `getOpenCodeModelUsage`) are all async.
+  - **`statSessionFile()`**: A helper that resolves DB virtual paths to `fs.promises.stat()` on the `opencode.db` file itself. **All `fs.promises.stat(sessionFile)` calls in loops that process session files must use `this.statSessionFile()` instead**, to avoid failures on DB virtual paths.
+  - **Key invariant**: When adding new code that processes session file paths (stat calls, path splitting, folder grouping), always check `isOpenCodeDbSession()` first and handle the virtual path appropriately (use `statSessionFile()` for stats, use `getOpenCodeDataDir()` for folder grouping).
+
+- **UI Components**:
+  1. **Status Bar**: A `vscode.StatusBarItem` (`statusBarItem`) shows a brief summary. Its tooltip provides more detail.
+  2. **Details Panel**: The `copilot-token-tracker.showDetails` command opens a `vscode.WebviewPanel`. The content for this panel is generated dynamically as an HTML string by the `getDetailsHtml` method.
+
+## Developer Workflow
+
+- **Setup**: Run `npm install` inside `vscode-extension/` to install dependencies.
+- **Build**: Run `npm run compile` from `vscode-extension/` to lint and build the extension using `esbuild`. The output is a single file: `vscode-extension/dist/extension.js`.
+- **Watch Mode**: For active development, use `npm run watch` from `vscode-extension/`. This will automatically recompile the extension on file changes.
+- **Testing/Debugging**: Press `F5` in VS Code to open the Extension Development Host. This will launch a new VS Code window with the extension running. `console.log` statements from `vscode-extension/src/extension.ts` will appear in the Developer Tools console of this new window (Help > Toggle Developer Tools).
+
+**Important build guidance:** After making changes to source code or related files (TypeScript, JavaScript, JSON, or other code files used by the extension), always run `npm run compile` from `vscode-extension/` to validate that the project still builds and lints cleanly before opening a pull request or releasing. You do not need to run the full compile step for documentation-only changes (Markdown files), but you should run it after any edits that touch source, configuration, or JSON data files.
+
+## Development Guidelines
+
+- **Minimal Changes**: Only modify files that are directly needed for the actual changes being implemented. Avoid touching unrelated files, configuration files, or dependencies unless absolutely necessary for the feature or fix.
+- **Focused Modifications**: Make surgical, precise changes that address the specific requirements without affecting other functionality.
+- **Preserve Existing Structure**: Maintain the existing code organization and file structure. Don't refactor or reorganize code unless it's essential for the task.
+
+## Logging Best Practices
+
+**CRITICAL**: Do NOT add debug logging statements like `this.log('[DEBUG] message')` or `console.log('[DEBUG] ...')` for troubleshooting during development. This approach has been found to flood the output channel and cause messages to disappear.
+
+### Why DEBUG Logs Are Problematic
+
+**Extension Host Flooding**: When DEBUG log statements are added to frequently-called methods in `extension.ts` (e.g., cache lookups, file processing loops, webview message handlers), they can generate hundreds of log entries per operation. VS Code's OutputChannel has a buffer limit, and excessive logging causes older messages to be pushed out and lost. This was observed when:
+- Cache hit/miss logging was added to session file processing
+- JSONL content reference counting was logged for each file
+- Webview message handlers logged every incoming message with full JSON payloads
+- Session data was logged with repository counts on each webview update
+
+**Symptom**: After operations like clearing the cache, expected log messages would disappear from the Output panel because they were pushed out of the buffer by DEBUG logs.
+
+### Extension vs Webview Logging
+
+These are two completely separate logging systems:
+
+| Context | Method | Destination | Visibility |
+|---------|--------|-------------|------------|
+| Extension (`vscode-extension/src/extension.ts`) | `this.log()`, `this.warn()`, `this.error()` | VS Code Output Channel | Output panel → "Copilot Token Tracker" |
+| Webview (`vscode-extension/src/webview/*/main.ts`) | `console.log()` | Browser DevTools | Help → Toggle Developer Tools in webview |
+
+- Clearing the output channel (`outputChannel.clear()`) does NOT affect webview console logs
+- Webview console.log statements do NOT appear in the Output panel
+- DEBUG prefixes in webviews were removed to maintain consistency with extension guidelines
+
+### Best Practices
+
+- **Use Existing Logs**: The extension already has comprehensive logging. Review existing log statements before adding new ones.
+- **Minimal Logging**: Only add logging if absolutely necessary for a new feature. Keep messages concise.
+- **Remove Debug Logs**: Any temporary debug logging added during development MUST be removed before committing.
+- **Log Methods**: Use appropriate severity:
+  - `log(message)` - Standard informational messages
+  - `warn(message)` - Warnings or recoverable errors
+  - `error(message)` - Critical errors
+- **No Debug Prefixes**: Avoid `[DEBUG]` markers. The log output is already timestamped.
+- **Avoid High-Frequency Logging**: Never log inside loops that process many items (files, sessions, cache entries).
+
+### Debugging Without Logs
+
+Prefer VS Code's debugger with breakpoints rather than adding log statements:
+1. Press `F5` to launch Extension Development Host
+2. Set breakpoints in `vscode-extension/src/extension.ts`
+3. Use the Debug Console to inspect variables
+
+## Key Files & Conventions
+
+- **`vscode-extension/src/extension.ts`**: The single source file containing all logic.
+  - `CopilotTokenTracker`: The main class.
+  - `calculateDetailedStats()`: The primary data aggregation method.
+  - `getDetailsHtml()`: The method responsible for rendering the webview's HTML content. All styling is inlined within this method's template string.
+- **`vscode-extension/src/README.md`**: **IMPORTANT**: Contains detailed instructions for updating the JSON data files. Always consult this file when updating tokenEstimators.json or modelPricing.json. It includes structure definitions, update procedures, and current pricing information.
+- **`vscode-extension/src/tokenEstimators.json`**: Character-to-token ratio estimators for different AI models. See `vscode-extension/src/README.md` for update instructions.
+- **`vscode-extension/src/modelPricing.json`**: Model pricing data with input/output costs per million tokens. Includes metadata about pricing sources and last update date. See `vscode-extension/src/README.md` for detailed update instructions and current pricing sources.
+- **`docs/FLUENCY-LEVELS.md`**: Documents the scoring rules for the Copilot Fluency Score dashboard (4 stages, 6 categories, thresholds, and boosters). **Keep this file up to date** when changing the `calculateMaturityScores()` method in `vscode-extension/src/extension.ts`.
+- **`vscode-extension/package.json`**: Defines activation events, commands, and build scripts.
+- **`vscode-extension/esbuild.js`**: The build script that bundles the TypeScript source and JSON data files. Also copies `sql-wasm.wasm` from `node_modules/sql.js/dist/` to `dist/` for OpenCode SQLite support.
+- **`vscode-extension/src/types/json.d.ts`**: Type declarations for JSON module imports and the `sql.js` module.
+
+## Webview Navigation Buttons
+
+To maintain a consistent, VS Code-native look across all webview panels (Details, Chart, Usage Analysis, Diagnostics), use the VS Code Webview UI Toolkit for top-level navigation buttons.
+
+- **Use `vscode-button`**: Prefer the toolkit button component for header navigation controls instead of custom `<button>` elements. Example usage in a webview script:
+
+  ```ts
+  const { provideVSCodeDesignSystem, vsCodeButton } = await import('@vscode/webview-ui-toolkit');
+  provideVSCodeDesignSystem().register(vsCodeButton());
+
+  // then create buttons in the DOM:
+  const btn = document.createElement('vscode-button');
+  btn.id = 'btn-details';
+  btn.textContent = '🤖 Details';
+  btn.setAttribute('appearance', 'primary');
+  ```
+
+- **Register the toolkit in each webview**: Each webview that uses `vscode-button` should import and register the toolkit in its `bootstrap()` or initialization function before rendering the layout.
+
+- **Wire messages unchanged**: Buttons should `postMessage` the same navigation commands (`showDetails`, `showChart`, `showUsageAnalysis`, `showDiagnostics`, `refresh`) so the extension can reuse existing panels. Do not change the command names.
+
+- **Checklist for PRs touching webviews**:
+  - Ensure the toolkit is registered before creating `vscode-button` elements.
+  - Keep navigation command names unchanged so `extension.ts` handlers continue to work.
+  - Run `npm run compile` and verify TypeScript and ESLint pass.
+  - Visually compare the header with the Details and other panels to confirm parity.
+
+## Adding a New Editor / Data Source
+
+When adding support for a new editor or data source, you must wire it into **both** this extension (`vscode-extension/src/`) **and** the CLI (`cli/src/`). See `.github/instructions/cli.instructions.md` for the CLI integration checklist.