feat: comprehensive telemetry instrumentation (#39)

* feat: comprehensive telemetry instrumentation for error visibility and data moat

Instrument 25 event types across the CLI lifecycle — auth, MCP servers, Python
engine, provider errors, permissions, upgrades, context utilization, agent
outcomes, workflow sequencing, and environment census. Adds telemetry docs page,
firewall endpoint, 25 unit tests, and privacy-safe helpers (categorizeToolName,
bucketCount). No behavioral changes; all telemetry is opt-out via config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address PR review feedback on telemetry instrumentation

- compaction.ts: scope compactionAttempt per-session via Map instead of
  module-level counter that leaked across sessions
- prompt.ts: remove hardcoded doom_loops/permission_denials fields from
  agent_outcome (were always 0); doom loops and permission denials are
  already tracked as separate events
- mcp/index.ts: make mcp_server_census fire-and-forget so listTools/
  listResources never block the MCP connect critical path
- processor.ts: use dedicated generationCounter for context_utilization
  instead of toolCallCounter which counts a different thing
- telemetry/index.ts: refactor categorizeToolName to use a pattern array
  with explicit ordering, making the match logic order-independent and
  easier to maintain
- installation/index.ts: add "other" to upgrade method union instead of
  silently casting unknown methods to "npm"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: anandgupta42

docs/docs/configure/telemetry.md

@@ -0,0 +1,122 @@
+# Telemetry
+
+Altimate Code collects anonymous usage data to help us improve the product. This page describes what we collect, why, and how to opt out.
+
+## What We Collect
+
+We collect the following categories of events:
+
+| Event | Description |
+|-------|-------------|
+| `session_start` | A new CLI session begins |
+| `session_end` | A CLI session ends (includes duration) |
+| `session_forked` | A session is forked from an existing one |
+| `generation` | An AI model generation completes (model ID, token counts, duration — no prompt content) |
+| `tool_call` | A tool is invoked (tool name and category — no arguments or output) |
+| `bridge_call` | A Python engine RPC call completes (method name and duration — no arguments) |
+| `command` | A CLI command is executed (command name only) |
+| `error` | An unhandled error occurs (error type and truncated message — no stack traces) |
+| `auth_login` | Authentication succeeds or fails (provider and method — no credentials) |
+| `auth_logout` | A user logs out (provider only) |
+| `mcp_server_status` | An MCP server connects, disconnects, or errors (server name and transport) |
+| `provider_error` | An AI provider returns an error (error type and HTTP status — no request content) |
+| `engine_started` | The Python engine starts or restarts (version and duration) |
+| `engine_error` | The Python engine fails to start (phase and truncated error) |
+| `upgrade_attempted` | A CLI upgrade is attempted (version and method) |
+| `permission_denied` | A tool permission is denied (tool name and source) |
+| `doom_loop_detected` | A repeated tool call pattern is detected (tool name and count) |
+| `compaction_triggered` | Context compaction runs (strategy and token counts) |
+| `tool_outputs_pruned` | Tool outputs are pruned during compaction (count) |
+| `environment_census` | Environment snapshot on project scan (warehouse types, dbt presence, feature flags — no hostnames) |
+| `context_utilization` | Context window usage per generation (token counts, utilization percentage, cache hit ratio) |
+| `agent_outcome` | Agent session outcome (agent type, tool/generation counts, cost, outcome status) |
+| `error_recovered` | Successful recovery from a transient error (error type, strategy, attempt count) |
+| `mcp_server_census` | MCP server capabilities after connect (tool and resource counts — no tool names) |
+| `context_overflow_recovered` | Context overflow is handled (strategy) |
+
+Each event includes a timestamp, anonymous session ID, and the CLI version.
+
+## Why We Collect Telemetry
+
+Telemetry helps us:
+
+- **Detect errors** — identify crashes, provider failures, and engine issues before users report them
+- **Improve reliability** — track MCP server stability, engine startup success rates, and upgrade outcomes
+- **Understand usage patterns** — know which tools and features are used so we can prioritize development
+- **Measure performance** — track generation latency, engine startup time, and bridge call duration
+
+## Disabling Telemetry
+
+To disable all telemetry collection, add this to your configuration file (`~/.config/altimate/config.json`):
+
+```json
+{
+  "telemetry": {
+    "disabled": true
+  }
+}
+```
+
+You can also set the environment variable:
+
+```bash
+export ALTIMATE_TELEMETRY_DISABLED=true
+```
+
+When telemetry is disabled, no events are sent and no network requests are made to the telemetry endpoint.
+
+## Privacy
+
+We take your privacy seriously. Altimate Code telemetry **never** collects:
+
+- SQL queries or query results
+- Code content, file contents, or file paths
+- Credentials, API keys, or tokens
+- Database connection strings or hostnames
+- Personally identifiable information beyond your email (used only for user correlation)
+- Tool arguments or outputs
+- AI prompt content or responses
+
+Error messages are truncated to 500 characters and scrubbed of file paths before sending.
+
+## Network
+
+Telemetry data is sent to Azure Application Insights:
+
+| Endpoint | Purpose |
+|----------|---------|
+| `eastus-8.in.applicationinsights.azure.com` | Telemetry ingestion |
+
+For a complete list of network endpoints, see the [Network Reference](../network.md).
+
+## For Contributors
+
+### Naming Convention
+
+Event type names use **snake_case** with a `domain_action` pattern:
+
+- `auth_login`, `auth_logout` — authentication events
+- `mcp_server_status`, `mcp_server_census` — MCP server lifecycle
+- `engine_started`, `engine_error` — Python engine events
+- `provider_error` — AI provider errors
+- `session_forked` — session lifecycle
+- `environment_census` — environment snapshot events
+- `context_utilization`, `context_overflow_recovered` — context management events
+- `agent_outcome` — agent session events
+- `error_recovered` — error recovery events
+
+### Adding a New Event
+
+1. **Define the type** — Add a new variant to the `Telemetry.Event` union in `packages/altimate-code/src/telemetry/index.ts`
+2. **Emit the event** — Call `Telemetry.track()` at the appropriate location
+3. **Update docs** — Add a row to the event table above
+
+### Privacy Checklist
+
+Before adding a new event, verify:
+
+- [ ] No SQL, code, or file contents are included
+- [ ] No credentials or connection strings are included
+- [ ] Error messages are truncated to 500 characters
+- [ ] File paths are not included in any field
+- [ ] Only tool names are sent, never arguments or outputs

docs/docs/network.md

@@ -39,6 +39,7 @@ altimate needs outbound HTTPS access to:
 | `registry.npmjs.org` | Package updates |
 | `models.dev` | Model catalog (can be disabled) |
 | Your warehouse endpoints | Database connections |
+| `eastus-8.in.applicationinsights.azure.com` | Telemetry (Azure Application Insights) |
 
 ### Disable Model Fetching
 

docs/mkdocs.yml

@@ -96,6 +96,7 @@ nav:
       - Appearance:
           - Themes: configure/themes.md
           - Keybinds: configure/keybinds.md
+      - Telemetry: configure/telemetry.md
       - Integrations:
           - LSP Servers: configure/lsp.md
           - MCP Servers: configure/mcp-servers.md

packages/altimate-code/src/bridge/engine.ts

@@ -17,6 +17,7 @@ import fs from "fs/promises"
 import path from "path"
 import { Global } from "../global"
 import { UI } from "../cli/ui"
+import { Telemetry } from "@/telemetry"
 
 declare const ALTIMATE_ENGINE_VERSION: string
 declare const ALTIMATE_CLI_VERSION: string
@@ -94,7 +95,17 @@ export async function ensureUv(): Promise<void> {
   await fs.mkdir(path.join(dir, "bin"), { recursive: true })
 
   const response = await fetch(url)
-  if (!response.ok) throw new Error(`Failed to download uv: ${response.statusText}`)
+  if (!response.ok) {
+    const errMsg = `Failed to download uv: ${response.statusText}`
+    Telemetry.track({
+      type: "engine_error",
+      timestamp: Date.now(),
+      session_id: Telemetry.getContext().sessionId,
+      phase: "uv_download",
+      error_message: errMsg.slice(0, 500),
+    })
+    throw new Error(errMsg)
+  }
   const buffer = Buffer.from(await response.arrayBuffer())
 
   const tmpFile = path.join(dir, "bin", asset)
@@ -146,8 +157,11 @@ export async function ensureEngine(): Promise<void> {
 
 async function ensureEngineImpl(): Promise<void> {
   const manifest = await readManifest()
+  const isUpgrade = manifest !== null
   if (manifest && manifest.engine_version === ALTIMATE_ENGINE_VERSION) return
 
+  const startTime = Date.now()
+
   await ensureUv()
 
   const uv = uvPath()
@@ -157,13 +171,35 @@ async function ensureEngineImpl(): Promise<void> {
   // Create venv if it doesn't exist
   if (!existsSync(venvDir)) {
     UI.println(`${UI.Style.TEXT_DIM}Creating Python environment...${UI.Style.TEXT_NORMAL}`)
-    execFileSync(uv, ["venv", "--python", "3.12", venvDir])
+    try {
+      execFileSync(uv, ["venv", "--python", "3.12", venvDir])
+    } catch (e: any) {
+      Telemetry.track({
+        type: "engine_error",
+        timestamp: Date.now(),
+        session_id: Telemetry.getContext().sessionId,
+        phase: "venv_create",
+        error_message: (e?.message ?? String(e)).slice(0, 500),
+      })
+      throw e
+    }
   }
 
   // Install/upgrade engine
   const pythonPath = enginePythonPath()
   UI.println(`${UI.Style.TEXT_DIM}Installing altimate-engine ${ALTIMATE_ENGINE_VERSION}...${UI.Style.TEXT_NORMAL}`)
-  execFileSync(uv, ["pip", "install", "--python", pythonPath, `altimate-engine==${ALTIMATE_ENGINE_VERSION}`])
+  try {
+    execFileSync(uv, ["pip", "install", "--python", pythonPath, `altimate-engine==${ALTIMATE_ENGINE_VERSION}`])
+  } catch (e: any) {
+    Telemetry.track({
+      type: "engine_error",
+      timestamp: Date.now(),
+      session_id: Telemetry.getContext().sessionId,
+      phase: "pip_install",
+      error_message: (e?.message ?? String(e)).slice(0, 500),
+    })
+    throw e
+  }
 
   // Get python version
   const pyVersion = execFileSync(pythonPath, ["--version"]).toString().trim()
@@ -178,6 +214,16 @@ async function ensureEngineImpl(): Promise<void> {
     installed_at: new Date().toISOString(),
   })
 
+  Telemetry.track({
+    type: "engine_started",
+    timestamp: Date.now(),
+    session_id: Telemetry.getContext().sessionId,
+    engine_version: ALTIMATE_ENGINE_VERSION,
+    python_version: pyVersion,
+    status: isUpgrade ? "upgraded" : "started",
+    duration_ms: Date.now() - startTime,
+  })
+
   UI.println(`${UI.Style.TEXT_SUCCESS}Engine ready${UI.Style.TEXT_NORMAL}`)
 }
 

packages/altimate-code/src/cli/cmd/auth.ts

@@ -10,6 +10,7 @@ import { Config } from "../../config/config"
 import { Global } from "../../global"
 import { Plugin } from "../../plugin"
 import { Instance } from "../../project/instance"
+import { Telemetry } from "../../telemetry"
 import type { Hooks } from "@altimateai/altimate-code-plugin"
 
 type PluginAuth = NonNullable<Hooks["auth"]>
@@ -78,6 +79,15 @@ async function handlePluginAuth(plugin: { auth: PluginAuth }, provider: string):
       const result = await authorize.callback()
       if (result.type === "failed") {
         spinner.stop("Failed to authorize", 1)
+        Telemetry.track({
+          type: "auth_login",
+          timestamp: Date.now(),
+          session_id: "cli",
+          provider_id: provider,
+          method: "oauth",
+          status: "error",
+          error: "OAuth auto authorization failed",
+        })
       }
       if (result.type === "success") {
         const saveProvider = result.provider ?? provider
@@ -98,6 +108,14 @@ async function handlePluginAuth(plugin: { auth: PluginAuth }, provider: string):
           })
         }
         spinner.stop("Login successful")
+        Telemetry.track({
+          type: "auth_login",
+          timestamp: Date.now(),
+          session_id: "cli",
+          provider_id: saveProvider,
+          method: "oauth",
+          status: "success",
+        })
       }
     }
 
@@ -110,6 +128,15 @@ async function handlePluginAuth(plugin: { auth: PluginAuth }, provider: string):
       const result = await authorize.callback(code)
       if (result.type === "failed") {
         prompts.log.error("Failed to authorize")
+        Telemetry.track({
+          type: "auth_login",
+          timestamp: Date.now(),
+          session_id: "cli",
+          provider_id: provider,
+          method: "oauth",
+          status: "error",
+          error: "OAuth code authorization failed",
+        })
       }
       if (result.type === "success") {
         const saveProvider = result.provider ?? provider
@@ -130,6 +157,14 @@ async function handlePluginAuth(plugin: { auth: PluginAuth }, provider: string):
           })
         }
         prompts.log.success("Login successful")
+        Telemetry.track({
+          type: "auth_login",
+          timestamp: Date.now(),
+          session_id: "cli",
+          provider_id: saveProvider,
+          method: "oauth",
+          status: "success",
+        })
       }
     }
 
@@ -142,6 +177,15 @@ async function handlePluginAuth(plugin: { auth: PluginAuth }, provider: string):
       const result = await method.authorize(inputs)
       if (result.type === "failed") {
         prompts.log.error("Failed to authorize")
+        Telemetry.track({
+          type: "auth_login",
+          timestamp: Date.now(),
+          session_id: "cli",
+          provider_id: provider,
+          method: "api_key",
+          status: "error",
+          error: "API key authorization failed",
+        })
       }
       if (result.type === "success") {
         const saveProvider = result.provider ?? provider
@@ -150,6 +194,14 @@ async function handlePluginAuth(plugin: { auth: PluginAuth }, provider: string):
           key: result.key,
         })
         prompts.log.success("Login successful")
+        Telemetry.track({
+          type: "auth_login",
+          timestamp: Date.now(),
+          session_id: "cli",
+          provider_id: saveProvider,
+          method: "api_key",
+          status: "success",
+        })
       }
       prompts.outro("Done")
       return true
@@ -270,6 +322,15 @@ export const AuthLoginCommand = cmd({
           const exit = await proc.exited
           if (exit !== 0) {
             prompts.log.error("Failed")
+            Telemetry.track({
+              type: "auth_login",
+              timestamp: Date.now(),
+              session_id: "cli",
+              provider_id: args.url!,
+              method: "api_key",
+              status: "error",
+              error: "Well-known auth command failed",
+            })
             prompts.outro("Done")
             return
           }
@@ -280,6 +341,14 @@ export const AuthLoginCommand = cmd({
             token: token.trim(),
           })
           prompts.log.success("Logged into " + args.url)
+          Telemetry.track({
+            type: "auth_login",
+            timestamp: Date.now(),
+            session_id: "cli",
+            provider_id: args.url!,
+            method: "api_key",
+            status: "success",
+          })
           prompts.outro("Done")
           return
         }
@@ -411,6 +480,14 @@ export const AuthLoginCommand = cmd({
           type: "api",
           key,
         })
+        Telemetry.track({
+          type: "auth_login",
+          timestamp: Date.now(),
+          session_id: "cli",
+          provider_id: provider,
+          method: "api_key",
+          status: "success",
+        })
 
         prompts.outro("Done")
       },
@@ -439,6 +516,12 @@ export const AuthLogoutCommand = cmd({
     })
     if (prompts.isCancel(providerID)) throw new UI.CancelledError()
     await Auth.remove(providerID)
+    Telemetry.track({
+      type: "auth_logout",
+      timestamp: Date.now(),
+      session_id: "cli",
+      provider_id: providerID,
+    })
     prompts.outro("Logout successful")
   },
 })