|
| 1 | +# Tracing |
| 2 | + |
| 3 | +Altimate Code captures detailed traces of every headless session — LLM generations, tool calls, token usage, cost, and timing — and saves them locally as JSON files. Traces are invaluable for debugging agent behavior, optimizing cost, and understanding how the agent solves problems. |
| 4 | + |
| 5 | +Tracing is **enabled by default** and requires no configuration. Traces are stored locally and never leave your machine unless you configure a remote exporter. |
| 6 | + |
| 7 | +## Quick Start |
| 8 | + |
| 9 | +```bash |
| 10 | +# Run a prompt — trace is saved automatically |
| 11 | +altimate-code run "optimize my most expensive queries" |
| 12 | +# → Trace saved: ~/.local/share/altimate-code/traces/abc123.json |
| 13 | + |
| 14 | +# List recent traces |
| 15 | +altimate-code trace list |
| 16 | + |
| 17 | +# View a trace in the browser |
| 18 | +altimate-code trace view abc123 |
| 19 | +``` |
| 20 | + |
| 21 | +## What's Captured |
| 22 | + |
| 23 | +Each trace records the full agent session: |
| 24 | + |
| 25 | +| Data | Description | |
| 26 | +|------|-------------| |
| 27 | +| **Generations** | Each LLM call with model, provider, finish reason, and variant | |
| 28 | +| **Token usage** | Input, output, reasoning, cache read, and cache write tokens per generation | |
| 29 | +| **Cost** | Per-generation and total session cost in USD | |
| 30 | +| **Tool calls** | Every tool invocation with input, output, duration, and status | |
| 31 | +| **Timing** | Start/end timestamps for every span (session, generation, tool) | |
| 32 | +| **Errors** | Error messages and status on failed tool calls or generations | |
| 33 | +| **Metadata** | Model, provider, agent, prompt, user ID, environment, tags | |
| 34 | + |
| 35 | +### Data Engineering Attributes |
| 36 | + |
| 37 | +When using SQL and dbt tools, traces automatically capture domain-specific data: |
| 38 | + |
| 39 | +| Category | Examples | |
| 40 | +|----------|----------| |
| 41 | +| **Warehouse** | Bytes scanned/billed, execution time, queue time, partitions pruned, cache hits, query ID, estimated cost | |
| 42 | +| **SQL** | Query text, dialect, validation results, lineage (input/output tables), schema changes | |
| 43 | +| **dbt** | Command, model status, materialization, rows affected, compiled SQL, test results, Jinja errors | |
| 44 | +| **Data Quality** | Row counts, null percentages, freshness, anomaly detection | |
| 45 | +| **Cost Attribution** | LLM cost + warehouse compute cost + storage delta = total cost, per user/team/project | |
| 46 | + |
| 47 | +These attributes are purely optional — traces are valid without them. They're populated automatically by tools that have access to warehouse metadata. |
| 48 | + |
| 49 | +## Configuration |
| 50 | + |
| 51 | +Add to your config file (`~/.config/altimate-code/altimate-code.json` or project-level `altimate-code.json`): |
| 52 | + |
| 53 | +```json |
| 54 | +{ |
| 55 | + "tracing": { |
| 56 | + "enabled": true, |
| 57 | + "dir": "~/.local/share/altimate-code/traces/", |
| 58 | + "maxFiles": 100, |
| 59 | + "exporters": [] |
| 60 | + } |
| 61 | +} |
| 62 | +``` |
| 63 | + |
| 64 | +| Option | Type | Default | Description | |
| 65 | +|--------|------|---------|-------------| |
| 66 | +| `enabled` | `boolean` | `true` | Enable or disable tracing | |
| 67 | +| `dir` | `string` | `~/.local/share/altimate-code/traces/` | Custom directory for trace files | |
| 68 | +| `maxFiles` | `number` | `100` | Max trace files to keep (oldest pruned automatically). Set to `0` for unlimited | |
| 69 | +| `exporters` | `array` | `[]` | Remote HTTP exporters (see below) | |
| 70 | + |
| 71 | +### Disabling Tracing |
| 72 | + |
| 73 | +```json |
| 74 | +{ |
| 75 | + "tracing": { |
| 76 | + "enabled": false |
| 77 | + } |
| 78 | +} |
| 79 | +``` |
| 80 | + |
| 81 | +Or per-run with the `--no-trace` flag: |
| 82 | + |
| 83 | +```bash |
| 84 | +altimate-code run --no-trace "quick question" |
| 85 | +``` |
| 86 | + |
| 87 | +## Viewing Traces |
| 88 | + |
| 89 | +### List Traces |
| 90 | + |
| 91 | +```bash |
| 92 | +altimate-code trace list |
| 93 | +``` |
| 94 | + |
| 95 | +Shows a table of recent traces with session ID, timestamp, duration, tokens, cost, tool calls, and status. |
| 96 | + |
| 97 | +``` |
| 98 | +SESSION WHEN DURATION TOKENS COST TOOLS STATUS PROMPT |
| 99 | +abc123def456 2m ago 45.2s 12,500 $0.0150 8 ok optimize my most expensive queries |
| 100 | +xyz789abc012 1h ago 12.8s 3,200 $0.0040 3 ok explain this model |
| 101 | +err456def789 3h ago 5.1s 1,800 $0.0020 2 error run dbt tests |
| 102 | +``` |
| 103 | + |
| 104 | +Options: |
| 105 | + |
| 106 | +| Flag | Description | |
| 107 | +|------|-------------| |
| 108 | +| `-n`, `--limit` | Number of traces to show (default: 20) | |
| 109 | + |
| 110 | +### View a Trace |
| 111 | + |
| 112 | +```bash |
| 113 | +altimate-code trace view <session-id> |
| 114 | +``` |
| 115 | + |
| 116 | +Opens a local web server with an interactive trace viewer in your browser. The viewer shows: |
| 117 | + |
| 118 | +- **Summary cards** — duration, token breakdown (input/output/reasoning/cache), cost, generations, tool calls, status |
| 119 | +- **Timeline** — horizontal bars for each span, color-coded by type (generation, tool, error) |
| 120 | +- **Detail panel** — click any span to see its model info, token counts, finish reason, input/output, and domain-specific attributes (warehouse metrics, dbt results, etc.) |
| 121 | + |
| 122 | +Options: |
| 123 | + |
| 124 | +| Flag | Description | |
| 125 | +|------|-------------| |
| 126 | +| `--port` | Port for the viewer server (default: random) | |
| 127 | +| `--live` | Auto-refresh every 2s for in-progress sessions | |
| 128 | + |
| 129 | +Partial session ID matching is supported — `altimate-code trace view abc` matches `abc123def456`. |
| 130 | + |
| 131 | +### Live Viewing (In-Progress Sessions) |
| 132 | + |
| 133 | +Traces are written incrementally — after every tool call and generation, a snapshot is flushed to disk. This means you can view a trace while the session is still running: |
| 134 | + |
| 135 | +```bash |
| 136 | +# In terminal 1: run a long task |
| 137 | +altimate-code run "refactor the entire pipeline" |
| 138 | + |
| 139 | +# In terminal 2: watch the trace live |
| 140 | +altimate-code trace view <session-id> --live |
| 141 | +``` |
| 142 | + |
| 143 | +The `--live` flag adds a green "LIVE" indicator and polls for updates every 2 seconds. The page auto-refreshes when new spans appear. |
| 144 | + |
| 145 | +### From the TUI |
| 146 | + |
| 147 | +Type `/trace` in the TUI to open the trace viewer for the current session in your browser. The viewer launches in live mode automatically, so you can watch spans appear as the agent works. |
| 148 | + |
| 149 | +## Remote Exporters |
| 150 | + |
| 151 | +Traces can be sent to remote backends via HTTP POST. Each exporter receives the full trace JSON on session completion. |
| 152 | + |
| 153 | +```json |
| 154 | +{ |
| 155 | + "tracing": { |
| 156 | + "exporters": [ |
| 157 | + { |
| 158 | + "name": "my-backend", |
| 159 | + "endpoint": "https://api.example.com/v1/traces", |
| 160 | + "headers": { |
| 161 | + "Authorization": "Bearer <token>" |
| 162 | + } |
| 163 | + } |
| 164 | + ] |
| 165 | + } |
| 166 | +} |
| 167 | +``` |
| 168 | + |
| 169 | +| Field | Type | Description | |
| 170 | +|-------|------|-------------| |
| 171 | +| `name` | `string` | Identifier for this exporter (used in logs) | |
| 172 | +| `endpoint` | `string` | HTTP endpoint to POST trace JSON to | |
| 173 | +| `headers` | `object` | Custom headers (e.g., auth tokens) | |
| 174 | + |
| 175 | +**How it works:** |
| 176 | + |
| 177 | +- All exporters run concurrently with the local file write via `Promise.allSettled` |
| 178 | +- A failing exporter never blocks local file storage or other exporters |
| 179 | +- If the server responds with `{ "url": "..." }`, the URL is displayed to the user |
| 180 | +- Exporters have a 10-second timeout |
| 181 | +- All export operations are best-effort — they never crash the CLI |
| 182 | + |
| 183 | +## Trace File Format |
| 184 | + |
| 185 | +Traces are stored as JSON files in the traces directory. The schema is versioned for forward compatibility. |
| 186 | + |
| 187 | +```json |
| 188 | +{ |
| 189 | + "version": 2, |
| 190 | + "traceId": "019cf4e2-...", |
| 191 | + "sessionId": "session-abc123", |
| 192 | + "startedAt": "2026-03-15T10:00:00.000Z", |
| 193 | + "endedAt": "2026-03-15T10:00:45.200Z", |
| 194 | + "metadata": { |
| 195 | + "model": "anthropic/claude-sonnet-4-20250514", |
| 196 | + "providerId": "anthropic", |
| 197 | + "agent": "builder", |
| 198 | + "variant": "high", |
| 199 | + "prompt": "optimize my most expensive queries", |
| 200 | + "userId": "user@example.com", |
| 201 | + "environment": "production", |
| 202 | + "version": "2.0.0", |
| 203 | + "tags": ["benchmark", "nightly"] |
| 204 | + }, |
| 205 | + "spans": [ |
| 206 | + { |
| 207 | + "spanId": "...", |
| 208 | + "parentSpanId": null, |
| 209 | + "name": "session-abc123", |
| 210 | + "kind": "session", |
| 211 | + "startTime": 1710500000000, |
| 212 | + "endTime": 1710500045200, |
| 213 | + "status": "ok" |
| 214 | + }, |
| 215 | + { |
| 216 | + "spanId": "...", |
| 217 | + "parentSpanId": "<session-span-id>", |
| 218 | + "name": "generation-1", |
| 219 | + "kind": "generation", |
| 220 | + "startTime": 1710500000100, |
| 221 | + "endTime": 1710500003500, |
| 222 | + "status": "ok", |
| 223 | + "model": { |
| 224 | + "modelId": "anthropic/claude-sonnet-4-20250514", |
| 225 | + "providerId": "anthropic" |
| 226 | + }, |
| 227 | + "finishReason": "stop", |
| 228 | + "cost": 0.005, |
| 229 | + "tokens": { |
| 230 | + "input": 1500, |
| 231 | + "output": 300, |
| 232 | + "reasoning": 100, |
| 233 | + "cacheRead": 200, |
| 234 | + "cacheWrite": 50, |
| 235 | + "total": 2150 |
| 236 | + } |
| 237 | + }, |
| 238 | + { |
| 239 | + "spanId": "...", |
| 240 | + "parentSpanId": "<generation-span-id>", |
| 241 | + "name": "sql_execute", |
| 242 | + "kind": "tool", |
| 243 | + "startTime": 1710500001000, |
| 244 | + "endTime": 1710500003000, |
| 245 | + "status": "ok", |
| 246 | + "tool": { "callId": "call-1", "durationMs": 2000 }, |
| 247 | + "input": { "query": "SELECT ..." }, |
| 248 | + "output": "10 rows returned", |
| 249 | + "attributes": { |
| 250 | + "de.warehouse.system": "snowflake", |
| 251 | + "de.warehouse.bytes_scanned": 45000000, |
| 252 | + "de.warehouse.estimated_cost_usd": 0.0012, |
| 253 | + "de.sql.validation.valid": true |
| 254 | + } |
| 255 | + } |
| 256 | + ], |
| 257 | + "summary": { |
| 258 | + "totalTokens": 2150, |
| 259 | + "totalCost": 0.005, |
| 260 | + "totalToolCalls": 1, |
| 261 | + "totalGenerations": 1, |
| 262 | + "duration": 45200, |
| 263 | + "status": "completed", |
| 264 | + "tokens": { |
| 265 | + "input": 1500, |
| 266 | + "output": 300, |
| 267 | + "reasoning": 100, |
| 268 | + "cacheRead": 200, |
| 269 | + "cacheWrite": 50 |
| 270 | + } |
| 271 | + } |
| 272 | +} |
| 273 | +``` |
| 274 | + |
| 275 | +### Span Types |
| 276 | + |
| 277 | +| Kind | Description | Key Fields | |
| 278 | +|------|-------------|------------| |
| 279 | +| `session` | Root span for the entire session | `input` (prompt), `output` (summary) | |
| 280 | +| `generation` | One LLM call (step-start to step-finish) | `model`, `finishReason`, `tokens`, `cost` | |
| 281 | +| `tool` | A tool invocation | `tool.callId`, `tool.durationMs`, `input`, `output` | |
| 282 | + |
| 283 | +### Domain Attribute Namespaces |
| 284 | + |
| 285 | +All domain-specific attributes use the `de.*` prefix and are stored in the `attributes` map on tool spans: |
| 286 | + |
| 287 | +| Prefix | Domain | |
| 288 | +|--------|--------| |
| 289 | +| `de.warehouse.*` | Warehouse metrics (bytes, credits, partitions, timing) | |
| 290 | +| `de.sql.*` | SQL quality (validation, lineage, schema changes) | |
| 291 | +| `de.dbt.*` | dbt operations (model status, tests, Jinja, DAG) | |
| 292 | +| `de.quality.*` | Data quality (row counts, freshness, anomalies) | |
| 293 | +| `de.cost.*` | Cost attribution (LLM + warehouse + storage) | |
| 294 | + |
| 295 | +## Crash Recovery |
| 296 | + |
| 297 | +Traces are designed to survive process crashes: |
| 298 | + |
| 299 | +1. **Immediate snapshot** — A trace file is written as soon as `startTrace()` is called, before any LLM interaction. Even if the process crashes immediately, a minimal trace file exists. |
| 300 | + |
| 301 | +2. **Incremental snapshots** — After every tool call and generation completion, the trace file is updated atomically (write to temp file, then rename). The file on disk always contains a valid, complete JSON document. |
| 302 | + |
| 303 | +3. **Crash handlers** — The `run` command registers `SIGINT`/`SIGTERM`/`beforeExit` handlers that flush the trace synchronously with a `"crashed"` status. |
| 304 | + |
| 305 | +4. **Status indicators** — Trace status tells you exactly what happened: |
| 306 | + |
| 307 | +| Status | Meaning | |
| 308 | +|--------|---------| |
| 309 | +| `completed` | Session finished normally | |
| 310 | +| `error` | Session finished with an error | |
| 311 | +| `running` | Session is still in progress (visible in live mode) | |
| 312 | +| `crashed` | Process was interrupted before the session completed | |
| 313 | + |
| 314 | +Crashed traces contain all data up to the last successful snapshot. You can view them normally with `altimate-code trace view`. |
| 315 | + |
| 316 | +## Historical Traces |
| 317 | + |
| 318 | +All traces are stored in the traces directory and persist across sessions. Use `trace list` to browse history: |
| 319 | + |
| 320 | +```bash |
| 321 | +# Show the last 50 traces |
| 322 | +altimate-code trace list -n 50 |
| 323 | + |
| 324 | +# View any historical trace |
| 325 | +altimate-code trace view <session-id> |
| 326 | +``` |
| 327 | + |
| 328 | +Traces are automatically pruned when `maxFiles` is exceeded (default: 100). The oldest traces are removed first. Set `maxFiles: 0` for unlimited retention. |
| 329 | + |
| 330 | +## Privacy |
| 331 | + |
| 332 | +Traces are stored **locally only** by default. They contain: |
| 333 | + |
| 334 | +- The prompt you sent |
| 335 | +- Tool inputs and outputs (SQL queries, file contents, command results) |
| 336 | +- Model responses |
| 337 | + |
| 338 | +If you configure remote exporters, trace data is sent to those endpoints. No trace data is included in the anonymous telemetry described in [Telemetry](telemetry.md). |
| 339 | + |
| 340 | +!!! warning "Sensitive Data" |
| 341 | + Traces may contain SQL queries, file paths, and command outputs from your session. If you share trace files or configure remote exporters, be aware that this data will be included. |
0 commit comments