|
2 | 2 |
|
3 | 3 | proto is built on [OpenTelemetry](https://opentelemetry.io/) — the vendor-neutral observability standard. All traces, spans, and metrics use OTLP format and can be exported to any compatible backend. |
4 | 4 |
|
5 | | -## Langfuse (built-in, recommended) |
| 5 | +## Opt-in by default |
6 | 6 |
|
7 | | -proto ships with a Langfuse exporter. Set these environment variables to activate it — no other configuration needed: |
| 7 | +**No telemetry leaves the host until you explicitly enable it.** This is true for every exporter — OTLP, Langfuse, file output. Set: |
8 | 8 |
|
9 | | -```bash |
10 | | -export LANGFUSE_PUBLIC_KEY="pk-lf-..." |
11 | | -export LANGFUSE_SECRET_KEY="sk-lf-..." |
12 | | -export LANGFUSE_BASE_URL="https://your-langfuse-instance.example.com" # optional, defaults to cloud |
| 9 | +```json |
| 10 | +{ "telemetry": { "enabled": true } } |
13 | 11 | ``` |
14 | 12 |
|
15 | | -What is traced: |
| 13 | +in `settings.json`, or pass `--telemetry` on the CLI. Without that flag, nothing is sent anywhere — even if Langfuse env vars are present. |
| 14 | + |
| 15 | +## Default ingress: `otel.proto-labs.ai` |
| 16 | + |
| 17 | +When opted in, traces ship to the homelab LGTM stack at `https://otel.proto-labs.ai` over OTLP/HTTP. The endpoint is bearer-token authenticated: |
| 18 | + |
| 19 | +```json |
| 20 | +{ |
| 21 | + "telemetry": { "enabled": true }, |
| 22 | + "env": { |
| 23 | + "OTEL_INGRESS_TOKEN": "<token from Infisical homelab-media/prod>" |
| 24 | + } |
| 25 | +} |
| 26 | +``` |
| 27 | + |
| 28 | +Without `OTEL_INGRESS_TOKEN`, the ingress returns 401 (debug-logged in `~/.proto/debug/latest`). |
| 29 | + |
| 30 | +## Langfuse (LLM-grade trace UI) |
| 31 | + |
| 32 | +Langfuse is wired in addition to the LGTM stack — keeps prompt-grade trace debugging available alongside Tempo's APM views. Activate by setting all of these in `settings.json` `env`: |
| 33 | + |
| 34 | +```json |
| 35 | +{ |
| 36 | + "telemetry": { "enabled": true }, |
| 37 | + "env": { |
| 38 | + "LANGFUSE_PUBLIC_KEY": "pk-lf-...", |
| 39 | + "LANGFUSE_SECRET_KEY": "sk-lf-...", |
| 40 | + "LANGFUSE_BASE_URL": "https://your-langfuse-instance.example.com" |
| 41 | + } |
| 42 | +} |
| 43 | +``` |
| 44 | + |
| 45 | +`LANGFUSE_BASE_URL` is optional; defaults to `https://cloud.langfuse.com`. |
| 46 | + |
| 47 | +What's traced: |
16 | 48 |
|
17 | 49 | - Every session turn |
18 | 50 | - All LLM calls (all providers) with token counts |
19 | 51 | - Tool calls with input/output |
20 | 52 | - Sub-agent spawns and completions |
| 53 | +- Model reasoning content as `gen_ai.response.thinking` span attribute (when surfaced by the model or gateway) |
21 | 54 |
|
22 | | -## OpenTelemetry configuration |
23 | | - |
24 | | -Configure via `settings.json` or environment variables: |
| 55 | +## Configuration reference |
25 | 56 |
|
26 | | -| Setting | Env var | CLI flag | Values | Default | |
27 | | -| -------------------- | -------------------------- | ---------------------------------- | --------------- | ------- | |
28 | | -| `telemetry.enabled` | `PROTO_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | `true`/`false` | `false` | |
29 | | -| `telemetry.target` | `PROTO_TELEMETRY_TARGET` | `--telemetry-target <local\|otel>` | `local`, `otel` | `local` | |
30 | | -| `telemetry.endpoint` | `PROTO_TELEMETRY_ENDPOINT` | `--telemetry-endpoint <url>` | OTLP URL | — | |
| 57 | +| Setting | Env var | CLI flag | Values | Default | |
| 58 | +| ------------------------ | -------------------------- | ----------------------------------- | --------------- | --------------------------------- | |
| 59 | +| `telemetry.enabled` | `PROTO_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | `true`/`false` | `false` _(opt-in)_ | |
| 60 | +| `telemetry.target` | `PROTO_TELEMETRY_TARGET` | `--telemetry-target <local\|otel>` | `local`, `otel` | `local` | |
| 61 | +| `telemetry.otlpEndpoint` | `PROTO_TELEMETRY_ENDPOINT` | `--telemetry-otlp-endpoint <url>` | OTLP URL | `https://otel.proto-labs.ai` | |
| 62 | +| `telemetry.otlpProtocol` | — | `--telemetry-otlp-protocol <proto>` | `http`, `grpc` | `http` | |
| 63 | +| `telemetry.logPrompts` | — | `--telemetry-log-prompts` | `true`/`false` | `true` | |
| 64 | +| `OTEL_INGRESS_TOKEN` | — | — | bearer token | — (required for default endpoint) | |
31 | 65 |
|
32 | 66 | ### File-based local output |
33 | 67 |
|
| 68 | +For local-only debugging without shipping anywhere: |
| 69 | + |
34 | 70 | ```json |
35 | 71 | { |
36 | 72 | "telemetry": { |
37 | 73 | "enabled": true, |
38 | 74 | "target": "local", |
39 | | - "logFile": "~/.proto/telemetry/traces.jsonl" |
| 75 | + "outfile": "~/.proto/telemetry/traces.jsonl" |
40 | 76 | } |
41 | 77 | } |
42 | 78 | ``` |
43 | 79 |
|
44 | | -### Export to any OTLP backend (Jaeger, Datadog, etc.) |
| 80 | +### Override to a different OTLP backend |
| 81 | + |
| 82 | +If you run a local OTel Collector or want to point at a different vendor: |
45 | 83 |
|
46 | 84 | ```json |
47 | 85 | { |
48 | 86 | "telemetry": { |
49 | 87 | "enabled": true, |
50 | | - "target": "otel", |
51 | | - "endpoint": "http://localhost:4318/v1/traces" |
| 88 | + "otlpEndpoint": "http://localhost:4318", |
| 89 | + "otlpProtocol": "http" |
52 | 90 | } |
53 | 91 | } |
54 | 92 | ``` |
55 | 93 |
|
| 94 | +For a local gRPC collector, set `otlpProtocol: "grpc"` and use port `4317`. |
| 95 | + |
56 | 96 | ## What is instrumented |
57 | 97 |
|
58 | 98 | - **Session turns** — user prompt, model response, duration |
59 | | -- **LLM calls** — provider, model, input/output tokens, latency |
| 99 | +- **LLM calls** — provider, model, input/output/thinking tokens, latency, reasoning content |
60 | 100 | - **Tool calls** — name, arguments, result, duration |
61 | 101 | - **Sub-agent lifecycle** — spawn, completion, token usage |
62 | 102 | - **Harness interventions** — doom loop detection, multi-sample retries |
| 103 | +- **Hook executions** — which hook fired, success, duration, exit code, captured stdout/stderr |
63 | 104 |
|
64 | 105 | ## Privacy |
65 | 106 |
|
66 | | -Traces include prompt content and tool outputs by default. For production environments, configure sampling or filtering at your OTLP collector level to avoid capturing sensitive data. |
| 107 | +The opt-in default means privacy is the baseline — silence unless you say otherwise. When opted in: |
| 108 | + |
| 109 | +- `telemetry.logPrompts` controls whether prompt content + reasoning text are attached to spans (default `true`). |
| 110 | +- Reasoning text and completion content are truncated at 10K chars on each span. |
| 111 | +- For production deployments shipping to shared infra, configure sampling/filtering at the OTel Collector layer. |
| 112 | +- Local sandboxed runs default to a per-process session ID; no stable user identifier is exported. |
0 commit comments