Skip to content

Commit 79afc74

Browse files
mtblzMateusz
mtblz
and
Mateusz
authored
docs: explain the OpenTelemetry logs setup in the chart README (#534)
Co-authored-by: Mateusz <m4itee@bularz.net>
1 parent bbea6b1 commit 79afc74

2 files changed

Lines changed: 232 additions & 0 deletions

File tree

‎charts/netdata/README.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2313,6 +2313,122 @@ In case of `child` instance it is a bit simpler. By default, hostPath: `/var/lib
23132313
in: `/var/lib/netdata`. You can disable it, but this option is pretty much required in a real life scenario, as without
23142314
it each pod deletion will result in a new replication node for a parent.
23152315

2316+
### Collecting logs with OpenTelemetry
2317+
2318+
Netdata can ingest, store, and visualize your cluster's container logs through OpenTelemetry.
2319+
This relies on two components — **both disabled by default**:
2320+
2321+
- **`netdataOpentelemetry`** — Netdata's OpenTelemetry receiver. It runs as its own `Deployment`
2322+
(the `netdata-otel` node) and listens for OTLP data on port `4317`. It receives, stores, and
2323+
displays the logs in the Netdata UI. It does **not** collect logs itself.
2324+
- **`otel-collector`** — the upstream [OpenTelemetry Collector](https://opentelemetry.io/docs/platforms/kubernetes/helm/collector/),
2325+
bundled as an optional subchart. It runs as a `DaemonSet` (one pod per node), reads each node's
2326+
local container log files, and forwards them over OTLP to `netdataOpentelemetry`.
2327+
2328+
`netdataOpentelemetry` is the destination; `otel-collector` is what feeds it. The collector is
2329+
bundled only as a convenient, working default, and is off by default because it is not the only
2330+
option. If you already run a log pipeline (Fluent Bit, Vector, an existing Collector, or any
2331+
OTLP-capable agent), leave `otel-collector` disabled and point that pipeline's OTLP exporter at
2332+
the `netdata-otel` service on port `4317` instead.
2333+
2334+
**Log flow:**
2335+
2336+
```
2337+
container stdout/stderr
2338+
│ the container runtime writes them to node-local files (/var/log/pods/…)
2339+
▼
2340+
otel-collector DaemonSet (one pod per node, needs host log access)
2341+
│ reads each node's log files, pushes over OTLP
2342+
▼
2343+
netdata-otel:4317 → stored and shown in the Netdata UI
2344+
```
2345+
2346+
#### Securing the endpoint with TLS
2347+
2348+
By default the `netdata-otel` receiver listens on port `4317` in **plaintext** — TLS is **disabled**
2349+
(`tls_cert_path`, `tls_key_path`, and `tls_ca_cert_path` are unset in
2350+
`netdataOpentelemetry.configs.otel.data`). The steps below turn it on with a self-signed
2351+
certificate. TLS affects **both sides**: the receiver must serve the certificate, and every client
2352+
(including the bundled `otel-collector`) must be switched to TLS, or it will stop delivering data.
2353+
2354+
**1. Generate a self-signed certificate and key** (Linux, `openssl`):
2355+
2356+
```
2357+
openssl req -x509 -newkey rsa:4096 -nodes \
2358+
-keyout tls.key -out tls.crt -days 365 \
2359+
-subj "/CN=netdata-otel"
2360+
```
2361+
2362+
**2. Create a Kubernetes TLS secret** from those files, in the chart's namespace:
2363+
2364+
```
2365+
kubectl create secret tls netdata-otel-tls \
2366+
--cert=tls.crt --key=tls.key \
2367+
--namespace <your-namespace>
2368+
```
2369+
2370+
**3. Mount the secret into the receiver and point the config at it.** The certificate paths live
2371+
inside `netdataOpentelemetry.configs.otel.data`, which is a single block — supply it in full with
2372+
the two `tls_*_path` values filled in (keep the `metrics` and `logs` sections in sync with the
2373+
chart's `values.yaml`). Mounting the secret alone does nothing until these paths are set:
2374+
2375+
```yaml
2376+
netdataOpentelemetry:
2377+
extraVolumes:
2378+
- name: otel-tls
2379+
secret:
2380+
secretName: netdata-otel-tls
2381+
extraVolumeMounts:
2382+
- name: otel-tls
2383+
mountPath: /etc/netdata/otel-certs
2384+
readOnly: true
2385+
configs:
2386+
otel:
2387+
data: |
2388+
endpoint:
2389+
path: "0.0.0.0:4317"
2390+
tls_cert_path: /etc/netdata/otel-certs/tls.crt
2391+
tls_key_path: /etc/netdata/otel-certs/tls.key
2392+
tls_ca_cert_path: null
2393+
metrics:
2394+
print_flattened: false
2395+
buffer_samples: 10
2396+
throttle_charts: 100
2397+
chart_configs_dir: otel.d/v1/metrics
2398+
logs:
2399+
journal_dir: otel/v1
2400+
size_of_journal_file: "100MB"
2401+
number_of_journal_files: 10
2402+
size_of_journal_files: "1GB"
2403+
duration_of_journal_files: "7 days"
2404+
duration_of_journal_file: "2 hours"
2405+
store_otlp_json: false
2406+
```
2407+
2408+
**4. Switch every client to TLS.** A TLS listener rejects plaintext connections, so any sender must
2409+
be reconfigured — otherwise logs silently stop arriving. For the bundled `otel-collector`, enable
2410+
TLS on its OTLP exporter. Because the certificate is self-signed, skip verification with
2411+
`insecure_skip_verify` — this keeps the connection encrypted but does not validate the certificate
2412+
chain (only the `tls` block is overridden; the exporter's `endpoint` is kept from the chart
2413+
defaults):
2414+
2415+
```yaml
2416+
otel-collector:
2417+
config:
2418+
exporters:
2419+
otlp:
2420+
tls:
2421+
insecure: false
2422+
insecure_skip_verify: true
2423+
```
2424+
2425+
Apply the same change to any external OTLP client — Fluent Bit, Vector, or another Collector —
2426+
pointing at the `netdata-otel` service.
2427+
2428+
> For production, replace the self-signed certificate with one issued by a trusted CA, give it a
2429+
> `CN`/`SAN` that matches the `netdata-otel` service DNS name, and have clients trust that CA via
2430+
> `tls_ca_cert_path` instead of skipping verification.
2431+
23162432
### Service discovery and supported services
23172433

23182434
Netdata's [service discovery](https://github.com/netdata/agent-service-discovery/), which is installed as part of the

‎templates/netdata-README.md.gotmpl‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,122 @@ In case of `child` instance it is a bit simpler. By default, hostPath: `/var/lib
221221
in: `/var/lib/netdata`. You can disable it, but this option is pretty much required in a real life scenario, as without
222222
it each pod deletion will result in a new replication node for a parent.
223223

224+
### Collecting logs with OpenTelemetry
225+
226+
Netdata can ingest, store, and visualize your cluster's container logs through OpenTelemetry.
227+
This relies on two components — **both disabled by default**:
228+
229+
- **`netdataOpentelemetry`** — Netdata's OpenTelemetry receiver. It runs as its own `Deployment`
230+
(the `netdata-otel` node) and listens for OTLP data on port `4317`. It receives, stores, and
231+
displays the logs in the Netdata UI. It does **not** collect logs itself.
232+
- **`otel-collector`** — the upstream [OpenTelemetry Collector](https://opentelemetry.io/docs/platforms/kubernetes/helm/collector/),
233+
bundled as an optional subchart. It runs as a `DaemonSet` (one pod per node), reads each node's
234+
local container log files, and forwards them over OTLP to `netdataOpentelemetry`.
235+
236+
`netdataOpentelemetry` is the destination; `otel-collector` is what feeds it. The collector is
237+
bundled only as a convenient, working default, and is off by default because it is not the only
238+
option. If you already run a log pipeline (Fluent Bit, Vector, an existing Collector, or any
239+
OTLP-capable agent), leave `otel-collector` disabled and point that pipeline's OTLP exporter at
240+
the `netdata-otel` service on port `4317` instead.
241+
242+
**Log flow:**
243+
244+
```
245+
container stdout/stderr
246+
│ the container runtime writes them to node-local files (/var/log/pods/…)
247+
▼
248+
otel-collector DaemonSet (one pod per node, needs host log access)
249+
│ reads each node's log files, pushes over OTLP
250+
▼
251+
netdata-otel:4317 → stored and shown in the Netdata UI
252+
```
253+
254+
#### Securing the endpoint with TLS
255+
256+
By default the `netdata-otel` receiver listens on port `4317` in **plaintext** — TLS is **disabled**
257+
(`tls_cert_path`, `tls_key_path`, and `tls_ca_cert_path` are unset in
258+
`netdataOpentelemetry.configs.otel.data`). The steps below turn it on with a self-signed
259+
certificate. TLS affects **both sides**: the receiver must serve the certificate, and every client
260+
(including the bundled `otel-collector`) must be switched to TLS, or it will stop delivering data.
261+
262+
**1. Generate a self-signed certificate and key** (Linux, `openssl`):
263+
264+
```
265+
openssl req -x509 -newkey rsa:4096 -nodes \
266+
-keyout tls.key -out tls.crt -days 365 \
267+
-subj "/CN=netdata-otel"
268+
```
269+
270+
**2. Create a Kubernetes TLS secret** from those files, in the chart's namespace:
271+
272+
```
273+
kubectl create secret tls netdata-otel-tls \
274+
--cert=tls.crt --key=tls.key \
275+
--namespace <your-namespace>
276+
```
277+
278+
**3. Mount the secret into the receiver and point the config at it.** The certificate paths live
279+
inside `netdataOpentelemetry.configs.otel.data`, which is a single block — supply it in full with
280+
the two `tls_*_path` values filled in (keep the `metrics` and `logs` sections in sync with the
281+
chart's `values.yaml`). Mounting the secret alone does nothing until these paths are set:
282+
283+
```yaml
284+
netdataOpentelemetry:
285+
extraVolumes:
286+
- name: otel-tls
287+
secret:
288+
secretName: netdata-otel-tls
289+
extraVolumeMounts:
290+
- name: otel-tls
291+
mountPath: /etc/netdata/otel-certs
292+
readOnly: true
293+
configs:
294+
otel:
295+
data: |
296+
endpoint:
297+
path: "0.0.0.0:4317"
298+
tls_cert_path: /etc/netdata/otel-certs/tls.crt
299+
tls_key_path: /etc/netdata/otel-certs/tls.key
300+
tls_ca_cert_path: null
301+
metrics:
302+
print_flattened: false
303+
buffer_samples: 10
304+
throttle_charts: 100
305+
chart_configs_dir: otel.d/v1/metrics
306+
logs:
307+
journal_dir: otel/v1
308+
size_of_journal_file: "100MB"
309+
number_of_journal_files: 10
310+
size_of_journal_files: "1GB"
311+
duration_of_journal_files: "7 days"
312+
duration_of_journal_file: "2 hours"
313+
store_otlp_json: false
314+
```
315+
316+
**4. Switch every client to TLS.** A TLS listener rejects plaintext connections, so any sender must
317+
be reconfigured — otherwise logs silently stop arriving. For the bundled `otel-collector`, enable
318+
TLS on its OTLP exporter. Because the certificate is self-signed, skip verification with
319+
`insecure_skip_verify` — this keeps the connection encrypted but does not validate the certificate
320+
chain (only the `tls` block is overridden; the exporter's `endpoint` is kept from the chart
321+
defaults):
322+
323+
```yaml
324+
otel-collector:
325+
config:
326+
exporters:
327+
otlp:
328+
tls:
329+
insecure: false
330+
insecure_skip_verify: true
331+
```
332+
333+
Apply the same change to any external OTLP client — Fluent Bit, Vector, or another Collector —
334+
pointing at the `netdata-otel` service.
335+
336+
> For production, replace the self-signed certificate with one issued by a trusted CA, give it a
337+
> `CN`/`SAN` that matches the `netdata-otel` service DNS name, and have clients trust that CA via
338+
> `tls_ca_cert_path` instead of skipping verification.
339+
224340
### Service discovery and supported services
225341

226342
Netdata's [service discovery](https://github.com/netdata/agent-service-discovery/), which is installed as part of the

0 commit comments

Comments
 (0)