deploy: harden otel-collector config (memory_limiter + sending_queue/retry)

The sample claimed it survives a backend outage but had neither a sending queue nor retry,
and no memory_limiter (OOM risk under backpressure). Add memory_limiter (first), explicit
batch sizing, and otlp retry_on_failure + in-memory sending_queue.

Author: RoyLin

CHANGELOG.md

@@ -11,6 +11,9 @@ All notable changes to a3s-observer will be documented in this file.
   fanotify `FAN_OPEN_PERM` v0.4.0); the file guard uses fanotify, not LSM-BPF, because `bpf`
   isn't in this kernel's `lsm=` set.
 - Removed redundant `unsafe` blocks in the eBPF probes (clean build).
+- `deploy/otel-collector.yaml` hardened for production: `memory_limiter` (OOM guard under
+  backpressure) + `sending_queue` / `retry_on_failure` so it actually survives a backend
+  outage (which the header claimed but the config didn't do).
 
 ## [0.5.1] — production hardening
 

deploy/otel-collector.yaml

@@ -31,19 +31,31 @@ receivers:
         parse_from: body
 
 processors:
-  batch: {}
+  # memory_limiter MUST be first — it sheds load before the Collector OOMs under backpressure
+  # (e.g. a slow/dead backend filling the sending queue).
+  memory_limiter:
+    check_interval: 5s
+    limit_percentage: 80
+    spike_limit_percentage: 20
+  batch:
+    timeout: 5s
+    send_batch_size: 1024
 
 exporters:
-  debug: # prints received records — use this to smoke-test the pipeline
-    verbosity: detailed
   otlp:
     endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT} # Tempo / Jaeger / Loki / vendor OTLP
     tls:
-      insecure: true
+      insecure: true # production: set false and configure CA / mTLS
+    retry_on_failure:
+      enabled: true
+    sending_queue:
+      enabled: true # buffer across a backend outage; add file_storage to also survive restarts
+  debug: # add to a pipeline's exporters to smoke-test (prints received records)
+    verbosity: detailed
 
 service:
   pipelines:
     logs:
       receivers: [filelog]
-      processors: [batch]
-      exporters: [debug, otlp]
+      processors: [memory_limiter, batch]
+      exporters: [otlp] # add `debug` to smoke-test the pipeline