docs: add issue 29 spec and baseline evidence

Refs: #29

Author: josecelano

docs/issues/ISSUE-29-research-high-cpu-load-after-udp-fix.md

@@ -0,0 +1,140 @@
+# Research high CPU load after UDP uptime recovery
+
+**Issue**: [#29](https://github.com/torrust/torrust-tracker-demo/issues/29)
+**Related**:
+[#21](https://github.com/torrust/torrust-tracker-demo/issues/21),
+[#19](https://github.com/torrust/torrust-tracker-demo/issues/19)
+
+## Overview
+
+The UDP uptime problem tracked in [#21](https://github.com/torrust/torrust-tracker-demo/issues/21)
+appears resolved: newTrackon is now healthy again after the conntrack fix and server resize.
+However, the live server is still running at a high CPU load, which leaves limited headroom for
+future traffic growth and may become the next reliability risk even before external uptime drops.
+
+This issue tracks a controlled investigation and remediation plan for the remaining CPU pressure.
+The work should change only one variable at a time, with an observation period after each change,
+so the impact of each action is measurable and attributable.
+
+## Current Baseline (2026-05-04)
+
+Live snapshot collected from the demo server on 2026-05-04:
+
+- Host load average: `8.50 / 8.27 / 8.20`
+- Host CPU summary (`mpstat -P ALL 1 1`):
+  - all CPUs: `%usr=35.33`, `%sys=15.38`, `%soft=19.69`, `%idle=29.20`
+  - CPU2: `%soft=100.00`, `%idle=0.00`
+- Top CPU consumers:
+  - `caddy`: about `279%`
+  - `torrust-tracker`: about `88%`
+  - `ksoftirqd/2`: visible in top CPU list
+- Docker container CPU snapshot:
+  - `caddy`: `295.60%`
+  - `tracker`: `91.84%`
+  - `mysql`: `2.63%`
+  - `grafana`: `0.29%`
+  - `prometheus`: `0.00%`
+- Current request rates from Prometheus:
+  - HTTP1: about `1982.32 req/s`
+  - UDP1: about `2124.28 req/s`
+  - Combined: about `4106.60 req/s`
+- Conntrack state:
+  - `nf_conntrack_count`: `423120`
+  - `nf_conntrack_max`: `1048576`
+  - utilization: about `40.35%`
+- RX steering state:
+  - `/sys/class/net/eth0/queues/rx-0/rps_cpus`: `00`
+  - `/sys/class/net/eth0/queues/rx-0/rps_flow_cnt`: `0`
+  - `net.core.rps_sock_flow_entries`: `0`
+
+## What We Know So Far
+
+### Conntrack is no longer the immediate bottleneck
+
+The live conntrack table is well below the configured limit, so this does not look like a repeat
+of the overflow problem fixed in [#21](https://github.com/torrust/torrust-tracker-demo/issues/21).
+
+### The packet path is still imbalanced on one CPU
+
+One CPU is saturated in softirq while the host still has idle capacity overall. That strongly
+suggests packet processing is concentrated on one RX path instead of being distributed across
+cores.
+
+The current live configuration confirms that RPS/RFS are disabled.
+
+### Caddy is also consuming several CPU cores
+
+This is not just a tracker-only problem. The HTTPS front-end is currently one of the largest CPU
+consumers on the host.
+
+The current Compose file exposes UDP 443 for Caddy:
+
+```yaml
+- "443:443/udp"
+```
+
+This is used for HTTP/3 over QUIC. It is not required for normal HTTPS over TCP. Standard HTTPS
+would continue to work without it; removing it would only disable HTTP/3.
+
+The deployed server currently mirrors the repository on this point: the live
+`/opt/torrust/docker-compose.yml` also exposes `443:443/udp`, and the host is listening on UDP 443.
+
+## Controlled Action Plan
+
+Important constraint: apply only one production change at a time and wait before taking the next
+step.
+
+### Phase 1 — Preserve a reproducible baseline
+
+- [x] Record a baseline evidence snapshot under `docs/issues/evidence/ISSUE-29/` before making
+      any changes. See `00-baseline-live-snapshot.md` and `2026-05-04-htop-snapshot.png`.
+- [x] Capture host load, per-CPU usage, top processes, docker stats, conntrack state, and RX
+      steering state.
+- [x] Record at minimum the live HTTP1 and UDP1 request rates from Prometheus.
+- [x] Record current newTrackon status for UDP1 and HTTP1.
+
+### Phase 2 — First isolated experiment: disable HTTP/3
+
+- [ ] Remove `"443:443/udp"` from the Caddy service in `server/opt/torrust/docker-compose.yml`.
+- [ ] Apply only that change on the live server and restart only Caddy.
+- [ ] Observe CPU, request rates, and external service health for an agreed window.
+- [ ] Decide whether Caddy CPU dropped materially enough to keep HTTP/3 disabled.
+
+Rationale: this is a small, isolated change that affects only HTTP/3/QUIC support and does not
+change normal HTTPS or the tracker's UDP listener on port 6969.
+
+### Phase 3 — Second isolated experiment: enable RPS/RFS
+
+- [ ] If CPU pressure remains high after Phase 2, enable RPS and RFS as documented in
+      `docs/udp-conntrack-runbook.md`.
+- [ ] Persist the configuration in the tracked server config.
+- [ ] Re-check whether softirq is still concentrated on one CPU.
+- [ ] Observe for an agreed window before taking further action.
+
+Rationale: this directly targets the observed one-core softirq saturation while leaving the
+application stack unchanged.
+
+### Phase 4 — Reassess architecture only after isolated tuning results exist
+
+- [ ] If both isolated experiments fail to provide enough headroom, evaluate moving the HTTPS
+      front-end or the UDP tracker onto separate hosts.
+- [ ] Treat host separation as a later step, not the first response.
+
+## Questions To Answer In This Issue
+
+1. How much of the current CPU load is explained by higher traffic versus avoidable packet-path
+   inefficiency?
+2. Does disabling HTTP/3 reduce Caddy CPU materially without unacceptable product impact?
+3. Does enabling RPS/RFS spread softirq work enough to restore headroom?
+4. After isolated tuning, is a single-host deployment still acceptable at current traffic?
+
+## Acceptance Criteria
+
+- [ ] A baseline evidence snapshot exists for the pre-change state, including HTTP1 and UDP1
+      request rates.
+- [ ] The first production change is isolated to a single variable and its effect is documented.
+- [ ] No follow-up production change is applied before the previous change has been observed.
+- [ ] A documented decision exists on whether `443:443/udp` should remain enabled for HTTP/3.
+- [ ] A documented decision exists on whether RPS/RFS should be deployed permanently.
+- [ ] The final issue conclusion states whether the current single-host design still has enough
+      CPU headroom.

docs/issues/evidence/ISSUE-29/00-baseline-live-snapshot.md

@@ -0,0 +1,88 @@
+<!-- cspell:ignore CPUPerc urlencode -->
+
+# ISSUE-29 Baseline Live Snapshot (Phase 1)
+
+## Context
+
+- Issue: [#29](https://github.com/torrust/torrust-tracker-demo/issues/29)
+- Capture timestamp (UTC): `2026-05-04T15:20:07Z`
+- Goal: record a pre-change baseline before any production tuning action.
+
+## Commands Used
+
+```bash
+ssh demotracker 'date -u +%Y-%m-%dT%H:%M:%SZ; uptime; nproc'
+ssh demotracker 'mpstat -P ALL 1 1'
+ssh demotracker 'ps -eo pid,comm,%cpu,%mem,stat --sort=-%cpu | head -20'
+ssh demotracker 'cd /opt/torrust && docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"'
+ssh demotracker 'sudo sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_udp_timeout net.netfilter.nf_conntrack_udp_timeout_stream'
+ssh demotracker 'cat /sys/class/net/eth0/queues/rx-0/rps_cpus; cat /sys/class/net/eth0/queues/rx-0/rps_flow_cnt; sudo sysctl net.core.rps_sock_flow_entries'
+ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(http_tracker_core_requests_received_total[5m]))"'
+ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(udp_tracker_server_requests_received_total[5m]))"'
+ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(http_tracker_core_responses_sent_total{result=\"error\"}[5m]))"'
+ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(udp_tracker_server_responses_sent_total{result=\"error\"}[5m]))"'
+```
+
+External uptime sampling source:
+
+```text
+https://newtrackon.com/raw
+```
+
+## Key Results
+
+### Host and CPU
+
+- `uptime`: `load average: 10.46, 9.44, 8.85`
+- `nproc`: `8`
+- `mpstat` (all CPUs): `%usr=32.81`, `%sys=16.01`, `%soft=20.08`, `%idle=30.97`
+- `mpstat` (CPU2): `%soft=100.00`, `%idle=0.00`
+
+### Top CPU Processes
+
+- `caddy`: about `279%`
+- `torrust-tracker`: about `88.4%`
+- `ksoftirqd/2`: about `14.6%`
+
+### Container CPU Snapshot
+
+- `caddy`: `323.16%`
+- `tracker`: `84.39%`
+- `mysql`: `6.63%`
+- `grafana`: `0.37%`
+- `prometheus`: `0.02%`
+
+### Conntrack and RX Steering
+
+- `nf_conntrack_max`: `1048576`
+- `nf_conntrack_count`: `424607`
+- `nf_conntrack_udp_timeout`: `10`
+- `nf_conntrack_udp_timeout_stream`: `15`
+- `/sys/class/net/eth0/queues/rx-0/rps_cpus`: `00`
+- `/sys/class/net/eth0/queues/rx-0/rps_flow_cnt`: `0`
+- `net.core.rps_sock_flow_entries`: `0`
+
+### Prometheus Request Rates
+
+- HTTP1 request rate: `1983.1789473684207 req/s`
+- UDP1 request rate: `2240.870175438596 req/s`
+- Combined request rate: `4224.0491228070167 req/s`
+- HTTP error rate query result: empty vector (`0` at sample time)
+- UDP error response rate: `26.34035087719298 req/s`
+
+### newTrackon Raw Snapshot (Sample)
+
+Observed at capture window from `newtrackon/raw`:
+
+- `https://http1.torrust-tracker-demo.com:443/announce` -> `Working`
+- `udp://udp1.torrust-tracker-demo.com:6969/announce` -> `Working`
+
+## Attached Evidence
+
+- `2026-05-04-htop-snapshot.png` (provided by maintainer)
+
+## Notes
+
+- This baseline is intentionally captured before changing any production setting.
+- The first planned production experiment remains Phase 2: disable Caddy HTTP/3
+  (`443:443/udp`) only, then observe before any further action.

docs/issues/evidence/ISSUE-29/2026-05-04-htop-snapshot.png

@@ -47,11 +47,13 @@ frontmatter
 fswc
 grafana
 healthcheck
+ksoftirqd
 leecher
 logfile
 misrouted
 mkpath
 Mailgun
+mpstat
 mysqladmin
 netnsid
 netfilter