Skip to content

Commit 3fc2ce8

Browse files
committed
docs(issue-29): add self-contained phase 3 handoff runbook
Expand phase 3 evidence with operator handoff details so a new maintainer can execute T+1h and T+next-day checkpoints without chat context. Includes: objective/success criteria, required access, exact checkpoint commands, required file updates, commit workflow, and guardrails for keeping live and repo configs in sync.
1 parent 353638b commit 3fc2ce8

1 file changed

Lines changed: 133 additions & 1 deletion

File tree

docs/issues/evidence/ISSUE-29/02-phase3-enable-rps-rfs-execution.md

Lines changed: 133 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
<!-- cspell:ignore CPUPerc ksoftirqd rps_cpus rps_flow_cnt -->
1+
<!-- cspell:ignore CPUPerc ksoftirqd rps_cpus rps_flow_cnt urlencode -->
22

33
# ISSUE-29 Phase 3 Execution - Enable RPS/RFS
44

@@ -116,3 +116,135 @@ Agreed observation windows for Phase 3:
116116

117117
Capture the same metrics at each checkpoint: `mpstat`, `docker stats`,
118118
Prometheus HTTP1/UDP1 rates, and a `newtrackon.com/raw` sample.
119+
120+
## Operator Handoff Guide (Self-Contained)
121+
122+
This section is intentionally explicit so another operator can continue Phase 3
123+
without prior context from chat history.
124+
125+
### What this investigation is trying to prove
126+
127+
Phase 3 hypothesis:
128+
129+
- Enabling RPS/RFS reduces the single-core softirq hotspot (previously CPU2 at
130+
about 100% softirq) by distributing packet processing across CPUs.
131+
132+
Success condition for follow-up checkpoints:
133+
134+
- CPU2 `%soft` remains materially below pre-change saturation.
135+
- `%soft` is distributed across multiple CPUs, not pinned to one core.
136+
- HTTP1/UDP1 external status remains `Working`.
137+
- No obvious regression in request rates relative to normal load.
138+
139+
### Current known-good live settings
140+
141+
The live host and this repository should both contain:
142+
143+
- `net.core.rps_sock_flow_entries = 32768`
144+
- `/sys/class/net/eth0/queues/rx-0/rps_cpus = ff`
145+
- `/sys/class/net/eth0/queues/rx-0/rps_flow_cnt = 4096`
146+
147+
Persistent files expected:
148+
149+
- Live host: `/etc/sysctl.d/98-rps-rfs.conf`, `/etc/cron.d/rps-rfs`
150+
- Repository mirror:
151+
- `server/etc/sysctl.d/98-rps-rfs.conf`
152+
- `server/etc/cron.d/rps-rfs`
153+
154+
### Required access and tools
155+
156+
- SSH alias `demotracker` must work from local machine.
157+
- Prometheus endpoint must be reachable on host at `http://127.0.0.1:9090`.
158+
- `mpstat` must be available on host.
159+
160+
### Checkpoint command bundle (copy/paste)
161+
162+
Run these commands from local workstation at each checkpoint.
163+
164+
1. Capture host/runtime metrics:
165+
166+
```bash
167+
ssh demotracker 'set -e
168+
echo "=== capture_utc ==="; date -u +%Y-%m-%dT%H:%M:%SZ
169+
echo "=== uptime ==="; uptime
170+
echo "=== rps_state ==="
171+
cat /proc/sys/net/core/rps_sock_flow_entries
172+
cat /sys/class/net/eth0/queues/rx-0/rps_cpus
173+
cat /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
174+
echo "=== mpstat ==="; mpstat -P ALL 1 1
175+
echo "=== top_cpu ==="; ps -eo pid,comm,%cpu,%mem,stat --sort=-%cpu | head -20
176+
echo "=== docker_stats ==="
177+
cd /opt/torrust && docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"'
178+
```
179+
180+
1. Capture Prometheus rates (numeric values):
181+
182+
```bash
183+
ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(http_tracker_core_requests_received_total[5m]))"' | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["data"]["result"][0]["value"][1])'
184+
ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(udp_tracker_server_requests_received_total[5m]))"' | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["data"]["result"][0]["value"][1])'
185+
```
186+
187+
1. Capture external probe sample:
188+
189+
```bash
190+
curl -s https://newtrackon.com/raw | grep -E 'https://http1.torrust-tracker-demo.com:443/announce|udp://udp1.torrust-tracker-demo.com:6969/announce' | head -10
191+
```
192+
193+
### What to add to this file at each checkpoint
194+
195+
For each checkpoint (`T+1 h`, `T+next day`), append a new section:
196+
197+
- `## T+1 h Observation (<timestamp>)` or
198+
`## T+next-day Observation (<timestamp>)`
199+
- Include:
200+
- capture timestamp
201+
- host load average
202+
- `mpstat` all-CPU summary
203+
- `mpstat` CPU2 `%soft` and `%idle`
204+
- whether `%soft` is distributed across CPUs
205+
- container CPU snapshot for `caddy`, `tracker`, `mysql`, `grafana`, `prometheus`
206+
- Prometheus HTTP1 and UDP1 rates
207+
- newtrackon status for HTTP1 and UDP1
208+
- Add a short assessment paragraph: improved/stable/regressed.
209+
210+
Then update table status in this file:
211+
212+
- mark checkpoint `complete`
213+
- replace generic time with actual capture time where appropriate
214+
215+
### Required updates outside this file
216+
217+
After each checkpoint, also update issue tracker doc:
218+
219+
- `docs/issues/ISSUE-29-research-high-cpu-load-after-udp-fix.md`
220+
221+
Update Phase 3 checklist items as evidence is completed.
222+
223+
### Commit workflow after each checkpoint
224+
225+
1. Run validations:
226+
227+
```bash
228+
./scripts/lint.sh
229+
```
230+
231+
1. Commit with signed Conventional Commit, for example:
232+
233+
```text
234+
docs(issue-29): record phase 3 T+1h observation
235+
```
236+
237+
1. Push:
238+
239+
```bash
240+
git push origin main
241+
```
242+
243+
### Guardrails
244+
245+
- Keep Phase 3 as one variable: do not change unrelated server settings during
246+
observation windows.
247+
- If any live server config is changed, mirror the exact same file content under
248+
`server/` in this repository.
249+
- If metrics look contradictory, capture raw command output first and defer
250+
conclusions until evidence is recorded.

0 commit comments

Comments
 (0)