|
1 | | -<!-- cspell:ignore CPUPerc ksoftirqd rps_cpus rps_flow_cnt --> |
| 1 | +<!-- cspell:ignore CPUPerc ksoftirqd rps_cpus rps_flow_cnt urlencode --> |
2 | 2 |
|
3 | 3 | # ISSUE-29 Phase 3 Execution - Enable RPS/RFS |
4 | 4 |
|
@@ -116,3 +116,135 @@ Agreed observation windows for Phase 3: |
116 | 116 |
|
117 | 117 | Capture the same metrics at each checkpoint: `mpstat`, `docker stats`, |
118 | 118 | Prometheus HTTP1/UDP1 rates, and a `newtrackon.com/raw` sample. |
| 119 | + |
| 120 | +## Operator Handoff Guide (Self-Contained) |
| 121 | + |
| 122 | +This section is intentionally explicit so another operator can continue Phase 3 |
| 123 | +without prior context from chat history. |
| 124 | + |
| 125 | +### What this investigation is trying to prove |
| 126 | + |
| 127 | +Phase 3 hypothesis: |
| 128 | + |
| 129 | +- Enabling RPS/RFS reduces the single-core softirq hotspot (previously CPU2 at |
| 130 | + about 100% softirq) by distributing packet processing across CPUs. |
| 131 | + |
| 132 | +Success condition for follow-up checkpoints: |
| 133 | + |
| 134 | +- CPU2 `%soft` remains materially below pre-change saturation. |
| 135 | +- `%soft` is distributed across multiple CPUs, not pinned to one core. |
| 136 | +- HTTP1/UDP1 external status remains `Working`. |
| 137 | +- No obvious regression in request rates relative to normal load. |
| 138 | + |
| 139 | +### Current known-good live settings |
| 140 | + |
| 141 | +The live host and this repository should both contain: |
| 142 | + |
| 143 | +- `net.core.rps_sock_flow_entries = 32768` |
| 144 | +- `/sys/class/net/eth0/queues/rx-0/rps_cpus = ff` |
| 145 | +- `/sys/class/net/eth0/queues/rx-0/rps_flow_cnt = 4096` |
| 146 | + |
| 147 | +Persistent files expected: |
| 148 | + |
| 149 | +- Live host: `/etc/sysctl.d/98-rps-rfs.conf`, `/etc/cron.d/rps-rfs` |
| 150 | +- Repository mirror: |
| 151 | + - `server/etc/sysctl.d/98-rps-rfs.conf` |
| 152 | + - `server/etc/cron.d/rps-rfs` |
| 153 | + |
| 154 | +### Required access and tools |
| 155 | + |
| 156 | +- SSH alias `demotracker` must work from local machine. |
| 157 | +- Prometheus endpoint must be reachable on host at `http://127.0.0.1:9090`. |
| 158 | +- `mpstat` must be available on host. |
| 159 | + |
| 160 | +### Checkpoint command bundle (copy/paste) |
| 161 | + |
| 162 | +Run these commands from local workstation at each checkpoint. |
| 163 | + |
| 164 | +1. Capture host/runtime metrics: |
| 165 | + |
| 166 | +```bash |
| 167 | +ssh demotracker 'set -e |
| 168 | +echo "=== capture_utc ==="; date -u +%Y-%m-%dT%H:%M:%SZ |
| 169 | +echo "=== uptime ==="; uptime |
| 170 | +echo "=== rps_state ===" |
| 171 | +cat /proc/sys/net/core/rps_sock_flow_entries |
| 172 | +cat /sys/class/net/eth0/queues/rx-0/rps_cpus |
| 173 | +cat /sys/class/net/eth0/queues/rx-0/rps_flow_cnt |
| 174 | +echo "=== mpstat ==="; mpstat -P ALL 1 1 |
| 175 | +echo "=== top_cpu ==="; ps -eo pid,comm,%cpu,%mem,stat --sort=-%cpu | head -20 |
| 176 | +echo "=== docker_stats ===" |
| 177 | +cd /opt/torrust && docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"' |
| 178 | +``` |
| 179 | + |
| 180 | +1. Capture Prometheus rates (numeric values): |
| 181 | + |
| 182 | +```bash |
| 183 | +ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(http_tracker_core_requests_received_total[5m]))"' | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["data"]["result"][0]["value"][1])' |
| 184 | +ssh demotracker 'curl -sG "http://127.0.0.1:9090/api/v1/query" --data-urlencode "query=sum(rate(udp_tracker_server_requests_received_total[5m]))"' | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["data"]["result"][0]["value"][1])' |
| 185 | +``` |
| 186 | + |
| 187 | +1. Capture external probe sample: |
| 188 | + |
| 189 | +```bash |
| 190 | +curl -s https://newtrackon.com/raw | grep -E 'https://http1.torrust-tracker-demo.com:443/announce|udp://udp1.torrust-tracker-demo.com:6969/announce' | head -10 |
| 191 | +``` |
| 192 | + |
| 193 | +### What to add to this file at each checkpoint |
| 194 | + |
| 195 | +For each checkpoint (`T+1 h`, `T+next day`), append a new section: |
| 196 | + |
| 197 | +- `## T+1 h Observation (<timestamp>)` or |
| 198 | + `## T+next-day Observation (<timestamp>)` |
| 199 | +- Include: |
| 200 | + - capture timestamp |
| 201 | + - host load average |
| 202 | + - `mpstat` all-CPU summary |
| 203 | + - `mpstat` CPU2 `%soft` and `%idle` |
| 204 | + - whether `%soft` is distributed across CPUs |
| 205 | + - container CPU snapshot for `caddy`, `tracker`, `mysql`, `grafana`, `prometheus` |
| 206 | + - Prometheus HTTP1 and UDP1 rates |
| 207 | + - newtrackon status for HTTP1 and UDP1 |
| 208 | +- Add a short assessment paragraph: improved/stable/regressed. |
| 209 | + |
| 210 | +Then update table status in this file: |
| 211 | + |
| 212 | +- mark checkpoint `complete` |
| 213 | +- replace generic time with actual capture time where appropriate |
| 214 | + |
| 215 | +### Required updates outside this file |
| 216 | + |
| 217 | +After each checkpoint, also update issue tracker doc: |
| 218 | + |
| 219 | +- `docs/issues/ISSUE-29-research-high-cpu-load-after-udp-fix.md` |
| 220 | + |
| 221 | +Update Phase 3 checklist items as evidence is completed. |
| 222 | + |
| 223 | +### Commit workflow after each checkpoint |
| 224 | + |
| 225 | +1. Run validations: |
| 226 | + |
| 227 | +```bash |
| 228 | +./scripts/lint.sh |
| 229 | +``` |
| 230 | + |
| 231 | +1. Commit with signed Conventional Commit, for example: |
| 232 | + |
| 233 | +```text |
| 234 | +docs(issue-29): record phase 3 T+1h observation |
| 235 | +``` |
| 236 | + |
| 237 | +1. Push: |
| 238 | + |
| 239 | +```bash |
| 240 | +git push origin main |
| 241 | +``` |
| 242 | + |
| 243 | +### Guardrails |
| 244 | + |
| 245 | +- Keep Phase 3 as one variable: do not change unrelated server settings during |
| 246 | + observation windows. |
| 247 | +- If any live server config is changed, mirror the exact same file content under |
| 248 | + `server/` in this repository. |
| 249 | +- If metrics look contradictory, capture raw command output first and defer |
| 250 | + conclusions until evidence is recorded. |
0 commit comments