Skip to content

Commit d86668d

Browse files
committed
docs(issue-29): update draft blog post with final phase 3 outcome
Add the T+next-day findings and final operational outcome for ISSUE-29. The draft now clarifies that RPS/RFS fixed the one-core softirq hotspot, but overall host load remained high, so capacity follow-up moved to ISSUE-30. Also updates references to include the May 6 htop snapshot. Refs: #29
1 parent 3877e76 commit d86668d

1 file changed

Lines changed: 32 additions & 10 deletions

File tree

docs/issues/evidence/ISSUE-29/03-draft-blog-post-rps-rfs-softirq-hotspot.md

Lines changed: 32 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,15 @@ We tested two isolated changes:
1818
1. Disable HTTP/3 (QUIC) on Caddy by removing UDP 443.
1919
2. Enable RPS/RFS to distribute packet processing across CPUs.
2020

21-
Result so far:
21+
Final result:
2222

2323
- Disabling HTTP/3 did **not** improve the bottleneck.
2424
- Enabling RPS/RFS immediately reduced CPU2 softirq from `100%` to `48.51%`
2525
and spread packet work across all 8 CPUs.
26+
- At T+next-day, distribution remained stable (`CPU2 %soft=49.49%`), and both
27+
public endpoints were still `Working`.
28+
- However, global host load remained high (`11.83 / 11.59 / 10.82`), so this
29+
fix improved balance but did not create enough long-term capacity headroom.
2630

2731
## What Problem We Detected
2832

@@ -217,15 +221,23 @@ https://http1.torrust-tracker-demo.com:443/announce -> Working
217221
udp://udp1.torrust-tracker-demo.com:6969/announce -> Working
218222
```
219223

220-
### Phase 3 interim conclusion
224+
### Phase 3 conclusion after T+1h and T+next-day checkpoints
221225

222-
The specific bottleneck we targeted improved immediately:
226+
The specific bottleneck we targeted improved and stayed improved:
223227

224-
- CPU2 stopped being hard-pinned at 100% softirq.
225-
- Softirq work spread across all cores.
228+
- Immediate: CPU2 stopped being hard-pinned at 100% softirq.
229+
- T+1h: CPU2 stayed around `49.48%`, with distributed softirq across all cores.
230+
- T+next-day: CPU2 stayed around `49.49%`, still distributed.
226231

227232
This is strong evidence that receive-side steering imbalance was a major cause.
228233

234+
But there is an important second-order learning:
235+
236+
- Fixing a one-core packet-processing hotspot does not automatically reduce
237+
total system load enough for comfortable operation.
238+
- Even with better CPU distribution, the host can remain near saturation under
239+
current traffic.
240+
229241
## What We Changed in Configuration Files
230242

231243
This section is explicit so readers can reproduce the patch.
@@ -290,13 +302,21 @@ Operational rule we followed:
290302

291303
This keeps production reproducible and reviewable.
292304

293-
## Remaining Work Before Publishing Final Results
305+
## Final Operational Decision and Follow-Up
306+
307+
Decisions taken after completing the full observation window:
294308

295-
At this draft stage, we still need:
309+
1. Keep RPS/RFS enabled permanently on the current host.
310+
2. Close ISSUE-29 as completed for the tuning scope.
311+
3. Track capacity follow-up in ISSUE-30 (server scale-up planning).
312+
313+
Interpretation:
296314

297-
1. Phase 3 T+next-day checkpoint.
298-
2. Final decision: keep RPS/RFS permanently (likely) or adjust values.
299-
3. Final conclusion on long-term headroom under sustained load.
315+
- ISSUE-29 succeeded technically for its target (remove single-core softirq
316+
saturation).
317+
- ISSUE-29 did not eliminate overall capacity risk at current traffic.
318+
- The next lever is capacity (scale-up), not another immediate packet-path
319+
tuning change.
300320

301321
## Suggested Blog Structure (for torrust.com/blog)
302322

@@ -310,11 +330,13 @@ At this draft stage, we still need:
310330
## References
311331

312332
- Issue plan: `docs/issues/ISSUE-29-research-high-cpu-load-after-udp-fix.md`
333+
- Follow-up scaling issue: `docs/issues/ISSUE-30-scale-up-server-for-sustained-load.md`
313334
- Phase 2 evidence: `docs/issues/evidence/ISSUE-29/01-phase2-disable-http3-execution.md`
314335
- Phase 3 evidence: `docs/issues/evidence/ISSUE-29/02-phase3-enable-rps-rfs-execution.md`
315336
- Htop snapshots:
316337
- `docs/issues/evidence/ISSUE-29/2026-05-04-htop-snapshot.png`
317338
- `docs/issues/evidence/ISSUE-29/2026-05-05-htop-snapshot.png`
339+
- `docs/issues/evidence/ISSUE-29/2026-05-06-htop-snapshot.png`
318340
- Background runbook: `docs/udp-conntrack-runbook.md`
319341
- Previous related blog post:
320342
- https://torrust.com/blog/nf-conntrack-overflow-docker-udp-tracker

0 commit comments

Comments
 (0)