Skip to content

Commit 5341ff9

Browse files
josecelanojosecelano
committed
docs(conntrack): add UDP conntrack runbook and check skill
- add a permanent runbook for diagnosing and fixing UDP conntrack saturation - add a reusable workspace skill for checking conntrack-related UDP loss - link infrastructure and issue docs to the new canonical guidance - add shared spell-check terms for the new runbook and skill Refs: #21
1 parent 7ca3751 commit 5341ff9

6 files changed

Lines changed: 281 additions & 2 deletions

File tree

‎.github/skills/check-udp-conntrack/skill.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
name: check-udp-conntrack
3+
description: Workflow for checking whether UDP packet loss or uptime degradation may be caused by conntrack saturation on the torrust-tracker-demo server. Use when diagnosing UDP timeouts, low newTrackon uptime, packet drops, conntrack pressure, UDP receive-buffer errors, or when validating whether conntrack tuning is still healthy.
4+
metadata:
5+
author: torrust
6+
version: "1.0"
7+
---
8+
9+
<!-- cspell:ignore Rcvbuf conntrack NoPorts -->
10+
11+
# Check UDP Conntrack
12+
13+
## Overview
14+
15+
Use this skill to investigate whether UDP instability is caused by kernel-side
16+
conntrack saturation or related packet-path pressure.
17+
18+
The canonical human-facing reference is:
19+
20+
- `docs/udp-conntrack-runbook.md`
21+
22+
Keep durable explanations and operational guidance in that document. This skill
23+
should stay focused on workflow and safe execution.
24+
25+
## When To Use
26+
27+
Use this skill when the user asks to:
28+
29+
- check whether conntrack is too small
30+
- diagnose UDP timeouts or packet loss
31+
- validate that current conntrack tuning is still active
32+
- verify whether the server is dropping UDP packets
33+
- assess whether current symptoms point to conntrack saturation or something else
34+
35+
## Workflow
36+
37+
1. Run the host checks from `docs/udp-conntrack-runbook.md`.
38+
2. Summarize the results in terms of:
39+
- conntrack occupancy
40+
- presence or absence of `table full` events
41+
- IPv4 and IPv6 UDP receive-buffer errors
42+
- whether `NoPorts` counters are relevant or benign
43+
3. Distinguish conntrack saturation from softirq/RX steering imbalance.
44+
4. If the user asks to document the result, update the relevant issue evidence
45+
or incident file and reference the runbook when appropriate.
46+
47+
## Interpretation Rules
48+
49+
- `nf_conntrack_count` near or equal to `nf_conntrack_max` means real pressure.
50+
- Any fresh `nf_conntrack: table full, dropping packet` message is a confirmed problem.
51+
- `UdpRcvbufErrors` or `Udp6RcvbufErrors` increasing during the incident means packet loss below the application layer.
52+
- `NoPorts` counters alone do not prove tracker loss.
53+
- High load average with one CPU dominated by `%soft` points to softirq concentration, not necessarily conntrack exhaustion.
54+
55+
## Safety Constraints
56+
57+
- Do not change sysctl values unless the user explicitly asks for a fix.
58+
- If applying a fix, update both runtime state and persistent files when appropriate.
59+
- Preserve issue-specific evidence in `docs/issues/evidence/ISSUE-<N>/`.
60+
- Do not present the skill as the primary source of truth; the runbook in `docs/` is the canonical explanation.

‎docs/infrastructure-resize-history.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,5 +39,7 @@ investigations (especially for UDP uptime on newTrackon).
3939

4040
1. Track UDP uptime daily for at least 7 days.
4141
2. Re-check host load and UDP receive buffer errors.
42+
For conntrack-specific diagnosis and remediation, use
43+
[udp-conntrack-runbook.md](udp-conntrack-runbook.md).
4244
3. Compare tracker error/aborted counters before vs after resize.
4345
4. Record final conclusion in this file and in the related issue.

‎docs/infrastructure.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ For raw command outputs (`ip addr`, `df -h`, etc.) see
66
[infrastructure-raw-outputs.md](infrastructure-raw-outputs.md).
77
For server resize and observed request-rate history see
88
[infrastructure-resize-history.md](infrastructure-resize-history.md).
9+
For UDP packet-loss diagnosis and conntrack tuning guidance see
10+
[udp-conntrack-runbook.md](udp-conntrack-runbook.md).
911

1012
## Server
1113

‎docs/issues/ISSUE-21-scale-up-server-for-udp-uptime.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,9 @@ is the main bottleneck and to restore/maintain UDP uptime at or above 99%.
3232

3333
Outcome: **Success**. See
3434
[03-pre-post-comparison.md](evidence/ISSUE-21/03-pre-post-comparison.md) for
35-
the final decision record.
35+
the final decision record. Permanent follow-up documentation now lives in
36+
[udp-conntrack-runbook.md](../../udp-conntrack-runbook.md), with a reusable
37+
workspace skill at `.github/skills/check-udp-conntrack/skill.md`.
3638

3739
## Goal
3840

@@ -118,7 +120,7 @@ The next available option selected for this experiment is:
118120
- [x] At least 7 days of post-resize observations recorded.
119121
- [x] UDP newTrackon uptime reaches and stays >= 99.0% during evaluation window.
120122
- [x] Pre/post comparison documented with clear conclusion.
121-
- [ ] Resize workflow skill added and referenced.
123+
- [x] Resize workflow skill added and referenced.
122124

123125
## Possible Outcomes
124126

‎docs/udp-conntrack-runbook.md‎

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
<!-- cspell:ignore Rcvbuf conntrack softirq recvmmsg NoPorts ksoftirqd nproc vmstat mpstat -->
2+
3+
# UDP Conntrack Runbook
4+
5+
Operational guide for detecting, fixing, and explaining UDP packet loss caused
6+
by conntrack saturation or related kernel-side packet-path pressure.
7+
8+
This runbook exists for reuse beyond issue-specific evidence. For the incident
9+
that led to the current tuning, see
10+
[ISSUE-21](issues/ISSUE-21-scale-up-server-for-udp-uptime.md) and the evidence
11+
under `docs/issues/evidence/ISSUE-21/`.
12+
13+
## When To Use This Runbook
14+
15+
Use this runbook when one or more of these symptoms appear:
16+
17+
- newTrackon or other external probes show intermittent UDP timeouts
18+
- UDP uptime drops while HTTP stays healthy
19+
- UDP request volume is high and Docker DNAT is in the packet path
20+
- `nf_conntrack` may be full or close to full
21+
- Host load looks odd relative to per-CPU usage and packet drops are suspected
22+
23+
## How To Detect The Problem
24+
25+
### External Symptoms
26+
27+
Common user-visible symptoms:
28+
29+
- External UDP probes alternate between working and timing out
30+
- Failures self-recover without a deploy or restart
31+
- HTTP tracker remains mostly healthy while UDP uptime degrades
32+
- Rolling uptime remains low for hours even after recent successful probes
33+
34+
### Host Checks
35+
36+
Run this on the live host:
37+
38+
```bash
39+
ssh demotracker '
40+
echo "=== conntrack counts ===" &&
41+
sudo sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count &&
42+
echo "=== UDP timeouts ===" &&
43+
sudo sysctl net.netfilter.nf_conntrack_udp_timeout \
44+
net.netfilter.nf_conntrack_udp_timeout_stream &&
45+
echo "=== dmesg table full ===" &&
46+
sudo dmesg -T | grep -i "nf_conntrack: table full" | tail -10 &&
47+
echo "(no output = no table-full events)" &&
48+
echo "=== UDP receive errors ===" &&
49+
cat /proc/net/snmp | grep -E "^Udp:" |
50+
awk "NR==1{for(i=1;i<=NF;i++) h[i]=\$i} NR==2{for(i=1;i<=NF;i++) print h[i]\": \"\$i}" |
51+
grep -E "RcvbufErrors|InErrors|NoPorts" &&
52+
echo "=== UDP6 receive errors ===" &&
53+
cat /proc/net/snmp6 | grep -E "Udp6RcvbufErrors|Udp6InErrors|Udp6NoPorts"
54+
'
55+
```
56+
57+
Interpret the output like this:
58+
59+
- `nf_conntrack_count == nf_conntrack_max`: immediate problem; table is full
60+
- `dmesg` contains `nf_conntrack: table full, dropping packet`: confirmed drops
61+
- `UdpRcvbufErrors > 0` or `Udp6RcvbufErrors > 0`: receive-buffer drops exist
62+
- `UdpNoPorts` or `Udp6NoPorts`: usually benign; probes to closed ports, not the tracker itself
63+
64+
### Optional Load Distribution Check
65+
66+
Use this when load average looks high but per-process CPU usage does not explain
67+
it clearly:
68+
69+
```bash
70+
ssh demotracker '
71+
uptime &&
72+
nproc &&
73+
mpstat -P ALL 1 1 2>/dev/null || echo "mpstat not available" &&
74+
ps -eo pid,comm,%cpu,%mem,stat --sort=-%cpu | head -15 &&
75+
vmstat 1 3
76+
'
77+
```
78+
79+
Interpretation:
80+
81+
- high `%soft` on one CPU means kernel packet handling is concentrated there
82+
- this points to softirq/RX steering imbalance, not necessarily tracker code problems
83+
- this is a separate bottleneck from conntrack table saturation
84+
85+
## How To Fix It
86+
87+
### Immediate Live Fix
88+
89+
Apply the kernel tuning live:
90+
91+
```bash
92+
ssh demotracker '
93+
sudo sysctl -w net.netfilter.nf_conntrack_max=1048576 &&
94+
sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout=10 &&
95+
sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout_stream=15
96+
'
97+
```
98+
99+
### Persist The Fix In This Repository
100+
101+
The persistent configuration lives in:
102+
103+
- `server/etc/sysctl.d/99-conntrack.conf`
104+
- `server/etc/modules-load.d/conntrack.conf`
105+
106+
Why both files matter:
107+
108+
- `99-conntrack.conf` stores the kernel parameter values
109+
- `conntrack.conf` preloads the `nf_conntrack` module at boot
110+
- without preloading, the `net.netfilter.*` keys may not exist yet when systemd applies sysctl files, so the values can be skipped after reboot
111+
112+
Current tuned values used by this repository:
113+
114+
| Key | Value |
115+
| ----------------------------------------------- | --------- |
116+
| `net.netfilter.nf_conntrack_max` | `1048576` |
117+
| `net.netfilter.nf_conntrack_udp_timeout` | `10` |
118+
| `net.netfilter.nf_conntrack_udp_timeout_stream` | `15` |
119+
120+
### Validate After The Change
121+
122+
Re-run the detection command above and confirm all of these:
123+
124+
- `nf_conntrack_count` is well below `nf_conntrack_max`
125+
- no fresh `table full` messages appear in `dmesg`
126+
- `UdpRcvbufErrors` and `Udp6RcvbufErrors` are stable or zero
127+
- external UDP probes recover and remain healthy for multiple hours or days
128+
129+
## Why This Works
130+
131+
### Packet Path
132+
133+
For the deployed tracker, the UDP receive path is approximately:
134+
135+
```text
136+
NIC -> kernel RX interrupt -> softirq/ksoftirqd -> conntrack + Docker DNAT -> socket buffer -> tracker recv loop -> spawned async task
137+
```
138+
139+
The important point is that conntrack lookup and DNAT happen in the kernel
140+
before the tracker reads the packet from the socket.
141+
142+
### Failure Mechanism
143+
144+
With Docker in the packet path, each UDP packet can create or refresh a
145+
conntrack entry.
146+
147+
If all of these are true at the same time:
148+
149+
- request rate is high
150+
- `nf_conntrack_max` is too small
151+
- UDP entry timeouts are too long
152+
153+
then the steady-state number of tracked UDP flows grows until the table is full.
154+
Once full, the kernel drops new packets before the tracker can read them.
155+
156+
### Why Increasing `nf_conntrack_max` Helps
157+
158+
Increasing `nf_conntrack_max` raises the ceiling for concurrent tracked flows,
159+
reducing the chance that bursts or sustained load fill the table.
160+
161+
### Why Reducing UDP Timeouts Helps
162+
163+
Reducing `nf_conntrack_udp_timeout` and
164+
`nf_conntrack_udp_timeout_stream` shortens how long old UDP entries stay in the
165+
table.
166+
167+
That reduces steady-state occupancy, which is often more important than raw CPU
168+
capacity for this failure mode.
169+
170+
### Why The Tracker Code Is Not The Root Cause
171+
172+
The tracker's UDP loop reads packets after the kernel has already:
173+
174+
- handled the RX interrupt/softirq work
175+
- performed conntrack lookup
176+
- applied Docker NAT rules
177+
- copied the packet into the socket receive buffer
178+
179+
If packets are being dropped because the conntrack table is full, the tracker
180+
never sees them.
181+
182+
## Separate Future Tuning: RPS/RFS
183+
184+
RPS and RFS are not part of the current fix, but they may matter if one CPU is
185+
dominated by softirq work while other CPUs remain idle.
186+
187+
- RPS spreads receive-side softirq work across CPUs
188+
- RFS tries to steer packets toward the CPU currently running the application thread that reads the socket
189+
190+
Use them only if softirq concentration becomes the next bottleneck. They solve a
191+
different problem from conntrack table saturation.
192+
193+
## Reference Values From The 2026-04-27 Verification
194+
195+
Recorded before merging PR #22:
196+
197+
- peak UDP tracker traffic observed over the prior 7 days: about `750 req/s`
198+
- peak HTTP tracker traffic observed over the prior 7 days: about `2000 req/s`
199+
- `nf_conntrack_count`: `341652`
200+
- `nf_conntrack_max`: `1048576`
201+
- utilization: `32.59%`
202+
- `UdpRcvbufErrors`: `0`
203+
- `Udp6RcvbufErrors`: `56` cumulative since boot, not material at observed load
204+
205+
## Related Files
206+
207+
- [docs/infrastructure.md](infrastructure.md)
208+
- [docs/infrastructure-resize-history.md](infrastructure-resize-history.md)
209+
- [server/etc/sysctl.d/99-conntrack.conf](../server/etc/sysctl.d/99-conntrack.conf)
210+
- [server/etc/modules-load.d/conntrack.conf](../server/etc/modules-load.d/conntrack.conf)
211+
- [docs/issues/ISSUE-21-scale-up-server-for-udp-uptime.md](issues/ISSUE-21-scale-up-server-for-udp-uptime.md)

‎project-words.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,10 @@ qdisc
6969
qlen
7070
repomix
7171
rgba
72+
runbook
7273
rustfmt
7374
shellcheck
75+
snmp
7476
sourceable
7577
signup
7678
tcpdump

0 commit comments

Comments
 (0)