Skip to content

Adaptive SQM: calc_burst 5 KB clamp caps throughput at ~65-70% of the configured rate on UCG Fiber (measured; rate-proportional burst restores ~830 of 900 Mbit) #1006

Description

@Optic00

Summary

Adaptive SQM already scales HTB burst/cburst and fq_codel memory_limit with the shaped rate (the "queue memory and burst sizes are scaled to your WAN speed" behavior), which is exactly the right idea.
However, calc_burst() clamps at 5000 bytes (~3 full-size packets).
On a UCG Fiber shaping a gigabit PPPoE WAN, that clamp is still the dominant throughput limiter: I measured ~630-660 Mbit/s at Adaptive SQM's exact operating point (burst 5000b, memory_limit 8Mb) versus ~830 Mbit/s with a rate-proportional burst of ~1 ms, at a configured rate of 900 Mbit/s and with no measurable latency regression.
I realize the 5 KB value is itself empirically grounded (the code comment says 8KB+ increased queue depth variance on the test platform), so the sweet spot appears to be platform-dependent.
Proposal: make the burst rate-proportional (~1 ms of line rate) instead of clamping at a fixed 5 KB, or at least raise the clamp on platforms where the ingress path runs on IFB without GRO.
I am preparing a PR with measurements and would adjust it to whatever direction you prefer.

Background - why I dug into this

My ISP currently has a destination-selective packet-loss problem under download load near line rate (same connection as in #959).
Ingress shaping turned out to be the only effective local mitigation: with Smart Queues active, loss on every affected destination drops to a measured 0%, presumably because the smoothed TCP aggregate stays below whatever triggers the faulty upstream element.
That makes SQM permanent infrastructure for me rather than an optional comfort feature - and permanently giving up ~40% of a gigabit line to the stock 1-packet burst is what made me chase where the throughput actually goes.
The fix itself is independent of that backstory: every Adaptive SQM user on a fast WAN gets the same bandwidth back.

Environment

  • UCG Fiber (4 cores), UniFi OS 5.1.19, Network 10.5.62
  • WAN: 1G/0.5G FTTH via PPPoE (VLAN 7); UniFi Smart Queues at 900 down / 512 up creating the usual HTB+fq_codel on ppp0 (egress) and ifbppp0 (ingress via IFB)
  • All measurements: same wired vantage host, same Ookla server, download direction, rate pinned at 900 Mbit via tc class change so only burst/memory vary
  • Note: the ingress IFB shows maxpacket 1492, i.e. no GRO aggregation; the shaper pays full per-packet cost on this path

Measurements (rate fixed at 900 Mbit, 2 runs per cell; the stock row includes a third run as an evening re-anchor)

burst/cburst memory_limit measured download
~1300b (UniFi stock, 1 packet) 4Mb (stock) 539 / 539 / 542 Mbit
5000b (= calc_burst clamp; tc rounds to 4837b) 8Mb (= calc_fq_mem at >= 750) 628 / 661 Mbit
5000b 32Mb 574 / 616 Mbit
32k 4Mb 772 / 758 Mbit
32k 32Mb 754 / 752 Mbit
128k 32Mb 831 / 803 Mbit
256k 32Mb 823 / 802 Mbit

Observations:

  • The two 5000b cells sit well below the 32k/128k cells (means: ~645 vs ~755-820); raising memory_limit from 8Mb to 32Mb at fixed 5000b does not recover any of that gap (574/616 vs 628/661 - if anything lower), so memory is not the binding constraint; the burst is.
  • Burst saturates around 128k: 256k gives no further gain, so ~830 Mbit is the hardware ceiling on this box (per-packet CPU on the GRO-less IFB path), not a burst-tunable limit. Raising the configured rate cap to 1024 with a 128k burst also did not help (816/815), confirming the limiter is burst-driven scheduling, not the rate ceiling.
  • With small bursts, HTB overlimits counts in the millions per speedtest (~0.57 overlimits per packet at 5000b), i.e. near-per-packet throttling; with burst = ~1 ms of line rate this collapses.
    Mechanism: with a burst of 1-3 packets, HTB can only release a few packets per timer/softirq wakeup, so throughput degenerates into a function of scheduling latency instead of the configured rate.
  • Latency cost of the larger burst: none measurable.
    Parallel 5 Hz RTT probes stayed flat during all cells (e.g. avg ~12.6 ms to a Hetzner target, ~10.2 ms to 1.1.1.1, identical across 5000b and 128k cells; a 128k burst at ~1 Gbit/s is a ~1 ms line-rate microburst).
  • Remaining ceiling ~830 Mbit/s on this hardware is plausibly the per-packet CPU path on IFB without GRO (no clean single-core saturation in 1 s samples, max ~86% busy / 60% softirq on one core).

Proposed change

In ScriptGenerator.cs calc_burst():

  • current: burst = rate_mbps * 5, clamped to [1500, 5000] bytes
  • proposed: burst = rate_mbps * 125 bytes (= rate x 1 ms), clamped to [1500, 131072] bytes

At 100 Mbit this yields 12.5k (vs 500 -> floor 1500 today, effectively unchanged behavior at low rates), at 900 Mbit ~112k.
calc_fq_mem() can stay as is; 8Mb showed no throughput penalty and no drop_overmemory in these tests (the earlier drop_overmemory I saw came from the 4Mb stock value under multi-stream load, which your existing scaling already fixes).
tune_tc_performance() and the cron re-apply path pick the new values up automatically, so the change is small and self-contained.

Since your 5 KB result came from real measurements on different hardware, the honest framing is: the optimal burst is platform-dependent, and a fixed 5 KB clamp leaves a lot of bandwidth on the table on at least UCG Fiber.
If you prefer, the clamp ceiling could also be a per-platform or advanced-config value instead of a hard-coded constant; happy to implement either variant.
If you can re-run your queue-depth-variance test with rate-proportional burst on your platform, that would settle whether the higher ceiling is safe everywhere.

The deeper lever (worth stating even if out of scope for this change)

The ~830 Mbit ceiling that remains after fixing the burst is consistent with per-packet CPU cost on the ingress IFB path: observed on the live gateway, the IFB qdisc reports maxpacket 1492 (it only ever sees MTU-sized packets, i.e. no GRO aggregation on this path), and once any shaping is active UniFi installs a match-all mirred redirect to ifbppp0, taking the whole download direction off the hardware fast path.
I did not see clean single-core saturation in 1 s samples (max ~86% busy / 60% softirq on one core), so I frame the CPU attribution as the most plausible explanation rather than proven.
The burst fix only amortizes that cost; enabling GRO on the IFB path would attack the ceiling itself.
(Curiously, the UCG Fiber firmware ships qca-nss-ppe-qdisc.ko implementing ppe_htb - a hardware HTB for the PPE engine - but no qdisc on the box uses it and there is no hardware fq_codel counterpart, so Smart Queues runs entirely on the CPU path.)
Separately, the 1-packet stock burst looks like a UniFi-side defect (a rate-shaped class should not ship a 1-packet burst at gigabit); we are reporting that to Ubiquiti in parallel, since a tool that rewrites a UniFi-managed live tc tree is maintaining a workaround, not consuming a stable interface.

End-to-end validation with the patched formula (deployed build)

I built NetworkOptimizer with calc_burst = rate_mbps * 125 clamped to [1500, 131072] and deployed Adaptive SQM to the UCG Fiber through the normal UI flow.
The deployed scripts applied rate 897Mbit burst 112125b (ingress; the configured 900 Mbit max capped at 95% -> 897, after a 905 Mbit calibration measurement) and 550Mbit burst 68750b (egress) on their own.
Resulting measurements (two vantage points, two servers): 810/514 and 832/521 Mbit down/up, 0% packet loss, idle-level ping - versus ~533/452 with the stock 1-packet burst before.
Upload benefits identically because calc_burst is shared between directions: the egress HTB ships the same 1-packet burst (1408b at a 512 Mbit cap), and the rate-proportional burst alone lifted upload from 464 to 487 at the unchanged 512 cap (~95% of cap, i.e. shaping-efficiency territory); the ~515-521 figures above are after I additionally raised the cap to 550.

Notes / caveats

  • The ablation matrix used runtime tc changes (values reset on every provisioning cycle); the end-to-end section used the real deployed persistence path (boot script + cron re-apply), which survives reprovisioning as designed.
  • Tested on UCG Fiber only. The controlled ablation matrix above is ingress-only; the end-to-end validation additionally covered egress (as reported in that section).
  • Latency across rate tiers was measured, not assumed: at 100/300/600 Mbit under saturating load (old vs new burst at identical memory, 400 raw RTT samples each), p99 was equal at 100 and 600 Mbit and clearly better with the new burst at 300 Mbit (p99 24.4 -> 14.6 ms to one target, 23.6 -> 18.4 to another) - plausibly because with a 1-3 packet burst, waiting for the next HTB timer tick is itself added latency. Below ~40 Mbit the formula stays within the stock 1500-5000b clamp range; at 100 Mbit it yields 12.5k (~1 ms), where no p99 regression was measurable. This is one platform; re-validating on your hardware (where the 5KB/8KB variance findings came from) would settle generality.
  • When UniFi QoS rules add child classes, the same burst/cburst is applied per child (whose ceil is the full rate, so it stays ~1 ms at ceil); inter-class fairness right after idle shifts by at most that ~1 ms window. Untested here - my tree has only the default class.
  • burst vs memory are separate effects: burst is the throughput lever; fq_codel memory_limit (your existing rate-scaled 8MB) only removes drop_overmemory, it does not raise throughput on its own. Note memory_limit is a global fq_codel safety limit, not a TCP window, and the packet limit (2000p) can bind first depending on skb truesize - worth keeping in mind when reasoning about the memory scaling.
  • Happy to share raw logs (tc counters, per-run outputs, RTT probe series) in the PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions