jonathaneoliver
diff --git a/‎README.md‎
Lines changed: 21 additions & 3 deletions b/‎README.md‎
Lines changed: 21 additions & 3 deletions
diff --git a/‎analytics/README.md‎
Lines changed: 212 additions & 0 deletions b/‎analytics/README.md‎
Lines changed: 212 additions & 0 deletions
@@ -257,11 +257,11 @@ ATS-style transfer timeouts the proxy enforces against the *client* — useful f
 
 When a timeout fires the network-log waterfall renders the row with `!⏱` so you can tell it apart from a fault-injection cut (`!✂`) or a player abort (`!↩`).
 
-### Bitrate chart (with buffer depth and FPS)
+### Bitrate chart (with RTT, buffer depth, and FPS)
 
 ![Playback state chart](docs/screenshots/playback-state-chart.png)
 
-The session card has a collapsible **Bitrate Chart** that stacks an events timeline + up to three time-series charts, all sharing a 10-minute rolling window and unified zoom/pan. Legend entries toggle series, scroll zooms, drag pans, `⏸` pauses live updates. The four panels share an x-axis so a bandwidth dip lines up visually with its buffer / FPS / variant-shift impact.
+The session card has a collapsible **Bitrate Chart** that stacks an events timeline + up to four time-series charts, all sharing a 10-minute rolling window and unified zoom/pan. Legend entries toggle series, scroll zooms, drag pans, `⏸` pauses live updates. The five panels share an x-axis so a bandwidth dip lines up visually with its RTT / buffer / FPS / variant-shift impact.
 
 - **Events timeline** *(top)* — swim-lane visualization of what the player and server are doing right now:
   - **PLAYER variants** — one lane per ladder rung (e.g. `1920×1080:7.1Mbps`, `1280×720:3.5Mbps`). Coloured blocks show which variant the player was on at each moment. The variant shift on a throughput collapse is visible as a downstep across lanes.
@@ -275,10 +275,14 @@ The session card has a collapsible **Bitrate Chart** that stacks an events timel
   - **Reference lines**: `Limit` (shaping ceiling, stepped when a pattern is active), `Server Rendition` (what the server believes it delivered), one line per ladder `Variant` (hidden by default).
   - **Events**: `STALL` and `RESTART` markers annotate player stalls and restarts.
   - **Y-axis**: `Auto` or fixed `5 / 10 / 20 / 30 / 40 / 50 / 100` Mbps — pin the scale when comparing two sessions side by side.
+- **RTT chart** — two independent round-trip-time signals on the same time axis (issues #401 / #404):
+  - **TCP_INFO RTT family** (purple lines, left Y-axis) — what the streaming TCP connection actually experiences. Sampled inside go-proxy via `getsockopt(TCP_INFO)` at 100 Hz, drained into 1 s windows: `RTT avg` (smoothed RFC 6298 SRTT), `RTT max` / `min` (per-window peak/trough envelope), `RTT lifetime min` (sticky path floor, dashed reference), `RTO` (right Y-axis, hidden by default — climbs above RTT during a wedge while smoothed RTT flatlines).
+  - **Path ping** (cyan line, left Y-axis) — out-of-band ICMP echo from go-proxy → `player_ip` at 1 Hz, routed through a high-priority band inside the per-port shaping class. Sees the configured netem delay but jumps the bulk segment queue. The closest thing to "what the path could deliver if you weren't loading it." Zero / gap when ICMP is filtered.
+  - **Why both?** The ping line is the network's contribution; the gap up to the TCP_INFO line is the application stack's contribution (queueing under throttle, delayed ACKs, receiver load). Together they decompose latency under shaping: rising netem moves both lines together; rising throttle inflates only TCP_INFO via bufferbloat. See [`analytics/README.md`](analytics/README.md#reading-the-rtt-chart-issues-401-404) for the deep interpretation guide and per-shaping-knob test recipes.
 - **Buffer depth chart** — player `buffered` TimeRanges (`player_metrics_buffer_depth_s`) on the left axis; **Wall-Clock Offset** (player playhead vs encoder PDT) on the right axis.
 - **FPS chart** — rendered and dropped frames/s from `player_metrics_frames_displayed` / `_dropped_frames`, 2 s sliding window, exponential smoothing (α = 0.15). Series: `FPS (smoothed)`, `Low FPS` (red below threshold), `FPS Baseline` (75th percentile), `Low Threshold` (`0.75 × baseline`), `Dropped Frames/s` (right Y-axis).
 
-The four panels' plot areas all align on the same right edge so vertical x-axis ticks line up across every chart and the events timeline above.
+The five panels' plot areas all align on the same right edge so vertical x-axis ticks line up across every chart and the events timeline above.
 
 ### Network log waterfall (HAR view)
 
@@ -545,6 +549,20 @@ Full API (`/api/content`, `/api/jobs`, `/api/sessions/*`, `/api/nftables/*`, etc
 | `mbps_transfer_rate` | 250 ms | Byte-change-gated rate during segment transfer, aligned to HTB burst edges. Reports at drain/refill boundaries |
 | `mbps_transfer_complete` | per segment | Total bytes / total time for one completed segment transfer (backlog drained to 0) |
 
+### Server-side RTT metrics
+
+Sampled inside go-proxy via `getsockopt(TCP_INFO)` on each session's most-recent connection. The 100 ms sampler folds reads into a 1 s window that drains on every snapshot tick — same cadence as the player-metrics PATCH heartbeat, so the RTT chart shares a time axis with the bitrate chart above it. Linux-only (the kernel option doesn't exist on macOS); the dev build compiles via a stub that emits zeros. All values in milliseconds.
+
+| Metric | Source field | What it measures |
+|---|---|---|
+| `client_rtt_ms` | `tcpi_rtt` (avg of 1 s window) | Smoothed RTT (RFC 6298 SRTT, kernel EWMA) |
+| `client_rtt_max_ms` | window max of `tcpi_rtt` | Peak smoothed RTT in window — catches sub-second spikes the kernel's EWMA would mask |
+| `client_rtt_min_ms` | window min of `tcpi_rtt` | Trough during the same 1 s window |
+| `client_rtt_min_lifetime_ms` | `tcpi_min_rtt` | Min RTT ever observed on this connection — the path floor |
+| `client_rtt_var_ms` | `tcpi_rttvar` | Smoothed mean deviation (jitter) |
+| `client_rto_ms` | `tcpi_rto` | Current retransmit timeout — rises during a wedge while smoothed RTT flatlines; the gap between `rto` and `rtt` is the canonical "kernel suspects this connection is stalling" signal |
+| `client_path_ping_rtt_ms` | ICMP echo, 1 Hz | **Out-of-band path latency** (issue #404). Independent of the streaming connection's queue contribution — TCP_INFO RTT inflates with throttle, but ICMP packets bypass the application's send queue, so this line stays at the LAN baseline when shaping kicks in. The vertical gap between this line and `client_rtt_ms` is the queueing delay the application is inducing on itself. Zero / gap when ICMP is filtered. |
+
 ### Metric semantics
 
 - **Limit value** (`nftables` shaping rate): configured ceiling for the session port; a control target, not a measured throughput.
 
@@ -77,6 +77,218 @@ docker compose up -d --force-recreate grafana
 - `TTL toDateTime(ts) + INTERVAL 30 DAY` enforces retention; tune via
   `ALTER TABLE ... MODIFY TTL`.
 
+### Reading the RTT chart (issues #401, #404)
+
+Two independent measurement schemes plotted on the same chart:
+
+- **TCP_INFO family** (`client_rtt_*_ms`, purple lines) — what the
+  *streaming TCP connection* actually experiences. Self-loaded:
+  every sample includes the application's own queue contribution.
+- **Path ping** (`client_path_ping_rtt_ms`, cyan line) — out-of-band
+  ICMP echo from go-proxy → player_ip at 1 Hz, routed through a
+  high-priority band (TC_PRIO_INTERACTIVE → tc band 0) inside the
+  per-port shaping class. Sees the configured netem delay but jumps
+  the bulk queue. The closest thing to "what the path could deliver
+  if you weren't loading it."
+
+#### The two probe paths through the kernel
+
+Both probes share the same physical egress (eth0) and the same per-
+port HTB class, but they take different *bands* through the prio
+scheduler that sits inside the class as the leaf qdisc:
+
+```
+egress eth0
+└── HTB root (1:1)
+    └── HTB class 1:<port>     (rate-limited at configured Mbps)
+        └── prio leaf qdisc     (3 bands, default priomap)
+            ├── band 0 (1810:1)  ← PROBE LANE
+            │   └── netem (configured delay/loss + 5% jitter)
+            ├── band 1 (1810:2)  ← BULK DATA LANE
+            │   └── netem (same delay/loss/jitter)
+            └── band 2 (1810:3)  unused
+
+filters at parent 1:0:
+  ip sport <port>           → flowid 1:<port>     (TCP segments → bulk lane via priomap default)
+  ip protocol icmp dst <ip> → flowid 1:<port>     (ICMP probes → probe lane via IP_TOS=0x10 → priomap band 0)
+```
+
+How a packet picks its band:
+
+- **Bulk segment data**: leaves the proxy with `skb->priority = 0`
+  (`TC_PRIO_BESTEFFORT`). Default prio priomap routes that to band 1
+  (the middle band) → classid `1810:2` → bulk netem.
+- **Path-ping ICMP**: socket has `IP_TOS = 0x10` set. Kernel's
+  `rt_tos2priority` table maps that to `TC_PRIO_INTERACTIVE` (priority
+  6). Default priomap routes that to band 0 → classid `1810:1` →
+  probe netem.
+
+Both bands carry **identical netem configuration** — `UpdateNetem`
+writes the same `delay/loss/jitter` to all three bands in lockstep,
+so the configured network conditions apply uniformly regardless of
+priority. The only thing the prio scheduler changes is *queueing
+order* when both bands have eligible packets: strict priority means
+band 0 always drains first.
+
+Net effect:
+
+- **Bulk** sits in band 1's netem queue, gets the configured delay,
+  then waits for HTB's rate-limit token. If the rate is full it
+  queues behind earlier bulk packets too — bufferbloat.
+- **Probe** sits in band 0's netem queue, gets the same configured
+  delay, then jumps every queued bulk packet on the next HTB dequeue.
+  Worst case it waits one MTU's serialization (≈ 12 ms at 1 Mbps,
+  2.4 ms at 5 Mbps) for a bulk packet already on the wire.
+
+So the ping line shows you *path + netem*, almost free of bufferbloat.
+The TCP_INFO lines show you *path + netem + bufferbloat + ACK overhead*
+— what bulk segment data really pays.
+
+#### Why we ship both signals
+
+They answer different questions. Either alone is misleading:
+
+- **Just TCP_INFO** would conflate "the path is bad" with "I'm
+  loading the path" — a player ABR algorithm seeing high RTT can't
+  tell whether to back off or just wait for its own queue to drain.
+  Lifetime min approximates the path floor but is sticky and slow
+  to update during sustained load.
+- **Just ping** would tell you nothing about what the streaming
+  connection actually experiences — the ABR doesn't react to ICMP,
+  it reacts to TCP behavior. A perfectly healthy ping line above a
+  stalling player would be a useless diagnostic.
+
+Together they decompose the latency budget: ping is the *physical
+network's contribution* (path + configured delay), and the gap up
+to TCP_INFO is the *application stack's contribution* (queueing
+under throttle, delayed ACKs, receiver load, reverse-path queuing).
+When bitrate drops or buffer drains, the chart tells you *which
+component* moved — was it the path getting longer, or my own
+queue piling up?
+
+#### Field reference
+
+- `client_rtt_ms` — kernel's smoothed RTT (RFC 6298 SRTT). Current
+  path latency with EWMA, lags real changes by a fraction of a second.
+- `client_rtt_max_ms` / `client_rtt_min_ms` — peak/trough of smoothed
+  RTT *within the 1 s emit window*. Catches sub-second spikes the
+  kernel's EWMA would otherwise mask.
+- `client_rtt_min_lifetime_ms` — connection's *path floor*: the best
+  RTT ever seen on that TCP connection. Sticky-low, never climbs back.
+- `client_rtt_var_ms` — kernel's smoothed mean deviation (jitter).
+- `client_rto_ms` — current retransmit timeout. Rises during a wedge
+  while smoothed RTT flatlines because no fresh ACKs are coming back.
+- `client_path_ping_rtt_ms` — ICMP echo round-trip, 1 Hz cadence.
+  Zero / absent when ICMP is filtered on the path.
+
+#### Why the ping line is *always* lower than TCP_INFO RTT
+
+Even on a perfectly idle, unshaped LAN, expect TCP_INFO RTT to sit
+above the ping line. Sources of inflation, all real, all working as
+designed:
+
+- **Delayed ACKs** — receiver TCP holds ACKs up to 40–200 ms (Linux/iOS
+  default ~40 ms) to coalesce them. ICMP echo replies aren't subject
+  to this; they go out immediately.
+- **TCP_INFO is the smoothed SRTT** — exponentially averaged across
+  recent ACKs, including ACKs generated under load. ICMP samples a
+  single round-trip on the kernel fast path.
+- **Receiver processing latency under load** — at multi-Mbps the
+  player's TCP stack is busy demuxing segment data; ACK generation
+  slips behind. ICMP echo handling skips userspace entirely.
+- **Reverse-path queuing** — the player's egress (ACKs going *back*)
+  has its own tiny outbound queue. ICMP replies skip it.
+
+So `TCP_INFO − ping` on healthy unshaped LAN ≈ delayed-ACK + receiver
+load + reverse queueing. This is the network stack's overhead, not
+a fault.
+
+#### Expected behavior under shaping
+
+The two signals respond differently to the two shaping knobs (netem
+delay and HTB rate limit). Useful test recipes:
+
+| Action | TCP_INFO RTT | Path ping | Why |
+|---|---|---|---|
+| **No shaping** | `path + ACK overhead + receiver load` (typically 5–50 ms LAN) | `~path RTT` (sub-ms LAN) | Baseline. The ping line is the floor; TCP_INFO is everything else the stack adds. |
+| **Set netem delay = 25 ms** | rises by ~25 ms (mean) | rises by ~25 ms (mean) | Both packets traverse the same per-band netem inside the HTB class. Matched movement. Per-packet variance is ±5 % of mean (~1 ms stddev at 25 ms — see jitter note below). |
+| **Set throttle = 1 Mbps** (no netem) | climbs into bufferbloat range (often 100s of ms during downloads) | unchanged from baseline | Bulk segment data fills the HTB queue; each MTU waits for a rate token. The probe escapes via prio band 0 — at most one MTU's serialization (~12 ms at 1 Mbps for 1500 B). |
+| **Throttle + netem combined** | `path + netem + bufferbloat + ACK overhead` (compounded) | `~path + netem + at-most-one-MTU` | Effects stack additively on bulk data. The ping line shows you what's *just* the configured delay so you can subtract bufferbloat by eye. |
+| **Toggle shaping mid-stream** | step changes correlate visibly with bitrate / buffer drops on the chart above | flat through bandwidth changes; steps on netem changes only | Whole point of having both signals. Bitrate dropped because shaping was applied → both lines confirm in different ways. |
+| **Drop packets via fault-injection** | smoothed RTT eventually flatlines (no fresh ACKs); `client_rto_ms` climbs as kernel doubles its timeout | `0` / gap (echo replies dropped too) | RTO − RTT divergence is the canonical wedge indicator. |
+
+Reading the chart in one sentence:
+**ping = network's contribution; (TCP_INFO − ping) = stack + bufferbloat
+contribution.**
+
+#### A note on jitter
+
+`UpdateNetem` adds a fixed 5 % normal-distributed jitter (`stddev =
+delay/20`). For configured 25 ms delay: stddev ≈ 1 ms, so ~99.7 % of
+per-packet delays land in [22, 28] ms. The ping per-window min sits
+at the configured floor; per-window max shows the tight scatter above.
+Configured delays ≤19 ms get zero jitter (integer divide rounds to 0)
+— fine for low-RTT testing where any noise would dominate the signal.
+
+If a future test needs higher jitter (ABR resilience to RTT scatter,
+out-of-order arrivals via wider Gaussian draws), add a separate
+`jitter_ms` parameter to the shaping API rather than re-deriving from
+the delay value. The 5 % default is tuned for "I configured 25 ms,
+the chart should read ~25 ms," not for stress-testing variance.
+
+#### Reading the RTO line (wedge detection)
+
+`client_rto_ms` is hidden by default — toggle it on from the chart
+legend when you suspect a wedge. RTO answers a different question
+than the rest of the chart: not *how long does a round-trip take*
+but *how long is the kernel willing to wait for one before giving
+up and retransmitting*.
+
+What it is. RTO is the kernel's **retransmission timeout** for the
+TCP connection. After sending a segment, the sender starts a timer;
+if no ACK arrives before the timer expires, the segment is
+retransmitted and the timer doubles. RFC 6298 sets the steady-state
+value to roughly `SRTT + 4 × RTTVAR` (smoothed RTT plus four times
+its smoothed deviation), with a kernel floor of 200 ms and ceiling
+of 120 s. So on a quiet, healthy connection RTO sits a small
+multiple of RTT above the smoothed-RTT line — *not* zero.
+
+Why it's the wedge canary. When ACKs stop flowing entirely (transport
+fault, dropped packets, broken middlebox), `tcpi_rtt` flatlines —
+no fresh ACK round-trips means no new samples to update the EWMA.
+Looking at the smoothed-RTT line alone, you can't tell whether the
+connection is healthy-and-idle or wedged-and-silent. RTO has its
+own state machine driven by the kernel's retransmission timer, not
+by ACK arrivals: every retry doubles it (`200ms → 400ms → 800ms →
+1.6s → 3.2s → 6.4s → 12.8s → 25.6s → 51.2s → 102.4s`, capped at
+the kernel max). So when the connection wedges:
+
+```
+RTT (purple) ──flat───────────────────────────────  ← no ACKs, no fresh samples
+RTO  (red) ───────⌐──┘─⌐──┘──⌐──┘──⌐────┘────  ← kernel doubling on each retry
+```
+
+The growing gap between the two is the unambiguous "kernel suspects
+this connection is stalling" signal. It appears within seconds of
+the wedge starting — much faster than a stall on the bitrate chart
+above (which only triggers after the player's buffer drains).
+
+What recovery looks like. Once ACKs start flowing again (fault
+cleared, retry succeeds), the kernel resets RTO to `SRTT + 4×RTTVAR`
+on the very next ACK. The red line snaps back down; the purple
+smoothed-RTT line resumes updating. So a recovered wedge is visible
+as a sawtooth-like RTO climb followed by an instant drop, with the
+RTT line resuming its normal track.
+
+Useful pairing. RTO + the path-ping line together disambiguate the
+wedge cause:
+
+| RTT line | RTO line | Path ping | What it means |
+|---|---|---|---|
+| flat | climbing | also gone (ICMP filtered/dropped) | Wedge or transport fault — entire path is dead from proxy's view. |
+| flat | climbing | still arriving normally | Wedge is TCP-specific — ICMP gets through but TCP is stuck. Likely middlebox dropping the connection or a broken player TCP stack, not a network outage. |
+| climbing slowly | tracking RTT (small multiple above) | climbing the same | Genuine path latency increase, not a wedge — RTO is just following healthy RTT growth. |
+
 ## Securing a WAN-exposed deployment
 
 Default docker-compose binds ClickHouse to `127.0.0.1` (host-only) and