Skip to content

Commit 18eef23

Browse files
haoruizhouhaoruizhou
committed
TCP retransmission documentation
1 parent c52733c commit 18eef23

1 file changed

Lines changed: 97 additions & 2 deletions

File tree

‎universal-telemetry-software/README.md‎

Lines changed: 97 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,101 @@ The deployed car service sets `ROLE=car` explicitly in `deploy/car-telemetry.ser
3737

3838
---
3939

40+
## Reliability & Retransmission
41+
42+
The link between car and base is a **best-effort UDP stream with a pull-based recovery channel over TCP**. The car never waits for acknowledgements — it fires CAN batches over UDP as fast as it reads them and keeps a short backlog in memory. The base station detects the gaps and asks for the missing pieces back. This keeps the live path low-latency while still recovering most dropped packets a few seconds later.
43+
44+
All of this lives in `src/data.py`: the car side in `run_car()` (`udp_sender`, `handle_resend`), the base side in `run_base()` (`udp_receiver`, `missing_reporter`).
45+
46+
### Data flow
47+
48+
```mermaid
49+
sequenceDiagram
50+
participant CAN as CAN bus
51+
participant Car as Car (udp_sender)
52+
participant RB as Ring buffer (60s)
53+
participant Base as Base (udp_receiver)
54+
participant MR as Base (missing_reporter)
55+
56+
CAN->>Car: raw frames
57+
Note over Car: batch 20 msgs OR 50ms<br/>assign seq_num (uint64)
58+
Car->>RB: store (seq, batch, t)
59+
Car--)Base: UDP :5005 [seq | count | msgs]
60+
Note over Base: track expected_seq<br/>seq > expected ⇒ gap<br/>add skipped seqs to missing set
61+
loop every 10s, if missing set non-empty
62+
MR->>Car: TCP :5006 {"missing": [seq, ...]}
63+
Car->>RB: look up seqs still in buffer
64+
Car-->>MR: JSON [{seq, msgs:[{t,id,d}]}]
65+
Note over MR: re-publish to Redis<br/>mark seq recovered
66+
end
67+
```
68+
69+
### Sender: batching, sequencing, ring buffer (car)
70+
71+
- **Batching** — CAN frames are accumulated and flushed when the batch hits `BATCH_SIZE` (20 messages) **or** `BATCH_TIMEOUT` (50 ms) elapses, whichever comes first. Each flush is one UDP datagram.
72+
- **Sequencing** — every batch gets a monotonic `seq_num` (uint64). This is the unit of loss detection and retransmission; individual CAN messages are never tracked, only whole batches.
73+
- **Wire format** — the UDP payload is a 10-byte header followed by fixed-size messages:
74+
75+
| Field | Type | Bytes | Notes |
76+
|-------|------|-------|-------|
77+
| `seq` | `uint64` big-endian (`!Q`) | 8 | batch sequence number |
78+
| `count` | `uint16` big-endian (`!H`) | 2 | number of messages in this batch |
79+
| `messages` | `count × CANMessage` | 20 each | packed via `CANMessage.pack()` |
80+
81+
- **Ring buffer** — after sending, the car also appends `(seq, batch, timestamp)` to an in-memory `deque` and evicts anything older than `BUFFER_DURATION` (**60 seconds**). This is the *only* copy available for retransmission. A gap that isn't requested within ~60s is gone for good on the live link (it may still be recovered later from the car's own CSV log, but not through this channel).
82+
83+
### Receiver: gap detection state machine (base)
84+
85+
The base tracks a single `expected_seq` and classifies every incoming datagram:
86+
87+
| Condition | Meaning | Action |
88+
|-----------|---------|--------|
89+
| `seq == expected_seq` | in order | process; `expected_seq = seq + 1` |
90+
| `seq > expected_seq` | **gap** | add every seq in `[expected_seq, seq)` to the `missing` set; process; advance `expected_seq` |
91+
| `seq < expected_seq`, in `missing` set | late/out-of-order arrival | remove from `missing`, count as recovered |
92+
| `seq < expected_seq`, not in `missing` | duplicate | ignore |
93+
| `seq < expected_seq − 1000` | **sequence reset** (car restarted) | reset `expected_seq`, clear `missing` |
94+
95+
The `missing` set is capped at **1000 entries**; when it overflows the oldest sequence is evicted (and will never be requested). A per-second status map (`0` = missing, `1` = UDP, `2` = TCP-recovered) is maintained purely for the PECAN link-health visualization.
96+
97+
### Recovery: request / response schema (TCP :5006)
98+
99+
A `missing_reporter` task wakes every `MISSING_CHECK_INTERVAL` (**10 seconds**). If the `missing` set is non-empty it opens a TCP connection to the car's resend server and sends:
100+
101+
```json
102+
{ "missing": [12043, 12044, 12051] }
103+
```
104+
105+
Only the **100 highest** (most recent) missing sequences are sent per cycle — `sorted(missing)[-100:]` — on the assumption that older gaps have already aged out of the car's 60s buffer. The car's `handle_resend` looks each one up in the ring buffer and returns whatever it still holds:
106+
107+
```json
108+
[
109+
{ "seq": 12043, "msgs": [ { "t": 1752345600.12, "id": 256, "d": "a1b2c3..." } ] }
110+
]
111+
```
112+
113+
Recovered messages are re-published to the same `can_messages` Redis channel as the live path, so downstream consumers (WebSocket bridge, TimescaleDB) see one merged stream.
114+
115+
> **Format note:** the resend response uses a JSON shape (`t`/`id`/`d`-hex) that differs from the binary `CANMessage` on the primary UDP path. Both are normalized to the same `{time, canId, data}` shape before hitting Redis, but the two encodings are worth keeping in mind when touching either path.
116+
117+
### Tuning knobs & limits
118+
119+
| Constant | Default | Location | Effect |
120+
|----------|---------|----------|--------|
121+
| `BATCH_SIZE` | `20` | `src/data.py` | messages per UDP datagram |
122+
| `BATCH_TIMEOUT` | `0.05` s | `src/data.py` | max latency before a partial batch is flushed |
123+
| `BUFFER_DURATION` | `60` s | `src/data.py` | how far back the car can retransmit |
124+
| `MISSING_CHECK_INTERVAL` | `10` s | `src/data.py` | how often the base requests resends |
125+
| `UDP_PORT` / `TCP_PORT` | `5005` / `5006` | `src/config.py` | live stream / recovery channel |
126+
127+
**Known limitations** (by design — this is a soft-real-time telemetry link, not a guaranteed-delivery bus):
128+
129+
- Recovery is best-effort. Nothing is retried indefinitely; a gap older than ~60s or beyond the 1000-entry `missing` cap is dropped from the live stream.
130+
- The `[-100:]` request slice favors recent gaps. Under a sustained large backlog, the oldest missing sequences can starve.
131+
- There is no end-to-end ACK, so the car has no knowledge of what the base did or didn't receive — all loss detection is receiver-driven.
132+
133+
---
134+
40135
## Hardware Setup (Ubuntu)
41136

42137
This section covers setting up a CAN HAT (e.g. MCP2515-based) on Ubuntu before running the software.
@@ -238,8 +333,8 @@ Images are built for both `linux/amd64` and `linux/arm64` (Raspberry Pi).
238333

239334
| Port | Protocol | Purpose |
240335
|------|----------|---------|
241-
| 5005 | UDP | CAN data streaming |
242-
| 5006 | TCP | Packet retransmission |
336+
| 5005 | UDP | CAN data streaming (see [Reliability & Retransmission](#reliability--retransmission)) |
337+
| 5006 | TCP | Packet retransmission (see [Reliability & Retransmission](#reliability--retransmission)) |
243338
| 6379 | TCP | Redis (internal) |
244339
| 8080 | HTTP | Status monitoring page |
245340
| 9080 | WebSocket | PECAN dashboard feed |

0 commit comments

Comments
 (0)