Commit d3e4633
authored
test(e2e): allocate HA node p2p ports above the ephemeral range to deflake e2e_ha_full (#24418)
## Summary
Deflakes `e2e_ha_full.parallel.test.ts`, which FLAKED in CI ([run
6d29cd3fcf374a68](http://ci.aztec-labs.com/6d29cd3fcf374a68)) reporting
`should not be affected by process.env.TZ changes` as the failing test.
## Root cause
- The reported test name is a red herring. The failure happened in
`beforeAll` while creating the second HA node (HA-1): `CodeError:
Transport (@libp2p/tcp) could not listen on any available address`, code
`ERR_NO_VALID_ADDRESSES`. jest attributed the `beforeAll` throw to the
first test in the suite (the timezone test, a pure Postgres test that
never ran) and skipped the other 7.
- The suite assigned **fixed** p2p ports `(config.p2pPort ?? 40400) + i
+ 1`, i.e. 40400 (bootstrap), 40401-40405 (HA nodes). These sit inside
the Linux default ephemeral port range (`32768-60999`, per
`/proc/sys/net/ipv4/ip_local_port_range`).
- The OS draws ephemeral ports from that range for the in-process prover
node (created by `setup()` with `p2pPort: 0`) and for outbound TCP
connections. When an ephemeral socket already held one of 40401-40405 at
bind time, that HA node's libp2p TCP `listen()` failed with
`EADDRINUSE`. libp2p runs the listen attempt under `Promise.allSettled`
and, finding no fulfilled result, throws the aggregate
`ERR_NO_VALID_ADDRESSES` without surfacing the underlying `EADDRINUSE` —
which is why no `EADDRINUSE` string appears in the logs.
- Confirmed against the passing retry (`061f6753f56b00fd`): identical
fixed ports, but there HA-0..HA-4 bound 40401-40405 cleanly. The only
divergence is HA-1 failing to bind 40402 in the failed run.
## Fix
- Allocate every node's p2p port via `get-port` from **61000-65535**,
above the ephemeral range, so neither in-process ephemeral sockets nor
concurrent CI jobs can be holding a node's port when libp2p binds it.
Applied to the bootstrap node (via `setup()` opts) and to each HA node.
- Set `p2pBroadcastPort` alongside `p2pPort`: discv5 defaults the
broadcast port by mutating the config object in place, and that mutated
value would otherwise leak from the bootstrap config into the HA node
configs (which are built by spreading `config`), making them advertise
the wrong port in their ENR.
## Verification
- TypeScript build of `@aztec/end-to-end` passes; prettier clean.
- A local repro of the flake is not feasible: it is a probabilistic
ephemeral-port collision that cannot be deterministically forced, and
the test requires the full docker-compose HA stack (Postgres +
Web3Signer). The fix is grounded in both the logs (HA-1 failing to bind
fixed port 40402 with `ERR_NO_VALID_ADDRESSES`) and source (fixed ports
inside the ephemeral range; `p2pPort: 0` prover node; discv5 binding UDP
on the same port) — moving the ports above the ephemeral range removes
the collision source entirely.1 parent 0e1827b commit d3e4633
1 file changed
Lines changed: 24 additions & 1 deletion
Lines changed: 24 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| |||
63 | 64 | | |
64 | 65 | | |
65 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
66 | 75 | | |
67 | 76 | | |
68 | 77 | | |
| |||
206 | 215 | | |
207 | 216 | | |
208 | 217 | | |
| 218 | + | |
| 219 | + | |
209 | 220 | | |
210 | 221 | | |
211 | 222 | | |
| |||
234 | 245 | | |
235 | 246 | | |
236 | 247 | | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
237 | 255 | | |
238 | 256 | | |
239 | 257 | | |
| |||
298 | 316 | | |
299 | 317 | | |
300 | 318 | | |
| 319 | + | |
301 | 320 | | |
302 | 321 | | |
303 | 322 | | |
| |||
313 | 332 | | |
314 | 333 | | |
315 | 334 | | |
316 | | - | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
317 | 340 | | |
318 | 341 | | |
319 | 342 | | |
| |||
0 commit comments