Skip to content

Commit d3e4633

Browse files
authored
test(e2e): allocate HA node p2p ports above the ephemeral range to deflake e2e_ha_full (#24418)
## Summary Deflakes `e2e_ha_full.parallel.test.ts`, which FLAKED in CI ([run 6d29cd3fcf374a68](http://ci.aztec-labs.com/6d29cd3fcf374a68)) reporting `should not be affected by process.env.TZ changes` as the failing test. ## Root cause - The reported test name is a red herring. The failure happened in `beforeAll` while creating the second HA node (HA-1): `CodeError: Transport (@libp2p/tcp) could not listen on any available address`, code `ERR_NO_VALID_ADDRESSES`. jest attributed the `beforeAll` throw to the first test in the suite (the timezone test, a pure Postgres test that never ran) and skipped the other 7. - The suite assigned **fixed** p2p ports `(config.p2pPort ?? 40400) + i + 1`, i.e. 40400 (bootstrap), 40401-40405 (HA nodes). These sit inside the Linux default ephemeral port range (`32768-60999`, per `/proc/sys/net/ipv4/ip_local_port_range`). - The OS draws ephemeral ports from that range for the in-process prover node (created by `setup()` with `p2pPort: 0`) and for outbound TCP connections. When an ephemeral socket already held one of 40401-40405 at bind time, that HA node's libp2p TCP `listen()` failed with `EADDRINUSE`. libp2p runs the listen attempt under `Promise.allSettled` and, finding no fulfilled result, throws the aggregate `ERR_NO_VALID_ADDRESSES` without surfacing the underlying `EADDRINUSE` — which is why no `EADDRINUSE` string appears in the logs. - Confirmed against the passing retry (`061f6753f56b00fd`): identical fixed ports, but there HA-0..HA-4 bound 40401-40405 cleanly. The only divergence is HA-1 failing to bind 40402 in the failed run. ## Fix - Allocate every node's p2p port via `get-port` from **61000-65535**, above the ephemeral range, so neither in-process ephemeral sockets nor concurrent CI jobs can be holding a node's port when libp2p binds it. Applied to the bootstrap node (via `setup()` opts) and to each HA node. - Set `p2pBroadcastPort` alongside `p2pPort`: discv5 defaults the broadcast port by mutating the config object in place, and that mutated value would otherwise leak from the bootstrap config into the HA node configs (which are built by spreading `config`), making them advertise the wrong port in their ENR. ## Verification - TypeScript build of `@aztec/end-to-end` passes; prettier clean. - A local repro of the flake is not feasible: it is a probabilistic ephemeral-port collision that cannot be deterministically forced, and the test requires the full docker-compose HA stack (Postgres + Web3Signer). The fix is grounded in both the logs (HA-1 failing to bind fixed port 40402 with `ERR_NO_VALID_ADDRESSES`) and source (fixed ports inside the ephemeral range; `p2pPort: 0` prover node; discv5 binding UDP on the same port) — moving the ports above the ephemeral range removes the collision source entirely.
1 parent 0e1827b commit d3e4633

1 file changed

Lines changed: 24 additions & 1 deletion

File tree

yarn-project/end-to-end/src/composed/ha/e2e_ha_full.parallel.test.ts

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ import { PostgresSlashingProtectionDatabase } from '@aztec/validator-ha-signer/d
3333
import { type DutyRow, DutyStatus, DutyType } from '@aztec/validator-ha-signer/types';
3434

3535
import { jest } from '@jest/globals';
36+
import getPort, { portNumbers } from 'get-port';
3637
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
3738
import { tmpdir } from 'node:os';
3839
import { join } from 'node:path';
@@ -63,6 +64,14 @@ const NODE_COUNT = 5;
6364
const VALIDATOR_COUNT = 4;
6465
const COMMITTEE_SIZE = 4;
6566

67+
// Allocate p2p listen ports from above the OS ephemeral range (Linux default tops out at 60999) so they
68+
// never collide with an ephemeral socket the OS may already have handed out -- e.g. the in-process prover
69+
// node (which listens on p2pPort 0) or any outbound connection. The previous fixed 4040x ports sat inside
70+
// the ephemeral range, so an ephemeral socket occasionally held a node's port at bind time, surfacing as
71+
// libp2p ERR_NO_VALID_ADDRESSES and aborting beforeAll. get-port also locks each returned port briefly, so
72+
// concurrent calls within this process never hand back the same one.
73+
const getFreeP2PPort = () => getPort({ port: portNumbers(61000, 65535) });
74+
6675
async function registerTestContract(wallet: TestWallet): Promise<TestContract> {
6776
const instance = await getContractInstanceFromInstantiationParams(TestContract.artifact, {
6877
constructorArgs: [],
@@ -206,6 +215,8 @@ describe('HA Full Setup', () => {
206215

207216
const initialValidators = createInitialValidatorsFromPrivateKeys(attesterPrivateKeys);
208217

218+
const bootstrapP2PPort = await getFreeP2PPort();
219+
209220
({ teardown, logger, wallet, aztecNode, config, accounts, dateProvider, deployL1ContractsValues, genesis } =
210221
await setup(
211222
// A single default initializerless account, created/funded/registered by setup with no on-chain
@@ -234,6 +245,13 @@ describe('HA Full Setup', () => {
234245
disableValidator: true,
235246
// Enable P2P for transaction gossip
236247
p2pEnabled: true,
248+
// Bind the bootstrap node above the ephemeral range too (see getFreeP2PPort), so it can't lose
249+
// its port to an ephemeral socket and abort the whole suite before any HA node is created. Set
250+
// the broadcast port explicitly to the same value: discv5 otherwise defaults p2pBroadcastPort to
251+
// p2pPort by mutating this config object in place, and that mutated value would then leak into the
252+
// HA nodes' configs below (built by spreading `config`), making them advertise the wrong port.
253+
p2pPort: bootstrapP2PPort,
254+
p2pBroadcastPort: bootstrapP2PPort,
237255
// Enable slashing for testing governance + slashing vote coordination
238256
slasherEnabled: true,
239257
slashingRoundSizeInEpochs: 1, // 32 slots (1 epoch)
@@ -298,6 +316,7 @@ describe('HA Full Setup', () => {
298316

299317
const dataDirectory = config.dataDirectory ? `${config.dataDirectory}-${i}` : undefined;
300318

319+
const nodeP2PPort = await getFreeP2PPort();
301320
const nodeConfig: AztecNodeConfig = {
302321
...config,
303322
nodeId,
@@ -313,7 +332,11 @@ describe('HA Full Setup', () => {
313332
disableValidator: false,
314333
// Enable P2P for transaction and block gossip
315334
p2pEnabled: true,
316-
p2pPort: (config.p2pPort ?? 40400) + i + 1,
335+
// Each HA node gets its own free port above the ephemeral range. Override the broadcast port too:
336+
// `...config` carries the bootstrap node's broadcast port (discv5 sets it in place), which would
337+
// otherwise make every HA node advertise the bootstrap's port instead of its own.
338+
p2pPort: nodeP2PPort,
339+
p2pBroadcastPort: nodeP2PPort,
317340
// Connect to bootstrap node for tx gossip
318341
bootstrapNodes: [bootstrapNodeEnr],
319342
web3SignerUrl,

0 commit comments

Comments
 (0)