test failed in CI: omicron-nexus app::sagas::instance_update::test::migration_update_can_complete_with_dead_switch

[sunshowers]

This test failed on a CI run on "main":

https://github.com/oxidecomputer/omicron/runs/84346615139

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KWCP96WFAQV8TZ8SDWAX5SZ2/sKuuwKDQ966gNkmmKQ2mtpfitWgbEWWMABvpbA5wJpsh3aG3/01KWCP9Y94CXKVQZ7W81BA01FR#S7176

Excerpt from the log showing the failure:

thread 'app::sagas::instance_update::test::migration_update_can_complete_with_dead_switch' (2) panicked at nexus/src/app/sagas/instance_update/mod.rs:2706:10:
called `Result::unwrap()` on an `Err` value: switch0 did not have 1 NAT entries after 60s

I spent some

[sunshowers]

I spent some time looking at this and it looks like the cause is a port reuse race.

The test is at

omicron/nexus/src/app/sagas/instance_update/mod.rs

Line 2658 in 9f3125d

let switch0_port = shutdown_switch0(&cptestctx).await;

.

We do:

let switch0_port = shutdown_switch0(&cptestctx).await;

// ...

restart_switch0(&cptestctx, switch0_port).await;

The idea is to reuse the same port across the shutdown and restart, so that clients don't have to be updated. But what appears to be happening in practice (based on all the clues from the logs) is that a different test's dendrite gets randomly assigned the same port, and so there ends up being cross-test communication. That's quite concerning to me from a test reliability perspective!

Some ways to fix this:

• finally invest in having a separate zone (and more specifically a separate TCP port namespace) for each test
• use nextest's slot numbers to generate fixed port numbers
• use Unix domain sockets rather than TCP sockets
• having a proxy server in the middle which stays bound to the port
• hardcoding ports (probably the worst option)

Thanks to @hawkw for letting me bounce ideas off of her.

[sunshowers]

opening for a few days. I'm 95% sure #10726 fixes this but not 100%.