e2e-tests: Fix bulletin_fetch flake by staging the relay snapshot per-validator

[BigTava]

Problem

bulletin_fetch flakes intermittently with a panic in zombienet-provider's archive.unpack().unwrap():

called `Result::unwrap()` on an `Err` value: Custom { kind: UnexpectedEof,
  error: TarError { desc: "failed to unpack `data/chains/.../db/full/000008.log`", ... } }

Reproduced 3/5 in a rerun loop on a single commit. Failure path was inside zombienet's local copy/read/extract pipeline.

Cause

alice and bob both passed the same relay tarball path to with_db_snapshot(...). zombienet-provider keys its internal extraction cache by sha256(path_string), so the two validators landed in the same cache slot and raced writing/reading the same intermediate <hash>.tgz in the namespace dir. When the read won the race against an in-progress write, tar parsing hit UnexpectedEof mid-entry.

[skunert]

Thanks! Can you maybe unify this with the other prefetchers we have in the tests?

I think the proper solution would be for zombiened-sdk to handle this gracefully, it seems totally normal to have multiple nodes reference the same snapshot. Opened an issue on zombienet-sdk, in the meantime we should keep the workarounds.

cc @pepoviola

[pepoviola]

Hi @BigTava / @skunert, thanks for reporting this issue. Looks like in native provider we took the decision of not copy the snap per node and we can have the described race condition when multiple nodes use the same snap. I easy workaround to use until I draft a new release with this fixed is to set in the global settings spawn_concurrency = 1, so this will make the spawning logic sequential an should works as expected.

I will work in the issue and ping you for bumping the version of zombienet when is ready.

Thx!

[skunert]

Okay then lets use spawn_concurrency = 1 for now, thanks!