Skip to content

fix(envd): fix data race in fan-out when subscriber is removed mid-iteration#2643

Open
arkamar wants to merge 3 commits into
mainfrom
fix/envd-subscriber-data-race
Open

fix(envd): fix data race in fan-out when subscriber is removed mid-iteration#2643
arkamar wants to merge 3 commits into
mainfrom
fix/envd-subscriber-data-race

Conversation

@arkamar
Copy link
Copy Markdown
Contributor

@arkamar arkamar commented May 13, 2026

run() copied only the slice header under RLock, sharing the backing array with remove(). When remove() shifted elements in-place via append(channels[:i], channels[i+1:]...), the fan-out's stale snapshot would skip one subscriber and deliver to another twice — silent stdout/stderr corruption. Deep-copy the slice under RLock so the iteration is immune to concurrent mutations.

Fixes: a67f983 ("fix(envd): fix fan-out deadlock when process subscriber disconnects (#2579)")

@codecov
Copy link
Copy Markdown

codecov Bot commented May 13, 2026

❌ 10 Tests Failed:

Tests completed Failed Passed Skipped
2618 10 2608 5
View the full list of 11 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/metrics::TestTeamMetrics

Flake rate in main: 70.20% (Passed 152 times, Failed 358 times)

Stack Traces | 1.99s run time
=== RUN   TestTeamMetrics
=== PAUSE TestTeamMetrics
=== CONT  TestTeamMetrics
    team_metrics_test.go:61: 
        	Error Trace:	.../api/metrics/team_metrics_test.go:61
        	Error:      	Should be true
        	Test:       	TestTeamMetrics
        	Messages:   	MaxConcurrentSandboxes should be >= 0
--- FAIL: TestTeamMetrics (1.99s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestEgressFirewallAllowAllDuplicate

Flake rate in main: 54.76% (Passed 152 times, Failed 184 times)

Stack Traces | 180s run time
=== RUN   TestEgressFirewallAllowAllDuplicate
=== PAUSE TestEgressFirewallAllowAllDuplicate
=== CONT  TestEgressFirewallAllowAllDuplicate
    sandbox_network_out_test.go:30: Building custom template for network egress tests...
    template.go:44: network-egress-test: [info] Building template ah1tn6xgmr11h2d34qcw/e2084f1e-7337-4a8d-aef1-40f07e868e94
    template.go:44: network-egress-test: [info] [base] FROM ubuntu:22.04 [ffd709f131f42dfab282de47a91dd2c139e900c1c11fc574b49b517a05ef0a32]
    template.go:44: network-egress-test: [info] Base Docker image size: 30 MB
    template.go:44: network-egress-test: [info] Creating file system and pulling Docker image
    template.go:44: network-egress-test: [info] Uncompressing layer sha256:f63eb04151bcac21ad049f8d781b97b219aba392c5457907f8f3e88e43eb48ec 30 MB
    template.go:44: network-egress-test: [info] Uncompressing layer sha256:79ffba3fb9035a81a3c984e8c30394021e1990057ec1c0645c3fa14eef21dc49 12 MB
    template.go:44: network-egress-test: [info] Uncompressing layer sha256:8c4b1b28875140ed3abacaf16ad0d696f6bef912f52d2148f261a23e3349465b 168 B
    template.go:44: network-egress-test: [info] Layers extracted
    template.go:44: network-egress-test: [info] Root filesystem structure: bin, boot, dev, etc, home, lib, lib32, lib64, libx32, media, mnt, opt, proc, root, run, sbin, srv, sys, tmp, usr, var
    template.go:44: network-egress-test: [info] Provisioning sandbox template
    template.go:44: network-egress-test: [info] Provisioning was successful, cleaning up
    template.go:44: network-egress-test: [info] Sandbox template provisioned
    template.go:44: network-egress-test: [info] [base] DEFAULT USER user [90bdd4afa342293c931373351bf578872dec9179214ba3e8bf9edba311466213]
    template.go:44: network-egress-test: [info] [builder 1/1] RUN sudo apt-get update && sudo apt-get install -y curl iputils-ping dnsutils openssh-client gnupg && sudo rm -rf .../lib/apt/lists/* [2463a9871c5acb096dc00a6b03e4965da11a9e03c5f0618cbf556435df9fae17]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Reading package lists...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Reading package lists...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Building dependency tree...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Reading state information...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: curl is already the newest version (7.81.0-1ubuntu1.24).
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: openssh-client is already the newest version (1:8.9p1-3ubuntu0.15).
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: openssh-client set to manually installed.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: The following additional packages will be installed:
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: bind9-dnsutils bind9-host bind9-libs dirmngr gnupg-l10n gnupg-utils gpg
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libassuan0 libicu70
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: libksba8 liblmdb0 libmaxminddb0 libnpth0 libuv1 libxml2 pinentry-curses
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Suggested packages:
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: scdaemon mmdb-bin pinentry-doc
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: The following NEW packages will be installed:
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: bind9-dnsutils bind9-host bind9-libs dirmngr dnsutils gnupg gnupg-l10n
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: iputils-ping libassuan0 libicu70 libksba8 liblmdb0 libmaxminddb0 libnpth0
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: libuv1 libxml2 pinentry-curses
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: 0 upgraded, 24 newly installed, 0 to remove and 2 not upgraded.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Need to get 15.3 MB of archives.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: After this operation, 48.2 MB of additional disk space will be used.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 iputils-ping amd64 3:20211215-1ubuntu0.1 [43.0 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libicu70 amd64 70.1-2 [10.6 MB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libxml2 amd64 2.9.13+dfsg-1ubuntu0.11 [765 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblmdb0 amd64 0.9.24-1build2 [47.6 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libmaxminddb0 amd64 1.5.2-1build2 [24.7 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuv1 amd64 1.43.0-1ubuntu0.1 [92.7 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 bind9-libs amd64 1:9.18.39-0ubuntu0.22.04.3 [1262 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 bind9-host amd64 1:9.18.39-0ubuntu0.22.04.3 [52.5 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 bind9-dnsutils amd64 1:9.18.39-0ubuntu0.22.04.3 [158 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libassuan0 amd64 2.5.5-1build1 [38.2 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpgconf amd64 2.2.27-3ubuntu2.5 [94.3 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libksba8 amd64 1.6.0-2ubuntu0.2 [119 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libnpth0 amd64 1.6-3build2 [8664 B]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dirmngr amd64 2.2.27-3ubuntu2.5 [293 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 dnsutils all 1:9.18.39-0ubuntu0.22.04.3 [3924 B]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-l10n all 2.2.27-3ubuntu2.5 [54.5 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-utils amd64 2.2.27-3ubuntu2.5 [309 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg amd64 2.2.27-3ubuntu2.5 [519 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 pinentry-curses amd64 1.1.1-1build2 [34.4 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-agent amd64 2.2.27-3ubuntu2.5 [209 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-client amd64 2.2.27-3ubuntu2.5 [62.7 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-server amd64 2.2.27-3ubuntu2.5 [57.6 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpgsm amd64 2.2.27-3ubuntu2.5 [197 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg all 2.2.27-3ubuntu2.5 [315 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stderr]: debconf: delaying package configuration, since apt-utils is not installed
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Fetched 15.3 MB in 0s (32.1 MB/s)
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package iputils-ping.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 75%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 80%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 85%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 90%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 95%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 100%(Reading database ... 12395 files and directories currently installed.)
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../00-iputils-ping_3%3a20211215-1ubuntu0.1_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking iputils-ping (3:20211215-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libicu70:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../01-libicu70_70.1-2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libicu70:amd64 (70.1-2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libxml2:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../02-libxml2_2.9.13+dfsg-1ubuntu0.11_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libxml2:amd64 (2.9.13+dfsg-1ubuntu0.11) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package liblmdb0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../03-liblmdb0_0.9.24-1build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking liblmdb0:amd64 (0.9.24-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libmaxminddb0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../04-libmaxminddb0_1.5.2-1build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libmaxminddb0:amd64 (1.5.2-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libuv1:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../05-libuv1_1.43.0-1ubuntu0.1_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libuv1:amd64 (1.43.0-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package bind9-libs:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../06-bind9-libs_1%3a9.18.39-0ubuntu0.22.04.3_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking bind9-libs:amd64 (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package bind9-host.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../07-bind9-host_1%3a9.18.39-0ubuntu0.22.04.3_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking bind9-host (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package bind9-dnsutils.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../08-bind9-dnsutils_1%3a9.18.39-0ubuntu0.22.04.3_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking bind9-dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libassuan0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../09-libassuan0_2.5.5-1build1_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libassuan0:amd64 (2.5.5-1build1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpgconf.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../10-gpgconf_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpgconf (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libksba8:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../11-libksba8_1.6.0-2ubuntu0.2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libksba8:amd64 (1.6.0-2ubuntu0.2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libnpth0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../12-libnpth0_1.6-3build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libnpth0:amd64 (1.6-3build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package dirmngr.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../13-dirmngr_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking dirmngr (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package dnsutils.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../14-dnsutils_1%3a9.18.39-0ubuntu0.22.04.3_all.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gnupg-l10n.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../15-gnupg-l10n_2.2.27-3ubuntu2.5_all.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gnupg-l10n (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gnupg-utils.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../16-gnupg-utils_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gnupg-utils (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../17-gpg_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package pinentry-curses.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../18-pinentry-curses_1.1.1-1build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking pinentry-curses (1.1.1-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg-agent.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../19-gpg-agent_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg-agent (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg-wks-client.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../20-gpg-wks-client_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg-wks-client (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg-wks-server.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../21-gpg-wks-server_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg-wks-server (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpgsm.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../22-gpgsm_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpgsm (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gnupg.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../23-gnupg_2.2.27-3ubuntu2.5_all.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gnupg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libksba8:amd64 (1.6.0-2ubuntu0.2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up liblmdb0:amd64 (0.9.24-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libmaxminddb0:amd64 (1.5.2-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libnpth0:amd64 (1.6-3build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libassuan0:amd64 (2.5.5-1build1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libuv1:amd64 (1.43.0-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gnupg-l10n (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpgconf (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up iputils-ping (3:20211215-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libicu70:amd64 (70.1-2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gnupg-utils (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up pinentry-curses (1.1.1-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg-agent (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent-browser.socket → .../systemd/user/gpg-agent-browser.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent-extra.socket → .../systemd/user/gpg-agent-extra.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent-ssh.socket → .../systemd/user/gpg-agent-ssh.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent.socket → .../systemd/user/gpg-agent.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpgsm (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up dirmngr (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/dirmngr.socket → .../systemd/user/dirmngr.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg-wks-server (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libxml2:amd64 (2.9.13+dfsg-1ubuntu0.11) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg-wks-client (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up bind9-libs:amd64 (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gnupg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up bind9-host (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up bind9-dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Processing triggers for libc-bin (2.35-0ubuntu3.13) ...
    template.go:44: network-egress-test: [info] [finalize] Finalizing template build [678d8c6396745e76c1e6fcb6ec74a06eb1cc3bffc1101dbaec5c89848fb75045]
    template.go:44: network-egress-test: [info] [optimize] Optimizing template [027a14840d213f9902f523344074c7f19992129998b9867f8c379d0012f44fe9]
    template.go:44: network-egress-test: [info] Build finished, took 2m50s
    sandbox_network_out_test.go:32: Build completed successfully
    sandbox_network_out_test.go:49: Network test template built: ah1tn6xgmr11h2d34qcw
    sandbox_network_out_test.go:489: Command [curl] output: event:{start:{pid:1322}}
Executing command curl in sandbox ixkjwyecsol7q0xgfzkyz
    sandbox_network_out_test.go:489: Command [curl] output: event:{end:{exit_code:28  exited:true  status:"exit status 28"  error:"exit status 28"}}
    sandbox_network_out_test.go:489: 
        	Error Trace:	.../api/sandboxes/sandbox_network_out_test.go:67
        	            				.../api/sandboxes/sandbox_network_out_test.go:489
        	Error:      	Received unexpected error:
        	            	command curl in sandbox im7k47e20c0zx5uhki46y failed with exit code 28
        	Test:       	TestEgressFirewallAllowAllDuplicate
        	Messages:   	Expected curl to 8.8.8.8 to succeed with duplicate 0.0.0.0/0 allow
--- FAIL: TestEgressFirewallAllowAllDuplicate (179.66s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestEgressFirewallAllowCIDRRange

Flake rate in main: 54.63% (Passed 152 times, Failed 183 times)

Stack Traces | 2.22s run time
=== RUN   TestEgressFirewallAllowCIDRRange
=== PAUSE TestEgressFirewallAllowCIDRRange
=== CONT  TestEgressFirewallAllowCIDRRange
    sandbox_network_out_test.go:179: Command [curl] output: event:{start:{pid:1322}}
    sandbox_network_out_test.go:179: Command [curl] output: event:{data:{stdout:"HTTP/2 301 \r\ndate: Wed, 13 May 2026 16:10:36 GMT\r\nlocation: https://one.one.one.one/\r\nserver: cloudflare\r\ncf-ray: 9fb2ef2a18483997-IAD\r\n\r\n"}}
    sandbox_network_out_test.go:179: Command [curl] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_network_out_test.go:179: Command [curl] completed successfully in sandbox ikywdgll4ky19wt4jz3ww
    sandbox_network_out_test.go:180: Command [curl] output: event:{start:{pid:1323}}
    sandbox_network_out_test.go:180: 
        	Error Trace:	.../api/sandboxes/sandbox_network_out_test.go:75
        	            				.../api/sandboxes/sandbox_network_out_test.go:180
        	Error:      	"failed to execute command curl in sandbox ikywdgll4ky19wt4jz3ww: invalid_argument: protocol error: incomplete envelope: unexpected EOF" does not contain "failed with exit code"
        	Test:       	TestEgressFirewallAllowCIDRRange
        	Messages:   	Expected connection failure message
--- FAIL: TestEgressFirewallAllowCIDRRange (2.22s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestUpdateNetworkConfig

Flake rate in main: 76.44% (Passed 160 times, Failed 519 times)

Stack Traces | 47s run time
=== RUN   TestUpdateNetworkConfig
=== PAUSE TestUpdateNetworkConfig
=== CONT  TestUpdateNetworkConfig
Executing command curl in sandbox igcpciuo43wzcy7rn63n1
--- FAIL: TestUpdateNetworkConfig (47.00s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false

Flake rate in main: 76.91% (Passed 154 times, Failed 513 times)

Stack Traces | 7.02s run time
=== RUN   TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false
Executing command curl in sandbox i2x3uwfu4ypncz0gmlbjn
    sandbox_network_update_test.go:372: Command [curl] output: event:{start:{pid:1365}}
    sandbox_network_update_test.go:372: Command [curl] output: event:{end:{exit_code:35  exited:true  status:"exit status 35"  error:"exit status 35"}}
Executing command curl in sandbox i2x3uwfu4ypncz0gmlbjn
    sandbox_network_update_test.go:372: Command [curl] output: event:{start:{pid:1366}}
    sandbox_network_update_test.go:372: Command [curl] output: event:{end:{exit_code:35  exited:true  status:"exit status 35"  error:"exit status 35"}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{start:{pid:1367}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{data:{stdout:"HTTP/2 302 \r\nx-content-type-options: nosniff\r\nlocation: https://dns.google/\r\ndate: Wed, 13 May 2026 16:11:55 GMT\r\ncontent-type: text/html; charset=UTF-8\r\nserver: HTTP server (unknown)\r\ncontent-length: 216\r\nx-xss-protection: 0\r\nx-frame-options: SAMEORIGIN\r\nalt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000\r\n\r\n"}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_network_update_test.go:391: Command [curl] completed successfully in sandbox i2x3uwfu4ypncz0gmlbjn
    sandbox_network_update_test.go:391: 
        	Error Trace:	.../api/sandboxes/sandbox_network_out_test.go:74
        	            				.../api/sandboxes/sandbox_network_update_test.go:60
        	            				.../api/sandboxes/sandbox_network_update_test.go:391
        	Error:      	An error is expected but got nil.
        	Test:       	TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false
        	Messages:   	https://8.8.8.8 should be blocked
--- FAIL: TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false (7.02s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost

Flake rate in main: 56.82% (Passed 266 times, Failed 350 times)

Stack Traces | 0s run time
=== RUN   TestBindLocalhost
=== PAUSE TestBindLocalhost
=== CONT  TestBindLocalhost
--- FAIL: TestBindLocalhost (0.00s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_0_0_0_0

Flake rate in main: 62.78% (Passed 150 times, Failed 253 times)

Stack Traces | 8.75s run time
=== RUN   TestBindLocalhost/bind_0_0_0_0
=== PAUSE TestBindLocalhost/bind_0_0_0_0
=== CONT  TestBindLocalhost/bind_0_0_0_0
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1253}}
Executing command python in sandbox imgzlxa1g0vn096tfuf7j
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_0_0_0_0
        	Messages:   	Unexpected status code 502 for bind address 0.0.0.0
--- FAIL: TestBindLocalhost/bind_0_0_0_0 (8.75s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_::1

Flake rate in main: 64.45% (Passed 150 times, Failed 272 times)

Stack Traces | 8.74s run time
=== RUN   TestBindLocalhost/bind_::1
=== PAUSE TestBindLocalhost/bind_::1
=== CONT  TestBindLocalhost/bind_::1
Executing command python in sandbox i6a3fy0xpcemoe7r9wpra
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1252}}
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_::1
        	Messages:   	Unexpected status code 502 for bind address ::1
--- FAIL: TestBindLocalhost/bind_::1 (8.74s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_localhost

Flake rate in main: 64.37% (Passed 150 times, Failed 271 times)

Stack Traces | 7.31s run time
=== RUN   TestBindLocalhost/bind_localhost
=== PAUSE TestBindLocalhost/bind_localhost
=== CONT  TestBindLocalhost/bind_localhost
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1253}}
Executing command python in sandbox is1hfbn9z9nim4921xurg
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_localhost
        	Messages:   	Unexpected status code 502 for bind address localhost
--- FAIL: TestBindLocalhost/bind_localhost (7.31s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 66.10% (Passed 160 times, Failed 312 times)

Stack Traces | 72.8s run time
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:26: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (72.82s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 67.11% (Passed 150 times, Failed 306 times)

Stack Traces | 30.1s run time
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1264}}
Executing command bash in sandbox ipf97l2h3jmvc20rdkusb (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 183 MB\nFree memory before tmpfs mount: 801 MB\nMemory to use in integrity test (80% of free, min 64MB): 640 MB\n"}}
Executing command bash in sandbox ipf97l2h3jmvc20rdkusb (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"640+0 records in\n640+0 records out\n671088640 bytes (671 MB, 640 MiB) copied, 4.83791 s, 139 MB/s\n\tCommand being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=640\"\n\tUser time (seconds): 0.00\n\tSystem time (sec"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 830 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"onds): 4.77\n\tPercent of CPU this job got: 98%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:04.84\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2652\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n\tMinor (reclaiming a frame) page faults: 343\n\tVoluntary context switches: 3\n\tInvoluntary context switches: 37\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox i351ymcwvd2h1knli2nqg
Executing command bash in sandbox i351ymcwvd2h1knli2nqg (user: root)
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{start:{pid:1280}}
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{data:{stdout:"c9acfe115333fc4245f4cbbeb35d17fc8d392039a0147fc82d6b7d06e2ced6af\n"}}
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:74: Command [bash] completed successfully in sandbox i351ymcwvd2h1knli2nqg
Executing command bash in sandbox i351ymcwvd2h1knli2nqg (user: root)
    sandbox_memory_integrity_test.go:99: Command [bash] output: event:{start:{pid:1283}}
    sandbox_memory_integrity_test.go:100: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:100
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox i351ymcwvd2h1knli2nqg: invalid_argument: protocol error: incomplete envelope: unexpected EOF
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (30.06s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b9a572a6d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/envd/internal/services/process/handler/multiplex_test.go Outdated
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a race condition in the MultiplexedChannel by creating a shallow copy of the subscribers slice before iteration, which prevents slice corruption during concurrent modifications. A regression test has been added to verify the fix and ensure stable delivery to subscribers. I have no further feedback.

Comment thread packages/envd/internal/services/process/handler/multiplex.go
@arkamar arkamar force-pushed the fix/envd-subscriber-data-race branch from 8b9a572 to 17fdd0a Compare May 13, 2026 15:05
Comment thread packages/envd/internal/services/process/handler/multiplex.go
Comment thread packages/envd/internal/services/process/handler/multiplex_test.go Outdated
@cursor
Copy link
Copy Markdown

cursor Bot commented May 13, 2026

PR Summary

Medium Risk
Touches concurrency in the process output fan-out loop; if incorrect it can still drop/duplicate messages or panic under load. The change is small but affects runtime behavior in multi-subscriber scenarios.

Overview
Fixes a race in MultiplexedChannel.run() where iterating over m.channels could observe in-place slice mutations from concurrent unsubscribe/removal, causing skipped/duplicated deliveries and potential corruption. The fan-out now snapshots the subscriber slice under RLock by deep-copying it before iterating, and bumps pkg.Version to 0.5.18.

Reviewed by Cursor Bugbot for commit 340d4c1. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread packages/envd/internal/services/process/handler/multiplex_test.go Outdated
Comment thread packages/envd/internal/services/process/handler/multiplex_test.go Outdated
arkamar added 2 commits May 13, 2026 17:33
…eration

run() copied only the slice header under RLock, sharing the backing
array with remove(). When remove() shifted elements in-place via
append(channels[:i], channels[i+1:]...), the fan-out's stale snapshot
would skip one subscriber and deliver to another twice — silent
stdout/stderr corruption. Deep-copy the slice under RLock so the
iteration is immune to concurrent mutations.

Fixes: a67f983 ("fix(envd): fix fan-out deadlock when process subscriber disconnects (#2579)")
@arkamar arkamar force-pushed the fix/envd-subscriber-data-race branch from 40f5326 to 8756a4c Compare May 13, 2026 15:33
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8756a4c. Configure here.

Comment thread packages/envd/internal/services/process/handler/multiplex_test.go Outdated
for v := range m.Source {
m.mu.RLock()
subs := m.channels
subs := append([]*subscriber[T](nil), m.channels...)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
subs := append([]*subscriber[T](nil), m.channels...)
subs := slices.Clone(m.channels)

@@ -54,7 +54,7 @@ func NewMultiplexedChannel[T any](buffer int) *MultiplexedChannel[T] {
func (m *MultiplexedChannel[T]) run() {
for v := range m.Source {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be able to move the allocation only to remove instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants