feat(rootfs): isolate envd in a dedicated network namespace#2700
Closed
ValentaTomas wants to merge 2 commits into
Closed
feat(rootfs): isolate envd in a dedicated network namespace#2700ValentaTomas wants to merge 2 commits into
ValentaTomas wants to merge 2 commits into
Conversation
Customer iptables in the default namespace can break envd's MMDS / /init routing (cf. customer DNAT'ing :80 to a different port and breaking the MMDS token fetch). Move eth0 + envd into envd-ns; route customer traffic through a veth pair. Customer can only modify their own namespace. Port forwarding chain: orchestrator -> eth0 (envd-ns) -> DNAT to 192.168.250.1 -> veth-customer (default ns) -> existing PREROUTING DNAT to 127.0.0.1. Port 49983 (envd) RETURN's from the envd-ns DNAT, so /init reaches envd directly without traversing the customer namespace.
PR SummaryMedium Risk Overview Reviewed by Cursor Bugbot for commit 91fea54. Bugbot is set up for automated code reviews on this repo. Configure here. |
❌ 19 Tests Failed:
View the full list of 22 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
The iptables DNAT rule in the setup script is restricted to the TCP protocol, which prevents other traffic types like UDP or ICMP from being forwarded to the customer namespace. Removing the protocol restriction from the DNAT rule ensures all non-envd traffic is correctly routed to the customer namespace.
Comment on lines
+57
to
+58
| # Forward all other inbound TCP to the customer-side veth peer. | ||
| ip netns exec "${NS}" iptables -t nat -A PREROUTING -i eth0 -p tcp -j DNAT --to-destination "${VETH_HOST_IP}" |
Contributor
There was a problem hiding this comment.
The iptables DNAT rule in the setup script is restricted to the TCP protocol, which prevents other traffic types like UDP or ICMP from being forwarded to the customer namespace. This breaks any UDP-based services or diagnostic tools that the customer might be running. Removing the protocol restriction from the DNAT rule ensures all non-envd traffic is correctly routed to the customer namespace.
# Forward all other inbound traffic to the customer-side veth peer.
ip netns exec "${NS}" iptables -t nat -A PREROUTING -i eth0 -j DNAT --to-destination "${VETH_HOST_IP}"
…in in unit Per Petr's review: split the imperative shell script into: - .netdev for the veth pair - .network for customer-side veth IP + gateway - e2b-netns.service uses individual ExecStart lines (with - prefix for idempotent ip-netns/ip-link operations); iptables rules remain wrapped in sh -c with -C check before -A so they don't accumulate across reboots.
Member
Author
|
Refactored per @arkamar's suggestion (91fea54):
Trade-offs vs. fully-declarative-everywhere:
LMK if you'd like to push further (e.g. install |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Child processes spawned by envd inherit wrong network namespace
- Removed NetworkNamespacePath from envd.service so envd runs in the default namespace, updated socat to bind to veth-customer IP (192.168.250.1), and removed port 49983 RETURN rule so all traffic gets DNAT'ed correctly.
Or push these changes by commenting:
@cursor push d12aa6ed65
Preview (d12aa6ed65)
diff --git a/packages/envd/internal/port/forward.go b/packages/envd/internal/port/forward.go
--- a/packages/envd/internal/port/forward.go
+++ b/packages/envd/internal/port/forward.go
@@ -25,7 +25,7 @@
PortStateDelete PortState = "DELETE"
)
-var defaultGatewayIP = net.IPv4(169, 254, 0, 21)
+var defaultGatewayIP = net.IPv4(192, 168, 250, 1)
type PortToForward struct {
socat *exec.Cmd
diff --git a/packages/orchestrator/pkg/template/build/core/rootfs/files/e2b-netns.service.tpl b/packages/orchestrator/pkg/template/build/core/rootfs/files/e2b-netns.service.tpl
--- a/packages/orchestrator/pkg/template/build/core/rootfs/files/e2b-netns.service.tpl
+++ b/packages/orchestrator/pkg/template/build/core/rootfs/files/e2b-netns.service.tpl
@@ -30,7 +30,6 @@
# Idempotent iptables: -C check before -A. Wrapped in sh because there is no
# declarative systemd directive for "add rule if not present".
ExecStart=/bin/sh -c 'ip netns exec envd-ns iptables -t nat -C POSTROUTING -o eth0 -j MASQUERADE 2>/dev/null || ip netns exec envd-ns iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE'
-ExecStart=/bin/sh -c 'ip netns exec envd-ns iptables -t nat -C PREROUTING -i eth0 -p tcp --dport 49983 -j RETURN 2>/dev/null || ip netns exec envd-ns iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 49983 -j RETURN'
ExecStart=/bin/sh -c 'ip netns exec envd-ns iptables -t nat -C PREROUTING -i eth0 -p tcp -j DNAT --to-destination 192.168.250.1 2>/dev/null || ip netns exec envd-ns iptables -t nat -A PREROUTING -i eth0 -p tcp -j DNAT --to-destination 192.168.250.1'
[Install]
diff --git a/packages/orchestrator/pkg/template/build/core/rootfs/files/envd.service.tpl b/packages/orchestrator/pkg/template/build/core/rootfs/files/envd.service.tpl
--- a/packages/orchestrator/pkg/template/build/core/rootfs/files/envd.service.tpl
+++ b/packages/orchestrator/pkg/template/build/core/rootfs/files/envd.service.tpl
@@ -13,9 +13,6 @@
Restart=always
User=root
Group=root
-# Run envd inside the dedicated network namespace set up by e2b-netns.service.
-# Customer iptables in the default namespace cannot reach this namespace.
-NetworkNamespacePath=/run/netns/envd-ns
Environment=GOTRACEBACK=all
LimitCORE=infinity
ExecStartPre=/bin/sh -c 'mountpoint -q /etc/ssl/certs || (mkdir -p /run/e2b/certs && mount --bind /run/e2b/certs /etc/ssl/certs) && ([ -s /etc/ssl/certs/ca-certificates.crt ] || update-ca-certificates)'You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 91fea54. Configure here.
| Group=root | ||
| # Run envd inside the dedicated network namespace set up by e2b-netns.service. | ||
| # Customer iptables in the default namespace cannot reach this namespace. | ||
| NetworkNamespacePath=/run/netns/envd-ns |
There was a problem hiding this comment.
Child processes spawned by envd inherit wrong network namespace
High Severity
NetworkNamespacePath=/run/netns/envd-ns causes all child processes forked by envd (socat port forwarders, user terminal sessions, command execution) to inherit envd-ns. The existing socat port forwarder binds to 169.254.0.21:port in envd-ns and tries to connect to localhost:port in envd-ns — but customer processes listen in the default namespace's loopback, which is unreachable from envd-ns. Additionally, any user-facing process spawning (terminals, exec) would place the user process in the wrong network namespace where veth-customer (192.168.250.1) isn't visible.
Reviewed by Cursor Bugbot for commit 91fea54. Configure here.
This was referenced May 17, 2026
Member
Author
|
Closing in favor of the dirty self-heal pin (#2701). The dedicated network namespace is still the right structural fix and we'll likely reopen / rebase this once the immediate Bitfrost mitigation lands and we have time to coordinate the systemd-networkd + iptables-in-customer-ns story properly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves `eth0` + envd into a new `envd-ns` so customer iptables changes in the default namespace can no longer break envd's MMDS / `/init` path (cf. the recent customer who DNAT'd :80 → :4200 and broke MMDS token fetch).
Topology
```
default ns (customer) envd-ns
veth-customer 192.168.250.1 veth-envd 192.168.250.2
eth0 169.254.0.21 (moved in)
```
Notes
Test plan