Egress tunnel capture + agent identity

[Nina Polshakova (npolshakova)]

POC for #126

Based on design discussed in https://docs.google.com/document/d/1KmpIFu2gnqy9gp95wASgIo_vkJ_dA1DZckV8upET6bs/edit?usp=sharing

• Tests pass
• Appropriate changes to documentation are included in the PR

Summary:
This is a proof of concept egress capture path for actors. It introduces a reusable internal/egress package that:

• starts local capture listeners
• derives the CONNECT authority from HTTP Host or TLS SNI, and opens a CONNECT-style tunnel to a resolved PEP address.

The gVisor and microvm runtimes wire this into actor network setup by redirecting actor HTTP/HTTPS egress traffic to local capture ports. The Linux-only glue shared by both runtimes (SO_ORIGINAL_DST lookup, nftables redirect rules, identity construction) lives in internal/ateomegress so the two runtimes stay in sync. Agentgateway is used as the receiving proxy to prove that captured actor traffic reaches the tunnel endpoint.

Capture is enabled per actor rather than by a global switch: ate-api watches Gateways labeled ate.dev/egress-pep, resolves the best-matching PEP for each actor at resume time (actor-specific > atespace-scoped > global), sends the PEP address down through atelet to ateom, and records the resolved address on the Actor status(egress_pep_address) so the binding is observable to the user.
NOTE: Currently, an actor with no matching PEP resolves to an empty address and runs with capture off.

Notes:

• Tested the gvisor setup and microvm setups in kind. Can split these up into separate chunks to make it easier to review, but this shows the shared code.
• Agent identity is currently passed as unsigned metadata headers:
  • x-ate-actor-id
  • x-ate-atespace
  • x-ate-actor-template
  • x-ate-actor-template-namespace
  • x-ate-original-destination
  • x-ate-connect-authority
• Signed agent/actor identity is out of scope for this PR and should replace the current metadata headers approach once agent identity goes in [Feature] Actor Identity #124
• PEP selection is label-based on Gateways and trusts every labeled Gateway in the cluster, so Gateway RBAC is the security boundary (documented in docs/egress-capture.md). A first-class PEP binding/control plane is out of scope of this PR — agentgateway is just used to prove the proxy gets traffic from the tunnel.
• The PEP binding is a snapshot taken at resume: a running actor keeps its PEP until it is suspended/paused and resumed. ate-api resolves it from the Gateway's status.addresses (falling back to the <gateway>.<namespace>.svc convention) and only considers Programmed=True Gateways.
• Enforcement is best-effort/observability-grade for now: capture is not yet acknowledged back from ateom, so a worker running an older ateom image can ignore the PEP while the Actor still reports a binding (TODO left in ateom.proto). TLS stays end-to-end; agentgateway only routes the encrypted bytes after CONNECT.

[Eitan Yarmush (EItanya)]

Is there a time we would want no egress capture?

[Nina Polshakova (npolshakova)]

That's a good question- I think in general, egress capture should always be enabled but you can imagine scenarios where that doesn't make sense:

1. No PEP egress is setup. In this case I think traffic should still be captured by the ateom proxy, but not forwarded to any PEP
2. Flexibility for workloads using traffic we do not capture yet (UDP/QUIC, non-80/443 TCP if we don't want to do the classifier, etc.)

[Nina Polshakova (npolshakova)]

Mike Morris (@mikemorris) This is kind of related to the discussion we had in slack. Like you brought up, if we're expecting all traffic to always egress through a PEP, then the PEP registration can be a first-class API. Are there cases though where we don't want the traffic to get captured/egress through a PEP?

[yanavlasov]

I agree with this approach overall. However there are problems that will need to be addressed:

1. there is no support for a fleet of gateways. I think right now the first server will always be picked.
2. Using CONNECT is quite limiting. Multiplexing TCP connections into a single H/2 connections can lead to issues at scale. Inner connections will not be loadbalanced well across multiple gateways. Also it complicates gateway's business logic if it need to enforce policies other than coarse destination based.

To improve the story with gateway pluggability, I propose to add support for PROXY protocol. This will make it easier to plug other gateways, like KAN or Envoy AI Gateway. This can be a substrate egress configuration option, which will make config more complex, and can be added as a followup (I volunteer myself for this if needed).

Eventually and option for terminating TLS at the proxy and re-encrypting using mTLS with agent identity could be added as well. This will make it even easier to plug other gateways as PEPs.

[yanavlasov]

Eventually even the thin proxy should be pluggable. Operators will want telemetry, tracing, access logging from it. It will need to be able to load balance across a dynamically scaled fleet of PEPs, implement retries, health checking, etc. We do not want to recreate all this stuff in a custom go proxy. We can start with just a DNS name and go forwarder, but eventually it will need evolve and replicating functionality that already exists.