Skip to content

Commit 2eecee1

Browse files
committed
update docs
Signed-off-by: npolshakova <nina.polshakova@solo.io>
1 parent 63301ae commit 2eecee1

2 files changed

Lines changed: 51 additions & 55 deletions

File tree

docs/egress-capture.md

Lines changed: 40 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -10,29 +10,30 @@ by default.
1010
Egress capture has no global on/off switch. ate-api watches Gateways labeled
1111
`ate.dev/egress-pep`. On actor resume, ate-api picks the best matching PEP
1212
Gateway for that actor and sends one optional PEP address to ateom. An empty PEP
13-
address means no redirect, so capture is enabled per actor purely by whether a
14-
matching labeled Gateway exists. The reusable capture core lives in
15-
`internal/egress`:
13+
address means no redirect, so capture is enabled per actor only when a matching
14+
programmed labeled Gateway with an HTTP listener resolves. The reusable capture
15+
core lives in `internal/egress`:
1616
it owns capture listeners, authority derivation, CONNECT tunnel transports, and
1717
byte proxying. The runtime-specific `ateom` egress proxy setup supplies the
1818
original-destination lookup and packet-capture rules.
1919

20-
The current gVisor implementation starts a local capture listener and installs
21-
actor-network redirects for TCP/80 and TCP/443. From the actor's point of view
22-
it still opens a normal HTTP or HTTPS connection to the original destination.
23-
MicroVM or future hypervisor implementations should reuse
24-
`internal/egress` for the local listener, authority derivation, tunnel
25-
transport, and byte proxying. Each runtime still provides its own egress proxy
26-
setup for redirecting actor traffic and recovering the original destination.
20+
The current gVisor and MicroVM implementations start a local capture listener
21+
and install actor-network redirects for TCP egress. From the actor's point of
22+
view it still opens a normal TCP connection to the original destination. Future
23+
hypervisor implementations should reuse `internal/egress` for the local
24+
listener, authority derivation, tunnel transport, and byte proxying. Each
25+
runtime still provides its own egress proxy setup for redirecting actor traffic
26+
and recovering the original destination.
2727

2828
The redirected connection lands on `ateom`, which records the original
2929
destination and derives a stable CONNECT authority from the first bytes of the
3030
actor connection:
3131

3232
| Actor traffic | Authority source | Example CONNECT authority |
3333
| --- | --- | --- |
34-
| HTTPS / TCP 443 | TLS ClientHello SNI | `httpbin.org:443` |
35-
| Plaintext HTTP / TCP 80 | HTTP `Host` header | `example.com:80` |
34+
| HTTPS / any TCP port | TLS ClientHello SNI + original destination port | `httpbin.org:443` |
35+
| Plaintext HTTP / any TCP port | HTTP `Host` header, defaulting to original destination port when the header has no port | `example.com:80` |
36+
| Other TCP | Original destination address | `203.0.113.10:2222` |
3637

3738
The shared capture core then opens a plaintext HTTP/2 CONNECT stream to the PEP
3839
address selected by ate-api. Only Gateways with condition `Programmed=True` are
@@ -45,11 +46,12 @@ stays pending). The port is the Gateway's HTTP listener port. Agentgateway maps
4546
the CONNECT authority to its configured TCP listener and routes the tunnel to a
4647
Kubernetes Service backed by an EndpointSlice.
4748

48-
The demo setup configures only `httpbin.org:443` for egress.
49-
Other hosts or plaintext HTTP destinations need their own agentgateway
50-
Service, EndpointSlice, listener, and route. For HTTPS, TLS is still end-to-end
51-
between the actor and the external service; agentgateway only routes the
52-
encrypted bytes after CONNECT succeeds.
49+
The demo setup configures only `httpbin.org:443` for egress. Any other CONNECT
50+
authority, including plaintext HTTP destinations or fallback original IP:port
51+
authorities, needs its own matching agentgateway Service, EndpointSlice,
52+
listener, and route. For HTTPS, TLS is still end-to-end between the actor and
53+
the external service; agentgateway only routes the encrypted bytes after
54+
CONNECT succeeds.
5355

5456
### Selecting a PEP for an actor
5557

@@ -73,7 +75,7 @@ highest-precedence match (`resolveEgressPEPAddress` in
7375
| Situation | Result |
7476
| --- | --- |
7577
| Multiple candidates tied at the top score | Lowest `(namespace, name)` wins; the others are **silently ignored** |
76-
| No labeled candidate | Empty PEP address → no redirect, capture off |
78+
| No matching programmed candidate | Empty PEP address → no redirect, capture off |
7779

7880
Don't deploy multiple PEPs at the same tier for the same actor — use the
7981
scoping labels to raise the intended one's tier instead of relying on name
@@ -88,8 +90,8 @@ label a Gateway can:
8890

8991
- **Intercept actor egress**: copy an actor's `ate.dev/atespace` +
9092
`ate.dev/actor` labels onto their own Gateway to out-score its real PEP.
91-
- **Block all resumes**: label a Gateway with no HTTP listener (broken PEP
92-
config fails resolution cluster-wide by design).
93+
- **Block resumes**: label a programmed Gateway with no HTTP listener (broken
94+
PEP config fails PEP resolution by design).
9395

9496
Substrate deliberately does not second-guess labeled Gateways. If Gateway RBAC
9597
can't be that strict, scope the watch to an allowlist of PEP namespaces in
@@ -163,7 +165,7 @@ sequenceDiagram
163165
rect rgb(255, 248, 225)
164166
Note over OM,EXT: Actor egress connection
165167
OM->>CAP: Actor conn (nftables redirect) to httpbin.org:443
166-
CAP->>CAP: Get original destination<br/>sniff SNI or Host
168+
CAP->>CAP: Get original destination<br/>classify SNI, Host, or original dst
167169
CAP->>PEP: HTTP/2 CONNECT httpbin.org:443<br/>with actor metadata
168170
PEP->>EXT: Route via TCPRoute
169171
Note over CAP,EXT: TLS remains end-to-end<br/>PEP routes encrypted bytes only
@@ -218,8 +220,9 @@ system. No fixed PEP address is configured; ate-api derives the address from the
218220
labeled Gateway's HTTP listener.
219221

220222
The install script resolves `httpbin.org` during install and creates the
221-
`httpbin-egress` Service and EndpointSlice for those IPs. `ateom` derives the
222-
CONNECT authority from SNI for this HTTPS demo.
223+
`httpbin-egress` Service and EndpointSlice for those IPs. For the default HTTPS
224+
demo request, `ateom` derives the CONNECT authority from TLS SNI and the
225+
original destination port.
223226

224227
Verify the static agentgateway resources and PEP label:
225228

@@ -299,8 +302,9 @@ curl -i -X POST --get \
299302
"http://localhost:8000"
300303
```
301304

302-
Do not use this query parameter for a different host unless you also update the
303-
agentgateway route. `ateom` will derive the new host from SNI, but the demo
305+
Do not use this query parameter for a different host or port unless you also
306+
update the agentgateway route. `ateom` will derive the new CONNECT authority
307+
from TLS SNI, the HTTP `Host` header, or the original destination, but the demo
304308
agentgateway config only routes `httpbin.org:443`.
305309

306310
## Verify capture was installed
@@ -490,9 +494,9 @@ EOF
490494
http://localhost:8000
491495
```
492496

493-
The authoritative signal is the ateom capture log, which names the PEP the
494-
actor actually tunnelled through (see "Verify capture was installed" for how
495-
to find the worker pod):
497+
The authoritative signal is the ateom capture log, which names the PEP
498+
configured for the actor's current activation (see "Verify capture was
499+
installed" for how to find the worker pod):
496500

497501
```bash
498502
kubectl logs -n "${ateom_ns}" "${ateom_pod}" -c ateom | grep "Started actor egress capture listener"
@@ -593,21 +597,12 @@ PEP address, so `egressPepAddress` on the actor can report a binding the
593597
sandbox does not enforce. If capture logs are missing despite a resolved PEP,
594598
check the WorkerPool's ateom image version.
595599

596-
If the capture listener logs are missing, confirm that the actor is running on a
597-
fresh worker pod created after egress was enabled:
600+
If the egress request fails after changing the `url` host or port, remember that
601+
this demo only configures agentgateway for `httpbin.org:443`. Add matching static
602+
agentgateway backend resources for the CONNECT authority that `ateom` will send:
598603

599-
```bash
600-
kubectl ate get actor my-egress-1 -a demo
601-
kubectl get pods -n ate-demo-egress -l ate.dev/worker-pool=egress
602-
```
603-
604-
If the egress request fails after changing the `url` host, remember that this demo only
605-
configures agentgateway for `httpbin.org:443`. Add matching static agentgateway
606-
backend resources for the new destination:
607-
608-
- HTTPS: Service, EndpointSlice, TCP listener on `443`, and TCPRoute.
609-
- Plaintext HTTP: Service, EndpointSlice, TCP listener on `80`, and TCPRoute.
610-
611-
Traffic without SNI or a plaintext HTTP `Host` header falls back to the captured
612-
original destination IP and port, which requires matching agentgateway routing
613-
for that address.
604+
- HTTPS: SNI plus the original destination port, for example `example.com:443`.
605+
- Plaintext HTTP: `Host` header authority, defaulting to the original
606+
destination port when the header has no port, for example `example.com:80`.
607+
- Other TCP: captured original destination IP and port, for example
608+
`203.0.113.10:2222`.

hack/install-ate.sh

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -571,8 +571,8 @@ deploy_atenet() {
571571

572572
# get_actor_status echoes the actor's status enum (e.g. STATUS_SUSPENDED).
573573
get_actor_status() {
574-
local actor_id="$1"
575-
local atespace="$2"
574+
local atespace="$1"
575+
local actor_id="$2"
576576
local json
577577

578578
if ! json=$(run_kubectl_ate get actor "${actor_id}" -a "${atespace}" -o json 2>/dev/null); then
@@ -584,14 +584,14 @@ get_actor_status() {
584584
# prepare_actor_for_delete suspends (or resumes then suspends) until DeleteActor
585585
# is allowed. Actors must be STATUS_SUSPENDED before deletion.
586586
prepare_actor_for_delete() {
587-
local actor_id="$1"
588-
local atespace="$2"
587+
local atespace="$1"
588+
local actor_id="$2"
589589
local timeout_secs="${3:-120}"
590590
local deadline=$((SECONDS + timeout_secs))
591591
local status
592592

593593
while ((SECONDS < deadline)); do
594-
if ! status=$(get_actor_status "${actor_id}" "${atespace}"); then
594+
if ! status=$(get_actor_status "${atespace}" "${actor_id}"); then
595595
return 0
596596
fi
597597

@@ -646,21 +646,22 @@ delete_demo_actors() {
646646
return 0
647647
fi
648648

649-
local ns tmpl atespace actor_id
649+
local ns tmpl actor_ref atespace actor_id
650650
while (($# > 0)); do
651651
ns="$1"
652652
tmpl="$2"
653653
shift 2
654654

655655
log_step "Deleting actors for ${ns}/${tmpl}"
656656
while IFS=$'\t' read -r atespace actor_id; do
657-
[[ -z "${actor_id}" ]] && continue
658-
log_step " preparing actor ${atespace}/${actor_id} for delete"
659-
prepare_actor_for_delete "${actor_id}" "${atespace}"
657+
[[ -z "${atespace}" || -z "${actor_id}" ]] && continue
658+
actor_ref="${atespace}/${actor_id}"
659+
log_step " preparing actor ${actor_ref} for delete"
660+
prepare_actor_for_delete "${atespace}" "${actor_id}"
660661
run_kubectl_ate delete actor "${actor_id}" -a "${atespace}"
661662
done < <(
662663
jq -r --arg ns "${ns}" --arg tmpl "${tmpl}" \
663-
'.actors[]? | select(.actorTemplateNamespace == $ns and .actorTemplateName == $tmpl) | "\(.atespace)\t\(.actorId)"' \
664+
'.actors[]? | select(.actorTemplateNamespace == $ns and .actorTemplateName == $tmpl) | [.atespace, .actorId] | @tsv' \
664665
<<<"${actors_json}"
665666
)
666667
done

0 commit comments

Comments
 (0)