11# a3s-observer
22
3- Kernel-level ** eBPF** observability — and optional intervention — for AI agents. It turns
3+ Kernel-level ** eBPF observability — and optional intervention — for AI agents.** It turns
44syscalls and network events into agent-semantic telemetry (which agent ran which tool, made
55which LLM call, touched which files, reached which endpoint) with ** zero changes to the agent
6- and no per-language instrumentation** . The same kernel vantage point can also ** intervene** —
7- deny an agent's egress or file access from an external policy.
6+ and no per-language instrumentation** — and the same kernel vantage point can ** intervene** :
7+ deny an agent's egress, file access, or process execution from an external policy.
88
9- Built and validated on Linux 6.8 (Aya, no uprobes ). Observe-only by default; intervention is
10- opt-in and never affects the observe path.
9+ Built and validated on Linux 6.8 (Aya). Observe-only by default; every intervention is opt-in
10+ and isolated from the observe path.
1111
1212## Architecture
1313
1414A ** minimal core** — probes → loader → identity → correlation → export — with everything else a
15- swappable trait. Two paths share one kernel vantage point: ** observe** is always-on and passive;
16- ** intervene** is opt-in and isolated, so a policy mistake can never break observability.
15+ swappable trait. Two paths share one kernel vantage point: ** observe** is always-on and passive
16+ (tracepoints can't block); ** intervene** is opt-in (cgroup-BPF / fanotify), so a policy mistake
17+ can never break observability.
1718
1819```
19- AI agent + its tool subprocesses unmodified · any language
20- │
21- │ execve · connect · TLS ClientHello · DNS · openat · read / recv
22- ════════════╪══════════════════════════════════════════════ KERNEL · eBPF (no uprobes)
23- ▼
24- OBSERVE (passive, always on) INTERVENE (opt-in)
25- exec connect sni dns connect4 egress guard (cgroup-scoped)
26- llm-metrics file fanotify file guard
27- │ ▲
28- │ ring buffers │ allow / DENY → EPERM
29- ════════════╪═════════════════════════════════════════════╪═══════════ USERSPACE
30- ▼ │
31- a3s-observer-collector (Aya) a3s-observer-enforce · fileguard
32- · identity k8s pod / proc / comm ▲
33- · correlate (pid,fd) → peer │ reads
34- · export NDJSON / human log external policy file
35- │ (your controller · OPA · a script)
36- ▼
37- OTel Collector → your backend
20+ AI agent + its tool subprocesses unmodified · any language
21+ │
22+ │ execve · connect · TLS ClientHello · DNS · openat · read/recv · SSL_*
23+ ══════════╪═══════════════════════════════════════════════ KERNEL (eBPF + fanotify)
24+ ▼
25+ OBSERVE (passive, always-on) INTERVENE (opt-in, external policy)
26+ exec connect sni dns enforce → cgroup/connect4: deny egress
27+ llm-metrics file* ssl-content* fileguard → fanotify: deny open + exec
28+ │
29+ │ ring buffers
30+ ══════════╪═══════════════════════════════════════════════ USERSPACE
31+ ▼
32+ a3s-observer-collector (Aya loader)
33+ identity (k8s pod / proc / comm) · correlate (pid,fd)→peer · export NDJSON
34+ │
35+ ▼
36+ OTel Collector → your backend * opt-in: A3S_OBSERVER_FILES / _SSL
3837```
3938
40- The four core pieces — ** probes** (language-agnostic kernel hooks), ** identity** (attribute an
41- event to an agent), ** correlation** (fuse provider + endpoint), ** export** (NDJSON) — plus the
42- opt-in ** guards** , are detailed below.
43-
4439## Observe — who / what / where
4540
46- One event answers ** who / what / where ** : who (process or k8s pod), what (tool, file, or LLM
47- provider + bytes/latency/ TTFT), where (peer IP / hostname).
41+ One event answers ** who** (process or k8s pod) / ** what** (tool, file, LLM provider + bytes /
42+ latency / TTFT, or plaintext) / ** where** (peer IP / hostname).
4843
49- | probe | kernel hook | signal |
44+ | signal | kernel hook | event |
5045| ---| ---| ---|
51- | ` exec ` | ` sys_enter_execve ` | tools / subprocesses (argv, comm, uid) |
52- | ` connect ` | ` sys_enter_connect ` | peer IP: port |
53- | ` sni ` | TLS ClientHello (plaintext ` server_name ` ) | LLM provider + endpoint |
54- | ` dns ` | ` sendto ` / ` sendmsg ` / ` sendmmsg ` to :53 | resolved hostnames |
55- | LLM metrics | per-socket ` read ` / ` recv ` + ` close ` | req/resp wire bytes, latency, TTFT |
56- | ` file ` | ` sys_enter_openat ` (write opens) | files written — ** opt-in ** (` A3S_OBSERVER_FILES=1 ` ; high-volume ) |
57- | ` ssl ` content | uprobes on OpenSSL ` SSL_write ` / ` SSL_read ` | request/response ** plaintext** (prompt/completion) — ** opt-in ** ( ` A3S_OBSERVER_SSL=1 ` ; OpenSSL-only, not language-agnostic ) |
46+ | ` exec ` | ` sys_enter_execve ` | ` ToolExec ` — tool / subprocess (argv, comm, uid) |
47+ | ` connect ` | ` sys_enter_connect ` | ` Egress ` — peer IP: port |
48+ | ` sni ` | TLS ClientHello (plaintext ` server_name ` ) | LLM ** provider** + endpoint |
49+ | ` dns ` | ` sendto ` / ` sendmsg ` / ` sendmmsg ` to :53 | ` Dns ` — resolved hostname |
50+ | llm metrics | per-socket ` read ` / ` recv ` + ` close ` | ` LlmCall ` — req/resp wire bytes, latency, TTFT |
51+ | ` file ` \* | ` sys_enter_openat ` (write opens) | ` FileAccess ` — files written (` A3S_OBSERVER_FILES=1 ` ) |
52+ | ` ssl ` \* | OpenSSL ` SSL_write ` / ` SSL_read ` uprobes | ` SslContent ` — request/response plaintext ( ` A3S_OBSERVER_SSL=1 ` ) |
5853
5954Userspace enriches each event with ** identity** (k8s cgroup→pod, ` /proc ` comm+ppid, or an
60- in-kernel ` comm ` fallback for short-lived processes) and a ` (pid,fd)→peer ` ** correlation** ,
61- then exports ** NDJSON** (or a human-readable log).
55+ in-kernel ` comm ` fallback for short-lived processes), a ` (pid,fd)→peer ` ** correlation** , and
56+ ** provider ** classification (SNI → 15 LLM providers); then exports ** NDJSON** (or a human log).
6257
63- ## Intervene (opt-in)
58+ ## Intervene — egress / file / exec (opt-in)
6459
65- The same vantage point can enforce an ** external** policy: the policy lives outside the binary
66- (a plain file any controller can write), and the kernel asks a guard allow/deny per action.
67- The observe-only core is untouched.
60+ The same vantage point enforces an ** external policy ** — a plain file any controller writes; the
61+ kernel asks a guard allow/deny per action. The observe-only core is untouched. Both guards are
62+ ** hot-reloaded ** and ** KVM-validated ** (a denied action returns ` EPERM ` ):
6863
6964| guard | mechanism | denies |
7065| ---| ---| ---|
71- | ` a3s-observer-enforce ` | ` cgroup/connect4 ` eBPF | egress to policy IPs/hosts — ** cgroup-scoped** , fail-open |
72- | ` a3s-observer-fileguard ` | fanotify ` FAN_OPEN_PERM ` + ` FAN_OPEN_EXEC_PERM ` | ` open() ` ** and** ` exec ` of policy-listed files; policy ** hot-reloaded ** |
66+ | ` a3s-observer-enforce ` | eBPF ` cgroup/connect4 ` | ` connect() ` to policy IPs/hosts — cgroup-scoped, fail-open, DNS-re-resolved |
67+ | ` a3s-observer-fileguard ` | fanotify ` FAN_OPEN_PERM ` + ` FAN_OPEN_EXEC_PERM ` | ` open() ` ** and** ` exec ` of policy-listed files |
7368
74- Both KVM-validated: a denied connect / file-open returns ` EPERM ` , everything else is
75- untouched. Drive it in-process (the ` Policy ` trait) or out-of-process — ` scripts/example-controller.py `
69+ Drive it in-process (the ` Policy ` trait) or out-of-process — ` scripts/example-controller.py `
7670turns observed events into a deny-list. See [ ` docs/enforcement.md ` ] ( docs/enforcement.md ) .
7771
7872## Why eBPF, and the boundary
7973
80- - ** Zero-instrumentation, language-agnostic** — observe or guard any agent
81- (Python/Node/Go/Rust) without touching its code, including its tool subprocesses.
82- - ** Sees what the app won't report** — real execs, file I/O, network egress.
83- - ** Kernel hooks only in the always-on core, no uprobes.** The deliberate trade-off: the core
84- gives ** no LLM prompt / completion content** . That content * is* available via an ** opt-in**
85- OpenSSL uprobe extension (` A3S_OBSERVER_SSL=1 ` ) that captures ` SSL_write ` /` SSL_read `
86- plaintext — kept out of the core because a uprobe binds to a library symbol, so it's OpenSSL
87- only, not language-agnostic. (ECH will eventually hide SNI → fall back to IP/DNS.)
88- - ** a3s-box:** a box is a separate guest kernel, so host-side eBPF sees box ** egress** (it
89- flows through the host net path) but not in-guest exec/file — those need an in-guest
90- collector (phase 2).
74+ - ** Zero-instrumentation, language-agnostic** — observe or guard any agent (Python/Node/Go/Rust)
75+ without touching its code, including its tool subprocesses.
76+ - ** Kernel hooks only in the always-on core, no uprobes** — so the core gives ** no LLM
77+ prompt/completion content** . That content is available via an ** opt-in** OpenSSL uprobe
78+ extension (` A3S_OBSERVER_SSL=1 ` ) — OpenSSL only (Python/Node/curl …, not Go ` crypto/tls ` ),
79+ kept out of the universal core because a uprobe binds to a library symbol. (ECH will
80+ eventually hide SNI → fall back to IP/DNS.)
81+ - ** a3s-box** — a box is a separate guest kernel, so host-side eBPF sees box ** egress** (it
82+ flows through the host net path) but not in-guest exec/file — those need an in-guest collector
83+ (phase 2).
9184
9285## Build & run
9386
94- eBPF needs nightly + ` rust-src ` + [ ` bpf-linker ` ] ( https://github.com/aya-rs/bpf-linker )
95- (it borrows rustc's bundled LLVM — no system LLVM required):
87+ eBPF needs nightly + ` rust-src ` + [ ` bpf-linker ` ] ( https://github.com/aya-rs/bpf-linker ) (it
88+ borrows rustc's bundled LLVM — no system LLVM required):
9689
9790``` bash
9891rustup toolchain install nightly --component rust-src
@@ -101,38 +94,47 @@ cargo build --release -p a3s-observer-collector # build.rs compiles + links t
10194
10295sudo ./target/release/a3s-observer-collector # human-readable log
10396A3S_OBSERVER_JSON=1 sudo -E ./target/release/a3s-observer-collector # NDJSON
104- A3S_OBSERVER_FILES=1 A3S_OBSERVER_JSON=1 sudo -E ./target/release/a3s-observer-collector # + file writes
10597```
10698
107- Linux only; needs root (CAP_BPF + CAP_PERFMON).
99+ Linux only; needs root (CAP_BPF + CAP_PERFMON). Env knobs: ` A3S_OBSERVER_JSON ` (NDJSON),
100+ ` A3S_OBSERVER_FILES ` (file writes — high-volume), ` A3S_OBSERVER_SSL ` (OpenSSL content),
101+ ` A3S_OBSERVER_HEARTBEAT ` (liveness file path).
102+
103+ Opt-in enforcement — run against an agent's cgroup and/or a deny-list file:
104+
105+ ``` bash
106+ sudo ./target/release/a3s-observer-enforce /sys/fs/cgroup/< agent> egress-deny.txt
107+ sudo ./target/release/a3s-observer-fileguard file-exec-deny.txt
108+ ```
108109
109110## Deploy
110111
111- a3s-observer captures and emits NDJSON; shipping it (batch / retry / route to a backend) is
112- the OpenTelemetry Collector's job, so in-process OTLP is intentionally ** not** built:
112+ a3s-observer emits NDJSON; shipping it (batch / retry / route to a backend) is the OpenTelemetry
113+ Collector's job, so in-process OTLP is intentionally ** not** built:
113114
114115```
115116a3s-observer → NDJSON → OTel Collector (filelog → OTLP) → your backend
116117```
117118
118- - Collector config: [ ` deploy/otel-collector.yaml ` ] ( deploy/otel-collector.yaml ) . Every event
119- is one valid-JSON line, so it also drops into vector / Loki / ` jq ` .
120- - ** Kubernetes:** CI publishes ` ghcr.io/a3s-lab/observer:<tag> ` on each tag
121- ([ ` image.yml ` ] ( .github/workflows/image.yml ) from [ ` deploy/Dockerfile ` ] ( deploy/Dockerfile ) );
122- deploy [ ` deploy/daemonset.yaml ` ] ( deploy/daemonset.yaml ) (NDJSON to stdout, pod identity from
123- ` /proc/<pid>/cgroup ` — no k8s API/RBAC). Mirror the image to a cluster-local registry for
124- nodes that can't reach ghcr.io.
119+ - Collector config: [ ` deploy/otel-collector.yaml ` ] ( deploy/otel-collector.yaml ) (` memory_limiter `
120+ + a retrying sending queue). Every event is one valid-JSON line — also drops into vector /
121+ Loki / ` jq ` .
122+ - ** Kubernetes:** CI publishes ` ghcr.io/a3s-lab/observer:<tag> ` (Trivy-scanned, cosign-signed,
123+ with SBOM + SLSA provenance); deploy [ ` deploy/daemonset.yaml ` ] ( deploy/daemonset.yaml ) (NDJSON
124+ to stdout, pod identity from ` /proc/<pid>/cgroup ` — no k8s API/RBAC; liveness probe,
125+ ` system-node-critical ` ). Mirror the image to a cluster-local registry for nodes that can't
126+ reach ghcr.io.
125127
126128## Workspace
127129
128130| crate | role |
129131| ---| ---|
130132| ` a3s-observer ` | contracts + data model (` IdentityResolver ` / ` ServiceClassifier ` / ` Exporter ` / ` Policy ` ) — host-buildable |
131133| ` a3s-observer-common ` | ` no_std ` types shared with the eBPF probes |
132- | ` a3s-observer-ebpf ` | the probes + egress guard, compiled to BPF bytecode |
134+ | ` a3s-observer-ebpf ` | probes + the ` connect4 ` egress guard, compiled to BPF bytecode |
133135| ` a3s-observer-collector ` | loader, correlation, export; plus the ` enforce ` and ` fileguard ` binaries |
134136
135- Rust + [ Aya] ( https://aya-rs.dev ) .
137+ Rust + [ Aya] ( https://aya-rs.dev ) . Validated on Linux 6.8.
136138
137139## Security
138140
0 commit comments