Merge pull request #35 from A3S-Lab/chore/sync-sentry-decoupled-l3-20260720

docs: sync Sentry decoupled L3 safeguards

Author: ZhiXiao-Lin

README.md

@@ -65,7 +65,7 @@ is Rust.
 | [a3s-runtime](crates/runtime/) | 0.2.0 | Provider-neutral execution contract and Runtime client. |
 | [a3s-box](crates/box/) | 3.0.5 | Docker-like MicroVM runtime for Linux OCI workloads. |
 | [a3s-observer](crates/observer/) | 0.11.0 | eBPF observability for LLM calls, tools, files, and egress. |
-| [a3s-sentry](crates/sentry/) | 0.7.0 | Tiered runtime security control. |
+| [a3s-sentry](crates/sentry/) | 0.7.0 | Tiered runtime security control with staged L3 dispatch and incomplete-evidence safeguards. |
 | [a3s-boot](crates/boot/) | 0.1.1 | Nest-inspired modular service framework for Rust APIs. |
 | [a3s-gateway](crates/gateway/) | 1.0.12 | Reverse proxy, routing, middleware, streaming, and scale-to-zero. |
 | [a3s-power](crates/power/) | 0.4.2 | Privacy-preserving LLM inference for TEE environments. |

apps/docs/content/docs/cn/sentry/configuration.mdx

@@ -128,10 +128,16 @@ rules = [
 
 ## Fail Mode
 
-Sentry 默认 fail-open。L1 escalate 后如果没有更深 tier 给出结论，该事件会 allow。慢 tier 导致 queue overload 时也遵循同样姿态。
+Sentry 默认 fail-open。L1 escalate 后如果没有更深 tier 给出结论，该事件会 allow。完整 evidence 的 escalation 被满 worker queue 拒绝时也遵循同样姿态。
 
 Fail-closed 会把 unresolved escalation 和 overload degradation 变成 block。只有在 L2/L3 容量和 operational blast radius 都明确时，才适合安全优先 workload。
 
+Incomplete `ToolExec` evidence 刻意采用不同语义。危险的已捕获 prefix 仍可
+block，但模糊的 truncated 或 incompletely reassembled command 在两种 fail mode
+下都会保留为 L1 escalation。Bundled daemon 会 audit 这个 unresolved result，
+包括 worker overload 场景；durable external L3 dispatch 应由 staged SDK
+consumer 负责。
+
 ## Deny Sinks
 
 Deny file 是 append-only operational state。Sentry 写入 egress、file、exec target；Observer guard 执行。缺少 deny sink 也是合法的，此时进程只用于 judgment 或 dry-run analysis。
@@ -145,4 +151,7 @@ Exec deny 以路径为中心。对裸 binary name 的 deny action 可能无法
 - `GET /metrics`，包含 judged events、blocks、overload degradation、enforce failures 等 counters。
 - `GET /healthz`，daemon 存活时返回 `200 ok`。
 
-应重点告警 `sentry_overload_degraded_total` 和 `sentry_enforce_failed_total`；它们都表示预期 block 可能没有落地。
+应重点告警 `sentry_overload_degraded_total` 和
+`sentry_enforce_failed_total`。前者统计被满 worker queue 拒绝的
+escalation：完整 evidence event 使用 fail mode，incomplete command evidence
+保持 unresolved；后者表示 deny-file write 失败。

apps/docs/content/docs/cn/sentry/deployment-runbook.mdx

@@ -56,7 +56,7 @@ Manifest 默认启用 dry-run，让首次 rollout 先 shadow policy decisions，
 
 | Metric | 为什么重要 | 第一反应 |
 | --- | --- | --- |
-| `sentry_overload_degraded_total` | Worker queue 满了，escalation 只能落到 fail mode。 | 增加 workers 或 queue，减少 escalation volume，调慢 tier，或有意识地选择 fail-closed。 |
+| `sentry_overload_degraded_total` | Escalation 被满 worker queue 拒绝。完整 evidence 使用 fail mode；incomplete command evidence 保持 unresolved。 | 增加 workers 或 queue，减少 escalation volume，调慢 tier，或为完整 evidence overload 有意识地选择 fail-closed。 |
 | `sentry_enforce_failed_total` | 某个 block 需要写 deny file，但写入失败。 | 检查 volume space、mount mode、file permissions 和 configured paths。 |
 
 `sentry_events_total` 和 `sentry_blocked_total` 用于观察流量与 block ratio trend。`/healthz` 只说明 daemon 存活，不证明每个 deny 都已落地。
@@ -73,7 +73,9 @@ L1 在每个 event 上运行并保持低成本。Escalation 进入 bounded worke
 - `A3S_SENTRY_AGENT_TIMEOUT`：L3 A3S Code investigation timeout。
 - `A3S_SENTRY_SPECULATE`：高 severity event 值得并行 L2/L3 时启用。
 
-如果 overload 上升，说明当前部署对 event mix 来说资源不足，或 deeper tier 对 escalation rate 来说太慢。
+如果 overload 上升，说明当前部署对 event mix 来说资源不足，或 deeper tier
+对 escalation rate 来说太慢。完整 evidence event 按 fail mode 解决；
+incomplete `ToolExec` evidence 保持为 audited L1 escalation。
 
 ## L2 Or L3 Outage
 
@@ -97,5 +99,6 @@ Sentry restart 期间，Observer guards 仍可继续执行已有 deny files。
 
 - Observer-event Sentry 是 reactive。它能 block 下一次匹配动作，不一定能 block 产生 event 的那一次动作。
 - Inline Gateway inspection 是路由到 proxy 的 LLM/MCP traffic 的 pre-forward path。
+- Bundled daemon 会 audit incomplete-evidence escalation，但不提供 durable external L3 queue；staged SDK consumer 必须负责持久化和 dispatch。
 - Deny 是 coarse 且 node-local 的：IP 或 executable-path block 可能影响同 node 上其它 workload。
 - L2 和 L3 会处理 attacker-influenced content。确定性的 L1 和 kernel guard enforcement 仍应作为底线。

apps/docs/content/docs/cn/sentry/operations.mdx

@@ -27,12 +27,13 @@ Guard 会热加载这些文件。Deny files 是运行态，不是 source-of-trut
 
 | Metric | 含义 |
 | --- | --- |
-| `sentry_overload_degraded_total` | Escalation 无法处理，落入配置的 fail mode。 |
+| `sentry_overload_degraded_total` | Escalation 被满 worker queue 拒绝。完整 evidence 使用 fail mode；incomplete command evidence 保持 unresolved。 |
 | `sentry_enforce_failed_total` | Decision 需要 enforce，但写 deny record 失败。 |
 | `sentry_blocked_total` | 按 tier 和 event family 统计 block。 |
 | `sentry_events_total` | 总判断事件数。 |
 
-在 fail-open 部署中，overload degradation 或 enforce failure 都意味着某次 block 可能没有落地。
+任何非零 overload rate 都表示 worker pool 资源不足。在 fail-open 部署中，
+完整 evidence 的 overflow 或 enforce failure 意味着某次 block 可能没有落地。
 
 ## Queue 和 Timeout 调优
 
@@ -44,8 +45,14 @@ L2 和 L3 在 ingest thread 之外的 worker pools 运行。需要调：
 - 深度 A3S Code investigation 的 L3 timeout。
 - 高 severity event 是否值得 L2/L3 parallel 的 speculation threshold。
 
+完整 evidence 的 queue overflow 会按配置 fail mode 解决。Incomplete
+`ToolExec` evidence 即使在 overload 时也保持为 audited L1 escalation。
+
 ## 边界
 
 - Observer-event Sentry 是 reactive；它可能阻断下一步 action，而不是产生 event 的那一步。
+- Bundled daemon 会 audit incomplete-evidence escalation，但不提供 durable
+  external L3 queue。需要 deeper work 时，staged SDK consumer 必须负责持久化
+  和 dispatch。
 - Deny 可能比较粗：binary path 与 IP-based enforcement 可能影响同节点上的多个进程。
 - L2/L3 会读取攻击者影响的内容。确定性的 L1 底线和外部 kernel guard 仍然重要。

apps/docs/content/docs/cn/sentry/pipeline.mdx

@@ -1,6 +1,6 @@
 ---
 title: Pipeline
-description: L1 rules、L2 LLM、L3 A3S Code investigation、SAE signals、fail posture 与 speculative parallelism。
+description: L1 rules、L2 LLM、staged L3 dispatch、incomplete-evidence safeguards、SAE signals、fail posture 与 speculation。
 ---
 
 # Sentry Pipeline
@@ -34,13 +34,24 @@ ACL policy 文件提供，并支持热加载。
 
 L1 是快速 triage 层，不是 sandbox。它刻意保守：遇到模糊情况就 escalate，而不是猜测。
 
+对于 `ToolExec`，Observer 会在无法捕获完整 command 时报告
+`argv_truncated` 或 `argv_incomplete`。危险的已捕获 prefix 仍可直接 block；
+其它不明确的 incomplete command 会保留为 L1 escalation。进程内 L2/L3 judge
+不会猜测缺失 suffix 是 benign。
+
 ## L2 和 L3
 
 L2 是事件分类器，适合 false-positive reduction 和快速语义判断。
 
 L3 是真实 A3S Code agent。它加载安全技能，考虑 actor、观察到的 event、可能 attack chain
 和 blast radius，然后返回结构化 verdict。它更慢，但更深。
 
+需要把快速判断与深度 investigation 分离的产品可以调用
+`Pipeline::evaluate_through_l2`。它返回 L1 decision、可选 L2 decision、
+effective decision、stage status 和 escalation cause，且不会调用 L3、
+应用 `fail_closed` 或启动 speculative L3。`Escalated` result 是需要持久化并
+dispatch 到 durable external L3 worker 的工作，不是 allow。
+
 ## Speculation
 
 默认情况下，L3 只在 L2 escalate 后启动。开启 speculative parallelism 后，高 severity
@@ -53,3 +64,6 @@ Sentry 默认 fail-open：无法解决的 escalation 会 allow。只有在部署
 或采用能承受 unknown blocking 的 allow-list 姿态时，才应启用 fail-closed。
 
 Rules-only fail-open 模式能执行直接 `block` 的规则，但不会执行只 `escalate` 的规则。
+Incomplete `ToolExec` evidence 是例外：无论 fail mode 如何，它都保留为
+unresolved L1 escalation。Bundled daemon 会 audit 这个 result，但不提供
+durable external L3 queue。

apps/docs/content/docs/cn/sentry/sdk.mdx

@@ -23,15 +23,28 @@ enforced = sentry.evaluate_and_enforce(tool_exec(2, ["/usr/bin/ncat", "host", "4
 
 ## TypeScript
 
-TypeScript SDK 通过 native package 暴露同一个 judge：
+TypeScript SDK 通过 native package 暴露同一个 judge。Staged method 在 N-API
+worker pool 上运行并返回 typed promise：
 
 ```ts
-import { Sentry, egress } from '@a3s-lab/sentry';
+import { Sentry, egress, fileAccess } from '@a3s-lab/sentry';
 
 const sentry = Sentry.create('sentry.acl');
 const decision = sentry.evaluate(egress(1, '169.254.169.254', 80));
+
+const fast = await sentry.evaluateThroughL2(
+  fileAccess(2, '/home/agent/.aws/credentials', false),
+);
+if (fast.stageStatus === 'escalated') {
+  await durableL3Queue.send(fast);
+}
 ```
 
+`evaluateThroughL2` 不会调用 L3，也不会通过 `fail_closed` 解决 unresolved
+escalation。它的 `ThroughL2Result` 包含 `l1Decision`、可选
+`l2Decision`、`effectiveDecision`、`stageStatus` 和可选
+`escalationCause`。
+
 ## 统一 ACL 配置
 
 SDK 配置可以描述：
@@ -48,3 +61,7 @@ SDK 配置可以描述：
 
 Rust host 可以直接用 `LiveRules`、`Pipeline` 和 judge implementations 构建 pipeline。
 当产品已经拥有 event source 和 enforcement sinks 时，这是最低层集成路径。
+
+当产品还拥有 durable L3 queue 时，使用 `Pipeline::evaluate_through_l2`。
+这个 method 在 fast tiers 后停止并保留 unresolved escalation；caller 必须
+持久化并 dispatch 该 result，不能把它当作 allow。

apps/docs/content/docs/en/sentry/configuration.mdx

@@ -128,10 +128,16 @@ For committed samples, prefer environment substitution in the host application o
 
 ## Fail Mode
 
-Sentry defaults to fail-open. If L1 escalates and no deeper tier resolves the event, the event is allowed. The same posture applies when a slow tier overloads the queue.
+Sentry defaults to fail-open. If L1 escalates and no deeper tier resolves the event, the event is allowed. Complete-evidence escalations rejected by a full worker queue use the same posture.
 
 Fail-closed changes unresolved escalation and overload degradation into blocks. Use it for safety-first workloads only when L2/L3 capacity and the operational blast radius are understood.
 
+Incomplete `ToolExec` evidence is deliberately different. A dangerous captured
+prefix can still block, but an ambiguous truncated or incompletely reassembled
+command remains an L1 escalation in both fail modes. The bundled daemon audits
+the unresolved result, including during worker overload; durable external L3
+dispatch belongs to a staged SDK consumer.
+
 ## Deny Sinks
 
 Deny files are append-only operational state. Sentry writes egress, file, and exec targets; Observer guards enforce them. Missing deny sinks are valid when the process is used only for judgment or dry-run analysis.
@@ -145,4 +151,7 @@ Exec deny is path-oriented. A deny action on a bare binary name may not map to a
 - `GET /metrics` with counters for judged events, blocks, overload degradation, and enforce failures.
 - `GET /healthz` with `200 ok` while the daemon is alive.
 
-Alert on `sentry_overload_degraded_total` and `sentry_enforce_failed_total`; both indicate that an intended block may not have landed.
+Alert on `sentry_overload_degraded_total` and `sentry_enforce_failed_total`.
+The first counts escalations rejected by the full worker queue:
+complete-evidence events use the fail mode, while incomplete command evidence
+remains unresolved. The second means a deny-file write failed.

apps/docs/content/docs/en/sentry/deployment-runbook.mdx

@@ -56,7 +56,7 @@ Page on rising values for:
 
 | Metric | Why it matters | First response |
 | --- | --- | --- |
-| `sentry_overload_degraded_total` | Escalations fell through to the fail mode because the worker queue was full. | Increase workers or queue, reduce escalation volume, tune the slow tier, or choose fail-closed deliberately. |
+| `sentry_overload_degraded_total` | Escalations were rejected by the full worker queue. Complete evidence uses the fail mode; incomplete command evidence remains unresolved. | Increase workers or queue, reduce escalation volume, tune the slow tier, or choose fail-closed deliberately for complete-evidence overflow. |
 | `sentry_enforce_failed_total` | A block wanted to write a deny file and failed. | Check volume space, mount mode, file permissions, and configured paths. |
 
 Use `sentry_events_total` and `sentry_blocked_total` for traffic and block-ratio trends. `/healthz` only says the daemon is alive; it does not prove that every deny landed.
@@ -73,7 +73,10 @@ Tune:
 - `A3S_SENTRY_AGENT_TIMEOUT` for the L3 A3S Code investigation.
 - `A3S_SENTRY_SPECULATE` for high-severity events where parallel L2/L3 is worth the extra cost.
 
-If overload rises, the deployment is under-provisioned for the current event mix or the deeper tier is too slow for the escalation rate.
+If overload rises, the deployment is under-provisioned for the current event
+mix or the deeper tier is too slow for the escalation rate. Complete-evidence
+events resolve through the fail mode; incomplete `ToolExec` evidence remains an
+audited L1 escalation.
 
 ## L2 Or L3 Outage
 
@@ -97,5 +100,6 @@ Observer guards can continue enforcing existing deny files while Sentry restarts
 
 - Observer-event Sentry is reactive. It can block the next matching action, not necessarily the action that produced the event.
 - Inline Gateway inspection is the pre-forward path for LLM/MCP traffic that is routed through the proxy.
+- The bundled daemon audits incomplete-evidence escalations but does not provide a durable external L3 queue; staged SDK consumers must own persistence and dispatch.
 - Denies are coarse and node-local: an IP or executable-path block can affect other workloads on the same node.
 - L2 and L3 process attacker-influenced content. Keep deterministic L1 and kernel guard enforcement as the floor.

apps/docs/content/docs/en/sentry/operations.mdx

@@ -29,13 +29,14 @@ most important counters to alert on are:
 
 | Metric | Meaning |
 | --- | --- |
-| `sentry_overload_degraded_total` | An escalation could not be processed and fell through to the configured fail mode. |
+| `sentry_overload_degraded_total` | An escalation was rejected by the full worker queue. Complete evidence uses the fail mode; incomplete command evidence remains unresolved. |
 | `sentry_enforce_failed_total` | A decision wanted to enforce, but writing the deny record failed. |
 | `sentry_blocked_total` | Blocks by tier and event family. |
 | `sentry_events_total` | Total events judged. |
 
-In fail-open deployments, overload degradation or enforce failure means a block
-may not have landed.
+Any non-zero overload rate means the worker pool is under-provisioned. In
+fail-open deployments, a complete-evidence overflow or enforce failure means a
+block may not have landed.
 
 ## Queue And Timeout Tuning
 
@@ -47,10 +48,17 @@ L2 and L3 run in worker pools away from the ingest thread. Tune:
 - L3 timeout for deeper A3S Code investigation.
 - Speculation threshold when high-severity events deserve L2/L3 in parallel.
 
+Complete-evidence queue overflow resolves through the configured fail mode.
+Incomplete `ToolExec` evidence remains an audited L1 escalation, even during
+overload.
+
 ## Boundaries
 
 - Observer-event Sentry is reactive; it may block the next action rather than the
   action that produced the event.
+- The bundled daemon audits incomplete-evidence escalations but does not provide
+  a durable external L3 queue. A staged SDK consumer must persist and dispatch
+  them when deeper work is required.
 - Denies can be coarse: binary path and IP-based enforcement may affect more
   than one process on the same node.
 - L2/L3 read attacker-influenced content. The deterministic L1 floor and

apps/docs/content/docs/en/sentry/pipeline.mdx

@@ -1,6 +1,6 @@
 ---
 title: Pipeline
-description: L1 rules, L2 LLM, L3 A3S Code investigation, SAE signals, fail posture, and speculative parallelism.
+description: L1 rules, L2 LLM, staged L3 dispatch, incomplete-evidence safeguards, SAE signals, fail posture, and speculation.
 ---
 
 # Sentry Pipeline
@@ -38,6 +38,12 @@ rules can be provided through an ACL policy file and are hot-reloaded.
 L1 is a fast triage layer, not a sandbox. It is intentionally conservative: if a
 case is ambiguous, it escalates rather than guessing.
 
+For `ToolExec`, Observer reports `argv_truncated` and `argv_incomplete` when it
+could not capture the complete command. A dangerous captured prefix can still
+block, but an otherwise ambiguous incomplete command remains an L1 escalation.
+The in-process L2/L3 judges are not asked to infer that a missing suffix was
+benign.
+
 ## L2 And L3
 
 L2 is a classifier over the event. It is useful for false-positive reduction and
@@ -47,6 +53,13 @@ L3 is an actual A3S Code agent. It loads security skills, considers the actor,
 the observed event, the possible attack chain, and the blast radius, then
 returns a structured verdict. It is slower but deeper.
 
+Products that separate fast judgment from deep investigation can call
+`Pipeline::evaluate_through_l2`. It returns the L1 decision, optional L2
+decision, effective decision, stage status, and escalation cause without
+invoking L3, applying `fail_closed`, or starting speculative L3. An
+`Escalated` result is work to persist and dispatch to a durable external L3
+worker, not an allow.
+
 ## Speculation
 
 By default, L3 starts only after L2 escalates. With speculative parallelism,
@@ -61,4 +74,6 @@ fail-closed only when the deployment has enough L2/L3 capacity or an allow-list
 posture that can tolerate blocking unknowns.
 
 Rules-only fail-open mode catches direct `block` rules but does not enforce
-rules that only escalate.
+rules that only escalate. Incomplete `ToolExec` evidence is the exception: it
+remains an unresolved L1 escalation regardless of fail mode. The bundled daemon
+audits that result but does not provide a durable external L3 queue.