Skip to content

Commit e133db0

Browse files
committed
chore(docs): update roadmap and design
1 parent 165a7c4 commit e133db0

6 files changed

Lines changed: 114 additions & 4 deletions

File tree

docs/design/ARCHITECTURE.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,8 @@ Each concept has a **source-of-truth document** and one or more documents that r
202202
| Agent swarm orchestration | ROADMAP.md (Iter 6) ||
203203
| Adaptive model router | ROADMAP.md (Iter 5) | COST_MODEL.md |
204204
| Capability-based security | ROADMAP.md (Iter 5) | SECURITY.md |
205+
| Centralized policy framework | ROADMAP.md (Iter 5), SECURITY.md (Policy enforcement and audit) | ORCHESTRATOR.md, OBSERVABILITY.md |
206+
| GitHub App + AgentCore Token Vault | ROADMAP.md (Iter 3c), SECURITY.md (Authentication) | ORCHESTRATOR.md (context hydration), COMPUTE.md |
205207
| Live session replay | ROADMAP.md (Iter 4) | API_CONTRACT.md |
206208
| PR iteration task type | API_CONTRACT.md, ORCHESTRATOR.md | USER_GUIDE.md, PROMPT_GUIDE.md, SECURITY.md, AGENT_HARNESS.md |
207209
| PR review task type | API_CONTRACT.md, ORCHESTRATOR.md | USER_GUIDE.md, PROMPT_GUIDE.md, SECURITY.md, AGENT_HARNESS.md |

docs/design/SECURITY.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,42 @@ The `functionArn` in `CustomStepConfig` should be validated at CDK synth time to
102102
- **Per-repo `egressAllowlist` is a declarative annotation**, not per-session enforcement. All agent sessions share the same VPC and DNS Firewall rules. Per-repo allowlists are aggregated (union) into the platform-wide policy.
103103
- **DNS Firewall does not prevent IP-based connections.** A direct connection to an IP address (e.g. `curl https://1.2.3.4/`) bypasses DNS resolution. This is acceptable for the "confused agent" threat model (the agent uses domain names in its tool calls) but does not defend against a sophisticated adversary. Closing this gap would require AWS Network Firewall (SNI-based filtering) at ~$274/month/endpoint.
104104

105+
## Policy enforcement and audit
106+
107+
The platform enforces policies at multiple points in the task lifecycle. Today, these policies are implemented inline across ~20 files (handlers, constructs, agent code). A centralized policy framework is planned (Iteration 5) to improve auditability, consistency, and change control.
108+
109+
### Current policy enforcement map
110+
111+
| Phase | Policy | Enforcement location | Audit trail |
112+
|---|---|---|---|
113+
| **Submission** | Input validation (format, ranges, lengths) | `validation.ts`, `create-task-core.ts` | HTTP 400 response only — no event emitted |
114+
| **Submission** | Repo onboarding gate | `repo-config.ts``create-task-core.ts` | HTTP 422 response only — no event emitted |
115+
| **Submission** | Guardrail input screening | `create-task-core.ts` (Bedrock Guardrails) | HTTP 400 response only — no event emitted |
116+
| **Submission** | Idempotency check | `create-task-core.ts` | HTTP 409 response only — no event emitted |
117+
| **Admission** | Concurrency limit | `orchestrator.ts` (`admissionControl`) | `admission_rejected` event emitted |
118+
| **Pre-flight** | GitHub reachability, repo access, PR access | `preflight.ts` | `preflight_failed` event emitted |
119+
| **Hydration** | Guardrail PR prompt screening | `context-hydration.ts` | `guardrail_blocked` event emitted |
120+
| **Hydration** | Budget/quota resolution (3-tier max_turns, 2-tier max_budget_usd) | `orchestrator.ts` (`hydrateAndTransition`) | Values persisted on task record — no policy decision event |
121+
| **Hydration** | Token budget for prompt assembly | `context-hydration.ts` | No event emitted |
122+
| **Session** | Tool access control (pr_review restrictions) | `agent/entrypoint.py` | No event emitted |
123+
| **Session** | Budget enforcement (turns, cost) | Claude Agent SDK | Agent SDK enforces; cost in task result |
124+
| **Finalization** | Build/lint verification | `agent/entrypoint.py` | Results in task record and PR body |
125+
| **Infrastructure** | DNS Firewall egress allowlist | `dns-firewall.ts`, `agent.ts` (CDK synth) | DNS query logs in CloudWatch |
126+
| **Infrastructure** | WAF rate limiting | `task-api.ts` (CDK synth) | WAF logs |
127+
| **State machine** | Valid transition enforcement | `task-status.ts`, `orchestrator.ts` | DynamoDB conditional writes |
128+
129+
### Audit gaps (planned remediation)
130+
131+
Submission-time policy decisions (validation, onboarding gate, guardrail screening, idempotency) currently return HTTP errors without emitting structured audit events. Budget resolution decisions are persisted but not logged as policy decisions with reason codes. Tool access selection is implicit (hardcoded in agent code) with no audit event.
132+
133+
**Planned (Iteration 5, Phase 1):** A unified `PolicyDecisionEvent` schema will normalize all policy decisions into structured events with: decision ID, policy name, version, phase, input hash, result, reason codes, and enforcement mode. See [ROADMAP.md Iteration 5](../guides/ROADMAP.md) for the full centralized policy framework design.
134+
135+
### Policy resolution (planned)
136+
137+
**Planned (Iteration 5, Phase 2):** Budget/quota/tool-access resolution will be consolidated into a `PolicyInput → PolicyDecision` module that computes effective values from the 3-tier hierarchy (per-task → Blueprint → platform default). This replaces the current scattered merge logic across handlers.
138+
139+
**Planned (Iteration 5, Phase 3):** Cedar authorization policies for multi-tenant access control. Cedar is preferred over OPA for this system: it is AWS-native, has formal verification guarantees, integrates with AgentCore Gateway, and policies can be evaluated in-process via the Cedar SDK without a separate service dependency.
140+
105141
## Memory-specific threats
106142

107143
### OWASP ASI06 — Memory and context poisoning

docs/guides/ROADMAP.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,24 @@ Deep research identified **9 memory-layer security gaps** in the current archite
271271
- **Advanced evaluation and feedback loop** — Extend the basic evaluation pipeline from Iteration 3d: ML-based or LLM-based trace analysis (not just rules), A/B prompt comparison framework, automated feedback into prompt templates (e.g. "for repo X, always run tests before opening PR"), and per-repo or per-failure-type improvement tracking. Evaluation results can update the repo's agent configuration stored during onboarding. **Optional patterns from adaptive teaching research** (e.g. plan → targeted critique → execution; separate **evaluator** vs **prompt/reflection** roles; fitness from LLM judging plus efficiency metrics; evolution of teaching templates from failed trajectories with Pareto-style candidate sets for diverse failure modes) can inform offline or scheduled improvement of Blueprint prompts and checklists without replacing ABCA's core orchestrator.
272272
- **Formal orchestrator verification (TLA+)** — Add a formal specification of the orchestrator in TLA+ and verify it with TLC model checking. Scope includes the task state machine (8 states, valid transitions, terminal states), concurrency admission control (atomic increment + max check), cancellation races (cancel arriving during any orchestration step), reconciler/orchestrator interleavings (counter drift correction while tasks are active), and the polling loop (agent writes terminal status, orchestrator observes and finalizes). Define invariants such as valid-state progression, no illegal transitions, and repo-level safety constraints (for example, at most one active `RUNNING` task per repo when configured). Keep the spec aligned with `src/constructs/task-status.ts` and orchestrator docs so regressions surface as model-check counterexamples before production.
273273
- **Guardrails (output and tool-call)** — Extend Bedrock Guardrails from input screening (implemented in Iteration 3c) to **output filtering** and **agent tool-call guardrails**. Apply content filters to model responses during agent execution, restrict sensitive content generation, and enforce organizational policies (e.g. "do not modify files in `/infrastructure`"). See [SECURITY.md](../design/SECURITY.md). Guardrails configuration can be per-repo (via onboarding) or platform-wide.
274-
- **Capability-based security model** — Fine-grained enforcement beyond Bedrock Guardrails, operating at three levels: (1) **Tool-level capabilities** — Bash command allowlist (git, npm, make permitted; curl, wget blocked), configurable per capability tier (standard / elevated / read-only). (2) **File-system scope** — Blueprint declares include/exclude path patterns; Write/Edit/Read tools are filtered to the declared scope. (3) **Input trust scoring** — Authenticated user input = trusted; external GitHub issues = untrusted; PR review comments entering memory = adversarial. Trust level selects the capability set. Essential once review feedback memory (Iter 3d) introduces attacker-controlled content into the agent's context. Blueprint `security` prop configures the capability profile per repo.
274+
- **Centralized policy framework** — Consolidate the platform's distributed policy decisions into a unified policy resolution module and audit layer. Policy logic today is scattered across 20+ files (input validation in `validation.ts` and `create-task-core.ts`, admission control in `orchestrator.ts`, guardrail screening in `context-hydration.ts`, budget resolution across `validation.ts`/`orchestrator.ts`/`entrypoint.py`, tool access in `entrypoint.py`, network egress in `dns-firewall.ts`/`agent.ts`, state transitions in `task-status.ts`/`orchestrator.ts`). This fragmentation makes it difficult to audit what policies exist, verify consistency, or change policy behavior without touching multiple files.
275+
276+
**Phase 1 — Policy audit normalization:**
277+
Define a stable `PolicyDecisionEvent` schema: `decision_id` (ULID), `policy_name` (e.g. `admission.concurrency`, `budget.max_turns`, `guardrail.input_screening`), `policy_version`, `phase` (`submission` | `admission` | `hydration` | `session_start` | `finalization`), `input_hash` (SHA-256 of the decision input for reproducibility), `result` (`allow` | `deny` | `modify`), `reason_codes[]`, `enforcement` (`enforced` | `observed`), and `task_id`. Emit a `policy_decision` event via `emitTaskEvent` at every existing enforcement point. Today, some decisions emit events (`admission_rejected`, `preflight_failed`, `guardrail_blocked`) while others silently return HTTP errors — normalize them all. This is pure instrumentation of existing code paths; no behavior change.
278+
279+
**Phase 2 — Policy resolution module (`policy.ts`):**
280+
Extract budget/quota/tool-access resolution logic into a single TypeScript module with a clear contract: `PolicyInput` (task metadata, user ID, repo config, blueprint config, phase) → `PolicyDecision` (computed budgets, tool profile, risk tier, redaction directives). This consolidates the 3-tier `max_turns` resolution, 2-tier `max_budget_usd` resolution, tool access determination (currently an `if` statement in `entrypoint.py`), and per-repo configuration merge (currently in `loadBlueprintConfig`). The module is called by existing handlers — no new service, no network hop, no new SPOF. Input validation (format checks, range checks) remains at the input boundary where it belongs; the policy module handles resolution and composition only.
281+
282+
**Phase 3 — Cedar authorization for multi-tenancy (with multi-user/team):**
283+
When multi-user/team support lands, introduce **Cedar** (not OPA) for tenant-specific authorization policies. Cedar is AWS-native, has formal verification guarantees, and integrates with AgentCore Gateway. Policies express rules like: "users in team X can submit tasks to repos A, B, C", "team Y has a monthly budget of $500", "repos tagged `critical` require `pr_review` before `new_task`". Cedar policies are stored alongside Blueprint config and evaluated at admission and finalization. This replaces the current single-dimensional ownership check (`record.user_id !== userId`) with multi-dimensional authorization (user, team, repo, action, risk level). Policy versioning, rollback, and observe-before-enforce semantics carry forward from Phase 1.
284+
285+
**Why not OPA:** OPA uses Rego (a custom DSL) and runs as a sidecar or external service. ABCA's policies change at the same cadence as infrastructure (deployed via CDK). A separate service with a separate language adds operational burden without proportionate benefit for a single-tenant platform. Cedar is a better fit: it's a typed language with formal verification, it's AWS-native (used by Amazon Verified Permissions and AgentCore Gateway), and policies can be evaluated in-process via the Cedar SDK without a separate service.
286+
287+
**What stays out of the policy framework:** Schema validation (repo format, `max_turns` range, task description length) stays at the input boundary. State machine transitions stay in the orchestrator. DNS Firewall stays in CDK. These are infrastructure invariants, not policy decisions — they don't vary by tenant, user, or context.
288+
289+
See [SECURITY.md](../design/SECURITY.md) (Policy enforcement and audit).
290+
291+
- **Capability-based security model** — Fine-grained enforcement beyond Bedrock Guardrails, operating at three levels: (1) **Tool-level capabilities** — Bash command allowlist (git, npm, make permitted; curl, wget blocked), configurable per capability tier (standard / elevated / read-only). (2) **File-system scope** — Blueprint declares include/exclude path patterns; Write/Edit/Read tools are filtered to the declared scope. (3) **Input trust scoring** — Authenticated user input = trusted; external GitHub issues = untrusted; PR review comments entering memory = adversarial. Trust level selects the capability set. Essential once review feedback memory (Iter 3d) introduces attacker-controlled content into the agent's context. Blueprint `security` prop configures the capability profile per repo. Capability tiers become inputs to the centralized policy framework (Phase 2) and can be governed by Cedar policies (Phase 3) when multi-tenancy is in place.
275292
- **Additional execution environment** — Support an alternative to AgentCore Runtime (e.g. ECS/Fargate, EKS) behind the **ComputeStrategy** interface (see [REPO_ONBOARDING.md](../design/REPO_ONBOARDING.md#compute-strategy-interface)). The orchestrator calls abstract methods (`startSession`, `stopSession`, `pollSession`); the implementation maps to AgentCore, Fargate, or EKS. Repos select the strategy via `compute_type` in their blueprint configuration. Reduces vendor lock-in and enables workloads that exceed AgentCore limits (e.g. GPU, larger images, longer sessions). The ComputeStrategy interface contract is defined in Iteration 3a; Iteration 5 adds alternative implementations.
276293
- **Full web dashboard** — Extend the control panel from Iteration 4: detailed dashboards (cost, performance, evaluation), reasoning trace viewer or log explorer (linked to OpenTelemetry traces from AgentCore), task submit/cancel from the UI, and admin views (system health, capacity, user management).
277294
- **Customization (advanced) with tiered tool access** — Agent can be extended with **MCP servers**, **plugins**, and **skills** beyond the basic prompt-from-repo customization in Iteration 3a. Composable tool sets per repo. MCP server discovery and lifecycle management. More tools increase behavioral unpredictability, so use a **tiered tool access model**: a minimal default tool set (bash allowlist, git, verify/lint/test) that all repos get, with MCP servers and plugins as opt-in per repo during onboarding. Per-repo tool profiles are stored in the onboarding config and loaded by the orchestrator. This balances flexibility with predictability. See [SECURITY.md](../design/SECURITY.md) and [REPO_ONBOARDING.md](../design/REPO_ONBOARDING.md).
@@ -308,7 +325,7 @@ Deep research identified **9 memory-layer security gaps** in the current archite
308325
- **Iteration 3e** — Memory security and integrity: input hardening (content sanitization, provenance tagging, integrity hashing), trust-aware retrieval (trust scoring, temporal decay, guardian validation), detection and response (anomaly detection, circuit breaker, quarantine, rollback), advanced protections (write-ahead validation, behavioral drift detection, cryptographic provenance, red teaming). Addresses OWASP ASI06 (Memory & Context Poisoning).
309326
- **Iteration 3bis** (hardening) — Orchestrator IAM grant for Memory (was silently AccessDenied), memory schema versioning (`schema_version: "2"`), Python repo format validation, severity-aware error logging in Python memory, narrowed entrypoint try-catch, orchestrator fallback episode observability, conditional writes in agent task_state.py (ConditionExpression guards), orchestrator Lambda error alarm (CloudWatch, retryAttempts: 0), concurrency counter reconciliation (scheduled Lambda, drift correction), multi-AZ NAT documentation (already configurable), Python unit tests (pytest), entrypoint decomposition (4 extracted subfunctions), dual prompt assembly deprecation docstring, graceful thread drain in server.py (shutdown hook + atexit), dead QUEUED state removal (8 states, 4 active).
310327
- **Iteration 4** — Additional git providers, visual proof (screenshots/videos), Slack channel, skills pipeline, user preference memory (Tier 3), control panel (restrict CORS to dashboard origin), real-time event streaming (WebSocket), live session replay and mid-task nudge, browser extension client, MFA for production.
311-
- **Iteration 5** — Snapshot-on-schedule pre-warming, multi-user/team, memory isolation for multi-tenancy, full cost management, adaptive model router with cost-aware cascade, advanced evaluation (optional adaptive-teaching / trajectory-driven prompt patterns), formal orchestrator verification with TLA+/TLC, Bedrock Guardrails output/tool-call (PII, denied topics, output filters) — input screening in 3c, capability-based security model, alternate runtime, advanced customization with tiered tool access (MCP/plugins via AgentCore Gateway), full dashboard, AI-specific WAF rules.
328+
- **Iteration 5** — Snapshot-on-schedule pre-warming, multi-user/team, memory isolation for multi-tenancy, full cost management, adaptive model router with cost-aware cascade, advanced evaluation (optional adaptive-teaching / trajectory-driven prompt patterns), formal orchestrator verification with TLA+/TLC, Bedrock Guardrails output/tool-call (PII, denied topics, output filters) — input screening in 3c, centralized policy framework (Phase 1: policy audit normalization with `PolicyDecisionEvent` schema across all enforcement points; Phase 2: policy resolution module consolidating budget/quota/tool-access logic; Phase 3: Cedar authorization for multi-tenancy — AWS-native, formal-verification-backed, integrates with AgentCore Gateway), capability-based security model (tiers feed into policy framework), alternate runtime, advanced customization with tiered tool access (MCP/plugins via AgentCore Gateway), full dashboard, AI-specific WAF rules.
312329
- **Iteration 6** — Agent swarm orchestration, skills learning, multi-repo, iterative feedback and multiplayer sessions, HITL approval, scheduled triggers, CDK constructs.
313330

314331
Design docs to keep in sync: [ARCHITECTURE.md](../design/ARCHITECTURE.md), [ORCHESTRATOR.md](../design/ORCHESTRATOR.md), [API_CONTRACT.md](../design/API_CONTRACT.md), [INPUT_GATEWAY.md](../design/INPUT_GATEWAY.md), [REPO_ONBOARDING.md](../design/REPO_ONBOARDING.md), [MEMORY.md](../design/MEMORY.md), [OBSERVABILITY.md](../design/OBSERVABILITY.md), [COMPUTE.md](../design/COMPUTE.md), [CONTROL_PANEL.md](../design/CONTROL_PANEL.md), [SECURITY.md](../design/SECURITY.md), [EVALUATION.md](../design/EVALUATION.md).

docs/src/content/docs/design/Architecture.md

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)