fix(agent): build/verify failure diagnosis — timeout classify, claude-binary Exec-format, failing-line surfacing#597
Conversation
The ECS compute strategy inlined the full orchestrator payload (incl. the large hydrated_context) into the AGENT_PAYLOAD container-override env var. ECS RunTask caps the TOTAL containerOverrides blob at 8192 bytes, so any real task was rejected before the container started: InvalidParameterException: Container Overrides length must be at most 8192 AgentCore is unaffected — it passes the payload in the InvokeAgentRuntime request body, which has no comparable limit. The bug only surfaces with a realistic hydrated payload, which is why the prior ECS smoke test (a small Rust cargo-check, #494) didn't catch it. Fix — stash the payload out-of-band and pass only a pointer: - New EcsPayloadBucket construct (mirrors TraceArtifactsBucket): BLOCK_ALL, enforceSSL, S3_MANAGED encryption, 1-day lifecycle TTL (payloads are ephemeral — read once at boot). Dedicated bucket so the ECS task role's S3 read is scoped to payloads only and can't touch attachments/traces. - ecs-strategy: when ECS_PAYLOAD_BUCKET is set, PutObject the payload to <task_id>/payload.json and pass AGENT_PAYLOAD_S3_URI in the override; the boot command fetches+parses it via boto3. Inline AGENT_PAYLOAD remains as a fallback (small payloads / no bucket), so nothing regresses. deleteEcsPayload helper removes the object. - orchestrate-task finalize: best-effort deleteEcsPayload for ECS tasks once terminal (the container has long since read it); lifecycle rule is the crash backstop. - EcsAgentCluster: accept payloadBucket, inject ECS_PAYLOAD_BUCKET env, grant the task role READ ONLY (untrusted repo code must not write/delete payloads; the trusted orchestrator owns write+delete). Session-role-aware. - task-orchestrator: ecsPayloadBucket prop → grantPut + grantDelete to the orchestrator; @aws-sdk/client-s3 added to bundling externals. - agent.ts: updated the commented uncomment-to-enable ECS scaffolding to wire the payload bucket. Tests: new bucket construct (TTL/SSL/block-public/autoDelete); strategy S3-write + URI-pointer + inline fallback + deleteEcsPayload (incl. best-effort swallow + no-op without bucket); cluster read-grant + env var + read-only (no put/delete). Full build green. Closes #502
Root cause of the exit-127 inert gate: the agent Dockerfile installs node/npm/ mise/uv but NOT yarn, so any repo whose build runs 'yarn install' (incl. ABCA) hit 'yarn: command not found' and the agent hand-built a ~/bin/yarn shim every run. corepack (ships with Node 24) installs the yarn/pnpm shims at image-build time. Systemic — fixes every yarn repo, no per-run shim. Needs redeploy to take effect. (cherry picked from commit 2fd5fca) (cherry picked from commit d5e0e64)
…ilds AgentCore's fixed microVM envelope OOM-kills CI-parity builds (live-caught dogfooding ABCA-on-ABCA: the ~2800-test `mise run build` killed the VM at the build-gate step). AgentCore memory isn't tunable, so route repos that set `compute_type: 'ecs'` to a Fargate task instead. - agent.ts: gate EcsAgentCluster + the orchestrator ecsConfig on the `compute_type` deploy context (default 'agentcore'). ECS resources only synthesize under `--context compute_type=ecs`, so default synth (and the bootstrap-coverage test) stays agentcore-only. Mirrors upstream's context-gating of the ECS construct. - ecs-agent-cluster.ts: size the Fargate task at 8 vCPU / 32 GB (was 2/4) — a valid ARM64 combo with headroom for the jest worker fleet + tsc that AgentCore's envelope can't give. Comment documents the live OOM. - test: assert the new 8192/32768 sizing. Verified: default synth has 0 ECS resources; `--context compute_type=ecs` synth emits AWS::ECS::Cluster + TaskDefinition. Full `mise run build` green. (cherry picked from commit 8cb7cfa) (cherry picked from commit 632a36b)
Live-fired the full `mise run build` on the 32GB ECS task (fork dogfood): install OK, build ran ~50 min then OOM-killed at the cap (ECS stoppedReason "OutOfMemoryError: container killed due to memory usage", exit 137; peak working set ~31.6 GB). Root cause: the monorepo build fans out 4 heavy jobs in PARALLEL (agent:quality ‖ cdk:build ‖ cli:build ‖ docs:build), each with its own worker fleet (jest, pytest, esbuild Lambda bundling) — 32 GB had no headroom for that concurrent peak, and the ~50-min wall-clock also blew the 1800s build-gate cap. Fix (keep the full build as the gate, give it room): - ecs-agent-cluster.ts: Fargate 8vCPU/32GB → 16vCPU/64GB (valid ARM64 combo; 16 vCPU admits 32–120 GB). 2× memory headroom for the parallel storm + more cores to shorten wall-clock. Comment records the full sizing history. - ecs-agent-cluster.ts: inject BUILD_VERIFY_TIMEOUT_S=3600 into the container env so a slow-but-healthy CI-parity build isn't mis-flagged as a timeout (post_hooks.py reads it; default 1800 stays for AgentCore repos). - tests: assert 16384/65536 + the new env var. Verified: default synth ECS-free; --context compute_type=ecs synth emits Cpu 16384 / Memory 65536 / BUILD_VERIFY_TIMEOUT_S 3600. Full build green. (cherry picked from commit 11fb3a5) (cherry picked from commit cd7b8eb)
…aren't dropped (ABCA-487) The ECS boot command hand-listed ~18 run_task kwargs and silently dropped the rest. On ECS this meant channel_source/channel_metadata never reached run_task, so resolve_linear_api_token never ran, LINEAR_API_TOKEN was never set, and the Linear/Jira reaction + channel MCP silently no-op'd — @bgagent tasks on ECS posted NO 👀 reaction or anything (live-caught: ABCA-487 on the fork, RUNNING with mcp_servers:[] and no reaction). Also dropped: build_command/lint_command (build-gate used defaults), cedar_policies + approval_* (HITL guardrails), base_branch/merge_branches (orchestration stacking), attachments, trace, user_id. Same "hand-rolled partial payload copy" root as #502. Fix — single source of truth, unit-testable: - pipeline.run_task_from_payload(p): maps the WHOLE payload dict to run_task's signature — rename prompt→task_description / model_id→anthropic_model, filter to accepted params (unknown keys ignored, not passed as **kwargs), coerce issue_number/pr_number→str + max_turns→int, aws_region falls back to env. _RUN_TASK_PARAMS computed once at import (patch-safe). - entrypoint exports it; ecs-strategy boot command now calls run_task_from_payload(p) instead of the hand-listed kwargs. - tests: 9 agent tests (rename, channel fields ABCA-487, build/cedar/branch, coercion, unknown-key drop, None-drop, aws_region, signature guard) + ecs-strategy asserts the boot command calls the mapper and no longer hand-lists. Full build green: agent + cdk 2852 + cli 571 + docs. Deployed to dev (--context compute_type=ecs; agent image rebuilt). (cherry picked from commit 777ee0e) (cherry picked from commit 1639c27)
…th-* (ABCA-488) A Linear/Jira-channel task resolves its per-workspace OAuth token from Secrets Manager (bgagent-linear-oauth-<slug>) at startup to fire the 👀→✅ reaction and drive the channel MCP. The AgentCore runtime role + orchestrator/fanout/ screenshot/linear-integration roles all have the bgagent-linear-oauth-* prefix grant, but the ECS task role only had GetSecretValue on the GitHub token. So on ECS the token fetch hit AccessDenied → LINEAR_API_TOKEN unset → linear_reactions skipped AND the Linear MCP stayed needs-auth (agent couldn't post comments either). Live-caught dogfooding on the fork (ABCA-488, ECS task 828dab35). Fourth ECS-parity gap after #502 (payload size) + ABCA-487 (dropped channel fields): the AgentCore path had a capability the ECS task role didn't. Fix: grant the ECS task role secretsmanager:GetSecretValue on the bgagent-linear-oauth-* and bgagent-jira-oauth-* prefixes (GetSecretValue only — the container reads; the orchestrator owns refresh/PutSecretValue). Nag reason updated + regression test asserting the prefix grant. Full build green (2853). (cherry picked from commit 5a886bb) (cherry picked from commit 49235d4)
#299) Live-caught: a :decompose on an ecs-configured repo ran on ECS (fix confirmed — no more 'ECS_CLUSTER_ARN missing') but then FAILED at delivery with 'ValueError: deliver_artifact: ARTIFACTS_BUCKET_NAME is not configured'. The AgentCore runtime sets ARTIFACTS_BUCKET_NAME; the EcsAgentCluster task def never got the bucket — an ECS-parity gap (same class as #502/ABCA-487). A repo-bound artifact workflow (coding/decompose-v1) delivers its plan JSON there. - EcsAgentCluster: new optional artifactsBucket prop → ARTIFACTS_BUCKET_NAME in the container env + grantReadWrite on the task role (read+write: the container DELIVERS the artifact, unlike the read-only payload bucket). IAM5 suppression extended to cover the second scoped S3 grant. - agent.ts: pass traceArtifactsBucket.bucket (same bucket the runtime uses). - 3 construct tests (env injected / read+write grant / omitted when absent). 2872 CDK + 571 CLI + 1184 agent green. Held on branch. (cherry picked from commit 7617d72) (cherry picked from commit e0dfc93)
… substrate A repo onboarded as compute_type=ecs on a stack deployed WITHOUT --context compute_type=ecs fails every task at session start (no ECS_* env vars). Catch the config/deploy mismatch early + make the runtime error actionable: - agent.ts: new ComputeSubstrate CfnOutput — 'ecs' when the gate is on, else 'agentcore' (ecs is additive; the AgentCore runtime is always present, so an agentcore repo works on either). - cli repo onboard: when --compute-type ecs, read ComputeSubstrate and refuse with a redeploy message if it's an explicit non-ecs value. Null (older stacks predating the output) is treated as unknown → proceed (runtime backstop). - ecs-strategy: the missing-env error now names the root cause + remedy (redeploy with the gate, or set the repo to agentcore) instead of a bare env-var list. - Tests: CDK output flips with the gate (agentcore default / ecs gated); CLI onboard refuses/allows/back-commpat; ecs construct unchanged. 2875 CDK + 575 CLI + 1184 agent green. Held on branch. (cherry picked from commit cb7c8c6) (cherry picked from commit 9048db0)
…baseline OOM ABCA-662's pre-agent baseline build was OOM-killed (exit 137) at 64 GB: dogfooding ABCA-on-ABCA, the full parallel `mise run build` (agent:quality ‖ cdk:build ‖ cli:build ‖ docs:build, each with its own worker fleet) peaked past 64 GB. Each build task is memory-ISOLATED (its own Fargate microVM), so limiting concurrency does NOT help a single over-64 GB build — only more per-task RAM or less build parallelism does. 120 GB is the MAX Fargate admits at 16 vCPU (32–120 in 8 GB steps), so this is the clean experiment: if a build still OOMs here, the only remaining lever is cutting the build's peak parallelism (serialize the mise DAG / cap jest --maxWorkers) — there is no more RAM to give. Pairs with the repo.py fix (an OOM-killed baseline is now infra, not 'already broken'), so even if a build does hit the ceiling the verdict stays honest. Tests updated to assert Memory 122880 (both BUILD-def assertions). (cherry picked from commit 25e2bb1)
The inline-fallback / no-op tests in the top-of-file describe blocks assume the OPTIONAL vars ECS_PAYLOAD_BUCKET and ECS_PLANNING_TASK_DEFINITION_ARN are ABSENT when the module is first imported (it reads them as module-level constants). A dev shell has neither set, so the suite passed locally — but the REAL ECS agent container HAS ECS_PAYLOAD_BUCKET set (#502 payload bucket), so on ECS the const was truthy and the 'no bucket → inline fallback' + 'no-op' assertions failed: FAIL test/handlers/shared/strategies/ecs-strategy.test.ts ● startSession › sends RunTaskCommand with correct params ... ● deleteEcsPayload without ECS_PAYLOAD_BUCKET › no-ops ... This was the residual fork baseline-build exit-1: 2 failed / 3093 passed, only ever reproducible inside the ECS microVM. Surfaced by the live-streaming build log (the buffered summary had hidden it). Fix: delete both optional vars before the top-of-file import so it's hermetic regardless of the runner's environment; the #502/#299 describe blocks already set them via jest.isolateModules. (cherry picked from commit 7a2bde9) (cherry picked from commit 8267c29)
…(cdk synth on fresh clone)
A CDK-based target repo's build gate runs `cdk synth`. When the stack is wired
to a concrete env ({account, region}), CDK does a synth-time availability-zone
context lookup (ec2:DescribeAvailabilityZones). A developer box caches the result
in the gitignored cdk.context.json so synth is hermetic; the agent clones fresh,
so there is no cache and the live lookup fires. The ECS task role lacked the
action, so synth failed with:
ERROR ...EcsAgentClusterTaskRole... is not authorized to perform:
ec2:DescribeAvailabilityZones ...
Synthesis finished with errors
→ a FALSE build-gate failure on code that builds fine everywhere else. Same
ECS-parity class as ABCA-488 (GetSecretValue) and F-2 (CreateEvent): a permission
the AgentCore path had but the ECS task role didn't.
Live-caught on the ABCA fork: baseline `mise run build` passed all 3095 tests
then died at `cdk synth -q`, surfaced only by the live-streaming build log.
Grant is a read-only describe (Resource:* — EC2 describe actions have no
resource-level scoping; IAM5 suppressed, no mutation/data access). +1 regression
test pins the statement.
(cherry picked from commit 3fc9060)
(cherry picked from commit 2bf12f4)
…class) Live-caught during the fork stress smoke (ABCA-495): the ECS TaskDefTaskRole got AccessDeniedException on bedrock-agentcore:CreateEvent for the AgentMemory resource, so the agent's cross-task learning writes (write_task_episode / write_repo_learnings) silently no-op'd (WARN) on EVERY ECS task — memory never persisted on the fork's ECS-only substrate. Same class as ABCA-488: the AgentCore runtime role gets memory access via agentMemory.grantReadWrite(runtime) in agent.ts, and the orchestrator gets it too, but the ECS task role never did. Fix: EcsAgentCluster gains an optional agentMemory prop; when provided the task role gets agentMemory.grantReadWrite (read actions + CreateEvent), scoped to the memory ARN (synth-verified: no wildcard resource, so the existing IAM5 nag suppression is unaffected). agent.ts passes agentMemory to the ECS cluster alongside the existing memoryId. Omitted in isolated construct tests / memory-less deploys → no grant (asserted). Tests: construct asserts CreateEvent on the task role scoped to MemoryArn when wired, and NO bedrock-agentcore grant when unwired. Full cdk build green (2883). (cherry picked from commit c23d49fbead30ed52860e3975815e3643b22ade0) (cherry picked from commit 44bb71e)
The //cdk:bootstrap task hint and DEPLOYMENT_ROLES.md both said to enable the ECS compute backend with 'mise //cdk:bootstrap -- --context ComputeTypes=agentcore,ecs'. That does NOT work: ComputeTypes is a CloudFormation *template parameter*, and this CDK version's 'cdk bootstrap' has no way to set one — there's no --parameters flag, and --context sets CDK construct context, not a CFN parameter. So the ECS policy (IaCRole-ABCA-Compute-ECS, conditional on ComputeTypes containing 'ecs') was never attached, and a --context compute_type=ecs deploy then failed with AccessDenied on ecs:* — live-caught deploying the substrate to dev. Correct it to the mechanism that actually works: bootstrap normally, then set the parameter directly on the CDKToolkit stack via 'aws cloudformation update-stack ... ParameterKey=ComputeTypes,ParameterValue=agentcore,ecs' (with a describe-stacks verify). Fixed in the mise task description + comment and both DEPLOYMENT_ROLES.md occurrences; Starlight mirror regenerated. Drift gate green. (cherry picked from commit a46683e)
…A-667, was 'Unexpected error') Replication finding: a live fork coding child (ABCA-667) failed with 'TimeoutExpired: Command [mise run build] timed out after 600 seconds' — the full ~2800-test fork build exceeded the 600s cap. That's the pre-agent verify-build subprocess timeout, which lands raw in error_message and fell through to 'Unexpected error' + generic guidance — exactly the transient-vs-real confusion this rework targets. It's not a crash and not broken code: the build ran too long. Add a TIMEOUT classification for / → 'Build/tests didn't finish in time', user class, retryable, remedy names the real fix (retry / raise BUILD_VERIFY_TIMEOUT_S or move to the larger ECS build box). Ordered before the generic poll-timeout pattern. Test uses the exact ABCA-667 shape. 60 classifier tests green. (cherry picked from commit 6394a59) (cherry picked from commit b9c1a94)
… (Exec format error) claude-code@2.1.191 uses a shim (bin/claude.exe) + platform-optional-dependency install model: its postinstall (install.cjs) is supposed to replace the shim with the platform-native ELF binary. When that postinstall silently falls back (e.g. it hits EACCES overwriting the shim under npm's root de-privileging), the error-shim stays on PATH and every agent run dies at the run_agent step with 'OSError: [Errno 8] Exec format error: \'claude\'' — even though the native arm64 binary is present in the image as an optional dep, just never wired up. Live-caught on ABCA-659's retry: all 3 ECS runs failed this way on a freshly rebuilt image (the prior working image ran the older single-binary 2.1.142). Re-run install.cjs explicitly after the npm install so the native binary is placed at build time, and hard-verify with 'claude --version' so a still-broken shim fails the image build instead of every task at runtime. (cherry picked from commit 7cb4a87) (cherry picked from commit 7de778b)
…debuggable The post-agent build gate logged only stderr on a non-zero exit. But build/test tooling (jest, tsc, the mise task DAG) writes the ACTUAL failing-task error to STDOUT, not stderr — stderr often carries only the runner's plan echo. So a red `mise run build` surfaced every task STARTING in CloudWatch but never WHICH one failed or why (ABCA-662: a 7s exit-1 with zero captured cause, indistinguishable from a phantom, and the reason Linear showed a build-fail GitHub CI passed). run_cmd now also tails the last 20 lines of stdout on failure (redacted — repo build output is untrusted). 4 tests pin: stdout surfaced, tailed-not-headed, redacted, and NOT dumped on success. (cherry picked from commit 67d84db) (cherry picked from commit bb578fd)
…ust a tail The stdout-on-failure logging (67d84db) tailed the last 20 lines — enough for a serial command, but NOT for `mise run build`, which runs 4 packages in PARALLEL and interleaves their output. The failing task's error lands in the MIDDLE while the tail captures whatever finished LAST (e.g. a PASSING package's coverage table) — so the log showed a green coverage table on a red build and named nothing (ABCA-662: three verification runs where the actual failing sub-task was never visible in CloudWatch). run_cmd now scans the WHOLE stdout for failure-signature lines (FAIL/✕/●/error TS/ELIFECYCLE/coverage-threshold/mise 'no task'/'task failed'/traceback/…), filters benign noise ('0 errors', '--no-error'), and logs those FIRST, then a trailing tail for context — deduped, capped at 40 lines. Falls back to the tail when nothing matches (unknown tool). This is the tooling gap that made the whole ABCA-662 build investigation slow: every failure was diagnosable only by luck or local repro. 5 tests incl. the mid-DAG-failure + coverage-threshold cases. (cherry picked from commit 3fe99c9) (cherry picked from commit aaa4f18)
scottschreckengaust
left a comment
There was a problem hiding this comment.
Review — Principal Architect (Stack A: ECS substrate, PR 3/4)
Base verified against fresh origin PR head (diffed vs #596). Mandatory agents run stack-wide (full table on the #596 review).
Verdict: Approve (with a nit)
Solid build/verify-diagnosis work, all three changes tied to live incidents:
_surface_failure_lines(agent/src/shell.py) — scans the WHOLE stdout for failure-signature lines, not just the tail. Correct for a parallel mise DAG that interleaves output so the failing task's line lands in the middle while the tail is a passing package's coverage table (ABCA-662). Capped at 40 + a 15-line tail, deduped, and redacted (redact_secrets) since repo build output is untrusted. Thereturn out or tailfallback guarantees a failure never logs nothing. Verifiedrun_cmd'scheck=Truepath still raises — the added stdout logging is purely additive, not a swallow.- Error-classifier TIMEOUT pattern (
cdk/src/handlers/shared/error-classifier.ts) — classifies a build/verify wall-clock kill distinctly from a crash (retryable: true, user-actionable remedy). Verified it's ordered before the generic catch-all and isn't shadowed by the earlierpoll timeout exceededrule. - Dockerfile claude
install.cjsre-run +claude --version— a real fix: claude-code@2.1.x ships an error-shim that its postinstall can silently leave in place, dying at runtime with "Exec format error". Re-running the installer and hard-verifying it exec's fails the build instead of the task.
Tests (test_shell.py::TestRunCmdFailureLogging, error-classifier.test.ts) cover the key cases well: stdout-logged-on-failure, mid-DAG line surfaced, coverage-threshold surfaced, benign "0 errors" not falsely elevated, redaction, success-doesn't-dump, no-marker→tail fallback, and the timeout classified as retryable-not-Unexpected.
Non-blocking nit
- N5 — Broad failure markers.
_FAILURE_LINE_MARKERSincludes broad substrings ("error:","failed") that could surface benign advisory lines. It's logging-only, capped, redacted, and the_FAILURE_LINE_NOISEallowlist covers common false positives — so worst case is slightly noisier CloudWatch, no functional impact. Flagging for awareness only.
Governance note
Branch pr/agent-build-robustness carries no issue number and the body references only stacked PRs (ADR-003). Same governance item raised on #596 — please attach an approved issue and a conforming branch name before merge. Code is approve-ready.
CI
All green (title, CodeQL ×3, dead-code advisory, secrets/deps/actions, build (agentcore)); MERGEABLE. No CDK construct/stack change → bootstrap synth-coverage N/A.
🤖 Generated with Claude Code
The `export const ECS_PAYLOAD_OBJECT_KEY_PREFIX = ''` constant was referenced nowhere — ecsPayloadKey() (ecs-strategy.ts) builds `<task_id>/payload.json` independently and already documents that layout in its own docstring. The dead export was a "prefix" whose value was '' with a docstring describing a key layout it didn't produce, and it raised the knip dead-code count, tripping the ratchet inside `mise run build`. Delete it; ecsPayloadKey remains the single source of truth for the key layout.
isadeks
left a comment
There was a problem hiding this comment.
Cross-reference: Dockerfile fix overlaps #613 (and complements #599)
Flagging a coordination point surfaced while reviewing #613 — the claude-binary Exec format error is being addressed across three open PRs:
- This PR (#597) and #613 both append
node install.cjsto the sameRUN … npm --prefix … update …line inagent/Dockerfile, so they will textually conflict — both can't merge as-is. #597's version is the superset: it also adds&& claude --version, hard-gating the image build so a silent postinstall fallback failsdocker buildrather than every task at runtime (strictly better than restore-and-hope), plus a comment explaining the shim mechanism. - #599 is complementary, not conflicting — it classifies the same
Exec format errorinto a user-facing message (error-classifier.ts, disjoint file). Prevention (here/#613) + legibility (#599) are the two halves of ABCA-659.
Suggested resolution: since #597's Dockerfile hunk supersedes #613's, either close #613 in favor of this PR, or — if this PR stays blocked behind its stack (#596 → #503; #503 merged, #596 still open) and the exec-format break is hurting people now — let #613 merge against main and drop the Dockerfile hunk here on rebase. Whoever lands first wins the line; the other should rebase around it.
(I verified install.cjs is idempotent and exists in @anthropic-ai/claude-code@2.1.191; couldn't reproduce the npm update clobber on linux/amd64, so it looks arm64/AgentCore-specific — consistent with the live-caught reports. The && claude --version gate here is the right belt-and-braces regardless.)
…+ nits
B1 (least-privilege regression): remove props.artifactsBucket.grantReadWrite
from the ECS task role. coding/decompose-v1 delivers its plan via the assumed
SessionRole (deliverers.py -> tenant_client), scoped to artifacts/${task_id}/*,
exactly like the AgentCore runtime (whose task role likewise has NO direct
artifacts grant). The whole-bucket grant over-privileged the untrusted-code
role and broke cross-task isolation (a task could read/clobber other tasks'
artifacts/<other_id>/, traces/, attachments/). Task role keeps only the
ARTIFACTS_BUCKET_NAME env. Correct the inverted 'parity' comment and drop the
now-stale artifacts clause from the cdk-nag IAM5 reason.
Test: flip 'grants READ+WRITE on artifacts' → asserts the task role has NO S3
Put/Delete action at all (the read-only #502 payload GetObject*/List* grant is
not flagged).
N4: add lint_command to _KNOWN_ORCHESTRATOR_KEYS (sibling of build_command) so a
future 'wired build_command, forgot lint_command' contract gap WARNs instead of
dropping silently.
(N1/N2/N3 comment nits were already addressed on the branch head; B1 dead-const
ECS_PAYLOAD_OBJECT_KEY_PREFIX no longer present. B2 governance handled separately.)
…noise + max_turns coercion Scott's fresh re-review (head 86ab249) is Approve-with-nits: B1 (over-privilege) and B2 (governance) resolved; remaining asks are doc-accuracy + WARN-channel hygiene. - N1: rewrite the artifactsBucket JSDoc (ecs-agent-cluster.ts) that still claimed a read/write task-role grant the B1 fix deliberately removed — now states env-only wiring with delivery through the task_id-scoped SessionRole (parity with the AgentCore runtime role, which has no artifacts grant). The grant block ~30 lines below was already corrected in the B1 commit; this was its residual half. - N2: same inverted-comment fix at the agent.ts wiring site — '(read+write grant in the construct)' → env-only + SessionRole delivery. - N3: drop github_token_secret_arn from _KNOWN_ORCHESTRATOR_KEYS. It is ALWAYS present and ALWAYS resolved via the GITHUB_TOKEN_SECRET_ARN env, so listing it fired the known-key WARN on 100% of ECS boots — pure noise diluting the channel meant for genuine future contract gaps. Now falls through as a quiet foreign-key drop; comment records why it must not be re-added. +test asserting no WARN. - N4: max_turns int() coercion accepted a bool (int(True)==1) and truncated a non-integral float (int(3.9)==3), and the comment falsely claimed it 'matches how every other field is handled' (it is the one non-str coercion). Now rejects bools + non-integral floats with a breadcrumb; valid int / int-string / int-float still pass. +test. Reachable only via a corrupt payload (orchestrator emits a real int), so cosmetic — bundled since cheap. - N5: no-op — the Dockerfile corepack line already has no 2>/dev/null. Gates: cdk 2355 tests + eslint + synth green; agent 1268 tests + ruff + ty green. Non-blocking #608 test-seam gaps (finalize deleteEcsPayload, boot-uri parse, entrypoint re-export, zero-ECS-synth assert) remain tracked, not in scope here.
…nt-build-robustness
The base branch was changed.
scottschreckengaust
left a comment
There was a problem hiding this comment.
Review — Principal Architect (re-review vs. clean main)
My earlier review here is dismissed — it was diffed against #596's branch. Both parents are now merged (#503, #596), so this re-review is against a clean main base; the diff is exactly the 5 files described in the body. #599 (the complementary Exec format error classifier) also merged since — I re-verified there's no overlap/shadowing. Mandatory agents re-run: code-reviewer, silent-failure-hunter, pr-test-analyzer, comment-analyzer. All findings independently verified against real code (I did not take agent output at face value — the loudest agent finding turned out wrong; see below).
Verdict: Request changes — one narrow blocker; everything else is approve-ready
The core work is genuinely good and each change is tied to a live incident. The blocker is not a crash — it's that this PR's own feature (accurate, actionable error legibility) ships a user-facing remedy that points an operator at a lever that does nothing, under a comment whose causal narrative the code contradicts. On a repo whose backing issue is literally titled "honest failure classification" (#610, approved), that's a contract defect, not a nit.
Blocking
B1 — The new TIMEOUT remedy misdirects the operator to a non-functional knob, and its anchoring comment is contradicted by the code. cdk/src/handlers/shared/error-classifier.ts:440-450
Two linked, verified problems in the new classification block:
BUILD_VERIFY_TIMEOUT_Sis consumed nowhere inagent/.grep -rn BUILD_VERIFY_TIMEOUT_S agent/returns nothing.verify_build/verify_lint(agent/src/post_hooks.py:19,39) callrun_cmd(["mise","run","build"], check=False)with notimeout=, so they always userun_cmd's hardcodedtimeout: int = 600(agent/src/shell.py:244). The env var is only set (CDKecs-agent-cluster.ts, value 3600) and asserted in a test — nothing reads it. So the remedy line "an admin can raiseBUILD_VERIFY_TIMEOUT_S" tells an operator to turn a knob that isn't wired to anything.- The comment's live-caught narrative is contradicted by the code. The comment (lines 438-444) says this was "live-caught on ABCA-667: the full
mise run buildexceeded the 600s default and surfaced as a bare 'Unexpected error'." Butpost_hooks.py:25,45catchsubprocess.TimeoutExpired, log"… timed out — treating as failed", andreturn False— i.e. the verify path converts a build timeout into a returncode failure and never emits a rawTimeoutExpired … timed out after N secondsstring. So the regex cannot fire for themise run buildverify path it names; it only fires for some other, unwrappedrun_cmdtimeout (clone/setup/etc.).
The classification itself (TIMEOUT, retryable: true, not "Unexpected error") is a real improvement and worth keeping — the defect is the operator-facing copy + the comment. Two acceptable fixes: (a) wire verify_build/verify_lint to read BUILD_VERIFY_TIMEOUT_S and pass it as timeout= (makes the remedy true), or (b) reword the comment to describe the pattern generically ("any uncaught run_cmd TimeoutExpired") and drop/soften the BUILD_VERIFY_TIMEOUT_S promise in the remedy. Either closes it; (b) is the smaller change if the wiring is out of scope. Please also reconcile the "600s default" here vs. the "1800s default" comment at ecs-agent-cluster.ts while you're in there.
Non-blocking (nits)
-
N1 —
errorClassomitted on the new pattern.error-classifier.ts:446-452is the only entry inPATTERNSthat omitserrorClass. This is functionally fine (absent ⇒ defaults toUSER,isTransientError()false ⇒ no auto-retry — which is arguably correct: a slow build re-run unchanged just times out again). But it's implicit against the file's own "New PATTERNS should always set it" convention (line 80-82). AdderrorClass: ErrorClass.USERwith a one-line comment stating the no-auto-retry intent, so a future maintainer doesn't "fix" it to TRANSIENT and start auto-retrying doomed builds. (I disagree with the reviewer suggestion to make it TRANSIENT — auto-retrying an unchanged slow build is wrong.) -
N2 — Regex misses float-formatted, non-"build" timeouts.
error-classifier.ts:445:timed out after \d+ ?swon't match… after 600.0 seconds. In practice this is theoretical —run_cmd'stimeoutis int-typed so Python always rendersN seconds, and the600.0/buildcase still matches via the second alternative. Cheap hardening:\d+(?:\.\d+)? ?s(econds)?. (Flagging because an agent called this a blocker — it isn't.) -
N3 — One misleading test gives false confidence.
agent/tests/test_shell.py:227test_benign_zero_errors_line_not_surfaced_as_failureuseseslint: 0 errors, 0 warnings, which matches no entry in_FAILURE_LINE_MARKERS, so the_FAILURE_LINE_NOISEallowlist guard (shell.py:88-90) is never exercised — the test passes identically if the allowlist is deleted. The genuinely uncovered branch: a line that hits a marker and a noise term (e.g.error: produced 0 errors after retry) positioned outside the tail window. Not blocking, but worth a real assertion since the allowlist is load-bearing. -
N4 — Truncation cap uncovered.
shell.py_MAX_SURFACED_FAILURE_LINES(40) truncation branch is never reached by the tests (the coverage-table test surfaces ~16 lines). Defensive-only; low priority. A >40-marker-line test would pin it. -
N5 (carried, waived by author) — Broad
_FAILURE_LINE_MARKERSsubstrings (error:,failed). Logging-only, capped, redacted, allowlisted; author's "add to allowlist rather than narrow markers" reasoning is sound. Awareness only.
Merge-coordination note (not a #597 defect)
#613 is still OPEN and edits the same agent/Dockerfile RUN line — the two will textually conflict. #597's hunk is the superset (it adds && claude --version to hard-gate the image build). Whoever merges first wins the line; the other rebases. Suggest closing #613 in favor of this, or dropping #597's Dockerfile hunk on rebase if #613 lands first. (The author already flagged this in a PR comment — captured here for the record.)
What's solid (verified, no action)
shell.py_surface_failure_lines+ stdout-on-failure logging —silent-failure-hunter+ my own trace confirm the change is purely additive: theif result.stdout:block sits beforeif check:, andraise RuntimeError(...)(line 282) is byte-for-byte unchanged — no raising path became logged-only.return out or tailguarantees a failure never logs nothing (empty-stdout branch is unreachable behindif result.stdout:). stdout isredact_secrets-wrapped (double-redacted vialog()). Correct fix for the ABCA-662 interleaved-DAG tail problem.- Dockerfile
install.cjsre-run +claude --versiongate — accurate comment, correct RUN; a broken postinstall shim now failsdocker buildinstead of every task. Local exec check, no network/auth. Strictly better than restore-and-hope. - #599 overlap — its
Exec format errorclassifier is complementary (classification) to this PR's Dockerfile prevention; no shadowing. The new TIMEOUT pattern is the firsttimed outmatcher in the array and sits beforepoll timeout exceeded— no ordering conflict.
Governance
Approved backing issue #610 (bug+approved) covers this error-legibility work — governance satisfied. Branch pr/agent-build-robustness doesn't encode the issue number (ADR-003 branch-name convention) — a de-facto-waived nit, not a blocker.
CI
All green (title, CodeQL ×3, dead-code advisory, secrets/deps/actions, build (agentcore)); MERGEABLE but BLOCKED on required review. No CDK construct/stack change → bootstrap synth-coverage N/A.
Review agents run
code-reviewer ✓ · silent-failure-hunter ✓ · pr-test-analyzer ✓ · comment-analyzer ✓. Omitted: type-design-analyzer (no new types — one regex pattern object + one module-level helper), security-review/IAM (no IAM/Cedar/network/secrets surface — the only security-adjacent line, secret redaction of newly-surfaced stdout, was checked by silent-failure-hunter and holds; note redact_secrets doesn't mask AWS keys, a pre-existing gap this PR widens slightly — worth a follow-up, out of scope here).
Human heuristics
- Proportionality — pass. A regex pattern + a ~30-line helper for a real, recurring diagnosis gap; no over-abstraction.
- Coherence — pass. Changes land in the right packages (
agent/runtime,cdk/classifier) per the routing table; consistent with existingPATTERNS/run_cmdidioms. - Clarity — concern (B1): the TIMEOUT remedy surfaces a plausible-but-misleading action (a dead knob) — exactly the AI004 "hides failure behind a plausible default" failure mode, here inverted into a plausible-but-wrong remedy.
- Appropriateness — pass. Tests assert on observed log output (contract), not internal calls; the MIDDLE-line test genuinely reproduces the ABCA-662 root cause. (One misleading test — N3.)
🤖 Generated with Claude Code
isadeks
left a comment
There was a problem hiding this comment.
Review — build/verify failure diagnosis (#597)
Re-reviewed against clean main (both stack parents #503/#596 merged; #599 also merged). Full diff is the 5 files described. I verified the code paths by reading them at the current head; agent suite 29/29 green, build (agentcore) green.
Overview. Four independent, incident-tied robustness fixes to the agent's build/verify path: (1) classify a build/verify TimeoutExpired as a user-actionable TIMEOUT instead of "Unexpected error"; (2) re-run install.cjs + claude --version in the Dockerfile so a broken claude-code shim fails the image build not every task; (3) log stdout on run_cmd failure; (4) surface the failing line from the middle of an interleaved parallel-DAG log. Each maps to a live incident (ABCA-662/667). Scope is tight and proportional.
Confirming @scottschreckengaust's blocker — B1 is correct (verified independently)
I traced both halves against the code at head; both hold:
BUILD_VERIFY_TIMEOUT_Sis read nowhere inagent/. It's only set in CDK (ecs-agent-cluster.ts:191= 3600) and asserted in a test.verify_build/verify_lint(post_hooks.py:19,39) callrun_cmd(["mise","run","build"], check=False)with notimeout=, so they userun_cmd's hardcodedtimeout: int = 600(shell.py:244). So the new remedy — "an admin can raiseBUILD_VERIFY_TIMEOUT_S" (error-classifier.ts:450) — points the operator at a lever wired to nothing.- The comment's causal story is contradicted by the code.
verify_build/verify_lintcatchsubprocess.TimeoutExpired(post_hooks.py:25,45), log "timed out — treating as failed", andreturn False— they convert a build timeout into a returncode failure and never emit the rawTimeoutExpired … timed out after N secondsstring. So the new regex cannot fire for themise run buildverify path the comment names as its ABCA-667 origin; it only matches some other unwrappedrun_cmdtimeout (clone/setup).
On a repo whose backing issue (#610) is "honest failure classification," shipping a remedy that misdirects and a comment the code contradicts is a genuine contract defect, not a nit — agreed it blocks. The classification itself (TIMEOUT / retryable: true / not "Unexpected error") is a real improvement worth keeping.
Fix — I'll take Scott's option (b) plus reconciliation (smaller than wiring the env var, which is arguably out of scope for this PR):
- Reword the comment to describe the pattern generically ("any uncaught
run_cmdTimeoutExpired", e.g. clone/setup), dropping the ABCA-667-mise-build narrative. - Soften the remedy: drop the
BUILD_VERIFY_TIMEOUT_Spromise (or make it conditional/"if wired") since nothing reads it; keep "retry / move to the larger ECS build compute." - Reconcile the "600s default" here vs the "1800s" comment at
ecs-agent-cluster.tswhile in there. - (If wiring
verify_build/verify_lintto readBUILD_VERIFY_TIMEOUT_Sand passtimeout=is preferred, that makes the remedy true and is the better long-term fix — but it's a behavior change beyond this PR's stated scope. I'll do (b) unless a maintainer wants (a).)
Nits — all verified accurate, will address
- N1 — the new pattern is the only
PATTERNSentry omittingerrorClass(confirmed: 28 classification blocks, 27 witherrorClass), against the file's own convention. Absent ⇒USER+ no auto-retry, which is correct here (auto-retrying an unchanged slow build just times out again). Will adderrorClass: ErrorClass.USERwith a comment stating the deliberate no-auto-retry intent, so nobody "fixes" it to TRANSIENT. - N3 —
test_benign_zero_errors_line_not_surfaced_as_failureuseseslint: 0 errors, 0 warnings, which matches no_FAILURE_LINE_MARKERSentry (error:≠errors), so the_FAILURE_LINE_NOISEallowlist guard is never exercised — the test passes even if the allowlist is deleted. Will add a real case: a line hitting both a marker and a noise term, outside the tail window, asserting it's dropped. - N2 (float-format regex), N4 (untested 40-line truncation cap) — low-priority hardening; will fold N2's
\d+(?:\.\d+)?in with B1 and add an N4 test.
Solid (verified, no action)
_surface_failure_lines+ stdout-on-failure logging is purely additive: theif result.stdout:block precedesif check:and theraise RuntimeError(...)is unchanged — no raising path became logged-only;return out or tailguarantees a failure never logs nothing; stdout isredact_secrets-wrapped. Correct fix for the interleaved-DAG tail problem.- Dockerfile
install.cjs+claude --versiongate — accurate comment, correct RUN, local-exec only. Strictly better than restore-and-hope. - #599 overlap is complementary (its
Exec format errorclassifier vs this PR's Dockerfile prevention); the new TIMEOUT pattern sits beforepoll timeout exceeded, no ordering conflict.
Coordination
- #613 still edits the same
DockerfileRUN line — this PR's hunk is the superset (adds&& claude --version). Whoever merges first wins; the other rebases. I'll close #613 in favor of this or drop this hunk if #613 lands first. - Follow-up (out of scope):
redact_secretsdoesn't mask AWS access keys — a pre-existing gap this PR's stdout-surfacing widens slightly. Worth a separate issue.
Plan: pushing a fix commit for B1 (option b + reconcile the 600/1800 comment) + N1 + N3, with N2/N4 folded in. Thanks @scottschreckengaust — the B1 catch is right, and I appreciate that you re-verified against real code rather than trusting the agent pass.
…1-N4) Scott's re-review (vs clean main) — one blocker + nits, all in the error-legibility feature this PR ships. - B1 (blocker): the new TIMEOUT classification pointed operators at BUILD_VERIFY_TIMEOUT_S — but NOTHING in agent/ reads it on this branch (verify_build/verify_lint call run_cmd with no timeout=, so they use the 600s default; the env is only SET in CDK). And the anchoring comment's 'live-caught on ABCA-667 via mise run build' narrative is contradicted by the code: the verify path CATCHES TimeoutExpired and returns a plain failure, so the regex never fires for that path. Took Scott's option (b): reworded the comment to describe the real trigger (any UNCAUGHT run_cmd TimeoutExpired — clone/setup), and softened the remedy to a generic 'retry / contact admin' with no dead-knob promise. (The actual BUILD_VERIFY_TIMEOUT_S wiring lives on the ECS-substrate track, not this main-bound branch — out of scope here.) Also reconciled the ecs-agent-cluster.ts comment to say the env is provisioned ahead of its consuming wiring rather than currently effective. - N1: added explicit errorClass: ErrorClass.USER with a comment stating the no-auto-retry intent (re-running an unchanged slow build just times out again; must NOT be 'fixed' to TRANSIENT). - N2: hardened the regex to match float seconds (\d+(?:\.\d+)?). - N3: replaced the false-confidence allowlist test with a real one — a line that hits a marker (error:) AND a noise term (0 errors), placed outside the tail window so it's only reachable via the marker scan; deleting _FAILURE_LINE_NOISE now fails the test. - N4: added a >40-marker-line test that pins the truncation-cap breadcrumb. Gates: cdk 2439 tests + synth + eslint; agent 1278 tests + ruff + ty.
|
Addressed all of it in B1 (blocker) — took your option (b). You're right:
N1 — added N2 — hardened the regex: N3 — replaced the false-confidence test. The old one used N4 — added a #613 Dockerfile coordination — noted; #597's hunk is the superset (adds Gates: cdk 2439 tests + synth + eslint green; agent 1278 tests + ruff + ty green. Pushing re-set the review to REVIEW_REQUIRED — ready for your re-look. |
scottschreckengaust
left a comment
There was a problem hiding this comment.
Review — Principal Architect (re-review of the B1 fix, 7e54b338)
My prior CHANGES_REQUESTED raised one blocker (B1) + nits N1–N4. I re-verified the fix commit 7e54b338 against real code (worktree at PR head, clean main base) rather than trusting the commit message — read error-classifier.ts, shell.py, post_hooks.py, and the two test files at head, re-ran the suites, and re-ran comment-analyzer on the reworded comments. B1 is genuinely resolved and every nit is addressed.
Verdict: Approve
The error-legibility feature now tells the truth — which, on a branch backed by #610 ("honest failure classification"), is exactly the bar.
B1 — resolved (option b, verified)
error-classifier.ts:436-460: the anchoring comment no longer claims the mise run build verify path as its ABCA-667 origin. It now correctly states the regex matches only uncaught run_cmd timeouts (clone/setup) and that the verify path (post_hooks.py:25,45) catches TimeoutExpired and returns a plain failure, so the pattern never fires for it. The remedy dropped the dead-knob promise (BUILD_VERIFY_TIMEOUT_S is read nowhere in agent/ — re-confirmed by grep) and now reads "retry / contact admin." ecs-agent-cluster.ts:186-194 reconciled: the misleading "1800s default" is gone, replaced with an honest "provisioned ahead of the consuming wiring" note (grep confirms nothing reads the env; verify still uses run_cmd's 600s default). comment-analyzer independently confirmed all reworded comments are accurate against the code.
Nits — all addressed
- N1 —
error-classifier.ts:456-459:errorClass: ErrorClass.USERnow set explicitly, with a comment stating the deliberate no-auto-retry intent (re-running an unchanged slow build just times out again — must not be "fixed" to TRANSIENT). Matches the file's own convention. - N2 —
error-classifier.ts:449: regex hardened to\d+(?:\.\d+)? ?s(econds)?— float-formatted seconds now match. - N3 —
test_shell.py:271:test_marker_line_with_noise_term_is_filtered_by_allowlistis now a real test of the load-bearing_FAILURE_LINE_NOISEallowlist — a line hitting both a marker (error:) and a noise term (0 errors), placed outside the tail window so it's only reachable via the marker scan. Deleting the allowlist now fails the test (verified the guard atshell.py:217-219is what it exercises). - N4 —
test_shell.py:290:test_surfaced_failure_lines_are_capped_and_truncation_markedpins the_MAX_SURFACED_FAILURE_LINEScap (40) + the… (more failure lines truncated)breadcrumb — the branch (shell.py:221-223) no prior test reached.
Tests & CI
- Ran locally at PR head:
agent/tests/test_shell.py→ 31 passed;cdk/test/handlers/shared/error-classifier.test.ts→ 77 passed (the isolated-run coverage-threshold warning is a single-file artifact, not a failure). - CI all green: CodeQL ×3, dead-code advisory, secrets/deps/actions, title,
build (agentcore).MERGEABLE. - No CDK construct/stack logic change (the
ecs-agent-cluster.tshunk is comment-only) → bootstrap synth-coverage N/A.
Review agents run
comment-analyzer ✓ (re-run on the fix — the delta is almost entirely comment rewording, its exact scope; all claims verified accurate). The full panel (code-reviewer, silent-failure-hunter, pr-test-analyzer) ran on my prior review of the same diff; the fix commit's executable delta is one regex char-class + one errorClass line, both re-verified by hand here. Omitted: type-design-analyzer (no new types), security-review/IAM (no IAM/Cedar/network/secrets surface).
Human heuristics
- Proportionality — pass. Comment/copy fix + two real tests; no over-engineering. Correctly declined to wire
BUILD_VERIFY_TIMEOUT_S(that behavior change belongs on the ECS-substrate track, not thismain-bound branch). - Coherence — pass. Changes stay in the right packages; consistent with existing
PATTERNS/run_cmdidioms. - Clarity — pass (was the B1 concern). The remedy no longer misdirects to a dead knob; comments now match the code.
- Appropriateness — pass. N3/N4 tests assert the load-bearing behavior (allowlist filtering, truncation cap), not just what the code happens to do.
Governance
Backing issue #610 (bug+approved) covers this work — satisfied. Branch pr/agent-build-robustness doesn't encode the issue number (ADR-003) — a de-facto-waived nit, not a blocker.
Coordination (not a #597 defect)
#613 is still OPEN and edits the same agent/Dockerfile RUN line; #597's hunk is the superset (adds && claude --version). Whoever merges first wins the line; the other rebases — author already committed to closing #613 in favor of this or dropping the hunk on rebase.
Nice work — the fix does exactly what "honest failure classification" asks. Approving.
🤖 Generated with Claude Code
Summary
Platform-agnostic robustness fixes for the agent's build/verify path, surfaced dogfooding ABCA-on-ABCA on the ECS build box but not ECS-specific — they improve failure diagnosis for any repo/substrate.
What changed
fix(errors): classify a build/verify command TIMEOUT — aTimeoutExpiredfrom a long build previously fell through to a bare "Unexpected error"; now classified as a user-actionable timeout (retry / raiseBUILD_VERIFY_TIMEOUT_S),retryable: true. Live-caught (ABCA-667).fix(agent): place the claude-code native binary at image build — claude-code@2.1.x ships an error-shim atbin/claude.exethat its postinstall replaces with the platform-native ELF; the postinstall silently fell back at build time, leaving the shim on PATH → every agent died withExec format error. Now re-runinstall.cjs+ hard-verifyclaude --versionso a broken shim fails the image build, not every task.fix(agent): log stdout onrun_cmdfailure — build-gate failures were undebuggable without the command output.fix(agent): surface the failing line from a parallel build DAG — a fan-outmise run buildfailure showed only a tail; now surfaces the line that actually names the failure.Testing
mise run buildgreen (CDK + agent + CLI); new/updated tests inerror-classifier.test.ts,test_shell.py.claude --version→ real 2.1.191, not the error shim).Notes for reviewers