Skip to content

fix(agent): build/verify failure diagnosis — timeout classify, claude-binary Exec-format, failing-line surfacing#597

Merged
isadeks merged 43 commits into
mainfrom
pr/agent-build-robustness
Jul 22, 2026
Merged

fix(agent): build/verify failure diagnosis — timeout classify, claude-binary Exec-format, failing-line surfacing#597
isadeks merged 43 commits into
mainfrom
pr/agent-build-robustness

Conversation

@isadeks

@isadeks isadeks commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Stacked on #596 (pr/ecs-substrate-hardening) → #503. Merge those first; this PR then retargets. Diff shown is against #596's branch.

Summary

Platform-agnostic robustness fixes for the agent's build/verify path, surfaced dogfooding ABCA-on-ABCA on the ECS build box but not ECS-specific — they improve failure diagnosis for any repo/substrate.

What changed

  • fix(errors): classify a build/verify command TIMEOUT — a TimeoutExpired from a long build previously fell through to a bare "Unexpected error"; now classified as a user-actionable timeout (retry / raise BUILD_VERIFY_TIMEOUT_S), retryable: true. Live-caught (ABCA-667).
  • fix(agent): place the claude-code native binary at image buildclaude-code@2.1.x ships an error-shim at bin/claude.exe that its postinstall replaces with the platform-native ELF; the postinstall silently fell back at build time, leaving the shim on PATH → every agent died with Exec format error. Now re-run install.cjs + hard-verify claude --version so a broken shim fails the image build, not every task.
  • fix(agent): log stdout on run_cmd failure — build-gate failures were undebuggable without the command output.
  • fix(agent): surface the failing line from a parallel build DAG — a fan-out mise run build failure showed only a tail; now surfaces the line that actually names the failure.

Testing

  • Full mise run build green (CDK + agent + CLI); new/updated tests in error-classifier.test.ts, test_shell.py.
  • Live-verified: the arm64-native-claude fix confirmed in the deployed image (claude --version → real 2.1.191, not the error shim).

Notes for reviewers

  • No compute-backend coupling — pure agent-runtime + error-classifier changes. Stacked here only because they were validated together with the ECS substrate work.

bgagent and others added 19 commits June 29, 2026 19:35
The ECS compute strategy inlined the full orchestrator payload (incl. the
large hydrated_context) into the AGENT_PAYLOAD container-override env var.
ECS RunTask caps the TOTAL containerOverrides blob at 8192 bytes, so any real
task was rejected before the container started:

  InvalidParameterException: Container Overrides length must be at most 8192

AgentCore is unaffected — it passes the payload in the InvokeAgentRuntime
request body, which has no comparable limit. The bug only surfaces with a
realistic hydrated payload, which is why the prior ECS smoke test (a small
Rust cargo-check, #494) didn't catch it.

Fix — stash the payload out-of-band and pass only a pointer:
- New EcsPayloadBucket construct (mirrors TraceArtifactsBucket): BLOCK_ALL,
  enforceSSL, S3_MANAGED encryption, 1-day lifecycle TTL (payloads are
  ephemeral — read once at boot). Dedicated bucket so the ECS task role's S3
  read is scoped to payloads only and can't touch attachments/traces.
- ecs-strategy: when ECS_PAYLOAD_BUCKET is set, PutObject the payload to
  <task_id>/payload.json and pass AGENT_PAYLOAD_S3_URI in the override; the
  boot command fetches+parses it via boto3. Inline AGENT_PAYLOAD remains as a
  fallback (small payloads / no bucket), so nothing regresses. deleteEcsPayload
  helper removes the object.
- orchestrate-task finalize: best-effort deleteEcsPayload for ECS tasks once
  terminal (the container has long since read it); lifecycle rule is the
  crash backstop.
- EcsAgentCluster: accept payloadBucket, inject ECS_PAYLOAD_BUCKET env, grant
  the task role READ ONLY (untrusted repo code must not write/delete payloads;
  the trusted orchestrator owns write+delete). Session-role-aware.
- task-orchestrator: ecsPayloadBucket prop → grantPut + grantDelete to the
  orchestrator; @aws-sdk/client-s3 added to bundling externals.
- agent.ts: updated the commented uncomment-to-enable ECS scaffolding to wire
  the payload bucket.

Tests: new bucket construct (TTL/SSL/block-public/autoDelete); strategy
S3-write + URI-pointer + inline fallback + deleteEcsPayload (incl. best-effort
swallow + no-op without bucket); cluster read-grant + env var + read-only
(no put/delete). Full build green.

Closes #502
Root cause of the exit-127 inert gate: the agent Dockerfile installs node/npm/
mise/uv but NOT yarn, so any repo whose build runs 'yarn install' (incl. ABCA)
hit 'yarn: command not found' and the agent hand-built a ~/bin/yarn shim every
run. corepack (ships with Node 24) installs the yarn/pnpm shims at image-build
time. Systemic — fixes every yarn repo, no per-run shim. Needs redeploy to take
effect.

(cherry picked from commit 2fd5fca)
(cherry picked from commit d5e0e64)
…ilds

AgentCore's fixed microVM envelope OOM-kills CI-parity builds (live-caught
dogfooding ABCA-on-ABCA: the ~2800-test `mise run build` killed the VM at the
build-gate step). AgentCore memory isn't tunable, so route repos that set
`compute_type: 'ecs'` to a Fargate task instead.

- agent.ts: gate EcsAgentCluster + the orchestrator ecsConfig on the
 `compute_type` deploy context (default 'agentcore'). ECS resources only
 synthesize under `--context compute_type=ecs`, so default synth (and the
 bootstrap-coverage test) stays agentcore-only. Mirrors upstream's
 context-gating of the ECS construct.
- ecs-agent-cluster.ts: size the Fargate task at 8 vCPU / 32 GB (was 2/4) —
 a valid ARM64 combo with headroom for the jest worker fleet + tsc that
 AgentCore's envelope can't give. Comment documents the live OOM.
- test: assert the new 8192/32768 sizing.

Verified: default synth has 0 ECS resources; `--context compute_type=ecs`
synth emits AWS::ECS::Cluster + TaskDefinition. Full `mise run build` green.

(cherry picked from commit 8cb7cfa)
(cherry picked from commit 632a36b)
Live-fired the full `mise run build` on the 32GB ECS task (fork dogfood):
install OK, build ran ~50 min then OOM-killed at the cap (ECS stoppedReason
"OutOfMemoryError: container killed due to memory usage", exit 137; peak
working set ~31.6 GB). Root cause: the monorepo build fans out 4 heavy jobs in
PARALLEL (agent:quality ‖ cdk:build ‖ cli:build ‖ docs:build), each with its
own worker fleet (jest, pytest, esbuild Lambda bundling) — 32 GB had no
headroom for that concurrent peak, and the ~50-min wall-clock also blew the
1800s build-gate cap.

Fix (keep the full build as the gate, give it room):
- ecs-agent-cluster.ts: Fargate 8vCPU/32GB → 16vCPU/64GB (valid ARM64 combo;
 16 vCPU admits 32–120 GB). 2× memory headroom for the parallel storm + more
 cores to shorten wall-clock. Comment records the full sizing history.
- ecs-agent-cluster.ts: inject BUILD_VERIFY_TIMEOUT_S=3600 into the container
 env so a slow-but-healthy CI-parity build isn't mis-flagged as a timeout
 (post_hooks.py reads it; default 1800 stays for AgentCore repos).
- tests: assert 16384/65536 + the new env var.

Verified: default synth ECS-free; --context compute_type=ecs synth emits
Cpu 16384 / Memory 65536 / BUILD_VERIFY_TIMEOUT_S 3600. Full build green.

(cherry picked from commit 11fb3a5)
(cherry picked from commit cd7b8eb)
…aren't dropped (ABCA-487)

The ECS boot command hand-listed ~18 run_task kwargs and silently dropped the
rest. On ECS this meant channel_source/channel_metadata never reached run_task,
so resolve_linear_api_token never ran, LINEAR_API_TOKEN was never set, and the
Linear/Jira reaction + channel MCP silently no-op'd — @bgagent tasks on ECS
posted NO 👀 reaction or anything (live-caught: ABCA-487 on the fork, RUNNING
with mcp_servers:[] and no reaction). Also dropped: build_command/lint_command
(build-gate used defaults), cedar_policies + approval_* (HITL guardrails),
base_branch/merge_branches (orchestration stacking), attachments, trace, user_id.
Same "hand-rolled partial payload copy" root as #502.

Fix — single source of truth, unit-testable:
- pipeline.run_task_from_payload(p): maps the WHOLE payload dict to run_task's
 signature — rename prompt→task_description / model_id→anthropic_model, filter
 to accepted params (unknown keys ignored, not passed as **kwargs), coerce
 issue_number/pr_number→str + max_turns→int, aws_region falls back to env.
 _RUN_TASK_PARAMS computed once at import (patch-safe).
- entrypoint exports it; ecs-strategy boot command now calls
 run_task_from_payload(p) instead of the hand-listed kwargs.
- tests: 9 agent tests (rename, channel fields ABCA-487, build/cedar/branch,
 coercion, unknown-key drop, None-drop, aws_region, signature guard) +
 ecs-strategy asserts the boot command calls the mapper and no longer hand-lists.

Full build green: agent + cdk 2852 + cli 571 + docs. Deployed to dev
(--context compute_type=ecs; agent image rebuilt).

(cherry picked from commit 777ee0e)
(cherry picked from commit 1639c27)
…th-* (ABCA-488)

A Linear/Jira-channel task resolves its per-workspace OAuth token from Secrets
Manager (bgagent-linear-oauth-<slug>) at startup to fire the 👀→✅ reaction and
drive the channel MCP. The AgentCore runtime role + orchestrator/fanout/
screenshot/linear-integration roles all have the bgagent-linear-oauth-* prefix
grant, but the ECS task role only had GetSecretValue on the GitHub token. So on
ECS the token fetch hit AccessDenied → LINEAR_API_TOKEN unset → linear_reactions
skipped AND the Linear MCP stayed needs-auth (agent couldn't post comments
either). Live-caught dogfooding on the fork (ABCA-488, ECS task 828dab35).

Fourth ECS-parity gap after #502 (payload size) + ABCA-487 (dropped channel
fields): the AgentCore path had a capability the ECS task role didn't.

Fix: grant the ECS task role secretsmanager:GetSecretValue on the
bgagent-linear-oauth-* and bgagent-jira-oauth-* prefixes (GetSecretValue only —
the container reads; the orchestrator owns refresh/PutSecretValue). Nag reason
updated + regression test asserting the prefix grant. Full build green (2853).

(cherry picked from commit 5a886bb)
(cherry picked from commit 49235d4)
#299)

Live-caught: a :decompose on an ecs-configured repo ran on ECS (fix confirmed —
no more 'ECS_CLUSTER_ARN missing') but then FAILED at delivery with
'ValueError: deliver_artifact: ARTIFACTS_BUCKET_NAME is not configured'. The
AgentCore runtime sets ARTIFACTS_BUCKET_NAME; the EcsAgentCluster task def never
got the bucket — an ECS-parity gap (same class as #502/ABCA-487). A repo-bound
artifact workflow (coding/decompose-v1) delivers its plan JSON there.

- EcsAgentCluster: new optional artifactsBucket prop → ARTIFACTS_BUCKET_NAME in
 the container env + grantReadWrite on the task role (read+write: the container
 DELIVERS the artifact, unlike the read-only payload bucket). IAM5 suppression
 extended to cover the second scoped S3 grant.
- agent.ts: pass traceArtifactsBucket.bucket (same bucket the runtime uses).
- 3 construct tests (env injected / read+write grant / omitted when absent).

2872 CDK + 571 CLI + 1184 agent green. Held on branch.

(cherry picked from commit 7617d72)
(cherry picked from commit e0dfc93)
… substrate

A repo onboarded as compute_type=ecs on a stack deployed WITHOUT --context
compute_type=ecs fails every task at session start (no ECS_* env vars). Catch
the config/deploy mismatch early + make the runtime error actionable:

- agent.ts: new ComputeSubstrate CfnOutput — 'ecs' when the gate is on, else
 'agentcore' (ecs is additive; the AgentCore runtime is always present, so an
 agentcore repo works on either).
- cli repo onboard: when --compute-type ecs, read ComputeSubstrate and refuse
 with a redeploy message if it's an explicit non-ecs value. Null (older stacks
 predating the output) is treated as unknown → proceed (runtime backstop).
- ecs-strategy: the missing-env error now names the root cause + remedy
 (redeploy with the gate, or set the repo to agentcore) instead of a bare
 env-var list.
- Tests: CDK output flips with the gate (agentcore default / ecs gated); CLI
 onboard refuses/allows/back-commpat; ecs construct unchanged.

2875 CDK + 575 CLI + 1184 agent green. Held on branch.

(cherry picked from commit cb7c8c6)
(cherry picked from commit 9048db0)
…baseline OOM

ABCA-662's pre-agent baseline build was OOM-killed (exit 137) at 64 GB: dogfooding
ABCA-on-ABCA, the full parallel `mise run build` (agent:quality ‖ cdk:build ‖
cli:build ‖ docs:build, each with its own worker fleet) peaked past 64 GB.

Each build task is memory-ISOLATED (its own Fargate microVM), so limiting
concurrency does NOT help a single over-64 GB build — only more per-task RAM or
less build parallelism does. 120 GB is the MAX Fargate admits at 16 vCPU (32–120
in 8 GB steps), so this is the clean experiment: if a build still OOMs here, the
only remaining lever is cutting the build's peak parallelism (serialize the mise
DAG / cap jest --maxWorkers) — there is no more RAM to give.

Pairs with the repo.py fix (an OOM-killed baseline is now infra, not 'already
broken'), so even if a build does hit the ceiling the verdict stays honest.
Tests updated to assert Memory 122880 (both BUILD-def assertions).

(cherry picked from commit 25e2bb1)
The inline-fallback / no-op tests in the top-of-file describe blocks assume the
OPTIONAL vars ECS_PAYLOAD_BUCKET and ECS_PLANNING_TASK_DEFINITION_ARN are ABSENT
when the module is first imported (it reads them as module-level constants). A
dev shell has neither set, so the suite passed locally — but the REAL ECS agent
container HAS ECS_PAYLOAD_BUCKET set (#502 payload bucket), so on ECS the const
was truthy and the 'no bucket → inline fallback' + 'no-op' assertions failed:

  FAIL test/handlers/shared/strategies/ecs-strategy.test.ts
    ● startSession › sends RunTaskCommand with correct params ...
    ● deleteEcsPayload without ECS_PAYLOAD_BUCKET › no-ops ...

This was the residual fork baseline-build exit-1: 2 failed / 3093 passed, only
ever reproducible inside the ECS microVM. Surfaced by the live-streaming build
log (the buffered summary had hidden it). Fix: delete both optional vars before
the top-of-file import so it's hermetic regardless of the runner's environment;
the #502/#299 describe blocks already set them via jest.isolateModules.

(cherry picked from commit 7a2bde9)
(cherry picked from commit 8267c29)
…(cdk synth on fresh clone)

A CDK-based target repo's build gate runs `cdk synth`. When the stack is wired
to a concrete env ({account, region}), CDK does a synth-time availability-zone
context lookup (ec2:DescribeAvailabilityZones). A developer box caches the result
in the gitignored cdk.context.json so synth is hermetic; the agent clones fresh,
so there is no cache and the live lookup fires. The ECS task role lacked the
action, so synth failed with:

  ERROR ...EcsAgentClusterTaskRole... is not authorized to perform:
        ec2:DescribeAvailabilityZones ...
  Synthesis finished with errors

→ a FALSE build-gate failure on code that builds fine everywhere else. Same
ECS-parity class as ABCA-488 (GetSecretValue) and F-2 (CreateEvent): a permission
the AgentCore path had but the ECS task role didn't.

Live-caught on the ABCA fork: baseline `mise run build` passed all 3095 tests
then died at `cdk synth -q`, surfaced only by the live-streaming build log.
Grant is a read-only describe (Resource:* — EC2 describe actions have no
resource-level scoping; IAM5 suppressed, no mutation/data access). +1 regression
test pins the statement.

(cherry picked from commit 3fc9060)
(cherry picked from commit 2bf12f4)
…class)

Live-caught during the fork stress smoke (ABCA-495): the ECS TaskDefTaskRole
got AccessDeniedException on bedrock-agentcore:CreateEvent for the AgentMemory
resource, so the agent's cross-task learning writes (write_task_episode /
write_repo_learnings) silently no-op'd (WARN) on EVERY ECS task — memory never
persisted on the fork's ECS-only substrate.

Same class as ABCA-488: the AgentCore runtime role gets memory access via
agentMemory.grantReadWrite(runtime) in agent.ts, and the orchestrator gets it
too, but the ECS task role never did.

Fix: EcsAgentCluster gains an optional agentMemory prop; when provided the task
role gets agentMemory.grantReadWrite (read actions + CreateEvent), scoped to the
memory ARN (synth-verified: no wildcard resource, so the existing IAM5 nag
suppression is unaffected). agent.ts passes agentMemory to the ECS cluster
alongside the existing memoryId. Omitted in isolated construct tests /
memory-less deploys → no grant (asserted).

Tests: construct asserts CreateEvent on the task role scoped to MemoryArn when
wired, and NO bedrock-agentcore grant when unwired. Full cdk build green (2883).

(cherry picked from commit c23d49fbead30ed52860e3975815e3643b22ade0)
(cherry picked from commit 44bb71e)
The //cdk:bootstrap task hint and DEPLOYMENT_ROLES.md both said to enable the ECS
compute backend with 'mise //cdk:bootstrap -- --context ComputeTypes=agentcore,ecs'.
That does NOT work: ComputeTypes is a CloudFormation *template parameter*, and this
CDK version's 'cdk bootstrap' has no way to set one — there's no --parameters flag,
and --context sets CDK construct context, not a CFN parameter. So the ECS policy
(IaCRole-ABCA-Compute-ECS, conditional on ComputeTypes containing 'ecs') was never
attached, and a --context compute_type=ecs deploy then failed with AccessDenied on
ecs:* — live-caught deploying the substrate to dev.

Correct it to the mechanism that actually works: bootstrap normally, then set the
parameter directly on the CDKToolkit stack via
'aws cloudformation update-stack ... ParameterKey=ComputeTypes,ParameterValue=agentcore,ecs'
(with a describe-stacks verify). Fixed in the mise task description + comment and
both DEPLOYMENT_ROLES.md occurrences; Starlight mirror regenerated. Drift gate green.

(cherry picked from commit a46683e)
…A-667, was 'Unexpected error')

Replication finding: a live fork coding child (ABCA-667) failed with
'TimeoutExpired: Command [mise run build] timed out after 600 seconds' — the full
~2800-test fork build exceeded the 600s cap. That's the pre-agent verify-build
subprocess timeout, which lands raw in error_message and fell through to
'Unexpected error' + generic guidance — exactly the transient-vs-real confusion
this rework targets. It's not a crash and not broken code: the build ran too long.

Add a TIMEOUT classification for  /
 → 'Build/tests didn't finish in time', user class,
retryable, remedy names the real fix (retry / raise BUILD_VERIFY_TIMEOUT_S or move
to the larger ECS build box). Ordered before the generic poll-timeout pattern.
Test uses the exact ABCA-667 shape. 60 classifier tests green.

(cherry picked from commit 6394a59)
(cherry picked from commit b9c1a94)
… (Exec format error)

claude-code@2.1.191 uses a shim (bin/claude.exe) + platform-optional-dependency
install model: its postinstall (install.cjs) is supposed to replace the shim
with the platform-native ELF binary. When that postinstall silently falls back
(e.g. it hits EACCES overwriting the shim under npm's root de-privileging), the
error-shim stays on PATH and every agent run dies at the run_agent step with
'OSError: [Errno 8] Exec format error: \'claude\'' — even though the native
arm64 binary is present in the image as an optional dep, just never wired up.
Live-caught on ABCA-659's retry: all 3 ECS runs failed this way on a freshly
rebuilt image (the prior working image ran the older single-binary 2.1.142).

Re-run install.cjs explicitly after the npm install so the native binary is
placed at build time, and hard-verify with 'claude --version' so a still-broken
shim fails the image build instead of every task at runtime.

(cherry picked from commit 7cb4a87)
(cherry picked from commit 7de778b)
…debuggable

The post-agent build gate logged only stderr on a non-zero exit. But build/test
tooling (jest, tsc, the mise task DAG) writes the ACTUAL failing-task error to
STDOUT, not stderr — stderr often carries only the runner's plan echo. So a red
`mise run build` surfaced every task STARTING in CloudWatch but never WHICH one
failed or why (ABCA-662: a 7s exit-1 with zero captured cause, indistinguishable
from a phantom, and the reason Linear showed a build-fail GitHub CI passed).

run_cmd now also tails the last 20 lines of stdout on failure (redacted — repo
build output is untrusted). 4 tests pin: stdout surfaced, tailed-not-headed,
redacted, and NOT dumped on success.

(cherry picked from commit 67d84db)
(cherry picked from commit bb578fd)
…ust a tail

The stdout-on-failure logging (67d84db) tailed the last 20 lines — enough for a
serial command, but NOT for `mise run build`, which runs 4 packages in PARALLEL
and interleaves their output. The failing task's error lands in the MIDDLE while
the tail captures whatever finished LAST (e.g. a PASSING package's coverage
table) — so the log showed a green coverage table on a red build and named
nothing (ABCA-662: three verification runs where the actual failing sub-task was
never visible in CloudWatch).

run_cmd now scans the WHOLE stdout for failure-signature lines (FAIL/✕/●/error
TS/ELIFECYCLE/coverage-threshold/mise 'no task'/'task failed'/traceback/…),
filters benign noise ('0 errors', '--no-error'), and logs those FIRST, then a
trailing tail for context — deduped, capped at 40 lines. Falls back to the tail
when nothing matches (unknown tool). This is the tooling gap that made the whole
ABCA-662 build investigation slow: every failure was diagnosable only by luck or
local repro. 5 tests incl. the mid-DAG-failure + coverage-threshold cases.

(cherry picked from commit 3fe99c9)
(cherry picked from commit aaa4f18)
@isadeks isadeks added bug Something isn't working agent-runtime Python agent container: pipeline, runner, hooks, prompts, tools, Dockerfile validation-loop Tasks related to improve the validation loop for ABCA's codebase P2 lowest priority labels Jul 14, 2026

@scottschreckengaust scottschreckengaust left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — Principal Architect (Stack A: ECS substrate, PR 3/4)

Base verified against fresh origin PR head (diffed vs #596). Mandatory agents run stack-wide (full table on the #596 review).

Verdict: Approve (with a nit)

Solid build/verify-diagnosis work, all three changes tied to live incidents:

  • _surface_failure_lines (agent/src/shell.py) — scans the WHOLE stdout for failure-signature lines, not just the tail. Correct for a parallel mise DAG that interleaves output so the failing task's line lands in the middle while the tail is a passing package's coverage table (ABCA-662). Capped at 40 + a 15-line tail, deduped, and redacted (redact_secrets) since repo build output is untrusted. The return out or tail fallback guarantees a failure never logs nothing. Verified run_cmd's check=True path still raises — the added stdout logging is purely additive, not a swallow.
  • Error-classifier TIMEOUT pattern (cdk/src/handlers/shared/error-classifier.ts) — classifies a build/verify wall-clock kill distinctly from a crash (retryable: true, user-actionable remedy). Verified it's ordered before the generic catch-all and isn't shadowed by the earlier poll timeout exceeded rule.
  • Dockerfile claude install.cjs re-run + claude --version — a real fix: claude-code@2.1.x ships an error-shim that its postinstall can silently leave in place, dying at runtime with "Exec format error". Re-running the installer and hard-verifying it exec's fails the build instead of the task.

Tests (test_shell.py::TestRunCmdFailureLogging, error-classifier.test.ts) cover the key cases well: stdout-logged-on-failure, mid-DAG line surfaced, coverage-threshold surfaced, benign "0 errors" not falsely elevated, redaction, success-doesn't-dump, no-marker→tail fallback, and the timeout classified as retryable-not-Unexpected.

Non-blocking nit

  • N5 — Broad failure markers. _FAILURE_LINE_MARKERS includes broad substrings ("error:", "failed") that could surface benign advisory lines. It's logging-only, capped, redacted, and the _FAILURE_LINE_NOISE allowlist covers common false positives — so worst case is slightly noisier CloudWatch, no functional impact. Flagging for awareness only.

Governance note

Branch pr/agent-build-robustness carries no issue number and the body references only stacked PRs (ADR-003). Same governance item raised on #596 — please attach an approved issue and a conforming branch name before merge. Code is approve-ready.

CI

All green (title, CodeQL ×3, dead-code advisory, secrets/deps/actions, build (agentcore)); MERGEABLE. No CDK construct/stack change → bootstrap synth-coverage N/A.

🤖 Generated with Claude Code

The `export const ECS_PAYLOAD_OBJECT_KEY_PREFIX = ''` constant was referenced
nowhere — ecsPayloadKey() (ecs-strategy.ts) builds `<task_id>/payload.json`
independently and already documents that layout in its own docstring. The dead
export was a "prefix" whose value was '' with a docstring describing a key
layout it didn't produce, and it raised the knip dead-code count, tripping the
ratchet inside `mise run build`. Delete it; ecsPayloadKey remains the single
source of truth for the key layout.

@isadeks isadeks left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-reference: Dockerfile fix overlaps #613 (and complements #599)

Flagging a coordination point surfaced while reviewing #613 — the claude-binary Exec format error is being addressed across three open PRs:

  • This PR (#597) and #613 both append node install.cjs to the same RUN … npm --prefix … update … line in agent/Dockerfile, so they will textually conflict — both can't merge as-is. #597's version is the superset: it also adds && claude --version, hard-gating the image build so a silent postinstall fallback fails docker build rather than every task at runtime (strictly better than restore-and-hope), plus a comment explaining the shim mechanism.
  • #599 is complementary, not conflicting — it classifies the same Exec format error into a user-facing message (error-classifier.ts, disjoint file). Prevention (here/#613) + legibility (#599) are the two halves of ABCA-659.

Suggested resolution: since #597's Dockerfile hunk supersedes #613's, either close #613 in favor of this PR, or — if this PR stays blocked behind its stack (#596#503; #503 merged, #596 still open) and the exec-format break is hurting people now — let #613 merge against main and drop the Dockerfile hunk here on rebase. Whoever lands first wins the line; the other should rebase around it.

(I verified install.cjs is idempotent and exists in @anthropic-ai/claude-code@2.1.191; couldn't reproduce the npm update clobber on linux/amd64, so it looks arm64/AgentCore-specific — consistent with the live-caught reports. The && claude --version gate here is the right belt-and-braces regardless.)

…+ nits

B1 (least-privilege regression): remove props.artifactsBucket.grantReadWrite
from the ECS task role. coding/decompose-v1 delivers its plan via the assumed
SessionRole (deliverers.py -> tenant_client), scoped to artifacts/${task_id}/*,
exactly like the AgentCore runtime (whose task role likewise has NO direct
artifacts grant). The whole-bucket grant over-privileged the untrusted-code
role and broke cross-task isolation (a task could read/clobber other tasks'
artifacts/<other_id>/, traces/, attachments/). Task role keeps only the
ARTIFACTS_BUCKET_NAME env. Correct the inverted 'parity' comment and drop the
now-stale artifacts clause from the cdk-nag IAM5 reason.

Test: flip 'grants READ+WRITE on artifacts' → asserts the task role has NO S3
Put/Delete action at all (the read-only #502 payload GetObject*/List* grant is
not flagged).

N4: add lint_command to _KNOWN_ORCHESTRATOR_KEYS (sibling of build_command) so a
future 'wired build_command, forgot lint_command' contract gap WARNs instead of
dropping silently.

(N1/N2/N3 comment nits were already addressed on the branch head; B1 dead-const
ECS_PAYLOAD_OBJECT_KEY_PREFIX no longer present. B2 governance handled separately.)
isadeks added 9 commits July 17, 2026 13:49
…noise + max_turns coercion

Scott's fresh re-review (head 86ab249) is Approve-with-nits: B1 (over-privilege)
and B2 (governance) resolved; remaining asks are doc-accuracy + WARN-channel
hygiene.

- N1: rewrite the artifactsBucket JSDoc (ecs-agent-cluster.ts) that still claimed
  a read/write task-role grant the B1 fix deliberately removed — now states
  env-only wiring with delivery through the task_id-scoped SessionRole (parity
  with the AgentCore runtime role, which has no artifacts grant). The grant block
  ~30 lines below was already corrected in the B1 commit; this was its residual
  half.
- N2: same inverted-comment fix at the agent.ts wiring site — '(read+write grant
  in the construct)' → env-only + SessionRole delivery.
- N3: drop github_token_secret_arn from _KNOWN_ORCHESTRATOR_KEYS. It is ALWAYS
  present and ALWAYS resolved via the GITHUB_TOKEN_SECRET_ARN env, so listing it
  fired the known-key WARN on 100% of ECS boots — pure noise diluting the channel
  meant for genuine future contract gaps. Now falls through as a quiet foreign-key
  drop; comment records why it must not be re-added. +test asserting no WARN.
- N4: max_turns int() coercion accepted a bool (int(True)==1) and truncated a
  non-integral float (int(3.9)==3), and the comment falsely claimed it 'matches
  how every other field is handled' (it is the one non-str coercion). Now rejects
  bools + non-integral floats with a breadcrumb; valid int / int-string / int-float
  still pass. +test. Reachable only via a corrupt payload (orchestrator emits a
  real int), so cosmetic — bundled since cheap.
- N5: no-op — the Dockerfile corepack line already has no 2>/dev/null.

Gates: cdk 2355 tests + eslint + synth green; agent 1268 tests + ruff + ty green.
Non-blocking #608 test-seam gaps (finalize deleteEcsPayload, boot-uri parse,
entrypoint re-export, zero-ECS-synth assert) remain tracked, not in scope here.
Base automatically changed from pr/ecs-substrate-hardening to main July 21, 2026 19:47
@ayushtr-aws
ayushtr-aws dismissed scottschreckengaust’s stale review July 21, 2026 19:47

The base branch was changed.

@scottschreckengaust scottschreckengaust left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — Principal Architect (re-review vs. clean main)

My earlier review here is dismissed — it was diffed against #596's branch. Both parents are now merged (#503, #596), so this re-review is against a clean main base; the diff is exactly the 5 files described in the body. #599 (the complementary Exec format error classifier) also merged since — I re-verified there's no overlap/shadowing. Mandatory agents re-run: code-reviewer, silent-failure-hunter, pr-test-analyzer, comment-analyzer. All findings independently verified against real code (I did not take agent output at face value — the loudest agent finding turned out wrong; see below).

Verdict: Request changes — one narrow blocker; everything else is approve-ready

The core work is genuinely good and each change is tied to a live incident. The blocker is not a crash — it's that this PR's own feature (accurate, actionable error legibility) ships a user-facing remedy that points an operator at a lever that does nothing, under a comment whose causal narrative the code contradicts. On a repo whose backing issue is literally titled "honest failure classification" (#610, approved), that's a contract defect, not a nit.


Blocking

B1 — The new TIMEOUT remedy misdirects the operator to a non-functional knob, and its anchoring comment is contradicted by the code. cdk/src/handlers/shared/error-classifier.ts:440-450

Two linked, verified problems in the new classification block:

  1. BUILD_VERIFY_TIMEOUT_S is consumed nowhere in agent/. grep -rn BUILD_VERIFY_TIMEOUT_S agent/ returns nothing. verify_build/verify_lint (agent/src/post_hooks.py:19,39) call run_cmd(["mise","run","build"], check=False) with no timeout=, so they always use run_cmd's hardcoded timeout: int = 600 (agent/src/shell.py:244). The env var is only set (CDK ecs-agent-cluster.ts, value 3600) and asserted in a test — nothing reads it. So the remedy line "an admin can raise BUILD_VERIFY_TIMEOUT_S" tells an operator to turn a knob that isn't wired to anything.
  2. The comment's live-caught narrative is contradicted by the code. The comment (lines 438-444) says this was "live-caught on ABCA-667: the full mise run build exceeded the 600s default and surfaced as a bare 'Unexpected error'." But post_hooks.py:25,45 catch subprocess.TimeoutExpired, log "… timed out — treating as failed", and return False — i.e. the verify path converts a build timeout into a returncode failure and never emits a raw TimeoutExpired … timed out after N seconds string. So the regex cannot fire for the mise run build verify path it names; it only fires for some other, unwrapped run_cmd timeout (clone/setup/etc.).

The classification itself (TIMEOUT, retryable: true, not "Unexpected error") is a real improvement and worth keeping — the defect is the operator-facing copy + the comment. Two acceptable fixes: (a) wire verify_build/verify_lint to read BUILD_VERIFY_TIMEOUT_S and pass it as timeout= (makes the remedy true), or (b) reword the comment to describe the pattern generically ("any uncaught run_cmd TimeoutExpired") and drop/soften the BUILD_VERIFY_TIMEOUT_S promise in the remedy. Either closes it; (b) is the smaller change if the wiring is out of scope. Please also reconcile the "600s default" here vs. the "1800s default" comment at ecs-agent-cluster.ts while you're in there.


Non-blocking (nits)

  • N1 — errorClass omitted on the new pattern. error-classifier.ts:446-452 is the only entry in PATTERNS that omits errorClass. This is functionally fine (absent ⇒ defaults to USER, isTransientError() false ⇒ no auto-retry — which is arguably correct: a slow build re-run unchanged just times out again). But it's implicit against the file's own "New PATTERNS should always set it" convention (line 80-82). Add errorClass: ErrorClass.USER with a one-line comment stating the no-auto-retry intent, so a future maintainer doesn't "fix" it to TRANSIENT and start auto-retrying doomed builds. (I disagree with the reviewer suggestion to make it TRANSIENT — auto-retrying an unchanged slow build is wrong.)

  • N2 — Regex misses float-formatted, non-"build" timeouts. error-classifier.ts:445: timed out after \d+ ?s won't match … after 600.0 seconds. In practice this is theoreticalrun_cmd's timeout is int-typed so Python always renders N seconds, and the 600.0/build case still matches via the second alternative. Cheap hardening: \d+(?:\.\d+)? ?s(econds)?. (Flagging because an agent called this a blocker — it isn't.)

  • N3 — One misleading test gives false confidence. agent/tests/test_shell.py:227 test_benign_zero_errors_line_not_surfaced_as_failure uses eslint: 0 errors, 0 warnings, which matches no entry in _FAILURE_LINE_MARKERS, so the _FAILURE_LINE_NOISE allowlist guard (shell.py:88-90) is never exercised — the test passes identically if the allowlist is deleted. The genuinely uncovered branch: a line that hits a marker and a noise term (e.g. error: produced 0 errors after retry) positioned outside the tail window. Not blocking, but worth a real assertion since the allowlist is load-bearing.

  • N4 — Truncation cap uncovered. shell.py _MAX_SURFACED_FAILURE_LINES (40) truncation branch is never reached by the tests (the coverage-table test surfaces ~16 lines). Defensive-only; low priority. A >40-marker-line test would pin it.

  • N5 (carried, waived by author) — Broad _FAILURE_LINE_MARKERS substrings (error:, failed). Logging-only, capped, redacted, allowlisted; author's "add to allowlist rather than narrow markers" reasoning is sound. Awareness only.


Merge-coordination note (not a #597 defect)

#613 is still OPEN and edits the same agent/Dockerfile RUN line — the two will textually conflict. #597's hunk is the superset (it adds && claude --version to hard-gate the image build). Whoever merges first wins the line; the other rebases. Suggest closing #613 in favor of this, or dropping #597's Dockerfile hunk on rebase if #613 lands first. (The author already flagged this in a PR comment — captured here for the record.)


What's solid (verified, no action)

  • shell.py _surface_failure_lines + stdout-on-failure loggingsilent-failure-hunter + my own trace confirm the change is purely additive: the if result.stdout: block sits before if check:, and raise RuntimeError(...) (line 282) is byte-for-byte unchanged — no raising path became logged-only. return out or tail guarantees a failure never logs nothing (empty-stdout branch is unreachable behind if result.stdout:). stdout is redact_secrets-wrapped (double-redacted via log()). Correct fix for the ABCA-662 interleaved-DAG tail problem.
  • Dockerfile install.cjs re-run + claude --version gate — accurate comment, correct RUN; a broken postinstall shim now fails docker build instead of every task. Local exec check, no network/auth. Strictly better than restore-and-hope.
  • #599 overlap — its Exec format error classifier is complementary (classification) to this PR's Dockerfile prevention; no shadowing. The new TIMEOUT pattern is the first timed out matcher in the array and sits before poll timeout exceeded — no ordering conflict.

Governance

Approved backing issue #610 (bug+approved) covers this error-legibility work — governance satisfied. Branch pr/agent-build-robustness doesn't encode the issue number (ADR-003 branch-name convention) — a de-facto-waived nit, not a blocker.

CI

All green (title, CodeQL ×3, dead-code advisory, secrets/deps/actions, build (agentcore)); MERGEABLE but BLOCKED on required review. No CDK construct/stack change → bootstrap synth-coverage N/A.

Review agents run

code-reviewer ✓ · silent-failure-hunter ✓ · pr-test-analyzer ✓ · comment-analyzer ✓. Omitted: type-design-analyzer (no new types — one regex pattern object + one module-level helper), security-review/IAM (no IAM/Cedar/network/secrets surface — the only security-adjacent line, secret redaction of newly-surfaced stdout, was checked by silent-failure-hunter and holds; note redact_secrets doesn't mask AWS keys, a pre-existing gap this PR widens slightly — worth a follow-up, out of scope here).

Human heuristics

  • Proportionality — pass. A regex pattern + a ~30-line helper for a real, recurring diagnosis gap; no over-abstraction.
  • Coherence — pass. Changes land in the right packages (agent/ runtime, cdk/ classifier) per the routing table; consistent with existing PATTERNS/run_cmd idioms.
  • Clarityconcern (B1): the TIMEOUT remedy surfaces a plausible-but-misleading action (a dead knob) — exactly the AI004 "hides failure behind a plausible default" failure mode, here inverted into a plausible-but-wrong remedy.
  • Appropriateness — pass. Tests assert on observed log output (contract), not internal calls; the MIDDLE-line test genuinely reproduces the ABCA-662 root cause. (One misleading test — N3.)

🤖 Generated with Claude Code

Comment thread cdk/src/handlers/shared/error-classifier.ts Outdated
Comment thread cdk/src/handlers/shared/error-classifier.ts

@isadeks isadeks left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — build/verify failure diagnosis (#597)

Re-reviewed against clean main (both stack parents #503/#596 merged; #599 also merged). Full diff is the 5 files described. I verified the code paths by reading them at the current head; agent suite 29/29 green, build (agentcore) green.

Overview. Four independent, incident-tied robustness fixes to the agent's build/verify path: (1) classify a build/verify TimeoutExpired as a user-actionable TIMEOUT instead of "Unexpected error"; (2) re-run install.cjs + claude --version in the Dockerfile so a broken claude-code shim fails the image build not every task; (3) log stdout on run_cmd failure; (4) surface the failing line from the middle of an interleaved parallel-DAG log. Each maps to a live incident (ABCA-662/667). Scope is tight and proportional.

Confirming @scottschreckengaust's blocker — B1 is correct (verified independently)

I traced both halves against the code at head; both hold:

  1. BUILD_VERIFY_TIMEOUT_S is read nowhere in agent/. It's only set in CDK (ecs-agent-cluster.ts:191 = 3600) and asserted in a test. verify_build/verify_lint (post_hooks.py:19,39) call run_cmd(["mise","run","build"], check=False) with no timeout=, so they use run_cmd's hardcoded timeout: int = 600 (shell.py:244). So the new remedy — "an admin can raise BUILD_VERIFY_TIMEOUT_S" (error-classifier.ts:450) — points the operator at a lever wired to nothing.
  2. The comment's causal story is contradicted by the code. verify_build/verify_lint catch subprocess.TimeoutExpired (post_hooks.py:25,45), log "timed out — treating as failed", and return False — they convert a build timeout into a returncode failure and never emit the raw TimeoutExpired … timed out after N seconds string. So the new regex cannot fire for the mise run build verify path the comment names as its ABCA-667 origin; it only matches some other unwrapped run_cmd timeout (clone/setup).

On a repo whose backing issue (#610) is "honest failure classification," shipping a remedy that misdirects and a comment the code contradicts is a genuine contract defect, not a nit — agreed it blocks. The classification itself (TIMEOUT / retryable: true / not "Unexpected error") is a real improvement worth keeping.

Fix — I'll take Scott's option (b) plus reconciliation (smaller than wiring the env var, which is arguably out of scope for this PR):

  • Reword the comment to describe the pattern generically ("any uncaught run_cmd TimeoutExpired", e.g. clone/setup), dropping the ABCA-667-mise-build narrative.
  • Soften the remedy: drop the BUILD_VERIFY_TIMEOUT_S promise (or make it conditional/"if wired") since nothing reads it; keep "retry / move to the larger ECS build compute."
  • Reconcile the "600s default" here vs the "1800s" comment at ecs-agent-cluster.ts while in there.
  • (If wiring verify_build/verify_lint to read BUILD_VERIFY_TIMEOUT_S and pass timeout= is preferred, that makes the remedy true and is the better long-term fix — but it's a behavior change beyond this PR's stated scope. I'll do (b) unless a maintainer wants (a).)

Nits — all verified accurate, will address

  • N1 — the new pattern is the only PATTERNS entry omitting errorClass (confirmed: 28 classification blocks, 27 with errorClass), against the file's own convention. Absent ⇒ USER + no auto-retry, which is correct here (auto-retrying an unchanged slow build just times out again). Will add errorClass: ErrorClass.USER with a comment stating the deliberate no-auto-retry intent, so nobody "fixes" it to TRANSIENT.
  • N3test_benign_zero_errors_line_not_surfaced_as_failure uses eslint: 0 errors, 0 warnings, which matches no _FAILURE_LINE_MARKERS entry (error:errors), so the _FAILURE_LINE_NOISE allowlist guard is never exercised — the test passes even if the allowlist is deleted. Will add a real case: a line hitting both a marker and a noise term, outside the tail window, asserting it's dropped.
  • N2 (float-format regex), N4 (untested 40-line truncation cap) — low-priority hardening; will fold N2's \d+(?:\.\d+)? in with B1 and add an N4 test.

Solid (verified, no action)

  • _surface_failure_lines + stdout-on-failure logging is purely additive: the if result.stdout: block precedes if check: and the raise RuntimeError(...) is unchanged — no raising path became logged-only; return out or tail guarantees a failure never logs nothing; stdout is redact_secrets-wrapped. Correct fix for the interleaved-DAG tail problem.
  • Dockerfile install.cjs + claude --version gate — accurate comment, correct RUN, local-exec only. Strictly better than restore-and-hope.
  • #599 overlap is complementary (its Exec format error classifier vs this PR's Dockerfile prevention); the new TIMEOUT pattern sits before poll timeout exceeded, no ordering conflict.

Coordination

  • #613 still edits the same Dockerfile RUN line — this PR's hunk is the superset (adds && claude --version). Whoever merges first wins; the other rebases. I'll close #613 in favor of this or drop this hunk if #613 lands first.
  • Follow-up (out of scope): redact_secrets doesn't mask AWS access keys — a pre-existing gap this PR's stdout-surfacing widens slightly. Worth a separate issue.

Plan: pushing a fix commit for B1 (option b + reconcile the 600/1800 comment) + N1 + N3, with N2/N4 folded in. Thanks @scottschreckengaust — the B1 catch is right, and I appreciate that you re-verified against real code rather than trusting the agent pass.

…1-N4)

Scott's re-review (vs clean main) — one blocker + nits, all in the error-legibility
feature this PR ships.

- B1 (blocker): the new TIMEOUT classification pointed operators at
  BUILD_VERIFY_TIMEOUT_S — but NOTHING in agent/ reads it on this branch
  (verify_build/verify_lint call run_cmd with no timeout=, so they use the 600s
  default; the env is only SET in CDK). And the anchoring comment's 'live-caught
  on ABCA-667 via mise run build' narrative is contradicted by the code: the
  verify path CATCHES TimeoutExpired and returns a plain failure, so the regex
  never fires for that path. Took Scott's option (b): reworded the comment to
  describe the real trigger (any UNCAUGHT run_cmd TimeoutExpired — clone/setup),
  and softened the remedy to a generic 'retry / contact admin' with no dead-knob
  promise. (The actual BUILD_VERIFY_TIMEOUT_S wiring lives on the ECS-substrate
  track, not this main-bound branch — out of scope here.) Also reconciled the
  ecs-agent-cluster.ts comment to say the env is provisioned ahead of its
  consuming wiring rather than currently effective.
- N1: added explicit errorClass: ErrorClass.USER with a comment stating the
  no-auto-retry intent (re-running an unchanged slow build just times out again;
  must NOT be 'fixed' to TRANSIENT).
- N2: hardened the regex to match float seconds (\d+(?:\.\d+)?).
- N3: replaced the false-confidence allowlist test with a real one — a line that
  hits a marker (error:) AND a noise term (0 errors), placed outside the tail
  window so it's only reachable via the marker scan; deleting _FAILURE_LINE_NOISE
  now fails the test.
- N4: added a >40-marker-line test that pins the truncation-cap breadcrumb.

Gates: cdk 2439 tests + synth + eslint; agent 1278 tests + ruff + ty.
@isadeks

isadeks commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Addressed all of it in 7e54b338.

B1 (blocker) — took your option (b). You're right: BUILD_VERIFY_TIMEOUT_S is read by nothing in agent/ on this branch (verify_build/verify_lint call run_cmd with no timeout= → the hardcoded 600s), and the comment's "live-caught via mise run build" narrative is contradicted by the verify path catching TimeoutExpired itself. So:

  • Reworded the comment to describe the real trigger — an uncaught run_cmd TimeoutExpired (clone/setup/etc.), explicitly noting the verify path catches its own timeout and doesn't reach here.
  • Softened the remedy to a generic "retry — it may just be slow; if a repo's build is legitimately long its environment needs more time/capacity, contact your ABCA admin" — no dead-knob promise. The real BUILD_VERIFY_TIMEOUT_S wiring (verify passing the env as timeout=) lives on the ECS-substrate track, not this main-bound branch, so I kept it out of scope rather than balloon this PR.
  • Also reconciled the ecs-agent-cluster.ts comment (your "while you're in there"): it now says the env is provisioned ahead of its consuming wiring rather than currently effective, and drops the "see post_hooks.py BUILD_VERIFY_TIMEOUT_S" cross-ref (that symbol isn't on this branch).

N1 — added errorClass: ErrorClass.USER explicitly, with a comment stating the no-auto-retry intent (agree with you, not the earlier suggestion: auto-retrying an unchanged slow build just times out again — must not be TRANSIENT).

N2 — hardened the regex: timed out after \d+(?:\.\d+)? ?s(econds)?.

N3 — replaced the false-confidence test. The old one used eslint: 0 errors which matches no marker, so the allowlist never ran. The new test uses error: eslint produced 0 errors after autofix retry — hits marker error: and noise 0 errors — placed outside the tail window so it's only reachable via the marker scan. Deleting _FAILURE_LINE_NOISE now fails it (verified).

N4 — added a > _MAX_SURFACED_FAILURE_LINES test that asserts the … (more failure lines truncated) breadcrumb fires and the matched-scan (pre-tail) is capped.

#613 Dockerfile coordination — noted; #597's hunk is the superset (adds && claude --version). Whoever merges first wins the line; I'll rebase the other.

Gates: cdk 2439 tests + synth + eslint green; agent 1278 tests + ruff + ty green. Pushing re-set the review to REVIEW_REQUIRED — ready for your re-look.

@scottschreckengaust scottschreckengaust left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — Principal Architect (re-review of the B1 fix, 7e54b338)

My prior CHANGES_REQUESTED raised one blocker (B1) + nits N1–N4. I re-verified the fix commit 7e54b338 against real code (worktree at PR head, clean main base) rather than trusting the commit message — read error-classifier.ts, shell.py, post_hooks.py, and the two test files at head, re-ran the suites, and re-ran comment-analyzer on the reworded comments. B1 is genuinely resolved and every nit is addressed.

Verdict: Approve

The error-legibility feature now tells the truth — which, on a branch backed by #610 ("honest failure classification"), is exactly the bar.

B1 — resolved (option b, verified)

error-classifier.ts:436-460: the anchoring comment no longer claims the mise run build verify path as its ABCA-667 origin. It now correctly states the regex matches only uncaught run_cmd timeouts (clone/setup) and that the verify path (post_hooks.py:25,45) catches TimeoutExpired and returns a plain failure, so the pattern never fires for it. The remedy dropped the dead-knob promise (BUILD_VERIFY_TIMEOUT_S is read nowhere in agent/ — re-confirmed by grep) and now reads "retry / contact admin." ecs-agent-cluster.ts:186-194 reconciled: the misleading "1800s default" is gone, replaced with an honest "provisioned ahead of the consuming wiring" note (grep confirms nothing reads the env; verify still uses run_cmd's 600s default). comment-analyzer independently confirmed all reworded comments are accurate against the code.

Nits — all addressed

  • N1error-classifier.ts:456-459: errorClass: ErrorClass.USER now set explicitly, with a comment stating the deliberate no-auto-retry intent (re-running an unchanged slow build just times out again — must not be "fixed" to TRANSIENT). Matches the file's own convention.
  • N2error-classifier.ts:449: regex hardened to \d+(?:\.\d+)? ?s(econds)? — float-formatted seconds now match.
  • N3test_shell.py:271: test_marker_line_with_noise_term_is_filtered_by_allowlist is now a real test of the load-bearing _FAILURE_LINE_NOISE allowlist — a line hitting both a marker (error:) and a noise term (0 errors), placed outside the tail window so it's only reachable via the marker scan. Deleting the allowlist now fails the test (verified the guard at shell.py:217-219 is what it exercises).
  • N4test_shell.py:290: test_surfaced_failure_lines_are_capped_and_truncation_marked pins the _MAX_SURFACED_FAILURE_LINES cap (40) + the … (more failure lines truncated) breadcrumb — the branch (shell.py:221-223) no prior test reached.

Tests & CI

  • Ran locally at PR head: agent/tests/test_shell.py31 passed; cdk/test/handlers/shared/error-classifier.test.ts77 passed (the isolated-run coverage-threshold warning is a single-file artifact, not a failure).
  • CI all green: CodeQL ×3, dead-code advisory, secrets/deps/actions, title, build (agentcore). MERGEABLE.
  • No CDK construct/stack logic change (the ecs-agent-cluster.ts hunk is comment-only) → bootstrap synth-coverage N/A.

Review agents run

comment-analyzer ✓ (re-run on the fix — the delta is almost entirely comment rewording, its exact scope; all claims verified accurate). The full panel (code-reviewer, silent-failure-hunter, pr-test-analyzer) ran on my prior review of the same diff; the fix commit's executable delta is one regex char-class + one errorClass line, both re-verified by hand here. Omitted: type-design-analyzer (no new types), security-review/IAM (no IAM/Cedar/network/secrets surface).

Human heuristics

  • Proportionality — pass. Comment/copy fix + two real tests; no over-engineering. Correctly declined to wire BUILD_VERIFY_TIMEOUT_S (that behavior change belongs on the ECS-substrate track, not this main-bound branch).
  • Coherence — pass. Changes stay in the right packages; consistent with existing PATTERNS/run_cmd idioms.
  • Clarity — pass (was the B1 concern). The remedy no longer misdirects to a dead knob; comments now match the code.
  • Appropriateness — pass. N3/N4 tests assert the load-bearing behavior (allowlist filtering, truncation cap), not just what the code happens to do.

Governance

Backing issue #610 (bug+approved) covers this work — satisfied. Branch pr/agent-build-robustness doesn't encode the issue number (ADR-003) — a de-facto-waived nit, not a blocker.

Coordination (not a #597 defect)

#613 is still OPEN and edits the same agent/Dockerfile RUN line; #597's hunk is the superset (adds && claude --version). Whoever merges first wins the line; the other rebases — author already committed to closing #613 in favor of this or dropping the hunk on rebase.

Nice work — the fix does exactly what "honest failure classification" asks. Approving.

🤖 Generated with Claude Code

@isadeks
isadeks added this pull request to the merge queue Jul 22, 2026
Merged via the queue into main with commit 7ae4af3 Jul 22, 2026
8 checks passed
@isadeks
isadeks deleted the pr/agent-build-robustness branch July 22, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-runtime Python agent container: pipeline, runner, hooks, prompts, tools, Dockerfile bug Something isn't working P2 lowest priority validation-loop Tasks related to improve the validation loop for ABCA's codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants