test: per-directory coverage thresholds + E2E cleanup hardening

tejaskash · tejaskash · commit 677ab958a1d0 · 2026-05-05T19:06:36.000-04:00
Day 5: Coverage gate
- Add scripts/check-coverage.mjs with ratchet thresholds per source directory
  (schema 78%, operations 52%, lib 82%, aws 40%, tui/hooks 14%,
  tui/components 68%, commands 46%). Target thresholds documented in the
  report, ratchet set ~5% below current to prevent regression.
- Also check PR changed-line coverage (&gt;=50%) and post a report comment.
- Switch davelosert/vitest-coverage-report-action to file-coverage-mode: changes
  so PR comments highlight uncovered new lines.

Day 6: E2E cleanup
- teardownE2EProject() now retries deploy up to 3x and throws on failure
  instead of swallowing silently. The remove step logs but does not retry.
- Tag CDK stacks with Environment=e2e-test / CreatedAt / CreatedBy when
  AGENTCORE_E2E_TEST=1 is set. The CLI's e2e helper sets this env var;
  user projects are unaffected.
- Shrink cleanupStaleCredentialProviders() window from 30min -&gt; 5min to
  catch orphans faster.
- byo-custom-jwt.test.ts Cognito cleanup now retries 3x with 5s delay
  instead of swallowing errors on first try.
- Add docs/test-resource-sweeper-spec.md for the infra team to implement
  a scheduled sweeper (CFN stacks, credential providers, log groups, ECR).
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.yml
@@ -134,16 +134,29 @@ jobs:
 
     steps:
       - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-node@v6
+        with:
+          node-version: 20.x
+          cache: 'npm'
       - name: Download coverage artifact
         uses: actions/download-artifact@v8
         with:
           name: coverage-report
           path: coverage/
-      - name: Coverage Report
+      - name: Coverage Report (PR comment)
         uses: davelosert/vitest-coverage-report-action@v2
         with:
           json-summary-path: coverage/coverage-summary.json
           json-final-path: coverage/coverage-final.json
           vite-config-path: vitest.config.ts
-          file-coverage-mode: none
-          coverage-thresholds: '{ "lines": 50, "branches": 50, "functions": 50, "statements": 50 }'
+          file-coverage-mode: changes
+      - name: Per-directory coverage gate
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+        run: node scripts/check-coverage.mjs
diff --git a/docs/test-resource-sweeper-spec.md b/docs/test-resource-sweeper-spec.md
@@ -0,0 +1,115 @@
+# E2E Test Resource Sweeper — Specification
+
+## Purpose
+
+The e2e test suite provisions real AWS resources: CloudFormation stacks, credential providers, CloudWatch log groups, ECR images, and S3 artifacts. When a test crashes, times out, or is cancelled mid-run, its `afterAll` teardown may not execute, leaving orphaned resources behind.
+
+Orphan cost estimates from the April 2026 audit put sustained leaks at $2.5k-$10k/week depending on which stacks escape (runtimes, memory, credential providers). Teardown is now fatal on failure (PR bundled with this spec), but belt-and-suspenders dictates a scheduled sweeper that catches anything slipping through.
+
+This document specifies what a daily sweeper must do. The infra team owns implementation.
+
+## Scope
+
+The sweeper runs on a schedule (daily at 04:00 UTC recommended) in the same AWS account used by CI. It identifies and deletes resources that meet all three criteria:
+
+1. Match a well-known e2e naming or tagging pattern.
+2. Created more than N hours ago (N=4 recommended — longer than the longest e2e run).
+3. Are not part of an active workflow run (check GitHub Actions API for running e2e jobs).
+
+Dry-run mode must print what would be deleted without acting. Default to dry-run for the first two weeks.
+
+## Resources to Sweep
+
+### 1. CloudFormation Stacks
+
+**Identification:** Stacks tagged with `Environment=e2e-test` (this PR adds the tag via `AGENTCORE_E2E_TEST=1` env var in `src/assets/cdk/bin/cdk.ts`). Also match by stack name prefix `AgentCore-E2e*` as a fallback for older stacks.
+
+**Action:** `DeleteStack`. Watch for `DELETE_FAILED` and surface to the on-call rotation — some failures (S3 buckets with objects, ENIs held by Lambda) need manual intervention.
+
+**API calls:**
+- `cloudformation:DescribeStacks` (paginate, filter by tag)
+- `cloudformation:DeleteStack`
+- `cloudformation:DescribeStackEvents` (on DELETE_FAILED for context)
+
+### 2. API Key Credential Providers
+
+**Identification:** Providers named with the `E2e` prefix and `createdTime` older than N hours.
+
+**Action:** `DeleteApiKeyCredentialProvider`. Silent failures acceptable — the CLI already logs these via `cleanupStaleCredentialProviders()`.
+
+**API calls:**
+- `bedrock-agentcore-control:ListApiKeyCredentialProviders` (paginate)
+- `bedrock-agentcore-control:DeleteApiKeyCredentialProvider`
+
+### 3. CloudWatch Log Groups
+
+**Identification:** Log groups under `/aws/bedrock-agentcore/runtimes/E2e*` and `/aws/codebuild/AgentCore-E2e*`.
+
+**Action:** `DeleteLogGroup` for groups older than N hours. Alternatively, set a retention policy of 3 days on all matched groups and let CloudWatch expire data. Retention is safer — deletion drops diagnostic context if someone is debugging a test failure.
+
+**API calls:**
+- `logs:DescribeLogGroups` (filter by prefix, paginate)
+- `logs:PutRetentionPolicy` (preferred) or `logs:DeleteLogGroup`
+
+### 4. ECR Repositories
+
+**Identification:** Repositories tagged with `Environment=e2e-test` or named with the `agentcore-e2e-*` prefix.
+
+**Action:** Delete images older than N hours. Keep the repository itself — CDK recreates images on every deploy, so repo deletion causes churn. Image cleanup is sufficient.
+
+**API calls:**
+- `ecr:DescribeRepositories` (filter by tag or name)
+- `ecr:ListImages`
+- `ecr:BatchDeleteImage`
+
+### 5. S3 (CDK Bootstrap Bucket)
+
+**Identification:** The bootstrap bucket (`cdk-*-assets-*`) is shared across all deploys in the account. Don't delete the bucket or its tagged objects — CDK uses content-hashed object keys and expects them to persist.
+
+**Recommendation:** Apply an S3 lifecycle policy to the bootstrap bucket: transition objects to Intelligent-Tiering after 30 days, expire non-current versions after 90 days. Do this once via Terraform/CLI, not via the sweeper.
+
+## Workflow Structure
+
+GitHub Actions workflow (`.github/workflows/e2e-sweeper.yml`) with:
+
+```yaml
+on:
+  schedule:
+    - cron: '0 4 * * *'  # Daily at 04:00 UTC
+  workflow_dispatch:
+    inputs:
+      dry-run:
+        type: boolean
+        default: true
+```
+
+Permissions: use the same OIDC role that e2e tests use, but with delete permissions for the resources above. Store the role ARN in `AWS_E2E_SWEEPER_ROLE_ARN`.
+
+Steps:
+
+1. Configure AWS credentials (OIDC).
+2. Check for running e2e jobs via `gh api /repos/aws/agentcore-cli/actions/runs?status=in_progress`. If any e2e workflow is running, skip the sweep (or narrow age threshold to 24 hours).
+3. Run sweep script (Node or Python) against each resource category.
+4. Post a summary to a Slack channel (resource counts deleted per category, failures).
+5. On any resource with repeated delete failures (>3 runs in a row), open a GitHub issue.
+
+## Safety Rails
+
+- **Hard age floor:** never delete anything younger than 2 hours, even if the script says to.
+- **Account allow-list:** the script must fail closed if `AWS_ACCOUNT_ID` is not in the expected list (CI account only).
+- **Kill switch:** check for a `SWEEPER_DISABLED` repo variable before running. On-call can flip this if the sweeper misbehaves.
+- **Rate limits:** cap deletes per category at 50 per run to avoid runaway behavior.
+
+## Implementation Order
+
+1. Start with CloudFormation stack sweeping (highest $ impact). Run in dry-run for one week.
+2. Add credential provider sweeping (already scoped by prefix, low risk).
+3. Add log group retention policies (set-and-forget, no scheduled action needed).
+4. Add ECR image cleanup (low $ impact; deferrable).
+5. Enable live deletes after two weeks of clean dry-run output.
+
+## References
+
+- CDK stack tagging: `src/assets/cdk/bin/cdk.ts` (tags applied when `AGENTCORE_E2E_TEST=1`)
+- Credential provider cleanup: `e2e-tests/e2e-helper.ts#cleanupStaleCredentialProviders`
+- E2E teardown: `e2e-tests/e2e-helper.ts#teardownE2EProject` (throws on repeated failure as of this PR)
diff --git a/e2e-tests/byo-custom-jwt.test.ts b/e2e-tests/byo-custom-jwt.test.ts
@@ -48,7 +48,36 @@ const region = process.env.AWS_REGION ?? 'us-east-1';
  * Run the local CLI build without skipping install (needed for deploy).
  */
 function runLocalCLI(args: string[], cwd: string): Promise<RunResult> {
-  return runCLI(args, cwd, { skipInstall: false });
+  return runCLI(args, cwd, {
+    skipInstall: false,
+    env: {
+      AGENTCORE_E2E_TEST: '1',
+      AGENTCORE_E2E_CREATOR: process.env.AGENTCORE_E2E_CREATOR ?? 'github-actions',
+    },
+  });
+}
+
+async function deleteCognitoResourceWithRetry(
+  label: string,
+  op: () => Promise<unknown>,
+  attempts = 3,
+  delayMs = 5000
+): Promise<void> {
+  for (let attempt = 1; attempt <= attempts; attempt++) {
+    try {
+      await op();
+      return;
+    } catch (err) {
+      const name = (err as { name?: string }).name ?? 'Unknown';
+      const msg = (err as { message?: string }).message ?? String(err);
+      if (attempt === attempts) {
+        console.error(`[cognito-cleanup] ${label} failed after ${attempts} attempts: [${name}] ${msg}`);
+        return;
+      }
+      console.warn(`[cognito-cleanup] ${label} attempt ${attempt}/${attempts} failed: [${name}] — retrying`);
+      await new Promise(resolve => setTimeout(resolve, delayMs));
+    }
+  }
 }
 
 describe.sequential('e2e: BYO agent with CUSTOM_JWT auth', () => {
@@ -201,21 +230,15 @@ describe.sequential('e2e: BYO agent with CUSTOM_JWT auth', () => {
 
     // ── Delete Cognito resources ──
     if (userPoolId) {
-      try {
-        await cognitoClient.send(new DeleteResourceServerCommand({ UserPoolId: userPoolId, Identifier: 'agentcore' }));
-      } catch {
-        /* best-effort */
-      }
-      try {
-        await cognitoClient.send(new DeleteUserPoolDomainCommand({ UserPoolId: userPoolId, Domain: domainPrefix }));
-      } catch {
-        /* best-effort */
-      }
-      try {
-        await cognitoClient.send(new DeleteUserPoolCommand({ UserPoolId: userPoolId }));
-      } catch {
-        /* best-effort */
-      }
+      await deleteCognitoResourceWithRetry('DeleteResourceServer', () =>
+        cognitoClient.send(new DeleteResourceServerCommand({ UserPoolId: userPoolId, Identifier: 'agentcore' }))
+      );
+      await deleteCognitoResourceWithRetry('DeleteUserPoolDomain', () =>
+        cognitoClient.send(new DeleteUserPoolDomainCommand({ UserPoolId: userPoolId, Domain: domainPrefix }))
+      );
+      await deleteCognitoResourceWithRetry('DeleteUserPool', () =>
+        cognitoClient.send(new DeleteUserPoolCommand({ UserPoolId: userPoolId }))
+      );
     }
 
     // ── Clean up temp directory ──
diff --git a/e2e-tests/e2e-helper.ts b/e2e-tests/e2e-helper.ts
@@ -308,7 +308,10 @@ export function createE2ESuite(cfg: E2EConfig) {
 export { hasAws, baseCanRun };
 
 export function runAgentCoreCLI(args: string[], cwd: string): Promise<RunResult> {
-  return spawnAndCollect('agentcore', args, cwd);
+  return spawnAndCollect('agentcore', args, cwd, {
+    AGENTCORE_E2E_TEST: '1',
+    AGENTCORE_E2E_CREATOR: process.env.AGENTCORE_E2E_CREATOR ?? 'github-actions',
+  });
 }
 
 // TODO: Replace with `agentcore add target` once the CLI command is re-introduced
@@ -347,7 +350,7 @@ async function deleteCredentialProvider(client: BedrockAgentCoreControlClient, n
  * Runs in beforeAll to prevent accumulation from previous runs that
  * crashed or timed out before their afterAll teardown could execute.
  */
-export async function cleanupStaleCredentialProviders(maxAgeMs: number = 30 * 60 * 1000): Promise<void> {
+export async function cleanupStaleCredentialProviders(maxAgeMs: number = 5 * 60 * 1000): Promise<void> {
   const region = process.env.AWS_REGION ?? 'us-east-1';
   const client = new BedrockAgentCoreControlClient({ region });
   const cutoff = new Date(Date.now() - maxAgeMs);
@@ -365,17 +368,45 @@ export async function cleanupStaleCredentialProviders(maxAgeMs: number = 30 * 60
 }
 
 export async function teardownE2EProject(projectPath: string, agentName: string, modelProvider: string): Promise<void> {
-  await spawnAndCollect('agentcore', ['remove', 'all', '--json'], projectPath);
-  const result = await spawnAndCollect('agentcore', ['deploy', '--yes', '--json'], projectPath);
-  if (result.exitCode !== 0) {
-    console.log('Teardown stdout:', result.stdout);
-    console.log('Teardown stderr:', result.stderr);
+  const failures: string[] = [];
+
+  const removeResult = await runAgentCoreCLI(['remove', 'all', '--json'], projectPath);
+  if (removeResult.exitCode !== 0) {
+    console.error(`[teardown] remove all failed (exit ${removeResult.exitCode})`);
+    console.error('[teardown] remove stdout:', removeResult.stdout);
+    console.error('[teardown] remove stderr:', removeResult.stderr);
+    failures.push(`remove all: exit ${removeResult.exitCode}`);
+  }
+
+  const MAX_DEPLOY_ATTEMPTS = 3;
+  let deploySucceeded = false;
+  for (let attempt = 1; attempt <= MAX_DEPLOY_ATTEMPTS; attempt++) {
+    const result = await runAgentCoreCLI(['deploy', '--yes', '--json'], projectPath);
+    if (result.exitCode === 0) {
+      deploySucceeded = true;
+      if (attempt > 1) console.error(`[teardown] deploy succeeded on attempt ${attempt}`);
+      break;
+    }
+    console.error(`[teardown] deploy attempt ${attempt}/${MAX_DEPLOY_ATTEMPTS} failed (exit ${result.exitCode})`);
+    console.error('[teardown] deploy stdout:', result.stdout);
+    console.error('[teardown] deploy stderr:', result.stderr);
+    if (attempt < MAX_DEPLOY_ATTEMPTS) {
+      await new Promise(resolve => setTimeout(resolve, 15000));
+    }
   }
+  if (!deploySucceeded) {
+    failures.push(`deploy teardown failed after ${MAX_DEPLOY_ATTEMPTS} attempts`);
+  }
+
   if (modelProvider !== 'Bedrock' && agentName) {
     const region = process.env.AWS_REGION ?? 'us-east-1';
     const client = new BedrockAgentCoreControlClient({ region });
     await deleteCredentialProvider(client, `${agentName}${modelProvider}`);
   }
+
+  if (failures.length > 0) {
+    throw new Error(`E2E teardown failed: ${failures.join('; ')}`);
+  }
 }
 
 export async function dumpImportDebugInfo(
diff --git a/scripts/check-coverage.mjs b/scripts/check-coverage.mjs
diff --git a/src/assets/cdk/bin/cdk.ts b/src/assets/cdk/bin/cdk.ts
