Skip to content

Commit ed7f9f4

Browse files
fix(ci): remove broken integ teardown safety net + map stack name to commit sha (#567)
* fix(ci): remove broken 'Ensure stack torn down' safety net from integ workflow (#566) The if: always() backstop ran raw aws cloudformation delete-stack / wait stack-delete-complete as the GitHub OIDC role, which has no cloudformation:DeleteStack or DescribeStacks permission. Both calls AccessDenied on every run; the un-guarded wait exits 255 and fails the job even when the integ test passed and integ-runner already destroyed the stack (e.g. run 28892156232: SUCCESS in 291s, then teardown-step failure). integ-runner remains the teardown path (cdk destroy on success and failure via the CDK bootstrap roles). Stranded-stack reclaim for cancelled/crashed runs is owned by #400 (non-blocking cleanup) and #72 (scheduled sweep). * feat(ci): map integ stack name to commit sha (int-<sha>) Replace the single hardcoded backgroundagent-integ stack name with a run-unique int-<short-sha> derived from the resolved head SHA, threaded into integ-runner via INTEG_STACK_NAME. The test falls back to a stable backgroundagent-integ-local name so the local mise //cdk:integ path is unchanged. A stranded stack from a cancelled/crashed run no longer collides with or head-of-line blocks a later run under one fixed name; the int- prefix marks it ephemeral for the scheduled sweep (#72). Concurrency-comment updated: run-unique names make single-concurrency a cost bound, not a correctness requirement. Partial pull-forward of #400's ephemeral-name ask, scoped to naming. * feat(ci): add run-number to integ stack name (int-<sha>-<run>) Same-commit re-runs (re-run, or dispatch + workflow_run on one SHA) now get distinct stack names, matching #400's proposed int-{short-sha}-{run} scheme. * Apply suggestions from code review Co-authored-by: Scott Schreckengaust <scottschreckengaust@users.noreply.github.com> --------- Co-authored-by: Scott Schreckengaust <scottschreckengaust@users.noreply.github.com>
1 parent 092361e commit ed7f9f4

2 files changed

Lines changed: 38 additions & 23 deletions

File tree

.github/workflows/integ.yml

Lines changed: 30 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,10 @@ on:
2929
types: [completed]
3030
workflow_dispatch: {}
3131

32-
# Only one integ run at a time against the shared account — overlapping deploys
33-
# would collide on the single hardcoded `backgroundagent-integ` stack name.
32+
# Only one integ run at a time against the shared account. Stack names are now
33+
# run-unique (`int-<sha>`, see the integ job) so this is no longer a correctness
34+
# requirement — it bounds shared-account cost/parallelism rather than preventing
35+
# a name collision.
3436
concurrency:
3537
group: cdk-integ
3638
cancel-in-progress: false
@@ -217,30 +219,36 @@ jobs:
217219
- name: Install dependencies
218220
run: yarn install --immutable
219221

220-
- name: Run integ tests (deploy → assert → destroy)
221-
run: mise //cdk:integ
222-
223-
# Safety net: integ-runner forces teardown on success and failure, but if
224-
# the run is cancelled or crashes mid-deploy the stack can be stranded in
225-
# the shared account. Delete it directly via CloudFormation so we never
226-
# leak billable resources.
222+
# Teardown: integ-runner forces `cdk destroy` on success and failure (via
223+
# the CDK bootstrap roles, which hold the CloudFormation permissions). No
224+
# raw-CLI safety net here: the GitHub OIDC role has no
225+
# cloudformation:DeleteStack/DescribeStacks, so such a step can only fail
226+
# (#566). If a cancelled/crashed run strands the stack, reclaim is the
227+
# stranded-stack cleanup owned by #400/#72 — and the run-unique name below
228+
# means a stranded stack never blocks a later run.
227229
#
228-
# NOTE: `cdk destroy backgroundagent-integ` would NOT work here — it
229-
# synthesizes the main app (src/main.ts), which does not contain the integ
230-
# stack, so it exits 0 having deleted nothing. Target the stack by its
231-
# literal CloudFormation name instead. delete-stack is idempotent (no-op if
232-
# already gone), so `|| true` only guards transient API errors.
233-
- name: Ensure stack torn down
234-
if: always()
230+
# INTEG_STACK_NAME maps the stack to the commit under test plus the run
231+
# number (`int-<sha>-<run>`), replacing the single fixed
232+
# `backgroundagent-integ` name. The run number makes same-commit re-runs
233+
# distinct; the `int-` prefix marks it ephemeral for #72's sweep. HEAD_SHA
234+
# is the resolved PR head (or dispatched ref) and RUN_NUMBER is a positive
235+
# integer, so both are already lowercase-alphanumeric — no sanitizing
236+
# needed to stay CloudFormation-valid.
237+
- name: Run integ tests (deploy → assert → destroy)
235238
env:
236-
AWS_REGION: ${{ vars.AWS_REGION || 'us-east-1' }}
237-
AWS_DEFAULT_REGION: ${{ vars.AWS_REGION || 'us-east-1' }}
239+
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
240+
RUN_NUMBER: ${{ github.run_number }}
238241
run: |
239242
set -euo pipefail
240-
aws cloudformation delete-stack --stack-name backgroundagent-integ || true
241-
# No `|| true` on the wait: a DELETE_FAILED must surface loudly so we
242-
# never silently leak billable resources in the shared account.
243-
aws cloudformation wait stack-delete-complete --stack-name backgroundagent-integ
243+
export INTEG_STACK_NAME="int-${HEAD_SHA:0:12}-${RUN_NUMBER}"
244+
echo "Integ stack name: $INTEG_STACK_NAME"
245+
mise //cdk:integ -- --all --require-approval never \
246+
-c stackName="integ-${{ github.run_id }}-${{ github.run_attempt }}" \
247+
-c github:sha="$(git rev-parse HEAD)" \
248+
-c github:ref="$(git branch --show-current || echo "detached")" \
249+
-c github:actor="${{ github.triggering_actor }}" \
250+
-c github:repository="$(gh repo view --json nameWithOwner --jq .nameWithOwner)" \
251+
-c github:clean="$([ -z "$(git status --porcelain)" ] && echo true || echo false)"
244252
245253
# Post the final integ-smoke status back to the PR head so the check flips from
246254
# pending to success/failure. Skipped for workflow_dispatch (no PR to gate).

cdk/test/integ/integ.task-api-smoke.ts

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,8 +89,15 @@ class TaskApiSmokeStack extends Stack {
8989
}
9090
}
9191

92+
// Stack name is run-unique in CI (`int-<short-sha>-<run-number>`, set by
93+
// integ.yml) so a stranded stack from a cancelled/crashed run never collides
94+
// with — or blocks — a later run under one fixed name. Falls back to a stable
95+
// local name so the `mise //cdk:integ` dev path is unchanged. The `int-` prefix
96+
// keeps it letter-led and the rest is lowercase-alphanumeric + hyphens, so the
97+
// name is CloudFormation-valid.
9298
const app = new App();
93-
const stack = new TaskApiSmokeStack(app, 'backgroundagent-integ');
99+
const stackName = process.env.INTEG_STACK_NAME || 'backgroundagent-integ-local';
100+
const stack = new TaskApiSmokeStack(app, stackName);
94101

95102
const integ = new IntegTest(app, 'TaskApiSmoke', {
96103
testCases: [stack],

0 commit comments

Comments
 (0)