Skip to content

fix: clean up leaked IAM groups in scheduled nuke jobs (OSS-4024)#1160

Open
james00012 wants to merge 1 commit into
masterfrom
fix/oss-4024-nuke-iam-groups
Open

fix: clean up leaked IAM groups in scheduled nuke jobs (OSS-4024)#1160
james00012 wants to merge 1 commit into
masterfrom
fix/oss-4024-nuke-iam-groups

Conversation

@james00012

@james00012 james00012 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Problem

IAM groups do not support tags, so the per-repo CircleCI cleanup jobs that filter by --include-tag "gw:repo=..." exclude them for safety:

Resource does not support tag filtering but include tag filters are specified - excluding for safety

The account-wide guardrail did not clean them either, because of a contradiction between two files:

# .github/nuke_config.yml
IAMGroups:
  # Delete all groups - no infrastructure groups to protect   <-- intent: delete
# .github/workflows/nuke-account.yml
COMMON_EXCLUDES: ... --exclude-resource-type iam-group ...    <-- type never processed

The CLI exclusion wins, so the IAMGroups block was dead code and has never executed. Nothing cleaned up IAM groups in any account. They accumulated until ConfigTests (677276116620) hit the GroupsPerAccount quota and CreateGroup began failing, which blocked the terraform-aws-security test suite:

Error: creating IAM Group (dxlh8x-test-group-number-3): CreateGroup, 409 LimitExceeded:
Cannot exceed quota for GroupsPerAccount

Ref: OSS-4024, OSS-4031.

Fix

Delete-all was not safe to rely on. The "no infrastructure groups to protect" comment had never actually been exercised (the type was CLI-excluded the whole time), and it could not be verified for ConfigTests or Sandbox. The guardrail protects an IAM user ^circle-ci-test$, so a real CI identity exists in those accounts; if its permissions came from a group membership, delete-all would strip CI permissions for ~20 repos every 3 hours. The allowlist gets the same cleanup with none of that risk.

Why here rather than in each consumer repo

Untaggable resources cannot be isolated by the gw:repo tag, so consumers have no good mechanism for them. Handling them centrally fixes every account and every consumer repo on the existing 3h cadence, with the naming conventions in one place. Taggable resources stay with the per-repo tag-filtered jobs as before.

Testing

Verified end to end against phxdevops using the real .github/nuke_config.yml:

  • Created four IAM groups: oss4024-test-group-number-1 and gw-test-group-alpha (match), oss4024-real-admins and circle-ci-test-perms (do not match).
  • The run found exactly the two matching groups and deleted them.
  • Both non-matching groups survived, confirming unknown/real groups are protected by the allowlist.
  • Account restored to its original state afterwards.

Also confirmed phxdevops has no IAM groups in normal operation and no IAM user (including circle-ci-test) depends on group membership.

Follow-up

The allowlist currently carries only terraform-aws-security's test-group convention. Other repos whose tests create IAM groups will need their patterns added. OSS-4031 tracks inventorying ConfigTests and Sandbox, which would also tell us whether delete-all is genuinely safe and could replace the allowlist later.

Summary by CodeRabbit

  • Chores
    • Updated automated account cleanup to target only IAM groups matching the test-group naming convention.
    • Removed the broad exclusion that prevented IAM group cleanup.
    • Preserved existing cleanup execution, reporting, and notification behavior.

IAM groups do not support tags, so the per-repo CircleCI cleanup jobs that
filter by the gw:repo tag exclude them for safety and never delete them.
The account-wide guardrail did not delete them either: nuke_config.yml
declared 'IAMGroups: delete all groups' but COMMON_EXCLUDES passed
--exclude-resource-type iam-group, which excludes the type at the CLI and
made that config block dead. As a result nothing cleaned up IAM groups in
any account, and they accumulated until ConfigTests hit the
GroupsPerAccount quota, failing CreateGroup and blocking the
terraform-aws-security test suite (OSS-4024).

Drop iam-group from COMMON_EXCLUDES so the type is processed, and scope
IAMGroups with an allowlist rather than the previously declared delete-all.
Only groups matching a known test naming convention are deleted, so any
real group is protected by default. Deleting all groups was not safe to
rely on: the 'no infrastructure groups to protect' claim had never been
exercised, and it could not be verified for ConfigTests or Sandbox.

Verified against the phxdevops account with the real guardrail config: of
four IAM groups, only the two matching test-group were deleted, and
oss4024-real-admins and circle-ci-test-perms were left intact.
@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 99b5adc2-7041-4e8b-aa54-788cfa303c80

📥 Commits

Reviewing files that changed from the base of the PR and between 40a183b and aafe050.

📒 Files selected for processing (2)
  • .github/nuke_config.yml
  • .github/workflows/nuke-account.yml
💤 Files with no reviewable changes (1)
  • .github/workflows/nuke-account.yml

Walkthrough

IAM group cleanup is enabled in the account workflow, while the nuke configuration restricts deletions to groups matching the test-group naming pattern.

Changes

IAM group cleanup

Layer / File(s) Summary
Selective IAM group cleanup
.github/nuke_config.yml, .github/workflows/nuke-account.yml
The workflow processes IAM groups, and the nuke configuration limits deletion to IAM groups whose names match test-group, with documentation explaining the allowlist.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: denis256

Poem

IAM groups step into the light,
Test names keep the scope just right.
The workflow clears its old exclude,
While safe rules guide what gets removed.
A tidy nuke, with guardrails bright.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: fixing leaked IAM group cleanup in scheduled nuke jobs.
Description check ✅ Passed It covers the problem, fix, testing, and follow-up, so the core PR details are there despite not matching the template headings exactly.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/oss-4024-nuke-iam-groups

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@james00012
james00012 marked this pull request as ready for review July 18, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants