fix: clean up leaked IAM groups in scheduled nuke jobs (OSS-4024)#1160
fix: clean up leaked IAM groups in scheduled nuke jobs (OSS-4024)#1160james00012 wants to merge 1 commit into
Conversation
IAM groups do not support tags, so the per-repo CircleCI cleanup jobs that filter by the gw:repo tag exclude them for safety and never delete them. The account-wide guardrail did not delete them either: nuke_config.yml declared 'IAMGroups: delete all groups' but COMMON_EXCLUDES passed --exclude-resource-type iam-group, which excludes the type at the CLI and made that config block dead. As a result nothing cleaned up IAM groups in any account, and they accumulated until ConfigTests hit the GroupsPerAccount quota, failing CreateGroup and blocking the terraform-aws-security test suite (OSS-4024). Drop iam-group from COMMON_EXCLUDES so the type is processed, and scope IAMGroups with an allowlist rather than the previously declared delete-all. Only groups matching a known test naming convention are deleted, so any real group is protected by default. Deleting all groups was not safe to rely on: the 'no infrastructure groups to protect' claim had never been exercised, and it could not be verified for ConfigTests or Sandbox. Verified against the phxdevops account with the real guardrail config: of four IAM groups, only the two matching test-group were deleted, and oss4024-real-admins and circle-ci-test-perms were left intact.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
💤 Files with no reviewable changes (1)
WalkthroughIAM group cleanup is enabled in the account workflow, while the nuke configuration restricts deletions to groups matching the ChangesIAM group cleanup
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Problem
IAM groups do not support tags, so the per-repo CircleCI cleanup jobs that filter by
--include-tag "gw:repo=..."exclude them for safety:The account-wide guardrail did not clean them either, because of a contradiction between two files:
The CLI exclusion wins, so the
IAMGroupsblock was dead code and has never executed. Nothing cleaned up IAM groups in any account. They accumulated until ConfigTests (677276116620) hit theGroupsPerAccountquota andCreateGroupbegan failing, which blocked the terraform-aws-security test suite:Ref: OSS-4024, OSS-4031.
Fix
iam-groupfromCOMMON_EXCLUDESso the type is processed. This follows the established pattern of enabling types individually once verified, as done for subnets/NAT (chore: enable subnet and NAT gateway cleanup in nuke workflow #1024), Route 53 (Enable Route 53 hosted zone cleanup and fix tag batching #1026), and IAM policies (fix: enable IAM policy cleanup to prevent policy limit exhaustion #1112). The blanket list came in wholesale with Migrate scheduled nuke jobs from CircleCI to GitHub Actions #997.IAMGroupswith an allowlist instead of the previously declared delete-all. Only groups matching a known test naming convention are deleted; any real group is protected by default.Delete-all was not safe to rely on. The "no infrastructure groups to protect" comment had never actually been exercised (the type was CLI-excluded the whole time), and it could not be verified for ConfigTests or Sandbox. The guardrail protects an IAM user
^circle-ci-test$, so a real CI identity exists in those accounts; if its permissions came from a group membership, delete-all would strip CI permissions for ~20 repos every 3 hours. The allowlist gets the same cleanup with none of that risk.Why here rather than in each consumer repo
Untaggable resources cannot be isolated by the
gw:repotag, so consumers have no good mechanism for them. Handling them centrally fixes every account and every consumer repo on the existing 3h cadence, with the naming conventions in one place. Taggable resources stay with the per-repo tag-filtered jobs as before.Testing
Verified end to end against phxdevops using the real
.github/nuke_config.yml:oss4024-test-group-number-1andgw-test-group-alpha(match),oss4024-real-adminsandcircle-ci-test-perms(do not match).Also confirmed phxdevops has no IAM groups in normal operation and no IAM user (including
circle-ci-test) depends on group membership.Follow-up
The allowlist currently carries only terraform-aws-security's
test-groupconvention. Other repos whose tests create IAM groups will need their patterns added. OSS-4031 tracks inventorying ConfigTests and Sandbox, which would also tell us whether delete-all is genuinely safe and could replace the allowlist later.Summary by CodeRabbit