|
| 1 | +# AWS Ghost Account Deployment Issue |
| 2 | + |
| 3 | +## Summary |
| 4 | + |
| 5 | +The nshm-hazard-graphql-api service was deployed to an unintended AWS account (a recently created account) instead of the expected account `461564345538`. |
| 6 | + |
| 7 | +**Root cause (confirmed)**: A default AWS Provider configured via the Serverless Dashboard Web UI was pointing at the wrong AWS account. The Serverless Framework CLI used this Provider's credentials in preference to the GitHub Actions secrets, silently deploying to the wrong account. The GitHub secrets themselves were never changed or rotated incorrectly. |
| 8 | + |
| 9 | +## Background |
| 10 | + |
| 11 | +### Deployment Configuration |
| 12 | + |
| 13 | +- **Framework**: Serverless Framework v4 |
| 14 | +- **CI/CD**: GitHub Actions using reusable workflows from `GNS-Science/nshm-github-actions` |
| 15 | +- **Target Account**: `461564345538` (referenced in serverless.yml IAM policies for DynamoDB, S3, ECR access) |
| 16 | +- **Authentication**: GitHub Secrets (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) |
| 17 | + |
| 18 | +### GitHub Secrets |
| 19 | + |
| 20 | +| Secret | Purpose | Age | |
| 21 | +|--------|---------|-----| |
| 22 | +| `AWS_ACCESS_KEY_ID` | AWS authentication | ~3 years | |
| 23 | +| `AWS_SECRET_ACCESS_KEY` | AWS authentication | ~3 years | |
| 24 | +| `SERVERLESS_ACCESS_KEY` | Serverless Framework Dashboard metrics | 3 years | |
| 25 | + |
| 26 | +## Analysis |
| 27 | + |
| 28 | +### How AWS Credentials Work |
| 29 | + |
| 30 | +AWS access keys are **tied to exactly one AWS account**. When keys are created, they are generated within a specific AWS account and can only authenticate to that account. Therefore: |
| 31 | + |
| 32 | +- If deployment landed in a different account, **the credentials must belong to that account** |
| 33 | +- Credentials cannot "accidentally" authenticate to a different account |
| 34 | + |
| 35 | +### Possible Causes |
| 36 | + |
| 37 | +1. **Silent Credential Rotation** |
| 38 | + - The GitHub secrets may have been updated with credentials from the new account |
| 39 | + - This could have happened during key rotation without clear documentation |
| 40 | + |
| 41 | +2. **Key Regeneration in Wrong Account** |
| 42 | + - During a scheduled key rotation, new keys were created in the new account |
| 43 | + - The old keys from `461564345538` were deactivated or deleted |
| 44 | + |
| 45 | +3. **Multiple Maintainers** |
| 46 | + - Another team member may have updated secrets without communication |
| 47 | + |
| 48 | +### What We Know |
| 49 | + |
| 50 | +- GitHub secrets have existed for ~3 years (claimed) |
| 51 | +- Deployment landed in a recently created AWS account |
| 52 | +- The service was expected to deploy to account `461564345538` |
| 53 | +- No explicit OIDC configuration exists in this repository (using static access keys) |
| 54 | + |
| 55 | +## Diagnostic Steps |
| 56 | + |
| 57 | +To confirm which account the current GitHub secrets belong to: |
| 58 | + |
| 59 | +```bash |
| 60 | +# Run locally or in GitHub Actions: |
| 61 | +AWS_ACCESS_KEY_ID=<secret_value> \ |
| 62 | +AWS_SECRET_ACCESS_KEY=<secret_value> \ |
| 63 | +aws sts get-caller-identity |
| 64 | +``` |
| 65 | + |
| 66 | +Expected output: |
| 67 | +```json |
| 68 | +{ |
| 69 | + "UserId": "AIDAI...", |
| 70 | + "Account": "123456789012", // <-- This will reveal the actual account |
| 71 | + "Arn": "arn:aws:iam::123456789012:user/..." |
| 72 | +} |
| 73 | +``` |
| 74 | + |
| 75 | +## Fix Options |
| 76 | + |
| 77 | +### Option A: Restore Credentials from Target Account |
| 78 | + |
| 79 | +**Quick fix using existing access key pattern.** |
| 80 | + |
| 81 | +#### Steps |
| 82 | + |
| 83 | +1. Log into AWS Account `461564345538` (expected target) |
| 84 | +2. Navigate to IAM → Users → Find the deployment user |
| 85 | +3. Create new access keys if needed, or locate existing active keys |
| 86 | +4. Update GitHub repository secrets: |
| 87 | + - `AWS_ACCESS_KEY_ID` → new key ID from account `461564345538` |
| 88 | + - `AWS_SECRET_ACCESS_KEY` → new secret from account `461564345538` |
| 89 | +5. Re-run deployment workflow |
| 90 | +6. Verify deployment lands in correct account |
| 91 | + |
| 92 | +#### Pros |
| 93 | +- Quick to implement |
| 94 | +- No infrastructure changes required |
| 95 | +- Familiar pattern for team |
| 96 | + |
| 97 | +#### Cons |
| 98 | +- Long-lived credentials remain a security risk |
| 99 | +- Keys can be leaked via logs, code, or credential exposure |
| 100 | +- Requires periodic rotation |
| 101 | +- No audit trail for which GitHub workflow used credentials |
| 102 | + |
| 103 | +--- |
| 104 | + |
| 105 | +### Option B: Migrate to OIDC Authentication (Recommended) |
| 106 | + |
| 107 | +**Secure, credential-less authentication using GitHub OIDC.** |
| 108 | + |
| 109 | +#### Steps |
| 110 | + |
| 111 | +1. **Create IAM OIDC Provider in Account `461564345538`** |
| 112 | + ```bash |
| 113 | + # If not already exists |
| 114 | + aws iam create-open-id-connect-provider \ |
| 115 | + --url https://token.actions.githubusercontent.com \ |
| 116 | + --client-id-list sts.amazonaws.com \ |
| 117 | + --thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1 |
| 118 | + ``` |
| 119 | + |
| 120 | +2. **Create IAM Role for GitHub Actions** |
| 121 | + ```json |
| 122 | + { |
| 123 | + "Version": "2012-10-17", |
| 124 | + "Statement": [ |
| 125 | + { |
| 126 | + "Effect": "Allow", |
| 127 | + "Principal": { |
| 128 | + "Federated": "arn:aws:iam::461564345538:oidc-provider/token.actions.githubusercontent.com" |
| 129 | + }, |
| 130 | + "Action": "sts:AssumeRoleWithWebIdentity", |
| 131 | + "Condition": { |
| 132 | + "StringEquals": { |
| 133 | + "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" |
| 134 | + }, |
| 135 | + "StringLike": { |
| 136 | + "token.actions.githubusercontent.com:sub": "repo:GNS-Science/nshm-hazard-graphql-api:*" |
| 137 | + } |
| 138 | + } |
| 139 | + } |
| 140 | + ] |
| 141 | + } |
| 142 | + ``` |
| 143 | + |
| 144 | +3. **Attach Required Policies to the Role** |
| 145 | + - Include all permissions from `.github/old_policy.json` |
| 146 | + - Include Serverless Framework deployment permissions |
| 147 | + |
| 148 | +4. **Update GitHub Secrets** |
| 149 | + - Remove `AWS_ACCESS_KEY_ID` |
| 150 | + - Remove `AWS_SECRET_ACCESS_KEY` |
| 151 | + - Add `AWS_ROLE_ARN` = `arn:aws:iam::461564345538:role/GitHubActionsDeployRole` |
| 152 | + |
| 153 | +5. **Update Reusable Workflow** (in `nshm-github-actions` repository) |
| 154 | + ```yaml |
| 155 | + - name: Configure AWS credentials |
| 156 | + uses: aws-actions/configure-aws-credentials@v4 |
| 157 | + with: |
| 158 | + role-to-assume: ${{ secrets.AWS_ROLE_ARN }} |
| 159 | + aws-region: us-east-1 |
| 160 | + ``` |
| 161 | +
|
| 162 | +6. **Test and Deploy** |
| 163 | + - Re-run workflow |
| 164 | + - Verify deployment lands in correct account |
| 165 | +
|
| 166 | +#### Pros |
| 167 | +- No long-lived credentials in GitHub |
| 168 | +- Automatic credential expiration (1 hour tokens) |
| 169 | +- Fine-grained access control per repository/branch |
| 170 | +- Full audit trail in AWS CloudTrail |
| 171 | +- Cannot be leaked via code or logs |
| 172 | +
|
| 173 | +#### Cons |
| 174 | +- More complex initial setup |
| 175 | +- Requires changes to shared workflow repository |
| 176 | +- May need coordination with other teams using the reusable workflow |
| 177 | +
|
| 178 | +--- |
| 179 | +
|
| 180 | +### Option C: Verify and Rotate Existing Keys |
| 181 | +
|
| 182 | +**If credentials haven't changed, verify they're correct.** |
| 183 | +
|
| 184 | +#### Steps |
| 185 | +
|
| 186 | +1. Run diagnostic command to identify account for current secrets |
| 187 | +2. If secrets belong to wrong account → proceed to Option A |
| 188 | +3. If secrets belong to correct account → investigate other causes: |
| 189 | + - Serverless Framework Dashboard settings |
| 190 | + - `--stage` or `--region` overrides |
| 191 | + - Multiple CloudFormation stacks with similar names |
| 192 | + |
| 193 | +## Recommendation |
| 194 | + |
| 195 | +1. **Immediate**: Run diagnostic to confirm which account the secrets belong to |
| 196 | +2. **Short-term**: If wrong account, restore correct credentials (Option A) |
| 197 | +3. **Long-term**: Migrate to OIDC (Option B) to prevent future credential-related issues |
| 198 | + |
| 199 | +## Prevention Measures |
| 200 | + |
| 201 | +- Add deployment account verification step to CI/CD workflow: |
| 202 | + ```yaml |
| 203 | + - name: Verify target account |
| 204 | + run: | |
| 205 | + ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) |
| 206 | + if [ "$ACCOUNT_ID" != "461564345538" ]; then |
| 207 | + echo "ERROR: Deploying to wrong account: $ACCOUNT_ID" |
| 208 | + exit 1 |
| 209 | + fi |
| 210 | + ``` |
| 211 | + |
| 212 | +- Document credential ownership and rotation schedule |
| 213 | +- Consider migrating all repositories to OIDC for consistency |
| 214 | + |
| 215 | +## References |
| 216 | + |
| 217 | +- [GitHub Actions OIDC with AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) |
| 218 | +- [Serverless Framework AWS Credentials](https://www.serverless.com/framework/docs/providers/aws/guide/credentials) |
| 219 | +- `.github/old_policy.json` - IAM policy for deployment permissions |
| 220 | +- `serverless.yml:110-111` - DynamoDB references to account `461564345538` |
0 commit comments