Skip to content

Commit 3bf4539

Browse files
chrisbcchrisbc
authored
Merge pull request #35 from GNS-Science/fix/aws_account_misdirection
Fix/aws account misdirection
2 parents aff9055 + 8e24706 commit 3bf4539

4 files changed

Lines changed: 263 additions & 0 deletions

File tree

‎.github/workflows/dev.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,29 @@ on:
1212
workflow_call:
1313

1414
jobs:
15+
verify-aws-account:
16+
runs-on: ubuntu-latest
17+
steps:
18+
- name: Configure AWS credentials
19+
uses: aws-actions/configure-aws-credentials@v4
20+
with:
21+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
22+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
23+
aws-region: us-east-1
24+
25+
- name: Verify target AWS account
26+
run: |
27+
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
28+
echo "AWS Account ID: $ACCOUNT_ID"
29+
EXPECTED_ACCOUNT="461564345538"
30+
if [ "$ACCOUNT_ID" != "$EXPECTED_ACCOUNT" ]; then
31+
echo "::error::Deploying to wrong AWS account: $ACCOUNT_ID (expected $EXPECTED_ACCOUNT)"
32+
exit 1
33+
fi
34+
echo "✓ Verified correct AWS account"
35+
1536
call-test-workflow:
37+
needs: verify-aws-account
1638
uses: GNS-Science/nshm-github-actions/.github/workflows/python-run-tests.yml@main
1739
with:
1840
operating-systems: "['ubuntu-latest']"

‎.github/workflows/release.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,29 @@ on:
1212
workflow_dispatch:
1313

1414
jobs:
15+
verify-aws-account:
16+
runs-on: ubuntu-latest
17+
steps:
18+
- name: Configure AWS credentials
19+
uses: aws-actions/configure-aws-credentials@v4
20+
with:
21+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
22+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
23+
aws-region: us-east-1
24+
25+
- name: Verify target AWS account
26+
run: |
27+
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
28+
echo "AWS Account ID: $ACCOUNT_ID"
29+
EXPECTED_ACCOUNT="461564345538"
30+
if [ "$ACCOUNT_ID" != "$EXPECTED_ACCOUNT" ]; then
31+
echo "::error::Deploying to wrong AWS account: $ACCOUNT_ID (expected $EXPECTED_ACCOUNT)"
32+
exit 1
33+
fi
34+
echo "✓ Verified correct AWS account"
1535
1636
release-and-distribute:
37+
needs: verify-aws-account
1738
uses: GNS-Science/nshm-github-actions/.github/workflows/python-release.yml@main
1839
with:
1940
python-version: '3.12'

‎.llm/log.jsonl‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
11
{"timestamp":"2026-02-24T11:30:00Z","session_start":"2026-02-23T00:00:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"glm-5:cloud","tool":"opencode","duration_min":90,"tokens_in":85000,"tokens_out":25000,"commits":["5057d4c"],"files":{"read":["nshm_hazard_graphql_api/schema/toshi_hazard/gridded_hazard.py","tests/conftest.py","tests/test_gridded_hazard.py","tests/test_gridded_hazard_with_nulls.py","tests/test_gridded_hazard_fix_issue_79.py","tests/test_gridded_hazard_helpers.py","tests/test_hazard_curve_migration.py"],"created":["tests/fixtures/gridded_hazard/DS/"],"modified":["nshm_hazard_graphql_api/schema/toshi_hazard/gridded_hazard.py","tests/conftest.py","tests/test_gridded_hazard.py","tests/test_gridded_hazard_with_nulls.py","tests/test_gridded_hazard_fix_issue_79.py","tests/test_gridded_hazard_helpers.py","tests/test_hazard_curve_migration.py"]},"summary":"Update gridded_hazard for THS 1.4.0: replaced deprecated get_one_gridded_hazard with get_gridded_hazard, updated model attributes (aggr, accel_levels), refactored tests from unittest to pytest style with parquet fixtures","user_notes":"new THS library version"}
22
{"timestamp":"2026-02-25T21:04:56Z","session_start":"2026-02-25T20:22:11Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4-6","tool":"opencode","duration_min":43,"tokens_in":0,"tokens_out":0,"commits":["732d6d5","67fb25a","bda10e9"],"files":{"read":["package.json","serverless.yml","pyproject.toml",".env",".env.tests","DEVELOPMENT.md","setup.cfg"],"created":["CLAUDE.md",".yarnrc.yml"],"modified":["DEVELOPMENT.md","serverless.yml",".env.tests","setup.cfg"]},"summary":"Fixed Yarn Berry PnP issue for serverless-wsgi local dev; corrected THS env var names (DATASET_AGGR_ENABLED->THS_DATASET_AGGR_ENABLED); added THS_DATASET_GRIDDED_URI; updated DEVELOPMENT.md with full config reference; removed redundant tox build env","user_notes":"fix env variable, and docs, and remove redundant tox step (build). Tested with graphql locally."}
33
{"timestamp":"2026-03-02T00:15:00Z","session_start":"2026-03-02T00:00:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4","tool":"claude-code","duration_min":15,"tokens_in":8500,"tokens_out":1800,"commits":[],"files":{"read":["CHANGELOG.md",".llm/.session"],"created":[],"modified":["CHANGELOG.md"]},"summary":"Updated CHANGELOG.md with version 0.4.0 entry for THS 1.4.2 upgrade and configuration changes","user_notes":"complete missing changelog, memory config"}
4+
{"timestamp":"2026-03-02T00:20:00Z","session_start":"2026-03-02T00:00:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4","tool":"claude-code","duration_min":20,"tokens_in":9000,"tokens_out":2000,"commits":[],"files":{"read":[],"created":[],"modified":[]},"summary":"Session completed; CHANGELOG.md updated for v0.4.0","user_notes":"that was done"}
5+
{"timestamp":"2026-03-02T00:55:00Z","session_start":"2026-03-02T00:25:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4","tool":"claude-code","duration_min":30,"tokens_in":28000,"tokens_out":7500,"commits":[],"files":{"read":["serverless.yml",".github/workflows/dev.yml",".github/workflows/release.yml",".github/old_policy.json",".llm/.session"],"created":["AWS_GHOST_ACCOUNT_DEPLOYMENT_ISSUE.md"],"modified":[]},"summary":"Investigated AWS ghost account deployment issue. Analyzed serverless.yml and CI/CD workflows. Documented how AWS credentials work, possible causes, and fix options (restore credentials vs migrate to OIDC). Created AWS_GHOST_ACCOUNT_DEPLOYMENT_ISSUE.md with full analysis and recommendations.","user_notes":"diagnose AWS ghost account issue"}

‎AWS_GHOST_ACCOUNT_DEPLOYMENT_ISSUE.md‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
# AWS Ghost Account Deployment Issue
2+
3+
## Summary
4+
5+
The nshm-hazard-graphql-api service was deployed to an unintended AWS account (a recently created account) instead of the expected account `461564345538`.
6+
7+
## Background
8+
9+
### Deployment Configuration
10+
11+
- **Framework**: Serverless Framework v4
12+
- **CI/CD**: GitHub Actions using reusable workflows from `GNS-Science/nshm-github-actions`
13+
- **Target Account**: `461564345538` (referenced in serverless.yml IAM policies for DynamoDB, S3, ECR access)
14+
- **Authentication**: GitHub Secrets (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
15+
16+
### GitHub Secrets
17+
18+
| Secret | Purpose | Age |
19+
|--------|---------|-----|
20+
| `AWS_ACCESS_KEY_ID` | AWS authentication | ~3 years |
21+
| `AWS_SECRET_ACCESS_KEY` | AWS authentication | ~3 years |
22+
| `SERVERLESS_ACCESS_KEY` | Serverless Framework Dashboard metrics | 3 years |
23+
24+
## Analysis
25+
26+
### How AWS Credentials Work
27+
28+
AWS access keys are **tied to exactly one AWS account**. When keys are created, they are generated within a specific AWS account and can only authenticate to that account. Therefore:
29+
30+
- If deployment landed in a different account, **the credentials must belong to that account**
31+
- Credentials cannot "accidentally" authenticate to a different account
32+
33+
### Possible Causes
34+
35+
1. **Silent Credential Rotation**
36+
- The GitHub secrets may have been updated with credentials from the new account
37+
- This could have happened during key rotation without clear documentation
38+
39+
2. **Key Regeneration in Wrong Account**
40+
- During a scheduled key rotation, new keys were created in the new account
41+
- The old keys from `461564345538` were deactivated or deleted
42+
43+
3. **Multiple Maintainers**
44+
- Another team member may have updated secrets without communication
45+
46+
### What We Know
47+
48+
- GitHub secrets have existed for ~3 years (claimed)
49+
- Deployment landed in a recently created AWS account
50+
- The service was expected to deploy to account `461564345538`
51+
- No explicit OIDC configuration exists in this repository (using static access keys)
52+
53+
## Diagnostic Steps
54+
55+
To confirm which account the current GitHub secrets belong to:
56+
57+
```bash
58+
# Run locally or in GitHub Actions:
59+
AWS_ACCESS_KEY_ID=<secret_value> \
60+
AWS_SECRET_ACCESS_KEY=<secret_value> \
61+
aws sts get-caller-identity
62+
```
63+
64+
Expected output:
65+
```json
66+
{
67+
"UserId": "AIDAI...",
68+
"Account": "123456789012", // <-- This will reveal the actual account
69+
"Arn": "arn:aws:iam::123456789012:user/..."
70+
}
71+
```
72+
73+
## Fix Options
74+
75+
### Option A: Restore Credentials from Target Account
76+
77+
**Quick fix using existing access key pattern.**
78+
79+
#### Steps
80+
81+
1. Log into AWS Account `461564345538` (expected target)
82+
2. Navigate to IAM → Users → Find the deployment user
83+
3. Create new access keys if needed, or locate existing active keys
84+
4. Update GitHub repository secrets:
85+
- `AWS_ACCESS_KEY_ID` → new key ID from account `461564345538`
86+
- `AWS_SECRET_ACCESS_KEY` → new secret from account `461564345538`
87+
5. Re-run deployment workflow
88+
6. Verify deployment lands in correct account
89+
90+
#### Pros
91+
- Quick to implement
92+
- No infrastructure changes required
93+
- Familiar pattern for team
94+
95+
#### Cons
96+
- Long-lived credentials remain a security risk
97+
- Keys can be leaked via logs, code, or credential exposure
98+
- Requires periodic rotation
99+
- No audit trail for which GitHub workflow used credentials
100+
101+
---
102+
103+
### Option B: Migrate to OIDC Authentication (Recommended)
104+
105+
**Secure, credential-less authentication using GitHub OIDC.**
106+
107+
#### Steps
108+
109+
1. **Create IAM OIDC Provider in Account `461564345538`**
110+
```bash
111+
# If not already exists
112+
aws iam create-open-id-connect-provider \
113+
--url https://token.actions.githubusercontent.com \
114+
--client-id-list sts.amazonaws.com \
115+
--thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1
116+
```
117+
118+
2. **Create IAM Role for GitHub Actions**
119+
```json
120+
{
121+
"Version": "2012-10-17",
122+
"Statement": [
123+
{
124+
"Effect": "Allow",
125+
"Principal": {
126+
"Federated": "arn:aws:iam::461564345538:oidc-provider/token.actions.githubusercontent.com"
127+
},
128+
"Action": "sts:AssumeRoleWithWebIdentity",
129+
"Condition": {
130+
"StringEquals": {
131+
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
132+
},
133+
"StringLike": {
134+
"token.actions.githubusercontent.com:sub": "repo:GNS-Science/nshm-hazard-graphql-api:*"
135+
}
136+
}
137+
}
138+
]
139+
}
140+
```
141+
142+
3. **Attach Required Policies to the Role**
143+
- Include all permissions from `.github/old_policy.json`
144+
- Include Serverless Framework deployment permissions
145+
146+
4. **Update GitHub Secrets**
147+
- Remove `AWS_ACCESS_KEY_ID`
148+
- Remove `AWS_SECRET_ACCESS_KEY`
149+
- Add `AWS_ROLE_ARN` = `arn:aws:iam::461564345538:role/GitHubActionsDeployRole`
150+
151+
5. **Update Reusable Workflow** (in `nshm-github-actions` repository)
152+
```yaml
153+
- name: Configure AWS credentials
154+
uses: aws-actions/configure-aws-credentials@v4
155+
with:
156+
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
157+
aws-region: us-east-1
158+
```
159+
160+
6. **Test and Deploy**
161+
- Re-run workflow
162+
- Verify deployment lands in correct account
163+
164+
#### Pros
165+
- No long-lived credentials in GitHub
166+
- Automatic credential expiration (1 hour tokens)
167+
- Fine-grained access control per repository/branch
168+
- Full audit trail in AWS CloudTrail
169+
- Cannot be leaked via code or logs
170+
171+
#### Cons
172+
- More complex initial setup
173+
- Requires changes to shared workflow repository
174+
- May need coordination with other teams using the reusable workflow
175+
176+
---
177+
178+
### Option C: Verify and Rotate Existing Keys
179+
180+
**If credentials haven't changed, verify they're correct.**
181+
182+
#### Steps
183+
184+
1. Run diagnostic command to identify account for current secrets
185+
2. If secrets belong to wrong account → proceed to Option A
186+
3. If secrets belong to correct account → investigate other causes:
187+
- Serverless Framework Dashboard settings
188+
- `--stage` or `--region` overrides
189+
- Multiple CloudFormation stacks with similar names
190+
191+
## Recommendation
192+
193+
1. **Immediate**: Run diagnostic to confirm which account the secrets belong to
194+
2. **Short-term**: If wrong account, restore correct credentials (Option A)
195+
3. **Long-term**: Migrate to OIDC (Option B) to prevent future credential-related issues
196+
197+
## Prevention Measures
198+
199+
- Add deployment account verification step to CI/CD workflow:
200+
```yaml
201+
- name: Verify target account
202+
run: |
203+
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
204+
if [ "$ACCOUNT_ID" != "461564345538" ]; then
205+
echo "ERROR: Deploying to wrong account: $ACCOUNT_ID"
206+
exit 1
207+
fi
208+
```
209+
210+
- Document credential ownership and rotation schedule
211+
- Consider migrating all repositories to OIDC for consistency
212+
213+
## References
214+
215+
- [GitHub Actions OIDC with AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
216+
- [Serverless Framework AWS Credentials](https://www.serverless.com/framework/docs/providers/aws/guide/credentials)
217+
- `.github/old_policy.json` - IAM policy for deployment permissions
218+
- `serverless.yml:110-111` - DynamoDB references to account `461564345538`

0 commit comments

Comments
 (0)