Skip to content

Commit 2db016c

Browse files
authored
Merge pull request #37 from GNS-Science/deploy-test
Promote DEPLOY-TEST to PROD new THS 1.4.2 (Parquet) removes DynamoDB dependency
2 parents f73f794 + 446ba9b commit 2db016c

37 files changed

Lines changed: 1990 additions & 2047 deletions

.bumpversion.cfg

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 0.3.1
2+
current_version = 0.4.0
33
commit = True
44
tag = False
55

.env.tests

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
1-
DATASET_AGGR_ENABLED=FALSE
2-
DATASET_AGGR_URI=
1+
THS_DATASET_AGGR_ENABLED=FALSE
2+
THS_DATASET_AGGR_URI=
3+
THS_DATASET_GRIDDED_URI=

.github/workflows/deploy-to-aws.yaml

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,35 @@ on:
1010
workflow_dispatch:
1111

1212
jobs:
13+
verify-aws-account:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: Configure AWS credentials
17+
uses: aws-actions/configure-aws-credentials@v4
18+
with:
19+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
20+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
21+
aws-region: us-east-1
22+
23+
- name: Verify target AWS account
24+
run: |
25+
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
26+
echo "AWS Account ID: $ACCOUNT_ID"
27+
EXPECTED_ACCOUNT="461564345538"
28+
if [ "$ACCOUNT_ID" != "$EXPECTED_ACCOUNT" ]; then
29+
echo "::error::Deploying to wrong AWS account: $ACCOUNT_ID (expected $EXPECTED_ACCOUNT)"
30+
exit 1
31+
fi
32+
echo "✓ Verified correct AWS account"
33+
1334
call-test-workflow:
1435
uses: ./.github/workflows/dev.yml
1536
secrets: inherit
1637

1738
call-deploy-workflow:
18-
needs: call-test-workflow
39+
needs:
40+
- verify-aws-account
41+
- call-test-workflow
1942
uses: GNS-Science/nshm-github-actions/.github/workflows/deploy-to-aws.yml@main
2043
with:
2144
python-version: '3.12'

.github/workflows/release.yml

Lines changed: 0 additions & 28 deletions
This file was deleted.

.gitignore

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,3 +121,15 @@ ENV/
121121

122122
# mkdocs build dir
123123
site/
124+
125+
# LLM session files (ephemeral)
126+
.llm/.session
127+
128+
# Yarn Berry cache
129+
.yarn/
130+
131+
# Local config
132+
.safety-project.ini
133+
134+
# Old/archived files
135+
.github/old_policy.json

.llm/log.jsonl

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{"timestamp":"2026-02-24T11:30:00Z","session_start":"2026-02-23T00:00:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"glm-5:cloud","tool":"opencode","duration_min":90,"tokens_in":85000,"tokens_out":25000,"commits":["5057d4c"],"files":{"read":["nshm_hazard_graphql_api/schema/toshi_hazard/gridded_hazard.py","tests/conftest.py","tests/test_gridded_hazard.py","tests/test_gridded_hazard_with_nulls.py","tests/test_gridded_hazard_fix_issue_79.py","tests/test_gridded_hazard_helpers.py","tests/test_hazard_curve_migration.py"],"created":["tests/fixtures/gridded_hazard/DS/"],"modified":["nshm_hazard_graphql_api/schema/toshi_hazard/gridded_hazard.py","tests/conftest.py","tests/test_gridded_hazard.py","tests/test_gridded_hazard_with_nulls.py","tests/test_gridded_hazard_fix_issue_79.py","tests/test_gridded_hazard_helpers.py","tests/test_hazard_curve_migration.py"]},"summary":"Update gridded_hazard for THS 1.4.0: replaced deprecated get_one_gridded_hazard with get_gridded_hazard, updated model attributes (aggr, accel_levels), refactored tests from unittest to pytest style with parquet fixtures","user_notes":"new THS library version"}
2+
{"timestamp":"2026-02-25T21:04:56Z","session_start":"2026-02-25T20:22:11Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4-6","tool":"opencode","duration_min":43,"tokens_in":0,"tokens_out":0,"commits":["732d6d5","67fb25a","bda10e9"],"files":{"read":["package.json","serverless.yml","pyproject.toml",".env",".env.tests","DEVELOPMENT.md","setup.cfg"],"created":["CLAUDE.md",".yarnrc.yml"],"modified":["DEVELOPMENT.md","serverless.yml",".env.tests","setup.cfg"]},"summary":"Fixed Yarn Berry PnP issue for serverless-wsgi local dev; corrected THS env var names (DATASET_AGGR_ENABLED->THS_DATASET_AGGR_ENABLED); added THS_DATASET_GRIDDED_URI; updated DEVELOPMENT.md with full config reference; removed redundant tox build env","user_notes":"fix env variable, and docs, and remove redundant tox step (build). Tested with graphql locally."}
3+
{"timestamp":"2026-03-02T00:15:00Z","session_start":"2026-03-02T00:00:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4","tool":"claude-code","duration_min":15,"tokens_in":8500,"tokens_out":1800,"commits":[],"files":{"read":["CHANGELOG.md",".llm/.session"],"created":[],"modified":["CHANGELOG.md"]},"summary":"Updated CHANGELOG.md with version 0.4.0 entry for THS 1.4.2 upgrade and configuration changes","user_notes":"complete missing changelog, memory config"}
4+
{"timestamp":"2026-03-02T00:20:00Z","session_start":"2026-03-02T00:00:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4","tool":"claude-code","duration_min":20,"tokens_in":9000,"tokens_out":2000,"commits":[],"files":{"read":[],"created":[],"modified":[]},"summary":"Session completed; CHANGELOG.md updated for v0.4.0","user_notes":"that was done"}
5+
{"timestamp":"2026-03-02T00:55:00Z","session_start":"2026-03-02T00:25:00Z","project_name":"nshm-hazard-graphql-api","project_root":"nshm-hazard-graphql-api","model":"claude-sonnet-4","tool":"claude-code","duration_min":30,"tokens_in":28000,"tokens_out":7500,"commits":[],"files":{"read":["serverless.yml",".github/workflows/dev.yml",".github/workflows/release.yml",".github/old_policy.json",".llm/.session"],"created":["AWS_GHOST_ACCOUNT_DEPLOYMENT_ISSUE.md"],"modified":[]},"summary":"Investigated AWS ghost account deployment issue. Analyzed serverless.yml and CI/CD workflows. Documented how AWS credentials work, possible causes, and fix options (restore credentials vs migrate to OIDC). Created AWS_GHOST_ACCOUNT_DEPLOYMENT_ISSUE.md with full analysis and recommendations.","user_notes":"diagnose AWS ghost account issue"}

.yarnrc.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
nodeLinker: node-modules
Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# AWS Ghost Account Deployment Issue
2+
3+
## Summary
4+
5+
The nshm-hazard-graphql-api service was deployed to an unintended AWS account (a recently created account) instead of the expected account `461564345538`.
6+
7+
**Root cause (confirmed)**: A default AWS Provider configured via the Serverless Dashboard Web UI was pointing at the wrong AWS account. The Serverless Framework CLI used this Provider's credentials in preference to the GitHub Actions secrets, silently deploying to the wrong account. The GitHub secrets themselves were never changed or rotated incorrectly.
8+
9+
## Background
10+
11+
### Deployment Configuration
12+
13+
- **Framework**: Serverless Framework v4
14+
- **CI/CD**: GitHub Actions using reusable workflows from `GNS-Science/nshm-github-actions`
15+
- **Target Account**: `461564345538` (referenced in serverless.yml IAM policies for DynamoDB, S3, ECR access)
16+
- **Authentication**: GitHub Secrets (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
17+
18+
### GitHub Secrets
19+
20+
| Secret | Purpose | Age |
21+
|--------|---------|-----|
22+
| `AWS_ACCESS_KEY_ID` | AWS authentication | ~3 years |
23+
| `AWS_SECRET_ACCESS_KEY` | AWS authentication | ~3 years |
24+
| `SERVERLESS_ACCESS_KEY` | Serverless Framework Dashboard metrics | 3 years |
25+
26+
## Analysis
27+
28+
### How AWS Credentials Work
29+
30+
AWS access keys are **tied to exactly one AWS account**. When keys are created, they are generated within a specific AWS account and can only authenticate to that account. Therefore:
31+
32+
- If deployment landed in a different account, **the credentials must belong to that account**
33+
- Credentials cannot "accidentally" authenticate to a different account
34+
35+
### Possible Causes
36+
37+
1. **Silent Credential Rotation**
38+
- The GitHub secrets may have been updated with credentials from the new account
39+
- This could have happened during key rotation without clear documentation
40+
41+
2. **Key Regeneration in Wrong Account**
42+
- During a scheduled key rotation, new keys were created in the new account
43+
- The old keys from `461564345538` were deactivated or deleted
44+
45+
3. **Multiple Maintainers**
46+
- Another team member may have updated secrets without communication
47+
48+
### What We Know
49+
50+
- GitHub secrets have existed for ~3 years (claimed)
51+
- Deployment landed in a recently created AWS account
52+
- The service was expected to deploy to account `461564345538`
53+
- No explicit OIDC configuration exists in this repository (using static access keys)
54+
55+
## Diagnostic Steps
56+
57+
To confirm which account the current GitHub secrets belong to:
58+
59+
```bash
60+
# Run locally or in GitHub Actions:
61+
AWS_ACCESS_KEY_ID=<secret_value> \
62+
AWS_SECRET_ACCESS_KEY=<secret_value> \
63+
aws sts get-caller-identity
64+
```
65+
66+
Expected output:
67+
```json
68+
{
69+
"UserId": "AIDAI...",
70+
"Account": "123456789012", // <-- This will reveal the actual account
71+
"Arn": "arn:aws:iam::123456789012:user/..."
72+
}
73+
```
74+
75+
## Fix Options
76+
77+
### Option A: Restore Credentials from Target Account
78+
79+
**Quick fix using existing access key pattern.**
80+
81+
#### Steps
82+
83+
1. Log into AWS Account `461564345538` (expected target)
84+
2. Navigate to IAM → Users → Find the deployment user
85+
3. Create new access keys if needed, or locate existing active keys
86+
4. Update GitHub repository secrets:
87+
- `AWS_ACCESS_KEY_ID` → new key ID from account `461564345538`
88+
- `AWS_SECRET_ACCESS_KEY` → new secret from account `461564345538`
89+
5. Re-run deployment workflow
90+
6. Verify deployment lands in correct account
91+
92+
#### Pros
93+
- Quick to implement
94+
- No infrastructure changes required
95+
- Familiar pattern for team
96+
97+
#### Cons
98+
- Long-lived credentials remain a security risk
99+
- Keys can be leaked via logs, code, or credential exposure
100+
- Requires periodic rotation
101+
- No audit trail for which GitHub workflow used credentials
102+
103+
---
104+
105+
### Option B: Migrate to OIDC Authentication (Recommended)
106+
107+
**Secure, credential-less authentication using GitHub OIDC.**
108+
109+
#### Steps
110+
111+
1. **Create IAM OIDC Provider in Account `461564345538`**
112+
```bash
113+
# If not already exists
114+
aws iam create-open-id-connect-provider \
115+
--url https://token.actions.githubusercontent.com \
116+
--client-id-list sts.amazonaws.com \
117+
--thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1
118+
```
119+
120+
2. **Create IAM Role for GitHub Actions**
121+
```json
122+
{
123+
"Version": "2012-10-17",
124+
"Statement": [
125+
{
126+
"Effect": "Allow",
127+
"Principal": {
128+
"Federated": "arn:aws:iam::461564345538:oidc-provider/token.actions.githubusercontent.com"
129+
},
130+
"Action": "sts:AssumeRoleWithWebIdentity",
131+
"Condition": {
132+
"StringEquals": {
133+
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
134+
},
135+
"StringLike": {
136+
"token.actions.githubusercontent.com:sub": "repo:GNS-Science/nshm-hazard-graphql-api:*"
137+
}
138+
}
139+
}
140+
]
141+
}
142+
```
143+
144+
3. **Attach Required Policies to the Role**
145+
- Include all permissions from `.github/old_policy.json`
146+
- Include Serverless Framework deployment permissions
147+
148+
4. **Update GitHub Secrets**
149+
- Remove `AWS_ACCESS_KEY_ID`
150+
- Remove `AWS_SECRET_ACCESS_KEY`
151+
- Add `AWS_ROLE_ARN` = `arn:aws:iam::461564345538:role/GitHubActionsDeployRole`
152+
153+
5. **Update Reusable Workflow** (in `nshm-github-actions` repository)
154+
```yaml
155+
- name: Configure AWS credentials
156+
uses: aws-actions/configure-aws-credentials@v4
157+
with:
158+
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
159+
aws-region: us-east-1
160+
```
161+
162+
6. **Test and Deploy**
163+
- Re-run workflow
164+
- Verify deployment lands in correct account
165+
166+
#### Pros
167+
- No long-lived credentials in GitHub
168+
- Automatic credential expiration (1 hour tokens)
169+
- Fine-grained access control per repository/branch
170+
- Full audit trail in AWS CloudTrail
171+
- Cannot be leaked via code or logs
172+
173+
#### Cons
174+
- More complex initial setup
175+
- Requires changes to shared workflow repository
176+
- May need coordination with other teams using the reusable workflow
177+
178+
---
179+
180+
### Option C: Verify and Rotate Existing Keys
181+
182+
**If credentials haven't changed, verify they're correct.**
183+
184+
#### Steps
185+
186+
1. Run diagnostic command to identify account for current secrets
187+
2. If secrets belong to wrong account → proceed to Option A
188+
3. If secrets belong to correct account → investigate other causes:
189+
- Serverless Framework Dashboard settings
190+
- `--stage` or `--region` overrides
191+
- Multiple CloudFormation stacks with similar names
192+
193+
## Recommendation
194+
195+
1. **Immediate**: Run diagnostic to confirm which account the secrets belong to
196+
2. **Short-term**: If wrong account, restore correct credentials (Option A)
197+
3. **Long-term**: Migrate to OIDC (Option B) to prevent future credential-related issues
198+
199+
## Prevention Measures
200+
201+
- Add deployment account verification step to CI/CD workflow:
202+
```yaml
203+
- name: Verify target account
204+
run: |
205+
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
206+
if [ "$ACCOUNT_ID" != "461564345538" ]; then
207+
echo "ERROR: Deploying to wrong account: $ACCOUNT_ID"
208+
exit 1
209+
fi
210+
```
211+
212+
- Document credential ownership and rotation schedule
213+
- Consider migrating all repositories to OIDC for consistency
214+
215+
## References
216+
217+
- [GitHub Actions OIDC with AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
218+
- [Serverless Framework AWS Credentials](https://www.serverless.com/framework/docs/providers/aws/guide/credentials)
219+
- `.github/old_policy.json` - IAM policy for deployment permissions
220+
- `serverless.yml:110-111` - DynamoDB references to account `461564345538`

CHANGELOG.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,23 @@
11
# Changelog
22

3+
## [0.4.0] - 2026-03-02
4+
5+
### Changed
6+
- Update to toshi-hazard-store 1.4.2 with new gridded API (replaces deprecated get_one_gridded_hazard)
7+
- Rename DATASET_AGGR_* env vars to THS_DATASET_AGGR_* for consistency
8+
- Add THS_DATASET_GRIDDED_URI configuration
9+
- Refactor tests from unittest to pytest with parquet fixtures
10+
- Remove mock-based testing in favor of fixture data
11+
12+
### Added
13+
- .yarnrc.yml with node-modules linker (fixes serverless-wsgi local dev)
14+
- CLAUDE.md for AI assistant context
15+
- LLM session logging to .llm/log.jsonl
16+
17+
### Removed
18+
- tox build environment (not needed for serverless deployment)
19+
- test_gridded_hazard_moto.py (no longer needed)
20+
321
## [0.3.1] - 2025-11-25
422

523
### Changed

0 commit comments

Comments
 (0)