feat: deploy API + workers to AWS Lambda via Terraform (#54)

* feat: deploy api + workers to AWS Lambda via Terraform

Move from docker-compose to a fully-managed AWS deployment:
- API as a Lambda behind API Gateway HTTP API at api.dumpus.app
- Worker as an SQS-triggered Lambda (replaces the always-on Celery worker)
- RDS Postgres 17 + fck-nat for ~$22/mo total
- Drops Celery + RabbitMQ entirely; QUEUE_BACKEND=sync runs the worker
  inline for local dev, QUEUE_BACKEND=sqs in production
- GitHub Actions deploy via OIDC: build container -> ECR -> update both Lambdas
- Terraform stack under infra/terraform with full bootstrap docs

* chore(tf): allow overwrite on api A record

During the OVH→Route53 cutover the api record is pre-populated with
the existing origin IP to avoid downtime while nameservers propagate.
allow_overwrite = true lets Terraform take ownership and swap it for
the API Gateway alias on apply.

* chore: stub sentiment scoring, drop nltk

Sentiment analysis pulled in nltk + a baked-in vader_lexicon (~50MB),
which inflated the Lambda image and added cold-start cost. Stub
count_sentiments() to a constant 0.5 for now. The TODO in tasks.py
calls out how to revive it.

- remove nltk from requirements
- drop the vader_lexicon download from both Dockerfiles
- drop scripts/download-ntk.py and the install-step Makefile reference

* docs: fold AWS deploy guide into the main README

Move the contents of infra/terraform/README.md into a new "Deploy to AWS"
section in the project README, so contributors discover it from one place.
Also refresh the self-hosting steps for the post-Celery world (sync mode,
no broker) and drop the now-stale celeryconfig.py troubleshooting tip.

* docs: depersonalize the AWS deploy section

The README is shipped with the project; cost figures, traffic estimates,
and credit accounting only make sense for one operator and confuse
everyone else who self-hosts. Drop those, generalize the log-group paths,
and tighten the language.

* feat(tf): add allowed_account_ids guard

Prevents an apply from touching the wrong AWS account when AWS_PROFILE
or env credentials accidentally point somewhere else (e.g. a company
account on the same workstation). When set in terraform.tfvars, the AWS
provider hard-fails before planning if the resolved account isn't in
the allow-list.

* chore: bump pinned versions to current

Latest releases checked against the upgrade guides; only one breaking
change in our resource set (aws_ami stricter validation).

- AWS provider 5.70 -> 6.0 (latest 6.43)
- Add explicit owner-id filter to aws_ami.fck_nat for v6's stricter
  most_recent validation
- apig-wsgi 2.18.0 -> 2.20.0
- actions/checkout v4 -> v6
- aws-actions/configure-aws-credentials v4 -> v6
- docker/setup-buildx-action v3 -> v4
- docker/build-push-action v6 -> v7

Python 3.10 is deprecated by Lambda on Oct 31 2026. Bumping the runtime
needs numpy/pandas/Flask version bumps and is left for a follow-up PR.

* chore(tf): expose region as an output

The bootstrap commands in the README reference `terraform output -raw region`,
which didn't exist. Adding it so those commands work as written.

* feat(tf): auto-push bootstrap image so first apply doesn't need a manual step

CreateFunction validates that the image_uri exists, so the very first apply
used to fail unless you'd manually pushed a :bootstrap image to ECR first.

Add a null_resource that runs once after the ECR repo is created: pulls
public.ecr.aws/lambda/python:3.10, retags it to :bootstrap, pushes to our
ECR. Lambdas depend on it, so apply ordering becomes ECR -> placeholder
image -> Lambdas. Idempotent: re-applies skip if the tag already exists.

After first apply, CI takes over. The placeholder image gets overwritten by
the first push to main, and Terraform never touches image_uri again
(lifecycle.ignore_changes already in place).

Requires docker + aws CLI on the apply host.

* chore: review-pass cleanup before merge

- secrets.tf: drop special chars from RDS master password. !#$%&*()-_=+[]{}<>:?
  included `:`, `?`, `&`, `=` which would break SQLAlchemy URL parsing
  any time the random password happened to contain one. 32 alnum chars give
  ~190 bits of entropy anyway. (real bug)
- Dockerfile.api: bookworm replaces buster (Debian 10 EOL'd Jun 2024).
- bootstrap.tf: pin the placeholder image to python:3.13 instead of 3.10
  (3.10 is being deprecated by Lambda). The placeholder is replaced on
  first deploy, so this is purely about which tag we pull from public ECR.
- main.tf: drop unused common_tags local and aws_caller_identity data.
- variables.tf: rephrase worker_reserved_concurrency description to avoid
  project-specific framing.
- terraform.tfvars.example: rewrite the image_tag comment so it matches
  the actual lifecycle (Terraform owns the first apply, CI owns updates,
  ignore_changes prevents drift).
- .gitignore: stop ignoring .terraform.lock.hcl — provider lock should be
  committed so applies are reproducible. Adds the lock at v6.43.0.

* chore(tf): drop the OVH migration note on api A record

The migration is one-time. Once it's done, the comment is archaeology;
allow_overwrite is meaningful on its own (lets Terraform take ownership
of an existing record).

* feat(ci): add infra workflow so tofu apply can run from GitHub Actions

Lets users with limited local bandwidth (provider download is ~600MB)
run plan/apply from CI instead. The workflow uses static AWS keys
stored as GH secrets for the bootstrap, and remote state in S3 (set up
out-of-band via 'aws s3 mb' before the first run). After the OIDC role
is created on first apply, deploy.yml takes over for code deploys.

terraform.ci.tfvars carries the non-sensitive vars (region, account ID,
domain). Sensitive vars (discord_secret, diswho_jwt_secret) come in via
TF_VAR_* env from GH secrets and never land in the repo.

* feat: load app secrets from Secrets Manager at runtime

Lambda env no longer carries the actual secret values — only a JSON map
of ARNs (SECRETS_ARN_MAP). On cold start, secrets_loader.py reads the
real values from Secrets Manager via boto3 and hydrates os.environ, so
the rest of the app keeps using os.getenv() unchanged.

What this buys us:
- secret values aren't visible to anyone with lambda:GetFunction
- rotate via 'aws secretsmanager put-secret-value' without redeploying
  the Lambda (versioned secret_string + ignore_changes prevents tofu
  apply from reverting the rotation)
- every secret read shows up in CloudTrail

Other changes:
- add WH_URL secret + variable so internal Discord webhook notifications
  work in prod (was missing entirely before)
- discord_secret defaults to '' since it's only used as a fallback when
  diswho_base_url is unset
- gitignore .env.* (except .env.example) so .env.bootstrap can't be
  accidentally committed
- workflow plumbs TF_VAR_wh_url alongside the other secret vars

* chore(tf): enable S3 backend now that the state bucket exists

* chore(tf): set diswho_base_url to the public diswho instance

* chore: remove DISCORD_SECRET from the deployment

The fallback path in util.py.fetch_diswho_user that uses DISCORD_SECRET
only runs when DISWHO_BASE_URL is unset. Our deployment has it set, so
the secret is dead config. Drop:

- the discord_secret TF variable
- the Secrets Manager entry + version
- the IAM read permission
- the Lambda env entry in SECRETS_ARN_MAP
- the TF_VAR_discord_secret plumbing in infra.yml
- the matching GH repo secret

util.py keeps the fallback so self-hosters without a diswho instance
can still configure DISCORD_SECRET via env.

* chore: switch region from eu-west-3 to eu-west-1

Better service availability and lower cost. Latency from FR is ~30ms
worse but still acceptable for this app's UX. Stack-wide change:

- variables.tf, terraform.ci.tfvars, terraform.tfvars.example: var.region default
- backend.tf: state bucket region (the bucket itself is being recreated
  in eu-west-1 out-of-band)
- infra.yml + deploy.yml: aws-region used by the workflows

* chore(tf): point backend at dumpus-prod-tfstate (matches stack naming)

* fix(tf): two apply-time errors

1. New AWS accounts have a 10-execution Lambda concurrency ceiling and
   require >=10 unreserved at all times, so reserved_concurrent_executions=1
   was rejected with InvalidParameterValueException. Make the variable
   optional (0 = no reservation), default to 0.

2. EIP association raced with EC2 instance creation — the instance was
   still in pending-instance-creation state when the EIP tried to attach
   to its ENI. Add depends_on so the EIP attaches first, then the
   instance picks up the already-EIP-bound ENI.

Author: Androz2091

.env.example

@@ -4,17 +4,13 @@ POSTGRES_PORT="6641"
 POSTGRES_DB="dumpus"
 POSTGRES_URL="postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASS}@localhost:${POSTGRES_PORT}/${POSTGRES_DB}"
 
-RABBITMQ_USER="dumpus"
-RABBITMQ_PASS="CHANGE_ME"
-RABBITMQ_PORT="5672"
-RABBITMQ_URL="amqp://${RABBITMQ_USER}:${RABBITMQ_PASS}@localhost:${RABBITMQ_PORT}/"
-
 API_PORT="5000"
 DL_ZIP_WHITELISTED_DOMAINS=""
 DL_ZIP_TMP_PATH="./tmp"
 
-CELERY_QUEUE="regular_process"
-CELERY_HOSTNAME="regular_process"
+# `sync` runs the worker inline (default for local dev).
+# `sqs` (production) sends to AWS SQS — also requires SQS_QUEUE_URL.
+QUEUE_BACKEND="sync"
 
 DISWHO_JWT_SECRET=""
 DISWHO_BASE_URL=""

.github/workflows/deploy.yml

@@ -0,0 +1,102 @@
+name: deploy
+
+on:
+  push:
+    branches: [main]
+    paths-ignore:
+      - "docs/**"
+      - "infra/**"
+      - "**.md"
+  workflow_dispatch:
+    inputs:
+      image_tag:
+        description: "Override image tag (defaults to git sha)"
+        required: false
+
+permissions:
+  id-token: write   # GitHub OIDC
+  contents: read
+
+env:
+  AWS_REGION: eu-west-1
+  ECR_REPO: dumpus-prod-lambda
+  API_FUNCTION: dumpus-prod-api
+  WORKER_FUNCTION: dumpus-prod-worker
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Resolve image tag
+        id: meta
+        run: |
+          if [ -n "${{ inputs.image_tag }}" ]; then
+            tag="${{ inputs.image_tag }}"
+          else
+            tag="${GITHUB_SHA::12}"
+          fi
+          echo "image_tag=$tag" >> "$GITHUB_OUTPUT"
+
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v6
+        with:
+          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
+          aws-region: ${{ env.AWS_REGION }}
+
+      - name: Login to ECR
+        id: ecr
+        uses: aws-actions/amazon-ecr-login@v2
+
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@v4
+
+      - name: Build & push Lambda image
+        id: build
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          file: Dockerfile.lambda
+          push: true
+          # Lambda only runs amd64 unless the function is configured for arm64.
+          # Keep amd64 unless we explicitly switch the function architecture.
+          platforms: linux/amd64
+          provenance: false
+          tags: |
+            ${{ steps.ecr.outputs.registry }}/${{ env.ECR_REPO }}:${{ steps.meta.outputs.image_tag }}
+            ${{ steps.ecr.outputs.registry }}/${{ env.ECR_REPO }}:latest
+          cache-from: type=gha,scope=lambda
+          cache-to: type=gha,scope=lambda,mode=max
+
+      - name: Update Lambda functions
+        env:
+          IMAGE_URI: ${{ steps.ecr.outputs.registry }}/${{ env.ECR_REPO }}:${{ steps.meta.outputs.image_tag }}
+        run: |
+          set -euo pipefail
+          for fn in "${API_FUNCTION}" "${WORKER_FUNCTION}"; do
+            echo "::group::Updating $fn"
+            aws lambda update-function-code \
+              --function-name "$fn" \
+              --image-uri "$IMAGE_URI" \
+              --publish \
+              > /dev/null
+            aws lambda wait function-updated --function-name "$fn"
+            echo "::endgroup::"
+          done
+
+      - name: Smoke test API
+        run: |
+          set -euo pipefail
+          for i in 1 2 3 4 5; do
+            code=$(curl -s -o /dev/null -w "%{http_code}" https://api.dumpus.app/health || echo "000")
+            if [ "$code" = "200" ]; then
+              echo "API healthy"
+              exit 0
+            fi
+            echo "attempt $i: $code, retrying..."
+            sleep 5
+          done
+          echo "API never returned 200" >&2
+          exit 1

.gitignore

@@ -11,10 +11,15 @@ node_test
 
 # Env
 .env
+.env.*
+!.env.example
 
 # Data
 scripts/res.txt
 *.db
 
 # JetBrains IDE
-.idea
+.idea
+
+# Claude Code
+.claude/

Dockerfile.api

@@ -1,18 +1,8 @@
-# Last stable Python version
-FROM python:3.10-slim-buster
+FROM python:3.10-slim-bookworm
 
 # Create application directory and move there
 WORKDIR /app
 
-# Install nltk
-RUN pip install --no-cache-dir --prefer-binary nltk
-
-# Copy the script that downloads the vader lexicon module
-COPY scripts/download-ntk.py ./
-
-# Execute that script
-RUN python download-ntk.py
-
 # Copy requirements file from the host to the container
 COPY requirements.txt ./
 

Dockerfile.flower

@@ -0,0 +1,15 @@
+# Single image used by both Lambda functions (api + worker). The active
+# handler is selected per-function via the Lambda ImageConfig.Command.
+FROM public.ecr.aws/lambda/python:3.10
+
+WORKDIR ${LAMBDA_TASK_ROOT}
+
+COPY requirements.txt ./
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Flat layout: src/* sits at the task root so existing top-level imports
+# (`from tasks import ...`) keep working.
+COPY src/ ${LAMBDA_TASK_ROOT}/
+
+# Default to the API handler. Override per function in Terraform/Lambda config.
+CMD [ "lambda_handlers.api.handler" ]

Dockerfile.lambda

@@ -14,16 +14,12 @@ install:
 	uv venv --python 3.10
 	. .venv/bin/activate
 	uv pip install -r requirements.txt
-	uv run ./scripts/download-ntk.py
 
 dev:
 	. .venv/bin/activate
 	bash -c " \
 		trap 'docker compose down' EXIT; \
-		docker compose up -d broker db && \
+		docker compose up -d db && \
 		cd src && \
-		tee \
-			>(docker compose logs -f) \
-			>(celery --app tasks worker --loglevel=info --queues=${CELERY_QUEUE} --hostname=${CELERY_HOSTNAME}@%h --concurrency=1) \
-			>(waitress-serve --port=${API_PORT} app:app) \
-	"
+		QUEUE_BACKEND=sync waitress-serve --port=${API_PORT} app:app \
+	"

Dockerfile.worker

@@ -6,6 +6,7 @@ API to extract statistics from the Discord Data Packages (GDPR packages). This A
 
 * [Architecture Documentation](#architecture-documentation)
 * [Self-hosting](#self-hosting)
+* [Deploy to AWS](#deploy-to-aws)
 * [API Documentation](#api-documentation)
 * [Troubleshooting](#troubleshooting)
 
@@ -32,18 +33,59 @@ Thus:
 
 Anyone can host their own Dumpus instance. The official Dumpus client can then be configured to use it.
 
+The worker no longer runs as a separate Celery process — set `QUEUE_BACKEND=sync` and the API processes packages inline on the request thread (good enough at small volume), or set `QUEUE_BACKEND=sqs` to dispatch to AWS SQS (used by the Lambda deployment).
+
 * clone <https://github.com/dumpus-app/dumpus-api>
-* you can use Docker (easy way)
-* or you can install everything by yourself:
+* easy way: `cp .env.example .env`, fill it in, then `make up`
+* manual:
     - install requirements with pip
-    - start a RabbitMQ server (or redis)
     - start a PostgreSQL server
-    - fill the .env file with your Redis and PostgreSQL creds
-    - start the API using `waitress-serve --port=5000 app:app`
-    - start one worker using `celery --app tasks worker --loglevel=info --queues=regular_process --hostname=regular_process@%h --concurrency=1`
+    - fill the .env file with your PostgreSQL creds
+    - start the API: `QUEUE_BACKEND=sync waitress-serve --port=5000 app:app`
 
 By default, Dumpus API will only treat zip files sent from `https://discord.click`. You can specify a `DL_ZIP_WHITELISTED_DOMAINS` environment variable to add other allowed domains.
 
+## Deploy to AWS
+
+A Terraform stack under `infra/terraform/` provisions a serverless deployment of the API on AWS:
+
+| Component        | AWS service              |
+| ---------------- | ------------------------ |
+| API              | Lambda (container image) behind API Gateway HTTP API |
+| Worker           | Lambda triggered by SQS                              |
+| Database         | RDS Postgres in private subnets                      |
+| Outbound NAT     | fck-nat instance (NAT Gateway replacement)           |
+| Secrets          | Secrets Manager + Lambda env                         |
+| TLS / DNS        | ACM cert + Route53 alias to API Gateway              |
+| CI               | GitHub OIDC role; build → ECR → `update-function-code` |
+
+### Bootstrap
+
+1. Create a public Route53 hosted zone for your domain and point your registrar's nameservers at it.
+2. `cp infra/terraform/terraform.tfvars.example infra/terraform/terraform.tfvars` and fill in `discord_secret`, `domain_name`, `github_repository`, region, etc.
+3. `cd infra/terraform && terraform init && terraform apply`.
+   This single apply does everything: a `null_resource` pushes a placeholder image (the public AWS Lambda Python base) into ECR with the `:bootstrap` tag, then the Lambda functions are created against that placeholder. Requires `docker` and `aws` CLI on the apply host.
+4. Set the GitHub repo secret `AWS_DEPLOY_ROLE_ARN` from `terraform output -raw github_deploy_role_arn`. From here on, every push to `main` builds the real image in CI and rolls both Lambdas — no more local builds needed.
+
+### Day-to-day deploys
+
+Push to `main` → `.github/workflows/deploy.yml` builds the container, pushes to ECR tagged with the git SHA, and rolls both Lambdas via OIDC. No long-lived AWS keys in GitHub.
+
+### Operations
+
+```bash
+aws logs tail /aws/lambda/<name-prefix>-<env>-api --follow
+aws logs tail /aws/lambda/<name-prefix>-<env>-worker --follow
+aws sqs receive-message --queue-url "$(terraform output -raw sqs_dlq_url)"
+```
+
+Things to keep in mind:
+
+- **API cold start** is a few seconds while pandas imports. Invisible on the async submit/poll flow; use provisioned concurrency if a sync endpoint must be sub-second.
+- **Worker `/tmp` cap** defaults to 5GB (max 10GB). Bump `worker_ephemeral_storage_mb` if users upload very large Discord exports.
+- **Worker timeout** is 15 min (Lambda hard cap). Failures land in the DLQ after one retry.
+- **fck-nat is a single instance.** Switch to a managed NAT Gateway if you need the extra availability — at the cost of a much higher fixed monthly bill.
+
 ## API Documentation
 
 One header is required for all the requests except the `POST /process` one:
@@ -167,12 +209,5 @@ Current error message codes:
 
 ## Troubleshooting
 
-* Server does not respond after `POST /process`. Try to remove this property from the celeryconfig.py file.
-```
-broker_use_ssl={
-    'ssl_cert_reqs': None
-}
-```
-
-* API server is crashing and say that Postgres is not supported.  
+* API server is crashing and says that Postgres is not supported.
 Make sure that your PostgreSQL server URL starts with **postgresql://** and not **postgres://**, which is no longer supported by SQLAlchemy.