feat: deploy API + workers to AWS Lambda via Terraform#54
Merged
Conversation
Move from docker-compose to a fully-managed AWS deployment: - API as a Lambda behind API Gateway HTTP API at api.dumpus.app - Worker as an SQS-triggered Lambda (replaces the always-on Celery worker) - RDS Postgres 17 + fck-nat for ~$22/mo total - Drops Celery + RabbitMQ entirely; QUEUE_BACKEND=sync runs the worker inline for local dev, QUEUE_BACKEND=sqs in production - GitHub Actions deploy via OIDC: build container -> ECR -> update both Lambdas - Terraform stack under infra/terraform with full bootstrap docs
During the OVH→Route53 cutover the api record is pre-populated with the existing origin IP to avoid downtime while nameservers propagate. allow_overwrite = true lets Terraform take ownership and swap it for the API Gateway alias on apply.
Sentiment analysis pulled in nltk + a baked-in vader_lexicon (~50MB), which inflated the Lambda image and added cold-start cost. Stub count_sentiments() to a constant 0.5 for now. The TODO in tasks.py calls out how to revive it. - remove nltk from requirements - drop the vader_lexicon download from both Dockerfiles - drop scripts/download-ntk.py and the install-step Makefile reference
Move the contents of infra/terraform/README.md into a new "Deploy to AWS" section in the project README, so contributors discover it from one place. Also refresh the self-hosting steps for the post-Celery world (sync mode, no broker) and drop the now-stale celeryconfig.py troubleshooting tip.
The README is shipped with the project; cost figures, traffic estimates, and credit accounting only make sense for one operator and confuse everyone else who self-hosts. Drop those, generalize the log-group paths, and tighten the language.
Androz2091
force-pushed
the
feat/aws-lambda-deploy
branch
from
57e4e3f to
f097593
Compare
Prevents an apply from touching the wrong AWS account when AWS_PROFILE or env credentials accidentally point somewhere else (e.g. a company account on the same workstation). When set in terraform.tfvars, the AWS provider hard-fails before planning if the resolved account isn't in the allow-list.
Latest releases checked against the upgrade guides; only one breaking change in our resource set (aws_ami stricter validation). - AWS provider 5.70 -> 6.0 (latest 6.43) - Add explicit owner-id filter to aws_ami.fck_nat for v6's stricter most_recent validation - apig-wsgi 2.18.0 -> 2.20.0 - actions/checkout v4 -> v6 - aws-actions/configure-aws-credentials v4 -> v6 - docker/setup-buildx-action v3 -> v4 - docker/build-push-action v6 -> v7 Python 3.10 is deprecated by Lambda on Oct 31 2026. Bumping the runtime needs numpy/pandas/Flask version bumps and is left for a follow-up PR.
The bootstrap commands in the README reference `terraform output -raw region`, which didn't exist. Adding it so those commands work as written.
…ual step CreateFunction validates that the image_uri exists, so the very first apply used to fail unless you'd manually pushed a :bootstrap image to ECR first. Add a null_resource that runs once after the ECR repo is created: pulls public.ecr.aws/lambda/python:3.10, retags it to :bootstrap, pushes to our ECR. Lambdas depend on it, so apply ordering becomes ECR -> placeholder image -> Lambdas. Idempotent: re-applies skip if the tag already exists. After first apply, CI takes over. The placeholder image gets overwritten by the first push to main, and Terraform never touches image_uri again (lifecycle.ignore_changes already in place). Requires docker + aws CLI on the apply host.
- secrets.tf: drop special chars from RDS master password. !#$%&*()-_=+[]{}<>:?
included `:`, `?`, `&`, `=` which would break SQLAlchemy URL parsing
any time the random password happened to contain one. 32 alnum chars give
~190 bits of entropy anyway. (real bug)
- Dockerfile.api: bookworm replaces buster (Debian 10 EOL'd Jun 2024).
- bootstrap.tf: pin the placeholder image to python:3.13 instead of 3.10
(3.10 is being deprecated by Lambda). The placeholder is replaced on
first deploy, so this is purely about which tag we pull from public ECR.
- main.tf: drop unused common_tags local and aws_caller_identity data.
- variables.tf: rephrase worker_reserved_concurrency description to avoid
project-specific framing.
- terraform.tfvars.example: rewrite the image_tag comment so it matches
the actual lifecycle (Terraform owns the first apply, CI owns updates,
ignore_changes prevents drift).
- .gitignore: stop ignoring .terraform.lock.hcl — provider lock should be
committed so applies are reproducible. Adds the lock at v6.43.0.
The migration is one-time. Once it's done, the comment is archaeology; allow_overwrite is meaningful on its own (lets Terraform take ownership of an existing record).
Lets users with limited local bandwidth (provider download is ~600MB) run plan/apply from CI instead. The workflow uses static AWS keys stored as GH secrets for the bootstrap, and remote state in S3 (set up out-of-band via 'aws s3 mb' before the first run). After the OIDC role is created on first apply, deploy.yml takes over for code deploys. terraform.ci.tfvars carries the non-sensitive vars (region, account ID, domain). Sensitive vars (discord_secret, diswho_jwt_secret) come in via TF_VAR_* env from GH secrets and never land in the repo.
Lambda env no longer carries the actual secret values — only a JSON map of ARNs (SECRETS_ARN_MAP). On cold start, secrets_loader.py reads the real values from Secrets Manager via boto3 and hydrates os.environ, so the rest of the app keeps using os.getenv() unchanged. What this buys us: - secret values aren't visible to anyone with lambda:GetFunction - rotate via 'aws secretsmanager put-secret-value' without redeploying the Lambda (versioned secret_string + ignore_changes prevents tofu apply from reverting the rotation) - every secret read shows up in CloudTrail Other changes: - add WH_URL secret + variable so internal Discord webhook notifications work in prod (was missing entirely before) - discord_secret defaults to '' since it's only used as a fallback when diswho_base_url is unset - gitignore .env.* (except .env.example) so .env.bootstrap can't be accidentally committed - workflow plumbs TF_VAR_wh_url alongside the other secret vars
1 task
The fallback path in util.py.fetch_diswho_user that uses DISCORD_SECRET only runs when DISWHO_BASE_URL is unset. Our deployment has it set, so the secret is dead config. Drop: - the discord_secret TF variable - the Secrets Manager entry + version - the IAM read permission - the Lambda env entry in SECRETS_ARN_MAP - the TF_VAR_discord_secret plumbing in infra.yml - the matching GH repo secret util.py keeps the fallback so self-hosters without a diswho instance can still configure DISCORD_SECRET via env.
Better service availability and lower cost. Latency from FR is ~30ms worse but still acceptable for this app's UX. Stack-wide change: - variables.tf, terraform.ci.tfvars, terraform.tfvars.example: var.region default - backend.tf: state bucket region (the bucket itself is being recreated in eu-west-1 out-of-band) - infra.yml + deploy.yml: aws-region used by the workflows
1. New AWS accounts have a 10-execution Lambda concurrency ceiling and require >=10 unreserved at all times, so reserved_concurrent_executions=1 was rejected with InvalidParameterValueException. Make the variable optional (0 = no reservation), default to 0. 2. EIP association raced with EC2 instance creation — the instance was still in pending-instance-creation state when the EIP tried to attach to its ENI. Add depends_on so the EIP attaches first, then the instance picks up the already-EIP-bound ENI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
api.dumpus.app; worker is an SQS-triggered Lambda that replaces the Celery workerenqueue.pypicks SQS in prod / a synchronous inline path for local devArchitecture
fck-natont4g.nanoreplaces the managed NAT Gateway/tmp, 15-min timeout, reserved concurrency 1What changed in app code
src/tasks.py:handle_package(Celery task) replaced byprocess_package(plain function). No Celery import, noceleryconfig.pysrc/enqueue.py:QUEUE_BACKEND=sqs(prod) orsync(local dev, default)src/lambda_handlers/api.py: Flask wrapped viaapig-wsgisrc/lambda_handlers/worker.py: SQS handler with partial-batch-failure reportingDockerfile.lambda: single image used by both functions; handler selected via LambdaImageConfig.Commanddocker-compose.yml,Makefile,.env.example: simplified — no more broker/worker containerRequired to deploy
dumpus.appin Route 53 (delegated from registrar)cp infra/terraform/terraform.tfvars.example terraform.tfvars, filldiscord_secretandgithub_repositoryterraform init && terraform applyAWS_DEPLOY_ROLE_ARNGitHub secret fromterraform output github_deploy_role_arnTest plan
terraform validateandterraform plansucceed against a fresh AWS accountterraform applycreates VPC / RDS / SQS / Lambdas / API GW / ACM cert without manual interventionActivestatecurl https://api.dumpus.app/healthreturns 200POST /processenqueues to SQS and the worker Lambda processes it end-to-end (status flips toPROCESSED)mainbuilds, pushes, redeploys both Lambdas, and the smoke-test passesmake devstill works inQUEUE_BACKEND=syncmode