Skip to content

feat: presigned-S3 download path for package blobs#57

Merged
Androz2091 merged 1 commit into
mainfrom
feat/s3-presigned-data
May 1, 2026
Merged

feat: presigned-S3 download path for package blobs#57
Androz2091 merged 1 commit into
mainfrom
feat/s3-presigned-data

Conversation

@Androz2091

Copy link
Copy Markdown
Member

Summary

Fixes the slow download path on /process/<id>/data (~17KB/s through API Gateway because of base64 round-trip + lack of streaming). Adds an additive /blob endpoint that returns a presigned S3 URL the client can fetch directly — bypassing API Gateway entirely.

How it works

Worker now dual-writes:

  1. encrypted_data row in Postgres (legacy, unchanged)
  2. Encrypted blob to a new private S3 bucket (dumpus-prod-package-data) under a deterministic key

API gets a new endpoint GET /process/<id>/blob that:

  • Same auth as /data (Bearer UPN matches package_id)
  • Returns {url, iv, ttl} JSON — url is a presigned S3 GET (5min default TTL), iv lets the client AES-decrypt locally
  • Returns 409 if the row exists but no S3 blob has been uploaded yet → caller falls back to /data
  • Returns 501 if PACKAGE_DATA_BUCKET env isn't set (local dev) → caller falls back to /data

/data is untouched. Migration is opt-in; frontend can switch when ready, then we deprecate /data in a follow-up.

Why safe

  • Pure additive: no existing endpoint or schema changed
  • Worker S3 upload failure is logged-and-swallowed (legacy DB write still happens)
  • New IAM perms are minimal and resource-scoped (s3:PutObject on the bucket for worker, s3:GetObject on the bucket for API)
  • VPC gateway endpoint for S3 means worker→S3 traffic doesn't transit fck-nat (free, and avoids putting big upload bytes through the t4g.nano)
  • 90-day lifecycle on the bucket auto-cleans old blobs

Required frontend change (separate, not blocking)

Switch from GET /data (binary plaintext SQLite) to:

  1. GET /blob → parse JSON
  2. fetch(url) → encrypted bytes
  3. AES-CBC decrypt with SHA256(UPN) as key, iv from the JSON

This actually implements the README's stated security model ("encryption key must always remain on the client side") — currently the server transiently decrypts.

Test plan

  • Apply infra (gh workflow run infra.yml -f action=apply) — should add S3 bucket, IAM statements, env vars, S3 VPC endpoint; no changes to existing resources
  • Push to main → deploy.yml ships new app code
  • Submit a real package via POST /process and wait for PROCESSED
  • GET /process/<id>/blob returns 200 + JSON with a working presigned URL
  • curl -L $url downloads the encrypted blob fast (direct from S3)
  • GET /process/<id>/data still works (legacy path)
  • GET /process/demo/blob returns 409 (demo never goes through worker)

API Gateway HTTP API + apig-wsgi base64-rounds binary responses and
doesn't stream them, which throttles downloads of the encrypted SQLite
to ~17KB/s (a 100MB package would take 100min). Bypass the bottleneck
by handing the client a presigned S3 URL.

Additive only — nothing existing is changed:

- New S3 bucket dumpus-prod-package-data (private, encrypted, 90d
  lifecycle, CORS for browser fetches). Worker uploads each finished
  encrypted blob via a VPC gateway endpoint so the bytes don't transit
  fck-nat.
- Worker dual-writes: still inserts the legacy DB row, ALSO uploads to
  S3. S3 failure is logged-and-ignored so the legacy /data path keeps
  working.
- New endpoint GET /process/<id>/blob returns
  {url, iv, ttl} — same auth contract as /data, returns 409 if the
  blob isn't in S3 yet (caller falls back to /data).
- /data is untouched. Frontend can migrate to /blob at its own pace,
  decrypt locally with the UPN, and we deprecate /data later.
@Androz2091 Androz2091 merged commit a37cbbc into main May 1, 2026
1 check passed
@Androz2091 Androz2091 deleted the feat/s3-presigned-data branch May 1, 2026 02:48
Androz2091 added a commit that referenced this pull request May 1, 2026
Demo doesn't go through the worker, so the previous S3 PR (#57) didn't
help it — /demo/data was still ~25s.

- Move demo handling into /blob (the new endpoint), not /data (the
  endpoint we're deprecating).
- /blob/demo: no auth, lazy-seeds S3 from generate_demo_database() on
  the first call after deploy, returns a presigned URL with iv=null.
- /blob/<real-package>: unchanged (auth + DB row + presigned URL with iv).
- /data left fully untouched.
- blob_storage.upload() replaces the misleadingly-named
  upload_encrypted; worker call site updated.
Androz2091 added a commit that referenced this pull request May 1, 2026
Demo doesn't go through the worker, so the previous S3 PR (#57) didn't
help it — /demo/data was still ~25s.

- Move demo handling into /blob (the new endpoint), not /data (the
  endpoint we're deprecating).
- /blob/demo: no auth, lazy-seeds S3 from generate_demo_database() on
  the first call after deploy, returns a presigned URL with iv=null.
- /blob/<real-package>: unchanged (auth + DB row + presigned URL with iv).
- /data left fully untouched.
- blob_storage.upload() replaces the misleadingly-named
  upload_encrypted; worker call site updated.
Androz2091 added a commit that referenced this pull request May 1, 2026
Demo doesn't go through the worker, so the previous S3 PR (#57) didn't
help it — /demo/data was still ~25s.

- Move demo handling into /blob (the new endpoint), not /data (the
  endpoint we're deprecating).
- /blob/demo: no auth, lazy-seeds S3 from generate_demo_database() on
  the first call after deploy, returns a presigned URL with iv=null.
- /blob/<real-package>: unchanged (auth + DB row + presigned URL with iv).
- /data left fully untouched.
- blob_storage.upload() replaces the misleadingly-named
  upload_encrypted; worker call site updated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant