feat: presigned-S3 download path for package blobs#57
Merged
Conversation
API Gateway HTTP API + apig-wsgi base64-rounds binary responses and
doesn't stream them, which throttles downloads of the encrypted SQLite
to ~17KB/s (a 100MB package would take 100min). Bypass the bottleneck
by handing the client a presigned S3 URL.
Additive only — nothing existing is changed:
- New S3 bucket dumpus-prod-package-data (private, encrypted, 90d
lifecycle, CORS for browser fetches). Worker uploads each finished
encrypted blob via a VPC gateway endpoint so the bytes don't transit
fck-nat.
- Worker dual-writes: still inserts the legacy DB row, ALSO uploads to
S3. S3 failure is logged-and-ignored so the legacy /data path keeps
working.
- New endpoint GET /process/<id>/blob returns
{url, iv, ttl} — same auth contract as /data, returns 409 if the
blob isn't in S3 yet (caller falls back to /data).
- /data is untouched. Frontend can migrate to /blob at its own pace,
decrypt locally with the UPN, and we deprecate /data later.
4 tasks
Androz2091
added a commit
that referenced
this pull request
May 1, 2026
Demo doesn't go through the worker, so the previous S3 PR (#57) didn't help it — /demo/data was still ~25s. - Move demo handling into /blob (the new endpoint), not /data (the endpoint we're deprecating). - /blob/demo: no auth, lazy-seeds S3 from generate_demo_database() on the first call after deploy, returns a presigned URL with iv=null. - /blob/<real-package>: unchanged (auth + DB row + presigned URL with iv). - /data left fully untouched. - blob_storage.upload() replaces the misleadingly-named upload_encrypted; worker call site updated.
Androz2091
added a commit
that referenced
this pull request
May 1, 2026
Demo doesn't go through the worker, so the previous S3 PR (#57) didn't help it — /demo/data was still ~25s. - Move demo handling into /blob (the new endpoint), not /data (the endpoint we're deprecating). - /blob/demo: no auth, lazy-seeds S3 from generate_demo_database() on the first call after deploy, returns a presigned URL with iv=null. - /blob/<real-package>: unchanged (auth + DB row + presigned URL with iv). - /data left fully untouched. - blob_storage.upload() replaces the misleadingly-named upload_encrypted; worker call site updated.
Androz2091
added a commit
that referenced
this pull request
May 1, 2026
Demo doesn't go through the worker, so the previous S3 PR (#57) didn't help it — /demo/data was still ~25s. - Move demo handling into /blob (the new endpoint), not /data (the endpoint we're deprecating). - /blob/demo: no auth, lazy-seeds S3 from generate_demo_database() on the first call after deploy, returns a presigned URL with iv=null. - /blob/<real-package>: unchanged (auth + DB row + presigned URL with iv). - /data left fully untouched. - blob_storage.upload() replaces the misleadingly-named upload_encrypted; worker call site updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the slow download path on
/process/<id>/data(~17KB/s through API Gateway because of base64 round-trip + lack of streaming). Adds an additive/blobendpoint that returns a presigned S3 URL the client can fetch directly — bypassing API Gateway entirely.How it works
Worker now dual-writes:
encrypted_datarow in Postgres (legacy, unchanged)dumpus-prod-package-data) under a deterministic keyAPI gets a new endpoint
GET /process/<id>/blobthat:/data(Bearer UPN matchespackage_id){url, iv, ttl}JSON —urlis a presigned S3 GET (5min default TTL),ivlets the client AES-decrypt locally/dataPACKAGE_DATA_BUCKETenv isn't set (local dev) → caller falls back to/data/datais untouched. Migration is opt-in; frontend can switch when ready, then we deprecate/datain a follow-up.Why safe
s3:PutObjecton the bucket for worker,s3:GetObjecton the bucket for API)Required frontend change (separate, not blocking)
Switch from
GET /data(binary plaintext SQLite) to:GET /blob→ parse JSONfetch(url)→ encrypted bytesSHA256(UPN)as key,ivfrom the JSONThis actually implements the README's stated security model ("encryption key must always remain on the client side") — currently the server transiently decrypts.
Test plan
gh workflow run infra.yml -f action=apply) — should add S3 bucket, IAM statements, env vars, S3 VPC endpoint; no changes to existing resourcesPOST /processand wait forPROCESSEDGET /process/<id>/blobreturns 200 + JSON with a working presigned URLcurl -L $urldownloads the encrypted blob fast (direct from S3)GET /process/<id>/datastill works (legacy path)GET /process/demo/blobreturns 409 (demo never goes through worker)