feat: Implement queue-based async DB export pipeline - #188

Merged

yash-pouranik merged 29 commits into

geturbackend:mainfrom

renganathc:feature/db-export-pipeline

May 28, 2026

Commits on May 28, 2026

feat: setup consumer server
renganathc
committed
feat: configure Express server with CORS, CSRF, rate limiting, and graceful shutdown
renganathc
committed
refractor: reuse existing auth middleware
renganathc
committed
feat: implement handler in project controller
renganathc
committed
feat: create project route and add db export endpoint
renganathc
committed
feat: add project route to express app
renganathc
committed
feat: add BullMQ queue for db export jobs
renganathc
committed
refactor: move db export job creation to dashboard API
renganathc
committed
refactor: move export endpoint to projects router and remove dbExport router
renganathc
committed
feat: implement project ownership verifcation and queuing logic in export handler
renganathc
committed
feat: implement plan based rate limiting
renganathc
committed
feat: cache project lookup in Redis before MongoDB fallback
renganathc
committed
feat: implement the DB export worker logic
renganathc
committed
feat: add export email job handling to email worker
renganathc
committed
feat: implement DB export worker completion and failure handling and initialize worker
renganathc
committed
feat: add sendExportReadyEmail function to emailService.js
renganathc
committed
fix: modify emailQueue.js to use sendExportReadyEmail fn
renganathc
committed
fix (package.json): modify dev script to also run consumer server
renganathc
committed
refactor: add dedicated consumer entrypoint and graceful shutdown
renganathc
committed
chore: add Dockerfile
renganathc
committed
feat(storage): Implement function to return unified S3Client for all storage providers
renganathc
committed
fix (storage): Remove bucket from getS3Storage return body, rename ambiguous env vars to standardized storage config names
renganathc
committed
feat: update export worker to use getS3CompatibleStorage function, stream using PassThrough
renganathc
committed
fix: CodeQL format string warning in export logger
renganathc
committed
refactor(export-worker): route exports through legacy getStorage abstraction
renganathc
committed
fix (db-export): modify db export controller and endpoint to accept and export individual collections
renganathc
committed
fix (db-export): modify export worker to ahndle individual collection exports
renganathc
committed
fix (email-queue): Implement error handling for 'send-export-email'
renganathc
committed
fix: update res structure in export controller and handle unknown job type in email queue
renganathc
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Implement queue-based async DB export pipeline - #188

feat: Implement queue-based async DB export pipeline#188
yash-pouranik merged 29 commits into
geturbackend:mainfrom
renganathc:feature/db-export-pipeline

Commits on May 28, 2026

feat: setup consumer server

feat: configure Express server with CORS, CSRF, rate limiting, and graceful shutdown

refractor: reuse existing auth middleware

feat: implement handler in project controller

feat: create project route and add db export endpoint

feat: add project route to express app

feat: add BullMQ queue for db export jobs

refactor: move db export job creation to dashboard API

refactor: move export endpoint to projects router and remove dbExport router

feat: implement project ownership verifcation and queuing logic in export handler

feat: implement plan based rate limiting

feat: cache project lookup in Redis before MongoDB fallback

feat: implement the DB export worker logic

feat: add export email job handling to email worker

feat: implement DB export worker completion and failure handling and initialize worker

feat: add sendExportReadyEmail function to emailService.js

fix: modify emailQueue.js to use sendExportReadyEmail fn

fix (package.json): modify dev script to also run consumer server

refactor: add dedicated consumer entrypoint and graceful shutdown

chore: add Dockerfile

feat(storage): Implement function to return unified S3Client for all storage providers

fix (storage): Remove bucket from getS3Storage return body, rename ambiguous env vars to standardized storage config names

feat: update export worker to use getS3CompatibleStorage function, stream using PassThrough

fix: CodeQL format string warning in export logger

refactor(export-worker): route exports through legacy getStorage abstraction

fix (db-export): modify db export controller and endpoint to accept and export individual collections

fix (db-export): modify export worker to ahndle individual collection exports

fix (email-queue): Implement error handling for 'send-export-email'

fix: update res structure in export controller and handle unknown job type in email queue