Add Stripe Sigma reporting data schema support with parallel backfill

[Copilot]

Extends sync engine to ingest data from Stripe Sigma (reporting database) alongside existing API-based syncing. Enables comprehensive data warehousing by combining real-time API data with historical reporting tables.

Core Changes

• Sigma Integration: Query and sync Sigma reporting tables using generated configs from Stripe's reporting schema

  • Schema snapshot + generation tooling (fetch-schema.ts) for reproducible config updates
  • Runtime configs in sigmaIngestionConfigs.ts for supported tables
  • Sigma resources integrated into resource registry alongside API resources

• Parallel Backfill: New processUntilDoneParallel() with configurable concurrency (default 10)

  • Two-phase strategy: fast parallel historical sweep → incremental catchup
  • Graceful handling of inaccessible tables with skip/error tracking
  • Preserves dependency ordering via resource registry order field

• Resource Registry Refactor: Moved from static property to builder method

  • Unified registry for API resources (order 1-17) and Sigma tables (order 18+)
  • Consistent interface: listFn/upsertFn for API, sigma config for reporting tables

• Database Migrations: Added migration runner (migrate.ts) with Postgres integration

• Testing: Integration test suites for both API and Sigma sync paths

Example Usage

// Backfill with parallel Sigma ingestion
const result = await stripeSync.processUntilDoneParallel({
  triggeredBy: 'manual-backfill',
  maxParallel: 10,
  skipInaccessibleSigmaTables: true
});

console.log(`Synced ${result.totalSynced} rows across ${Object.keys(result.totals).length} objects`);

Notes

• Large generated files (~82K lines): schema_artifact.json and sigmaIngestionConfigs.ts
• Documentation added for Sigma schema generation workflow

Original prompt

Pull Request: https://github.com/stripe-experiments/sync-engine/pull/96/files

Summarize the changes for me

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.