feat: sequin backfill indexes

[epipav]

No description provided.

[cursor]

PR Summary

Medium Risk
Many concurrent index builds on large partitioned tables can stress production DB during migration and leave invalid indexes if runs fail; changes are additive but operationally sensitive.

Overview
Adds a Flyway migration that introduces partitioned btree indexes so Sequin backfill work can scan versions and package_dependencies in a stable sort order.

For versions, it creates a parent index on (last_synced_at, id, package_id), builds matching indexes concurrently on partitions versions_p0–p31, cleans up any invalid leftover partition indexes from prior failed runs, then attaches each partition index to the parent.

The same pattern is applied to package_dependencies with (updated_at, id, depends_on_id) across partitions p0–p63.

^{Reviewed by Cursor Bugbot for commit 2f59444. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

Adds Sequin backfill sort indexes for package versions and dependencies.

Changes:

• Adds composite indexes for Sequin cursor-based backfills.
• Uses concurrent index creation to limit write disruption.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

backend/src/osspckgs/migrations/V1784745489__sequin_backfill_sort_column_indexes.sql:11

• Scope the catalog lookup and drop to the target schema. pg_class covers every schema, so an invalid same-named index elsewhere can enter this loop; the unqualified DROP INDEX may then resolve to and delete the valid public index instead. Select the namespace and drop a schema-qualified identifier.

    SELECT c.relname FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
    WHERE NOT i.indisvalid AND c.relname ~ '^versions_p[0-9]+_last_synced_at_id_package_id_idx$'
  LOOP
    EXECUTE format('DROP INDEX IF EXISTS %I', idx);

backend/src/osspckgs/migrations/V1784745489__sequin_backfill_sort_column_indexes.sql:106

• Scope this cleanup to the target schema as well. Because the scan is database-wide but the drop is unqualified, a matching invalid index in another schema can cause PostgreSQL to drop a valid public index with the same name.

    SELECT c.relname FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
    WHERE NOT i.indisvalid AND c.relname ~ '^package_dependencies_p[0-9]+_updated_at_id_depends_on_id_idx$'
  LOOP
    EXECUTE format('DROP INDEX IF EXISTS %I', idx);

[epipav]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2f59444. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

backend/src/osspckgs/migrations/V1784745489__sequin_backfill_sort_column_indexes.sql:11

• This retry cleanup uses a non-concurrent DROP INDEX on a partition that receives live writes. PostgreSQL takes an exclusive table lock for a normal drop, so after any failed concurrent build this migration can queue behind traffic and stall subsequent writes—the exact outage risk the concurrent builds avoid. Use top-level DROP INDEX CONCURRENTLY statements for the leaf indexes (they cannot run inside this DO block), as established by V1784718694__repos_github_lower_url_guard.sql:14-20.

    EXECUTE format('DROP INDEX IF EXISTS %I', idx);

backend/src/osspckgs/migrations/V1784745489__sequin_backfill_sort_column_indexes.sql:106

• The same retry-path locking issue affects the 64 package_dependencies partitions: a normal DROP INDEX takes an exclusive table lock and can stall continuous dependency ingestion after a failed build. Move this cleanup to top-level DROP INDEX CONCURRENTLY statements before rebuilding each leaf index; PostgreSQL does not allow the concurrent form inside a DO block.

    EXECUTE format('DROP INDEX IF EXISTS %I', idx);