Skip to content

Commit 05f5548

Browse files
authored
fix(migrations): drop CONCURRENTLY from 0201 indexes (#830)
## Summary - Switches `0201_backfill_missing_reward_disbursements.sql` from `CREATE INDEX CONCURRENTLY` to plain `CREATE INDEX` inside the migration's `BEGIN/COMMIT`. - Both indexes (`sol_reward_disbursements (challenge_id, specifier)` and `sol_claimable_accounts (ethereum_address, mint, slot DESC)`) are now atomic with the backfill INSERT — if anything fails, the schema rolls back cleanly. ## Why `CREATE INDEX CONCURRENTLY` waits on a `virtualxid` lock for every transaction open during its build phases — not just transactions that touch the target table, but every one in the cluster. The legacy Python `index_rewards_manager` Celery task on discovery-provider keeps ~3-minute transactions open against `challenge_disbursements` continuously. As fast as one ends, another is already open. So the CONCURRENTLY build can wait indefinitely without ever seeing a quiet moment — and it did, for 10+ minutes blocked on `Lock/virtualxid` in tonight's deploy. Trade-off accepted: regular `CREATE INDEX` takes a `ShareLock` on the target table for the duration of the build, blocking writes. But both target tables are written only by the Go indexer, and only on reward_manager `EvaluateAttestations` and claimable token `Create` instructions — sparse on-chain. At current row counts each build completes in seconds; the blocked writes just queue on pgxpool and resume right after. ## Test plan - [ ] Cancel any in-flight 0201 attempt and drop any invalid index it left behind: ```sql SELECT pg_cancel_backend(pid) FROM pg_stat_activity WHERE query ILIKE 'CREATE INDEX CONCURRENTLY%'; DROP INDEX IF EXISTS sol_reward_disbursements_challenge_specifier_idx; DROP INDEX IF EXISTS sol_claimable_accounts_eth_mint_slot_idx; ``` - [ ] Roll the new image; migration Job's `bridge migrate` should complete in well under a minute. - [ ] Verify both indexes exist as `indisvalid = true`: ```sql SELECT indexrelid::regclass, indisvalid FROM pg_index WHERE indexrelid::regclass::text IN ( 'sol_reward_disbursements_challenge_specifier_idx', 'sol_claimable_accounts_eth_mint_slot_idx' ); ``` - [ ] Verify missing-row count drops as expected (per #829's test plan). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent ff5ea5e commit 05f5548

1 file changed

Lines changed: 20 additions & 11 deletions

File tree

ddl/migrations/0201_backfill_missing_reward_disbursements.sql

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -15,24 +15,33 @@
1515
-- account. Rows whose user record no longer exists are intentionally skipped;
1616
-- they would need on-chain signature replay (via program.Indexer) to recover.
1717

18-
-- CREATE INDEX CONCURRENTLY cannot run inside an explicit transaction, so
19-
-- these statements stay outside the BEGIN/COMMIT below. psql executes each in
20-
-- its own implicit transaction. Both indexes pay off well beyond this
21-
-- migration: the first lets the dedup LEFT JOIN on (challenge_id, specifier)
22-
-- use an index instead of a sequential scan; the second lets the live
23-
-- reward_manager indexer (and this migration's LATERAL lookup) resolve a
24-
-- user's current claimable account in O(log n) rather than scanning the table.
18+
BEGIN;
19+
20+
-- Build both indexes inside the migration transaction (non-CONCURRENTLY) so
21+
-- the writes are atomic with the backfill and not subject to virtualxid waits
22+
-- against concurrent transactions on this DB (notably the legacy Python
23+
-- index_rewards_manager Celery task on discovery-provider, which keeps long
24+
-- challenge_disbursements transactions open and can make CONCURRENTLY hang
25+
-- indefinitely). Regular CREATE INDEX takes a ShareLock on the target table:
26+
-- writes are blocked for the duration of the build, but both target tables
27+
-- have light write load (reward_manager EvaluateAttestations and claimable
28+
-- token creations are sparse on-chain) and the build itself completes in
29+
-- seconds at current row counts.
30+
--
31+
-- Both indexes pay off well beyond this migration: the first lets the dedup
32+
-- LEFT JOIN on (challenge_id, specifier) use an index instead of a sequential
33+
-- scan; the second lets the live reward_manager indexer (and the CTE below)
34+
-- resolve a user's current claimable account in O(log n) rather than scanning
35+
-- the table.
2536

26-
CREATE INDEX CONCURRENTLY IF NOT EXISTS
37+
CREATE INDEX IF NOT EXISTS
2738
sol_reward_disbursements_challenge_specifier_idx
2839
ON sol_reward_disbursements (challenge_id, specifier);
2940

30-
CREATE INDEX CONCURRENTLY IF NOT EXISTS
41+
CREATE INDEX IF NOT EXISTS
3142
sol_claimable_accounts_eth_mint_slot_idx
3243
ON sol_claimable_accounts (ethereum_address, mint, slot DESC);
3344

34-
BEGIN;
35-
3645
-- Skip the on_sol_reward_disbursement trigger for this transaction. The
3746
-- trigger fires per-row to create challenge_reward notifications and a
3847
-- pg_notify announcement for the Python ChallengeEventBus. For a one-shot

0 commit comments

Comments
 (0)