fix(operator): un-gate readiness on needs-reindex-all, REINDEX DATABASE CONCURRENTLY

[passcod]

🤖

Summary

Two changes that together unblock nauru-prod (and any replica whose snapshot triggers pg_resetwal):

1. Drop the amcheck-driven smart pass for needs-reindex-all; replace it with a blind REINDEX DATABASE per user DB — CONCURRENTLY on PG ≥ 12 so clients can keep using the old indexes during the rebuild.
2. Drop the readiness probe's gate on /pgdata/needs-reindex-all. Keep the gate on the locale-only /pgdata/needs-reindex (small, fast, proven).

Why

Verified live on nauru-prod after #68 deployed. Every restore attempt failed at the operator's 30-minute deployment_ready_timeout. Tracing inside a fresh pod:

• /pgdata/needs-reindex-all flag present (pg_resetwal did fire — expected on this source).
• pg_stat_activity shows the background reindex hook running.
• bt_index_check('pg_type_oid_index'::regclass) (a tiny system catalog index, normally a few KB) — state active, empty wait_event, query_start growing linearly with wall clock. 4+ minutes on that one index with no visible progress.

Same shape as the CREATE TABLE spinloop documented earlier on this dataset: vanilla DDL pegs 100% CPU forever inside postgres with no visible reason. The smart pass via amcheck can't avoid the hammer — it IS a hammer that triggers the same wedge.

REINDEX DATABASE CONCURRENTLY reads the heap (not the index) to rebuild from scratch — a completely different code path that isn't subject to whatever amcheck is hitting in the corrupt index pages. PG ≥ 12 syntax; falls back to blocking REINDEX DATABASE on older.

But REINDEX DATABASE on the prod-sized DBs takes hours. Gating readiness on it makes deployment_ready_timeout (30 min default) trip every time and the restore fails. So the second change is necessary: let the pod become Ready as soon as postgres accepts connections; let the reindex run in background. CONCURRENTLY means the corrupt indexes stay queryable until the atomic swap, narrowing the window where clients can see corruption.

Caveats

• REINDEX DATABASE CONCURRENTLY skips system catalogs (PG won't CONCURRENTLY them). For an analytics replica that's the right trade — user-data indexes are what client queries hit; system catalog corruption surfaces as different errors and is rare.

Trade-off

Clients hitting a not-yet-reindexed corrupt index get the explicit unexpected zero page at block N error from postgres — same error the downstream users originally reported. With CONCURRENTLY the window is narrow (queries hit the old index until the atomic swap); they retry, succeed once the rebuild lands.

Strictly better than the alternative (permanently-failed restore, replica stuck indefinitely on the previous Active).

Shape

• builders.rs: the needs-reindex-all branch in the postgres container startup hook replaces the amcheck loop with REINDEX DATABASE CONCURRENTLY "$db" per user database (with a PG < 12 fallback). Comment explains the wedge and why amcheck is wrong here.
• builders.rs: readiness probe drops the needs-reindex-all check. Comment explains.
• Tests: updated assertions — runtime hook uses REINDEX DATABASE CONCURRENTLY, gates on PG_MAJOR ≥ 12, no bt_index_check; readiness probe gates only on the locale-only flag.

Recovery

For replicas already in the failure loop (pre-fix gate), the readiness probe will start returning success on its next check after the operator rolls the new image (the needs-reindex-all file may still be there, but the probe no longer cares). The background reindex keeps churning. For the current live nauru-prod pod, I removed the flag by hand to unblock immediately — the next restore will exercise the new code path.