Skip to content

Commit 21194f6

Browse files
bitnerclaude
andcommitted
Docs: make max_locks_per_transaction guidance prominent and complete
The lock-table wall is the most common way to get bitten at scale, so surface it loudly and document it fully: - Add a prominent warning callout right under the intro (before Quick Start). - Expand the Scaling bullet: CREATE INDEX fails too (not just queries), the exact error, how to set max_locks_per_transaction (ALTER SYSTEM + restart), and sizing (lock table ~= max_locks_per_transaction * (max_connections + max_prepared_transactions); a statement over N partitions needs ~2N locks). - Correct the threshold: with default max_locks_per_transaction=64 and max_connections=100 (~6400 lock slots) the wall is around a few thousand partitions, not ~10k. - Tighten the Limitations bullet to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 5210c50 commit 21194f6

1 file changed

Lines changed: 54 additions & 13 deletions

File tree

README.md

Lines changed: 54 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,22 @@ cannot eliminate. Pruning is conservative: a partition is removed only when its
1010
provably cannot contain a matching row, so results are always identical to running
1111
without it.
1212

13+
> ### ⚠️ Many partitions? Raise `max_locks_per_transaction` first
14+
>
15+
> Pruning (and the index build) on a non-key column requires PostgreSQL to lock **every**
16+
> partition in one transaction. On the **default `max_locks_per_transaction = 64`** this
17+
> exhausts the lock table at roughly **a few thousand partitions**, with:
18+
>
19+
> ```
20+
> ERROR: out of shared memory
21+
> HINT: You might need to increase "max_locks_per_transaction".
22+
> ```
23+
>
24+
> If you have thousands of partitions, **raise `max_locks_per_transaction` (it requires a
25+
> restart) before creating the index or querying** — see
26+
> [Scaling and partition count](#scaling-and-partition-count) for sizing. This is a
27+
> PostgreSQL limit on wide non-key access, not specific to this extension.
28+
1329
## Quick Start
1430
1531
Summaries are built and maintained through a custom index access method, so pruning
@@ -168,14 +184,38 @@ planner hook that prunes a non-key column before expansion, so this O(n) cost is
168184

169185
Two practical consequences and how to handle them:
170186

171-
- **Lock table exhaustion (a hard wall, ~10k partitions on defaults).** Any query touching
172-
a non-key column must lock *every* partition (and its indexes) while planning. With the
173-
default `max_locks_per_transaction = 64`, a query over ~10,000 partitions fails with
174-
`ERROR: out of shared memory` / `You might need to increase "max_locks_per_transaction"`.
175-
This is a PostgreSQL limit on wide non-key scans, not specific to this extension (native
176-
key pruning avoids it by never locking pruned partitions). **Mitigation:** raise
177-
`max_locks_per_transaction` (e.g. to a few thousand) and restart — it preallocates
178-
shared memory for the lock table, pushing the wall out in proportion.
187+
- **Lock table exhaustion (the hard wall).** Two operations lock *every* partition (and its
188+
indexes) in a single transaction:
189+
- **`CREATE INDEX … USING table_range`**, which builds one child index per partition — so
190+
on too many partitions the index can't even be *built* (it fails and rolls back, leaving
191+
no summary, so queries then scan everything);
192+
- **any query on a non-key column**, whose planning expands and locks all partitions.
193+
194+
On the default **`max_locks_per_transaction = 64`** the lock table holds only ~6,400 locks
195+
(with default `max_connections = 100`), so at roughly **a few thousand partitions** — where
196+
`2 × partitions` exceeds that — you get:
197+
198+
```
199+
ERROR: out of shared memory
200+
HINT: You might need to increase "max_locks_per_transaction".
201+
```
202+
203+
This is a PostgreSQL limit on wide non-key access, not specific to this extension — native
204+
key pruning avoids it by never locking pruned partitions.
205+
206+
**How to fix it.** Raise `max_locks_per_transaction`. It is a postmaster-level setting, so
207+
it **requires a restart**:
208+
209+
```sql
210+
ALTER SYSTEM SET max_locks_per_transaction = 4096; -- then restart PostgreSQL
211+
```
212+
213+
Sizing: the lock table holds about `max_locks_per_transaction × (max_connections +
214+
max_prepared_transactions)` locks, and one statement over *N* partitions needs roughly
215+
`2 × N` of them (a heap + an index lock per partition). Pick a value so that product
216+
comfortably exceeds `2 × N` for your largest partitioned table, with headroom for
217+
concurrency. With default `max_connections`, a few thousand (e.g. `4096`) covers tens of
218+
thousands of partitions; each lock slot costs only a few hundred bytes of shared memory.
179219
- **Planning time grows with partition count.** Even below the lock wall, planning scales
180220
linearly — though the per-partition constant is now small (~3–4 µs warm, on par with
181221
`CHECK` constraint exclusion) thanks to the per-plan compilation and backend summary
@@ -244,11 +284,12 @@ range-type tests, which exercise the same code path.
244284

245285
## Limitations
246286

247-
- **Planning is O(partitions)** for non-key predicates, with a hard lock-table wall around
248-
~10k partitions on default settings. See [Scaling](#scaling-and-partition-count) for the
249-
cause and mitigations (`max_locks_per_transaction`, prepared statements, fewer/larger
250-
partitions). Native partition-key pruning does not have this limit; table_range is for
251-
the cases native pruning cannot handle.
287+
- **Lock-table wall on many partitions.** Both `CREATE INDEX … USING table_range` and
288+
queries on a non-key column lock every partition at once, so on the default
289+
`max_locks_per_transaction = 64` they fail (`out of shared memory`) at roughly a few
290+
thousand partitions. **Raise `max_locks_per_transaction` (needs a restart)** — see
291+
[Scaling](#scaling-and-partition-count) for sizing. Native partition-key pruning does not
292+
have this limit; table_range is for the cases native pruning cannot handle.
252293
- Pruning is a **planning-time cost / execution-time win** tradeoff: on small partitions
253294
the per-plan overhead can exceed the scan it saves. Measure with
254295
`table_range.enable_pruning`.

0 commit comments

Comments
 (0)