fix(plan): bound right dedup memory

[aunjgr]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #22151

What this PR does / why we need it:

RIGHT DEDUP builds the target map and then inserts every incoming key while probing. That probe-side growth cannot transition into the current build-side spill engine, so large insert-select statements can retain tens of GiB and OOM a CN.

This PR:

• estimates RIGHT DEDUP's retained hash-map allocation using the runtime table's capacity and load-factor rules;
• accounts for target plus incoming keys and sums chained primary/unique-key DEDUP maps;
• falls back the entire chain to normal DEDUP when the estimate reaches the configured join spill threshold or default CN-relative stage budget;
• treats missing, non-finite, and sentinel default statistics conservatively;
• preserves the row-count semantics of small join_spill_mem values;
• keeps FLOAT normal DEDUP unshuffled because FLOAT hash shuffle is not supported yet, avoiding an invalid plan while retaining single-CN spill.

The default combined budget is approximately one eighth of CN memory after the file-cache reservation. Normal DEDUP hash-shuffles supported keys; FLOAT keys stay unshuffled and spill locally.

Validation:

• .agents/skills/mo-dev/scripts/mo-cgo-test -count=1 -timeout=180s ./pkg/sql/plan
• go build -mod=readonly ./pkg/sql/plan
• go vet -mod=readonly ./pkg/sql/plan
• go build -mod=readonly ./pkg/container/hashtable
• go vet -mod=readonly ./pkg/container/hashtable
• go test -count=1 -timeout=120s ./pkg/container/hashtable
• ../mo-tester/run.sh -n -g -p test/distributed/cases/operator/in_range_operator.sql (368/368 passed)

Design decision: this selects the already-spillable normal DEDUP implementation for memory-unsafe plans instead of attempting to switch RIGHT DEDUP after it has streamed rows downstream. RIGHT DEDUP remains a small-input optimization; absent/default statistics take the safe fallback. FLOAT shuffle support is intentionally deferred to a separate PR.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[XuPeng-SH]

Deep review completed against f860ce2.

Verified the full closure of the change:

• RIGHT DEDUP memory accounting matches the executor hash-map type selection, load factors, growth steps, and chained-map coexistence. Unknown/non-finite/overflow estimates conservatively fall back to normal DEDUP.
• The fallback runs before child swapping and is followed by stats/shuffle recomputation, so it does not leave partially converted physical state.
• FLOAT DEDUP keys stay off the runtime hash-shuffle path, which does not support FLOAT.
• Broadcast workers still merge shared matched/capture state; shuffle workers finalize and clean their disjoint partitions independently, including Reset/reuse paths.

Local verification (CGo-transitive wrapper): full and -race tests passed for ./pkg/container/hashtable, ./pkg/sql/colexec/dedupjoin, and ./pkg/sql/plan. git diff --check also passed. No blocking issue found.

[XuPeng-SH]

Follow-up: I also completed real SQL execution against the exact reviewed head f860ce293fb80f0b25d8b95b4a76d3300e23d905, using an isolated locally built mo-service and data generated by generate_series.

Results:

• Chained PK + UK, 4,000 target rows + 10,000 input rows, join_spill_mem=50000: verbose/physical plans showed two chained RIGHT DEDUP / right dedup join operators. Execution produced 14,000 rows, 14,000 distinct PKs, and 14,000 distinct UKs.
• Same shape with join_spill_mem=25000: the whole chain fell back to two normal DEDUP operators. Execution produced the same 14,000/14,000/14,000 result.
• BIGINT PK, 4,000 target rows + 400,000 input rows, forced fallback: physical plan used 10 hash-shuffle partitions and 10 independent dedup workers. Final result was 404,000 rows and 404,000 distinct PKs, confirming no partition-finalize output was lost.
• DOUBLE PK with the same 400,000-row input: plan remained non-shuffled, executed without panic, and produced 404,000 rows / 404,000 distinct PKs.
• Mixed input containing one existing PK: both RIGHT DEDUP and fallback DEDUP returned the expected duplicate-key error; both target tables remained at 4,000 rows with zero partial new rows.

One observed conservative behavior: small or not-yet-flushed target/index tables can still present DefaultStats, so the guard intentionally falls back to normal DEDUP until non-default table stats are available. This is a performance choice, not a correctness issue. No blocker found in the real execution pass.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-07-21 16:54 UTC · Rule: main · triggered by rule Automatic queue on approval for main
• ✅ Checks passed · in-place
• ✅ Merged — 2026-07-21 20:16 UTC · at ce8c250df1e756d78f49516b76f759e5bcbf7cb7 · squash

This pull request spent 3 hours 22 minutes 22 seconds in the queue, including 1 hour 5 minutes 1 second running CI.

Required conditions to merge

• #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • fix(plan): bound right dedup memory #25936
• github-review-approved [🛡 GitHub branch protection] (documentation)
  • fix(plan): bound right dedup memory #25936
• github-review-decision = APPROVED [🛡 GitHub branch protection] (documentation)
  • fix(plan): bound right dedup memory #25936
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-neutral = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-skipped = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / UT Test on Ubuntu/x86
  • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
  • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Utils CI / Coverage
  • check-neutral = Matrixone Utils CI / Coverage
  • check-skipped = Matrixone Utils CI / Coverage
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / SCA Test on Linux/arm64
  • check-neutral = Matrixone CI / SCA Test on Linux/arm64
  • check-skipped = Matrixone CI / SCA Test on Linux/arm64

[XuPeng-SH]

Follow-up byte-threshold validation on exact head f860ce2 (correcting the earlier row-threshold-only experiment):

• Built real data with generate_series: target = 4,000 rows, source = 10,000 rows, with BIGINT PK plus VARCHAR UNIQUE key.
• Flushed target, hidden unique-index table, and source, then ANALYZE. EXPLAIN VERBOSE reported exact outcnt 4,000 / 4,000 / 10,000.
• For 14,000 resident keys per map, the runtime-aligned estimators give 512 KiB (int map) + 1 MiB (string map) = 1,572,864 bytes combined.
• join_spill_mem = 1,572,864 produced two RIGHT DEDUP joins.
• join_spill_mem = 1,572,863 (one byte lower) produced two normal DEDUP joins.

This confirms the planner uses the combined resident-map byte estimate and switches the whole chained dedup stage at the exact strict-greater-than memory boundary. I also reran the estimator, combined-map planner, ShouldSpill, and hashbuild memory-threshold spill tests; all passed.

Scope note: this validates the OOM-prevention planning decision without intentionally exhausting machine memory; it is not a literal OS-OOM reproduction.