release-25.4.10-rc: opt: add optimizer_span_limit and use in GenerateConstrainedScans

[michae2]

Backport 1/2 commits from #167180.

/cc @cockroachdb/release

---

opt: add optimizer_span_limit and use in GenerateConstrainedScans

This commit creates a new session variable optimizer_span_limit, and
uses it to bound the number of spans in constrained scans produced by
GenerateConstrainedScans.

Fixes: #167620

Release note (sql change): Add a new session variable
optimizer_span_limit which bounds the number of spans the optimizer
will allow in a single constrained index scan. If a single IN set has
more items than this limit, that IN set will not be used to build a
constrained index scan. If the cross product of two or more IN sets
would produce more spans than this limit for a composite index, then
only a prefix of the IN sets will be used to produce spans.

For example, for the following table and query, only the predicates on
columns a and b will be used to construct the constrained scan of
abc_idx, because including the predicate on column c would produce
more spans than optimizer_span_limit:

CREATE TABLE abc (a INT, b INT, c INT, INDEX abc_idx (a, b, c));
SET optimizer_span_limit = 10;
SELECT * FROM abc WHERE a IN (1, 3, 5) AND b IN (2, 4, 6) AND c IN (7, 9, 11);

Co-Authored-By: roachdev-claude roachdev-claude-bot@cockroachlabs.com

---

Release justification: disabled-by-default fix for customer workload. Fixes ENGREQ-424.

[blathers-crl]

Thanks for opening a backport.

Before merging, please confirm that it falls into one of the following categories (select one):

• Non-production code changes OR fixes for serious issues. Non-production includes test-only changes, build system changes, etc. Serious issues are defined in the policy as correctness, stability, or security issues, data corruption/loss, significant performance regressions, breaking working and widely used functionality, or an inability to detect and debug production issues.
• Other approved changes. These changes must be gated behind a disabled-by-default feature flag unless there is a strong justification not to. Reference the approved ENGREQ ticket in the PR body (e.g., "Fixes ENGREQ-123").

Add a brief release justification to the PR description explaining your selection.

Also, confirm that the change does not break backward compatibility and complies with all aspects of the backport policy.

All backports must be reviewed by the TL and EM for the owning area.

[cockroach-teamcity]

This change is 

[blathers-crl]

✅ PR #168892 is compliant with backport policy

Confidence: high
Feature flag detected: Yes
Backward compatible: true
Explanation: The pull request introduces a new session variable optimizer_span_limit that is disabled by default. This session variable limits the number of spans the optimizer will allow in a single constrained index scan, effectively acting as a feature flag for the new behavior introduced. Since it is disabled by default, it complies with the backport policy for feature flags. Additionally, the PR includes a 'Release justification: disabled-by-default fix for customer workload. Fixes ENGREQ-424.' which explicitly states the change is gated by a feature that is disabled by default and is intended to address a specific customer request (ENGREQ-424), further supporting the exemption from typical backport requirements.

✅ ENGREQ Check Passed: ENGREQ ticket referenced in PR body.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[DrewKimball]

@DrewKimball reviewed 20 files and all commit messages, and made 1 comment.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on mw5h).

[mw5h]

@mw5h made 1 comment.
Reviewable status: complete! 2 of 0 LGTMs obtained (waiting on michae2).

[michae2]

Thanks!