sql/tests: TestRandomSyntaxSQLSmith failed

[cockroach-teamcity]

sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on release-25.2.16-rc @ b408a4a7555d83a2fbffdf90b9b20956bee4556f:

github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt(0xc011fc2a00, {0x85055a0, 0xc00f988690}, {0x0, 0x0, 0x0}, 0xc018e05688)
	pkg/sql/opt/optbuilder/builder.go:383 +0x458
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRootWithScope(0xc011fc2a00, {0x85055a0, 0xc00f988690}, {0x0, 0x0, 0x0}, 0xc018e05688)
	pkg/sql/opt/optbuilder/builder.go:321 +0x1d0
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot(0xc011fc2a00?, {0x85055a0?, 0xc00f988690?}, {0x0?, 0x0?, 0x0?})
	pkg/sql/opt/optbuilder/builder.go:302 +0x114
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build(0xc011fc2a00)
	pkg/sql/opt/optbuilder/builder.go:281 +0x2c8
github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo(0xc00b4a1588, {0x84dee90, 0xc0242f9680})
	pkg/sql/plan_opt.go:846 +0x253
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan(0xc00b4a06f8, {0x84dee90?, 0xc0242f95e0?})
	pkg/sql/plan_opt.go:260 +0x189
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan(0xc00b4a0008, {0x84dee90, 0xc0242f95e0}, 0xc00b4a06f8)
	pkg/sql/conn_executor_exec.go:3364 +0xcb
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc00b4a0008, {0x84dee90, 0xc0242f95e0}, 0xc00b4a06f8, {0x76d5a0731510, 0xc00b2d6b60})
	pkg/sql/conn_executor_exec.go:2911 +0x905
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenStateWithPausablePortal(0xc00b4a0008, {0x84dd878, 0xc00f988730}, {{0x85055a0, 0xc00f988690}, {0x0, 0x0, 0x0}, {0xc00e215300, 0x1823}, ...}, ...)
	pkg/sql/conn_executor_exec.go:2142 +0x4ae5
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x84dd878?, 0xc00f988730?})
	pkg/sql/conn_executor_exec.go:165 +0xb0
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x84dd878?, {0x84dd878?, 0xc00f988730?}, {0x85055a0?, 0xc00f988690?}, 0x76d587cddeb8?, 0x480?)
	pkg/sql/conn_executor_exec.go:4553 +0x2ec
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc00b4a0008, {0x84dd878, 0xc00f988730}, {{0x85055a0, 0xc00f988690}, {0x0, 0x0, 0x0}, {0xc00e215300, 0x1823}, ...}, ...)
	pkg/sql/conn_executor_exec.go:164 +0x79c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1({{{0x85055a0, 0xc00f988690}, {0x0, 0x0, 0x0}, {0xc00e215300, 0x1823}, 0x0, 0x1}, 0x1e80a192103, ...}, ...)
	pkg/sql/conn_executor.go:2465 +0x3dd
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc00b4a0008)
	pkg/sql/conn_executor.go:2470 +0x888
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc00b4a0008, {0x84dee90, 0xc011428640}, 0xc0235940f0?, 0x0?, 0xc001e75210?)
	pkg/sql/conn_executor.go:2378 +0x208
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc02c971a40?, {0x84dee90?, 0xc011428640?}, {0xc01b16a7e0?}, 0x2?, 0x0?)
	pkg/sql/conn_executor.go:1038 +0xb9
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands(0xc00b2d6408, {0x84dee90, 0xc011428640}, {0x0, 0x4, {0x1, {0x7520ad0, 0x3}, {0xc00ebdf980, 0xf}, ...}, ...}, ...)
	pkg/sql/pgwire/conn.go:252 +0x4a8
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func4()
	pkg/sql/pgwire/server.go:1201 +0xec
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl in goroutine 8751043
	pkg/sql/pgwire/server.go:1198 +0x5d5

goroutine 7875607 [select]:
github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
	pkg/util/admission/work_queue.go:422 +0x85
created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 7874877
	pkg/util/admission/work_queue.go:419 +0x3be

goroutine 7875611 [select]:
github.com/cockroachdb/cockroach/pkg/util/admission.makeStoreWorkQueue.func1()
	pkg/util/admission/work_queue.go:2313 +0x85
created by github.com/cockroachdb/cockroach/pkg/util/admission.makeStoreWorkQueue in goroutine 7874877
	pkg/util/admission/work_queue.go:2310 +0x47d

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• sql/tests: TestRandomSyntaxSQLSmith failed #166976 sql/tests: TestRandomSyntaxSQLSmith failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-25.2]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-62527

[spilchen]

/investigate

[github-actions]

Investigation: TestRandomSyntaxSQLSmith

Investigated failure: issue body
Failure SHA: b408a4a7555d83a2fbffdf90b9b20956bee4556f
Confidence: high

What This Test Does

TestRandomSyntaxSQLSmith uses SQLSmith to generate random SQL queries and execute them against a single-node CockroachDB server for 5 minutes. It detects "crashers" (internal errors, panics, or statement execution timeouts) and reports them as test failures.

Where the Failure Occurs

A randomly generated query calls st_clipbybox2d(geometry, st_expand('BOX(-10 -10,10 10)'::BOX2D, -0.737..., 'NaN'::FLOAT8)::BOX2D). The st_expand with a NaN delta_y parameter produces a BOX2D with NaN coordinates. During optimizer constant folding (FoldFunction), st_clipbybox2d is evaluated at planning time (it's marked Immutable), and the GEOS C library function CR_GEOS_ClipByRect hangs indefinitely when given NaN bounding box coordinates.

The stuck goroutine at geos.go:655:

goroutine 8751062 [syscall, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/geo/geos._Cfunc_CR_GEOS_ClipByRect(
    0x76d59c65e000, {0xc020dded00, 0x5d4},
    0xc02286a58891be3c,   // LoX
    0x7ff8000000000001,   // LoY: NaN
    0x402286a58891be3c,   // HiX
    0x7ff8000000000001,   // HiY: NaN
    0xc009f515c0)

The cascade: CGO call hangs → 35s statement exec timeout → test can't reconnect (rsg_test.go:750) → server can't shut down (pgwire blocked waiting for stuck goroutine) → 1h test timeout.

Analysis

Hypothesis: GEOS ClipByRect hangs when passed NaN bounding box coordinates (high confidence).

The ClipByRect function at topology_operations.go:65-74 performs no validation on the bounding box coordinates before passing them to the GEOS C library. When SQLSmith generates a query with NaN coordinates in a BOX2D (via st_expand with NaN delta), the GEOS library enters an infinite loop or extremely long computation.

This is a confirmed duplicate of the failure in #166976. The prior investigation found the exact same root cause — identical function (CR_GEOS_ClipByRect), identical NaN pattern (0x7ff8000000000001), identical failure cascade. The only difference is the branch: #166976 was on release-25.2, this one is on release-25.2.16-rc.

Potential fixes:

1. Add NaN/Inf validation in ClipByRect before calling into CGO — return an error if any coordinate is NaN or Inf.
2. More broadly, add NaN/Inf validation at the BOX2D level so that no GEOS function receives pathological inputs.

Related Issues and PRs

• #166976 — Same failure, same root cause on release-25.2. The prior investigation (with 👍 from maintainer) identified this exact bug. A comment suggested "Consider hypothesis 1 as a fix" (NaN validation in ClipByRect). No fix has been merged yet.
• #139845 — geo: VoronoiDiagram doesn't respect context cancellation. Same class of bug — GEOS CGO calls that hang without respecting Go context cancellation. Still open.
• #159671 — sql: missing cancellation checks in some geo functions. Closed. Added cancellation checks to some geo functions, but st_clipbybox2d wasn't covered.

Timeline

• This is the same bug as sql/tests: TestRandomSyntaxSQLSmith failed #166976, which was filed on 2026-04-02 on release-25.2.
• This occurrence is on release-25.2.16-rc (SHA b408a4a755), dated 2026-04-05.
• No fix has been merged between these two occurrences.
• The underlying vulnerability (GEOS functions not validating NaN inputs) has existed since st_clipbybox2d was first implemented.

Detailed evidence and investigation notes

Log Excerpts

Stuck GEOS CGO call (from goroutine dump in test.log, line 675171):

goroutine 8751062 [syscall, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/geo/geos._Cfunc_CR_GEOS_ClipByRect(...)
    _cgo_gotypes.go:168 +0x4d
github.com/cockroachdb/cockroach/pkg/geo/geos.ClipByRect(...)
    pkg/geo/geos/geos.go:655 +0xcd
github.com/cockroachdb/cockroach/pkg/geo/geomfn.ClipByRect(...)
    pkg/geo/geomfn/topology_operations.go:69 +0x97
github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func585(...)
    pkg/sql/sem/builtins/geo_builtins.go:4293 +0x132
...
github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*CustomFuncs).FoldFunction(...)
    pkg/sql/opt/norm/fold_constants_funcs.go:693 +0x428

The triggering query (from test.log, line 666520):

st_clipbybox2d(
    '...':::GEOMETRY,
    st_expand('BOX(-10 -10,10 10)':::BOX2D, (-0.7370183298197691):::FLOAT8, 'NaN':::FLOAT8)::BOX2D
)

Test failure sequence:

rsg_test.go:298: SELECT 1 executed successfully: probably a slow statement
rsg_test.go:245: Context cancelled while executing: "USE defaultdb"
rsg_test.go:750: couldn't reconnect to db after crasher: server panic: statement exec timeout
...
-- Test timed out at 2026-04-05 06:11:06 UTC --
panic: test timed out after 1h0m0s
    running tests:
        TestRandomSyntaxSQLSmith (29m53s)

Code References

No NaN validation in ClipByRect (topology_operations.go:65-74):

func ClipByRect(g geo.Geometry, b geo.CartesianBoundingBox) (geo.Geometry, error) {
    if g.Empty() {
        return g, nil
    }
    clipByRectEWKB, err := geos.ClipByRect(g.EWKB(), b.LoX, b.LoY, b.HiX, b.HiY)
    // No validation of LoX, LoY, HiX, HiY for NaN/Inf
    ...
}

st_expand allows NaN to propagate (geo_builtins.go:6265-6286):

Fn: func(_ context.Context, _ *eval.Context, args tree.Datums) (tree.Datum, error) {
    bbox := tree.MustBeDBox2D(args[0])
    deltaX := float64(tree.MustBeDFloat(args[1]))
    deltaY := float64(tree.MustBeDFloat(args[2]))
    bboxBuffered := bbox.Buffer(deltaX, deltaY)
    // NaN delta produces a BOX2D with NaN coordinates
    ...
}

Things Ruled Out

• Different root cause from sql/tests: TestRandomSyntaxSQLSmith failed #166976: The goroutine stacks are identical — same function, same NaN pattern. This is the same bug.
• Server crash or wedge: The SELECT 1 test succeeded, confirming the server was responsive. Only the specific GEOS CGO call was stuck.
• Test infrastructure issue: While the test's timeout cascade could be more graceful, the underlying problem is the GEOS hang.

Other Failure Occurrences

This issue (#167548) and #166976 are the same failure mode. The issue body's stack trace (starting at buildStmt) is from the optimizer call chain that eventually reaches FoldFunction → ClipByRect → stuck GEOS call. The goroutine dump in the full test.log confirms this.

Tooling Feedback

No tools were missing for this investigation.

---

Workflow run

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[spilchen]

Looks like a duplicate of #166976. Removing release-blocker label.