sql/importer: TestUniqueUUID failed

[github-actions]

sql/importer.TestUniqueUUID failed on master @ 7db484e7464473c1097707eb6a1daae0fbe93a37:

net.(*conn).Read(0x28bae02a0378, {0x28bae344e800?, 0x28bae344e800?, 0x5?})
	GOROOT/src/net/net.go:196 +0x45
github.com/cockroachdb/cmux.(*MuxConn).Read(0x28bb1ee76480, {0x28bae344e800, 0x800, 0x800})
	external/com_github_cockroachdb_cmux/cmux.go:241 +0x14f
crypto/tls.(*atLeastReader).Read(0x28bab628a240, {0x28bae344e800?, 0x28bab3aa6780?, 0x28babf5b5a70?})
	GOROOT/src/crypto/tls/conn.go:815 +0x3b
bytes.(*Buffer).ReadFrom(0x28bac1ad0d28, {0x7605640, 0x28bab628a240})
	GOROOT/src/bytes/buffer.go:229 +0x98
crypto/tls.(*Conn).readFromUntil(0x28bac1ad0a88, {0x7fbe74752160, 0x28bb1ee76480}, 0x28babf5b5c30?)
	GOROOT/src/crypto/tls/conn.go:837 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0x28bac1ad0a88, 0x0)
	GOROOT/src/crypto/tls/conn.go:626 +0x3db
crypto/tls.(*Conn).readRecord(...)
	GOROOT/src/crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0x28bac1ad0a88, {0x28bb1eeb7000, 0x1000, 0x4?})
	GOROOT/src/crypto/tls/conn.go:1393 +0x145
storj.io/drpc/drpcwire.(*Reader).read(0x28bb1ee9a150, {0x28bb1eeb7000, 0x1000, 0x1000})
	external/io_storj_drpc/drpcwire/reader.go:71 +0x71
storj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0x28bb1ee9a150, {0x28bac8215420?, 0xda?, 0xe0?})
	external/io_storj_drpc/drpcwire/reader.go:116 +0x4cb
storj.io/drpc/drpcmanager.(*Manager).manageReader(0x28bb1eca94a0)
	external/io_storj_drpc/drpcmanager/manager.go:236 +0xd5
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 142157
	external/io_storj_drpc/drpcmanager/manager.go:128 +0x44c

goroutine 40916 [sync.WaitGroup.Wait, 14 minutes, "rangefeed":"settings-watcher"]:
sync.runtime_SemacquireWaitGroup(0x28ba9da5e808?, 0xe8?)
	GOROOT/src/runtime/sema.go:114 +0x2e
sync.(*WaitGroup).Wait(0x28baa981d6c8)
	GOROOT/src/sync/waitgroup.go:206 +0x85
golang.org/x/sync/errgroup.(*Group).Wait(0x28baa981d6c0)
	external/org_golang_x_sync/errgroup/errgroup.go:56 +0x1e
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0x28baa981d6c0?, {0x765a8b8?, 0x28bab5cc6aa0?}, 0x28bab5cd22b8?})
	pkg/util/ctxgroup/ctxgroup.go:148 +0x5c
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x765a8f0?, 0x28bab2024ae0?}, {0x28bab6d9fcb0, 0x2, 0x65d0ec0?})
	pkg/util/ctxgroup/ctxgroup.go:224 +0x11d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0x28bab6bacc40, {0x765a8f0, 0x28bab2024ae0}, {0x76c43b8, 0x28bab6bafd60}, 0x0)
	pkg/kv/kvclient/rangefeed/rangefeed.go:396 +0xb3d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func1({0x765a8b8?, 0x28bab6bafdb0?})
	pkg/kv/kvclient/rangefeed/rangefeed.go:256 +0x2e5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x765a8b8, 0x28bab6bafdb0})
	pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 27914
	pkg/util/stop/stopper.go:445 +0x125

goroutine 133211 [chan receive, 2 minutes]:
storj.io/drpc/drpcpool.(*poolConn[...]).monitorStream(0x76ce3c0, {0x769bf68, 0x28bb09626c88?}, {0x7690380?, 0x28badb977700}, 0x28bacab2d120)
	external/io_storj_drpc/drpcpool/conn.go:108 +0x4a
created by storj.io/drpc/drpcpool.(*poolConn[...]).NewStream in goroutine 133193
	external/io_storj_drpc/drpcpool/conn.go:102 +0x2e5

Parameters:

• attempt=1
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-64450

[spilchen]

Triager Pipeline Status

Field	Value
Stage	review-loop
Run ID	2026-06-02-070
Updated	2026-06-02T12:03:48Z
Confidence (flake)	—
Confidence (fix)	—
Result	in-progress

[spilchen]

/investigate

[github-actions]

Investigation: sql/importer.TestUniqueUUID

Investigated failure: issue body
Failure SHA: 7db484e7464473c1097707eb6a1daae0fbe93a37
Confidence: high

What This Test Does

TestUniqueUUID creates a 3-node test cluster and imports 50,000 rows (500 × 100) three times into tables r1, r2, and r3 to verify that UUID/random default values don't produce collisions during IMPORT INTO.

Where the Failure Occurs

The test times out after 15 minutes at import_stmt_test.go:4456, which is the third IMPORT INTO r3 (b) CSV DATA (...) statement. The test goroutine is blocked in pq.(*conn).recvMessage waiting for the SQL response.

The underlying IMPORT job (ID 1180565822566137857) is itself blocked at import_job.go:509 in WaitForJobsIgnoringJobErrors, waiting for an INSPECT validation job (1180565838521663489) to complete.

Analysis

Likely cause: Metamorphic sync INSPECT validation causes timeout (high confidence)

The IMPORT code path includes a metamorphic constant (import_job.go:170-174) that randomly selects between async (non-blocking) and sync (blocking) INSPECT validation after each import. When sync is selected, the IMPORT statement blocks until the INSPECT job completes.

From the node logs, each INSPECT job takes 4-6 minutes:

Operation	Start	End	Duration
Import r1 + INSPECT	21:38:06	21:42:21	~4m 15s
Import r2 + INSPECT	21:42:22	21:48:15	~5m 53s
Import r3 + INSPECT	21:48:17	timeout 21:50:23	>2m (killed)
Total	21:35:48 (cluster start)	21:50:23	14m 35s > 15m limit

The cumulative overhead of three synchronous INSPECT jobs (~14 minutes of INSPECT time) pushes the test past the 15-minute timeout.

This is the same class of bug fixed in commit 7c5a50047a3 for TestImportCSVStmt — that commit added SET CLUSTER SETTING bulkio.import.row_count_validation.mode = 'off' to prevent the same timeout pattern. However, TestUniqueUUID was not included in that fix.

Recommended fix: Add sqlDB.Exec(t, "SET CLUSTER SETTING bulkio.import.row_count_validation.mode = 'off'") to TestUniqueUUID after the existing elastic CPU control disable (line 4439), matching the pattern used by TestImportCSVStmt.

Related Issues and PRs

• sql/importer: TestUniqueUUID failed #171332 (this issue): Current failure
• sql/importer: TestUniqueUUID failed #166130 (closed): Prior TestUniqueUUID timeout, fixed by PR #166303 which disabled elastic CPU control — a different root cause
• sql/importer: TestUniqueUUID failed #122587 (closed): Earlier TestUniqueUUID timeout from 2024, fixed by #122561
• Commit 7c5a50047a3: Fixed the identical INSPECT timeout in TestImportCSVStmt but did not cover TestUniqueUUID
• Commit e56c0b95fdf: Introduced the metamorphic sync/async INSPECT validation setting (2025-10-07)

Timeline

• 2025-10-07: Metamorphic INSPECT validation introduced (e56c0b95fdf)
• 2026-03-19: Prior TestUniqueUUID timeout (sql/importer: TestUniqueUUID failed #166130), fixed by disabling elastic CPU control
• 2026-05-19: TestImportCSVStmt INSPECT timeout fixed (7c5a50047a3), TestUniqueUUID not covered
• 2026-06-01: This failure — TestUniqueUUID hits the same INSPECT timeout pattern

Detailed evidence and investigation notes

Log Excerpts

Test timeout panic (test.log:98-100):

panic: test timed out after 15m0s
	running tests:
		TestUniqueUUID (14m35s)

IMPORT job blocked on INSPECT (test.log:30372-30381):

goroutine 132815 [select, "job":"IMPORT id=1180565822566137857", "n":"1"]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0x28baa990e770)
	pkg/util/retry/retry.go:211 +0x3dd
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).waitForJobsToBeTerminalOrPaused(...)
	pkg/jobs/wait.go:102 +0x1aa
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).WaitForJobsIgnoringJobErrors(...)
	pkg/jobs/wait.go:68 +0x170
github.com/cockroachdb/cockroach/pkg/sql/importer.(*importResumer).Resume.func7()
	pkg/sql/importer/import_job.go:509 +0x125

Third INSPECT job started but never completed (importertest.log:4768-4774):

I260601 21:48:22.408601 132815 sql/inspect/inspect_job_entry.go:72 ⋮ [...] created and started inspect job 1180565838521663489
I260601 21:48:22.408629 132815 jobs/wait.go:65 ⋮ [...] waiting for 1 [1180565838521663489] queued jobs to complete
I260601 21:48:22.733290 133758 sql/inspect/inspect_resumer.go:39 ⋮ [...] starting INSPECT job
I260601 21:48:23.105456 133758 sql/inspect/inspect_resumer.go:466 ⋮ [...] protected timestamp created for INSPECT job

Metamorphic constant definition (import_job.go:170-174):

var importRowCountValidationMetamorphicValue = metamorphic.ConstantWithTestChoice(
	"import-row-count-validation",
	"async", // background validation (default)
	"sync",  // blocking validation for tests
)

Code References

• Test code: import_stmt_test.go:4417-4457
• Metamorphic setting: import_job.go:170-174
• Sync validation wait: import_job.go:505-509
• Prior fix for TestImportCSVStmt: import_stmt_test.go:1944

Things Ruled Out

• Elastic CPU throttling: Already disabled via the fix from sql/importer: TestUniqueUUID failed #166130 (PR importer: reduce TestImportData runtime to fix timeout #166303). The cluster setting bulkio.import.elastic_control.enabled = false is present in the test at line 4439.
• DRPC-related regression: While DRPC is enabled ("DRPC is enabled" appears in the test log), the goroutine dump shows no DRPC-specific hangs. The DRPC goroutines in the stack trace are normal background connection management, not blocking the import.
• Infrastructure/CI slowness: While CI slowness could contribute, the root cause is structural — three synchronous INSPECT jobs on 50K-row tables simply take too long regardless of machine speed.
• Data size increase: The data size (50K rows) hasn't changed; it's the addition of INSPECT overhead that makes the total exceed the time budget.

Other Failure Occurrences

The prior failures (#166130, #122587) were both timeouts but with different root causes:

• sql/importer: TestUniqueUUID failed #122587 (2024): Pure timeout, no INSPECT feature existed yet
• sql/importer: TestUniqueUUID failed #166130 (2026-03): Elastic CPU throttling was the root cause
• sql/importer: TestUniqueUUID failed #171332 (this issue): Synchronous INSPECT validation overhead

Tooling Feedback

No issues encountered with available tools. The EngFlow artifact download script worked well.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[spilchen]

Issue Classification

Field	Value
Classification	test-flake
Confidence	9/10
Scope	test-infrastructure

Reasoning: The test times out because a metamorphic constant randomly selects synchronous INSPECT validation, adding ~4-6 minutes per import across three imports, exceeding the 15-minute test timeout. The production code works correctly; the fix is test-only — adding SET CLUSTER SETTING bulkio.import.row_count_validation.mode = 'off' to TestUniqueUUID, matching the identical fix already applied to TestImportCSVStmt in commit 7c5a50047a3. This is a missed test when that prior fix was applied.

[spilchen]

Fix Assessment

Field	Value
Confidence	9/10
Safe to auto-apply	yes
Scope	test-infrastructure
Decision	advance

Reasoning: The fix is a one-line cluster setting addition (bulkio.import.row_count_validation.mode = 'off') to test setup code, exactly matching the pattern already applied to TestImportCSVStmt at line 1944 of the same file. The change is purely test-infrastructure — it disables metamorphic INSPECT validation overhead that causes timeout, without touching any production code paths. The root cause analysis is thorough, with clear log evidence showing three synchronous INSPECT jobs consuming ~14 minutes of the 15-minute budget.