sql/tests: TestRandomSyntaxGeneration failed [issue with domain-of-array]

[cockroach-teamcity]

sql/tests.TestRandomSyntaxGeneration failed with artifacts on master @ f1b136a030d439e5dc89a27cb4de08045dca8e77:

    rsg_test.go:992: 3m35s of 5m0s: 223011 executions, 11330 successful
    rsg_test.go:992: 3m40s of 5m0s: 227031 executions, 11497 successful
    rsg_test.go:992: 3m45s of 5m0s: 232169 executions, 11690 successful
    rsg_test.go:992: 3m50s of 5m0s: 237518 executions, 11878 successful
    rsg_test.go:992: 3m55s of 5m0s: 245505 executions, 12188 successful
    rsg_test.go:992: 4m0s of 5m0s: 250320 executions, 12392 successful
    rsg_test.go:992: 4m5s of 5m0s: 252437 executions, 12462 successful
    rsg_test.go:992: 4m10s of 5m0s: 253572 executions, 12510 successful
    rsg_test.go:992: 4m15s of 5m0s: 255879 executions, 12582 successful
    rsg_test.go:992: 4m20s of 5m0s: 258313 executions, 12680 successful
panic: /Table/3/1/138/2/1: failed to unmarshal descriptor {[209 240 220 87 3 26 144 1 8 104 16 133 1 26 7 95 102 97 109 105 108 121 32 138 1 40 1 58 57 8 15 16 0 24 0 42 0 80 169 142 6 90 41 8 15 16 0 24 0 42 0 80 169 142 6 90 19 8 6 16 0 24 0 42 0 80 162 9 96 0 106 4 8 0 16 0 96 0 122 4 16 170 142 6 96 0 64 0 72 1 82 0 96 137 1 104 0 114 48 10 11 10 5 97 100 109 105 110 16 2 24 2 10 13 10 6 112 117 98 108 105 99 16 128 4 24 0 10 10 10 4 114 111 111 116 16 2 24 2 18 4 114 111 111 116 24 3 122 0 160 1 0] 1775366109.921448016,0}
	(1) attached stack trace
	  -- stack trace:
	  | runtime.gopanic
	  | 	GOROOT/src/runtime/panic.go:783
	  | [...repeated from below...]
	Wraps: (2) attached stack trace
	  -- stack trace:
	  | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic
	  | 	pkg/util/log/logcrash/crash_reporting.go:432
	  | github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqlwatcher.(*SQLWatcher).watchForDescriptorUpdates.func1
	  | 	pkg/spanconfig/spanconfigsqlwatcher/sqlwatcher.go:215
	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvent
	  | 	pkg/kv/kvclient/rangefeed/rangefeed.go:453
	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents
	  | 	pkg/kv/kvclient/rangefeed/rangefeed.go:439
	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func3
	  | 	pkg/kv/kvclient/rangefeed/rangefeed.go:393
	  | github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1
	  | 	pkg/util/ctxgroup/ctxgroup.go:200
	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
	  | runtime.goexit
	  | 	src/runtime/asm_amd64.s:1693
	Wraps: (3) /Table/3/1/138/2/1: failed to unmarshal descriptor {[209 240 220 87 3 26 144 1 8 104 16 133 1 26 7 95 102 97 109 105 108 121 32 138 1 40 1 58 57 8 15 16 0 24 0 42 0 80 169 142 6 90 41 8 15 16 0 24 0 42 0 80 169 142 6 90 19 8 6 16 0 24 0 42 0 80 162 9 96 0 106 4 8 0 16 0 96 0 122 4 16 170 142 6 96 0 64 0 72 1 82 0 96 137 1 104 0 114 48 10 11 10 5 97 100 109 105 110 16 2 24 2 10 13 10 6 112 117 98 108 105 99 16 128 4 24 0 10 10 10 4 114 111 111 116 16 2 24 2 18 4 114 111 111 116 24 3 122 0 160 1 0] 1775366109.921448016,0}
	Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *errutil.leafError

goroutine 2040 [running, "job":"AUTO SPAN CONFIG RECONCILIATION id=1164232118191226881", "n":"1"]:
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0115fe380?, {0x87fb698?, 0xc0115eb1d0?}, 0xc0115e29f0?})
	pkg/util/ctxgroup/ctxgroup.go:151 +0xbb
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x87fb6d0?, 0xc0115ed800?}, {0xc011375cb0, 0x2, 0x6bf50c0?})
	pkg/util/ctxgroup/ctxgroup.go:224 +0x11d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0114f4000, {0x87fb6d0, 0xc0115ed800}, {0x886ab60, 0xc01149d1d0}, 0x0)
	pkg/kv/kvclient/rangefeed/rangefeed.go:396 +0xabd
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func1({0x87fb698?, 0xc01149d220?})
	pkg/kv/kvclient/rangefeed/rangefeed.go:256 +0x2e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x87fb698, 0xc01149d220})
	pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 1425
	pkg/util/stop/stopper.go:445 +0x125

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• sql/tests: TestRandomSyntaxGeneration failed [CREATE TENANT statement is not valid after parsing then formatting] #132558 sql/tests: TestRandomSyntaxGeneration failed [CREATE TENANT statement is not valid after parsing then formatting] [C-test-failure O-robot P-3 T-db-server branch-release-24.1.6-rc]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-62525

Epic CRDB-62146

[cockroach-teamcity]

sql/tests.TestRandomSyntaxGeneration failed with artifacts on master @ 7e489320238b6feb3fcf94afa5727b7dff7cf51f:

    rsg_test.go:992: 1m20s of 5m0s: 77877 executions, 5312 successful
    rsg_test.go:992: 1m25s of 5m0s: 80574 executions, 5449 successful
    rsg_test.go:992: 1m30s of 5m0s: 86860 executions, 5770 successful
    rsg_test.go:992: 1m35s of 5m0s: 94066 executions, 6074 successful
    rsg_test.go:992: 1m40s of 5m0s: 95016 executions, 6122 successful
    rsg_test.go:992: 1m45s of 5m0s: 98841 executions, 6298 successful
    rsg_test.go:992: 1m50s of 5m0s: 102754 executions, 6470 successful
    rsg_test.go:992: 1m55s of 5m0s: 104664 executions, 6557 successful
    rsg_test.go:992: 2m0s of 5m0s: 105989 executions, 6618 successful
    rsg_test.go:992: 2m5s of 5m0s: 107241 executions, 6687 successful
panic: /Tenant/10/Table/3/1/124/2/1: unable to unmarshal descriptor {[232 43 9 96 3 26 152 1 8 104 16 29 26 24 95 110 111 116 104 105 110 103 95 97 102 116 101 114 95 114 101 116 117 114 110 105 110 103 32 124 40 1 58 51 8 15 16 0 24 0 42 0 80 155 142 6 90 35 8 15 16 0 24 0 42 0 80 155 142 6 90 13 8 17 16 0 24 0 42 0 80 187 8 96 0 96 0 122 4 16 156 142 6 96 0 64 0 72 1 82 0 96 123 104 0 114 48 10 11 10 5 97 100 109 105 110 16 2 24 2 10 13 10 6 112 117 98 108 105 99 16 128 4 24 0 10 10 10 4 114 111 111 116 16 2 24 2 18 4 114 111 111 116 24 3 122 0 160 1 0] 1775538825.020570980,0}
	(1) attached stack trace
	  -- stack trace:
	  | runtime.gopanic
	  | 	GOROOT/src/runtime/panic.go:783
	  | [...repeated from below...]
	Wraps: (2) attached stack trace
	  -- stack trace:
	  | github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic
	  | 	pkg/util/log/logcrash/crash_reporting.go:432
	  | github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).watchForUpdates.func1
	  | 	pkg/sql/catalog/lease/lease.go:3090
	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvent
	  | 	pkg/kv/kvclient/rangefeed/rangefeed.go:453
	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents
	  | 	pkg/kv/kvclient/rangefeed/rangefeed.go:439
	  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func3
	  | 	pkg/kv/kvclient/rangefeed/rangefeed.go:393
	  | github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1
	  | 	pkg/util/ctxgroup/ctxgroup.go:200
	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
	  | runtime.goexit
	  | 	src/runtime/asm_amd64.s:1693
	Wraps: (3) /Tenant/10/Table/3/1/124/2/1: unable to unmarshal descriptor {[232 43 9 96 3 26 152 1 8 104 16 29 26 24 95 110 111 116 104 105 110 103 95 97 102 116 101 114 95 114 101 116 117 114 110 105 110 103 32 124 40 1 58 51 8 15 16 0 24 0 42 0 80 155 142 6 90 35 8 15 16 0 24 0 42 0 80 155 142 6 90 13 8 17 16 0 24 0 42 0 80 187 8 96 0 96 0 122 4 16 156 142 6 96 0 64 0 72 1 82 0 96 123 104 0 114 48 10 11 10 5 97 100 109 105 110 16 2 24 2 10 13 10 6 112 117 98 108 105 99 16 128 4 24 0 10 10 10 4 114 111 111 116 16 2 24 2 18 4 114 111 111 116 24 3 122 0 160 1 0] 1775538825.020570980,0}
	Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *errutil.leafError

goroutine 3175 [running]:
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc001f466c0?, {0x8802cb8?, 0xc001e4a4b0?}, 0xc001c0c3f0?})
	pkg/util/ctxgroup/ctxgroup.go:151 +0xbb
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x8802cf0?, 0xc0017d6180?}, {0xc00c3b5cb0, 0x2, 0x6bfaf60?})
	pkg/util/ctxgroup/ctxgroup.go:224 +0x11d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc002336380, {0x8802cf0, 0xc0017d6180}, {0x8872220, 0xc001f51a90}, 0x0)
	pkg/kv/kvclient/rangefeed/rangefeed.go:396 +0xabd
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func1({0x8802cb8?, 0xc001f51ae0?})
	pkg/kv/kvclient/rangefeed/rangefeed.go:256 +0x2e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x8802cb8, 0xc001f51ae0})
	pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 23
	pkg/util/stop/stopper.go:445 +0x125

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• sql/tests: TestRandomSyntaxGeneration failed #167562 sql/tests: TestRandomSyntaxGeneration failed [C-test-failure O-robot T-sql-foundations branch-release-24.1 release-blocker]
• sql/tests: TestRandomSyntaxGeneration failed [CREATE TENANT statement is not valid after parsing then formatting] #132558 sql/tests: TestRandomSyntaxGeneration failed [CREATE TENANT statement is not valid after parsing then formatting] [C-test-failure O-robot P-3 T-db-server branch-release-24.1.6-rc]

This test on roachdash | Improve this report!

[fqazi]

/investigate

[fqazi]

I wonder on how we got such a corrupt descriptor that isn't even a valid protobuf 😱

[github-actions]

Investigation: TestRandomSyntaxGeneration

Investigated failure: most recent failure
Failure SHA: 7e489320238b6feb3fcf94afa5727b7dff7cf51f
Confidence: moderate

What This Test Does

TestRandomSyntaxGeneration generates random SQL statements from CockroachDB's grammar (sql.y) and executes them against a test server for 5 minutes. It verifies that randomly generated SQL does not cause panics or internal errors, even when the statements themselves are semantically invalid.

Where the Failure Occurs

Both occurrences panic in rangefeed event processing when attempting to deserialize a type descriptor from the system.descriptor table. The actual error from descbuilder.FromSerializedValue is not included in the panic message — both callers discard the error and substitute their own format string.

Failure 1 (issue body): Panics in spanconfigsqlwatcher.(*SQLWatcher).watchForDescriptorUpdates.func1 at sqlwatcher.go:215. The failing descriptor is at key /Table/3/1/138/2/1 — an ALIAS type descriptor named _family (ID 138, parent_id 104).

Failure 2 (comment): Panics in lease.(*Manager).watchForUpdates.func1 at lease.go:3090. The failing descriptor is at key /Tenant/10/Table/3/1/124/2/1 — an ALIAS type descriptor named _nothing_after_returning (ID 124, parent_id 104).

Both descriptors are type ALIAS descriptors (kind=1), which are implicit array type companions automatically created when a user-defined type (ENUM, COMPOSITE, or DOMAIN) is created.

Analysis

Hypothesis 1: Domain type descriptors cause deserialization failures (moderate confidence)

Commit bd888fd625e ("sql: add CREATE/DROP DOMAIN type support", 2026-03-08) added DOMAIN=5 to the TypeDescriptor.Kind enum and a new Domain sub-message at field 21. The CREATE DOMAIN syntax is present in sql.y at line 12846, so the random syntax generator can produce domain creation statements.

When a domain type is created, CreateUserDefinedArrayTypeDesc at create_type.go:247-252 creates an ALIAS companion using types.MakeDomain(). The MakeDomain function at types.go:1433-1446 copies the base type's InternalType and sets a UDT OID, then this gets wrapped in types.MakeArray() and stored as the ALIAS descriptor's alias field.

The deserialization path through types.T.Unmarshal() → upgradeType() at types.go:2621-2766 may encounter issues with domain-typed array elements. Specifically:

• Domain types share their base type's Family but use UDT OIDs, which may conflict with upgradeType's OID-setting logic for certain families
• The ArrayFamily case in upgradeType recursively calls upgradeType() on array contents, which could propagate errors from domain element type handling

Manual protobuf decoding of the bytes in both failures shows structurally valid TypeDescriptor data, suggesting the error comes from RunPostDeserializationChanges or types.T.upgradeType() rather than raw protobuf parsing.

Hypothesis 2: Non-domain type interaction with new proto fields (low confidence)

The DOMAIN commit also changed the TypeDescriptor proto's "Next field" comment from 19 to 22, and added field 21 (domain). While field 20 (replicated_pcr_version) was already present, the field numbering is correct and there should be no encoding conflict. However, if any proto code generation issue caused field 20 data to be misinterpreted, it could cascade.

Key limitation: The actual deserialization error is lost because both callers (sqlwatcher.go:215-221 and lease.go:3089-3091) call logcrash.ReportOrPanic with a format string that includes only the key and value, but NOT the error returned by FromSerializedValue. Fixing this logging gap would make future investigations of this failure much easier.

Recommendations:

1. Include the actual error in the ReportOrPanic calls in both sqlwatcher.go and lease.go
2. Add a unit test that creates a DOMAIN type, then verifies its companion ALIAS descriptor can be round-tripped through FromSerializedValue
3. Test types.T.Unmarshal with a serialized domain-typed array to ensure upgradeType() handles it correctly

Related Issues and PRs

• #167562 — Same test failure on release-24.1, but with a different failure mode (nil pointer dereference in gcjob.refreshTenant). This is a separate bug.
• #111482 — Closed (unactionable). Same "failed to unmarshal descriptor" error in spanconfig watcher on v23.1.10 from Sentry. Suggests this class of failure has occurred before, possibly due to descriptor table corruption or unexpected descriptor formats.
• #132558 — Different failure mode (CREATE TENANT parse/format round-trip), listed as related but not the same root cause.

Timeline

• 2026-03-08: Commit bd888fd625e adds CREATE DOMAIN support, adding DOMAIN=5 to TypeDescriptor.Kind and CREATE DOMAIN to the SQL grammar
• 2026-03-08: Commit b7ea2683075 adds domain constraint enforcement
• 2026-04-05: First failure occurrence at SHA f1b136a030d4 (issue body) — panics in spanconfig SQL watcher
• 2026-04-07: Second failure occurrence at SHA 7e489320238b (comment) — panics in lease manager's descriptor watcher

Both failures occurred after the DOMAIN commits were merged, and both involve ALIAS type descriptors (array companions for user-defined types).

Detailed evidence and investigation notes

Log Excerpts

Failure 2 (from test.log):

panic: /Tenant/10/Table/3/1/124/2/1: unable to unmarshal descriptor
  {[232 43 9 96 3 26 152 1 8 104 16 29 ...] 1775538825.020570980,0}

goroutine 3175 [running]:
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).watchForUpdates.func1
    pkg/sql/catalog/lease/lease.go:3090
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvent
    pkg/kv/kvclient/rangefeed/rangefeed.go:453

Decoded protobuf for failure 2: ALIAS type descriptor named _nothing_after_returning, ID 124 (verified: a legitimate array companion for a UDT called nothing_after_returning).

Code References

The error reporting gap in sqlwatcher.go:

// pkg/spanconfig/spanconfigsqlwatcher/sqlwatcher.go:213-221
b, err := descbuilder.FromSerializedValue(&value)
if err != nil {
    logcrash.ReportOrPanic(ctx, &s.settings.SV,
        "%s: failed to unmarshal descriptor %v",
        ev.Key, value)  // BUG: `err` is not included!
    return
}

The same gap in lease.go:

// pkg/sql/catalog/lease/lease.go:3088-3091
b, err := descbuilder.FromSerializedValue(&ev.Value)
if err != nil {
    logcrash.ReportOrPanic(ctx, &m.storage.settings.SV,
        "%s: unable to unmarshal descriptor %v", ev.Key, ev.Value)  // BUG: `err` is not included!
}

Things Ruled Out

• Proto field encoding conflicts: Fields 20 and 21 in TypeDescriptor are correctly assigned and don't conflict.
• Extended MVCC encoding issues: Both values use standard encoding (tag byte = 3, not 101).
• Raw protobuf corruption: Manual byte-level decoding shows structurally valid TypeDescriptor messages with correct field tags, lengths, and values.
• MustSetModificationTime failure: The decoded bytes show version=1 and empty modification_time, which is a valid combination that doesn't trigger assertion errors.
• Issue sql/tests: TestRandomSyntaxGeneration failed #167562 (release-24.1): Different failure mode entirely — nil pointer dereference in gcjob.refreshTenant, not a descriptor unmarshal issue.

Other Failure Occurrences

The two failures in this issue thread have the same failure mode (descriptor unmarshal panic in rangefeed watchers) but differ in which watcher panics:

• Failure 1: spanconfig SQL watcher (system tenant, goroutine for AUTO SPAN CONFIG RECONCILIATION)
• Failure 2: lease manager watcher (tenant 10, external process virtual cluster)

Both involve ALIAS type descriptors, suggesting a common root cause related to how certain UDT array companion descriptors are serialized/deserialized.

Tooling Feedback

• The most impactful missing capability was the ability to run a small Go program to attempt protoutil.Unmarshal on the descriptor bytes directly. This would have immediately revealed the exact deserialization error.
• The CRDB server logs (10MB+ each) were available but did not contain the unmarshal error because the panic bypasses the logging system. The actual error is discarded in the ReportOrPanic call, which is the primary obstacle to diagnosing this issue.

---

Investigation run: https://github.com/cockroachdb/cockroach/actions/runs/24087012188

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[fqazi]

Lets confirm the hypothesis that this is only linked domain supprt, and if it is we can move it into the relevant epic.

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.