release-26.2: roachtest: schema_change_workload_failure failed

[cockroach-teamcity]

roachtest.schema_change_workload_failure failed with artifacts on release-26.2 @ 86ffd9f5725977297c6a936068131eccc8b289c0:

Failed with:

test backup-restore/small-ranges failed: (test_runner.go:2697).func1: task ``run-schemachange`` returned error: schema change workload failed: full command output in run_094412.186139110_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1 [owner=sql-foundations]
(monitor.go:267).Wait: monitor failure: error waiting for job 1174761151549243393 to finish: context canceled
test artifacts and logs in: /artifacts/backup-restore/small-ranges/cpu_arch=arm64/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21344617-1778567041-46-n4cpu4-0001	136.111.112.25	10.128.0.59
teamcity-21344617-1778567041-46-n4cpu4-0002	34.134.155.243	10.128.0.65
teamcity-21344617-1778567041-46-n4cpu4-0003	34.63.113.217	10.128.0.67
teamcity-21344617-1778567041-46-n4cpu4-0004	34.70.254.51	10.128.0.72

Parameters:

• arch=arm64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=false
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-63856

[spilchen]

/investigate

[github-actions]

Investigation: schema_change_workload_failure (backup-restore/small-ranges)

Investigated failure: #170170
Failure SHA: 86ffd9f5725977297c6a936068131eccc8b289c0
Confidence: high

What This Test Does

The backup-restore/small-ranges test runs backup and restore operations while concurrently running TPCC, bank, and schema change workloads. It uses small range sizes (4 MiB max) to stress the backup/restore path with many small ranges. The test creates backups, verifies them, and validates round-trip correctness.

Where the Failure Occurs

The schema change workload (schemachange) fails during operation generation when getTableColumns queries a table (schema_w2_3.table_w4_27) whose descriptor is being concurrently dropped by another worker. The query in operation_generator.go:3642 returns:

ERROR: descriptor is being dropped (SQLSTATE XXUUU)

This error propagates through selectStmt → randOp → runInTxn, where it is treated as a fatal unexpected error at schemachange.go:601-605:

return errors.Mark(
    w.WrapWithErrorState(
        errors.Wrap(err, "***UNEXPECTED ERROR; Failed to generate a random operation")),
    errRunInTxnFatalSentinel,
)

Analysis

Hypothesis: The schema change workload doesn't handle XXUUU (ObjectNotInPrerequisiteState) errors during operation generation — Confidence: high

The workload runs 5 concurrent schema change workers. When one worker drops a table while another worker is generating an operation that queries that table's columns, the second worker encounters descriptor is being dropped. In randOp (operation_generator.go:207-220), the only errors that trigger a retry of operation generation are:

1. pgx.ErrNoRows (when there's no PgError attached)
2. ErrSchemaChangesDisallowedDueToPkSwap

The XXUUU PgError from getTableColumns does not match either of these, so it falls through to line 220 as "failed generating operation: selectStmt", which is then treated as fatal in runInTxn.

The workload already handles pgcode.ObjectNotInPrerequisiteState as an expected error in many execution contexts (e.g., addColumn, dropColumn, alterType, createIndex, etc. — over 10 places in operation_generator.go), but it does not handle this error during operation generation in randOp.

Fix: In randOp, the error handling at lines 208-220 should also treat pgcode.ObjectNotInPrerequisiteState PgErrors as retriable (similar to pgx.ErrNoRows), allowing the generator to pick a different operation when it encounters a descriptor being dropped. Alternatively, getTableColumns could return pgx.ErrNoRows when the table's descriptor is being dropped, since a dropping table effectively has no usable columns.

Related Issues and PRs

• #170037 — Same schema_change_workload_failure on master, but with a different XXUUU error: RangeMaxBytes 33554432 less than minimum allowed 67108864. This is also an unhandled ObjectNotInPrerequisiteState error, but occurs during statement execution rather than generation. The backup-restore/small-ranges test's use of reduced kv.range_max_bytes (4 MiB) combined with the workload's ALTER DATABASE ADD REGION likely triggers a range size validation error. Both are manifestations of the same category of problem: insufficient handling of XXUUU errors in the schemachange workload.
• #168451 — Previous backup-restore/small-ranges failure (master), also with the schema_change_workload_failure title override from the same mechanism.
• #169693 — Recent cherry-pick that adds WithTitleOverride("schema_change_workload_failure") so these failures are grouped under a distinct issue title rather than polluting the backup-restore test's issue history. The failure SHA 86ffd9f is this exact commit.

Timeline

• The schema_change_workload_failure title override was added in PR release-26.2: roachtest: use title override for schema change workload failures #169693, cherry-picked to release-26.2 as commit 86ffd9f (the failure SHA). This means the failure mechanism has existed for some time but only recently started being tracked separately.
• Issue roachtest: schema_change_workload_failure failed #170037 (master) was filed on May 9, 2026, with an additional failure on May 11.
• Issue release-26.2: roachtest: schema_change_workload_failure failed #170170 (release-26.2) was filed on May 12, 2026.

Detailed evidence and investigation notes

Log Excerpts

Workload error from run_094412.186139110_n4_COCKROACHRANDOMSEED4.log:

E260512 09:46:52.397597 1 workload/cli/run.go:629  [-] 3  workload run error:
***UNEXPECTED ERROR; Failed to generate a random operation:
  failed generating operation: selectStmt:
    getting table columns from schema_w2_3.table_w4_27:
      ERROR: descriptor is being dropped (SQLSTATE XXUUU)

Stack trace (key frames):

github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).getTableColumns
    pkg/workload/schemachange/operation_generator.go:3644
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).selectStmt
    pkg/workload/schemachange/operation_generator.go:5000
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
    pkg/workload/schemachange/operation_generator.go:207
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
    pkg/workload/schemachange/schemachange.go:590

Code References

The error originates from ErrDescriptorDropped in pkg/sql/catalog/errors.go:62:

var ErrDescriptorDropped = errors.New("descriptor is being dropped")

The getTableColumns query at operation_generator.go:3637-3642 uses SHOW COLUMNS FROM which internally resolves the table descriptor and fails when the descriptor is in a dropping state.

In randOp at operation_generator.go:208-220, only pgx.ErrNoRows and ErrSchemaChangesDisallowedDueToPkSwap are retried:

if errors.Is(err, pgx.ErrNoRows) &&
    !errors.HasType(err, &pgconn.PgError{}) {
    continue
}
if errors.Is(err, ErrSchemaChangesDisallowedDueToPkSwap) {
    continue
}
return nil, errors.Wrapf(err, "failed generating operation: %s", op)

Things Ruled Out

• Backup/restore bug: The backup job (1174761151549243393) was still running when the schema change workload crashed. The backup failure is just a consequence of the context being canceled after the schema change workload error.
• Node crash or OOM: No fatal logs were found; all nodes were healthy.
• Infrastructure issue: The error is a clear application-level SQL error, not related to cloud infrastructure, disk, or network.

Other Failure Occurrences

Issue	Branch	Error	SQLSTATE
#170170	release-26.2	descriptor is being dropped	XXUUU
#170037 (1st)	master	RangeMaxBytes less than minimum allowed	XXUUU
#170037 (2nd)	master	RangeMaxBytes less than minimum allowed	XXUUU

All three failures share the same SQLSTATE (XXUUU / ObjectNotInPrerequisiteState) but have different underlying error messages. They all crash the schemachange workload because XXUUU errors are not handled as expected/retriable in the relevant code paths.

Tooling Feedback

No issues encountered with available tools. All artifacts were accessible and the investigation could be completed with the available tooling.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[spilchen]

I think we have seen this before that Brendan looked at.

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.

[bghal]

The type-specific errors in #169991 will cause a different error to trip this but need to handle that new error (similar to #169992).