feat(frontend): support schema evolution in data branch diff/merge by VioletQwQ-0 · Pull Request #25882 · matrixorigin/matrixone

VioletQwQ-0 · 2026-07-20T04:28:49Z

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

This PR continues the unfinished work from #24666 while preserving its original commits and authorship.

It supports schema evolution in DATA BRANCH DIFF/MERGE by matching columns with stable identity, separating physical and visible common-column indexes, projecting compatible historical batches, comparing common columns, preserving target-only data, and maintaining ALTER lineage ownership.

Review fixes on the latest head:

project reordered and target-only historical layouts without moving incompatible vectors;
reject same-name DROP/ADD column redefinitions by Seqnum, while preserving pure FIRST / AFTER reorderings;
retain ALTER snapshot/metadata pairs while table-, database-, or account-level historical owners still cover the component, including current-table-first deletion;
reclaim deleted intermediate ALTER generations after the final logical or historical owner disappears;
preserve explicit-transaction ALTER semantics for ordinary snapshot/PITR-covered tables.

Validation

go test ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile ./pkg/vm/engine/test -count=1
focused go test -race for identity mapping, projection, and lineage-owner lifecycle paths
go vet ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile ./pkg/vm/engine/test
go build ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile ./pkg/vm/engine/test
git diff upstream/main --check
full mo-pr-preflight-review and mo-self-review

The BVT fixtures are included for DIFF/MERGE identity redefinition and table/account/database historical-owner cleanup; they will be executed by CI because this local worktree has no standalone mo-tester runner.

Replace strict schema equivalence check with named-column compatibility check that allows compatible schema evolution (extra columns). Project base-side batches to target layout before hashmap encoding. Limit row comparison to common columns only. Fixes matrixorigin#24549

Fixes index out of range panic when base data batch has more columns than base table definition (e.g., composite PK columns). Clamp baseColCount to len(baseColToTarIdx) and validate tarColIdx against len(colNames) before accessing out.Vecs.

When target table has extra columns not present in base (schema evolution), INSERT SQL for MERGE must only reference common columns. Filter visibleNames in newPKBatchInfo and use commonIdxes in appendBatchRowsAsSQLValues/writeInsertRowValues when tarOnlyIdxes is non-empty. Also update diff_schema_evolve.result to expect 5 columns including the extra column c in DIFF output.

…olution - compareRowInWrappedBatches: use batchIdx for Vecs access, colIdx for pk check - baseColToTarIdx: build after colNames populated, filter hidden cols, map to colNames positions - data_branch_pick: add missing idxes parameter for writeInsertRowValues

The merge_schema_evolve.sql test creates snapshots sp0 and sp1 but never drops them. Since snapshot names are globally unique (checked by sname without level filtering), these leftover snapshots conflict with clone_reference.sql which creates account-level sp0, causing 6 BVT failures and dropping the pass rate below 100%. Add explicit drop snapshot sp1/sp0 cleanup matching the pattern in diff_schema_evolve.sql.

# Conflicts: # pkg/frontend/data_branch_output.go # pkg/frontend/data_branch_pick.go # pkg/frontend/data_branch_test.go

Expand diff_schema_evolve (8 cases) and merge_schema_evolve (5 cases) to cover: basic evolution, multiple extra columns, UPDATE on common col, UPDATE on target-only col (no diff), DELETE, composite PK, idempotent merge, type mismatch rejection, and PK removal rejection. Update branch_unhappy_path and branch_edge_cases to reflect the new schema evolution behavior: ALTER ADD COLUMN no longer errors, and type mismatch on common columns is the new rejection path.

The getTupleColumnValue function had an off-by-one bug for tuples with totalColCnt+1 elements. When projectBaseBatchToTarget creates a batch with [RowID, data...] (no commit_ts), the value tuple has totalColCnt+1 elements with RowID at index 0. The old code used tuple[colIdx] which read RowID bytes as data column values, causing 'unexpected byte slice for fixed-width column type INT' errors. Fix: use tuple[colIdx+1] for totalColCnt+1 tuples, matching the existing totalColCnt+2 case which already skips the leading RowID. Also add WHEN CONFLICT ACCEPT to merge_schema_evolve.sql Cases 2 and 4 which have UPDATE conflicts (same PK, different values on common columns). Without this clause, merge defaults to CONFLICT_FAIL.

VioletQwQ-0 · 2026-07-22T05:29:27Z

Addressed all blocking review feedback on the current review head and resolved the main conflicts in 08cd75df6.

Changes:

Reject same-name endpoint columns when their stable Seqnum identity differs, while preserving pure FIRST / AFTER reorderings whose identity is unchanged.
Enforce the same identity rule while mapping historical generations: common-column identity drift fails closed, while target-only drift is explicitly projected as NULL for endpoint hydration.
Preserve ALTER-owned branch snapshots while a table-, database-, or account-level historical source still owns the lineage component. This keeps the scope evidence available after current-table-first DROP TABLE and DATA BRANCH DELETE TABLE; the snapshot/metadata pair becomes reclaimable after the last owner is removed.
Merged the latest upstream/main without rebasing or rewriting the inherited history, preserving the new DATA BRANCH OUTPUT AS stable-identity and snapshot-range semantics.

Coverage:

Unit regressions for endpoint and historical Seqnum drift, pure reorder compatibility, target-only unmapping, account/database owner retention, owner removal, LCA identity mapping, and snapshot-range fallback.
DIFF and MERGE BVT regressions for UPDATE old b -> DROP b -> ADD b.
Current-table-first BVT coverage for table/account/database historical owners, including ordinary DROP TABLE and DATA BRANCH DELETE TABLE.

Validation on 08cd75df6:

go test ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile ./pkg/vm/engine/test -count=1
focused go test -race for identity mapping and historical-owner lifecycle paths
go vet ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile ./pkg/vm/engine/test
go build ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile ./pkg/vm/engine/test
git diff upstream/main --check
full mo-pr-preflight-review and mo-self-review: zero open merge blockers

The new BVT fixtures were not executed locally because this worktree has no standalone mo-tester runner; the new-head CI will validate them.

XuPeng-SH

Requesting changes on 08cd75d. One blocking historical-schema path remains.

[P1] Project incompatible target-only columns out of LCA probes — pkg/frontend/data_branch_hashdiff.go:114

The latest fix correctly maps an incompatible target-only historical column to -1 in dataBranchSourceColToTargetIdx, but lcaProbeColumnLayout independently selects every LCA column whose ColId and Seqnum match the target, without checking its logical type or targetOnlyIdxes. handleDelsOnLCA then appends that old LCA vector into the endpoint-typed dBat; appendLCAProbeValue rejects every non-ENUM type mismatch with unexpected LCA probe type conversion.

This is reachable when the fork point contains c INT, one descendant changes c to VARCHAR, and the sibling drops c. The endpoint compatibility check accepts c as target-only, but an UPDATE of an LCA row forces the probe to read c INT and append it into target c VARCHAR. A review-only unit with LCA c INT and target c VARCHAR sharing the same ColId/Seqnum expected the probe layout to contain only a; the current head returned [a,c]. DIFF/MERGE therefore fail before comparison even though c is excluded from comparison and MERGE apply.

Please thread the endpoint target-only/type-compatibility rule into LCA layout construction so such columns are omitted and reconstructed as NULL, matching the historical CollectChanges projection. Add DIFF/MERGE coverage with a common ancestor containing c INT, one endpoint MODIFY c VARCHAR, the sibling DROP c, and an updated ancestor row that exercises handleDelsOnLCA.

The prior Seqnum-redefinition and account/database-owner blockers are closed. The four affected package test suites and focused race x10 pass; the failing review-only case exposes the missing LCA branch in the current matrix.

VioletQwQ-0 · 2026-07-22T07:01:23Z

Fixed in 7a0fb11. The LCA probe now excludes endpoint target-only columns on both the SQL and reader paths, including the no-column-metadata fallback. Common columns still fail closed on identity/type mismatches, and full-layout reader results now resolve ENUM metadata by target index. This also prevents hidden RowID identity collisions from being appended as user column values, which was the source of the ROWID-to-INT/VARCHAR CI failures.

Validation:

go test ./pkg/frontend -count=1
go test -race ./pkg/frontend -run "TestHashDiff_HasLCAUpdateIgnoresTargetOnlyColumns|TestLCAProbeColumnLayout|TestLCAProbeResultTargetIndexes|TestHandleDelsOnLCA_SQLPaths" -count=1
git diff --check

Added DIFF and MERGE BVT regressions for the common-ancestor c INT -> target c VARCHAR / sibling DROP c path. CI is rerunning on the new head.

VioletQwQ-0 · 2026-07-22T08:00:25Z

Updated head: 5b57d3fa0.

I rechecked the complete PR diff and all previous change-request clusters. The earlier target-only projection, explicit ALTER semantics, deleted-generation compaction, owner-scope cleanup, DROP/ADD identity, and LCA target-only type fixes remain intact.

This update fixes the remaining composed historical-schema case found during preflight:

treat a rebuilt hidden __mo_cpkey_col as the same derived composite key only when its ordered component names, Seqnums, types, and nullability still match;
prefer that canonical identity over an accidental old RowID numeric-ID collision in the LCA path;
quote reserved/spaced composite-key names in the LCA VALUES join;
reject ALTER lineage timestamp overflow before UpdateSnapshot;
verify doDropSnapshot rolls back if branch-snapshot compaction fails after the user snapshot delete.

Validation:

go test ./pkg/frontend ./pkg/frontend/databranchutils ./pkg/sql/compile -count=1
focused -race tests x10 for composite-key mapping, drop rollback, and timestamp overflow
go vet on the same packages
isolated live MO DIFF/MERGE with quoted composite PK, pre-ALTER UPDATE, COPY ALTER, and post-ALTER INSERT
git diff --check
full 255-symbol preflight + mo-self-review: PASS, no PENDING findings

Please re-review this head.

XuPeng-SH

Approved on exact head 5b57d3fa0ec18283cb15397262e5b163d04c00a6. No blocking findings remain.

I revalidated the complete PR diff and every prior P1 cluster: target-only historical type projection, explicit ALTER transaction semantics, deleted intermediate-generation reclamation, account/database historical ownership after current-table-first deletion, DROP/ADD Seqnum identity, and incompatible target-only LCA probes. The latest derived composite-key fix is fail-closed on ordered component names, component Seqnums/types/nullability, handles accidental RowID identity collisions, and quotes PK identifiers in both VALUES aliases and join expressions.

Audit area	Q1 trigger/ownership	Q2 cleanup/failure path	Q3 completion/reuse
LCA SQL/reader probe	Input PK batch and snapshot reader callbacks have explicit ownership; concurrent callbacks are serialized	temporary/filter/output batches close on success and error; target-only columns are reconstructed as NULL	full/projected layouts map by target index; duplicate/missing keys complete deterministically
Historical generation projection	vectors are transferred once from source batches	hydration/probe errors clean projected data and tombstones	Seqnum/type mismatches fail closed except the verified derived composite key identity
Snapshot/PITR lineage compaction	owner mutation and compaction share one explicit transaction	any compaction DELETE failure rolls back the owner mutation and paired metadata work	deleted generations remain retained while owned and are reclaimed after the final owner disappears

Validation on this head:

go test ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile -count=1
focused go test -race for LCA layout/reader/SQL, rebuilt composite PK mapping, DROP snapshot rollback, and timestamp overflow
go vet ./pkg/frontend/databranchutils ./pkg/frontend ./pkg/sql/compile
git diff --check origin/main...HEAD

GitHub SCA and completed checks are green; UT/BVT/coverage jobs were still running with no failures at review submission.

ULookup and others added 27 commits May 28, 2026 00:30

Merge remote-tracking branch 'upstream/main' into pr-24666-work

d23ad0d

# Conflicts: # pkg/frontend/data_branch_output.go # pkg/frontend/data_branch_pick.go # pkg/frontend/data_branch_test.go

Merge remote-tracking branch 'upstream/main' into pr-24666-work

8bee19d

Merge branch 'main' into fix/issue-24549-schema-evolution-diff-merge

5e041da

fix(frontend): align data branch column indexes

568d5f7

Merge branch 'main' into fix/issue-24549-schema-evolution-diff-merge

a0ed6e0

fix(frontend): align data branch column indexes

3cf182e

fix(frontend): harden schema compatibility checks

ee0b8aa

fix(frontend): relax schema evolution metadata checks

d5a5c6a

fix(frontend): preserve branch lineage after schema evolution

04467ee

fix(frontend): probe LCA with common columns

e90a008

fix(frontend): preserve data branch changes across alter generations

8a998b7

test(frontend): cover sparse schema conflict updates

f0a5cb8

test(frontend): fix cluster-by schema evolution BVT

e8ff378

test(frontend): isolate lineage alter transaction BVT

31713c1

test(frontend): restore lineage transaction update case

7f51daf

fix(frontend): use live context for branch clone lock

e0cc767

fix(frontend): preserve ancestor branch updates

11def75

fix(frontend): preserve bounded historical branch updates

04b977c

fix: reclaim unowned historical alter lineage

efb637b

fix: preserve altered branch lineage ownership

2174674

VioletQwQ-0 temporarily deployed to ci July 20, 2026 04:29 — with GitHub Actions Inactive

VioletQwQ-0 temporarily deployed to ci July 22, 2026 05:28 — with GitHub Actions Inactive

VioletQwQ-0 temporarily deployed to ci July 22, 2026 05:29 — with GitHub Actions Inactive

VioletQwQ-0 had a problem deploying to ci July 22, 2026 05:29 — with GitHub Actions Failure

VioletQwQ-0 requested review from XuPeng-SH July 22, 2026 05:29

XuPeng-SH requested changes Jul 22, 2026

View reviewed changes

fix(frontend): exclude target-only columns from LCA probes

7a0fb11

VioletQwQ-0 had a problem deploying to ci July 22, 2026 07:01 — with GitHub Actions Failure

VioletQwQ-0 temporarily deployed to ci July 22, 2026 07:01 — with GitHub Actions Inactive

VioletQwQ-0 had a problem deploying to ci July 22, 2026 07:02 — with GitHub Actions Error

VioletQwQ-0 temporarily deployed to ci July 22, 2026 07:02 — with GitHub Actions Inactive

fix(frontend): preserve derived composite key lineage

5b57d3f

VioletQwQ-0 requested review from XuPeng-SH and gouhongshen and removed request for XuPeng-SH and gouhongshen July 22, 2026 08:00

VioletQwQ-0 temporarily deployed to ci July 22, 2026 08:00 — with GitHub Actions Inactive

VioletQwQ-0 had a problem deploying to ci July 22, 2026 08:00 — with GitHub Actions Failure

VioletQwQ-0 temporarily deployed to ci July 22, 2026 08:00 — with GitHub Actions Inactive

VioletQwQ-0 temporarily deployed to ci July 22, 2026 08:01 — with GitHub Actions Inactive

VioletQwQ-0 had a problem deploying to ci July 22, 2026 08:01 — with GitHub Actions Failure

XuPeng-SH approved these changes Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(frontend): support schema evolution in data branch diff/merge#25882

feat(frontend): support schema evolution in data branch diff/merge#25882
VioletQwQ-0 wants to merge 38 commits into
matrixorigin:mainfrom
VioletQwQ-0:codex/takeover-24666-data-branch-schema-evolution

VioletQwQ-0 commented Jul 20, 2026 •

edited

Loading

Uh oh!

VioletQwQ-0 commented Jul 22, 2026

Uh oh!

XuPeng-SH left a comment

Uh oh!

VioletQwQ-0 commented Jul 22, 2026

Uh oh!

VioletQwQ-0 commented Jul 22, 2026

Uh oh!

XuPeng-SH left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

VioletQwQ-0 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

Validation

Uh oh!

VioletQwQ-0 commented Jul 22, 2026

Uh oh!

XuPeng-SH left a comment

Choose a reason for hiding this comment

Uh oh!

VioletQwQ-0 commented Jul 22, 2026

Uh oh!

VioletQwQ-0 commented Jul 22, 2026

Uh oh!

XuPeng-SH left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

VioletQwQ-0 commented Jul 20, 2026 •

edited

Loading