Fix: plan unexecuted migrations regardless of major.minor order

[sonika-shah]

Summary

processNativeMigrations currently drops any unexecuted migration whose major.minor is lower than the current max's major.minor (sameOrHigherMajorMinor check). In upgrade paths where the ledger has a gap below the current max, those missing versions are silently skipped forever, and MigrationValidationClient reports them as Missing migrations that were not executed [...].

This PR removes the sameOrHigherMajorMinor gate. The ledger check (!executedMigrations.contains(version)) is sufficient — any version recorded in SERVER_CHANGE_LOG is still skipped, any version not recorded is planned. Reprocessing on the max version is preserved, so the continuous-release behavior (new SQL appended to the current version's schemaChanges.sql, gated by statement-level checksums in SERVER_MIGRATION_SQL_LOGS) is unaffected.

After the fix, processNativeMigrations matches processExtensionMigrations, which already uses a containment-only filter.

Why

Two real-world scenarios hit this gate today:

1. Historical gap. A dump whose ledger contains 1.5.0 … 1.5.11, 1.9.0 … but is missing 1.5.15. With max = 1.9.x, sameOrHigherMajorMinor("1.5.15", "1.9.x") returns false, so 1.5.15 is never planned on subsequent upgrades — even when the folder is present on disk and the migration hasn't been run.

2. Backport below current max. When 1.11.11 / 1.11.12 are released after 1.12.1 has been cut, a cluster upgrading from 1.12.1 → 1.12.2 sees max = 1.12.1 and drops the backported 1.11.x migrations. This is the original incident behind Minor migration fix #26592 on the 1.12.x line.

Both cases already work on the 1.12.x release branches (which carry #26592's containment-only filter). This PR brings main to parity.

What changes

• MigrationWorkflow.java
  • Remove sameOrHigherMajorMinor helper and its usage inside processNativeMigrations.
  • Retain reprocessing-on-maxVer (continuous-release scenario from Fix #26570: Add continuous migrations #26571).
• MigrationWorkflowTest.java
  • Update three tests that asserted the old exclusion behavior.
  • Add a historical-gap test (1.5.15 below executed max 1.10.5).
  • Remove the standalone sameOrHigherMajorMinorComparisons test (method is gone).

Guard rails that stay intact

• Version-level: SERVER_CHANGE_LOG containment check — any already-executed version is skipped.
• Statement-level: SERVER_MIGRATION_SQL_LOGS checksum check inside the SQL runner — already-applied statements skip even during reprocess.
• Idempotency contract for reprocessed Java migrations — unchanged from Fix #26570: Add continuous migrations #26571.

Test plan

• mvn test -Dtest=MigrationWorkflowTest — 25/25 pass
• Manual: restore a 1.9.x dump with a missing 1.5.15 row in SERVER_CHANGE_LOG, run ./bootstrap/openmetadata-ops.sh migrate, verify health check passes and 1.5.15 is recorded.
• Manual: 1.11.x backport path — cluster on 1.12.1, upgrade to build containing 1.11.11 folder on disk, verify it's planned and executed.

Related

• Fix #26570: Add continuous migrations #26571 — introduced the continuous-release reprocessing path (preserved here).
• Minor migration fix #26592 — equivalent fix already on 1.12.x line.

---

Summary by Gitar

• Refactored migration resolution:
  • Replaced hardcoded class instantiation with ServiceLoader to support MigrationProcessExtensionProvider for pluggable extension migration logic.
  • Introduced resolveMigrationProcess to handle delegation between native migrations and extension providers, ensuring Java data migrations are not incorrectly re-run for extension versions.
• Test enhancements:
  • Added a regression test loadMigrationsFallsBackToNoOpWhenNoExtensionProviderHandlesVersion to ensure extension versions correctly default to MigrationProcessImpl when no provider is found.

_{This will update automatically on new commits.}

[Copilot]

Pull request overview

Updates the server migration planner so that any native migration present on disk but missing from SERVER_CHANGE_LOG is planned, even if its major.minor is lower than the currently executed max. This prevents “missing migrations” from being silently skipped forever in upgrade paths with ledger gaps or backported patch releases.

Changes:

• Remove the sameOrHigherMajorMinor filter from processNativeMigrations, relying solely on execution-history containment while preserving “reprocess max version” behavior.
• Update/rename affected unit tests to assert inclusion of backported lower-minor migrations.
• Add a unit test covering the “historical gap below executed max” scenario; remove the now-obsolete sameOrHigherMajorMinor comparison test.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
openmetadata-service/src/main/java/org/openmetadata/service/migration/api/MigrationWorkflow.java	Removes major.minor gating so unexecuted native migrations below the executed max are still planned; keeps max-version reprocessing intact.
openmetadata-service/src/test/java/org/openmetadata/service/migration/api/MigrationWorkflowTest.java	Updates expectations for backported/lower-minor inclusion, adds historical-gap regression test, and deletes the helper-method test that no longer applies.

[github-actions]

🟡 Playwright Results — all passed (12 flaky)

✅ 3963 passed · ❌ 0 failed · 🟡 12 flaky · ⏭️ 86 skipped

Shard	Passed	Failed	Flaky	Skipped
✅ Shard 1	299	0	0	4
🟡 Shard 2	742	0	3	8
🟡 Shard 3	744	0	4	7
🟡 Shard 4	758	0	1	18
✅ Shard 5	687	0	0	41
🟡 Shard 6	733	0	4	8

🟡 12 flaky test(s) (passed on retry)

• Features/ActivityAPI.spec.ts › Activity event is created when description is updated (shard 2, 1 retry)
• Features/ActivityAPI.spec.ts › Activity event shows the actor who made the change (shard 2, 1 retry)
• Features/Glossary/GlossaryWorkflow.spec.ts › should start term as Draft when glossary has reviewers (shard 2, 1 retry)
• Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
• Features/Table.spec.ts › Tags term should be consistent for search (shard 3, 1 retry)
• Flow/PersonaDeletionUserProfile.spec.ts › User profile loads correctly before and after persona deletion (shard 3, 1 retry)
• Flow/PersonaFlow.spec.ts › Set default persona for team should work properly (shard 3, 1 retry)
• Pages/DataContracts.spec.ts › Create Data Contract and validate for Dashboard (shard 4, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
• Pages/ODCSImportExport.spec.ts › Multi-object ODCS contract - object selector shows all schema objects (shard 6, 1 retry)
• Pages/UserDetails.spec.ts › Admin user can edit teams from the user profile (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Pull request overview

Fixes migration planning so that any unexecuted native migration present on disk is planned, even if its major.minor is lower than the current max executed version—eliminating a long-standing gap/backport skip scenario and aligning behavior with extension migration planning.

Changes:

• Remove sameOrHigherMajorMinor gating from native migration selection (containment-only filter + reprocess max preserved).
• Update existing unit tests to reflect the new inclusion behavior for backported/lower-minor migrations.
• Add a new unit test covering a “historical gap below max executed” scenario.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
openmetadata-service/src/main/java/org/openmetadata/service/migration/api/MigrationWorkflow.java	Removes major/minor order filtering so unexecuted native migrations are always planned; keeps reprocessing of max executed native version.
openmetadata-service/src/test/java/org/openmetadata/service/migration/api/MigrationWorkflowTest.java	Updates assertions for backport behavior, adds a historical-gap regression test, and removes the obsolete helper-method test.

[gitar-bot]

Code Review ✅ Approved

Migration execution logic now correctly handles unexecuted migrations regardless of major or minor version ordering. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[Copilot]

Pull request overview

This PR fixes native migration planning so that any migration present on disk but missing from SERVER_CHANGE_LOG will be planned, even if its major.minor is lower than the currently max-executed major.minor. This aligns native migration behavior with extension migration planning and prevents historical gaps/backported patch migrations from being skipped indefinitely.

Changes:

• Removed the sameOrHigherMajorMinor filter from native migration planning so unexecuted versions are selected purely by execution history.
• Updated existing unit tests to reflect the new inclusion behavior for backported lower-minor versions.
• Added a new unit test covering a “historical gap below max” scenario (e.g., missing 1.5.15 while max executed is 1.10.5).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
openmetadata-service/src/main/java/org/openmetadata/service/migration/api/MigrationWorkflow.java	Removes the major.minor gating in processNativeMigrations, relying on executed-version containment while preserving max-version reprocessing.
openmetadata-service/src/test/java/org/openmetadata/service/migration/api/MigrationWorkflowTest.java	Updates assertions for backport behavior, adds a historical-gap test case, and removes the deleted helper’s test.

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud