fix(opensearch): repair bare orphan index on bootstrap without discarding populated indices (#36237)

[fabrizzio-dotCMS]

Problem

Re-fix for #36237 (QA failed the prior PRs #36238/#36240 on TC-003).

The idempotent-bootstrap reuse path introduced in #36238 re-asserted the custom mapping via putMapping against an orphaned cluster index (one that exists in the cluster but is missing from the dotCMS index store). On a bare orphan this failed:

INFO  Bootstrap: OS index already exists, reusing and re-asserting mapping: ...working_….os
ERROR MappingOperationsOS - putMapping failed for index ...working_….os — HTTP 400   (×8)

Root cause: the content mapping references the custom analyzer my_analyzer, defined in os-content-settings.json. Analyzers are static index settings that can only be applied at index creation — so a putMapping-only re-assert against a bare index is rejected (analyzer [my_analyzer] not found) and the index is left half-mapped, with the dotCMS dynamic templates missing.

How the orphan arises

In the migration catch-up path the OS index name is derived deterministically by mirroring the ES name (working_T0 → cluster_X.working_T0.os), not generated with a fresh timestamp. If a prior bootstrap created that physical index but crashed before committing its VersionedIndices store pointer, the next restart re-derives the same name, finds it already in the cluster, and the create fails with resource_already_exists.

Fix

Decide the orphan's fate by document count, so a populated index is never discarded:

• Empty orphan (0 docs) → delete and recreate from scratch, restoring full settings + base mapping + custom mapping. An empty index has no data and no reindex progress, so recreating it costs nothing operationally — and it's the only case putMapping-reuse could not repair.
• Populated orphan (>0 docs), or unknown count → reuse in place, untouched (not deleted, not recreated, not remapped). A dotCMS-created index already carries the full mapping; deleting it would force a full reindex (hours, degraded/inconsistent search) — not justified to clean up an orphan. On any uncertainty (the count probe fails) we err toward reuse.

The delete only ever fires against a demonstrably empty orphan, and only in the bootstrap path for a store slot that is not registered — never against the active production index.

Tests

• Unit (ContentletIndexAPIImplBootstrapTest, 8/8): empty→recreate, populated→reuse-untouched, doc-count-probe-fails→reuse, empty-orphan-delete-fails→still-creates, missing→create, create-fails→no-mapping, existence-probe-fails→create, OS-tag propagation.
• Integration (ContentletIndexAPIImplMigrationIntegrationTest): new regression IT creates a bare orphan against a real OpenSearch cluster, runs the bootstrap seam, and asserts the recreated index carries the dotCMS dynamic templates (template_1) and the my_analyzer setting. Verified green against OpenSearch:

  Tests run: 23, Failures: 0, Errors: 0, Skipped: 6  → BUILD SUCCESS
  ✅ bootstrap bare-orphan Phase 1 — recreated with full settings + mapping
  

Closes #36237

🤖 Generated with Claude Code

This PR fixes: #36237

[claude]

Claude finished @fabrizzio-dotCMS's task in 2m 22s —— View job

---

Rollback Safety Analysis

• Read rollback-unsafe categories reference
• Get full PR diff
• Analyze diff against all unsafe categories
• Apply appropriate label

---

Result: ✅ Safe To Rollback

This PR changes 3 files, all scoped to OpenSearch bootstrap logic and tests:

File	Type
ContentletIndexAPIImpl.java	Bootstrap orphan-handling logic
ContentletIndexAPIImplBootstrapTest.java	Unit tests
ContentletIndexAPIImplMigrationIntegrationTest.java	Integration tests

Analysis Against Every Unsafe Category

Category	Verdict
C-1 Structural data model change	✅ None — no DB schema changes, no runonce tasks
C-2 Elasticsearch/OpenSearch mapping change	✅ None — the fix is about not calling putMapping on a bare orphan during bootstrap. No ES/OS index mapping fields, types, or analyzers are modified in the live index; the code changes only affect the bootstrap path for an un-registered (orphaned) cluster index.
C-3 Content JSON model version bump	✅ None — no CURRENT_MODEL_VERSION or contentlet_as_json changes
C-4 DROP TABLE / DROP COLUMN	✅ None
H-1 One-way data migration	✅ None — the deletion path only targets a demonstrably empty orphan not registered in the store; no business data is transformed or lost
H-2 RENAME TABLE / COLUMN	✅ None
H-3 PK restructuring	✅ None
H-4 New content type field type	✅ None
H-5 Binary storage provider change	✅ None
H-6 DROP PROCEDURE / FUNCTION	✅ None
H-7 NOT NULL column without default	✅ None
H-8 VTL viewtool contract change	✅ None
M-1 Non-broadening column type change	✅ None
M-2 Push publishing bundle format	✅ None
M-3 REST/GraphQL API contract change	✅ None
M-4 OSGi public interface change	✅ None

Reasoning

The entire diff is contained within the bootstrap seam of ContentletIndexAPIImpl.createContentIndex() — a path that only runs during dotCMS startup when an OS index name collision is detected. No database schema is touched, no index mappings are altered on any existing live index, no data is transformed, and no API contracts are changed. Rolling back to N-1 simply reverts to the prior bootstrap behavior (reuse-and-putMapping), which is the very state that existed before these commits. No manual cluster intervention would be needed after rollback.

[github-actions]

🤖 dotBot Review (Bedrock)

Reviewed 3 file(s); 2 candidate(s) → 0 confirmed, 0 uncertain (unverified, kept for review).

✅ No issues found after verification.

---

_{us.deepseek.r1-v1:0 · Run: #28390053907 · tokens: in: 28972 · out: 11674 · total: 40646 · calls: 7 · est. ~$0.102}

[fabrizzio-dotCMS]

Addressed the 🟡 Medium finding (failed orphan deletion → create failure) in 47b6899.

The empty-orphan branch no longer falls through to a create after a non-acknowledged delete. If the delete isn't confirmed (so the index may still exist), the orphan is now reused in place instead of attempting a create that would throw resource_already_exists — which bootstrapAndPointOS doesn't catch (it only handles IOException) and would abort bootstrap, i.e. the exact original #36237 failure. The orphan is empty so nothing is lost, and a later clean restart recreates it properly once the cluster is healthy.

Unit test test_emptyOrphanDeleteFails_reusedInPlace_notRecreated updated to assert no create / no remap on a non-acknowledged delete. All 8 unit tests green.

[fabrizzio-dotCMS]

Addressed the 🔴 Critical finding (failed orphan deletion leaves broken index state) in 8b8da22.

You're right — the previous revision's return true on an unacknowledged delete would register a bare, un-repairable orphan (no custom analyzer → mapping cannot be fixed in place), leaving search permanently broken.

New behavior when the empty-orphan delete is not acknowledged — re-probe existence:

• index is gone (delete took effect without an ack) → recreate cleanly;
• index still present → throw and fail bootstrap loudly with a clear message, rather than recreating (would throw resource_already_exists) or registering a half-mapped index. This is an abnormal cluster state (not the orphan-name collision the method resolves), so failing loud — consistent with fix(opensearch): fail loudly on migration init failure + surface ES shard failures (#36237) #36240 — is the correct outcome.

Unit tests split into test_emptyOrphanDeleteUnacked_butIndexGone_recreates and test_emptyOrphanDeleteFails_indexStillExists_failsLoud. All 9 unit tests green.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request