OpenSearch: index bootstrap not idempotent — orphaned cluster index aborts startup

[fabrizzio-dotCMS]

Problem

During the OpenSearch migration, index bootstrap is not idempotent against an orphaned cluster index — an index that physically exists in the cluster but is missing from the dotCMS index store.

This happens when a previous startup created a physical index but never committed its store pointer (e.g. the OS VersionedIndices row was left uncommitted after a partial/interrupted init, or the OS live create / pointOS failed mid-bootstrap).

Observed symptoms (Phase 1)

1. On restart, the catchup path re-derives the same logical name and the create fails:

   org.opensearch.client.opensearch._types.OpenSearchException: Request failed:
   [resource_already_exists_exception] index [cluster_XXXX.working_YYYYYYYY.os/...] already exists
       at com.dotcms.content.index.opensearch.OSIndexAPIImpl.createIndex(OSIndexAPIImpl.java)
       at ...ContentletIndexAPIImpl.bootstrapAndPointOS(...)
       at ...ContentletIndexAPIImpl.initOSCatchup(...)
       at ...ContentletIndexAPIImpl.checkAndInitializeIndex(...)
   

   The exception bubbles to checkAndInitializeIndex() and is swallowed, leaving the instance half-initialised.

2. Because init aborted, the read provider then queries an index that was never finished, surfacing as:

   Elasticsearch SEARCH error ... search_phase_execution_exception, reason=all shards failed
   

Root cause

ContentletIndexAPIImpl.createContentIndex(name, shards, tag) (used only by the ES/OS bootstrap paths) always issues a create. The indexReadyOS()/indexReadyES() checks key off the store (VersionedIndices / indicies), not the physical cluster state, so when the two diverge the bootstrap retries a create that can never succeed.

Fix

Make createContentIndex(name, shards, tag) idempotent: if the physical index already exists, reuse it, re-assert the custom mapping (putMapping is additive), and let the caller's point() re-register it in the store instead of issuing a failing create.

Follow-ups (out of scope of the fix PR)

• checkAndInitializeIndex() swallows bootstrap failures and leaves the instance running half-initialised — should fail loudly instead.
• ContentFactoryIndexOperationsES logs only the wrapper message on search_phase_execution_exception, hiding the per-shard ShardSearchFailure root cause — should surface it.

[fabrizzio-dotCMS]

PRs

• fix(opensearch): make index bootstrap idempotent for orphaned cluster indices (#36237) #36238 — fix(opensearch): make index bootstrap idempotent for orphaned cluster indices — the root-cause fix.
• fix(opensearch): fail loudly on migration init failure + surface ES shard failures (#36237) #36240 — fix(opensearch): fail loudly on migration init failure + surface ES shard failures — covers the two follow-ups below (defense-in-depth + observability).

The two PRs are independent and can merge in any order.

Follow-up status

• checkAndInitializeIndex() swallowing bootstrap failures — fixed in fix(opensearch): fail loudly on migration init failure + surface ES shard failures (#36237) #36240 (re-throws when a migration is active; legacy ES-only installs keep best-effort boot).
• ContentFactoryIndexOperationsES hiding the per-shard ShardSearchFailure root cause — fixed in fix(opensearch): fail loudly on migration init failure + surface ES shard failures (#36237) #36240 (describeEsFailure() surfaces detailed message + per-shard reasons + suppressed causes).

[github-actions]

PRs linked to this issue

• fix(opensearch): make index bootstrap idempotent for orphaned cluster indices (#36237) by @fabrizzio-dotCMS
• fix(opensearch): repair bare orphan index on bootstrap without discarding populated indices (#36237) by @fabrizzio-dotCMS

[fabrizzio-dotCMS]

🧪 Note for testers — how to verify this fix

Merged: #36238 idempotent bootstrap, #36240 fail-loudly + better ES error log, and the TC-003 re-fix #36353 (repair a bare orphan by doc-count: empty → recreate with full settings+mapping, populated → reuse untouched). Below is the manual QA checklist to run against a post-merge build (nightly / staging / RC). All checks are manual; the unit coverage from #36238/#36353 already runs in CI.

Migration phase: unless a test case says otherwise, run every scenario in an OpenSearch migration, Phase 1 (shadow) — that is where the bootstrap creates the OS indices and where the orphaned-index divergence originates (the pointOS/create can be left half-committed). TC-005 is the only ES-only (no migration) case.

Heads-up: TC-002 (the main one) requires recreating the "index exists in the cluster but is missing from dotCMS's store" divergence. Do not try to fabricate the index name by hand — the physical name carries a yyyyMMddHHmmss timestamp (cluster_<id>.working_<yyyyMMddHHmmss>), so you cannot predict the name of a future boot. Instead, start from an index dotCMS already created (real timestamp included) and break only the store side: drop / leave-uncommitted its VersionedIndices store row while leaving the physical index alive in the cluster. If you don't have DB/cluster console access, ask a dev to leave the store pointer uncommitted after a create.

TC-001 — Healthy startup on a clean OpenSearch migration (Happy Path)

Phase: OpenSearch migration — Phase 1 (shadow).

1. Start a fresh post-merge build in an OpenSearch migration scenario (Phase 1 shadow).
2. Let it boot fully and log in.
3. Open System → Maintenance → Index and confirm working/live indices are present and green.
4. Create one piece of content and search for it.

Expected: boots with no FATAL/ERROR about index creation, indices healthy, content searchable. No resource_already_exists or "all shards failed".

TC-002 — Verify #36237: restart with an orphaned cluster index comes up healthy (High) ⭐

Phase: OpenSearch migration — Phase 1 (shadow).

1. Bring up the build in an OpenSearch migration scenario (Phase 1 shadow) and let it boot. dotCMS creates its working index (e.g. cluster_xxx.working_<yyyyMMddHHmmss>) and registers it in the index store. Note the exact name from System → Maintenance → Index or the startup log — this real, already-generated name is what you reuse (do not invent a timestamp).
2. Stop dotCMS (leave the cluster running so the physical index stays alive).
3. Simulate the orphan — break the store, not the cluster: delete (or leave uncommitted) the VersionedIndices store row that points at that working index, while leaving the physical index untouched in the cluster. Confirm via the cluster API that the index still physically exists but dotCMS's store no longer knows about it (the "exists in cluster, missing from store" divergence). On restart, the catchup path re-derives this same logical name and tries to re-create it.
4. Start dotCMS again and watch the startup log.
5. After boot: log in, open the index screen, create content, and search for it.

Expected: startup completes normally. Log shows INFO Bootstrap: ... index already exists, reusing and re-asserting mapping: <name>, and no resource_already_exists_exception, no Failed to create new indexes. dotCMS is fully started (not half-initialised): index screen healthy, new content saves and is searchable, no search_phase_execution_exception / "all shards failed".

TC-003 — Bare orphan index gets repaired on reuse (Medium) — verify #36353

Phase: OpenSearch migration — Phase 1 (shadow).

🛑 Read this before running — do NOT validate by diffing the index mapping. The previous two rejections compared the _mapping of the original (populated) index against the repaired (empty) one and flagged ~87 "missing" fields (title, inode, identifier, conhost, contenttype, moddate, deleted, live, working, permissions, categoryperms, and all *_dotraw variants). That is expected and is not a bug. dotCMS never writes those system fields as a static mapping — os-content-mapping.json has no properties section, only dynamic_templates. Those fields are created automatically by OpenSearch's dynamic templates the first time a document is indexed. The original index had them only because content had already been indexed into it; a freshly repaired empty index has not been indexed yet, so they legitimately don't exist until content flows in. A raw before/after mapping diff will always show them missing on an empty index and tells you nothing about whether the repair worked. Validate functionally (steps below) instead.

What #36353 actually does with a bare orphan (index exists in cluster but missing from dotCMS's store):

• Empty orphan (0 docs): dotCMS deletes it and recreates it from scratch with full settings (incl. the my_analyzer analyzer) + base mapping (dynamic_templates) + per-content-type custom mappings.
• Populated orphan (>0 docs, or doc-count probe fails): dotCMS reuses it untouched (it already carries the full mapping; deleting would force a costly full reindex).

Repro (empty/bare orphan path):

1. Boot dotCMS in Phase 1 (shadow), let it fully start. Note the working/live index names (System → Maintenance → Index, or SELECT index_name, index_type FROM indicies WHERE index_version IS NOT NULL;).
2. Stop only the dotCMS service (keep the cluster running).
3. Simulate the orphan: DELETE FROM indicies WHERE index_version IS NOT NULL; (drops the store pointer). Then delete the physical OS indices and recreate them bare (PUT /<name> with {} — no settings, no mapping), so they have 0 docs and no mapping.
4. Restart dotCMS and watch the startup log.
5. After boot: log in, create content (or run a reindex from System → Maintenance) using a content type that has a custom field (e.g. a date or tag field), then search and filter on both a system field (contentType, conHost, modDate) and the custom field.

Expected:

• Startup completes normally. No putMapping failed ... HTTP 400, no analyzer [my_analyzer] not found, no resource_already_exists_exception, no Failed to create new indexes.
• The repaired index carries what dotCMS controls explicitly: the my_analyzer setting, the dynamic templates (e.g. template_1 for *_dotraw), and the per-content-type mappings. (Check GET /<name>/_settings for my_analyzer and GET /<name>/_mapping for dynamic_templates + the content-type objects — not the system-field count.)
• The functional check is the pass/fail criterion: after indexing content, searching/filtering on system fields and on the custom field works correctly. The dynamic system fields reappear in the mapping automatically once content is indexed.

If you want a mapping-level comparison to come out equal, capture the after mapping only after re-indexing content into the repaired index — then it matches the populated before. Comparing an empty repaired index against a populated original is not a valid check.

Optional — valid "index → compare mappings" procedure

If you still want to confirm at the mapping level that the fields are added back, index content into the repaired index first, then diff:

1. Index content into the repaired index. Either works:
   • Recommended (exercises the repaired index itself): create / save / publish several contentlets — ideally across different content types, including one that goes through a workflow and one with the custom field (date/tag). Each indexed document fires the dynamic_templates and materialises the fields.
   • ⚠️ "Reindex All" from Maintenance creates a brand-new index, it does not backfill the repaired one — that tests creation, not the in-place repair.
2. Re-capture the mapping and diff against before:

   curl -sk "https://localhost:9201/${WORKING}/_mapping" -u 'admin:...' | python3 -m json.tool > working-after-reindex.json
   diff <(python3 -c "import json;print(json.dumps(json.load(open('working-before.json')),sort_keys=True,indent=2))") \
        <(python3 -c "import json;print(json.dumps(json.load(open('working-after-reindex.json')),sort_keys=True,indent=2))")

What you should see:

• System fields (title, inode, conhost, contenttype, moddate, deleted, live, working, *_dotraw, …) reappear as soon as the first document is indexed (they are present on every doc).
• Per-content-type fields were already there from boot (applied by addCustomMapping, independent of indexed data).
• Exact equality with before (all 123 fields) requires indexing the same variety of content the original had — e.g. wfassign / wfscheme / wfstep only appear if you index content with a workflow, and urlmap only with content that has a URL map. A single contentlet brings back the bulk; full parity needs those cases covered.

Fast no-indexing sanity check (confirms the repair applied what dotCMS controls, without indexing anything):

# the custom analyzer must exist (its absence was the original HTTP 400 cause)
curl -sk "https://localhost:9201/${WORKING}/_settings" -u 'admin:...' | grep -i my_analyzer
# dynamic_templates + per-content-type objects must be present
curl -sk "https://localhost:9201/${WORKING}/_mapping" -u 'admin:...' | python3 -m json.tool | grep -iE "template_1|dynamic_templates"

If my_analyzer + dynamic_templates + the content-type objects are present, the repair worked — the system fields are an automatic consequence of indexing.

TC-004 — Verify #36240: migration startup that can't initialise stops loudly (High) ⭐

Phase: OpenSearch migration — Phase 1 (shadow), migration active.

1. Configure the build for an active OpenSearch migration (migration started — Phase 1 shadow).
2. Make index initialisation fail unrecoverably at boot — e.g. point the OpenSearch connection at an unreachable host/port.
3. Start dotCMS and observe startup + log.

Expected: startup aborts loudly — FATAL Failed to create new indexes followed by Index initialization failed during OpenSearch migration; aborting startup to avoid serving an unfinished index store. Cause: .... dotCMS does not come up serving requests half-initialised. (After fixing connectivity and restarting, it boots normally.)

TC-005 — Legacy Elasticsearch-only install (no migration) still boots best-effort (Medium)

Phase: Elasticsearch-only — no migration (this is the only non-migration case).

1. Configure the build as a classic Elasticsearch-only install with no migration started.
2. Start dotCMS, log in, confirm search works.
3. (Optional) Introduce a transient index hiccup at boot in this ES-only mode and confirm startup is not hard-aborted.

Expected: boots normally as before — the "abort startup" behavior from TC-004 applies only when a migration is active.

TC-006 — Verify #36240: "all shards failed" log shows the real per-shard cause (Medium)

Phase: OpenSearch migration — Phase 1 (shadow). (The formatting change is in the ES read path; reproduce it within the Phase 1 shadow scenario so the run stays consistent with the rest of the suite.)

1. Trigger a search against a missing/unhealthy index to reproduce search_phase_execution_exception ("all shards failed").
2. Open the dotCMS log.

Expected: the "Elasticsearch SEARCH error" block now includes the detailed message plus per-shard reasons (shard[0]: <reason>, capped at 5 with "... and N more") and any caused by: suppressed causes — root cause diagnosable from the log alone, no literal null rendered.

---

Out of scope: permissions, sites, languages, content version state, workflow, push publish, REST, and UI — the change is purely in the index bootstrap/startup path. Orphan-reuse logic is already unit-covered by ContentletIndexAPIImplBootstrapTest (in CI via #36238). The fail-loudly path and the ES error formatting have no automated test yet — worth a follow-up IT.