fix(opensearch): make index bootstrap idempotent for orphaned cluster indices (#36237) (#36238)

## Proposed Changes

Fixes #36237.

Makes the OpenSearch/Elasticsearch index **bootstrap idempotent**
against an *orphaned cluster index* — one that physically exists in the
cluster but is missing from the dotCMS index store.

### Root cause

`ContentletIndexAPIImpl.createContentIndex(name, shards, tag)` (used
only by the ES/OS bootstrap paths) always issues a create. Readiness is
decided by `indexReadyOS()`/`indexReadyES()`, which key off the
**store** (`VersionedIndices` / `indicies`), not the physical cluster
state. When a prior startup created the physical index but never
committed its store pointer (partial/interrupted init, failed `pointOS`,
etc.), the two diverge: on restart the catchup path re-derives the same
logical name and the create fails with `resource_already_exists`.

That exception bubbles to `checkAndInitializeIndex()` and is swallowed,
leaving the instance **half-initialised** — and the read provider then
queries an unfinished index, surfacing as
`search_phase_execution_exception / all shards failed`.

### The fix

In `createContentIndex(name, shards, tag)`, before creating, check
`providerApi.indexExists(physicalName)`:
- **Already exists** → reuse it, re-assert the custom mapping
(`putMapping` is additive/idempotent, so a previously unmapped orphan is
repaired), and return `true`. The caller's `point()` re-registers it in
the store.
- **Does not exist** → create as before.

This is scoped to the two callers (`bootstrapAndPointES` /
`bootstrapAndPointOS`); no public/interface signatures change. The
`(IndexAPIImpl) indexAPI` cast mirrors the existing
`indexReadyES()/indexReadyOS()` usage.

## Testing

- `./mvnw compile -pl :dotcms-core` (JDK 25) — compiles clean.
- No signature changes; `ContentletIndexAPIImplPhaseTest` fakes never
exercise the bootstrap path, so existing unit coverage is unaffected.
- A meaningful idempotency IT requires a live cluster
(`opensearch-upgrade` profile) since the bootstrap path casts to
`IndexAPIImpl` — can be added on request.

## Out of scope (tracked in #36237)

- `checkAndInitializeIndex()` swallowing bootstrap failures instead of
failing loudly.
- `ContentFactoryIndexOperationsES` hiding the per-shard
`ShardSearchFailure` root cause behind the generic "all shards failed"
wrapper.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- re-run link-issue: Team : Scout added to #36237 -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: fabrizzio-dotCMS

dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java

@@ -652,22 +652,75 @@ public synchronized boolean createContentIndex(final String indexName, final int
 
 
     /**
-     * Create an index exclusively in one of the SE Providers
-     * @param indexName
-     * @param shards
-     * @param tag
-     * @return
-     * @throws IOException
+     * Create an index exclusively in one of the SE Providers.
+     *
+     * <p><b>Idempotent bootstrap.</b> If the physical index already exists in the target
+     * cluster it is reused instead of issuing a create. This guards against an orphaned
+     * cluster index — present in the cluster but missing from the index store — left behind
+     * when a previous bootstrap created the index but never committed its store pointer
+     * (e.g. the OS {@code VersionedIndices} row, or after a partial/interrupted startup).
+     * Without this guard the restart re-derives the same logical name, the create fails with
+     * {@code resource_already_exists}, and {@code checkAndInitializeIndex()} aborts — leaving
+     * the instance half-initialised. The custom mapping is (re)applied either way
+     * ({@code putMapping} is additive/idempotent), so a previously unmapped orphan is repaired,
+     * and the caller's {@code point()} re-registers the index in the store.</p>
+     *
+     * @param indexName logical index name (no cluster prefix, no vendor tag)
+     * @param shards    number of shards to create with (ignored when the index already exists)
+     * @param tag       target provider ({@link IndexTag#ES} or {@link IndexTag#OS})
+     * @return {@code true} when the index exists (reused) or was created successfully
+     * @throws IOException on a hard creation failure
      */
     private boolean createContentIndex(final String indexName, final int shards, IndexTag tag)
             throws IOException {
-        final MappingHelper helper = MappingHelper.getInstance();
+        final IndexAPIImpl impl = (IndexAPIImpl) indexAPI;
 
-        ContentletIndexOperations ops = router.esImpl();
-        if(tag == IndexTag.OS) {
-           ops = router.osImpl();
-        }
+        final ContentletIndexOperations ops = tag == IndexTag.OS ? router.osImpl() : router.esImpl();
+        final IndexAPI providerApi = tag == IndexTag.OS ? impl.osImpl() : impl.esImpl();
+
+        return createContentIndex(indexName, shards, tag, ops, providerApi,
+                MappingHelper.getInstance());
+    }
+
+    /**
+     * Idempotent-bootstrap core of {@link #createContentIndex(String, int, IndexTag)}, with the
+     * provider collaborators injected so the orphan-reuse decision can be unit-tested without a
+     * running cluster or the {@link MappingHelper} singleton. See the public-facing overload's
+     * javadoc for the behaviour contract.
+     *
+     * @param indexName   logical index name (no cluster prefix, no vendor tag)
+     * @param shards      number of shards to create with (ignored when the index already exists)
+     * @param tag         target provider ({@link IndexTag#ES} or {@link IndexTag#OS})
+     * @param ops         vendor write operations for {@code tag} (creation + physical-name mapping)
+     * @param providerApi vendor index API for {@code tag} (existence probe)
+     * @param helper      mapping helper used to (re)assert the custom mapping
+     * @return {@code true} when the index exists (reused) or was created successfully
+     * @throws IOException on a hard creation failure
+     */
+    boolean createContentIndex(final String indexName, final int shards, final IndexTag tag,
+            final ContentletIndexOperations ops, final IndexAPI providerApi,
+            final MappingHelper helper) throws IOException {
         final String physicalName = ops.toPhysicalName(indexName);
+
+        // Reuse an orphaned cluster index rather than failing the create (see method javadoc).
+        // The existence probe is best-effort: any failure is treated as "does not exist" so we
+        // fall through to the create path rather than aborting bootstrap on a transient error.
+        // The failure is logged at DEBUG so a real connectivity/config problem (which would then
+        // also surface on the create attempt) stays traceable instead of being silently swallowed.
+        final boolean alreadyExists = Try.of(() -> providerApi.indexExists(physicalName))
+                .onFailure(e -> Logger.debug(this,
+                        "Bootstrap existence probe failed for " + physicalName
+                        + " — treating as 'does not exist' and attempting create: "
+                        + e.getMessage(), e))
+                .getOrElse(false);
+        if (alreadyExists) {
+            Logger.info(this, String.format(
+                    "Bootstrap: %s index already exists, reusing and re-asserting mapping: %s",
+                    tag, physicalName));
+            helper.addCustomMapping(List.of(indexName), tag);
+            return true;
+        }
+
         final boolean contentIndex = ops.createContentIndex(physicalName, shards);
         if (contentIndex) {
             helper.addCustomMapping(List.of(indexName), tag);

dotCMS/src/test/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImplBootstrapTest.java

@@ -0,0 +1,180 @@
+package com.dotcms.content.elasticsearch.business;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import com.dotcms.content.elasticsearch.util.MappingHelper;
+import com.dotcms.content.index.ContentletIndexOperations;
+import com.dotcms.content.index.IndexAPI;
+import com.dotcms.content.index.IndexTag;
+import com.dotcms.content.index.VersionedIndicesAPI;
+import java.io.IOException;
+import java.util.List;
+import org.junit.Test;
+
+/**
+ * Unit tests for the idempotent index-bootstrap logic introduced for orphaned cluster indices
+ * (#36237 / PR #36238).
+ *
+ * <p>Exercises {@link ContentletIndexAPIImpl#createContentIndex(String, int, IndexTag,
+ * ContentletIndexOperations, IndexAPI, MappingHelper)} — the collaborator-injected seam of the
+ * private {@code createContentIndex(indexName, shards, tag)} — so the orphan-reuse decision can be
+ * validated without a running Elasticsearch/OpenSearch cluster or the {@code MappingHelper}
+ * singleton. All collaborators are Mockito doubles.</p>
+ *
+ * <p>The behaviour under test:</p>
+ * <ul>
+ *   <li>an index already present in the cluster is <b>reused</b> (no create) and its mapping is
+ *       re-asserted — the orphaned-index repair path;</li>
+ *   <li>a missing index is created and, on success, mapped;</li>
+ *   <li>a failed create does not apply a mapping;</li>
+ *   <li>a failing existence probe is treated as "does not exist", so bootstrap falls through to
+ *       the create path instead of aborting.</li>
+ * </ul>
+ */
+public class ContentletIndexAPIImplBootstrapTest {
+
+    private static final String CLUSTER_PREFIX = "cluster_test.";
+    private static final String LOGICAL_NAME = "working_T0";
+    private static final String PHYSICAL_NAME = CLUSTER_PREFIX + LOGICAL_NAME;
+    private static final int SHARDS = 1;
+
+    /**
+     * Builds an instance solely so the package-private seam can be invoked. The constructor
+     * dependencies are irrelevant to the seam, which operates exclusively on its injected
+     * collaborators.
+     */
+    private static ContentletIndexAPIImpl newApi() {
+        return new ContentletIndexAPIImpl(
+                mock(ContentletIndexOperations.class),
+                mock(ContentletIndexOperations.class),
+                mock(IndexAPI.class),
+                mock(IndiciesAPI.class),
+                mock(VersionedIndicesAPI.class));
+    }
+
+    /**
+     * Given : the physical index already exists in the target cluster (an orphaned cluster index
+     *         left behind by a previous bootstrap that never committed its store pointer).
+     * When  : createContentIndex() runs during bootstrap.
+     * Then  : the index is reused (no create is issued), the custom mapping is re-asserted to
+     *         repair a possibly-unmapped orphan, and the method returns true.
+     */
+    @Test
+    public void test_orphanIndexExists_reusesAndReassertsMapping_skipsCreate() throws IOException {
+        final ContentletIndexOperations ops = mock(ContentletIndexOperations.class);
+        final IndexAPI providerApi = mock(IndexAPI.class);
+        final MappingHelper helper = mock(MappingHelper.class);
+
+        when(ops.toPhysicalName(LOGICAL_NAME)).thenReturn(PHYSICAL_NAME);
+        when(providerApi.indexExists(PHYSICAL_NAME)).thenReturn(true);
+
+        final boolean result = newApi()
+                .createContentIndex(LOGICAL_NAME, SHARDS, IndexTag.ES, ops, providerApi, helper);
+
+        assertTrue("Existing (orphaned) index must be reused and reported as available", result);
+        verify(ops, never()).createContentIndex(anyString(), anyInt());
+        verify(helper).addCustomMapping(List.of(LOGICAL_NAME), IndexTag.ES);
+    }
+
+    /**
+     * Given : the physical index does not exist in the target cluster.
+     * When  : createContentIndex() runs and the create succeeds.
+     * Then  : the index is created with the resolved physical name and shard count, the custom
+     *         mapping is applied, and the method returns true.
+     */
+    @Test
+    public void test_indexMissing_createSucceeds_appliesMapping() throws IOException {
+        final ContentletIndexOperations ops = mock(ContentletIndexOperations.class);
+        final IndexAPI providerApi = mock(IndexAPI.class);
+        final MappingHelper helper = mock(MappingHelper.class);
+
+        when(ops.toPhysicalName(LOGICAL_NAME)).thenReturn(PHYSICAL_NAME);
+        when(providerApi.indexExists(PHYSICAL_NAME)).thenReturn(false);
+        when(ops.createContentIndex(PHYSICAL_NAME, SHARDS)).thenReturn(true);
+
+        final boolean result = newApi()
+                .createContentIndex(LOGICAL_NAME, SHARDS, IndexTag.ES, ops, providerApi, helper);
+
+        assertTrue("A successful create must be reported as available", result);
+        verify(ops).createContentIndex(PHYSICAL_NAME, SHARDS);
+        verify(helper).addCustomMapping(List.of(LOGICAL_NAME), IndexTag.ES);
+    }
+
+    /**
+     * Given : the physical index does not exist.
+     * When  : createContentIndex() runs and the create fails (returns false).
+     * Then  : no mapping is applied and the method returns false — the failure is not masked.
+     */
+    @Test
+    public void test_indexMissing_createFails_noMapping() throws IOException {
+        final ContentletIndexOperations ops = mock(ContentletIndexOperations.class);
+        final IndexAPI providerApi = mock(IndexAPI.class);
+        final MappingHelper helper = mock(MappingHelper.class);
+
+        when(ops.toPhysicalName(LOGICAL_NAME)).thenReturn(PHYSICAL_NAME);
+        when(providerApi.indexExists(PHYSICAL_NAME)).thenReturn(false);
+        when(ops.createContentIndex(PHYSICAL_NAME, SHARDS)).thenReturn(false);
+
+        final boolean result = newApi()
+                .createContentIndex(LOGICAL_NAME, SHARDS, IndexTag.ES, ops, providerApi, helper);
+
+        assertFalse("A failed create must not be reported as available", result);
+        verify(helper, never()).addCustomMapping(List.of(LOGICAL_NAME), IndexTag.ES);
+    }
+
+    /**
+     * Given : the existence probe itself fails (e.g. a transient cluster error).
+     * When  : createContentIndex() runs.
+     * Then  : the probe failure is swallowed and treated as "does not exist", so bootstrap falls
+     *         through to the create path rather than aborting. Here the subsequent create
+     *         succeeds, so the index is created and mapped.
+     */
+    @Test
+    public void test_existenceProbeThrows_treatedAsMissing_proceedsToCreate() throws IOException {
+        final ContentletIndexOperations ops = mock(ContentletIndexOperations.class);
+        final IndexAPI providerApi = mock(IndexAPI.class);
+        final MappingHelper helper = mock(MappingHelper.class);
+
+        when(ops.toPhysicalName(LOGICAL_NAME)).thenReturn(PHYSICAL_NAME);
+        when(providerApi.indexExists(PHYSICAL_NAME))
+                .thenThrow(new RuntimeException("cluster unreachable"));
+        when(ops.createContentIndex(PHYSICAL_NAME, SHARDS)).thenReturn(true);
+
+        final boolean result = newApi()
+                .createContentIndex(LOGICAL_NAME, SHARDS, IndexTag.ES, ops, providerApi, helper);
+
+        assertTrue("A failed existence probe must fall through to a (successful) create", result);
+        verify(ops).createContentIndex(PHYSICAL_NAME, SHARDS);
+        verify(helper).addCustomMapping(List.of(LOGICAL_NAME), IndexTag.ES);
+    }
+
+    /**
+     * Given : an OS-tagged bootstrap of an already-existing index.
+     * When  : createContentIndex() runs with {@link IndexTag#OS}.
+     * Then  : the mapping is re-asserted against the OS provider — the tag is propagated unchanged
+     *         to the mapping helper so the correct vendor is targeted.
+     */
+    @Test
+    public void test_osTag_isPropagatedToMappingHelper() throws IOException {
+        final ContentletIndexOperations ops = mock(ContentletIndexOperations.class);
+        final IndexAPI providerApi = mock(IndexAPI.class);
+        final MappingHelper helper = mock(MappingHelper.class);
+
+        final String osPhysical = PHYSICAL_NAME + ".os";
+        when(ops.toPhysicalName(LOGICAL_NAME)).thenReturn(osPhysical);
+        when(providerApi.indexExists(osPhysical)).thenReturn(true);
+
+        final boolean result = newApi()
+                .createContentIndex(LOGICAL_NAME, SHARDS, IndexTag.OS, ops, providerApi, helper);
+
+        assertTrue(result);
+        verify(helper).addCustomMapping(List.of(LOGICAL_NAME), IndexTag.OS);
+    }
+}