fix(opensearch): fail loudly on migration init failure + surface ES shard failures (#36237) (#36240)

## Proposed Changes

The two diagnostics/robustness follow-ups tracked in #36237 (the
bootstrap-idempotency fix is #36238). Both target the same incident
class where a migration startup problem silently degraded into a
confusing downstream symptom.

### 1. `ContentletIndexAPIImpl.checkAndInitializeIndex()` — fail loudly
during migration

The catch-all swallowed **every** bootstrap exception — including the
Phase 3 `DotRuntimeException` that is thrown a few lines above
specifically to "abort loudly". The instance then came up
**half-initialised**, and the read provider queried an
unfinished/missing index, surfacing downstream as `all shards failed`.

Now, when a migration is active (`isMigrationStarted() ||
isReadEnabled() || isMigrationComplete()`), the failure is re-thrown to
abort startup so the operator restores connectivity/config and restarts.
**Legacy ES-only installs (migration not started) keep the historical
best-effort boot** — no behavior change for non-migration deployments.

### 2. `ContentFactoryIndexOperationsES` — surface the per-shard root
cause

Both the search and count catch blocks logged only
`e.getCause().getMessage()`, hiding the actionable detail behind the
generic `search_phase_execution_exception / all shards failed` wrapper.

Added `describeEsFailure(ElasticsearchException)` which renders:
- `getDetailedMessage()` (full ES type/reason chain),
- the per-shard `ShardSearchFailure` reasons (capped at 5, with an "...
and N more" summary),
- any suppressed server-side causes.

The enriched message still contains the substrings
`shouldQueryCache(...)` keys off, so the error-caching behavior is
unchanged.

## Testing

- `./mvnw compile -pl :dotcms-core` (JDK 25) — compiles clean.
- No public/interface signatures changed.

## Relationship to other PRs

- #36238 — makes index bootstrap idempotent (removes the most common
trigger of the swallowed failure). This PR is the defense-in-depth +
observability layer on top; the two are independent and can merge in any
order.

Part of #36237 (does not close it — the parent issue is also addressed
by #36238).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: fabrizzio-dotCMS

dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentFactoryIndexOperationsES.java

@@ -35,6 +35,7 @@
 import org.elasticsearch.action.search.SearchPhaseExecutionException;
 import org.elasticsearch.action.search.SearchRequest;
 import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.action.search.ShardSearchFailure;
 import org.elasticsearch.client.RequestOptions;
 import org.elasticsearch.client.core.CountRequest;
 import org.elasticsearch.client.core.CountResponse;
@@ -83,6 +84,48 @@ private boolean shouldQueryCache(final String exceptionMsg) {
                 exception.contains("search_phase_execution_exception");
     }
 
+    /** Cap on per-shard failures rendered in a log line; the rest are summarised as a count. */
+    private static final int MAX_SHARD_FAILURES_LOGGED = 5;
+
+    /**
+     * Builds a diagnostic message that surfaces the <em>root</em> cause of an ES search/count
+     * failure rather than the generic wrapper. For {@code search_phase_execution_exception}
+     * ("all shards failed") the actionable detail lives in the per-shard failures and the
+     * suppressed server-side exceptions — {@code getMessage()}/{@code getCause()} alone hide it,
+     * which previously made these failures undiagnosable from the logs.
+     *
+     * @param e the ES exception caught from a search or count request
+     * @return a multi-line message including the detailed message, per-shard reasons (capped),
+     *         and any suppressed causes
+     */
+    private static String describeEsFailure(final ElasticsearchException e) {
+        final StringBuilder sb = new StringBuilder(e.getDetailedMessage());
+
+        if (e instanceof SearchPhaseExecutionException) {
+            final ShardSearchFailure[] failures = ((SearchPhaseExecutionException) e).shardFailures();
+            if (failures != null && failures.length > 0) {
+                final int shown = Math.min(failures.length, MAX_SHARD_FAILURES_LOGGED);
+                for (int i = 0; i < shown; i++) {
+                    sb.append('\n')
+                      .append("  shard[").append(i).append("]: ").append(failures[i].reason());
+                }
+                if (failures.length > shown) {
+                    sb.append('\n')
+                      .append("  ... and ").append(failures.length - shown).append(" more shard failure(s)");
+                }
+            }
+        }
+
+        for (final Throwable suppressed : e.getSuppressed()) {
+            // A suppressed Throwable may carry a null message (e.g. a bare NPE); fall back to
+            // its toString() so the line never renders the literal "null".
+            final String suppressedMsg = suppressed.getMessage();
+            sb.append('\n').append("  caused by: ")
+              .append(null != suppressedMsg ? suppressedMsg : suppressed.toString());
+        }
+        return sb.toString();
+    }
+
     /**
      * if enabled SearchRequests are executed and then cached
      * @param searchRequest
@@ -105,7 +148,7 @@ private SearchHits cachedIndexSearch(final SearchRequest searchRequest) {
             }
             return hits;
         } catch (final ElasticsearchStatusException | IndexNotFoundException | SearchPhaseExecutionException e) {
-            final String exceptionMsg = (null != e.getCause() ? e.getCause().getMessage() : e.getMessage());
+            final String exceptionMsg = describeEsFailure(e);
             Logger.warn(this.getClass(), "----------------------------------------------");
             Logger.warn(this.getClass(), String.format("Elasticsearch SEARCH error in index '%s'", (searchRequest.indices()!=null) ? String.join(",", searchRequest.indices()): "unknown"));
             Logger.warn(this.getClass(), String.format("Thread: %s", Thread.currentThread().getName() ));
@@ -171,7 +214,7 @@ public Long cachedIndexCount(final CountRequest countRequest) {
             }
             return count;
         } catch (final ElasticsearchStatusException | IndexNotFoundException | SearchPhaseExecutionException e) {
-            final String exceptionMsg = (null != e.getCause() ? e.getCause().getMessage() : e.getMessage());
+            final String exceptionMsg = describeEsFailure(e);
             Logger.warn(this.getClass(), "----------------------------------------------");
             Logger.warn(this.getClass(), String.format("Elasticsearch error in index '%s'", (countRequest.indices()!=null) ? String.join(",", countRequest.indices()): "unknown"));
             Logger.warn(this.getClass(), String.format("ES Query: %s", String.valueOf(countRequest.source()) ));

dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java

@@ -521,7 +521,7 @@ private boolean indexReadyOS() {
     @CloseDBIfOpened
     public synchronized void checkAndInitializeIndex() {
         try {
-            if (isMigrationStarted() || isReadEnabled() || isMigrationComplete()) {
+            if (isMigrationStarted()) {
                 if (!IndexStartupValidator.validateIndexingConfig()) {
                     if (isMigrationComplete()) {
                         // Phase 3: ES is decommissioned. Rolling back to Phase 0 would route
@@ -554,6 +554,18 @@ public synchronized void checkAndInitializeIndex() {
         } catch (Exception e) {
             Logger.fatal(this.getClass(), "Failed to create new indexes:" + e.getMessage(),e);
 
+            // During an active OpenSearch migration a half-initialised index store is worse
+            // than a hard stop: the read provider would serve an unfinished or missing index
+            // (surfacing downstream as "all shards failed"), and a Phase 3 validation failure
+            // intentionally throws above (line ~530) only to be swallowed here. Fail loudly so
+            // the operator restores connectivity/config and restarts, rather than silently
+            // coming up broken. Legacy ES-only installs (migration not started) keep the
+            // historical best-effort behaviour and still boot.
+            if (isMigrationStarted()) {
+                throw new DotRuntimeException(
+                        "Index initialization failed during OpenSearch migration; aborting startup"
+                        + " to avoid serving an unfinished index store. Cause: " + e.getMessage(), e);
+            }
         }
     }