Parquet-back raw-PUT test indices on the analytics-engine route

With -Dtests.analytics.parquet_indices=true, indices created by a raw
document PUT (e.g. `PUT /test/_doc/1` in a test's init()) bypass
AnalyticsIndexConfig.applyIndexCreationSettings, so they inherit the
composite *value* — and are therefore routed to the analytics engine by
RestUnifiedQueryAction.isAnalyticsIndex — but not the
`pluggable.dataformat.enabled` flag. They are then stored as a
plain-Lucene EngineBackedIndexer whose acquireReader() is unimplemented,
and the query fails with
`StreamException[INTERNAL] Failed to start streaming fragment`.

Apply the cluster-level composite defaults in setUpIndices() so every
index — including raw-PUT ones — is stored as a parquet-backed
DataFormatAwareEngine that is actually scannable by the analytics engine
it routes to. No-op unless tests.analytics.parquet_indices=true, so
normal CI is unchanged.

Signed-off-by: Kai Huang <ahkcs@amazon.com>

Author: ahkcs

integ-test/src/test/java/org/opensearch/sql/legacy/SQLIntegTestCase.java

@@ -66,6 +66,12 @@ public void setUpIndices() throws Exception {
       initClient();
     }
 
+    // When -Dtests.analytics.parquet_indices=true, make every index (including ones a test
+    // auto-creates via a raw document PUT, which bypasses createIndexByRestClient) parquet-backed
+    // composite, so it is stored as a DataFormatAwareEngine and is actually scannable by the
+    // analytics engine it routes to. Must run before init() creates any index.
+    TestUtils.AnalyticsIndexConfig.applyClusterSettings(client());
+
     if (shouldResetQuerySizeLimit()) {
       resetQuerySizeLimit();
     }

integ-test/src/test/java/org/opensearch/sql/legacy/TestUtils.java

@@ -89,6 +89,42 @@ static void applyIndexCreationSettings(JSONObject jsonObject) {
       jsonObject.put("settings", settings);
     }
 
+    /**
+     * Apply the cluster-level composite defaults so EVERY index — including ones auto-created by a
+     * raw document {@code PUT} that bypasses {@link #applyIndexCreationSettings} (e.g. {@code PUT
+     * /test/_doc/1} in a test's {@code init()}) — is stored as a parquet-backed composite index
+     * ({@code DataFormatAwareEngine}) and is therefore scannable by the analytics engine.
+     *
+     * <p>Without this, a raw-PUT index inherits the composite <em>value</em> (so {@code
+     * RestUnifiedQueryAction#isAnalyticsIndex} routes it to the analytics engine) but not the
+     * {@code .enabled} flag, so it is stored as a plain-Lucene {@code EngineBackedIndexer} whose
+     * {@code acquireReader()} is unimplemented — surfacing at query time as {@code
+     * StreamException[INTERNAL] "Failed to start streaming fragment"}. Setting the {@code .enabled}
+     * flag (plus the parquet primary / lucene secondary formats) at the cluster level makes the
+     * stored format match the routing decision.
+     *
+     * <p>Existing system indices are created at cluster startup (before this runs) and templated
+     * system indices carry explicit settings, so the cluster default does not retroactively alter
+     * them. The static {@code cluster.pluggable.dataformat.restrict.allowlist} guard is not
+     * dynamically updateable, so it must live in the cluster's node config if a run ever needs to
+     * exempt additional index name prefixes. No-op when disabled; idempotent, so it is safe to
+     * re-apply before each test.
+     */
+    public static void applyClusterSettings(RestClient client) {
+      if (!isEnabled()) {
+        return;
+      }
+      Request request = new Request("PUT", "/_cluster/settings");
+      request.setJsonEntity(
+          "{\"persistent\":{"
+              + "\"cluster.pluggable.dataformat.enabled\":true,"
+              + "\"cluster.pluggable.dataformat\":\"composite\","
+              + "\"cluster.composite.primary_data_format\":\"parquet\","
+              + "\"cluster.composite.secondary_data_formats\":[\"lucene\"]"
+              + "}}");
+      performRequest(client, request);
+    }
+
     /**
      * Returns the {@code _bulk} refresh query string for the current index type. Parquet-backed
      * indices in the analytics-backend-lucene composite engine don't yet implement {@code