Make analytics-engine an optional dependency via AnalyticsFrontEndExtension SPI

End-to-end consumer-side wiring for the AnalyticsFrontEndExtension SPI
(analytics-framework). Lets the SQL plugin install and run on stock
OpenSearch distributions that don't ship analytics-engine, while
preserving the analytics routing behavior when both plugins are
co-installed.

Build wiring (plugin/build.gradle):
- Mark analytics-engine as ;optional=true in extendedPlugins. OpenSearch's
  PluginsService.checkBundleJarHell skips both URL-overlap and
  class-level JarHell against optional deps, so SQL bundles its own
  copies of the previously-stripped jars (Calcite, Guava, etc.) again.
- Drop the entire bundlePlugin exclusion list — the hand-maintained
  jar-deduplication hack goes away with the SPI.
- Vendor the rebuilt analytics-framework-3.7.0-SNAPSHOT.jar (with the
  new AnalyticsFrontEndExtension + AnalyticsServices types) and the
  rebuilt analytics-engine-3.7.0-SNAPSHOT.zip (with the producer-side
  Guice TypeListener that pushes services to consumers).

SPI consumer (SQLAnalyticsFrontEndExtension.java + META-INF/services
registration):
- Implements AnalyticsFrontEndExtension; setAnalyticsServices stashes
  the executor + schemaProvider into AnalyticsExecutorHolder.
- Kept in a class separate from SQLPlugin so SQLPlugin's bytecode
  doesn't reference any analytics-framework class. When AE is absent,
  ServiceLoader never touches this class and SQL boots without
  analytics-framework on its runtime classpath.

Holder isolation (AnalyticsExecutorHolder.java):
- Refactored to store services as Object internally — no
  analytics-framework type appears in any signature loaded at SQL
  plugin startup. Callers cast at use sites that are already gated on
  a non-null value.
- Added schemaProvider alongside the executor.

Lazy resolution at the request boundary:
- TransportPPLQueryAction: drop the @Inject QueryPlanExecutor
  constructor parameter (the cross-plugin Guice dependency that broke
  the optional path). Replace with analyticsHandler() — null-checked
  lazy resolution from the holder; falls through to the legacy PPL
  path when AE is absent.
- SQLPlugin#createSqlAnalyticsRouter: same lazy-supplier pattern
  against the new Object-typed holder + RestUnifiedQueryAction
  factory.
- RestUnifiedQueryAction: add fromUnknownExecutor(Object, Object)
  factory (the only cast site for the analytics-framework types).
  Replaced static OpenSearchSchemaBuilder.buildSchema() call with the
  injected SchemaProvider so the runtime no longer has a hard static
  reference to analytics-engine.

Verified end-to-end with bin/opensearch-plugin install on a real
OpenSearch 3.7 distribution:
- WITH analytics-engine: all three plugins install; Lucene PPL works;
  parquet_* PPL routes through the analytics engine (confirmed via
  /_plugins/_ppl/_explain showing Calcite logical plan).
- WITHOUT analytics-engine: SQL + job-scheduler install (warning
  logged for missing optional AE); Lucene PPL works; parquet_* PPL
  falls through to legacy with a clean IndexNotFoundException — no
  NoClassDefFoundError, no startup crash.

Pairs with the producer-side wiring in opensearch-project/OpenSearch.

Signed-off-by: Kai Huang <ahkcs@amazon.com>

Author: ahkcs

libs/analytics-engine-3.7.0-SNAPSHOT.zip

@@ -55,15 +55,21 @@ opensearchplugin {
     name 'opensearch-sql'
     description 'OpenSearch SQL'
     classname 'org.opensearch.sql.plugin.SQLPlugin'
-    extendedPlugins = ['opensearch-job-scheduler', 'analytics-engine']
+    extendedPlugins = ['opensearch-job-scheduler', 'analytics-engine;optional=true']
     licenseFile rootProject.file("LICENSE.txt")
     noticeFile rootProject.file("NOTICE")
 }
 
-// Exclude jars provided by the analytics-engine plugin (shared via extendedPlugins classloader).
-// MUST match what's actually in the analytics-engine ZIP to avoid both jar hell (when analytics-engine
-// ships it) and ClassNotFoundException at runtime (when it doesn't). Last verified against
-// analytics-engine-3.7.0-SNAPSHOT — keep this list aligned when bumping versions.
+// Strip jars provided by analytics-engine to avoid runtime classloader confusion when both
+// plugins are co-installed (Janino's runtime code generation breaks when two versions of
+// Calcite/Janino are visible). With extendedPlugins ';optional=true', JarHell skips overlap
+// checks but Janino still ends up confused, so we keep this hand-maintained exclusion list as
+// the lesser evil. When analytics-engine is absent, classes that need these jars remain unloaded
+// because the SPI consumer (SQLAnalyticsFrontEndExtension) never fires and analytics-routing code
+// paths stay inert via the holder's null gate.
+//
+// When bumping analytics-engine, re-verify this list against `unzip -l <new analytics-engine zip>`
+// — anything analytics-engine ships at runtime must appear here to prevent dual-version conflicts.
 bundlePlugin {
     exclude 'calcite-core-*.jar'
     exclude 'calcite-linq4j-*.jar'
@@ -81,15 +87,10 @@ bundlePlugin {
     exclude 'jackson-annotations-*.jar'
     exclude 'jackson-databind-*.jar'
     exclude 'httpcore5-5*.jar'
-    // TODO: Remove the three httpcore5/httpclient5 exclusions below — and ideally this entire
-    // bundlePlugin exclusion block — once analytics-engine becomes an optional dependency via the
-    // AnalyticsFrontEndExtension SPI (opensearch-project/OpenSearch#21449). The shared-classloader
-    // jar deduplication that requires this hand-maintained list goes away with the SPI.
     exclude 'httpcore5-h2-*.jar'
     exclude 'httpcore5-reactive-*.jar'
     exclude 'httpclient5-*.jar'
     exclude 'commons-text-*.jar'
-    // Calcite transitive deps now bundled in analytics-engine 3.7 — exclude from sql to avoid jar hell.
     exclude 'commons-math3-*.jar'
     exclude 'commons-dbcp2-*.jar'
     exclude 'commons-pool2-*.jar'

libs/analytics-framework-3.7.0-SNAPSHOT.jar

@@ -0,0 +1,28 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package org.opensearch.sql.plugin;
+
+import org.opensearch.analytics.spi.AnalyticsFrontEndExtension;
+import org.opensearch.analytics.spi.AnalyticsServices;
+import org.opensearch.sql.plugin.rest.AnalyticsExecutorHolder;
+
+/**
+ * SPI consumer that publishes the analytics-engine services into {@link AnalyticsExecutorHolder}
+ * when analytics-engine is installed.
+ *
+ * <p>Kept separate from {@link SQLPlugin} so that SQLPlugin's bytecode does not reference any
+ * analytics-framework class. When analytics-engine is absent, OpenSearch never invokes {@code
+ * ServiceLoader.load(AnalyticsFrontEndExtension.class)} (because no plugin defining the SPI is
+ * present), so this class is never resolved and SQL boots without needing analytics-framework on
+ * its runtime classpath.
+ */
+public class SQLAnalyticsFrontEndExtension implements AnalyticsFrontEndExtension {
+
+  @Override
+  public void setAnalyticsServices(AnalyticsServices services) {
+    AnalyticsExecutorHolder.set(services.queryPlanExecutor(), services.schemaProvider());
+  }
+}

plugin/build.gradle

@@ -204,11 +204,14 @@ private BiFunction<SQLQueryRequest, RestChannel, Boolean> createSqlAnalyticsRout
     java.util.function.Supplier<RestUnifiedQueryAction> handlerSupplier =
         () -> {
           if (cached[0] == null) {
-            var executor = AnalyticsExecutorHolder.get();
-            if (executor == null) {
+            Object executor = AnalyticsExecutorHolder.getQueryPlanExecutor();
+            Object schemaProvider = AnalyticsExecutorHolder.getSchemaProvider();
+            if (executor == null || schemaProvider == null) {
               return null;
             }
-            cached[0] = new RestUnifiedQueryAction(client, clusterService, executor);
+            cached[0] =
+                RestUnifiedQueryAction.fromUnknownExecutor(
+                    client, clusterService, executor, schemaProvider);
           }
           return cached[0];
         };

plugin/src/main/java/org/opensearch/sql/plugin/SQLAnalyticsFrontEndExtension.java

@@ -5,32 +5,48 @@
 
 package org.opensearch.sql.plugin.rest;
 
-import org.apache.calcite.rel.RelNode;
-import org.opensearch.analytics.exec.QueryPlanExecutor;
-
 /**
- * Bridge for sharing the analytics-engine {@link QueryPlanExecutor} between the PPL transport
- * action (where Guice resolves the binding via {@code @Inject}) and the REST-only SQL router (where
- * Guice cannot, because {@code SQLPlugin#getRestHandlers} runs before the Node-level injector
- * satisfies {@code @Inject} parameters).
+ * Static bridge that lets {@code SQLAnalyticsFrontEndExtension} (the SPI consumer) hand off the
+ * analytics-engine services to the SQL request paths that need them.
  *
- * <p>Why a static holder: cross-plugin Guice injection needs a class registered in the Node
- * injector, and {@link org.opensearch.sql.plugin.SQLPlugin}'s SQL routing path is built in {@code
- * getRestHandlers} — outside any Guice-managed lifecycle. Persisting the executor in this holder
- * once {@link org.opensearch.sql.plugin.transport.TransportPPLQueryAction} is constructed lets the
- * SQL router read the same instance without going back through the injector.
+ * <p>Stored as {@link Object} on purpose. Callers cast at use sites that are already gated on a
+ * non-null value, ensuring no analytics-framework class is referenced from any signature loaded at
+ * SQL plugin startup. When analytics-engine is not installed, the SPI never fires, the holder stays
+ * null, and {@code TransportPPLQueryAction} / {@code SQLPlugin#createSqlAnalyticsRouter} fall
+ * through to the legacy paths without ever touching analytics-framework types.
  */
 public final class AnalyticsExecutorHolder {
 
-  private static volatile QueryPlanExecutor<RelNode, Iterable<Object[]>> executor;
+  private static volatile Object queryPlanExecutor;
+  private static volatile Object schemaProvider;
 
   private AnalyticsExecutorHolder() {}
 
-  public static void set(QueryPlanExecutor<RelNode, Iterable<Object[]>> instance) {
-    executor = instance;
+  /**
+   * Set both services in one call. Invoked by {@code SQLAnalyticsFrontEndExtension} from the SPI
+   * push lifecycle. Either argument may be {@code null} (not expected today, but tolerated).
+   */
+  public static void set(Object queryPlanExecutor, Object schemaProvider) {
+    AnalyticsExecutorHolder.queryPlanExecutor = queryPlanExecutor;
+    AnalyticsExecutorHolder.schemaProvider = schemaProvider;
+  }
+
+  /**
+   * Returns the analytics-engine query plan executor as {@link Object}. Returns {@code null} when
+   * analytics-engine is not installed (SPI never fired). Callers cast to {@code
+   * QueryPlanExecutor<RelNode, Iterable<Object[]>>} only inside code paths that are gated on a
+   * non-null return value — see {@code RestUnifiedQueryAction#fromUnknownExecutor}.
+   */
+  public static Object getQueryPlanExecutor() {
+    return queryPlanExecutor;
   }
 
-  public static QueryPlanExecutor<RelNode, Iterable<Object[]>> get() {
-    return executor;
+  /**
+   * Returns the analytics-engine schema provider as {@link Object}. Returns {@code null} when
+   * analytics-engine is not installed. Callers cast to {@code SchemaProvider} only inside code
+   * paths that are gated on a non-null return value.
+   */
+  public static Object getSchemaProvider() {
+    return schemaProvider;
   }
 }

plugin/src/main/java/org/opensearch/sql/plugin/SQLPlugin.java

@@ -23,7 +23,7 @@
 import org.apache.logging.log4j.Logger;
 import org.apache.logging.log4j.ThreadContext;
 import org.opensearch.analytics.exec.QueryPlanExecutor;
-import org.opensearch.analytics.schema.OpenSearchSchemaBuilder;
+import org.opensearch.analytics.schema.SchemaProvider;
 import org.opensearch.cluster.service.ClusterService;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.core.action.ActionListener;
@@ -59,14 +59,33 @@ public class RestUnifiedQueryAction {
   private final AnalyticsExecutionEngine analyticsEngine;
   private final NodeClient client;
   private final ClusterService clusterService;
+  private final SchemaProvider schemaProvider;
 
   public RestUnifiedQueryAction(
       NodeClient client,
       ClusterService clusterService,
-      QueryPlanExecutor<RelNode, Iterable<Object[]>> planExecutor) {
+      QueryPlanExecutor<RelNode, Iterable<Object[]>> planExecutor,
+      SchemaProvider schemaProvider) {
     this.client = client;
     this.clusterService = clusterService;
     this.analyticsEngine = new AnalyticsExecutionEngine(planExecutor);
+    this.schemaProvider = schemaProvider;
+  }
+
+  /**
+   * Construct from holder-stashed services typed as {@link Object} so callers don't take a
+   * compile-time reference on analytics-framework. The cast is confined to this method — invoking
+   * it loads the analytics-framework types for the first time, and the caller must gate on a
+   * non-null executor (i.e. analytics-engine is installed).
+   */
+  @SuppressWarnings("unchecked")
+  public static RestUnifiedQueryAction fromUnknownExecutor(
+      NodeClient client, ClusterService clusterService, Object executor, Object schemaProvider) {
+    return new RestUnifiedQueryAction(
+        client,
+        clusterService,
+        (QueryPlanExecutor<RelNode, Iterable<Object[]>>) executor,
+        (SchemaProvider) schemaProvider);
   }
 
   /**
@@ -161,7 +180,7 @@ private static UnifiedQueryContext buildParsingContext(QueryType queryType) {
   private UnifiedQueryContext buildContext(QueryType queryType, boolean profiling) {
     return UnifiedQueryContext.builder()
         .language(queryType)
-        .catalog(SCHEMA_NAME, OpenSearchSchemaBuilder.buildSchema(clusterService.state()))
+        .catalog(SCHEMA_NAME, schemaProvider.buildSchema(clusterService.state()))
         .defaultNamespace(SCHEMA_NAME)
         .profiling(profiling)
         .build();

plugin/src/main/java/org/opensearch/sql/plugin/rest/AnalyticsExecutorHolder.java

@@ -13,11 +13,9 @@
 import java.util.Locale;
 import java.util.Optional;
 import java.util.function.Supplier;
-import org.apache.calcite.rel.RelNode;
 import org.opensearch.action.ActionRequest;
 import org.opensearch.action.support.ActionFilters;
 import org.opensearch.action.support.HandledTransportAction;
-import org.opensearch.analytics.exec.QueryPlanExecutor;
 import org.opensearch.cluster.service.ClusterService;
 import org.opensearch.common.inject.Guice;
 import org.opensearch.common.inject.Inject;
@@ -62,7 +60,16 @@ public class TransportPPLQueryAction
 
   private final Supplier<Boolean> pplEnabled;
 
-  private final RestUnifiedQueryAction unifiedQueryHandler;
+  private final NodeClient client;
+
+  private final ClusterService clusterService;
+
+  /**
+   * Lazily-resolved analytics handler. {@code null} until the first analytics-routed request, and
+   * remains {@code null} forever if analytics-engine is not installed (the SPI never fires, so
+   * {@link AnalyticsExecutorHolder#getQueryPlanExecutor()} stays null).
+   */
+  private volatile RestUnifiedQueryAction unifiedQueryHandler;
 
   /** Constructor of TransportPPLQueryAction. */
   @Inject
@@ -72,10 +79,12 @@ public TransportPPLQueryAction(
       NodeClient client,
       ClusterService clusterService,
       DataSourceServiceImpl dataSourceService,
-      org.opensearch.common.settings.Settings clusterSettings,
-      QueryPlanExecutor<RelNode, Iterable<Object[]>> queryPlanExecutor) {
+      org.opensearch.common.settings.Settings clusterSettings) {
     super(PPLQueryAction.NAME, transportService, actionFilters, TransportPPLQueryRequest::new);
 
+    this.client = client;
+    this.clusterService = clusterService;
+
     ModulesBuilder modules = new ModulesBuilder();
     modules.add(new OpenSearchPluginModule());
     modules.add(
@@ -86,9 +95,6 @@ public TransportPPLQueryAction(
           b.bind(DataSourceService.class).toInstance(dataSourceService);
         });
     this.injector = Guice.createInjector(modules);
-    AnalyticsExecutorHolder.set(queryPlanExecutor);
-    this.unifiedQueryHandler =
-        new RestUnifiedQueryAction(client, clusterService, queryPlanExecutor);
     this.pplEnabled =
         () ->
             MULTI_ALLOW_EXPLICIT_INDEX.get(clusterSettings)
@@ -98,6 +104,31 @@ public TransportPPLQueryAction(
                         .getSettingValue(Settings.Key.PPL_ENABLED);
   }
 
+  /**
+   * Resolves the analytics-engine-backed handler lazily. Returns {@code null} when analytics-engine
+   * is not installed; callers fall through to the legacy PPL pipeline. Synchronized double-checked
+   * cache so we only build the handler once on the first analytics request.
+   */
+  private RestUnifiedQueryAction analyticsHandler() {
+    RestUnifiedQueryAction cached = unifiedQueryHandler;
+    if (cached != null) {
+      return cached;
+    }
+    Object executor = AnalyticsExecutorHolder.getQueryPlanExecutor();
+    Object schemaProvider = AnalyticsExecutorHolder.getSchemaProvider();
+    if (executor == null || schemaProvider == null) {
+      return null;
+    }
+    synchronized (this) {
+      if (unifiedQueryHandler == null) {
+        unifiedQueryHandler =
+            RestUnifiedQueryAction.fromUnknownExecutor(
+                client, clusterService, executor, schemaProvider);
+      }
+      return unifiedQueryHandler;
+    }
+  }
+
   /**
    * {@inheritDoc} Transform the request and call super.doExecute() to support call from other
    * plugins.
@@ -134,16 +165,20 @@ protected void doExecute(
     QueryContext.setProfile(transformedRequest.profile());
     ActionListener<TransportPPLQueryResponse> clearingListener = wrapWithProfilingClear(listener);
 
-    // Route to analytics engine for non-Lucene (e.g., Parquet-backed) indices
-    if (unifiedQueryHandler.isAnalyticsIndex(transformedRequest.getRequest(), QueryType.PPL)) {
+    // Route to analytics engine for non-Lucene (e.g., Parquet-backed) indices.
+    // analyticsHandler() returns null when analytics-engine isn't installed — we fall through
+    // to the regular Lucene PPL path so non-analytics queries still work.
+    RestUnifiedQueryAction analytics = analyticsHandler();
+    if (analytics != null
+        && analytics.isAnalyticsIndex(transformedRequest.getRequest(), QueryType.PPL)) {
       if (transformedRequest.isExplainRequest()) {
-        unifiedQueryHandler.explain(
+        analytics.explain(
             transformedRequest.getRequest(),
             QueryType.PPL,
             transformedRequest.mode(),
             createExplainResponseListener(transformedRequest, clearingListener));
       } else {
-        unifiedQueryHandler.execute(
+        analytics.execute(
             transformedRequest.getRequest(),
             QueryType.PPL,
             transformedRequest.profile(),

plugin/src/main/java/org/opensearch/sql/plugin/rest/RestUnifiedQueryAction.java

@@ -0,0 +1,6 @@
+#
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+#
+
+org.opensearch.sql.plugin.SQLAnalyticsFrontEndExtension

plugin/src/main/java/org/opensearch/sql/plugin/transport/TransportPPLQueryAction.java

@@ -13,6 +13,7 @@
 import org.junit.Before;
 import org.junit.Test;
 import org.opensearch.analytics.exec.QueryPlanExecutor;
+import org.opensearch.analytics.schema.SchemaProvider;
 import org.opensearch.cluster.service.ClusterService;
 import org.opensearch.sql.executor.QueryType;
 import org.opensearch.transport.client.node.NodeClient;
@@ -30,7 +31,11 @@ public void setUp() {
     @SuppressWarnings("unchecked")
     QueryPlanExecutor<RelNode, Iterable<Object[]>> executor = mock(QueryPlanExecutor.class);
     action =
-        new RestUnifiedQueryAction(mock(NodeClient.class), mock(ClusterService.class), executor);
+        new RestUnifiedQueryAction(
+            mock(NodeClient.class),
+            mock(ClusterService.class),
+            executor,
+            mock(SchemaProvider.class));
   }
 
   @Test