Commit 24dd79c
authored
Wire analytics-engine as extendedPlugins dependency (#5302)
* [Mustang] Wire analytics-engine as extendedPlugins dependency (#5247)
Step 1: Plugin wiring and dependency integration.
- Add analytics-engine as extendedPlugins in plugin/build.gradle
- Vendor analytics-framework JAR (interfaces) and analytics-engine
ZIP (plugin) built from OpenSearch sandbox/3.6 branch
- Delete local QueryPlanExecutor interface, use upstream
org.opensearch.analytics.exec.QueryPlanExecutor from JAR
- Replace StubSchemaProvider with OpenSearchSchemaBuilder which reads
real index mappings from ClusterState
- Delete StubSchemaProvider (no longer needed)
- Exclude shared JARs (Calcite, Guava, commons-*, etc.) from SQL
plugin bundle to avoid jar hell with analytics-engine classloader
- Load analytics-engine plugin in integTest and remoteCluster test
clusters before opensearch-sql-plugin
- Create parquet_logs and parquet_metrics indices in ITs so
OpenSearchSchemaBuilder can resolve the schema
- Update explain expected files for alphabetical field ordering
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Add analytics-engine plugin to all test clusters
Every test cluster that loads opensearch-sql-plugin needs the
analytics-engine plugin because SQL declares it as extendedPlugins.
Added to yamlRestTest, integTestWithSecurity, remoteIntegTestWithSecurity,
and integJdbcTest clusters.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Add analytics-engine plugin to doctest test cluster
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Add commons-text to analytics-engine ZIP and fix isAnalyticsIndex NPE
commons-text is needed by Calcite (parent classloader) for fuzzy matching
but was only in the SQL plugin (child classloader). Also use lightweight
parsing context in isAnalyticsIndex to avoid requiring cluster state.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Fix Janino classloader issue when analytics-engine is parent plugin
Calcite's EnumerableInterpretable.getBindable() hardcodes
EnumerableInterpretable.class.getClassLoader() for Janino compilation.
When analytics-engine is the parent classloader via extendedPlugins,
this returns the parent classloader which cannot see SQL plugin classes,
causing CompileException for any Enumerable code generation.
Override implement() in OpenSearchCalcitePreparingStmt to use our own
compileWithPluginClassLoader() which does the same code generation but
uses CalciteToolsHelper.class.getClassLoader() (SQL plugin's child
classloader) so Janino can resolve both parent and child classes.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Reverse classloader: make SQL plugin parent of analytics-engine
The previous approach (analytics-engine as parent) caused Janino
classloader issues in 4 Calcite code paths. Reversing the relationship
makes SQL plugin the parent, so Janino uses the SQL plugin classloader
which can see both Calcite and SQL plugin classes.
Changes:
- Remove extendedPlugins=['analytics-engine'] from SQL plugin
- Add ExtensiblePlugin interface to SQLPlugin
- Rebuild analytics-engine ZIP with extended.plugins=opensearch-sql
and without Calcite JARs (inherited from parent)
- Move OpenSearchSchemaBuilder to analytics-framework JAR
- Change analytics-framework from compileOnly to api in core
- Reverse test cluster plugin load order (SQL first)
- Revert all Janino classloader fixes (no longer needed)
- Add classloader architecture options doc
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Remove classloader options doc from PR
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Remove unnecessary Janino classloader workarounds
With SQL plugin as parent (Option C), EnumerableInterpretable and
RexExecutable use the SQL plugin classloader which sees all classes.
The implementEnumerable(), compileWithPluginClassLoader(), and
PluginClassLoaderRexExecutor workarounds are no longer needed.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Revert "Remove unnecessary Janino classloader workarounds"
This reverts commit 23e0d13.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Revert "Remove classloader options doc from PR"
This reverts commit 0ad18a8.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Revert "Reverse classloader: make SQL plugin parent of analytics-engine"
This reverts commit be27381.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Revert "Fix Janino classloader issue when analytics-engine is parent plugin"
This reverts commit 2c4c401.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Patch Calcite CALCITE-3745: use thread context classloader for Janino
Calcite hardcodes EnumerableInterpretable.class.getClassLoader() and
RexExecutable.class.getClassLoader() for Janino compilation. When
analytics-engine is the parent classloader, this returns the parent
which cannot see SQL plugin classes, causing CompileException.
Patch calcite-core to prefer Thread.currentThread().getContextClassLoader()
(CALCITE-3745), and set the context classloader to the SQL plugin's
classloader before Calcite execution in two places:
- CalciteToolsHelper.OpenSearchRelRunners.run()
- CalciteScriptEngine.compile()
This keeps analytics-engine as the natural parent while fixing all
Janino classloader issues with a minimal, well-understood patch.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Fix CalciteExplainIT boolean string literal explain plan difference
With the patched Calcite classloader, SAFE_CAST('TRUE') is no longer
silently folded to a boolean literal in the logical plan. Split the
expected YAML so boolean literal tests (male = true) and string literal
tests (male = 'TRUE') have separate expected outputs. The physical plan
(term query pushdown) is identical in both cases.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Add setContextClassLoader wrapper to QueryService Calcite paths
The analyze() and convertToCalcitePlan() steps in executeWithCalcite()
and explainWithCalcite() trigger RexSimplify which uses Janino for
constant folding. Without the context classloader set, UDF expressions
like SAFE_CAST('TRUE') can't be folded, causing explain plan differences
vs main branch. Adding the wrapper here covers the entire Calcite
lifecycle for both execute and explain paths.
Revert the separate boolean string literal YAML since all three boolean
tests now produce the same plan (SAFE_CAST folds correctly).
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Refactor context classloader wrapping into CalciteClassLoaderHelper
Extract the CALCITE-3745 context classloader fix into a dedicated
CalciteClassLoaderHelper utility class. Remove redundant wrapper from
OpenSearchRelRunners.run() (already covered by QueryService callers).
Three call sites remain:
- QueryService.executeWithCalcite() — covers planning + execution
- QueryService.explainWithCalcite() — covers planning + explain
- CalciteScriptEngine.compile() — covers pushdown scripts (shard thread)
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Revert CalciteToolsHelper comment to sync with main
Signed-off-by: Kai Huang <ahkcs@amazon.com>
---------
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>1 parent 151cffc commit 24dd79c
25 files changed
Lines changed: 253 additions & 142 deletions
File tree
- core
- src
- main/java/org/opensearch/sql
- calcite/utils
- executor
- analytics
- test/java/org/opensearch/sql/executor/analytics
- doctest
- integ-test
- src/test
- java/org/opensearch/sql/ppl
- resources/expectedOutput/analytics
- libs
- opensearch/src/main/java/org/opensearch/sql/opensearch/storage/script
- plugin
- src
- main/java/org/opensearch/sql/plugin
- rest
- analytics/stub
- transport
- test/java/org/opensearch/sql/plugin/rest
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
| 67 | + | |
66 | 68 | | |
67 | 69 | | |
68 | 70 | | |
| |||
Lines changed: 60 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
Lines changed: 26 additions & 17 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
139 | 140 | | |
140 | 141 | | |
141 | 142 | | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
150 | 155 | | |
151 | 156 | | |
152 | 157 | | |
| |||
169 | 174 | | |
170 | 175 | | |
171 | 176 | | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
| 177 | + | |
177 | 178 | | |
178 | | - | |
179 | | - | |
180 | | - | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
181 | 190 | | |
182 | | - | |
| 191 | + | |
183 | 192 | | |
184 | 193 | | |
185 | 194 | | |
| |||
Lines changed: 3 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
41 | | - | |
| 42 | + | |
42 | 43 | | |
43 | | - | |
| 44 | + | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
| |||
Lines changed: 0 additions & 32 deletions
This file was deleted.
Lines changed: 6 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
38 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
39 | 43 | | |
40 | 44 | | |
41 | 45 | | |
42 | 46 | | |
43 | | - | |
| 47 | + | |
44 | 48 | | |
45 | 49 | | |
46 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
195 | 195 | | |
196 | 196 | | |
197 | 197 | | |
| 198 | + | |
198 | 199 | | |
199 | 200 | | |
200 | 201 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
270 | 270 | | |
271 | 271 | | |
272 | 272 | | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
273 | 277 | | |
274 | 278 | | |
275 | 279 | | |
276 | 280 | | |
277 | 281 | | |
| 282 | + | |
278 | 283 | | |
279 | 284 | | |
280 | 285 | | |
281 | 286 | | |
282 | 287 | | |
283 | 288 | | |
284 | 289 | | |
| 290 | + | |
285 | 291 | | |
286 | 292 | | |
287 | 293 | | |
288 | 294 | | |
289 | 295 | | |
290 | 296 | | |
291 | 297 | | |
| 298 | + | |
292 | 299 | | |
293 | 300 | | |
294 | 301 | | |
295 | 302 | | |
296 | 303 | | |
| 304 | + | |
297 | 305 | | |
298 | 306 | | |
299 | 307 | | |
300 | 308 | | |
301 | 309 | | |
| 310 | + | |
302 | 311 | | |
303 | 312 | | |
304 | 313 | | |
| |||
352 | 361 | | |
353 | 362 | | |
354 | 363 | | |
355 | | - | |
| 364 | + | |
| 365 | + | |
356 | 366 | | |
| 367 | + | |
357 | 368 | | |
358 | 369 | | |
359 | 370 | | |
| |||
Lines changed: 18 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| 13 | + | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
| |||
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
28 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
29 | 46 | | |
30 | 47 | | |
31 | 48 | | |
| |||
Lines changed: 43 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
43 | | - | |
44 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
45 | 86 | | |
46 | 87 | | |
47 | 88 | | |
| |||
0 commit comments