Describe the bug
After upgrading from OpenSearch 3.1 to 3.6.0, PPL queries containing Chinese string literals fail with a CalciteException. The equivalent SQL query works perfectly on the same 3.6.0 cluster, and the identical PPL query worked without issues on 3.1. This is a confirmed regression in the PPL/Calcite integration layer regarding non-ASCII character encoding handling
Related component
Search:Query Capabilities
To Reproduce
- Ensure an index exists with documents containing Chinese field values (e.g., status: "未处置")
- Execute the following PPL query
source=your_index | where status = '未处置'
- Observe the CalciteException error
- Execute the equivalent SQL query to confirm it works:
SELECT * FROM your_index WHERE status = '未处置'
Expected behavior
The PPL query should execute successfully and return matching documents, consistent with the behavior in OpenSearch 3.1.x and the equivalent SQL query on 3.6.0.
Additional Details
Plugins
opensearch-sql
opensearch-dashboards
opensearch-security
opensearch-alerting
opensearch-notifications
opensearch-observability
Screenshots
Host/Environment (please complete the following information):
- OS: Ubuntu
- Version 24.04LTS
Additional context
org.apache.calcite.runtime.CalciteException: Failed to encode '未处置' in character set 'ISO-8859-1'
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:511)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:605)
at org.apache.calcite.util.NlsString.(NlsString.java:155)
at org.apache.calcite.util.NlsString.(NlsString.java:123)
at org.apache.calcite.rex.RexBuilder.clean(RexBuilder.java:2296)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:2070)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:2032)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitLiteral(CalciteRexNodeVisitor.java:127)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitLiteral(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.Literal.accept(Literal.java:57)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.tryMakeBooleanNotEquals(CalciteRexNodeVisitor.java:302)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitNot(CalciteRexNodeVisitor.java:199)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitNot(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.Not.accept(Not.java:31)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitAnd(CalciteRexNodeVisitor.java:176)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitAnd(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.And.accept(And.java:32)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitFilter(CalciteRelNodeVisitor.java:289)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitFilter(CalciteRelNodeVisitor.java:189)
at org.opensearch.sql.ast.tree.Filter.accept(Filter.java:41)
at org.opensearch.sql.ast.AbstractNodeVisitor.visitChildren(AbstractNodeVisitor.java:117)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitChildren(CalciteRelNodeVisitor.java:209)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitProject(CalciteRelNodeVisitor.java:432)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitProject(CalciteRelNodeVisitor.java:189)
at org.opensearch.sql.ast.tree.Project.accept(Project.java:65)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.analyze(CalciteRelNodeVisitor.java:204)
at org.opensearch.sql.executor.QueryService.analyze(QueryService.java:281)
at org.opensearch.sql.executor.QueryService.lambda$executeWithCalcite$0(QueryService.java:146)
at org.opensearch.sql.calcite.CalcitePlanContext.run(CalcitePlanContext.java:158)
at org.opensearch.sql.executor.QueryService.executeWithCalcite(QueryService.java:135)
at org.opensearch.sql.executor.QueryService.execute(QueryService.java:101)
at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:82)
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$schedule$0(OpenSearchQueryManager.java:84)
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$0(OpenSearchQueryManager.java:111)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:952)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.base/java.lang.Thread.run(Thread.java:1474)
Describe the bug
After upgrading from OpenSearch 3.1 to 3.6.0, PPL queries containing Chinese string literals fail with a CalciteException. The equivalent SQL query works perfectly on the same 3.6.0 cluster, and the identical PPL query worked without issues on 3.1. This is a confirmed regression in the PPL/Calcite integration layer regarding non-ASCII character encoding handling
Related component
Search:Query Capabilities
To Reproduce
Expected behavior
The PPL query should execute successfully and return matching documents, consistent with the behavior in OpenSearch 3.1.x and the equivalent SQL query on 3.6.0.
Additional Details
Plugins
opensearch-sql
opensearch-dashboards
opensearch-security
opensearch-alerting
opensearch-notifications
opensearch-observability
Screenshots
Host/Environment (please complete the following information):
Additional context
org.apache.calcite.runtime.CalciteException: Failed to encode '未处置' in character set 'ISO-8859-1'
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:511)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:605)
at org.apache.calcite.util.NlsString.(NlsString.java:155)
at org.apache.calcite.util.NlsString.(NlsString.java:123)
at org.apache.calcite.rex.RexBuilder.clean(RexBuilder.java:2296)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:2070)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:2032)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitLiteral(CalciteRexNodeVisitor.java:127)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitLiteral(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.Literal.accept(Literal.java:57)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.tryMakeBooleanNotEquals(CalciteRexNodeVisitor.java:302)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitNot(CalciteRexNodeVisitor.java:199)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitNot(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.Not.accept(Not.java:31)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitAnd(CalciteRexNodeVisitor.java:176)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitAnd(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.And.accept(And.java:32)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitFilter(CalciteRelNodeVisitor.java:289)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitFilter(CalciteRelNodeVisitor.java:189)
at org.opensearch.sql.ast.tree.Filter.accept(Filter.java:41)
at org.opensearch.sql.ast.AbstractNodeVisitor.visitChildren(AbstractNodeVisitor.java:117)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitChildren(CalciteRelNodeVisitor.java:209)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitProject(CalciteRelNodeVisitor.java:432)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitProject(CalciteRelNodeVisitor.java:189)
at org.opensearch.sql.ast.tree.Project.accept(Project.java:65)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.analyze(CalciteRelNodeVisitor.java:204)
at org.opensearch.sql.executor.QueryService.analyze(QueryService.java:281)
at org.opensearch.sql.executor.QueryService.lambda$executeWithCalcite$0(QueryService.java:146)
at org.opensearch.sql.calcite.CalcitePlanContext.run(CalcitePlanContext.java:158)
at org.opensearch.sql.executor.QueryService.executeWithCalcite(QueryService.java:135)
at org.opensearch.sql.executor.QueryService.execute(QueryService.java:101)
at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:82)
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$schedule$0(OpenSearchQueryManager.java:84)
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$0(OpenSearchQueryManager.java:111)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:952)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.base/java.lang.Thread.run(Thread.java:1474)