Skip to content

[BUG] PPL CalciteException: Failed to encode Chinese characters in ISO-8859-1 on 3.6.0 (works on 3.1) #21880

Description

@lukeyan2023

Describe the bug

After upgrading from OpenSearch 3.1 to 3.6.0, PPL queries containing Chinese string literals fail with a CalciteException. The equivalent SQL query works perfectly on the same 3.6.0 cluster, and the identical PPL query worked without issues on 3.1. This is a confirmed regression in the PPL/Calcite integration layer regarding non-ASCII character encoding handling

Related component

Search:Query Capabilities

To Reproduce

  1. Ensure an index exists with documents containing Chinese field values (e.g., status: "未处置")
  2. Execute the following PPL query
source=your_index | where status = '未处置'
  1. Observe the CalciteException error
  2. Execute the equivalent SQL query to confirm it works:
SELECT * FROM your_index WHERE status = '未处置'

Expected behavior

The PPL query should execute successfully and return matching documents, consistent with the behavior in OpenSearch 3.1.x and the equivalent SQL query on 3.6.0.

Additional Details

Plugins
opensearch-sql
opensearch-dashboards
opensearch-security
opensearch-alerting
opensearch-notifications
opensearch-observability

Screenshots

Image

Host/Environment (please complete the following information):

  • OS: Ubuntu
  • Version 24.04LTS

Additional context
org.apache.calcite.runtime.CalciteException: Failed to encode '未处置' in character set 'ISO-8859-1'
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:511)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:605)
at org.apache.calcite.util.NlsString.(NlsString.java:155)
at org.apache.calcite.util.NlsString.(NlsString.java:123)
at org.apache.calcite.rex.RexBuilder.clean(RexBuilder.java:2296)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:2070)
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:2032)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitLiteral(CalciteRexNodeVisitor.java:127)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitLiteral(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.Literal.accept(Literal.java:57)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.tryMakeBooleanNotEquals(CalciteRexNodeVisitor.java:302)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitNot(CalciteRexNodeVisitor.java:199)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitNot(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.Not.accept(Not.java:31)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitAnd(CalciteRexNodeVisitor.java:176)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.visitAnd(CalciteRexNodeVisitor.java:89)
at org.opensearch.sql.ast.expression.And.accept(And.java:32)
at org.opensearch.sql.calcite.CalciteRexNodeVisitor.analyze(CalciteRexNodeVisitor.java:94)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitFilter(CalciteRelNodeVisitor.java:289)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitFilter(CalciteRelNodeVisitor.java:189)
at org.opensearch.sql.ast.tree.Filter.accept(Filter.java:41)
at org.opensearch.sql.ast.AbstractNodeVisitor.visitChildren(AbstractNodeVisitor.java:117)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitChildren(CalciteRelNodeVisitor.java:209)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitProject(CalciteRelNodeVisitor.java:432)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.visitProject(CalciteRelNodeVisitor.java:189)
at org.opensearch.sql.ast.tree.Project.accept(Project.java:65)
at org.opensearch.sql.calcite.CalciteRelNodeVisitor.analyze(CalciteRelNodeVisitor.java:204)
at org.opensearch.sql.executor.QueryService.analyze(QueryService.java:281)
at org.opensearch.sql.executor.QueryService.lambda$executeWithCalcite$0(QueryService.java:146)
at org.opensearch.sql.calcite.CalcitePlanContext.run(CalcitePlanContext.java:158)
at org.opensearch.sql.executor.QueryService.executeWithCalcite(QueryService.java:135)
at org.opensearch.sql.executor.QueryService.execute(QueryService.java:101)
at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:82)
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$schedule$0(OpenSearchQueryManager.java:84)
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$0(OpenSearchQueryManager.java:111)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:952)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.base/java.lang.Thread.run(Thread.java:1474)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    🆕 New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions