Skip to content

Commit f3cde1d

Browse files
penghuopenghuo
authored
Update PPL Limitation Docs (opensearch-project#3656)
* Update PPL Limitation Docs Signed-off-by: Peng Huo <penghuo@gmail.com> * update table Signed-off-by: Peng Huo <penghuo@gmail.com> * update doc Signed-off-by: Peng Huo <penghuo@gmail.com> * Update Signed-off-by: Peng Huo <penghuo@gmail.com> --------- Signed-off-by: Peng Huo <penghuo@gmail.com>
1 parent 85bad19 commit f3cde1d

1 file changed

Lines changed: 71 additions & 6 deletions

File tree

‎docs/user/ppl/limitations/limitations.rst‎

Lines changed: 71 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
21
===========
32
Limitations
43
===========
@@ -9,13 +8,79 @@ Limitations
98
:local:
109
:depth: 2
1110

11+
Inconsistent Field Types across indices
12+
=======================================
13+
14+
* If the same field has different types across indices (e.g., ``field`` is a ``string`` in one index and an ``integer`` in another), PPL selects a field type from one of the indices—this selection is non-deterministic. Fields with other types are ignored during query execution.
15+
* For ``object`` fields, `PPL merges subfields from different indices to tolerate schema variations <https://github.com/opensearch-project/sql/issues/3625>`_.
16+
17+
Unsupported OpenSearch Field Types
18+
==================================
19+
20+
PPL does not support all `OpenSearch data types <https://docs.opensearch.org/docs/latest/field-types/supported-field-types/index/>`_. (e.g., ``flattened``, some complex ``nested`` usages). Unsupported fields are excluded from ``DESCRIBE`` and ``SOURCE`` outputs. At runtime: Queries referencing unsupported fields fail with semantic or resolution errors. Such fields are ignored in projections unless explicitly filtered out or removed at ingestion.
21+
22+
+---------------------------+---------+
23+
| OpenSearch Data Type | PPL |
24+
+===========================+=========+
25+
| knn_vector | Ignored |
26+
+---------------------------+---------+
27+
| Range field types | Ignored |
28+
+---------------------------+---------+
29+
| Object - flat_object | Ignored |
30+
+---------------------------+---------+
31+
| Object - join | Ignored |
32+
+---------------------------+---------+
33+
| String - Match-only text | Ignored |
34+
+---------------------------+---------+
35+
| String - Wildcard | Ignored |
36+
+---------------------------+---------+
37+
| String - token_count | Ignored |
38+
+---------------------------+---------+
39+
| String - constant_keyword | Ignored |
40+
+---------------------------+---------+
41+
| Autocomplete | Ignored |
42+
+---------------------------+---------+
43+
| Geoshape | Ignored |
44+
+---------------------------+---------+
45+
| Cartesian field types | Ignored |
46+
+---------------------------+---------+
47+
| Rank field types | Ignored |
48+
+---------------------------+---------+
49+
| Star-tree | Ignored |
50+
+---------------------------+---------+
51+
| derived | Ignored |
52+
+---------------------------+---------+
53+
| Percolator | Ignored |
54+
+---------------------------+---------+
1255

13-
Introduction
14-
============
56+
Field Parameters
57+
================
1558

16-
In this doc, the restrictions and limitations of PPL is covered as follows.
59+
For a field to be queryable in PPL, the following index settings must be enabled:
1760

18-
Limitations on Fields
61+
+------------------+--------------------------------------------------+--------------------------------------------------+
62+
| Setting | Description | Required For |
63+
+==================+==================================================+==================================================+
64+
| _source: true | Stores the original JSON document | Required for fetch raw data. |
65+
+------------------+--------------------------------------------------+--------------------------------------------------+
66+
| index: true | Enables field indexing | Required for filtering, search, and aggregations |
67+
+------------------+--------------------------------------------------+--------------------------------------------------+
68+
| doc_values: true | Enables columnar access for aggregations/sorting | Required for `stats`, `sort` |
69+
+------------------+--------------------------------------------------+--------------------------------------------------+
70+
71+
72+
Nested Field Behavior
1973
=====================
2074

21-
We are not supporting use `alias field type <https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html>`_ as identifier. It will throw exception ``can't resolve Symbol``.
75+
* There are `limitations <https://github.com/opensearch-project/sql/issues/52>`_ regarding the nested levels and query types that needs improvement.
76+
77+
Multi-value Field Behavior
78+
==========================
79+
80+
OpenSearch does not natively support the ARRAY data type but does allow multi-value fields implicitly. The
81+
SQL/PPL plugin adheres strictly to the data type semantics defined in index mappings. When parsing OpenSearch
82+
responses, it expects data to match the declared type and does not account for data in array format. If the
83+
plugins.query.field_type_tolerance setting is enabled, the SQL/PPL plugin will handle array datasets by returning
84+
scalar data types, allowing basic queries (e.g., source = tbl | where condition). However, using multi-value
85+
fields in expressions or functions will result in exceptions. If this setting is disabled or absent, only the
86+
first element of an array is returned, preserving the default behavior.

0 commit comments

Comments
 (0)