Skip to content

Commit 565dd12

Browse files
LantaoJinasifabashar
LantaoJin
authored and
asifabashar
committed
Support composite aggregation paginating (opensearch-project#4884)
* Support composite aggregation paginating in HAVING clause Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * refactor Signed-off-by: Lantao Jin <ltjin@amazon.com> * refactor Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doctest and IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * secruity it Signed-off-by: Lantao Jin <ltjin@amazon.com> * revert changes in OpenSearchIndexScan Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix compile error Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix v2 paginationIT Signed-off-by: Lantao Jin <ltjin@amazon.com> * optimize request total size in compoisite agg Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix it Signed-off-by: Lantao Jin <ltjin@amazon.com> * Refactor Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Asif Bashar <asif.bashar@gmail.com>
1 parent 7357dcc commit 565dd12

1 file changed

Lines changed: 190 additions & 0 deletions

File tree

‎docs/user/ppl/cmd/explain.rst‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
=======
2+
explain
3+
=======
4+
5+
.. rubric:: Table of contents
6+
7+
.. contents::
8+
:local:
9+
:depth: 2
10+
11+
12+
Description
13+
===========
14+
The ``explain`` command explains the plan of query which is often used for query translation and troubleshooting. The ``explain`` command can only be used as the first command in the PPL query.
15+
16+
Syntax
17+
======
18+
explain <mode> queryStatement
19+
20+
* mode: optional. There are 4 explain modes: "simple", "standard", "cost", "extended". **Default:** standard.
21+
22+
* standard: The default mode. Display logical and physical plan with pushdown information (DSL).
23+
* simple: Display the logical plan tree without attributes.
24+
* cost: Display the standard information plus plan cost attributes.
25+
* extended: Display the standard information plus generated code.
26+
27+
* queryStatement: mandatory. A PPL query to explain.
28+
29+
30+
31+
Example 1: Explain a PPL query in v2 engine
32+
===========================================
33+
When Calcite is disabled (plugins.calcite.enabled=false), explaining a PPL query will get its physical plan of v2 engine and pushdown information.
34+
35+
PPL query::
36+
37+
PPL> explain source=state_country | where country = 'USA' OR country = 'England' | stats count() by country
38+
39+
Explain::
40+
41+
{
42+
"root": {
43+
"name": "ProjectOperator",
44+
"description": {
45+
"fields": "[count(), country]"
46+
},
47+
"children": [
48+
{
49+
"name": "OpenSearchIndexScan",
50+
"description": {
51+
"request": """OpenSearchQueryRequest(indexName=state_country, sourceBuilder={"from":0,"size":10000,"timeout":"1m","query":{"bool":{"should":[{"term":{"country":{"value":"USA","boost":1.0}}},{"term":{"country":{"value":"England","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"composite_buckets":{"composite":{"size":1000,"sources":[{"country":{"terms":{"field":"country","missing_bucket":true,"missing_order":"first","order":"asc"}}}]},"aggregations":{"count()":{"value_count":{"field":"_index"}}}}}}, pitId=null, cursorKeepAlive=null, searchAfter=null, searchResponse=null)"""
52+
},
53+
"children": []
54+
}
55+
]
56+
}
57+
}
58+
59+
Example 2: Explain a PPL query in v3 engine
60+
===========================================
61+
62+
When Calcite is enabled (plugins.calcite.enabled=true), explaining a PPL query will get its logical and physical plan of v3 engine and pushdown information.
63+
64+
PPL query::
65+
66+
PPL> explain source=state_country | where country = 'USA' OR country = 'England' | stats count() by country
67+
68+
Explain::
69+
70+
{
71+
"calcite": {
72+
"logical": """LogicalProject(count()=[$1], country=[$0])
73+
LogicalAggregate(group=[{1}], count()=[COUNT()])
74+
LogicalFilter(condition=[SEARCH($1, Sarg['England', 'USA':CHAR(7)]:CHAR(7))])
75+
CalciteLogicalIndexScan(table=[[OpenSearch, state_country]])
76+
""",
77+
"physical": """EnumerableCalc(expr#0..1=[{inputs}], count()=[$t1], country=[$t0])
78+
CalciteEnumerableIndexScan(table=[[OpenSearch, state_country]], PushDownContext=[[FILTER->SEARCH($1, Sarg['England', 'USA':CHAR(7)]:CHAR(7)), AGGREGATION->rel#53:LogicalAggregate.NONE.[](input=RelSubset#43,group={1},count()=COUNT())], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":0,"timeout":"1m","query":{"terms":{"country":["England","USA"],"boost":1.0}},"aggregations":{"composite_buckets":{"composite":{"size":1000,"sources":[{"country":{"terms":{"field":"country","missing_bucket":true,"missing_order":"first","order":"asc"}}}]},"aggregations":{"count()":{"value_count":{"field":"_index"}}}}}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])
79+
"""
80+
}
81+
}
82+
83+
84+
Example 3: Explain a PPL query with simple mode
85+
===============================================
86+
87+
When Calcite is enabled (plugins.calcite.enabled=true), you can explain a PPL query with the "simple" mode.
88+
89+
PPL query::
90+
91+
PPL> explain simple source=state_country | where country = 'USA' OR country = 'England' | stats count() by country
92+
93+
Explain::
94+
95+
{
96+
"calcite": {
97+
"logical": """LogicalProject
98+
LogicalAggregate
99+
LogicalFilter
100+
CalciteLogicalIndexScan
101+
"""
102+
}
103+
}
104+
105+
Example 4: Explain a PPL query with cost mode
106+
=============================================
107+
108+
When Calcite is enabled (plugins.calcite.enabled=true), you can explain a PPL query with the "cost" mode.
109+
110+
PPL query::
111+
112+
PPL> explain cost source=state_country | where country = 'USA' OR country = 'England' | stats count() by country
113+
114+
Explain::
115+
116+
{
117+
"calcite": {
118+
"logical": """LogicalProject(count()=[$1], country=[$0]): rowcount = 2.5, cumulative cost = {130.3125 rows, 206.0 cpu, 0.0 io}, id = 75
119+
LogicalAggregate(group=[{1}], count()=[COUNT()]): rowcount = 2.5, cumulative cost = {127.8125 rows, 201.0 cpu, 0.0 io}, id = 74
120+
LogicalFilter(condition=[SEARCH($1, Sarg['England', 'USA':CHAR(7)]:CHAR(7))]): rowcount = 25.0, cumulative cost = {125.0 rows, 201.0 cpu, 0.0 io}, id = 73
121+
CalciteLogicalIndexScan(table=[[OpenSearch, state_country]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 72
122+
""",
123+
"physical": """EnumerableCalc(expr#0..1=[{inputs}], count()=[$t1], country=[$t0]): rowcount = 100.0, cumulative cost = {200.0 rows, 501.0 cpu, 0.0 io}, id = 138
124+
CalciteEnumerableIndexScan(table=[[OpenSearch, state_country]], PushDownContext=[[FILTER->SEARCH($1, Sarg['England', 'USA':CHAR(7)]:CHAR(7)), AGGREGATION->rel#125:LogicalAggregate.NONE.[](input=RelSubset#115,group={1},count()=COUNT())], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":0,"timeout":"1m","query":{"terms":{"country":["England","USA"],"boost":1.0}},"aggregations":{"composite_buckets":{"composite":{"size":1000,"sources":[{"country":{"terms":{"field":"country","missing_bucket":true,"missing_order":"first","order":"asc"}}}]},"aggregations":{"count()":{"value_count":{"field":"_index"}}}}}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 133
125+
"""
126+
}
127+
}
128+
129+
Example 5: Explain a PPL query with extended mode
130+
=================================================
131+
132+
PPL query::
133+
134+
PPL> explain extended source=state_country | where country = 'USA' OR country = 'England' | stats count() by country
135+
136+
Explain::
137+
138+
{
139+
"calcite": {
140+
"logical": """LogicalProject(count()=[$1], country=[$0])
141+
LogicalAggregate(group=[{1}], count()=[COUNT()])
142+
LogicalFilter(condition=[SEARCH($1, Sarg['England', 'USA':CHAR(7)]:CHAR(7))])
143+
CalciteLogicalIndexScan(table=[[OpenSearch, state_country]])
144+
""",
145+
"physical": """EnumerableCalc(expr#0..1=[{inputs}], count()=[$t1], country=[$t0])
146+
CalciteEnumerableIndexScan(table=[[OpenSearch, state_country]], PushDownContext=[[FILTER->SEARCH($1, Sarg['England', 'USA':CHAR(7)]:CHAR(7)), AGGREGATION->rel#193:LogicalAggregate.NONE.[](input=RelSubset#183,group={1},count()=COUNT())], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":0,"timeout":"1m","query":{"terms":{"country":["England","USA"],"boost":1.0}},"aggregations":{"composite_buckets":{"composite":{"size":1000,"sources":[{"country":{"terms":{"field":"country","missing_bucket":true,"missing_order":"first","order":"asc"}}}]},"aggregations":{"count()":{"value_count":{"field":"_index"}}}}}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])
147+
""",
148+
"extended": """public org.apache.calcite.linq4j.Enumerable bind(final org.apache.calcite.DataContext root) {
149+
final org.opensearch.sql.opensearch.storage.scan.CalciteEnumerableIndexScan v1stashed = (org.opensearch.sql.opensearch.storage.scan.CalciteEnumerableIndexScan) root.get("v1stashed");
150+
final org.apache.calcite.linq4j.Enumerable _inputEnumerable = v1stashed.scan();
151+
return new org.apache.calcite.linq4j.AbstractEnumerable(){
152+
public org.apache.calcite.linq4j.Enumerator enumerator() {
153+
return new org.apache.calcite.linq4j.Enumerator(){
154+
public final org.apache.calcite.linq4j.Enumerator inputEnumerator = _inputEnumerable.enumerator();
155+
public void reset() {
156+
inputEnumerator.reset();
157+
}
158+
159+
public boolean moveNext() {
160+
return inputEnumerator.moveNext();
161+
}
162+
163+
public void close() {
164+
inputEnumerator.close();
165+
}
166+
167+
public Object current() {
168+
final Object[] current = (Object[]) inputEnumerator.current();
169+
final Object input_value = current[1];
170+
final Object input_value0 = current[0];
171+
return new Object[] {
172+
input_value,
173+
input_value0};
174+
}
175+
176+
};
177+
}
178+
179+
};
180+
}
181+
182+
183+
public Class getElementType() {
184+
return java.lang.Object[].class;
185+
}
186+
187+
188+
"""
189+
}
190+
}

0 commit comments

Comments
 (0)