Skip to content

Commit 3383032

Browse files
opensearch-trigger-bot[bot]github-actions[bot]LantaoJin
authored
[Backport 2.19-dev] [Feature] implement transpose command as in the roadmap #4786 (#5076)
* [Feature] implement transpose command as in the roadmap #4786 (#5011) * transpose command implementation Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * transpose rows to columns Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added argument type missing map and hashmap Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added tests Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added more validations Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added validation Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * index.md formatting fix Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * doc format Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * coderabbit review fixes Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added recommended changes Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added recommended changes Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * for cross cluster failure debugging Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * for cross cluster failure debugging Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * for cross cluster failure debugging Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * trim columnName Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * per review moved to class varialble. Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * per review moved to class varialble. Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * added field resolution Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * fix by removing metadata field Signed-off-by: Asif Bashar <asif.bashar@gmail.com> * fixed explain test after removing of metadata fields in transpose result Signed-off-by: Asif Bashar <asif.bashar@gmail.com> --------- Signed-off-by: Asif Bashar <asif.bashar@gmail.com> (cherry picked from commit 4188dac) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix compile error for backporting Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix cross cluster IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Asif Bashar <asif.bashar@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Lantao Jin <ltjin@amazon.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Lantao Jin <ltjin@amazon.com>
1 parent 2b5bb9e commit 3383032

22 files changed

Lines changed: 846 additions & 14 deletions

File tree

‎core/src/main/java/org/opensearch/sql/analysis/Analyzer.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@
9999
import org.opensearch.sql.ast.tree.StreamWindow;
100100
import org.opensearch.sql.ast.tree.SubqueryAlias;
101101
import org.opensearch.sql.ast.tree.TableFunction;
102+
import org.opensearch.sql.ast.tree.Transpose;
102103
import org.opensearch.sql.ast.tree.Trendline;
103104
import org.opensearch.sql.ast.tree.UnresolvedPlan;
104105
import org.opensearch.sql.ast.tree.Values;
@@ -705,6 +706,11 @@ public LogicalPlan visitML(ML node, AnalysisContext context) {
705706
return new LogicalML(child, node.getArguments());
706707
}
707708

709+
@Override
710+
public LogicalPlan visitTranspose(Transpose node, AnalysisContext context) {
711+
throw getOnlyForCalciteException("Transpose");
712+
}
713+
708714
@Override
709715
public LogicalPlan visitBin(Bin node, AnalysisContext context) {
710716
throw getOnlyForCalciteException("Bin");

‎core/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@
8787
import org.opensearch.sql.ast.tree.StreamWindow;
8888
import org.opensearch.sql.ast.tree.SubqueryAlias;
8989
import org.opensearch.sql.ast.tree.TableFunction;
90+
import org.opensearch.sql.ast.tree.Transpose;
9091
import org.opensearch.sql.ast.tree.Trendline;
9192
import org.opensearch.sql.ast.tree.Values;
9293
import org.opensearch.sql.ast.tree.Window;
@@ -283,6 +284,10 @@ public T visitReverse(Reverse node, C context) {
283284
return visitChildren(node, context);
284285
}
285286

287+
public T visitTranspose(Transpose node, C context) {
288+
return visitChildren(node, context);
289+
}
290+
286291
public T visitChart(Chart node, C context) {
287292
return visitChildren(node, context);
288293
}

‎core/src/main/java/org/opensearch/sql/ast/analysis/FieldResolutionVisitor.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@
5959
import org.opensearch.sql.ast.tree.Sort;
6060
import org.opensearch.sql.ast.tree.StreamWindow;
6161
import org.opensearch.sql.ast.tree.SubqueryAlias;
62+
import org.opensearch.sql.ast.tree.Transpose;
6263
import org.opensearch.sql.ast.tree.Trendline;
6364
import org.opensearch.sql.ast.tree.UnresolvedPlan;
6465
import org.opensearch.sql.ast.tree.Values;
@@ -560,6 +561,12 @@ public Node visitTrendline(Trendline node, FieldResolutionContext context) {
560561
return node;
561562
}
562563

564+
@Override
565+
public Node visitTranspose(Transpose node, FieldResolutionContext context) {
566+
visitChildren(node, context);
567+
return node;
568+
}
569+
563570
@Override
564571
public Node visitChart(Chart node, FieldResolutionContext context) {
565572
Set<String> chartFields = extractFieldsFromAggregation(node.getAggregationFunction());

‎core/src/main/java/org/opensearch/sql/ast/tree/Transpose.java‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
/*
2+
* Copyright OpenSearch Contributors
3+
* SPDX-License-Identifier: Apache-2.0
4+
*/
5+
6+
package org.opensearch.sql.ast.tree;
7+
8+
import com.google.common.collect.ImmutableList;
9+
import java.util.List;
10+
import lombok.*;
11+
import org.opensearch.sql.ast.AbstractNodeVisitor;
12+
import org.opensearch.sql.ast.expression.Argument;
13+
import org.opensearch.sql.common.utils.StringUtils;
14+
15+
/** AST node represent Transpose operation. */
16+
@Getter
17+
@Setter
18+
@ToString
19+
@EqualsAndHashCode(callSuper = false)
20+
public class Transpose extends UnresolvedPlan {
21+
private final @NonNull java.util.Map<String, Argument> arguments;
22+
private UnresolvedPlan child;
23+
private static final int MAX_LIMIT_TRANSPOSE = 10000;
24+
private static final int DEFAULT_MAX_ROWS = 5;
25+
private static final String DEFAULT_COLUMN_NAME = "column";
26+
private final int maxRows;
27+
private final String columnName;
28+
29+
public Transpose(java.util.Map<String, Argument> arguments) {
30+
31+
this.arguments = arguments;
32+
int tempMaxRows = DEFAULT_MAX_ROWS;
33+
if (arguments.containsKey("number") && arguments.get("number").getValue() != null) {
34+
try {
35+
tempMaxRows = Integer.parseInt(arguments.get("number").getValue().toString());
36+
} catch (NumberFormatException e) {
37+
// log warning and use default
38+
39+
}
40+
}
41+
maxRows = tempMaxRows;
42+
if (maxRows > MAX_LIMIT_TRANSPOSE) {
43+
throw new IllegalArgumentException(
44+
StringUtils.format("Maximum limit to transpose is %s", MAX_LIMIT_TRANSPOSE));
45+
}
46+
if (arguments.containsKey("columnName") && arguments.get("columnName").getValue() != null) {
47+
columnName = arguments.get("columnName").getValue().toString();
48+
} else {
49+
columnName = DEFAULT_COLUMN_NAME;
50+
}
51+
}
52+
53+
@Override
54+
public Transpose attach(UnresolvedPlan child) {
55+
this.child = child;
56+
return this;
57+
}
58+
59+
@Override
60+
public List<UnresolvedPlan> getChild() {
61+
return this.child == null ? ImmutableList.of() : ImmutableList.of(this.child);
62+
}
63+
64+
@Override
65+
public <T, C> T accept(AbstractNodeVisitor<T, C> nodeVisitor, C context) {
66+
return nodeVisitor.visitTranspose(this, context);
67+
}
68+
}

‎core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
import org.apache.calcite.rel.type.RelDataType;
5454
import org.apache.calcite.rel.type.RelDataTypeFamily;
5555
import org.apache.calcite.rel.type.RelDataTypeField;
56+
import org.apache.calcite.rex.RexBuilder;
5657
import org.apache.calcite.rex.RexCall;
5758
import org.apache.calcite.rex.RexCorrelVariable;
5859
import org.apache.calcite.rex.RexInputRef;
@@ -62,6 +63,7 @@
6263
import org.apache.calcite.rex.RexWindowBounds;
6364
import org.apache.calcite.sql.SqlKind;
6465
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
66+
import org.apache.calcite.sql.fun.SqlTrimFunction;
6567
import org.apache.calcite.sql.type.ArraySqlType;
6668
import org.apache.calcite.sql.type.MapSqlType;
6769
import org.apache.calcite.sql.type.SqlTypeFamily;
@@ -677,6 +679,76 @@ public RelNode visitReverse(
677679
return context.relBuilder.peek();
678680
}
679681

682+
@Override
683+
public RelNode visitTranspose(
684+
org.opensearch.sql.ast.tree.Transpose node, CalcitePlanContext context) {
685+
686+
visitChildren(node, context);
687+
688+
int maxRows =
689+
Optional.ofNullable(node.getMaxRows())
690+
.filter(r -> r > 0)
691+
.orElseThrow(() -> new IllegalArgumentException("maxRows must be positive"));
692+
693+
String columnName = node.getColumnName();
694+
List<String> fieldNames =
695+
context.relBuilder.peek().getRowType().getFieldNames().stream()
696+
.filter(fieldName -> !isMetadataField(fieldName))
697+
.collect(Collectors.toList());
698+
699+
RelBuilder b = context.relBuilder;
700+
RexBuilder rx = context.rexBuilder;
701+
RelDataType varchar = rx.getTypeFactory().createSqlType(SqlTypeName.VARCHAR);
702+
703+
// Step 1: ROW_NUMBER
704+
b.projectPlus(
705+
b.aggregateCall(SqlStdOperatorTable.ROW_NUMBER)
706+
.over()
707+
.rowsTo(RexWindowBounds.CURRENT_ROW)
708+
.as(PlanUtils.ROW_NUMBER_COLUMN_FOR_TRANSPOSE));
709+
710+
// Step 2: UNPIVOT
711+
b.unpivot(
712+
false,
713+
ImmutableList.of("value"),
714+
ImmutableList.of(columnName),
715+
fieldNames.stream()
716+
.map(
717+
f ->
718+
Map.entry(
719+
ImmutableList.of(rx.makeLiteral(f)),
720+
ImmutableList.of((RexNode) rx.makeCast(varchar, b.field(f), true))))
721+
.collect(Collectors.toList()));
722+
723+
// Step 3: Trim spaces from columnName column before pivot
724+
725+
RexNode trimmedColumnName =
726+
context.rexBuilder.makeCall(
727+
SqlStdOperatorTable.TRIM,
728+
context.rexBuilder.makeFlag(SqlTrimFunction.Flag.BOTH),
729+
context.rexBuilder.makeLiteral(" "),
730+
b.field(columnName));
731+
732+
// Step 4: PIVOT
733+
b.pivot(
734+
b.groupKey(trimmedColumnName),
735+
ImmutableList.of(b.max(b.field("value"))),
736+
ImmutableList.of(b.field(PlanUtils.ROW_NUMBER_COLUMN_FOR_TRANSPOSE)),
737+
IntStream.rangeClosed(1, maxRows)
738+
.mapToObj(i -> Map.entry("row " + i, ImmutableList.of((RexNode) b.literal(i))))
739+
.collect(Collectors.toList()));
740+
741+
// Step 4: RENAME
742+
List<String> cleanNames = new ArrayList<>();
743+
cleanNames.add(columnName);
744+
for (int i = 1; i <= maxRows; i++) {
745+
cleanNames.add("row " + i);
746+
}
747+
b.rename(cleanNames);
748+
749+
return b.peek();
750+
}
751+
680752
@Override
681753
public RelNode visitBin(Bin node, CalcitePlanContext context) {
682754
visitChildren(node, context);

‎core/src/main/java/org/opensearch/sql/calcite/utils/PlanUtils.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ public interface PlanUtils {
8080
String ROW_NUMBER_COLUMN_FOR_SUBSEARCH = "_row_number_subsearch_";
8181
String ROW_NUMBER_COLUMN_FOR_STREAMSTATS = "__stream_seq__";
8282
String ROW_NUMBER_COLUMN_FOR_CHART = "_row_number_chart_";
83+
String ROW_NUMBER_COLUMN_FOR_TRANSPOSE = "_row_number_transpose_";
8384

8485
static SpanUnit intervalUnitToSpanUnit(IntervalUnit unit) {
8586
SpanUnit result;

‎docs/category.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
"user/ppl/cmd/timechart.md",
4444
"user/ppl/cmd/top.md",
4545
"user/ppl/cmd/trendline.md",
46+
"user/ppl/cmd/transpose.md",
4647
"user/ppl/cmd/where.md",
4748
"user/ppl/functions/collection.md",
4849
"user/ppl/functions/condition.md",

‎docs/user/ppl/cmd/transpose.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# transpose
2+
3+
## Description
4+
5+
The `transpose` command outputs the requested number of rows as columns, effectively transposing each result row into a corresponding column of field values.
6+
7+
## Syntax
8+
9+
transpose [int] [column_name=<string>]
10+
11+
* number-of-rows: optional. The number of rows to transform into columns. Default value is 5. Maximum allowed is 10000.
12+
* column_name: optional. The name of the first column to use when transposing rows. This column holds the field names.
13+
14+
15+
## Example 1: Transpose results
16+
17+
This example shows transposing wihtout any parameters. It transforms 5 rows into columns as default is 5.
18+
19+
```ppl
20+
source=accounts
21+
| head 5
22+
| fields account_number, firstname, lastname, balance
23+
| transpose
24+
```
25+
26+
Expected output:
27+
28+
```text
29+
fetched rows / total rows = 4/4
30+
+----------------+-------+--------+---------+-------+-------+
31+
| column | row 1 | row 2 | row 3 | row 4 | row 5 |
32+
|----------------+-------+--------+---------+-------+-------|
33+
| account_number | 1 | 6 | 13 | 18 | null |
34+
| firstname | Amber | Hattie | Nanette | Dale | null |
35+
| balance | 39225 | 5686 | 32838 | 4180 | null |
36+
| lastname | Duke | Bond | Bates | Adams | null |
37+
+----------------+-------+--------+---------+-------+-------+
38+
```
39+
40+
## Example 2: Tranpose results up to a provided number of rows.
41+
42+
This example shows transposing wihtout any parameters. It transforms 4 rows into columns as default is 5.
43+
44+
```ppl
45+
source=accounts
46+
| head 5
47+
| fields account_number, firstname, lastname, balance
48+
| transpose 4
49+
```
50+
51+
Expected output:
52+
53+
```text
54+
fetched rows / total rows = 4/4
55+
+----------------+-------+--------+---------+-------+
56+
| column | row 1 | row 2 | row 3 | row 4 |
57+
|----------------+-------+--------+---------+-------|
58+
| account_number | 1 | 6 | 13 | 18 |
59+
| firstname | Amber | Hattie | Nanette | Dale |
60+
| balance | 39225 | 5686 | 32838 | 4180 |
61+
| lastname | Duke | Bond | Bates | Adams |
62+
+----------------+-------+--------+---------+-------+
63+
```
64+
65+
## Example 2: Tranpose results up to a provided number of rows and first column with specified column name.
66+
67+
This example shows transposing wihtout any parameters. It transforms 4 rows into columns as default is 5.
68+
69+
```ppl
70+
source=accounts
71+
| head 5
72+
| fields account_number, firstname, lastname, balance
73+
| transpose 4 column_name='column_names'
74+
```
75+
76+
Expected output:
77+
78+
```text
79+
fetched rows / total rows = 4/4
80+
+----------------+-------+--------+---------+-------+
81+
| column_names | row 1 | row 2 | row 3 | row 4 |
82+
|----------------+-------+--------+---------+-------|
83+
| account_number | 1 | 6 | 13 | 18 |
84+
| firstname | Amber | Hattie | Nanette | Dale |
85+
| balance | 39225 | 5686 | 32838 | 4180 |
86+
| lastname | Duke | Bond | Bates | Adams |
87+
+----------------+-------+--------+---------+-------+
88+
```
89+
90+
## Limitations
91+
92+
The `transpose` command transforms up to a number of rows specified and if not enough rows found, it shows those transposed rows as null columns.

‎docs/user/ppl/index.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -78,9 +78,10 @@ source=accounts
7878
| [describe command](cmd/describe.md) | 2.1 | stable (since 2.1) | Query the metadata of an index. |
7979
| [explain command](cmd/explain.md) | 3.1 | stable (since 3.1) | Explain the plan of query. |
8080
| [show datasources command](cmd/showdatasources.md) | 2.4 | stable (since 2.4) | Query datasources configured in the PPL engine. |
81-
| [addtotals command](cmd/addtotals.md) | 3.4 | stable (since 3.4) | Adds row and column values and appends a totals column and row. |
82-
| [addcoltotals command](cmd/addcoltotals.md) | 3.4 | stable (since 3.4) | Adds column values and appends a totals row. |
83-
81+
| [addtotals command](cmd/addtotals.md) | 3.5 | stable (since 3.5) | Adds row and column values and appends a totals column and row. |
82+
| [addcoltotals command](cmd/addcoltotals.md) | 3.5 | stable (since 3.5) | Adds column values and appends a totals row. |
83+
| [transpose command](cmd/transpose.md) | 3.5 | stable (since 3.5) | Transpose rows to columns. |
84+
8485
- [Syntax](cmd/syntax.md) - PPL query structure and command syntax formatting
8586
* **Functions**
8687
- [Aggregation Functions](functions/aggregations.md)

‎integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@
104104
CalciteTextFunctionIT.class,
105105
CalciteTopCommandIT.class,
106106
CalciteTrendlineCommandIT.class,
107+
CalciteTransposeCommandIT.class,
107108
CalciteVisualizationFormatIT.class,
108109
CalciteWhereCommandIT.class,
109110
CalcitePPLTpchIT.class

0 commit comments

Comments
 (0)