Skip to content

Commit fa2ec9e

Browse files
committed
fix IT due to performance change
Signed-off-by: xinyual <xinyual@amazon.com>
1 parent 1624912 commit fa2ec9e

3 files changed

Lines changed: 14 additions & 56 deletions

File tree

docs/user/ppl/cmd/appendpipe.rst

Lines changed: 3 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -66,23 +66,7 @@ PPL query::
6666
| 101 | M | null |
6767
+----------+--------+-------+
6868

69-
Example 3: Append rows with column type conflict
70-
=============================================
71-
72-
This example shows how column type conflicts are handled when appending results. Same name columns with different types will generate two different columns in appended result.
73-
74-
PPL query::
75-
76-
os> source=accounts | stats sum(age) as total by gender, state | sort -total | head 5 | appendpipe [ stats sum(total) as total by gender | eval state = 123 ];
77-
fetched rows / total rows = 6/6
78-
+------+--------+-------+--------+
79-
| sum | gender | state | state0 |
80-
|------+--------+-------+--------|
81-
| 36 | M | TN | null |
82-
| 33 | M | MD | null |
83-
| 32 | M | IL | null |
84-
| 28 | F | VA | null |
85-
| 28 | F | null | 123 |
86-
| 101 | M | null | 123 |
87-
+------+--------+-------+--------+
69+
Limitations
70+
===========
8871

72+
* **Schema Compatibility**: Same as command ``append``, when fields with the same name exist between the main search and sub-search but have incompatible types, the query will fail with an error. To avoid type conflicts, ensure that fields with the same name have the same data type, or use different field names (e.g., by renaming with ``eval`` or using ``fields`` to select non-conflicting columns).

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLAppendPipeCommandIT.java

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -75,20 +75,16 @@ public void testAppendpipeWithMergedColumn() throws IOException {
7575

7676
@Test
7777
public void testAppendpipeWithConflictTypeColumn() throws IOException {
78-
JSONObject actual =
79-
executeQuery(
80-
String.format(
81-
Locale.ROOT,
82-
"source=%s | stats sum(age) as sum by gender | appendpipe [ eval sum = cast(sum as"
83-
+ " double) ] | head 5",
84-
TEST_INDEX_ACCOUNT));
85-
verifySchemaInOrder(
86-
actual, schema("sum", "bigint"), schema("gender", "string"), schema("sum0", "double"));
87-
verifyDataRows(
88-
actual,
89-
rows(14947, "F", null),
90-
rows(15224, "M", null),
91-
rows(null, "F", 14947d),
92-
rows(null, "M", 15224d));
78+
Exception exception =
79+
assertThrows(
80+
Exception.class,
81+
() ->
82+
executeQuery(
83+
String.format(
84+
Locale.ROOT,
85+
"source=%s | stats sum(age) as sum by gender | appendpipe [ eval sum ="
86+
+ " cast(sum as double) ] | head 5",
87+
TEST_INDEX_ACCOUNT)));
88+
assertTrue(exception.getMessage().contains("due to incompatible types"));
9389
}
9490
}

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAppendPipeTest.java

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -59,26 +59,4 @@ public void testAppendPipeWithMergedColumns() {
5959
+ "FROM `scott`.`EMP`";
6060
verifyPPLToSparkSQL(root, expectedSparkSql);
6161
}
62-
63-
@Test
64-
public void testAppendPipeWithConflictTypeColumn() {
65-
String ppl = "source=EMP | fields DEPTNO | appendpipe [ | fields DEPTNO | eval DEPTNO = 20 ]";
66-
RelNode root = getRelNode(ppl);
67-
String expectedLogical =
68-
"LogicalUnion(all=[true])\n"
69-
+ " LogicalProject(DEPTNO=[$7], DEPTNO0=[null:INTEGER])\n"
70-
+ " LogicalTableScan(table=[[scott, EMP]])\n"
71-
+ " LogicalProject(DEPTNO=[null:TINYINT], DEPTNO0=[20])\n"
72-
+ " LogicalTableScan(table=[[scott, EMP]])\n";
73-
verifyLogical(root, expectedLogical);
74-
verifyResultCount(root, 28);
75-
76-
String expectedSparkSql =
77-
"SELECT `DEPTNO`, CAST(NULL AS INTEGER) `DEPTNO0`\n"
78-
+ "FROM `scott`.`EMP`\n"
79-
+ "UNION ALL\n"
80-
+ "SELECT CAST(NULL AS TINYINT) `DEPTNO`, 20 `DEPTNO0`\n"
81-
+ "FROM `scott`.`EMP`";
82-
verifyPPLToSparkSQL(root, expectedSparkSql);
83-
}
8462
}

0 commit comments

Comments
 (0)