Skip to content

Commit 37b5c66

Browse files
yuancuyuancu
committed
Support time modifiers in search command (opensearch-project#4224)
* Implement absolute time range in search command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Unit test search with absolute time range Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Rephrase timeRange and timeModifier Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Switch to earliest and latest udf Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Add a convert util Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Verify time correctness during coversion Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Fix quarter parsing bugs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Fix week snap parsing Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Remove old implementation that translates time modifier to time filter Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix anomalyzed test & add a todo for an ignored test Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Support now() in time range Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix time modifier explain ITs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Support unixtimestamp (second) as a time modifier value Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update docs for search command with time modifiers Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test accessing fields with name earliest and latest in search command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update doctest in condition.rst due to the update in the implementation of earliest and latest conditions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update PPLQueryDataAnonymizerTest.java Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update explain ITs to use yaml plan files Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update a link to OpenSearch exists Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Support using timesnaps without quotes Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add a unit test for direct format - additionally rename parseRelativeTime to resolveTimeModifier Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add support to ISO 8601 date format to time modifier, as it is now widely supported in PPL Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update syntax by reusing SPANLENGTH definition Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update explain IT for search with time modifier Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add integration tests for time modifiers Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Parse timestamp string with multiple parsers in a loop Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Remove opensearch test dependency from core module Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix unit tests Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Minor updates to explain limitations in search doc Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> (cherry picked from commit e468513)
1 parent 782347f commit 37b5c66

34 files changed

Lines changed: 1707 additions & 136 deletions

‎core/src/main/java/org/opensearch/sql/calcite/plan/OpenSearchConstants.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ public interface OpenSearchConstants {
2020

2121
String METADATA_FIELD_ROUTING = "_routing";
2222

23+
String IMPLICIT_FIELD_TIMESTAMP = "@timestamp";
24+
2325
java.util.Map<String, ExprType> METADATAFIELD_TYPE_MAP =
2426
Map.of(
2527
METADATA_FIELD_ID, ExprCoreType.STRING,

‎core/src/main/java/org/opensearch/sql/expression/function/udf/condition/EarliestFunction.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,6 @@ public static Boolean earliest(Object... inputs) {
6868
ZonedDateTime earliest =
6969
getRelativeZonedDateTime(
7070
expression, ZonedDateTime.ofInstant(clock.instant(), clock.getZone()));
71-
return earliest.isBefore(candidateDatetime);
71+
return !earliest.isAfter(candidateDatetime);
7272
}
7373
}

‎core/src/main/java/org/opensearch/sql/expression/function/udf/condition/LatestFunction.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,6 @@ public static Boolean latest(Object... inputs) {
6868
ZonedDateTime latest =
6969
getRelativeZonedDateTime(
7070
expression, ZonedDateTime.ofInstant(clock.instant(), clock.getZone()));
71-
return latest.isAfter(candidateDatetime);
71+
return !latest.isBefore(candidateDatetime);
7272
}
7373
}

0 commit comments

Comments
 (0)