You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -88,7 +88,7 @@ Recently we have been actively improving our query engine primarily for better c
88
88
89
89
## Documentation
90
90
91
-
Please refer to the [SQL Language Reference Manual](./docs/user/index.rst), [Piped Processing Language (PPL) Reference Manual](./docs/user/ppl/index.rst), [OpenSearch SQL/PPL Engine Development Manual](./docs/dev/index.md) and [Technical Documentation](https://opensearch.org/docs/latest/search-plugins/sql/index/) for detailed information on installing and configuring plugin.
91
+
Please refer to the [SQL Language Reference Manual](./docs/user/index.rst), [Piped Processing Language (PPL) Reference Manual](./docs/user/ppl/index.md), [OpenSearch SQL/PPL Engine Development Manual](./docs/dev/index.md) and [Technical Documentation](https://opensearch.org/docs/latest/search-plugins/sql/index/) for detailed information on installing and configuring plugin.
Copy file name to clipboardExpand all lines: docs/dev/intro-v3-engine.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,9 +26,9 @@ Find more details in [V3 Architecture](./intro-v3-architecture.md).
26
26
27
27
In the initial release of the V3 engine (3.0.0), the main new features focus on enhancing the PPL language while maintaining maximum compatibility with V2 behavior.
Copy file name to clipboardExpand all lines: docs/user/ppl/admin/connectors/s3glue_connector.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ This page covers s3Glue datasource configuration and also how to query and s3Glu
8
8
9
9
*`EMRServerless Spark Execution Engine Config Setting`: Since we execute s3Glue queries on top of spark execution engine, we require this configuration.
10
10
11
-
More details: [ExecutionEngine Config](../../../interfaces/asyncqueryinterface.md#id2)
11
+
More details: [ExecutionEngine Config](../../../interfaces/asyncqueryinterface.rst#id2)
12
12
*`S3`: This is where the data lies.
13
13
*`Glue` Metadata store: Glue takes care of table metadata.
14
14
*`Opensearch IndexStore`: Index for s3 data lies in opensearch and also acts as temporary buffer for query results.
@@ -73,5 +73,5 @@ Sample Queries
73
73
* Create Covering Index Query: `create index clientip_year on my_glue.default.http_logs (clientip, year) WITH (auto_refresh=true)`
74
74
* Create Skipping Index: `create skipping index on mys3.default.http_logs (status VALUE_SET)`
75
75
76
-
These queries would work only top of async queries. Documentation: [Async Query APIs](../../../interfaces/asyncqueryinterface.md)
76
+
These queries would work only top of async queries. Documentation: [Async Query APIs](../../../interfaces/asyncqueryinterface.rst)
77
77
Documentation for Index Queries: https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md
Copy file name to clipboardExpand all lines: docs/user/ppl/admin/connectors/security_lake_connector.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ Security Lake connector provides a way to query Security Lake tables.
7
7
8
8
*`EMRServerless Spark Execution Engine Config Setting`: Since we execute s3Glue queries on top of spark execution engine, we require this configuration.
9
9
10
-
More details: [ExecutionEngine Config](../../../interfaces/asyncqueryinterface.md#id2)
10
+
More details: [ExecutionEngine Config](../../../interfaces/asyncqueryinterface.rst#id2)
11
11
*`S3`: This is where the data lies.
12
12
*`Glue`: Metadata store: Glue takes care of table metadata.
13
13
*`Lake Formation`: AWS service that performs authorization on Security Lake tables
@@ -59,5 +59,5 @@ Sample Queries
59
59
* Select Query : `select * from mysl.amazon_security_lake_glue_db_eu_west_1.amazon_security_lake_table_eu_west_1_vpc_flow_2_0 limit 1`
60
60
* Create Covering Index Query: `create index srcip_time on mysl.amazon_security_lake_glue_db_eu_west_1.amazon_security_lake_table_eu_west_1_vpc_flow_2_0 (src_endpoint.ip, time) WITH (auto_refresh=true)`
61
61
62
-
These queries would work only top of async queries. Documentation: [Async Query APIs](../../../interfaces/asyncqueryinterface.md)
62
+
These queries would work only top of async queries. Documentation: [Async Query APIs](../../../interfaces/asyncqueryinterface.rst)
63
63
Documentation for Index Queries: https://github.com/opensearch-project/opensearch-spark/blob/main/docs/index.md
The relevance functions are available to execute only in OpenSearch DSL but not in memory as of now, so the relevance search might fail for queries that are too complex to translate into DSL if the relevance function is following after a complex PPL query. To make your queries always work-able, it is recommended to place the relevance commands as close to the search command as possible, to ensure the relevance functions are eligible to push down. For example, a complex query like `search source = people | rename firstname as name | dedup account_number | fields name, account_number, balance, employer | where match(employer, 'Open Search') | stats count() by city` could fail because it is difficult to translate to DSL, but it would be better if we rewrite it to an equivalent query as `search source = people | where match(employer, 'Open Search') | rename firstname as name | dedup account_number | fields name, account_number, balance, employer | stats count() by city` by moving the where command with relevance function to the second command right after the search command, and the relevance would be optimized and executed smoothly in OpenSearch DSL. See [Optimization](../../optimization/optimization.md) to get more details about the query engine optimization.
505
+
The relevance functions are available to execute only in OpenSearch DSL but not in memory as of now, so the relevance search might fail for queries that are too complex to translate into DSL if the relevance function is following after a complex PPL query. To make your queries always work-able, it is recommended to place the relevance commands as close to the search command as possible, to ensure the relevance functions are eligible to push down. For example, a complex query like `search source = people | rename firstname as name | dedup account_number | fields name, account_number, balance, employer | where match(employer, 'Open Search') | stats count() by city` could fail because it is difficult to translate to DSL, but it would be better if we rewrite it to an equivalent query as `search source = people | where match(employer, 'Open Search') | rename firstname as name | dedup account_number | fields name, account_number, balance, employer | stats count() by city` by moving the where command with relevance function to the second command right after the search command, and the relevance would be optimized and executed smoothly in OpenSearch DSL. See [Optimization](../../optimization/optimization.rst) to get more details about the query engine optimization.
Copy file name to clipboardExpand all lines: docs/user/ppl/general/datatypes.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -58,7 +58,7 @@ Notes: Not all the PPL Type has correspond OpenSearch Type. e.g. data and time.
58
58
Numeric values ranged from -2147483648 to +2147483647 are recognized as integer with type name `int`. For others outside the range, `bigint` integer will be the data type after parsed.
59
59
## Date and Time Data Types
60
60
61
-
The date and time data types are the types that represent temporal values and PPL plugin supports types including DATE, TIME, TIMESTAMP and INTERVAL. By default, the OpenSearch DSL uses date type as the only date and time related type, which has contained all information about an absolute time point. To integrate with PPL language, each of the types other than timestamp is holding part of temporal or timezone information, and the usage to explicitly clarify the date and time types is reflected in the datetime functions (see [Functions](functions.md) for details), where some functions might have restrictions in the input argument type.
61
+
The date and time data types are the types that represent temporal values and PPL plugin supports types including DATE, TIME, TIMESTAMP and INTERVAL. By default, the OpenSearch DSL uses date type as the only date and time related type, which has contained all information about an absolute time point. To integrate with PPL language, each of the types other than timestamp is holding part of temporal or timezone information, and the usage to explicitly clarify the date and time types is reflected in the datetime functions (see [Functions](../functions/datetime.md) for details), where some functions might have restrictions in the input argument type.
62
62
### Date
63
63
64
64
Date represents the calendar date regardless of the time zone. A given date value represents a 24-hour period, or say a day, but this period varies in different timezones and might have flexible hours during Daylight Savings Time programs. Besides, the date type does not contain time information as well. The supported range is '1000-01-01' to '9999-12-31'.
@@ -91,7 +91,7 @@ Interval data type represents a temporal duration or a period. The syntax is as
91
91
| --- | --- |
92
92
| Interval | INTERVAL expr unit |
93
93
94
-
The expr is any expression that can be iterated to a quantity value eventually, see [Expressions](expressions.md) for details. The unit represents the unit for interpreting the quantity, including MICROSECOND, SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER and YEAR.The INTERVAL keyword and the unit specifier are not case sensitive. Note that there are two classes of intervals. Year-week intervals can store years, quarters, months and weeks. Day-time intervals can store days, hours, minutes, seconds and microseconds. Year-week intervals are comparable only with another year-week intervals. These two types of intervals can only comparable with the same type of themselves.
94
+
The expr is any expression that can be iterated to a quantity value eventually, see [Expressions](../functions/expressions.md) for details. The unit represents the unit for interpreting the quantity, including MICROSECOND, SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER and YEAR.The INTERVAL keyword and the unit specifier are not case sensitive. Note that there are two classes of intervals. Year-week intervals can store years, quarters, months and weeks. Day-time intervals can store days, hours, minutes, seconds and microseconds. Year-week intervals are comparable only with another year-week intervals. These two types of intervals can only comparable with the same type of themselves.
95
95
### Conversion between date and time types
96
96
97
97
Basically the date and time types except interval can be converted to each other, but might suffer some alteration of the value or some information loss, for example extracting the time value from a timestamp value, or convert a date value to a timestamp value and so forth. Here lists the summary of the conversion rules that PPL plugin supports for each of the types:
0 commit comments