BUILD: fix linkchecker in 2.x (opensearch-project#3793) (opensearch-project#3795)

* BUILD: fix linkchecker in 2.x



* ignore github url



* add missing doc links



* fix typo



---------


(cherry picked from commit 7268bc5)

Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Author: opensearch-trigger-bot[bot]

.github/workflows/link-checker.yml

@@ -19,7 +19,7 @@ jobs:
         id: lychee
         uses: lycheeverse/lychee-action@master
         with:
-          args: --accept=200,403,429,999  "./**/*.html" "./**/*.md" "./**/*.txt" --exclude "https://aws.oss.sonatype.*" "http://localhost*" "https://localhost" "https://odfe-node1:9200/" "https://community.tableau.com/docs/DOC-17978" ".*family.zzz" "https://pypi.python.org/pypi/opensearchsql/" "opensearch*" ".*@amazon.com" ".*email.com" "git@github.com" "http://timestamp.verisign.com/scripts/timstamp.dll" ".*/PowerBIConnector/bin/Release"
+          args: --accept=200,403,429,999  "./**/*.html" "./**/*.md" "./**/*.txt" --exclude "https://aws.oss.sonatype.*|http://localhost.*|https://localhost|https://odfe-node1:9200/|https://community.tableau.com/docs/DOC-17978|.*family.zzz|https://pypi.python.org/pypi/opensearchsql/|opensearch*|.*@amazon.com|.*email.com|.*github.com|https://github.com/.*|http://timestamp.verisign.com/scripts/timstamp.dll|.*/PowerBIConnector/bin/Release"
         env:
           GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
       - name: Fail if there were link errors

docs/dev/datasource-prometheus.md

@@ -0,0 +1,252 @@
+## 1.Overview
+
+In this document, we will propose a solution in OpenSearch Observability to query log data stored in S3.
+
+### 1.1.Problem Statements
+
+Currently, OpenSearch Observability is collection of plugins and applications that let you visualize data-driven events by using Piped Processing Language to explore, discover, and query data stored in OpenSearch. The major requirements we heard from customer are
+
+* **cost**, regarding to hardware cost of setup an OpenSearch cluster.
+* **ingestion performance,** it is not easy to supporting high throughput raw log ingestion.
+* **flexibility,** OpenSearch index required user know their query pattern before ingest data. Which is not flexible.
+
+Can build a new solution for OpenSearch observably uses and leverage S3 as storage. The benefits are
+
+* **cost efficiency**, comparing with OpenSearch, S3 is cheap.
+* **high ingestion throughput**, S3 is by design to support high throughput write.
+* **flexibility**, user do not need to worry about index mapping define and reindex. everything could be define at query tier.
+* **scalability**, user do not need to optimize their OpenSearch cluster for write. S3 is auto scale.
+* **data durability**, S3 provide 11s 9 of data durability.
+
+With all these benefits, are there any concerns? The **ability to query S3 in OpenSearch and query performance** are the major concerns. In this doc, we will provide the solution to solve these two major concerns.
+
+## 2.Terminology
+
+* **Catalog**. OpenSearch access external data source through catalog. For example, S3 catalog. Each catalog has attributes, most important attributes is data source access credentials.
+* **Table**: To access external data source. User should create external table to describe the schema and location. Table is the virtual concept which does not mapping to OpenSearch index.
+* **Materialized View**: User could create view from existing tables. Each view is 1:1 mapping to OpenSearch index. There are two types of views
+    * (1) Permanent view (default) which is fully managed by user.
+    * (2) Transient View which is maintained by Maximus automatically. user could decide when to drop the transient view.
+
+## 3.Requirements
+
+### 3.1.Use Cases
+
+#### Use Case - 1: pre-build and query metrics from log on s3
+
+**_Create_**
+
+* User could ingest the log or events directly to s3 with existing ingestion tools (e.g. [fluentbit](https://docs.fluentbit.io/manual/pipeline/outputs/s3)). The ingested log should be partitioned. e.g. _s3://my-raw-httplog/region/yyyy/mm/dd_.
+* User configure s3 Catalog in OpenSearch. By default, s3 connector use [ProfileCredentialsProvider](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/profile/ProfileCredentialsProvider.html).
+
+```
+settings.s3.access_key: xxxxx
+settings.s3.secret_key: xxxxx
+```
+
+* User create table in OpenSearch of their log data on s3. Maximus will create table httplog in the s3 catalog
+
+```
+CREATE EXTERNAL TABLE `s3`.`httplog`(
+   @timestamp timestamp,
+   clientip string,
+   request string,
+   state integer,
+   size long
+   )
+ROW FORMAT SERDE
+   'json'
+   'grok', <timestamp> <className>
+PARTITION BY
+   ${region}/${year}/${month}/${day}/
+LOCATION
+   's3://my-raw-httplog/';
+```
+
+* User create the view *failEventsMetrics*. Note: User could only create view from schema-defined table in Maximus
+
+```
+CREATE MATERIALIZED VIEW failEventsMetrics (
+  cnt    long,
+  time   timestamp,
+  status string
+)
+AS source=`s3`.`httpLog` status != 200 | status count() as cnt by span(5mins), status
+WITH (
+    REFRESH=AUTO
+)
+```
+
+* Maximus will (1) create view *failEventsMetrics* in the default catalog. (2) create index *failEventsMetrics* in the OpenSearch cluster. (3) refresh *failEventsMetrics* index with logs on s3*.* _Notes: Create view return success when operation (1) and (2) success. Refresh view is an async task will be executed in background asynchronously._
+* User could describe the view to monitor the status.
+
+```
+DESCRIBE/SHOW MATERIALIZED VIEW failEventsMetrics
+
+# Return
+status: INIT | IN_PROGRESS | READY
+```
+
+_**Query**_
+
+* User could query the view.
+
+```
+source=failEventsMetrics time in last 7 days
+```
+
+* User could query the table, Thunder will rewrite table query as view query when optimizing the query.
+
+```
+source=`s3`.`httpLog` status != 200 | status count() as cnt by span(1hour), status
+```
+
+_**Drop**_
+
+
+* User could drop the table. Maximus will delete httpLog metadata from catalog and associated view drop.
+
+```
+DROP TABLE `s3`.`httpLog`
+```
+
+* User could drop the view. Maximus will delete failEventsMetrics metadata from catalog and delete failEventsMetrics indices.
+
+```
+DROP MATERIALIZED VIEW failEventsMetrics
+```
+
+**Access Control**
+
+* User could not configure any permission on table.
+* User could not directly configure any permission on view. But user could configure permission on index associated with the view.
+  ![image1](https://user-images.githubusercontent.com/2969395/182239505-a8541ec1-f46f-4b91-882a-8a4be36f5aea.png)
+
+#### Use Case - 2: Ad-hoc s3 query in OpenSearch
+
+**_Create_**
+
+* Similar as previous example, User could ingest the log or events directly to s3 with existing ingestion tools. create catalog and table from data on s3.
+
+_**Query**_
+
+* User could query table without create view. At runtime, Maximus will create the transient view and populate the view with required data.
+
+```
+source=`s3`.`httpLog` status != 200 | status count() as cnt by span(5mins), status
+```
+
+_**List**_
+
+* User could list all the view of a table no matter how the view is created. User could query/drop/describe the temp view create by Maximus, as same as user created view.
+
+```
+LIST VIEW on `s3`.`httpLog`
+
+# return
+httplog-tempView-xxxx
+```
+
+**Access Control**
+
+* User could not configure any permission on table. During query time, Maximus will use the credentials to access s3 data. It is user’s ownership to configure the permission on s3 data.
+* User could not directly configure any permission on view. But user could configure permission on index associated with the view.
+  ![image2](https://user-images.githubusercontent.com/2969395/182239672-72b2cfc6-c22e-4279-b33e-67a85ee6a778.png)
+
+### 3.2.Function Requirements
+
+#### Query S3
+
+* User could query time series data on S3 by using PPL in OpenSearch.
+* For querying time series data in S3, user must create a **Catalog** for S3 with required setting.
+    * access key and secret key
+    * endpoint
+* For querying time series data in S3, user must create a **Table** of time series data on S3.
+
+#### View
+
+* Support create materialized view from time series data on S3.
+* Support fully materialized view refresh
+* Support manually materialized view incrementally refresh
+* Support automatically materialized view incrementally refresh
+* Support hybrid scan
+* Support drop materialized view
+* Support show materialized view
+
+#### Query Optimization
+
+* Inject optimizer rule to rewrite the query with MV to avoid S3 scan
+
+#### Automatic query acceleration
+
+* Automatically select view candidate based on OpenSearch - S3 query execution metrics
+* Store workload and selected view info for visualization and user intervention
+* Automatically create/refresh/drop view.
+
+#### S3 data format
+
+* The time series data could be compressed with gzip.
+* The time series data file format could be.
+
+    * JSON
+    * TSV
+
+* If the time series data should be partitioned and have snapshot id. Query engine could support automatically incremental refresh and hybrid scan.
+
+#### Resource Management
+
+* Support circuit breaker based resource control when executing a query.
+* Support task based resource manager
+
+#### Fault Tolerant
+
+* For fast query process, we scarify Fault Tolerant. Support query fast failure in case of hardware failure.
+
+#### Setting
+
+* Support configure of x% of disk automatically create view should used.
+
+### 3.3.Non Function Requirements
+
+#### Security:
+
+There are three types of privileges that are related to materialized views
+
+* Privileges directly on the materialized view itself
+* Privileges on the objects (e.g. table) that the materialized view accesses.
+
+*_materialized view itself_*
+
+* User could not directly configure any access control on view. But user could configure any access control on index associated with the view. In the other words, materialized inherits all the access control on the index associated with view.
+* When **automatically** **refresh view**, Maximus will use the backend role. It required the backend role has permission to index data.
+
+*_objects (e.g. table) that the materialized view accesses_*
+
+* As with non-materialized views, a user who wishes to access a materialized view needs privileges only on the view, not on the underlying object(s) that the view references.
+
+*_table_*
+
+* User could not configure any privileges on table. *_Note: because the table query could be rewrite as view query If the user do not have required permission to access the view. User could still get no permission exception._*
+
+*_objects (e.g. table) that the table refer_*
+
+* The s3 access control is only evaluated during s3 access. if the table access is rewrite as view access. The s3 access control will not take effect.
+
+*_Encryption_*
+
+* Materialized view data will be encrypted at rest.
+
+#### Others
+
+* Easy to use: the solution should be designed easy to use. It should just work out of the box and provide good performance with minimal setup.
+* **Performance**: Use **http_log** dataset to benchmark with OpenSearch cluster.
+* Scalability: The solution should be scale horizontally.
+* **Serverless**: The solution should be designed easily deployed as Serverless infra.
+* **Multi-tenant**: The solution should support multi tenant use case.
+* **Metrics**: Todo
+* **Log:** Todo
+
+## 4.What is, and what is not
+
+* We design and optimize for observability use cases only. Not OLTP and OLAP.
+* We only support time series log data on S3. We do not support query arbitrary data on S3.

docs/dev/datasource-query-s3.md

@@ -0,0 +1,39 @@
+# OpenSearch SQL Engine Architecture
+
+---
+## 1.Overview
+
+The OpenSearch SQL (OD-SQL) project is developed based on NLPChina project (https://github.com/NLPchina/elasticsearch-sql) which has been deprecated now ([attributions](https://github.com/opensearch-project/sql/blob/main/docs/attributions.md)). Over the one year in development, a lot of features have been added to the OD-SQL project on top of the existing older NLPChina project. The purpose of this document is to explain the OD-SQL current architecture going ahead.
+
+---
+## 2.High Level View
+
+In the high level, the OD-SQL Engine could be divided into four major sub-module.
+
+* *Parser*: Currently, there are two Lex&Parser coexists. The Druid Lex&Parser is the original one from NLPChina. The input AST of Core Engine is from the Druid Lex&Parser. The [ANTLR](https://github.com/opensearch-project/sql/blob/main/legacy/src/main/antlr/OpenSearchLegacySqlLexer.g4) Lex&Parser is added by us to customized the verification and exception handling.
+* *Analyzer*: The analyzer module take the output from ANTLR Lex&Parser then perform syntax and semantic analyze.
+* *Core Engine*: The QueryAction take the output from Druid Lex&Parser and translate to the OpenSearch DSL if possible. This is an NLPChina original module. The QueryPlanner Builder is added by us to support the JOIN and Post-processing logic. The QueryPlanner will take the output from Druid Lex&Parser and build the PhysicalPlan
+* *Execution*: The execution module execute QueryAction or QueryPlanner and return the response to the client. Different from the Frontend, Analyzer and Core Engine which running on the Transport Thread and can’t do any blocking operation. The Execution module running on the client threadpool and can perform the blocking operation.
+
+There are also others modules include in the OD-SQL engine.
+
+* _Documentation_: it is used to auto-generated documentation.
+* _Metrics_: it is used to collect OD-SQL related metrics.
+* _Resource Manager_:  it is used to monitor the memory consumption when performing join operation to avoid the impact to OpenSearch availability.
+
+![Architecture Overview](img/architecture-overview.png)
+
+---
+## 3.Journey of the query in OD-SQL engine.
+
+The following diagram take a sample query and explain how the query flow within different modules.
+
+![Architecture Journey](img/architecture-journey.png)
+
+1. The ANTRL parser based on grammar file (https://github.com/opensearch-project/sql/blob/main/legacy/src/main/antlr/OpenSearchLegacySqlParser.g4) to auto generate the AST. 
+2. The Syntax and Semantic Analyzer will walk through the AST and verify whether the query is follow the grammar and supported by the OD-SQL. e.g. *SELECT * FROM semantics WHERE LOG(age, city) = 1, *will throw exception with message* Function [LOG] cannot work with [INTEGER, KEYWORD]. *and sample usage message* Usage: LOG(NUMBER T) → DOUBLE.
+3. The Druid Lex&Parser takes the input query and generate the druid AST which is different from the AST generated by the ANTRL. This module is the open source library (https://github.com/alibaba/druid) used by NLPChina originally.
+4. The QueryPlanner Builder take the AST as input and generate the LogicalPlan from it. Then it optimize the LogicalPlan to PhysicalPlan.(In current implementation, only rule-based model is implemented). The major part of PhysicalPlan generation use NLPChina’s original logic to translate the SQL expression in AST to OpenSearch DSL.
+5. The QueryPlanner executor execute the PhysicalPlan in worker thread.
+6. The formatter will reformat the response data to the required format. The default format is JDBC format.
+

docs/dev/img/hash-join-benchmark/newoldhasherr.png

@@ -0,0 +1,76 @@
+# SQL Engine V2 - Release Notes
+
+---
+## 1.Motivations
+
+The current SQL query engine provides users the basic query capability for using familiar SQL rather than complex OpenSearch DSL. Based on NLPchina ES-SQL, many new features have been added additionally, such as semantic analyzer, semi-structured data query support, Hash Join etc. However, as we looked into more advanced SQL features, challenges started emerging especially in terms of correctness and extensibility (see [Attributions](../attributions.md)). After thoughtful consideration, we decided to develop a new query engine to address all the problems met so far.
+
+
+---
+## 2.What's New
+
+With the architecture and extensibility improved significantly, the following SQL features are able to be introduced in the new query engine:
+
+* **Language Structure**
+    * [Identifiers](../../docs/user/general/identifiers.rst): added support for identifier names with special characters
+    * [Data types](../../docs/user/general/datatypes.rst): added support for date and interval types
+    * [Expressions](../../docs/user/dql/expressions.rst): complex nested expression support
+    * [SQL functions](../../docs/user/dql/functions.rst): more date function support, `ADDDATE`, `DATE_ADD`, `DATE_SUB`, `DAY`, `DAYNAME`, `DAYOFMONTH`, `DAYOFWEEK`, `DAYOFYEAR`, `FROM_DAYS`, `HOUR`, `MICROSECOND`, `MINUTE`, `QUARTER`, `SECOND`, `SUBDATE`, `TIME`, `TIME_TO_SEC`, `TO_DAYS`, `WEEK`
+    * [Comments](../../docs/user/general/comments.rst): SQL comment support
+* **Basic queries**
+    * [HAVING without GROUP BY clause](../../docs/user/dql/aggregations.rst#having-without-group-by)
+    * [Aggregate over arbitrary expression](../../docs/user/dql/aggregations.rst#expression)
+    * [Ordering by NULLS FIRST/LAST](../../docs/user/dql/basics.rst#example-2-specifying-order-for-null)
+    * [Ordering by aggregate function](../../docs/user/dql/basics.rst#example-3-ordering-by-aggregate-functions)
+* **Complex queries**
+    * [Subqueries in FROM clause](../../docs/user/dql/complex.rst#example-2-subquery-in-from-clause): support arbitrary nesting level and aggregation
+* **Advanced Features**
+    * [Window functions](../../docs/user/dql/window.rst): ranking and aggregate window functions
+    * [Selective aggregation](../../docs/user/dql/aggregations.rst#filter-clause): by standard `FILTER` function
+* **Beyond SQL**
+    * [Semi-structured data query](../../docs/user/beyond/partiql.rst#example-2-selecting-deeper-levels): support querying OpenSearch object fields on arbitrary level
+    * OpenSearch multi-field: handled automatically and users won't have the access, ex. `text` is converted to `text.keyword` if it’s a multi-field
+
+As for correctness, besides full coverage of unit and integration test, we developed a new comparison test framework to ensure correctness by comparing with other databases. Please find more details in [Testing](./testing-comparison-test.md).
+
+
+---
+## 3.What're Changed
+
+### 3.1 Breaking Changes
+
+Because of implementation changed internally, you can expect Explain output in a different format. For query protocol, there are slightly changes in the default response format:
+
+* **Total**: The `total` field represented how many documents matched in total no matter how many returned (indicated by `size` field). However, this field becomes meaningless because of post processing on DSL response in the new query engine. Thus, for now the total number is always same as size field.
+
+### 3.2 Fallback Mechanism
+
+For these unsupported features, the query will be forwarded to the old query engine by fallback mechanism. To avoid impact on your side, normally you won't see any difference in a query response. If you want to check if and why your query falls back to be handled by old SQL engine, please explain your query and check OpenSearch log for "Request is falling back to old SQL engine due to ...".
+
+For the following features unsupported in the new engine, the query will be forwarded to the old query engine and thus you cannot use new features listed above:
+
+* **Cursor**: request with `fetch_size` parameter
+* **JSON response format**: was used to return OpenSearch DSL which is not accessible now. Replaced by default format in the new engine which is also in JSON.
+* **Nested field query**: including supports for nested field query
+* **JOINs**: including all types of JOIN queries
+* **OpenSearch functions**: fulltext search, metric and bucket functions
+
+### 3.3 Limitations
+
+You can find all the limitations in [Limitations](../../docs/user/limitations/limitations.rst). 
+
+
+---
+## 4.How it's Implemented
+
+If you're interested in the new query engine, please find more details in [Developer Guide](../../DEVELOPER_GUIDE.rst), [Architecture](./intro-architecture.md) and other docs in the dev folder.
+
+
+---
+## 5.What's Next
+
+As aforementioned, there are still popular SQL features unsupported in the new query engine yet. In particular, the following items are on our roadmap with high priority:
+
+1. Nested field queries
+2. JOIN support
+3. OpenSearch functions