Update 3.4 release Note (#4923) (#4935)

* Update 3.4 doc



* update



* updates



* Update



---------


(cherry picked from commit c87f99f)

Signed-off-by: Kai Huang <ahkcs@amazon.com>

Author: ahkcs

prometheus/src/test/java/org/opensearch/sql/prometheus/storage/PrometheusStorageFactoryTest.java

@@ -130,7 +130,7 @@ void testGetStorageEngineWithWrongAuthType() {
         .thenReturn(Collections.emptyList());
     PrometheusStorageFactory prometheusStorageFactory = new PrometheusStorageFactory(settings);
     HashMap<String, String> properties = new HashMap<>();
-    properties.put("prometheus.uri", "https://test.com");
+    properties.put("prometheus.uri", "https://opensearch.org");
     properties.put("prometheus.auth.type", "random");
     properties.put("prometheus.auth.region", "us-east-1");
     properties.put("prometheus.auth.secret_key", "accessKey");
@@ -150,7 +150,7 @@ void testGetStorageEngineWithNONEAuthType() {
         .thenReturn(Collections.emptyList());
     PrometheusStorageFactory prometheusStorageFactory = new PrometheusStorageFactory(settings);
     HashMap<String, String> properties = new HashMap<>();
-    properties.put("prometheus.uri", "https://test.com");
+    properties.put("prometheus.uri", "https://opensearch.org");
     StorageEngine storageEngine = prometheusStorageFactory.getStorageEngine(properties);
     Assertions.assertTrue(storageEngine instanceof PrometheusStorageEngine);
   }

release-notes/opensearch-sql.release-notes-3.4.0.0.md

@@ -0,0 +1,155 @@
+
+## Version 3.4.0 Release Notes
+
+Compatible with OpenSearch and OpenSearch Dashboards version 3.4.0
+
+### Features
+* Support `chart` command in PPL ([#4579](https://github.com/opensearch-project/sql/pull/4579))
+* Support `Streamstats` command with calcite ([#4297](https://github.com/opensearch-project/sql/pull/4297))
+* Support `multisearch` command in calcite ([#4332](https://github.com/opensearch-project/sql/pull/4332))
+* Add replace command with Calcite ([#4451](https://github.com/opensearch-project/sql/pull/4451))
+
+### Enhancements
+* Add `bucket_nullable` argument for `Streamstats` command ([#4831](https://github.com/opensearch-project/sql/pull/4831))
+* Add `regexp_replace()` function as alias of `replace()` ([#4765](https://github.com/opensearch-project/sql/pull/4765))
+* Convert `dedup` pushdown to composite + top_hits ([#4844](https://github.com/opensearch-project/sql/pull/4844))
+* Merge group fields for aggregate if having dependent group fields ([#4703](https://github.com/opensearch-project/sql/pull/4703))
+* Merge the implementation of `timechart` and `chart` ([#4755](https://github.com/opensearch-project/sql/pull/4755))
+* PPL tostring() implementation issue #4492 ([#4497](https://github.com/opensearch-project/sql/pull/4497))
+* Perform RexNode expression standardization for script push down ([#4795](https://github.com/opensearch-project/sql/pull/4795))
+* Pushdown sort by complex expressions to scan ([#4750](https://github.com/opensearch-project/sql/pull/4750))
+* Pushdown the `top` `rare` commands to nested aggregation ([#4707](https://github.com/opensearch-project/sql/pull/4707))
+* Refactor alias type field by adding another project with alias ([#4881](https://github.com/opensearch-project/sql/pull/4881))
+* Remove count aggregation for sort on aggregate measure ([#4867](https://github.com/opensearch-project/sql/pull/4867))
+* Remove redundant push-down-filters derived for bucket-non-null agg ([#4843](https://github.com/opensearch-project/sql/pull/4843))
+* Remove unnecessary filter for DateHistogram aggregation ([#4877](https://github.com/opensearch-project/sql/pull/4877))
+* Specify timestamp field with `timefield` in timechart command ([#4784](https://github.com/opensearch-project/sql/pull/4784))
+* Support `appendpipe` command in PPL ([#4602](https://github.com/opensearch-project/sql/pull/4602))
+* Support `mvdedup` eval function ([#4828](https://github.com/opensearch-project/sql/pull/4828))
+* Support `mvindex` eval function ([#4794](https://github.com/opensearch-project/sql/pull/4794))
+* Support push down sort on aggregation measure for more than one agg call ([#4759](https://github.com/opensearch-project/sql/pull/4759))
+* Support wildcard for replace command ([#4698](https://github.com/opensearch-project/sql/pull/4698))
+* Add `bucket_nullable` argument for `Eventstats` ([#4817](https://github.com/opensearch-project/sql/pull/4817))
+* Bin command error message enhancement ([#4690](https://github.com/opensearch-project/sql/pull/4690))
+* Update clickbench queries with parameter bucket_nullable=false ([#4732](https://github.com/opensearch-project/sql/pull/4732))
+* Support 'usenull' option in PPL `top` and `rare` commands ([#4696](https://github.com/opensearch-project/sql/pull/4696))
+* Support millisecond span ([#4672](https://github.com/opensearch-project/sql/pull/4672))
+* Enhance dynamic source clause to support only metadata filters ([#4554](https://github.com/opensearch-project/sql/pull/4554))
+* Support push down sort after limit ([#4657](https://github.com/opensearch-project/sql/pull/4657))
+* Add `per_minute`, `per_hour`, `per_day` function support ([#4531](https://github.com/opensearch-project/sql/pull/4531))
+* Pushdown sort aggregate metrics ([#4603](https://github.com/opensearch-project/sql/pull/4603))
+* Allow renaming group-by fields to existing field names ([#4586](https://github.com/opensearch-project/sql/pull/4586))
+* Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions ([#4599](https://github.com/opensearch-project/sql/pull/4599))
+* Pushdown case function in aggregations as range queries ([#4400](https://github.com/opensearch-project/sql/pull/4400))
+* Update GEOIP function to support IP types as input ([#4613](https://github.com/opensearch-project/sql/pull/4613))
+* Pushdown distinct count approx ([#4614](https://github.com/opensearch-project/sql/pull/4614))
+* Optimize pushdown script size with necessary fields per expression ([#4615](https://github.com/opensearch-project/sql/pull/4615))
+* Support referring to implicit `@timestamp` field in span ([#4138](https://github.com/opensearch-project/sql/pull/4138))
+* Make composite bucket size configurable ([#4544](https://github.com/opensearch-project/sql/pull/4544))
+* Add internal MAP_REMOVE function for Calcite PPL ([#4511](https://github.com/opensearch-project/sql/pull/4511))
+* Add MAP_APPEND internal function to Calcite PPL ([#4515](https://github.com/opensearch-project/sql/pull/4515))
+* Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance ([#4569](https://github.com/opensearch-project/sql/pull/4569))
+* [Enhancement] Error handling for illegal character usage in java regex named capture group ([#4434](https://github.com/opensearch-project/sql/pull/4434))
+* Add JSON_EXTRACT_ALL internal function for Calcite PPL ([#4489](https://github.com/opensearch-project/sql/pull/4489))
+* Set 0 and negative value of subsearch.maxout as unlimited ([#4534](https://github.com/opensearch-project/sql/pull/4534))
+* Add configurable system limitations for `subsearch` and `join` command ([#4501](https://github.com/opensearch-project/sql/pull/4501))
+* Add `per_second` function support for `timechart` command ([#4464](https://github.com/opensearch-project/sql/pull/4464))
+* Add MAP_CONCAT internal function ([#4477](https://github.com/opensearch-project/sql/pull/4477))
+* Support Regex for replace eval function ([#4456](https://github.com/opensearch-project/sql/pull/4456))
+* Add data anonymizer for spath command ([#4479](https://github.com/opensearch-project/sql/pull/4479))
+* Support eval returns decimal division result instead of integer ([#4440](https://github.com/opensearch-project/sql/pull/4440))
+* Add mvappend function for Calcite PPL ([#4438](https://github.com/opensearch-project/sql/pull/4438))
+* PPL `fillnull` command enhancement ([#4421](https://github.com/opensearch-project/sql/pull/4421))
+* Support format=yaml in Explain API ([#4446](https://github.com/opensearch-project/sql/pull/4446))
+### Bug Fixes
+* Add hashCode() and equals() to the value class of ExprJavaType ([#4885](https://github.com/opensearch-project/sql/pull/4885))
+* BucketAggretationParser should handle more non-composite bucket types ([#4706](https://github.com/opensearch-project/sql/pull/4706))
+* Do not remove nested fields in resolving AllFieldsExcludeMeta ([#4708](https://github.com/opensearch-project/sql/pull/4708))
+* Fix binning udf resolution / Add type coercion support for binning UDFs ([#4742](https://github.com/opensearch-project/sql/pull/4742))
+* Fix bug that `Streamstats` command incorrectly treats null as a valid group ([#4777](https://github.com/opensearch-project/sql/pull/4777))
+* Fix filter push down producing redundant filter queries ([#4744](https://github.com/opensearch-project/sql/pull/4744))
+* Fix function identify problem in converting to sql dialect ([#4793](https://github.com/opensearch-project/sql/pull/4793))
+* Fix search anonymizer only ([#4783](https://github.com/opensearch-project/sql/pull/4783))
+* Fix sub-fields accessing of generated structs ([#4683](https://github.com/opensearch-project/sql/pull/4683))
+* Fix wrong parameter and return result logic for LogPatternAggFunction ([#4868](https://github.com/opensearch-project/sql/pull/4868))
+* Grouping key field type can only be overwritten when the `ExprCoreType`s are different ([#4850](https://github.com/opensearch-project/sql/pull/4850))
+* Fix eval on grouped fields after timechart ([#4758](https://github.com/opensearch-project/sql/pull/4758))
+* Support access to nested field of struct after fields command ([#4719](https://github.com/opensearch-project/sql/pull/4719))
+* Support escaped field names in SPath parsing ([#4813](https://github.com/opensearch-project/sql/pull/4813))
+* Support script pushdown in sort-on-measure pushdown rewriting ([#4749](https://github.com/opensearch-project/sql/pull/4749))
+* Support serializing external OpenSearch UDFs at pushdown time ([#4618](https://github.com/opensearch-project/sql/pull/4618))
+* Support using decimal as span literals ([#4717](https://github.com/opensearch-project/sql/pull/4717))
+* Translate `SAFE_CAST` to `TRY_CAST` in Spark SQL ([#4788](https://github.com/opensearch-project/sql/pull/4788))
+* Update syntax: like(string, PATTERN[, case_sensitive]) ([#4837](https://github.com/opensearch-project/sql/pull/4837))
+* [BugFix] Fix Memory Exhaustion for Multiple Filtering Operations in PPL ([#4841](https://github.com/opensearch-project/sql/pull/4841))
+* Fix CVE-2025-48924 ([#4665](https://github.com/opensearch-project/sql/pull/4665))
+* [BugFix] Fix unexpected shift of extraction for `rex` with nested capture groups in named groups ([#4641](https://github.com/opensearch-project/sql/pull/4641))
+* Fix asc/desc keyword behavior for sort command ([#4651](https://github.com/opensearch-project/sql/pull/4651))
+* Fixes for `Multisearch` and `Append` command ([#4512](https://github.com/opensearch-project/sql/pull/4512))
+* Make nested alias type support referring to outer context ([#4673](https://github.com/opensearch-project/sql/pull/4673))
+* Use table scan rowType in filter pushdown to fix rename issue ([#4670](https://github.com/opensearch-project/sql/pull/4670))
+* Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations ([#4621](https://github.com/opensearch-project/sql/pull/4621))
+* Fix bin nested fields issue ([#4606](https://github.com/opensearch-project/sql/pull/4606))
+* Change ComparableLinkedHashMap to compare Key than Value ([#4648](https://github.com/opensearch-project/sql/pull/4648))
+* Replace all dots in fields of table scan's PhysType ([#4633](https://github.com/opensearch-project/sql/pull/4633))
+* Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue ([#4629](https://github.com/opensearch-project/sql/pull/4629))
+* Fix filter parsing failure on date fields with non-default format ([#4616](https://github.com/opensearch-project/sql/pull/4616))
+* Fix compile issue in main ([#4608](https://github.com/opensearch-project/sql/pull/4608))
+* Fix push down failure for min/max on derived field ([#4572](https://github.com/opensearch-project/sql/pull/4572))
+* Add value type hint for derived aggregate group by field ([#4583](https://github.com/opensearch-project/sql/pull/4583))
+* Fix sort push down into agg after project already pushed ([#4546](https://github.com/opensearch-project/sql/pull/4546))
+* Update request builder after pushdown sort into agg buckets ([#4541](https://github.com/opensearch-project/sql/pull/4541))
+* Including metadata fields type when doing agg/filter script push down ([#4522](https://github.com/opensearch-project/sql/pull/4522))
+* Fix percentile bug ([#4539](https://github.com/opensearch-project/sql/pull/4539))
+* Fix mapping after aggregation push down ([#4500](https://github.com/opensearch-project/sql/pull/4500))
+* Throw an error when the conditions of case are not boolean values ([#4520](https://github.com/opensearch-project/sql/pull/4520))
+* Fallback to sub-aggregation if composite aggregation doesn't support ([#4413](https://github.com/opensearch-project/sql/pull/4413))
+* Fix the bug of explicit makeNullLiteral for UDT fields ([#4475](https://github.com/opensearch-project/sql/pull/4475))
+* Fix missing keywordsCanBeId ([#4491](https://github.com/opensearch-project/sql/pull/4491))
+* Fix issue 4441 ([#4449](https://github.com/opensearch-project/sql/pull/4449))
+* Fix join type ambiguous issue when specify the join type with sql-like join criteria ([#4474](https://github.com/opensearch-project/sql/pull/4474))
+* Remove shared mutable optimizer field that caused race condition ([#4454](https://github.com/opensearch-project/sql/pull/4454))
+* Reverting to _doc + _id ([#4435](https://github.com/opensearch-project/sql/pull/4435))
+
+### Infrastructure
+* Add config for CodeRabbit review ([#4890](https://github.com/opensearch-project/sql/pull/4890))
+* Split bwc-tests to bwc-rolling-upgrade and bwc-full-restart ([#4716](https://github.com/opensearch-project/sql/pull/4716))
+* Update github workflows to move from macos-13 to 14 ([#4779](https://github.com/opensearch-project/sql/pull/4779))
+* Fix the flaky CalcitePPLTcphIT ([#4846](https://github.com/opensearch-project/sql/pull/4846))
+* Fix UT failure and Linkchecker failure ([#4809](https://github.com/opensearch-project/sql/pull/4809))
+* Adding IT suite for PPL-based dashboards in Neo for CloudWatch Lake ([#4695](https://github.com/opensearch-project/sql/pull/4695))
+* Add allowed_warnings in yaml restful tests ([#4731](https://github.com/opensearch-project/sql/pull/4731))
+* Mitigate the CI failure caused by 500 Internal Server Error ([#4646](https://github.com/opensearch-project/sql/pull/4646))
+* Publish internal modules separately for downstream reuse ([#4484](https://github.com/opensearch-project/sql/pull/4484))
+* Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT ([#4623](https://github.com/opensearch-project/sql/pull/4623))
+* Onboarding async query core and grammar files to maven snapshots ([#4598](https://github.com/opensearch-project/sql/pull/4598))
+* Onboarding new maven snapshots publishing to s3 (sql) ([#4588](https://github.com/opensearch-project/sql/pull/4588))
+* Fix JsonExtractAllFunctionIT failure ([#4556](https://github.com/opensearch-project/sql/pull/4556))
+* Check server status before starting Prometheus ([#4537](https://github.com/opensearch-project/sql/pull/4537))
+* Update stalled action ([#4485](https://github.com/opensearch-project/sql/pull/4485))
+* Switch to Guice#createInjector and add concurrent SQL/PPL regression ITs ([#4462](https://github.com/opensearch-project/sql/pull/4462))
+* Update delete_backport_branch workflow to include release-chores branches ([#4025](https://github.com/opensearch-project/sql/pull/4025))
+* Add ignorePrometheus Flag for integTest and docTest ([#4442](https://github.com/opensearch-project/sql/pull/4442))
+
+### Documentation
+* Update PPL Command Documentation ([#4562](https://github.com/opensearch-project/sql/pull/4562))
+* Doc update for `json_valid` ([#4803](https://github.com/opensearch-project/sql/pull/4803))
+* Enhance tests and doc for eval isnull/isnotnull functions ([#4724](https://github.com/opensearch-project/sql/pull/4724))
+* Update search.rst documentation ([#4686](https://github.com/opensearch-project/sql/pull/4686))
+* Add more examples to the `where` command doc ([#4457](https://github.com/opensearch-project/sql/pull/4457))
+* Update eventstats.rst ([#4447](https://github.com/opensearch-project/sql/pull/4447))
+
+### Maintenance
+* Bump Calcite to 1.41.0 ([#4714](https://github.com/opensearch-project/sql/pull/4714))
+* Execute yamlRestTest in integration job ([#4838](https://github.com/opensearch-project/sql/pull/4838))
+* Fix test failures due to version in mapping ([#4748](https://github.com/opensearch-project/sql/pull/4748))
+* Support timeouts for Calcite queries ([#4857](https://github.com/opensearch-project/sql/pull/4857))
+* [Maintenance] Enforce PR label of 'bugFix' instead of 'bug' ([#4773](https://github.com/opensearch-project/sql/pull/4773))
+* [3.4.0] Bump Gradle to 9.2.0 and GitHub Action JDK to 25 ([#4824](https://github.com/opensearch-project/sql/pull/4824))
+* Fix clickbench query 43 ([#4861](https://github.com/opensearch-project/sql/pull/4861))
+* Update big5 ppl queries and check plans ([#4668](https://github.com/opensearch-project/sql/pull/4668))
+* Revert "Update grammar files and developer guide (#4301)" ([#4643](https://github.com/opensearch-project/sql/pull/4643))
+* Increment version to 3.4.0-SNAPSHOT ([#4452](https://github.com/opensearch-project/sql/pull/4452))
+* Revert partial of #4401 ([#4503](https://github.com/opensearch-project/sql/pull/4503))
+* Implement one-batch lookahead for index enumerators ([#4345](https://github.com/opensearch-project/sql/pull/4345))
+* Refactor name resolution in Calcite PPL ([#4393](https://github.com/opensearch-project/sql/pull/4393))