opensearch-sql.release-notes-3.4.0.0.md

Version 3.4.0 Release Notes

Compatible with OpenSearch and OpenSearch Dashboards version 3.4.0

Features

• Support chart command in PPL (#4579)
• Support Streamstats command with calcite (#4297)
• Support multisearch command in calcite (#4332)
• Support replace command in Calcite (#4451)
• Support mvdedup eval function (#4828)
• Support mvindex eval function (#4794)
• Support mvappend function (#4438)
• Support per_second function for timechart command (#4464)
• Support per_minute, per_hour, per_day function (#4531)
• Support appendpipe command in PPL (#4602)
• Support tostring() eval function (#4497)

Enhancements

• Add bucket_nullable argument for Streamstats command (#4831)
• Add regexp_replace() function as alias of replace() (#4765)
• Convert dedup pushdown to composite + top_hits (#4844)
• Merge group fields for aggregate if having dependent group fields (#4703)
• Merge the implementation of timechart and chart (#4755)
• Perform RexNode expression standardization for script push down (#4795)
• Pushdown sort by complex expressions to scan (#4750)
• Pushdown the top rare commands to nested aggregation (#4707)
• Refactor alias type field by adding another project with alias (#4881)
• Remove count aggregation for sort on aggregate measure (#4867)
• Remove redundant push-down-filters derived for bucket-non-null agg (#4843)
• Remove unnecessary filter for DateHistogram aggregation (#4877)
• Specify timestamp field with timefield in timechart command (#4784)
• Support push down sort on aggregation measure for more than one agg call (#4759)
• Support wildcard for replace command (#4698)
• Add bucket_nullable argument for Eventstats (#4817)
• Bin command error message enhancement (#4690)
• Update clickbench queries with parameter bucket_nullable=false (#4732)
• Support 'usenull' option in PPL top and rare commands (#4696)
• Support millisecond span (#4672)
• Enhance dynamic source clause to support only metadata filters (#4554)
• Support push down sort after limit (#4657)
• Pushdown sort aggregate metrics (#4603)
• Allow renaming group-by fields to existing field names (#4586)
• Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (#4599)
• Pushdown case function in aggregations as range queries (#4400)
• Update GEOIP function to support IP types as input (#4613)
• Pushdown distinct count approx (#4614)
• Optimize pushdown script size with necessary fields per expression (#4615)
• Support referring to implicit @timestamp field in span (#4138)
• Make composite bucket size configurable (#4544)
• Add internal MAP_REMOVE function for Calcite PPL (#4511)
• Add MAP_APPEND internal function to Calcite PPL (#4515)
• Use _doc + _shard_doc as sort tiebreaker to get better performance (#4569)
• [Enhancement] Error handling for illegal character usage in java regex named capture group (#4434)
• Add JSON_EXTRACT_ALL internal function for Calcite PPL (#4489)
• Set 0 and negative value of subsearch.maxout as unlimited (#4534)
• Add configurable system limitations for subsearch and join command (#4501)
• Add MAP_CONCAT internal function (#4477)
• Support Regex for replace eval function (#4456)
• Add data anonymizer for spath command (#4479)
• Support eval returns decimal division result instead of integer (#4440)
• PPL fillnull command enhancement (#4421)
• Support format=yaml in Explain API (#4446)

Bug Fixes

• Add hashCode() and equals() to the value class of ExprJavaType (#4885)
• BucketAggretationParser should handle more non-composite bucket types (#4706)
• Do not remove nested fields in resolving AllFieldsExcludeMeta (#4708)
• Fix binning udf resolution / Add type coercion support for binning UDFs (#4742)
• Fix bug that Streamstats command incorrectly treats null as a valid group (#4777)
• Fix filter push down producing redundant filter queries (#4744)
• Fix function identify problem in converting to sql dialect (#4793)
• Fix search anonymizer only (#4783)
• Fix sub-fields accessing of generated structs (#4683)
• Fix wrong parameter and return result logic for LogPatternAggFunction (#4868)
• Grouping key field type can only be overwritten when the ExprCoreTypes are different (#4850)
• Fix eval on grouped fields after timechart (#4758)
• Support access to nested field of struct after fields command (#4719)
• Support escaped field names in SPath parsing (#4813)
• Support script pushdown in sort-on-measure pushdown rewriting (#4749)
• Support serializing external OpenSearch UDFs at pushdown time (#4618)
• Support using decimal as span literals (#4717)
• Translate SAFE_CAST to TRY_CAST in Spark SQL (#4788)
• Update syntax: like(string, PATTERN[, case_sensitive]) (#4837)
• [BugFix] Fix Memory Exhaustion for Multiple Filtering Operations in PPL (#4841)
• Fix CVE-2025-48924 (#4665)
• [BugFix] Fix unexpected shift of extraction for rex with nested capture groups in named groups (#4641)
• Fix asc/desc keyword behavior for sort command (#4651)
• Fixes for Multisearch and Append command (#4512)
• Make nested alias type support referring to outer context (#4673)
• Use table scan rowType in filter pushdown to fix rename issue (#4670)
• Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (#4621)
• Fix bin nested fields issue (#4606)
• Change ComparableLinkedHashMap to compare Key than Value (#4648)
• Replace all dots in fields of table scan's PhysType (#4633)
• Return comparable LinkedHashMap in valueForCalcite() of ExprTupleValue (#4629)
• Fix filter parsing failure on date fields with non-default format (#4616)
• Fix compile issue in main (#4608)
• Fix push down failure for min/max on derived field (#4572)
• Add value type hint for derived aggregate group by field (#4583)
• Fix sort push down into agg after project already pushed (#4546)
• Update request builder after pushdown sort into agg buckets (#4541)
• Including metadata fields type when doing agg/filter script push down (#4522)
• Fix percentile bug (#4539)
• Fix mapping after aggregation push down (#4500)
• Throw an error when the conditions of case are not boolean values (#4520)
• Fallback to sub-aggregation if composite aggregation doesn't support (#4413)
• Fix the bug of explicit makeNullLiteral for UDT fields (#4475)
• Fix missing keywordsCanBeId (#4491)
• Fix issue 4441 (#4449)
• Fix join type ambiguous issue when specify the join type with sql-like join criteria (#4474)
• Remove shared mutable optimizer field that caused race condition (#4454)
• Reverting to _doc + _id (#4435)

Infrastructure

• Add config for CodeRabbit review (#4890)
• Split bwc-tests to bwc-rolling-upgrade and bwc-full-restart (#4716)
• Update github workflows to move from macos-13 to 14 (#4779)
• Fix the flaky CalcitePPLTcphIT (#4846)
• Fix UT failure and Linkchecker failure (#4809)
• Adding IT suite for PPL-based dashboards in Neo for CloudWatch Lake (#4695)
• Add allowed_warnings in yaml restful tests (#4731)
• Mitigate the CI failure caused by 500 Internal Server Error (#4646)
• Publish internal modules separately for downstream reuse (#4484)
• Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (#4623)
• Onboarding async query core and grammar files to maven snapshots (#4598)
• Onboarding new maven snapshots publishing to s3 (sql) (#4588)
• Fix JsonExtractAllFunctionIT failure (#4556)
• Check server status before starting Prometheus (#4537)
• Update stalled action (#4485)
• Switch to Guice#createInjector and add concurrent SQL/PPL regression ITs (#4462)
• Update delete_backport_branch workflow to include release-chores branches (#4025)
• Add ignorePrometheus Flag for integTest and docTest (#4442)

Documentation

• Update PPL Command Documentation (#4562)
• Doc update for json_valid (#4803)
• Enhance tests and doc for eval isnull/isnotnull functions (#4724)
• Update search.rst documentation (#4686)
• Add more examples to the where command doc (#4457)
• Update eventstats.rst (#4447)

Maintenance

• Bump Calcite to 1.41.0 (#4714)
• Execute yamlRestTest in integration job (#4838)
• Fix test failures due to version in mapping (#4748)
• Support timeouts for Calcite queries (#4857)
• [Maintenance] Enforce PR label of 'bugFix' instead of 'bug' (#4773)
• [3.4.0] Bump Gradle to 9.2.0 and GitHub Action JDK to 25 (#4824)
• Fix clickbench query 43 (#4861)
• Update big5 ppl queries and check plans (#4668)
• Revert "Update grammar files and developer guide (#4301)" (#4643)
• Increment version to 3.4.0-SNAPSHOT (#4452)
• Revert partial of #4401 (#4503)
• Implement one-batch lookahead for index enumerators (#4345)
• Refactor name resolution in Calcite PPL (#4393)