Skip to content

Reverse indexed-parquet segment iteration for opposite-direction sorts#22056

Merged
bharath-techie merged 2 commits into
opensearch-project:mainfrom
alchemist51:sort-aware-segment-reversal
Jun 15, 2026
Merged

Reverse indexed-parquet segment iteration for opposite-direction sorts#22056
bharath-techie merged 2 commits into
opensearch-project:mainfrom
alchemist51:sort-aware-segment-reversal

Conversation

@alchemist51

@alchemist51 alchemist51 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Description

Mirror of Lucene's ContextIndexSearcher.shouldUseTimeSeriesDescSortOptimization for the indexed-parquet scan path (IndexedTableProvider / QueryShardExec). When index.sort.field is configured and the query's leading ORDER BY runs counter to the catalog's stored direction, reverse the per-shard segment iteration so a TopK above the scan pulls the highest-priority segment first and parquet page-stats pruning kicks in for the rest.

Reuses PR #22041's plumbing. That PR threads index.sort.field / index.sort.order from IndexSettings through DatafusionReaderManagerDatafusionReaderReaderHandledf_create_reader to ShardView.sort_fields / ShardView.sort_orders. This PR adds a second consumer of those same fields, on the indexed path, with no new Java or FFM surface area.

Where the optimization fires

indexed_executor.rs::execute_indexed_with_context_inner, after from_substrait_plan decodes the LogicalPlan:

let mut segments = segments;
if should_reverse_segments(analyze_top_sort(&logical_plan).as_ref(), &sort_fields, &sort_orders) {
    reverse_segment_iteration_order(&mut segments);
}

Three module-local helpers:

  • analyze_top_sort(&LogicalPlan) -> Option<TopSort> — walks Projection / Limit / SubqueryAlias / Distinct / Filter to find the first LogicalPlan::Sort. Returns the leading sort key's bare column name and direction. Returns None if no top-level Sort, or if the leading sort key is anything but a plain Expr::Column (function on the sort key breaks catalog monotonicity).
  • should_reverse_segments(top_sort, sort_fields, sort_orders) -> bool — true iff the catalog has a sort, the query's leading sort column matches sort_fields[0] exactly (case-sensitive, matching [Sandbox] Propagate index.sort.field to DataFusion ListingTable #22041's Column::from_name), and the directions differ. Multi-key sorts use only sort_fields[0] — segments aren't monotonic on a non-leading key.
  • reverse_segment_iteration_order(&mut [SegmentFileInfo]) — flips the slice in place. Per-segment global_base is deliberately not recomputed; see below.

QTF correctness — why global_base must be preserved

The __row_id__ emitted by the indexed scan is segment.global_base + position. QTF's fetch phase (api::fetch_by_row_ids) goes through a different code path (ShardTableProvider, not IndexedTableProvider) and rebuilds segments via build_segments(ShardView.object_metas, ShardView.writer_generations) — always in catalog order. Fetch then partition_points the cached row IDs against catalog-order global_bases to find the owning segment.

For the round trip to hold, query-phase IDs must match catalog-order bases. reverse_segment_iteration_order therefore flips iteration order only — each segment retains its catalog-order global_base. Reversal becomes purely a scheduling decision; the row-id space is unchanged. The optimization composes safely with QTF.

What this PR does not change

  • Vanilla path (ListingTable / DataSourceExec) — covered by [Sandbox] Propagate index.sort.field to DataFusion ListingTable #22041 via ListingOptions::with_file_sort_order(...) (different mechanism: advertise output_ordering so the planner can drop SortExec).
  • QueryShardExec.properties.equivalence_properties — no output-ordering claim. The TopK / SortPreservingMergeExec above the scan still re-sorts. Reversal is purely a pruning aid for sort-with-limit shapes; it doesn't try to make the scan's output-ordering claim load-bearing. That's a separate, more invasive change.

Tests

Rust unit tests in indexed_executor::tests:

  • analyze_top_sort_* — top-level desc, descends through Limit/Projection, None for no-sort, None for function sort key.
  • should_reverse_segments_* — matching field opposite direction, matching same direction, catalog desc query asc, no query sort, no catalog sort, non-leading catalog field, case-sensitive name.
  • reverse_segments_preserves_global_base, reverse_segments_empty_is_noop, reverse_segments_single_keeps_its_base.

Rust e2e tests in indexed_table/tests_e2e/sort_reverse_row_id.rs — pin the load-bearing invariant directly: build 3- and 4-segment fixtures with emit_row_ids: true, run the query in natural order vs reverse_segment_iteration_order, assert the set of emitted __row_id__ values is identical (sorted set equality). If global_base cumulation drifted under reversal, fetch would return wrong rows; this catches it locally.

Java IT in sandbox/qa/analytics-engine-coordinator/.../SortReverseQtfIT.java (parquet primary + Lucene secondary, 2 shards, 3 flushes → 2-4 parquet segments per shard, index.sort.field=ts asc):

  • testReversalTriggers_orderByTsDescLimit_correctResults — captures the reversal log line via MockLogAppender on RustLoggerBridge (proves the optimization fired) and asserts top-5 rows by ts DESC are correct via the QTF query→fetch round-trip.
  • testNoReversal_orderByTsAscLimit_correctResultsUnseenEventExpectation confirms reversal does NOT fire when query direction matches catalog; results still correct.
  • testReversal_keywordFilterDelegationPath_correctResultsWHERE category = 'C0' ORDER BY ts DESC LIMIT 4 exercises the keyword-filter delegation path (predicate shipped to Lucene perf peer) + reversal; asserts QTF row-id round-trip composes with delegation.

Verification

  • cargo test --lib: 1045 / 1045 pass (1043 pre-existing + 22 new unit tests + 2 new e2e tests, after dedupe).
  • :server, :sandbox:plugins:composite-engine, :sandbox:plugins:analytics-backend-datafusion, :sandbox:plugins:analytics-backend-lucene, :sandbox:qa:analytics-engine-coordinator all compileJava / compileTestJava / compileInternalClusterTestJava clean.

Benchmark results — full clickbench corpus

Measured on EC2 r8g.2xlarge (ARM64, 8 vCPU, 64 GB) with the full 100M-row OSB clickbench corpus (parquet primary + Lucene secondary, OSB sort [CounterID desc, EventDate desc, UserID desc, EventTime desc, WatchID desc]).

Same EC2 box, same data, same script, same 13 queries (10 reversal-eligible + 3 controls), 3 passes per query. Build A = upstream main without this PR; Build B = upstream main + this PR. min_ms is reported (warm cache, ignores pass-1 cold-cache outliers; mean is misleading because Q01 cold-cache is ~2300 ms before page cache warms).

All queries: where <field> = <literal> | sort + CounterID | head 1000 (catalog leading is CounterID desc+ triggers reversal). Selectivity is the number of rows matching the filter against the 100M corpus.

Query Filter (selectivity) Build A (no fix) Build B (with fix) Δ ms Δ %
Q01 BrowserLanguage = 'S0' (~92M) 478 388 −90 −18.8%
Q02 HitColor = '5' (~80M) 402 339 −63 −15.7%
Q03 BrowserCountry = 'h1' (~28M) 324 210 −114 −35.2%
Q04 MobilePhoneModel = 'iPad' (~5M) 328 186 −142 −43.3%
Q07 UTMSource = 'feedburner' (~572K) 67 67 0 flat
Q08 MobilePhoneModel = 'iPhone' (~229K) 89 88 −1 flat
Q10 compound (~thousands) 73 75 +2 flat
C1 same direction as catalog (sort - CounterID) 243 242 −1 flat ✓
C2 non-leading catalog key (sort + EventDate) 234 233 −1 flat ✓

(Q05/Q06/Q09 returned 0 rows on this corpus slice — literal absent from the data — and are omitted; their flat timings are uninformative.)

Reading the result:

  • High-selectivity queries (Q01–Q04, 5M–92M matching rows): 16% → 43% latency reduction. These are the cases where head 1000 against tens of millions of matching rows actually has work to do — TopK has to scan many segments before its threshold tightens. With reversal, the highest-CounterID segment is hit first, the threshold tightens fast, and parquet page stats prune most of the remaining segments.
  • Low-selectivity queries (Q07–Q10, ≤600K matching rows): flat. Finding the top 1000 is cheap regardless of scheduling order — work is dominated by the keyword-filter scan, not segment iteration. The optimization correctly does no harm where it can't help.
  • Control queries (C1 same direction, C2 non-leading sort key): identical timings between builds. Negative confirmation that the change is neutral on shapes where reversal must not fire.

Variance across warm passes is ~3–5 ms, well below the 60–140 ms deltas on Q01–Q04.

Related Issues / PRs

Builds on #22041 (vanilla-path index.sort.field propagation).

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit c38fabb.

PathLineSeverityDescription
sandbox/qa/analytics-engine-coordinator/build.gradle61highNew project dependency added: 'analytics-backend-lucene' module added as internalClusterTestImplementation. Per mandatory review rules, all dependency changes must be flagged regardless of apparent legitimacy. Maintainers should verify this is an expected internal module addition and that the referenced project path resolves to the intended artifact.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 1 | Medium: 0 | Low: 0


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch 2 times, most recently from 8d5b4ce to c38fabb Compare June 8, 2026 17:28
@alchemist51 alchemist51 added the skip-diff-analyzer Maintainer to skip code-diff-analyzer check, after reviewing issues in AI analysis. label Jun 8, 2026
@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from c38fabb to 1a53c01 Compare June 8, 2026 17:37
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit 291a49f)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ Recommended focus areas for review

Possible Issue

analyze_top_sort descends through Filter nodes, but filters can appear anywhere in the plan tree, not just at the top. If a filter appears after a join or aggregation that breaks monotonicity, the function will incorrectly find a Sort below it and claim the catalog ordering applies. This can cause incorrect reversal when the sort key's monotonicity was destroyed by an intermediate operation.

pub(crate) fn analyze_top_sort(plan: &datafusion::logical_expr::LogicalPlan) -> Option<TopSort> {
    use datafusion::logical_expr::{Expr, LogicalPlan};
    let mut current = plan;
    loop {
        match current {
            LogicalPlan::Sort(s) => {
                let leading = s.expr.first()?;
                let col = match &leading.expr {
                    Expr::Column(c) => c.name.clone(),
                    _ => return None,
                };
                return Some(TopSort {
                    column: col,
                    descending: !leading.asc,
                });
            }
            LogicalPlan::Projection(p) => current = p.input.as_ref(),
            LogicalPlan::Limit(l) => current = l.input.as_ref(),
            LogicalPlan::SubqueryAlias(a) => current = a.input.as_ref(),
            LogicalPlan::Distinct(d) => match d {
                datafusion::logical_expr::Distinct::All(input) => current = input.as_ref(),
                datafusion::logical_expr::Distinct::On(on) => current = on.input.as_ref(),
            },
            LogicalPlan::Filter(f) => current = f.input.as_ref(),
            _ => return None,
        }
    }
Possible Issue

should_reverse_segments performs case-insensitive comparison on sort direction (catalog_order.eq_ignore_ascii_case("desc")) but case-sensitive comparison on field names. If the Java side ever passes "DESC" or "Desc" instead of lowercase "desc", the direction comparison will work but may not match the intended semantics if the code elsewhere assumes lowercase. The inconsistency in case-sensitivity between field name and direction could lead to subtle bugs if the input format varies.

pub(crate) fn should_reverse_segments(
    top_sort: Option<&TopSort>,
    sort_fields: &[String],
    sort_orders: &[String],
) -> bool {
    let Some(top) = top_sort else { return false };
    let Some(catalog_field) = sort_fields.first() else { return false };
    let Some(catalog_order) = sort_orders.first() else { return false };
    if top.column != *catalog_field {
        return false;
    }
    let catalog_descending = catalog_order.eq_ignore_ascii_case("desc");
    top.descending != catalog_descending
}

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

Latest suggestions up to 291a49f

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Add depth limit to prevent infinite loops

The infinite loop lacks an explicit depth limit, which could cause stack overflow or
infinite loops if the plan contains unexpected cycles or deeply nested structures.
Add a maximum iteration depth guard to prevent potential issues.

sandbox/plugins/analytics-backend-datafusion/rust/src/indexed_executor.rs [222-249]

 pub(crate) fn analyze_top_sort(plan: &datafusion::logical_expr::LogicalPlan) -> Option<TopSort> {
     use datafusion::logical_expr::{Expr, LogicalPlan};
     let mut current = plan;
+    let mut depth = 0;
+    const MAX_DEPTH: usize = 100;
     loop {
+        if depth >= MAX_DEPTH {
+            return None;
+        }
+        depth += 1;
         match current {
             LogicalPlan::Sort(s) => {
                 let leading = s.expr.first()?;
                 let col = match &leading.expr {
                     Expr::Column(c) => c.name.clone(),
                     _ => return None,
                 };
                 return Some(TopSort {
                     column: col,
                     descending: !leading.asc,
                 });
             }
             LogicalPlan::Projection(p) => current = p.input.as_ref(),
             LogicalPlan::Limit(l) => current = l.input.as_ref(),
             LogicalPlan::SubqueryAlias(a) => current = a.input.as_ref(),
             LogicalPlan::Distinct(d) => match d {
                 datafusion::logical_expr::Distinct::All(input) => current = input.as_ref(),
                 datafusion::logical_expr::Distinct::On(on) => current = on.input.as_ref(),
             },
             LogicalPlan::Filter(f) => current = f.input.as_ref(),
             _ => return None,
         }
     }
 }
Suggestion importance[1-10]: 6

__

Why: Adding a depth limit is a reasonable defensive measure against pathological query plans, though DataFusion's logical plans are typically well-formed and unlikely to contain cycles. The suggestion improves robustness with minimal overhead, making it a valid but moderate-impact improvement for edge-case protection.

Low
Avoid redundant defensive copy

The List.copyOf() call is redundant when sortFields or sortOrders is already an
immutable list (e.g., List.of()). Consider using defensive copying only when the
input is mutable to avoid unnecessary allocations.

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/DatafusionReaderManager.java [80-81]

-this.sortFields = sortFields == null ? List.of() : List.copyOf(sortFields);
-this.sortOrders = sortOrders == null ? List.of() : List.copyOf(sortOrders);
+this.sortFields = sortFields == null || sortFields.isEmpty() ? List.of() : List.copyOf(sortFields);
+this.sortOrders = sortOrders == null || sortOrders.isEmpty() ? List.of() : List.copyOf(sortOrders);
Suggestion importance[1-10]: 3

__

Why: The suggestion to check isEmpty() before calling List.copyOf() is incorrect. List.copyOf() is needed to create a defensive immutable copy regardless of whether the input is empty, as the input could be a mutable list. The isEmpty() check would not avoid the allocation since List.of() and List.copyOf(emptyList) both create objects. The current code correctly handles null and creates defensive copies.

Low

Previous suggestions

Suggestions up to commit 35801ae
CategorySuggestion                                                                                                                                    Impact
General
Validate parallel array lengths match

The function doesn't validate that sort_fields and sort_orders have the same length.
If they're mismatched (e.g., sort_fields has 2 elements but sort_orders has 1), the
function could make incorrect decisions. Add a length check to ensure the arrays are
parallel before proceeding.

sandbox/plugins/analytics-backend-datafusion/rust/src/indexed_executor.rs [262-275]

 pub(crate) fn should_reverse_segments(
     top_sort: Option<&TopSort>,
     sort_fields: &[String],
     sort_orders: &[String],
 ) -> bool {
     let Some(top) = top_sort else { return false };
     let Some(catalog_field) = sort_fields.first() else { return false };
     let Some(catalog_order) = sort_orders.first() else { return false };
+    if sort_fields.len() != sort_orders.len() {
+        return false;
+    }
     if top.column != *catalog_field {
         return false;
     }
     let catalog_descending = catalog_order.eq_ignore_ascii_case("desc");
     top.descending != catalog_descending
 }
Suggestion importance[1-10]: 5

__

Why: Adding a length check for sort_fields and sort_orders would improve robustness by catching configuration errors early. However, these arrays are sourced from the same Java-side IndexSettings and are expected to be parallel by construction, making this a defensive check rather than fixing a likely bug.

Low
Suggestions up to commit 2fa982e
CategorySuggestion                                                                                                                                    Impact
General
Validate parallel array lengths match

The function doesn't validate that sort_fields and sort_orders have the same length.
If they're mismatched (e.g., one field but no order), the function could return
incorrect results or panic on subsequent accesses.

sandbox/plugins/analytics-backend-datafusion/rust/src/indexed_executor.rs [262-275]

 pub(crate) fn should_reverse_segments(
     top_sort: Option<&TopSort>,
     sort_fields: &[String],
     sort_orders: &[String],
 ) -> bool {
     let Some(top) = top_sort else { return false };
     let Some(catalog_field) = sort_fields.first() else { return false };
     let Some(catalog_order) = sort_orders.first() else { return false };
+    if sort_fields.len() != sort_orders.len() {
+        return false;
+    }
     if top.column != *catalog_field {
         return false;
     }
     let catalog_descending = catalog_order.eq_ignore_ascii_case("desc");
     top.descending != catalog_descending
 }
Suggestion importance[1-10]: 5

__

Why: Adding a length check for sort_fields and sort_orders improves robustness by catching configuration errors early. However, the function already safely handles mismatches by returning false when either array is empty, and the parallel arrays are sourced from the same configuration on the Java side, making mismatches unlikely in practice.

Low
Optimize defensive copying for empty lists

The List.copyOf() call is redundant when sortFields or sortOrders is already an
immutable list (e.g., List.of()). Consider using defensive copying only when the
input is mutable to avoid unnecessary allocations.

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/DatafusionReaderManager.java [80-81]

-this.sortFields = sortFields == null ? List.of() : List.copyOf(sortFields);
-this.sortOrders = sortOrders == null ? List.of() : List.copyOf(sortOrders);
+this.sortFields = sortFields == null || sortFields.isEmpty() ? List.of() : List.copyOf(sortFields);
+this.sortOrders = sortOrders == null || sortOrders.isEmpty() ? List.of() : List.copyOf(sortOrders);
Suggestion importance[1-10]: 3

__

Why: The suggestion to check isEmpty() before List.copyOf() is a minor optimization, but List.copyOf() already handles empty lists efficiently by returning an immutable empty list. The performance gain is negligible, and the added complexity doesn't justify the change.

Low
Suggestions up to commit ece6b9e
CategorySuggestion                                                                                                                                    Impact
General
Guard reversal logic against empty segments

The reversal logic mutates segments in place without checking if it's empty. While
reverse_segment_iteration_order handles empty slices safely, the log message will
still execute and print misleading debug output when there are no segments to
reverse. Add an early return or guard to skip the entire block when
segments.is_empty().

sandbox/plugins/analytics-backend-datafusion/rust/src/indexed_executor.rs [843-851]

 let mut segments = segments;
-if should_reverse_segments(analyze_top_sort(&logical_plan).as_ref(), &sort_fields, &sort_orders) {
+if !segments.is_empty() && should_reverse_segments(analyze_top_sort(&logical_plan).as_ref(), &sort_fields, &sort_orders) {
     log_debug!(
         "indexed_executor: reversing segment iteration (catalog leading sort={:?} {:?}, query opposite)",
         sort_fields.first(),
         sort_orders.first()
     );
     reverse_segment_iteration_order(&mut segments);
 }
Suggestion importance[1-10]: 4

__

Why: While the suggestion correctly identifies that reverse_segment_iteration_order handles empty slices safely (as verified by the test reverse_segments_empty_is_noop), adding an !segments.is_empty() guard prevents unnecessary log output when there are no segments. However, this is a minor optimization that doesn't fix a bug, as the current code is functionally correct.

Low
Suggestions up to commit 3536f9d
CategorySuggestion                                                                                                                                    Impact
Possible issue
Validate parallel array lengths match

Validate that sort_fields and sort_orders have the same length before accessing
their first elements. Mismatched array lengths could lead to inconsistent behavior
where one array has elements but the other doesn't, causing the function to return
false when it should validate the configuration.

sandbox/plugins/analytics-backend-datafusion/rust/src/indexed_executor.rs [262-275]

 pub(crate) fn should_reverse_segments(
     top_sort: Option<&TopSort>,
     sort_fields: &[String],
     sort_orders: &[String],
 ) -> bool {
     let Some(top) = top_sort else { return false };
+    if sort_fields.len() != sort_orders.len() {
+        return false;
+    }
     let Some(catalog_field) = sort_fields.first() else { return false };
     let Some(catalog_order) = sort_orders.first() else { return false };
     if top.column != *catalog_field {
         return false;
     }
     let catalog_descending = catalog_order.eq_ignore_ascii_case("desc");
     top.descending != catalog_descending
 }
Suggestion importance[1-10]: 7

__

Why: The suggestion adds a defensive check to ensure sort_fields and sort_orders have matching lengths before accessing their elements. While the PR already ensures these arrays are parallel at construction time (lines 80-81 in DatafusionReaderManager.java use List.copyOf for both), adding this validation provides defense-in-depth against potential future bugs or misuse. However, since the arrays are already guaranteed to be parallel by the Java-side construction, this is more of a safety improvement than fixing a critical issue.

Medium
Suggestions up to commit b714624
CategorySuggestion                                                                                                                                    Impact
General
Validate parallel array lengths match

Verify that sort_fields and sort_orders have the same length before comparing their
first elements. Mismatched array lengths could indicate a configuration error that
should be caught early rather than silently returning false.

sandbox/plugins/analytics-backend-datafusion/rust/src/indexed_executor.rs [262-275]

 pub(crate) fn should_reverse_segments(
     top_sort: Option<&TopSort>,
     sort_fields: &[String],
     sort_orders: &[String],
 ) -> bool {
     let Some(top) = top_sort else { return false };
+    if sort_fields.len() != sort_orders.len() {
+        return false;
+    }
     let Some(catalog_field) = sort_fields.first() else { return false };
     let Some(catalog_order) = sort_orders.first() else { return false };
     if top.column != *catalog_field {
         return false;
     }
     let catalog_descending = catalog_order.eq_ignore_ascii_case("desc");
     top.descending != catalog_descending
 }
Suggestion importance[1-10]: 7

__

Why: Adding a length check for the parallel arrays sort_fields and sort_orders is a reasonable defensive measure that catches configuration errors early. However, the PR already ensures these arrays are parallel at construction time (line 80-81 in DatafusionReaderManager.java uses List.copyOf for both), making this a safety enhancement rather than a critical fix.

Medium

@alchemist51 alchemist51 changed the title [Sandbox] Reverse indexed-parquet segment iteration for opposite-direction sorts Reverse indexed-parquet segment iteration for opposite-direction sorts Jun 8, 2026
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

✅ Gradle check result for 1a53c01: SUCCESS

@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.36%. Comparing base (6a87a06) to head (291a49f).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main   #22056   +/-   ##
=========================================
  Coverage     73.35%   73.36%           
- Complexity    75854    75880   +26     
=========================================
  Files          6070     6070           
  Lines        344609   344609           
  Branches      49576    49576           
=========================================
+ Hits         252799   252812   +13     
+ Misses        71684    71674   -10     
+ Partials      20126    20123    -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 1a53c01 to 0295440 Compare June 8, 2026 18:37
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 0295440

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

✅ Gradle check result for 0295440: SUCCESS

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 0295440 to 3dc538e Compare June 8, 2026 21:13
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 3dc538e

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

✅ Gradle check result for 3dc538e: SUCCESS

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 3dc538e to 60d3bdc Compare June 9, 2026 06:19
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 60d3bdc

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

✅ Gradle check result for 60d3bdc: SUCCESS

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 60d3bdc to decfdd4 Compare June 9, 2026 07:27
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit decfdd4

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

❌ Gradle check result for decfdd4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from decfdd4 to ca6250c Compare June 9, 2026 09:53
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit ca6250c

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from ca6250c to 846878c Compare June 9, 2026 10:37
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 846878c

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 846878c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@alchemist51 alchemist51 marked this pull request as ready for review June 9, 2026 11:21
@alchemist51 alchemist51 requested a review from a team as a code owner June 9, 2026 11:21
@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 846878c to 32dd4f9 Compare June 9, 2026 15:18
@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 67bfb10: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit ccfe95e

@github-actions

Copy link
Copy Markdown
Contributor

✅ Gradle check result for ccfe95e: SUCCESS

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit b714624

@github-actions

Copy link
Copy Markdown
Contributor

✅ Gradle check result for b714624: SUCCESS

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from b714624 to 3536f9d Compare June 11, 2026 03:33
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 3536f9d

@bharath-techie bharath-techie left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

✅ Gradle check result for 3536f9d: SUCCESS

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit ece6b9e

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for ece6b9e: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from ece6b9e to 2fa982e Compare June 15, 2026 06:23
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 2fa982e

…ction sorts

Mirror of `ContextIndexSearcher.shouldUseTimeSeriesDescSortOptimization` for
the indexed-parquet path. When `index.sort.field` is configured and the
query's leading ORDER BY runs counter to the catalog's stored direction,
reverse the per-shard segment iteration so a TopK above the scan pulls the
highest-priority segment first and parquet page stats prune the rest.

Reuses PR opensearch-project#22041's `ShardView.sort_fields` / `sort_orders` plumbing — no new
Java/FFM surface. Only the indexed scan path (`IndexedTableProvider` /
`QueryShardExec`) consumes them; the vanilla `ListingTable` path remains
covered by opensearch-project#22041.

`indexed_executor.rs`:
- `analyze_top_sort(&LogicalPlan)` walks Projection/Limit/SubqueryAlias/
  Distinct/Filter to find the leading top-of-plan ORDER BY. Returns None if
  the leading sort key isn't a plain `Expr::Column` (function on the sort
  key breaks catalog monotonicity).
- `should_reverse_segments` triggers iff the catalog has a sort, the query's
  leading sort column matches `sort_fields[0]` (case-sensitive, matching
  PR opensearch-project#22041), and the directions differ.
- `reverse_segment_iteration_order` flips `Vec<SegmentFileInfo>` in place.
  Per-segment `global_base` is deliberately NOT recomputed — see below.
- Wired into `execute_indexed_with_context_inner` after `from_substrait_plan`
  decodes the LogicalPlan.

`SessionContextHandle` carries `sort_fields` / `sort_orders` cloned from
`ShardView` so both `create_session_context` and
`create_session_context_indexed` populate the indexed-path consumer.

The `__row_id__` emitted by the indexed scan is `segment.global_base + position`.
QTF's fetch phase (`api::fetch_by_row_ids`) goes through a different code path
(`ShardTableProvider`, not `IndexedTableProvider`) and rebuilds segments via
`build_segments(ShardView.object_metas, ShardView.writer_generations)` — always
in catalog order. Fetch then partition_points the cached row IDs against
catalog-order `global_base`s to find the owning segment.

For the round trip to hold, query-phase IDs must match catalog-order bases.
`reverse_segment_iteration_order` therefore flips iteration order ONLY —
each segment retains its catalog-order `global_base`. Reversal becomes purely
a scheduling decision; the row-id space is unchanged.

Rust unit tests in `indexed_executor::tests`:
- `analyze_top_sort` (top-level desc, descends through Limit/Projection,
  None for no-sort, None for function sort key).
- `should_reverse_segments` (matching field opposite direction, matching
  same direction, catalog desc query asc, no query sort, no catalog sort,
  non-leading catalog field, case-sensitive name).
- `reverse_segments_preserves_global_base`, `reverse_segments_empty_is_noop`,
  `reverse_segments_single_keeps_its_base`.

Rust e2e tests in `indexed_table/tests_e2e/sort_reverse_row_id.rs`:
- `three_segments_row_ids_invariant_under_reversal`,
  `four_segments_row_ids_invariant_under_reversal` — pin the load-bearing
  invariant: same shard-global `__row_id__` set whether iteration is
  natural or reversed.

Java IT in `sandbox/qa/analytics-engine-coordinator/.../SortReverseQtfIT.java`
(parquet+lucene composite, 2 shards, 3 flushes → 2-4 segments per shard):
- `testReversalTriggers_orderByTsDescLimit_correctResults` — captures the
  reversal log line via `MockLogAppender` on `RustLoggerBridge` AND asserts
  result correctness (top-k by ts DESC).
- `testNoReversal_orderByTsAscLimit_correctResults` — `UnseenEventExpectation`
  proves reversal does NOT fire; results still correct.
- `testReversal_keywordFilterDelegationPath_correctResults` — keyword filter
  shipped to Lucene perf peer + reversal; verifies QTF row-id round-trip
  composes with delegation.

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 2fa982e to 35801ae Compare June 15, 2026 07:08
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 35801ae

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 35801ae: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 35801ae to 7ed685e Compare June 15, 2026 09:14
Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
@alchemist51 alchemist51 force-pushed the sort-aware-segment-reversal branch from 7ed685e to 291a49f Compare June 15, 2026 09:15
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 291a49f

@github-actions

Copy link
Copy Markdown
Contributor

✅ Gradle check result for 291a49f: SUCCESS

@bharath-techie bharath-techie merged commit e2cc297 into opensearch-project:main Jun 15, 2026
16 checks passed
KhishorekumarBS pushed a commit to KhishorekumarBS/OpenSearch that referenced this pull request Jul 3, 2026
opensearch-project#22056)

* [Sandbox] Reverse indexed-parquet segment iteration for opposite-direction sorts

Mirror of `ContextIndexSearcher.shouldUseTimeSeriesDescSortOptimization` for
the indexed-parquet path. When `index.sort.field` is configured and the
query's leading ORDER BY runs counter to the catalog's stored direction,
reverse the per-shard segment iteration so a TopK above the scan pulls the
highest-priority segment first and parquet page stats prune the rest.

Reuses PR opensearch-project#22041's `ShardView.sort_fields` / `sort_orders` plumbing — no new
Java/FFM surface. Only the indexed scan path (`IndexedTableProvider` /
`QueryShardExec`) consumes them; the vanilla `ListingTable` path remains
covered by opensearch-project#22041.

`indexed_executor.rs`:
- `analyze_top_sort(&LogicalPlan)` walks Projection/Limit/SubqueryAlias/
  Distinct/Filter to find the leading top-of-plan ORDER BY. Returns None if
  the leading sort key isn't a plain `Expr::Column` (function on the sort
  key breaks catalog monotonicity).
- `should_reverse_segments` triggers iff the catalog has a sort, the query's
  leading sort column matches `sort_fields[0]` (case-sensitive, matching
  PR opensearch-project#22041), and the directions differ.
- `reverse_segment_iteration_order` flips `Vec<SegmentFileInfo>` in place.
  Per-segment `global_base` is deliberately NOT recomputed — see below.
- Wired into `execute_indexed_with_context_inner` after `from_substrait_plan`
  decodes the LogicalPlan.

`SessionContextHandle` carries `sort_fields` / `sort_orders` cloned from
`ShardView` so both `create_session_context` and
`create_session_context_indexed` populate the indexed-path consumer.

The `__row_id__` emitted by the indexed scan is `segment.global_base + position`.
QTF's fetch phase (`api::fetch_by_row_ids`) goes through a different code path
(`ShardTableProvider`, not `IndexedTableProvider`) and rebuilds segments via
`build_segments(ShardView.object_metas, ShardView.writer_generations)` — always
in catalog order. Fetch then partition_points the cached row IDs against
catalog-order `global_base`s to find the owning segment.

For the round trip to hold, query-phase IDs must match catalog-order bases.
`reverse_segment_iteration_order` therefore flips iteration order ONLY —
each segment retains its catalog-order `global_base`. Reversal becomes purely
a scheduling decision; the row-id space is unchanged.

Rust unit tests in `indexed_executor::tests`:
- `analyze_top_sort` (top-level desc, descends through Limit/Projection,
  None for no-sort, None for function sort key).
- `should_reverse_segments` (matching field opposite direction, matching
  same direction, catalog desc query asc, no query sort, no catalog sort,
  non-leading catalog field, case-sensitive name).
- `reverse_segments_preserves_global_base`, `reverse_segments_empty_is_noop`,
  `reverse_segments_single_keeps_its_base`.

Rust e2e tests in `indexed_table/tests_e2e/sort_reverse_row_id.rs`:
- `three_segments_row_ids_invariant_under_reversal`,
  `four_segments_row_ids_invariant_under_reversal` — pin the load-bearing
  invariant: same shard-global `__row_id__` set whether iteration is
  natural or reversed.

Java IT in `sandbox/qa/analytics-engine-coordinator/.../SortReverseQtfIT.java`
(parquet+lucene composite, 2 shards, 3 flushes → 2-4 segments per shard):
- `testReversalTriggers_orderByTsDescLimit_correctResults` — captures the
  reversal log line via `MockLogAppender` on `RustLoggerBridge` AND asserts
  result correctness (top-k by ts DESC).
- `testNoReversal_orderByTsAscLimit_correctResults` — `UnseenEventExpectation`
  proves reversal does NOT fire; results still correct.
- `testReversal_keywordFilterDelegationPath_correctResults` — keyword filter
  shipped to Lucene perf peer + reversal; verifies QTF row-id round-trip
  composes with delegation.

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

* trigger CI rerun (gradle-check Jenkins null-result flake)

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

---------

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-diff-analyzer Maintainer to skip code-diff-analyzer check, after reviewing issues in AI analysis.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants