Skip to content

test(spark): characterize VortexScanBuilder predicate pushdown splitting#8857

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-scan-builder-tests
Open

test(spark): characterize VortexScanBuilder predicate pushdown splitting#8857
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-scan-builder-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

VortexScanBuilder (java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java)
decides which Spark V2 predicates are pushed into the Vortex scan and which are returned
to Spark for post-scan evaluation, and it currently has no direct test coverage. The
split logic encodes correctness-critical rules: predicates referencing partition columns
must not be pushed (partition values live in directory paths, not inside the Vortex
files), untranslatable operators must be returned to Spark, and pushedPredicates()
must not expose internal state. A regression in any of these silently drops filters or
scans the wrong data. This continues the characterization-test series from #8770,
#8782, #8783, and #8850.

What changes are included in this PR?

A new VortexScanBuilderTest with 9 tests covering:

  • pushPredicates on data columns: a supported predicate on a data column is pushed
    and removed from the post-scan set; predicates referencing columns absent from the
    read schema, or using operators SparkPredicateToVortexExpression cannot translate
    (e.g. a raw LIKE), are returned to Spark and never recorded as pushed.
  • pushPredicates on partition columns: a predicate on an identity-partitioned
    column is left to Spark; mixed batches split correctly (data-column predicate pushed,
    partition-column predicate post-scan); an And spanning a data column and a partition
    column is kept whole on the Spark side.
  • pushedPredicates: empty before any push, and returns a defensive copy so callers
    cannot mutate the builder's internal state.
  • build: fails fast with IllegalStateException when no paths were added.

This is a test-only change; no production code is modified. All tests pass locally
on both matrix legs (:vortex-spark_2.12:test and :vortex-spark_2.13:test with
--tests 'dev.vortex.spark.read.VortexScanBuilderTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

@robert3005

Copy link
Copy Markdown
Contributor

These are good checks. I realise I have implemented some of this for iceberg but haven't done similar thing here

@robert3005 robert3005 added the changelog/chore A trivial change label Jul 20, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 44 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation true_count_vortex_buffer[128] 580.6 ns 522.2 ns +11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-scan-builder-tests (0f90592) with develop (b7b01d3)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants