[GLUTEN-12143][VL] Route bitmap_construct_agg to native Velox execution by minni31 · Pull Request #12142 · apache/gluten

minni31 · 2026-05-26T11:14:28Z

Route `bitmap_construct_agg` to Native Velox Execution

This PR enables native Velox execution for the bitmap_construct_agg aggregate function, avoiding fallback to vanilla Spark.

Changes

Expression Registration (ExpressionNames.scala): Added BITMAP_CONSTRUCT_AGG constant.
Shim Registration (Spark35Shims.scala, Spark40Shims.scala, Spark41Shims.scala): Register Sig[BitmapConstructAgg] mapping to the substrait function name.
Native Validator (SubstraitToVeloxPlanValidator.cc): Added "bitmap_construct_agg" to the supportedAggFuncs set so the native validator accepts this function.
Plan-shape assertion test (GlutenBitmapExpressionsQuerySuite.scala for spark35/40/41): Verifies the query plan contains HashAggregateExecBaseTransformer (native execution) rather than falling back.

Test Results

Spark Version	Tests	Result
Spark 3.5	11/11	✅ Pass
Spark 4.0	11/11	✅ Pass
Spark 4.1	13/13	✅ Pass

All tests include the new plan-shape assertion test confirming native Velox execution.

Related issue: #12143

Register bitmap_construct_agg as a supported aggregate expression in the Velox backend, allowing it to be executed natively instead of falling back to vanilla Spark. Changes: - Add BITMAP_CONSTRUCT_AGG constant to ExpressionNames - Register Sig[BitmapConstructAgg] in Spark 3.5, 4.0, and 4.1 shims - Add bitmap_construct_agg to C++ plan validator allowed list Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-26T14:28:40Z

Run Gluten Clickhouse CI on x86

weiting-chen

Thanks for the clean implementation — the three-layer plumbing (constant + shim registration + C++ validator) follows the established pattern correctly.

Two suggestions for consideration (non-blocking):

weiting-chen · 2026-05-26T14:35:04Z

      Sig[RegrSXY](ExpressionNames.REGR_SXY),
-      Sig[RegrReplacement](ExpressionNames.REGR_REPLACEMENT)
+      Sig[RegrReplacement](ExpressionNames.REGR_REPLACEMENT),
+      Sig[BitmapConstructAgg](ExpressionNames.BITMAP_CONSTRUCT_AGG)


Consider adding plan-shape assertion in tests

Problem: GlutenBitmapExpressionsQuerySuite inherits upstream Spark tests that only use checkAnswer() — they verify correctness but do not assert that native Velox execution is actually active. A regression that silently disables offloading would go undetected since vanilla Spark also produces correct results.

Evidence:

class GlutenBitmapExpressionsQuerySuite extends BitmapExpressionsQuerySuite with GlutenSQLTestsTrait {} // All upstream tests only call checkAnswer(df, expected)

Suggested Fix: Add at least one Gluten-specific test asserting the native transformer is present:

test("bitmap_construct_agg routes to native") { val df = spark.sql("SELECT bitmap_construct_agg(col) FROM values (1L), (2L) AS t(col)") val plan = df.queryExecution.executedPlan assert(collect(plan) { case h: HashAggregateExecTransformer => h }.nonEmpty, "Expected native HashAggregateExecTransformer in plan") }

Added in all spark version. Thanks @weiting-chen

weiting-chen · 2026-05-26T14:35:04Z

      "regr_sxy",
-      "regr_replacement"};
+      "regr_replacement",
+      "bitmap_construct_agg"};


Follow-up opportunity: register bitmap_or_agg (and bitmap_and_agg for Spark 4.1)

Problem: bitmap_or_agg was introduced in the same Spark 3.5 commit as bitmap_construct_agg, and bitmap_and_agg in Spark 4.1. If native Velox implementations exist for these, registering them together would avoid unnecessary columnar-to-row transitions when users combine bitmap functions in the same query stage.

Investigation Needed: Confirm whether bitmap_or_agg and bitmap_and_agg have native Velox implementations (look for registration in cpp/velox/udf/ or the Velox aggregate function registry). If yes, consider adding them in a follow-up PR:

"bitmap_construct_agg", "bitmap_or_agg", "bitmap_and_agg"};

I've Velox PR for bitmap_or_agg in review phase. Will add follow up PR for Gluten once Velox PR merges. Thanks!

Adds a test verifying that bitmap_construct_agg routes to native Velox execution (HashAggregateExecBaseTransformer) rather than falling back to vanilla Spark. Test added for Spark 3.5, 4.0, and 4.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-26T18:06:34Z

Run Gluten Clickhouse CI on x86

yaooqinn · 2026-05-27T07:50:44Z

+1. Three-layer plumbing (Sig[BitmapConstructAgg] on all 3 shims + supportedAggFuncs entry + plan-shape assertion) follows the same shape as the existing BloomFilterAgg registration in this file, which is the right precedent here.

Grounded the plan-shape test guard: bitmap_bit_position is registered in gen-function-support-docs.py (BitmapBitPosition -> bitmap_bit_position), so the inner expression won't silently fallback and mask a bitmap_construct_agg regression. Good defensive test shape.

bitmap_construct_agg offloaded to Velox throws GlutenException instead of SparkArrayIndexOutOfBoundsException. Exclude these error-path tests following the established pattern for native execution error type mismatches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-27T12:44:28Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-05-28T11:05:28Z

Run Gluten Clickhouse CI on x86

yaooqinn · 2026-05-29T02:41:48Z

Following up on my earlier +1 — looked deeper after the ClickHouse CI failures and I think the registration is at the wrong layer.

CH CI is PR-tied (not infra flake). 5 tests fail in GlutenBitmapExpressionsQuerySuite (CH backend) with:

GlutenException: Unknown aggregate function bitmap_construct_agg
  at AggregateFunctionParserFactory::get (cpp-ch/local-engine/Parser/AggregateFunctionParser.cpp:255)

Root cause: Sig[BitmapConstructAgg] is registered in shims/spark35,40,41/SparkXShims.aggregateExpressionMappings, which is all-backend — so the CH planner also accepts bitmap_construct_agg and pushes it down to CH native, where the parser isn't registered, hence the failure.

Sibling pattern — BloomFilterAgg:

backends-velox/.../VeloxSparkPlanExecApi.scala:1120 — Sig[VeloxBloomFilterAggregate](ExpressionNames.BLOOM_FILTER_AGG) is registered in extraExpressionMappings, which is Velox-only. BLOOM_FILTER_AGG does not appear in any SparkXShims.aggregateExpressionMappings.

Suggest moving Sig[BitmapConstructAgg] from the three SparkXShims files to VeloxSparkPlanExecApi.extraExpressionMappings. BITMAP_CONSTRUCT_AGG constant in shims/common/ExpressionNames.scala can stay (it's just a string), or move it with the Sig — either is fine.

That should keep CH on the vanilla fallback path and let Velox continue to route natively.

(Sorry for the late catch — my +1 cited bitmap_bit_position registration as a fallback guard, but that doesn't protect against the outer bitmap_construct_agg Sig being visible to non-Velox backends. The shim-vs-backend layering is the actual gate.)

…pings Move Sig[BitmapConstructAgg] from SparkXShims.aggregateExpressionMappings (all-backend) to VeloxSparkPlanExecApi.extraExpressionMappings (Velox-only). This prevents ClickHouse backend from attempting to push down bitmap_construct_agg, which it does not support. Follows the same pattern as BloomFilterAgg registration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-29T05:33:25Z

Run Gluten Clickhouse CI on x86

…ionMappings BitmapConstructAgg only exists in Spark 3.5+. Use runtime class loading with scala.util.Try to gracefully handle Spark 3.3/3.4 where the class is absent. This keeps the registration Velox-only (CH unaffected) and avoids compilation failures on older Spark versions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-29T07:36:37Z

Run Gluten Clickhouse CI on x86

baibaichen · 2026-06-02T07:17:35Z

      // For test purpose.
      Sig[VeloxDummyExpression](VeloxDummyExpression.VELOX_DUMMY_EXPRESSION)
-    )
+    ) ++ scala.util.Try(


This reflection-based registration is a workaround, not the right long-term shape — let's move it to the shims.

Done. Moved to Shim and put it in blacklist on CH backend.

baibaichen · 2026-06-02T07:22:52Z

The CH CI failure (bitmap_construct_agg routes to native *** FAILED *** — List() was empty Expected native HashAggregateExecBaseTransformer in plan) needs to be fixed by adding a blacklist entry on the CH side, following the same pattern used for regr_slope / regr_intercept / regr_sxy / regr_replacement: https://github.com/apache/gluten/blob/main/backends-clickhouse/src/main/scala/org/apache/gluten/utils/CHExpressionUtil.scala#L200

Move Sig[BitmapConstructAgg] back to shims/aggregateExpressionMappings (spark35/40/41) following the established pattern for version-specific expressions. Add BITMAP_CONSTRUCT_AGG -> DefaultValidator() to CH_AGGREGATE_FUNC_BLACKLIST so ClickHouse backend falls back to vanilla Spark instead of attempting unsupported native push-down. Remove the Class.forName reflection workaround from VeloxSparkPlanExecApi. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-02T09:22:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-02T09:24:45Z

Run Gluten Clickhouse CI on x86

The plan-shape assertion test verifies native Velox execution, which is not applicable to the ClickHouse backend. Exclude it using .excludeCH() following the same pattern as GlutenBloomFilterAggregateQuerySuite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-02T14:05:32Z

Run Gluten Clickhouse CI on x86

minni31 · 2026-06-03T05:34:29Z

Following up on my earlier +1 — looked deeper after the ClickHouse CI failures and I think the registration is at the wrong layer.

CH CI is PR-tied (not infra flake). 5 tests fail in GlutenBitmapExpressionsQuerySuite (CH backend) with:
GlutenException: Unknown aggregate function bitmap_construct_agg
  at AggregateFunctionParserFactory::get (cpp-ch/local-engine/Parser/AggregateFunctionParser.cpp:255)
Root cause: Sig[BitmapConstructAgg] is registered in shims/spark35,40,41/SparkXShims.aggregateExpressionMappings, which is all-backend — so the CH planner also accepts bitmap_construct_agg and pushes it down to CH native, where the parser isn't registered, hence the failure.

Sibling pattern — BloomFilterAgg:

backends-velox/.../VeloxSparkPlanExecApi.scala:1120 — Sig[VeloxBloomFilterAggregate](ExpressionNames.BLOOM_FILTER_AGG) is registered in extraExpressionMappings, which is Velox-only. BLOOM_FILTER_AGG does not appear in any SparkXShims.aggregateExpressionMappings.

Suggest moving Sig[BitmapConstructAgg] from the three SparkXShims files to VeloxSparkPlanExecApi.extraExpressionMappings. BITMAP_CONSTRUCT_AGG constant in shims/common/ExpressionNames.scala can stay (it's just a string), or move it with the Sig — either is fine.

That should keep CH on the vanilla fallback path and let Velox continue to route natively.

(Sorry for the late catch — my +1 cited bitmap_bit_position registration as a fallback guard, but that doesn't protect against the outer bitmap_construct_agg Sig being visible to non-Velox backends. The shim-vs-backend layering is the actual gate.)

I discussed with @baibaichen and he suggested to put this function in blacklist at CH backend and put it in SHIM layers only for Velox. Updated the code. Thanks @yaooqinn for the review.

minni31 · 2026-06-03T05:35:19Z

The CH CI failure (bitmap_construct_agg routes to native *** FAILED *** — List() was empty Expected native HashAggregateExecBaseTransformer in plan) needs to be fixed by adding a blacklist entry on the CH side, following the same pattern used for regr_slope / regr_intercept / regr_sxy / regr_replacement: https://github.com/apache/gluten/blob/main/backends-clickhouse/src/main/scala/org/apache/gluten/utils/CHExpressionUtil.scala#L200

Done, addressed. Please help with review @baibaichen

baibaichen · 2026-06-04T13:39:34Z

LGTM

baibaichen · 2026-06-04T13:41:31Z

    .exclude("column pruning - non-readable file")
  enableSuite[GlutenBitmapExpressionsQuerySuite]
+    // bitmap_construct_agg is not supported natively in CH backend.
+    .excludeCH("bitmap_construct_agg routes to native")


This one checks the native plan path, specifically added to test the native plan. Since for CH backend it runs on spark, this test fail in CH backend.

minni31 changed the title ~~[CORE] Route bitmap_construct_agg to native Velox execution~~ [VL] Route bitmap_construct_agg to native Velox execution May 26, 2026

minni31 added 2 commits May 26, 2026 17:08

Merge branch 'main' into bitmap-construct-agg

a8413de

Merge branch 'main' into bitmap-construct-agg

15b7e64

minni31 changed the title ~~[VL] Route bitmap_construct_agg to native Velox execution~~ [GLUTEN-12143][VL] Route bitmap_construct_agg to native Velox execution May 26, 2026

Merge branch 'main' into bitmap-construct-agg

2368a41

github-actions Bot added CORE works for Gluten Core VELOX labels May 26, 2026

weiting-chen approved these changes May 26, 2026

View reviewed changes

weiting-chen reviewed May 26, 2026

View reviewed changes

Merge branch 'main' into bitmap-construct-agg

67a14ee

baibaichen reviewed Jun 2, 2026

View reviewed changes

Minni Mittal and others added 2 commits June 2, 2026 09:10

Merge branch 'main' into bitmap-construct-agg

4023e19

github-actions Bot added the CLICKHOUSE label Jun 2, 2026

baibaichen approved these changes Jun 4, 2026

View reviewed changes

baibaichen reviewed Jun 4, 2026

View reviewed changes

Comment thread gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala

minni31 mentioned this pull request Jun 4, 2026

[VL] Translate Velox bitmap_construct_agg exceptions to Spark-native exception types #12238

Open

baibaichen merged commit 2731470 into apache:main Jun 4, 2026
63 checks passed

Uh oh!

Conversation

minni31 commented May 26, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Route bitmap_construct_agg to Native Velox Execution

Changes

Test Results

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

weiting-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

yaooqinn commented May 27, 2026

Uh oh!

github-actions Bot commented May 27, 2026

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

yaooqinn commented May 29, 2026

Uh oh!

github-actions Bot commented May 29, 2026

Uh oh!

github-actions Bot commented May 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

baibaichen commented Jun 2, 2026

Uh oh!

github-actions Bot commented Jun 2, 2026

Uh oh!

github-actions Bot commented Jun 2, 2026

Uh oh!

github-actions Bot commented Jun 2, 2026

Uh oh!

minni31 commented Jun 3, 2026

Uh oh!

minni31 commented Jun 3, 2026

Uh oh!

baibaichen commented Jun 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

minni31 commented May 26, 2026 •

edited by github-actions Bot

Loading

Route `bitmap_construct_agg` to Native Velox Execution