Skip to content

test(spark): characterize HadoopUtils S3 and Azure credential forwarding#8850

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-hadoop-utils-tests
Open

test(spark): characterize HadoopUtils S3 and Azure credential forwarding#8850
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-hadoop-utils-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

HadoopUtils (java/vortex-spark/src/main/java/dev/vortex/spark/config/HadoopUtils.java)
is the bridge that forwards S3 and Azure credentials from the Spark session's Hadoop
Configuration into the Vortex object-store property keys, and it currently has no test
coverage. A silently broken key mapping here doesn't fail loudly — it surfaces as
authentication errors (or unintentionally anonymous access via the Azure
skip_signature fallback) only when running against real cloud storage. This continues
the characterization-test series from #8770 (ArrowUtils), #8782 (SparkToArrowSchema),
and #8783 (PartitionPathUtils).

What changes are included in this PR?

A new HadoopUtilsTest covering both public methods:

  • s3PropertiesFromHadoopConf: maps fs.s3a.access.key / fs.s3a.secret.key /
    fs.s3a.session.token / fs.s3a.endpoint.region to the corresponding aws_* keys;
    qualifies a bare fs.s3a.endpoint with https:// while preserving endpoints that
    already carry an http:// or https:// scheme; ignores unrelated Hadoop keys and
    returns an empty map for an empty configuration.
  • azurePropertiesFromHadoopConf: extracts the storage account key from any
    fs.azure.account.key-prefixed entry and the SAS token from any
    fs.azure.sas.fixed.token.-prefixed entry; sets azure_skip_signature=true whenever
    no account key is present (including the SAS-token-only case); ignores unrelated
    fs.azure.* keys.

This is a test-only change; no production code is modified. 10 tests, passing
locally on both matrix legs (:vortex-spark_2.12:test and :vortex-spark_2.13:test
with --tests 'dev.vortex.spark.config.HadoopUtilsTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

@codspeed-hq

codspeed-hq Bot commented Jul 21, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 44 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation true_count_vortex_buffer[128] 580.6 ns 522.2 ns +11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-hadoop-utils-tests (f8cb4b4) with develop (c5e075a)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@jackylee-ch
jackylee-ch force-pushed the spark-hadoop-utils-tests branch from f8cb4b4 to e4fa40e Compare July 21, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants