Skip to content

Add fabric-cosmos-spark-auth_4-0_2-13 for Spark 4.0 / Fabric Runtime 2.0 #49096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xinlian12 merged 6 commits into from
May 8, 2026
Merged

Add fabric-cosmos-spark-auth_4-0_2-13 for Spark 4.0 / Fabric Runtime 2.0 #49096

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4190ed7
feat: add fabric-cosmos-spark-auth_4-0_2-13 for Spark 4.0
xinlian12 May 7, 2026
4c4a9ce
fix: address PR review comments on README
xinlian12 May 7, 2026
2478584
fix: add fabric-cosmos-spark-auth_4-0_2-13 to cspell ignorePaths
xinlian12 May 7, 2026
57ba2de
fix: update synapseutils_2.13 version to 1.6.2
xinlian12 May 7, 2026
9f4647e
fix: skip Scala compilation on JDK < 17 for fabric-cosmos-spark-auth_…
xinlian12 May 8, 2026
e37d24a
Register fabric-cosmos-spark-auth_4-0_2-13 README in .docsettings.yml
xinlian12 May 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .vscode/cspell.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@
"sdk/cosmos/azure-cosmos-spark_4-1_2-13/**",
"sdk/cosmos/azure-cosmos-spark-account-data-resolver-sample/**",
"sdk/cosmos/fabric-cosmos-spark-auth_3/**",
"sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/**",
"sdk/cosmos/azure-cosmos-encryption/**",
"sdk/cosmos/azure-cosmos-spark_3_2-12/**",
"sdk/spring/azure-spring-data-cosmos/**",
Expand Down
1 change: 1 addition & 0 deletions eng/.docsettings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ known_content_issues:
- ['sdk/cosmos/azure-cosmos-spark_4-1_2-13/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-spark-account-data-resolver-sample/README.md', '#3113']
- ['sdk/cosmos/fabric-cosmos-spark-auth_3/README.md', '#3113']
- ['sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-spark_3_2-12/dev/README.md', '#3113']
- ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/catalog-api.md', '#3113']
- ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md', '#3113']
Expand Down
1 change: 1 addition & 0 deletions eng/versioning/external_dependencies.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,7 @@ cosmos_com.google.guava:guava;33.0.0-jre
cosmos_io.dropwizard.metrics:metrics-core;4.1.0
cosmos_org.hdrhistogram:HdrHistogram;2.1.12
cosmos_com.microsoft.azure.synapse:synapseutils_2.12;1.5.4
cosmos_com.microsoft.azure.synapse:synapseutils_2.13;1.6.2

## Cosmos Spark connector under sdk\cosmos\azure-cosmos-spark_3-<version>_2-12\pom.xml
# Cosmos Spark connector runtime dependencies - provided by Spark runtime/host
Expand Down
1 change: 1 addition & 0 deletions eng/versioning/version_client.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ com.azure.cosmos.spark:azure-cosmos-spark_4;0.0.1-beta.1;0.0.1-beta.1
com.azure.cosmos.spark:azure-cosmos-spark_4-0_2-13;4.48.0;4.49.0-beta.1
com.azure.cosmos.spark:azure-cosmos-spark_4-1_2-13;4.48.0;4.49.0-beta.1
com.azure.cosmos.spark:fabric-cosmos-spark-auth_3;1.1.0;1.2.0-beta.1
com.azure.cosmos.spark:fabric-cosmos-spark-auth_4-0_2-13;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-cosmos-tests;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-data-appconfiguration;1.9.1;1.10.0-beta.1
com.azure.cosmos.kafka:azure-cosmos-kafka-connect;2.10.0;2.11.0-beta.1
Expand Down
15 changes: 15 additions & 0 deletions sdk/cosmos/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ trigger:
- sdk/cosmos/azure-cosmos-spark_4/
- sdk/cosmos/azure-cosmos-spark_4-1_2-13/
- sdk/cosmos/fabric-cosmos-spark-auth_3/
- sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/
- sdk/cosmos/azure-cosmos-test/
- sdk/cosmos/azure-cosmos-tests/
- sdk/cosmos/azure-cosmos-kafka-connect/
Expand All @@ -43,6 +44,7 @@ trigger:
- sdk/cosmos/azure-cosmos-spark_4/pom.xml
- sdk/cosmos/azure-cosmos-spark_4-1_2-13/pom.xml
- sdk/cosmos/fabric-cosmos-spark-auth_3/pom.xml
- sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/pom.xml
- sdk/cosmos/azure-cosmos-kafka-connect/pom.xml

pr:
Expand Down Expand Up @@ -72,6 +74,7 @@ pr:
- sdk/cosmos/azure-cosmos-spark_4/
- sdk/cosmos/azure-cosmos-spark_4-1_2-13/
- sdk/cosmos/fabric-cosmos-spark-auth_3/
- sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/
- sdk/cosmos/faq/
- sdk/cosmos/azure-cosmos-kafka-connect/
exclude:
Expand All @@ -89,6 +92,7 @@ pr:
- sdk/cosmos/azure-cosmos-spark_4/pom.xml
- sdk/cosmos/azure-cosmos-spark_4-1_2-13/pom.xml
- sdk/cosmos/fabric-cosmos-spark-auth_3/pom.xml
- sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/pom.xml
- sdk/cosmos/azure-cosmos-test/pom.xml
- sdk/cosmos/azure-cosmos-tests/pom.xml
- sdk/cosmos/azure-cosmos-kafka-connect/pom.xml
Expand Down Expand Up @@ -129,6 +133,10 @@ parameters:
displayName: 'fabric-cosmos-spark-auth_3'
type: boolean
default: true
- name: release_fabriccosmossparkauth4
displayName: 'fabric-cosmos-spark-auth_4-0_2-13'
type: boolean
default: true
- name: release_azurecosmostest
displayName: 'azure-cosmos-test'
type: boolean
Expand Down Expand Up @@ -201,6 +209,13 @@ extends:
skipPublishDocGithubIo: true
skipPublishDocMs: true
releaseInBatch: ${{ parameters.release_fabriccosmossparkauth3 }}
- name: fabric-cosmos-spark-auth_4-0_2-13
groupId: com.azure.cosmos.spark
safeName: fabriccosmossparkauth4
uberJar: true
skipPublishDocGithubIo: true
skipPublishDocMs: true
releaseInBatch: ${{ parameters.release_fabriccosmossparkauth4 }}
- name: azure-cosmos-test
groupId: com.azure
safeName: azurecosmostest
Expand Down
6 changes: 6 additions & 0 deletions sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
## Release History

### 1.0.0-beta.1 (Unreleased)

#### Features Added
* Added Spark 4.0 / Scala 2.13 variant of the Fabric account data resolver for Azure Cosmos DB Spark Connector. This allows signed-in users in Fabric Runtime 2.0 to authenticate to CosmosDB using Microsoft Entra ID.
19 changes: 19 additions & 0 deletions sdk/cosmos/fabric-cosmos-spark-auth_4-0_2-13/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
## Azure Cosmos DB OLTP Spark 4 Connector Fabric Account Data Resolver
Comment thread
xinlian12 marked this conversation as resolved.

This project provides an implementation of the `AccountDataResolver` interface for the Azure Cosmos DB Spark Connector, specifically designed to work with Azure Fabric accounts.
It allows a signed-in user to authenticate with Microsoft Entra ID within the Azure Fabric environment. To use this implementation, you have to upload the jar file for this project to your fabric environment. It is also necessary to grant the signed-in user's
identity with the correct permissions to perform data plane operations. Follow the instructions [here](https://learn.microsoft.com/azure/cosmos-db/nosql/how-to-grant-data-plane-access?tabs=built-in-definition%2Ccsharp&pivots=azure-interface-cli) to grant the relevant permissions.
To learn more about Microsoft Entra ID (AAD) authentication using the Azure Cosmos DB Spark Connector, see the
[AAD Auth Documentation](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3/docs/AAD-Auth.md). Fabric Native accounts only support gateway mode, so the `spark.cosmos.useGatewayMode` must be set to `true` in the Spark configuration for these accounts.

Comment thread
xinlian12 marked this conversation as resolved.
### Configuration options

| Config Property Name | Default | Description |
|:----------------------------------------------|:--------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| `spark.cosmos.accountDataResolverServiceName` | None | Set this value to `com.azure.cosmos.spark.fabric.FabricAccountDataResolver` to use this implementation of the `AccountDataResolver` |
| `spark.cosmos.auth.aad.audience` | `https://cosmos.azure.com/.default` | Set this value to change the audience used to obtain the Entra Id token. |
Comment thread
xinlian12 marked this conversation as resolved.

### Current Limitations

* There is no support for the [catalog api](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3/docs/catalog-api.md) operations using this implementation of the `AccountDataResolver` with a CosmosDB Fabric Native Account.
* Only Spark 4.0 is supported with this implementation of the `AccountDataResolver`.
Loading
Loading