[SparkConnector]updateTransactionalBulkConfig (#48008)

* add transactional bulk config in configuration reference

---------

Co-authored-by: Annie Liang <anniemac@Annies-MacBook-Pro.local>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: 3 people

sdk/cosmos/azure-cosmos-spark_3/docs/configuration-reference.md

@@ -67,6 +67,7 @@
 | `spark.cosmos.write.point.maxConcurrency`                        | None            | Cosmos DB Item Write Max concurrency. If not specified it will be determined based on the Spark executor VM Size                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
 | `spark.cosmos.write.bulk.maxPendingOperations`                   | None            | Cosmos DB Item Write bulk mode maximum pending operations. Defines a limit of bulk operations being processed concurrently. If not specified it will be determined based on the Spark executor VM Size. If the volume of data is large for the provisioned throughput on the destination container, this setting can be adjusted by following the estimation of `1000 x Cores`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 | `spark.cosmos.write.bulk.enabled`                                | `true`          | Cosmos DB Item Write bulk enabled                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `spark.cosmos.write.bulk.transactional`                          | `false`         | Enable transactional batch mode for bulk writes. **Requires Spark 3.5 or later.** When enabled, all operations for the same partition key are executed atomically (all succeed or all fail). Only supports upsert operations. Cannot exceed 100 operations or 2MB per partition key. **Note**: For containers using hierarchical partition keys (HPK), transactional scope applies only to **logical partitions** (complete partition key paths), not partial top-level keys. See [Transactional Batch documentation](https://learn.microsoft.com/azure/cosmos-db/transactional-batch) and the Spark connector [ingestion guidance](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3/docs/scenarios/Ingestion.md) for details.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
 | `spark.cosmos.write.bulk.targetedPayloadSizeInBytes`             | `220201`        | When the targeted payload size is reached for buffered documents, the request is sent to the backend. The default value is optimized for small documents <= 10 KB - when documents often exceed 110 KB, it can help to increase this value to up to about `1500000` (should still be smaller than 2 MB).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | `spark.cosmos.write.bulk.initialBatchSize`                       | `100`           | Cosmos DB initial bulk micro batch size - a micro batch will be flushed to the backend when the number of documents enqueued exceeds this size - or the target payload size is met. The micro batch size is getting automatically tuned based on the throttling rate. By default the initial micro batch size is 100. Reduce this when you want to avoid that the first few requests consume too many RUs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
 | `spark.cosmos.write.bulk.maxBatchSize`                           | `100`           | Cosmos DB max. bulk micro batch size - a micro batch will be flushed to the backend when the number of documents enqueued exceeds this size - or the target payload size is met. The micro batch size is getting automatically tuned based on the throttling rate. By default the max. micro batch size is 100. Use this setting only when migrating Spark 2.4 workloads - for other scenarios relying on the auto-tuning combined with throughput control will result in better experience.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |

sdk/cosmos/azure-cosmos-spark_3/src/main/scala/com/azure/cosmos/spark/CosmosConfig.scala

@@ -1510,8 +1510,7 @@ private object CosmosWriteConfig {
     defaultValue = Option.apply(false),
     parseFromStringFunction = bulkTransactionalAsString => bulkTransactionalAsString.toBoolean,
     helpMessage = "Cosmos DB Item Write bulk transactional batch mode enabled - requires bulk write to be enabled. " +
-      "Spark 3.5+ provides automatic distribution/ordering for transactional batch. " +
-      "On Spark 3.3/3.4, transactional batch is supported but requires manual sorting for optimal performance.")
+      "Requires Spark 3.5 or later.")
 
   private val microBatchPayloadSizeInBytes = CosmosConfigEntry[Int](key = CosmosConfigNames.WriteBulkPayloadSizeInBytes,
     defaultValue = Option.apply(BatchRequestResponseConstants.DEFAULT_MAX_DIRECT_MODE_BATCH_REQUEST_BODY_SIZE_IN_BYTES),