diff --git a/.vscode/cspell.json b/.vscode/cspell.json index 062e4900303..4dc0c9d0ea8 100644 --- a/.vscode/cspell.json +++ b/.vscode/cspell.json @@ -110,7 +110,10 @@ "webpki", "worktree", "decorrelates", - "finalizer" + "finalizer", + "callsite", + "callsites", + "footgun" ], "dictionaryDefinitions": [ { diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/HEDGING_SPEC.md b/sdk/cosmos/azure_data_cosmos_driver/docs/HEDGING_SPEC.md new file mode 100644 index 00000000000..fa10118e23a --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/HEDGING_SPEC.md @@ -0,0 +1,2333 @@ +# Cross-Region Hedging Availability Strategy Spec + +**Status:** Draft +**Date:** 2026-05-14 +**Authors:** (team) +**Crate:** `azure_data_cosmos_driver` + +--- + +## Table of Contents + +1. [Goals & Motivation](#1-goals--motivation) +2. [Background: .NET SDK Implementation](#2-background-net-sdk-implementation) +3. [Architectural Overview](#3-architectural-overview) +4. [Configuration Surface](#4-configuration-surface) +5. [Eligibility Rules](#5-eligibility-rules) +6. [Hedging Algorithm](#6-hedging-algorithm) +7. [Final Result Classification](#7-final-result-classification) +8. [Operation Pipeline Integration](#8-operation-pipeline-integration) +9. [Interaction with Existing Systems](#9-interaction-with-existing-systems) +10. [Diagnostics & Observability](#10-diagnostics--observability) +11. [Options API Design](#11-options-api-design) +12. [Cancellation & Resource Cleanup](#12-cancellation--resource-cleanup) +13. [Write Hedging (Removed)](#13-write-hedging-removed) +14. [Error Handling & Edge Cases](#14-error-handling--edge-cases) +15. [Test Plan](#15-test-plan) +16. [Implementation Phases](#16-implementation-phases) +17. [Open Questions](#17-open-questions) + +--- + +## 1. Goals & Motivation + +### Problem + +When a Cosmos DB region experiences elevated latency (but is not fully down), the +existing failover/circuit-breaker mechanisms (PPAF/PPCB) do **not** trigger because +the region eventually returns successful responses. Users see tail-latency spikes +(p99, p99.9) on operations routed to the slow region. + +### Solution: Speculative Hedging + +**Hedging** sends the same request to an alternate region after a latency threshold +is exceeded, and returns whichever **final** (non-transient) response arrives first. +This bounds tail latency +at roughly `threshold + cross-region-RTT` instead of waiting for the slow region to +respond. + +### Goals + +1. **Reduce tail latency** — p99/p99.9 point-read latencies bounded + by a user-configured threshold plus cross-region RTT. +2. **Transparent to application** — the caller sees a single `CosmosResponse`; the + hedging mechanism is invisible unless inspected via diagnostics. +3. **Configurable** — a single `threshold` knob is user-controlled at both + client and per-operation levels; opt-out via `AvailabilityStrategy::Disabled`. +4. **Complementary to failover** — hedging handles *latency*; PPAF/PPCB handle + *failures*. They compose without interference, and a repeated + alternate-region win feeds back into PPCB to mark the primary partition + degraded (see §9.5). +5. **Resource-safe** — at most one alternate-region request is in flight at + any time (max two concurrent requests per logical operation), and the + loser is cancelled cooperatively (best-effort — in-flight HTTP + requests cannot be aborted mid-stream; see §14.2) to bound RU and + transport waste over time. +6. **Zero-overhead happy path** — when the primary returns before the + threshold elapses, `execute_hedged()` must not allocate the hedge + task, the `FuturesUnordered`, or any per-hedge state (see §6.5). + +### Non-Goals + +- Hedging within a single region (e.g., across gateway nodes). +- Hedging writes of any kind. Write hedging on multi-master amplifies 409 / + 412 surface area and has near-zero adoption in the Java SDK; the Rust + driver does not hedge writes in any phase. (If service-side idempotency + keys land later, a separate proposal can revisit this.) +- Fanning out to more than one alternate region. The .NET / Java + N-region progressive fan-out is dropped in favor of a single-alternate + region model (max two concurrent requests). +- Automatic threshold tuning based on observed latency histograms (future work). +- Coupling hedging activation to PPAF. Hedging is independent of PPAF + in this driver — see §5.2. + +All Cosmos DB operation types are addressed by the phased rollout below. +Nothing is permanently excluded — stored procedure execution and +adaptive-tuning are deferred to the Future bucket pending a separate +design review. + +### Operation-type scope (phased) + +| Operation type | Phase 1 | Phase 2 | Future | +|---|:---:|:---:|:---:| +| Document point reads (GetItem) | ✅ | ✅ | ✅ | +| Queries (`QueryItems`) — page-level | ❌ | ✅ | ✅ | +| `ReadMany` — page-level | ❌ | ✅ | ✅ | +| Change feed — page-level | ❌ | ✅ | ✅ | +| Metadata operations (Database / Container / Offer / Throughput) | ❌ | ✅ | ✅ | +| Document writes (Create/Replace/Upsert/Delete/Patch) — any topology | ❌ | ❌ | ❌ | +| Stored procedure execution (`ExecuteJavaScript`) | ❌ | ❌ | 🟡 candidate | + +> **Triggers and UDFs** are not standalone operations — they ride along +> as request headers on document operations and are therefore hedged +> automatically with the document op they decorate. Only stored +> procedure execution is a standalone server-side execution and +> deferred to Future. + +Phase 1 ships document point reads only — the smallest correct surface +that exercises `execute_hedged()`, region pinning, cancellation, and the +PPCB feedback loop end-to-end. Phase 2 widens to feed-style operations +(Query / ReadMany / ChangeFeed), each hedged **per page**, plus +metadata operations. The exact integration with the `FeedRange` +abstraction is being co-designed with the feed-operation spec (see +§16). Writes are not in scope for any phase. + +--- + +## 2. Background: .NET SDK Implementation + +The .NET Cosmos DB SDK (v3) implements hedging via the +`CrossRegionHedgingAvailabilityStrategy` class. This section documents its design +to inform the Rust implementation. + +### 2.1 Public API + +```csharp +// Factory method on AvailabilityStrategy (public abstract class) +public static AvailabilityStrategy CrossRegionHedgingStrategy( + TimeSpan threshold, // Time before first hedge fires + TimeSpan? thresholdStep, // Time between subsequent hedges + bool enableMultiWriteRegionHedge = false); // Opt-in for writes on MM + +// Per-request override +requestOptions.AvailabilityStrategy = AvailabilityStrategy.DisabledStrategy(); +``` + +### 2.2 Configuration Model + +| Parameter | Description | Default | Constraints | +|-----------|-------------|---------|-------------| +| `threshold` | Delay before firing the first hedge request | (required) | `> 0` | +| `thresholdStep` | Delay between subsequent hedge requests | (required) | `> 0` | +| `enableMultiWriteRegionHedge` | Allow hedging for writes on multi-write accounts | `false` | Opt-in; increases 409/412 risk | + +### 2.3 Eligibility — `ShouldHedge()` + +Hedging applies **only** to document-level operations: + +1. `ResourceType == Document` — metadata (Database, Container, etc.) is excluded. +2. **Reads**: Always eligible. +3. **Writes**: Only if `enableMultiWriteRegionHedge == true` AND the account + supports multi-region writes for this resource/operation type. +4. **Single-region accounts**: Bypassed (no alternate region to hedge to). + +### 2.4 Execution — `ExecuteAvailabilityStrategyAsync()` + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ ExecuteAvailabilityStrategyAsync │ +│ │ +│ 1. Clone request body (CloneableStream) │ +│ 2. Get applicable regions via GlobalEndpointManager │ +│ (respects ExcludeRegions, read vs. write list) │ +│ 3. For requestNumber = 0..regions.len(): │ +│ a. awaitTime = (requestNumber == 0) ? threshold : thresholdStep │ +│ b. Start hedge timer (Task.Delay(awaitTime)) │ +│ c. Clone request, set ExcludeRegions to exclude all regions │ +│ except regions[requestNumber] (primary request keeps all) │ +│ d. Fire CloneAndSendAsync → RequestSenderAndResultCheckAsync │ +│ e. Race: Task.WhenAny(requestTask, hedgeTimer, ...) │ +│ ├─ Timer wins → continue to next iteration (launch next │ +│ │ hedge), keep running tasks alive │ +│ ├─ Request wins + IsFinalResult → cancel all others, return │ +│ └─ Request wins + transient → remove task, continue racing │ +│ 4. After all regions attempted, drain remaining tasks: │ +│ a. Wait for each remaining task via WhenAny │ +│ b. First IsFinalResult or last remaining task → return │ +│ 5. If all tasks fail/cancel, throw last exception │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +**Key observations:** + +- **Primary request is request #0** — it uses the same region the SDK would normally + pick. `ExcludeRegions` is NOT set for it, so it follows normal routing. +- **Hedge requests (1..N)** — each excludes all regions except one target region, + forcing routing to that specific region. +- **Concurrent fan-out** — all fired requests run in parallel. Timers gate when + new hedges are launched, but previously launched requests continue. +- **Early termination** — the first `IsFinalResult` response cancels all other + in-flight requests via a linked `CancellationTokenSource`. + +### 2.5 Final Result Classification — `IsFinalResult()` + +A response is "final" (non-transient) if: + +| Condition | Final? | +|-----------|--------| +| Any 1xx, 2xx, 3xx | Yes | +| 400 Bad Request | Yes | +| 401 Unauthorized | Yes | +| 405 Method Not Allowed | Yes | +| 409 Conflict | Yes | +| 412 Precondition Failed | Yes | +| 413 Request Entity Too Large | Yes | +| 404 with sub-status 0 (Unknown) | Yes | +| All other 4xx/5xx | **No** (transient) | + +Non-final (transient) responses do NOT terminate hedging — the SDK keeps waiting +for other in-flight requests that might succeed. + +### 2.6 PPAF / PPCB Integration + +When PPAF (Per-Partition Automatic Failover) is enabled on the account and the +user has not specified a custom availability strategy, the .NET SDK +**automatically enables** a default cross-region hedging strategy +(`IsSDKDefaultStrategyForPPAF = true`). This provides latency protection +alongside PPAF's failover protection. + +**Default values used by .NET (PPAF-driven)** +([source](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/0830090ca4dc47b71398ea0871cbe00b591aa8fc/docs/Cross%20Region%20Request%20Hedging.md)): + +- **Threshold:** `min(1000ms, RequestTimeout / 2)` (falls back to `1000ms` if + `RequestTimeout == 0`) +- **Threshold step:** `500ms` +- **Write hedging:** disabled + +PPCB (Per-Partition Circuit Breaker) on its own does **not** auto-enable +hedging in .NET. However, .NET implicitly turns PPCB on whenever PPAF is +enabled +([CosmosClientOptions.cs `EnablePartitionLevelCircuitBreaker`](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/71d317ffc6b6d62199b35c06a372825849e406fc/Microsoft.Azure.Cosmos/src/CosmosClientOptions.cs)), +so a PPAF-enabled deployment ends up running with all three (PPAF + PPCB + +hedging) active simultaneously. + +**The Rust driver diverges from .NET here.** Hedging in this driver is +activated **independently of PPAF and PPCB** — whenever the account +has ≥ 2 applicable preferred regions and the user has not opted out, +the driver-default `HedgingStrategy` is used (§5.2). Rationale: the +Rust driver is greenfield and has no backward-compatibility constraint +that forced .NET v3 / Java v4 to gate hedging on PPAF. PPCB is fed by +hedging via `record_consecutive_hedge_win` (§9.5) but does not gate +the hedge decision. Opt-out is via `AvailabilityStrategy::Disabled` or +`AZURE_COSMOS_HEDGING_DISABLED=true`. See §5.2 for the full +activation rules. + +### 2.7 Diagnostics + +The .NET SDK attaches three diagnostic data points to the winning response: +- **Hedge Config** — threshold/step/write-enabled settings string. +- **Hedge Context** — list of regions that were contacted (up to and including the + winning request's ordinal). +- **Response Region** — the target region name of the winning request. + +--- + +## 3. Architectural Overview + +### 3.1 Where Hedging Sits in the Driver + +Hedging is an **in-pipeline decision** taken by +`evaluate_transport_result` (TPS §3.4) and dispatched by STAGE 7 of +the operation pipeline (TPS §4.1). When the evaluator decides a +hedged attempt is warranted, it returns +`OperationAction::Hedge { secondary_routing }`; STAGE 7 then calls +`execute_hedged()` (TPS §4.2), which races the primary attempt +against a single secondary attempt in `secondary_routing.region`. + +``` +CosmosDriver.execute_operation() + │ + ▼ + execute_operation_pipeline() [TPS §4.1, unchanged] + │ STAGE 1 snapshot → STAGE 2 routing → STAGE 3 build request + │ + ▼ + STAGE 4 transport attempt (primary) + │ + ▼ + STAGE 5 evaluate_transport_result [TPS §3.4] + │ returns (OperationAction, Vec) + │ └─ may produce OperationAction::Hedge { secondary_routing } + │ + ▼ + STAGE 6 apply LocationEffects [unchanged] + │ + ▼ + STAGE 7 dispatch on OperationAction + │ └─ OperationAction::Hedge → execute_hedged() [TPS §4.2] + │ │ + │ ▼ + │ tokio::select! + │ primary_fut vs sleep(threshold) + │ then race primary_fut vs secondary_fut + │ │ + │ ▼ + │ first non-transient response wins; + │ loser is dropped (cancelled by Drop) + │ │ + ◄─────────────────────────────────┘ + │ + ▼ + CosmosResponse + HedgeDiagnostics (attached on the winning response) +``` + +**Rationale.** Hedging composes with the existing per-attempt retry +layer rather than wrapping it. The `evaluate_transport_result` decision +function is the single place every routing/retry/hedge decision is +made, so hedging eligibility and secondary-region selection sit +exactly where they belong. There is no parallel orchestrator above the +pipeline and no `cosmos_driver.rs`-level wrapper. + +### 3.2 Core Data Flow + +``` + STAGE 5 + evaluate_transport_result(...) + returns OperationAction::Hedge { + secondary_routing: RoutingDecision { + region: regions[1], + excluded_regions: user ∪ (all_regions \ regions[1]), + } + } + │ + ▼ + STAGE 7 + execute_hedged( + primary_routing, // STAGE 2 result + secondary_routing, // from the action + threshold, + … + ) + │ + ┌───────────────────────────────┐ + │ tokio::pin!(primary_fut); │ + │ tokio::select! biased; { │ + │ primary_fut → return, │ ← zero-overhead + │ sleep(threshold) → hedge │ happy path + │ } │ + └─────────────┬─────────────────┘ + ▼ + ┌─────────────────────────────────┐ + │ Build secondary transport │ + │ with ExecutionContext:: │ + │ Hedging marker (§10). │ + │ Race primary vs secondary │ + │ via tokio::select!. │ + └─────────────┬───────────────────┘ + ▼ + First final result wins; the loser's future is + dropped (its in-flight transport `Drop`s emit a + cancellation signal at the next await point). + │ + ▼ + Return CosmosResponse + HedgeDiagnostics +``` + +### 3.3 Design Principles + +1. **Single decision enum.** Hedging is dispatched via + `OperationAction::Hedge { secondary_routing }` returned by + `evaluate_transport_result`. There is no separate eligibility + gate above the pipeline and no parallel orchestrator. +2. **Structural cancellation.** The secondary's future is owned by + `tokio::select!` inside `execute_hedged()`; dropping it cancels + the in-flight transport via the standard `Drop` chain. No + `CancellationToken` is required in the hedge path. +3. **Immutable request cloning.** The `CosmosOperation` (which + contains an `Arc`-backed body, headers, partition key) is cheap + to clone. The secondary uses a different `RoutingDecision` (and + therefore a different `ExcludeRegions` set) but shares the same + `CosmosOperation`. +4. **Respect existing systems.** Hedging composes with PPAF / PPCB, + session consistency, and throughput control because each attempt + re-enters the per-attempt pipeline (TPS §4). A repeated + alternate-region win is fed back into PPCB via + `record_hedge_win()` (§9.5). + +### 3.4 Reconciliation with `TRANSPORT_PIPELINE_SPEC.md` §4.2 — **Resolved** + +This spec adopts the TPS §4.2 in-pipeline shape verbatim: +hedging is selected by `evaluate_transport_result` returning +`OperationAction::Hedge { secondary_routing }` (TPS §3.4) and is +executed by `execute_hedged()` from the STAGE 7 dispatch (TPS §4.1). + +The pseudocode in §6 is the **normative semantics** of the +`OperationAction::Hedge` arm: function signature, race shape, +zero-overhead happy path, and the diagnostics / PPCB-feedback +callsites are all load-bearing on the TPS-side implementation. + +**Invariants the merged TPS-side implementation MUST preserve:** + +- At most one alternate-region request in flight at any time + (§6.5 #1). +- `ExcludeRegions` is the region-pinning mechanism for the secondary + (§6.3, §8.4); the evaluator computes the set when it produces + `secondary_routing`. +- Zero-overhead happy path when the primary wins before the + threshold timer fires — no `Arc<>`, no clones, no + `CancellationToken` in this branch (§6.5 #3). +- Hedging-win feedback into PPCB via `record_hedge_win()` (§9.5) + is invoked by `execute_hedged()` immediately on a secondary win. +- App-cancellation re-raises with hedge diagnostics preserved + (§6.5 #7, §14.2). + +**Cross-spec follow-ups outside this spec:** + +- TPS §4.2 currently states *"For write operations, hedging is only + enabled on multi-write-region (MWR) accounts"*. This spec drops + write hedging entirely (§1 Non-Goals, per Fabian F5). The TPS + §4.2 wording must be updated to match — tracked as an open + follow-up against the TPS owner. +- TPS §4.2 specifies a **dynamic P99-based threshold clamped to + 50–4000 ms**; this spec specifies a **static + `min(1000ms, request_timeout / 2)` driver default** (§5.2). The + threshold-policy disagreement is tracked as a new open question + (§17 Q11) pending cross-team alignment. + +--- + +## 4. Configuration Surface + +### 4.1 HedgingStrategy Type + +```rust +/// A validated, non-zero hedging threshold. +/// +/// Newtype around `Duration` whose only constructor rejects zero — +/// trades a fallible-construct ceremony per call site for a single +/// `Option`-returning constructor that compiles away when the input is +/// known at compile time. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct HedgeThreshold(Duration); + +impl HedgeThreshold { + /// Returns `None` if `duration` is zero. + pub const fn new(duration: Duration) -> Option { + if duration.is_zero() { None } else { Some(Self(duration)) } + } + + pub const fn get(self) -> Duration { self.0 } +} + +/// Cross-region hedging availability strategy. +/// +/// When the primary request does not complete within `threshold`, the driver +/// sends a single speculative request to the next preferred region. +/// The first non-transient response wins; the loser is cancelled. +/// +/// At most one alternate-region request is in flight at any time — the +/// driver does not fan out to a third region. +#[derive(Clone, Copy, Debug)] +#[non_exhaustive] +pub struct HedgingStrategy { + threshold: HedgeThreshold, +} + +impl HedgingStrategy { + /// Creates a new hedging strategy with the given threshold. + pub const fn new(threshold: HedgeThreshold) -> Self { + Self { threshold } + } + + /// Returns the threshold before the alternate-region hedge fires. + pub const fn threshold(self) -> HedgeThreshold { self.threshold } +} +``` + +There is no separate "write hedging" flag, no `threshold_step`, no +fallible constructor, and no SDK-default factory. The single knob is +`threshold`; everything else (eligibility, region selection, cancellation) +is driven by spec rules rather than configuration. + +> **Divergence from .NET and Java.** +> +> - **No N-region fan-out / no `threshold_step`.** .NET's +> `thresholdStep` and Java's `DEFAULT_THRESHOLD_STEP = 100ms` exist +> only to schedule the second, third, …, Nth hedge. The Rust driver +> caps fan-out at one alternate region (max two concurrent +> requests), so a step is unnecessary. +> - **No write hedging knob.** .NET's `enableMultiWriteRegionHedge` +> and Java's implicit write-hedging path are intentionally dropped +> (see §1 Non-Goals). +> - **No fallible constructor on `HedgingStrategy`.** Validation moves +> into the `HedgeThreshold` newtype (`Option` on zero), so the +> only error surface is at the boundary where the duration is +> constructed. +> - **No `HedgingStrategy::default()` / no-arg constructor.** Users +> always pick a threshold explicitly. The driver-wide default-on +> activation (§5.2) uses an internal default that is not part of the +> public API. + +### 4.2 Disabled Strategy + +```rust +/// Sentinel value used to disable hedging for a specific operation when a +/// client-level strategy is configured. +/// +/// ```rust +/// let options = ItemReadOptionsBuilder::new() +/// .with_availability_strategy(AvailabilityStrategy::Disabled) +/// .build(); +/// ``` +#[derive(Clone, Debug)] +pub enum AvailabilityStrategy { + /// Cross-region hedging strategy. + Hedging(HedgingStrategy), + /// Explicitly disable the client-level strategy for this request. + Disabled, +} +``` + +> **Interaction with default-on activation (§5.2).** Because hedging +> is on by default for accounts that satisfy §5.1, setting +> `AvailabilityStrategy::Disabled` at the **client** level is the +> code-level kill switch: it suppresses the §5.2 driver default for +> every operation on that client and is equivalent (in effect) to +> `AZURE_COSMOS_HEDGING_DISABLED=true` at deploy time. Setting +> `Disabled` on a single operation suppresses only that operation; +> sibling operations continue to use the client-level strategy or the +> §5.2 default. The full precedence chain is in §11.3.1. + +### 4.3 Integration with OperationOptions + +```rust +// In OperationOptions (layered resolution: operation > account > runtime > env) +pub struct OperationOptions { + // ... existing fields ... + + /// Availability strategy controlling cross-region hedging. + /// + /// When set to `Some(AvailabilityStrategy::Hedging(..))`, the driver will + /// speculatively send requests to alternate regions after the configured + /// threshold. Set to `Some(AvailabilityStrategy::Disabled)` to suppress a + /// client-level strategy for this request. + pub availability_strategy: Option, +} +``` + +### 4.4 Environment Variable Support + +| Variable | Description | Default | +|----------|-------------|---------| +| `AZURE_COSMOS_HEDGING_THRESHOLD_MS` | Overrides the driver default threshold in milliseconds. Zero or non-numeric values are ignored. | (driver default — see §5.2) | +| `AZURE_COSMOS_HEDGING_DISABLED` | When `true`, disables hedging entirely at runtime regardless of code-level config. Useful as a deployment-time kill switch. | `false` | + +The env-var threshold sits at priority 3 in the resolution order +(§11.3.1) — it overrides the built-in default but is overridden by any +code-level `AvailabilityStrategy` set on the client or operation. +`AZURE_COSMOS_HEDGING_DISABLED=true` is equivalent to setting +`AvailabilityStrategy::Disabled` at the client level. + +There is no env var for `threshold_step`, write hedging, or SDK-default +suppression because none of those features exist (see §4.1 divergence +note). + +--- + +## 5. Eligibility Rules + +### 5.1 `should_hedge()` — Pure Function + +`should_hedge()` is a pure helper consulted by +`evaluate_transport_result` (TPS §3.4) when deciding whether to +return `OperationAction::Hedge`. It is called once per per-attempt +pipeline iteration; there is no orchestrator above the pipeline that +calls it separately. + +```rust +/// Determines whether the given operation should use hedging. +/// +/// Returns `false` if: +/// - No hedging strategy is resolved (or explicitly `Disabled`) +/// - Application preferred-region list empty or has < 2 entries after +/// `ExcludeRegions` filtering +/// - Operation is not in the phase-allowed ResourceType set +/// - Operation is a write (writes are never hedged — see §1 Non-Goals) +fn should_hedge( + strategy: &HedgingStrategy, + operation: &CosmosOperation, + account_state: &AccountEndpointState, +) -> bool +``` + +**Decision matrix** — evaluated in order; first matching row wins: + +| # | Condition | Hedge? | +|---:|-----------|--------| +| 1 | No strategy resolved (or `AvailabilityStrategy::Disabled`) | No | +| 2 | Application preferred-region list empty | No | +| 3 | `ResourceType` not in the **phase-allowed set** † | No | +| 4 | Operation is a write (any topology) | No | +| 5 | Applicable `preferred_read_endpoints` (after `ExcludeRegions`) has < 2 entries | No | +| 6 | Read with ≥ 2 applicable read endpoints | **Yes** | + +The "≥ 2 applicable endpoints" check is computed against the +post-`ExcludeRegions` list, not the raw account region count — a user +who excludes all-but-one region at the operation level will (correctly) +skip hedging even on a multi-region account. + +> **† Phase-allowed `ResourceType` set.** Row 3's allowed set evolves +> with the implementation phases in §16: +> +> | Phase | Allowed `ResourceType` set | +> |---|---| +> | 1 (MVP) | `{Document}` for point reads only — enforced by an additional `OperationType` guard inside the predicate (reads only, no writes). | +> | 2 | Phase 1 set ∪ feed-style operations (Query / ReadMany / ChangeFeed — still `ResourceType.Document` but `OperationType` differs) ∪ `{Database, Container, Offer, Throughput}` (metadata reads). | +> | Future | Phase 2 set ∪ `{StoredProcedure}` (sprocs only — triggers / UDFs are not standalone operations). | +> +> Phase 1 implementations should hard-code the allowed `OperationType` +> set to `{Read}`. Each subsequent phase widens the constants in one +> place; no other change to §5.1 is required. + +### 5.2 Default activation + +Hedging is **on by default** for accounts that satisfy the §5.1 +eligibility rules — it is independent of PPAF and PPCB. Rationale: the +Rust driver is greenfield, so we do not need the .NET / Java +PPAF-coupled opt-in to preserve backward compatibility. Tail-latency +protection is a generally useful default; users who do not want it can +opt out at any layer via `AvailabilityStrategy::Disabled` (§4.2) or +the `AZURE_COSMOS_HEDGING_DISABLED` env var (§4.4). + +**Driver default values** (used when no user strategy is configured): + +- **Threshold:** `min(1000ms, request_timeout / 2)`, with `1000ms` as + the fallback when `request_timeout` is unset or zero. Matches the + .NET v3 default-threshold formula; chosen because it is conservative + on accounts with sub-second p50 latency and degrades gracefully + under high configured request timeouts. + +**Activation preconditions** — even with hedging on by default, the +`OperationAction::Hedge` is not produced at runtime for operations that fail any of: + +1. **At least two applicable preferred regions.** Single-region + accounts and operations whose `ExcludeRegions` filter leaves only + one applicable region skip hedging. +2. **Application-level region configuration is required.** The driver + must have a non-empty preferred-region list (set via + `DriverOptions::preferred_regions` or + `OperationOptions::application_preferred_regions`) so the + evaluator has a deterministic alternate-region target. Without + it, hedging is silently skipped. + +Both checks are enforced inside `should_hedge()`; failure of either +short-circuits the hedge decision before the primary request is sent. + +> **Cross-SDK comparison.** .NET v3 couples hedging activation to +> PPAF (auto-enable only when PPAF is on); Java v4 does the same and +> additionally exposes +> `COSMOS.IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF=false` as a +> runtime opt-out. Both coupling decisions exist because PPAF in +> those SDKs is opt-in and signals an "availability-oriented" +> deployment. The Rust driver is greenfield and treats hedging as a +> stand-alone tail-latency tool, so this coupling is dropped — +> hedging is on by default and PPAF / PPCB activation is unrelated. +> Java's threshold (`500ms`) and .NET's (`min(1000ms, RequestTimeout/2)`) +> bracket the Rust default; users targeting Java parity should configure +> `500ms` explicitly. + +--- + +## 6. Hedging Algorithm + +> **Reading guide.** This section is the **normative semantics** of +> the `OperationAction::Hedge` arm dispatched by STAGE 7 of the +> operation pipeline (TPS §4.1). The signatures below match the TPS +> shape so that anyone implementing TPS §4.2 can lift the body +> directly. The race shape, zero-overhead happy path, app-cancel +> harvesting, and PPCB-feedback callsite are load-bearing. + +### 6.1 Overview + +Hedging is dispatched in two places: + +1. **`evaluate_transport_result` (TPS §3.4)** decides whether the + per-attempt result warrants a hedge. When `should_hedge()` (§5.1) + says yes and the primary has not yet produced a final result, the + evaluator returns + `OperationAction::Hedge { secondary_routing: RoutingDecision }`. + The secondary `RoutingDecision` is computed by the evaluator + (§6.2 / §6.3) and carries the secondary region plus the + `ExcludeRegions` set that pins the hedge to that region. +2. **`execute_hedged()` (called from STAGE 7)** consumes both + `RoutingDecision`s, races the primary attempt against a single + secondary attempt, and returns the first non-transient response. + +Both functions live in `operation_pipeline.rs`; there is no separate +orchestrator and no `cosmos_driver.rs`-level wrapper. + +```rust +// Selected by TPS §3.4. +OperationAction::Hedge { + secondary_routing: RoutingDecision, +} + +// Dispatched by TPS §4.1 STAGE 7. +async fn execute_hedged( + operation: &CosmosOperation, + options: &OperationOptions, + primary_routing: &RoutingDecision, + secondary_routing: &RoutingDecision, + threshold: HedgeThreshold, + session: &SessionState, + transport: &AdaptiveTransport, + credential: &Credential, + diagnostics: &mut DiagnosticsContextBuilder, + deadline: Option, +) -> azure_core::Result; +``` + +`execute_hedged()` fires **at most two** concurrent transport +attempts: the primary at `t=0`, and a single secondary attempt at +`t=threshold` if the primary has not yet produced a final result. +There is no third hedge, no `threshold_step`, and no N-region +fan-out. + +### 6.2 Region Selection (computed by `evaluate_transport_result`) + +The primary uses normal routing — STAGE 2's `resolve_endpoint()` +result. The secondary is `applicable_read_endpoints[1]` after +`ExcludeRegions` filtering — i.e. the second region in the user's +preferred-region list that is not currently excluded. If no such +region exists, the evaluator does **not** return +`OperationAction::Hedge` (§5.1 row 5); the operation pipeline takes +some other action (`Complete`, `FailoverRetry`, `SessionRetry`, or +`Abort`). + +``` +regions = applicable_read_endpoints(excluded_regions) +// regions[0] = primary (normal routing — primary_routing.region) +// regions[1] = secondary (the hedge target — secondary_routing.region) +// regions[2..] = unused by hedging in this driver +``` + +### 6.3 Request Routing via ExcludeRegions (also evaluator-side) + +The secondary is pinned to its target region by setting +`secondary_routing.excluded_regions = user ∪ (all_regions \ regions[1])`. +This is computed by the evaluator when it builds the +`secondary_routing: RoutingDecision`; `execute_hedged()` itself does +no routing math. + +| Request | ExcludeRegions | Target | +|---|---|---| +| Primary | (the user's original exclusion set, if any) | regions[0] (normal routing) | +| Secondary | user-original ∪ `(all_regions \ regions[1])` | regions[1] | + +This piggybacks on the existing `ExcludeRegions` mechanism in +`resolve_endpoint()` (TPS §4.1 STAGE 2), requiring no changes to the +endpoint resolution logic, and composes with any user-specified +`ExcludeRegions` (the secondary's exclusion set is the *union* of the +user's and the per-hedge pin). + +### 6.4 Execution Flow (Pseudocode) + +`execute_hedged()` races the primary attempt against a single +threshold timer. If the timer wins, it builds the secondary transport +request from `secondary_routing` and races primary vs secondary via +`tokio::select!`. There is no `CancellationToken`: the loser's future +is dropped, which `Drop`s the in-flight transport (TPS §5.1) and emits +the cancellation signal at the next transport await point. + +```rust +async fn execute_hedged( + operation: &CosmosOperation, + options: &OperationOptions, + primary_routing: &RoutingDecision, + secondary_routing: &RoutingDecision, + threshold: HedgeThreshold, + session: &SessionState, + transport: &AdaptiveTransport, + credential: &Credential, + diagnostics: &mut DiagnosticsContextBuilder, + deadline: Option, +) -> Result { + // ── Build the primary transport request. The secondary is NOT built + // yet — every allocation below is gated on the threshold timer + // firing (§6.5 invariant #3). ── + let primary_req = build_transport_request( + operation, primary_routing, session, options, deadline, + ); + let primary_fut = execute_transport_pipeline( + primary_req, transport, credential, diagnostics, + ); + tokio::pin!(primary_fut); + + // ── Happy path: just await the primary against the threshold timer. ── + tokio::select! { + biased; + + // Primary returned before the threshold. Attach a "no hedge fired" + // HedgeDiagnostics and return — zero-overhead happy path (§6.5 #3). + result = &mut primary_fut => { + diagnostics.record_attempt(&result); + return evaluate_and_return( + operation, result, diagnostics, + HedgeDiagnostics::primary_only(threshold, &primary_routing.region), + ); + } + + // Threshold elapsed → fall through to the hedged race below. + _ = sleep(threshold.get()) => {} + } + + // ── Spawn the secondary. From here on, both futures are pinned to + // this stack frame. The shared hub-region latch (§9.6) is + // constructed here too, after the threshold fires. ── + let shared_hub_region_latch = if hub_region_latch_eligible(operation, options) { + Some(Arc::new(AtomicBool::new(false))) + } else { + None + }; + + let secondary_req = build_transport_request( + operation, secondary_routing, session, options, deadline, + ) + .with_execution_context(ExecutionContext::Hedging) + .with_shared_hub_region_latch(shared_hub_region_latch.clone()); + + let secondary_fut = execute_transport_pipeline( + secondary_req, transport, credential, diagnostics, + ); + tokio::pin!(secondary_fut); + + // ── Race primary vs secondary. First final result wins; the loser's + // future is dropped (Drop chain cancels the in-flight transport). + // A transient result on either side keeps the *other* side racing. + // Application cancellation is observed by the surrounding + // `select!` arms via the deadline — no CancellationToken tree. ── + let mut last_transient: Option<(Side, azure_core::Error)> = None; + let mut primary_done = false; + let mut secondary_done = false; + + while !(primary_done && secondary_done) { + tokio::select! { + biased; + + // App-cancel observed via deadline → harvest the most-advanced + // pipeline within HARVEST_WINDOW for diagnostics, then re-raise. + _ = app_cancel_signal(deadline) => { + let err = harvest_app_cancel_error( + &mut primary_fut, primary_done, + &mut secondary_fut, secondary_done, + threshold, primary_routing, secondary_routing, + ).await; + return Err(err); + } + + r = &mut primary_fut, if !primary_done => { + primary_done = true; + diagnostics.record_primary_attempt(&r); + match classify(r) { + Outcome::Final(resp) => { + return Ok(decorate( + resp, threshold, + primary_routing, secondary_routing, + Side::Primary, + )); + } + Outcome::Transient(err) => { + last_transient = Some((Side::Primary, err)); + } + } + } + + r = &mut secondary_fut, if !secondary_done => { + secondary_done = true; + diagnostics.record_hedged_attempt(&r); + match classify(r) { + Outcome::Final(resp) => { + // Repeated secondary wins feed back into PPCB — see §9.5. + record_hedge_win(&primary_routing.partition_key_range_id, + &primary_routing.region); + return Ok(decorate( + resp, threshold, + primary_routing, secondary_routing, + Side::Secondary, + )); + } + Outcome::Transient(err) => { + last_transient = Some((Side::Secondary, err)); + } + } + } + } + } + + // ── Both sides terminated transient — surface the most recent error. ── + Err(last_transient.map(|(_, e)| e).unwrap_or_else(|| { + azure_core::Error::message( + azure_core::error::ErrorKind::Other, + "hedging completed without producing a response", + ) + })) +} +``` + +#### 6.4.1 Helper sketches + +```rust +enum Side { Primary, Secondary } +enum Outcome { Final(CosmosResponse), Transient(azure_core::Error) } + +fn classify(r: Result) -> Outcome { + match r { + Ok(resp) if is_final_result(resp.status()) => Outcome::Final(resp), + Ok(resp) => Outcome::Transient(transient_from_response(resp)), + Err(err) => Outcome::Transient(err), + } +} + +/// Build the secondary `RoutingDecision` inside `evaluate_transport_result`. +/// This is what populates `OperationAction::Hedge { secondary_routing }`; +/// `execute_hedged()` does NOT compute routing. +fn build_secondary_routing( + primary: &RoutingDecision, + user_excluded: &[Region], + regions: &[Region], +) -> RoutingDecision { + let mut excluded: Vec = user_excluded.to_vec(); + for (i, r) in regions.iter().enumerate() { + if i != /* secondary index */ 1 && !excluded.contains(r) { + excluded.push(r.clone()); + } + } + RoutingDecision { + region: regions[1].clone(), + excluded_regions: excluded, + partition_key_range_id: primary.partition_key_range_id.clone(), + // ... other RoutingDecision fields inherited from primary ... + } +} + +fn decorate( + mut resp: CosmosResponse, + threshold: HedgeThreshold, + primary: &RoutingDecision, + secondary: &RoutingDecision, + winner: Side, +) -> CosmosResponse { + let regions_contacted = match winner { + Side::Primary => vec![primary.region.clone()], + Side::Secondary => vec![primary.region.clone(), secondary.region.clone()], + }; + let response_region = match winner { + Side::Primary => primary.region.clone(), + Side::Secondary => secondary.region.clone(), + }; + resp.attach_hedge_diagnostics(HedgeDiagnostics { + strategy_config: HedgingStrategyConfig { threshold }, + regions_contacted, + response_region, + total_requests_launched: if matches!(winner, Side::Secondary) { 2 } else { 2 }, + was_hedge: matches!(winner, Side::Secondary), + }); + resp +} +``` + +#### 6.4.2 Ownership & Sharing + +`execute_hedged()` avoids `tokio::spawn` and `FuturesUnordered` +entirely in both the happy path and the hedged race — both futures +are pinned on the stack and polled by `tokio::select!`. This keeps +the allocator out of the hot path and removes the `'static + Send` +constraint that a `JoinHandle` model would impose. + +**Cancellation is structural, not signalled.** When the primary wins, +the secondary's pinned future is dropped — its in-flight transport +`Drop`s and emits the cancellation signal at the next transport +`await` point (TPS §5.1). There is no `CancellationToken` in the +hedge path. The same property holds for the surrounding code: if the +caller drops `execute_hedged()`'s future (e.g. via +`tokio::time::timeout` one layer up), both the primary and secondary +futures are dropped together by the standard structured-concurrency +chain. + +### 6.5 Key Invariants + +1. **Max two concurrent transport attempts.** Primary + at most one + secondary. The driver does not fan out to a third region under any + circumstance. +2. **Primary fires immediately.** Zero additional latency on the + happy path — `execute_hedged()`'s entry is a single + `tokio::select!` with two arms (primary future, threshold timer). +3. **Zero-overhead happy path.** If the primary returns a final + result before the threshold timer elapses, `execute_hedged()` + MUST NOT: + - Build the secondary `transport_request`. + - Construct the `Arc` shared hub-region latch (§9.6). + - Allocate any `Vec` per-hedge. + + This is a load-bearing performance constraint and is gated by the + `hedging_zero_overhead_happy_path_no_allocs` benchmark in §15. +4. **Region pinning is hard.** The secondary's per-attempt retry + layer honors its `ExcludeRegions` set for every retry trigger + class — it cannot fall back to the primary's region (§8.4). +5. **Cancellation is structural.** Dropping the loser's future via + `tokio::select!` cancels its in-flight transport via the standard + `Drop` chain. No `CancellationToken` is required in the hedge + path; the per-attempt deadline check inside the transport + pipeline (TPS §5.1) handles deadline-based cancellation. +6. **Single writer to diagnostics.** Only the winning response gets + `HedgeDiagnostics` attached; when the primary wins before the + threshold elapses, a synthetic "primary-only" diagnostics record + is used so consumers can tell *"hedging was selected but never + fanned out"* apart from *"hedging was not selected"*. The on-wire + marker for a hedged secondary attempt is + `ExecutionContext::Hedging` (TPS §3.4) on the secondary's + `transport_request`. +7. **App-cancel preserves hedge trace.** When the application's + cancellation (observed via the deadline) fires while both + attempts are racing, `execute_hedged()` harvests the in-flight + futures within a bounded `HARVEST_WINDOW = 50ms` and attaches the + most-advanced result's diagnostics to the returned + `application_cancelled_error()`. Mirrors .NET v3's behavior of + awaiting the most-advanced task before re-raising. +8. **Secondary wins feed back into PPCB.** A win by the secondary is + an out-of-band signal that the primary partition is degraded — + `execute_hedged()` records it via `record_hedge_win()` (§9.5) so + PPCB can mark the partition `Unhealthy` after the configured + number of consecutive secondary wins. +9. **Single decision enum.** Hedging is selected by + `evaluate_transport_result` returning + `OperationAction::Hedge { secondary_routing }`; there is no + parallel orchestrator and no separate cancellation tree above the + pipeline. The `OperationAction::Hedge` arm is the **only** entry + point to `execute_hedged()`. + +--- + +## 7. Final Result Classification + +### 7.1 `is_final_result()` — Pure Function + +```rust +/// Determines whether a response status code is a final (non-transient) result. +/// +/// Final results terminate hedging immediately. Transient results allow other +/// in-flight hedges to continue racing for a better outcome. +/// +/// Note: 403 (with or without sub-status `3` WriteForbidden) is **transient** +/// for hedging — it typically indicates the targeted region cannot serve the +/// request right now (account-level failover, write-region change). The retry +/// pipeline running *inside* each hedge may treat `403/3` as a redirect +/// trigger; that is independent of this classification. +fn is_final_result(status: &CosmosStatus) -> bool { + let code = status.http_status_code; + let sub = status.sub_status_code; + + // All 1xx, 2xx, 3xx → final + if code < 400 { + return true; + } + + // Specific client errors that are definitively non-transient + matches!(code, + 400 // Bad Request + | 401 // Unauthorized + | 405 // Method Not Allowed + | 409 // Conflict + | 412 // Precondition Failed + | 413 // Request Entity Too Large + ) || (code == 404 && sub == 0) // Not Found with no sub-status +} +``` + +### 7.2 Transient vs. Non-Transient Responses + +| Status | Sub-Status | Transient? | Rationale | +|--------|------------|------------|-----------| +| 200 | * | No (final) | Success | +| 304 | * | No (final) | Not Modified | +| 400 | * | No (final) | Client error — won't succeed in another region | +| 401 | * | No (final) | Auth failure — same credentials everywhere | +| 403 | 0 (no sub) | **Yes** | Forbidden — may indicate a regional failover in progress; another region may serve | +| 403 | 3 | **Yes** | WriteForbidden — region may be failing over | +| 404 | 0 | No (final) | Resource genuinely not found | +| 404 | 1002 | **Yes** | ReadSessionNotAvailable — session lag | +| 405 | * | No (final) | Wrong HTTP method | +| 408 | * | **Yes** | Timeout — another region may be faster | +| 409 | * | No (final) | Conflict — deterministic | +| 410 | * | **Yes** | Gone — partition may have moved | +| 412 | * | No (final) | Precondition — deterministic | +| 413 | * | No (final) | Payload too large — same everywhere | +| 429 | * | **Yes** | Throttled — another region may have capacity | +| 500 | * | **Yes** | Internal error — may be region-specific | +| 503 | * | **Yes** | Unavailable — another region may be healthy | + +> **Note on 403 sub-statuses.** The driver classifies any 403 (with or +> without `WriteForbidden` sub-status `3`) as **transient** for hedging +> purposes — a 403 typically signals that the targeted region cannot +> currently serve the request (account-level failover in progress, write +> region change, etc.), and the *correct* action is to keep racing other +> in-flight hedges. This matches .NET v3's `IsFinalResult` behavior. Note +> that the *retry layer* may treat `403/3` differently (PPAF write retry +> consumes it as a write-redirect signal); the hedging classification +> here governs only whether hedging itself terminates early, not the +> retry pipeline that runs *inside* each hedge. + +--- + +## 8. Operation Pipeline Integration + +### 8.1 Entry Point Changes + +**`cosmos_driver.rs::execute_operation()` does not change.** Hedging +is selected entirely inside the per-attempt pipeline by +`evaluate_transport_result` (TPS §3.4) returning +`OperationAction::Hedge { secondary_routing }`, and dispatched by +STAGE 7 of `execute_operation_pipeline()` (TPS §4.1) calling +`execute_hedged()` (§6.1, TPS §4.2). + +The only changes outside the transport pipeline are: + +- `OperationOptions` gains an `availability_strategy: + Option` field (§4.3). +- `evaluate_transport_result` consults the resolved + `HedgingStrategy` (§4) and `should_hedge()` (§5.1) when deciding + whether to return `OperationAction::Hedge`. + +The driver does not need a top-level wrapper: `execute_operation()` +calls `execute_operation_pipeline()` exactly as it does today. + +### 8.2 Operation Cloning + +Each hedged invocation needs its own: +- `OperationRetryState` — independent retry counters per region +- `DiagnosticsContextBuilder` — separate diagnostics chain per attempt +- `OperationOptions` — different `ExcludedRegions` per hedge + +Items shared (via `Arc` or reference): +- `CosmosOperation` — immutable; body is `Bytes` (cheaply cloneable) +- `LocationStateStore` — lock-free; multiple readers are safe +- `SessionManager` — designed for concurrent access +- `Credential` — `Arc`-wrapped +- **Hub-region-processing-only latch** — a single `Arc` + is shared across the primary and the alternate hedge for the + lifetime of the outer operation. See §9.6 for the full rationale; + the short version is that the per-`OperationRetryState` + `hub_region_processing_only` field added by [PR #4389][pr-4389] is + otherwise per-hedge, which would force the alternate hedge to + independently re-discover the hub region via its own 404/1002 + cycle. .NET v3 hit and fixed this in + [azure-cosmos-dotnet-v3 PR #5815][dotnet-pr-5815] via the + `CrossRegionAvailabilityContext` shared object; the Rust driver + adopts the equivalent shared signal. + +[pr-4389]: https://github.com/Azure/azure-sdk-for-rust/pull/4389 +[dotnet-pr-5815]: https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5815 + +### 8.3 Cancellation Propagation + +Cancellation in the hedge path is **structural**, not signalled. +`execute_hedged()` owns both futures via `tokio::select!`; the loser +is dropped, which `Drop`s its in-flight transport (TPS §5.1) and +emits the cancellation signal at the next transport `await` point. +No `CancellationToken` is constructed. + +``` + execute_hedged() + │ + ┌─────────────────────────┐ + │ tokio::select! biased; { │ + │ primary_fut, │ + │ secondary_fut, │ + │ deadline_signal(), │ + │ } │ + └────────────┬──────────────┘ + ▼ + winner returned → loser future dropped + │ + ▼ + Drop chain runs through the transport pipeline + → in-flight HTTP/AMQP request is cancelled at the + next await point (TPS §5.1). +``` + +Application-cancellation enters via the per-attempt deadline (the +same mechanism used everywhere else in the transport pipeline). When +the deadline fires while both attempts are racing, +`execute_hedged()` harvests the most-advanced future for +diagnostics within `HARVEST_WINDOW = 50ms` (§6.5 #7) before +returning the cancellation error. + +### 8.4 Local-Only Retries Inside a Hedge (Contract) + +> **Contract:** Each hedged pipeline invocation runs the **full operation +> pipeline including the retry layer**, but the retry layer must perform +> **local-only retries** — it is forbidden from re-routing the request to +> a different region than the one targeted by that hedge. + +This matches the .NET v3 behavior documented in +[Cross Region Request Hedging.md](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/0830090ca4dc47b71398ea0871cbe00b591aa8fc/docs/Cross%20Region%20Request%20Hedging.md): +*"hedged requests are restricted to the region they are sent out in so no +cross region retries will be made, only local retries."* + +**Mechanism.** `execute_hedged()` enforces this implicitly via +`ExcludeRegions` (see §6.3): hedge `N` is sent with +`ExcludeRegions = all_regions \ { regions[N] }`. The retry layer's +region-fallback logic (PPAF / PPCB / 503 retry) consults +`ExcludeRegions` when picking the next endpoint and therefore has no +alternate region to fall back to — every retry attempt re-resolves to the +same target region. + +**Why this matters.** Without this property, two hedges launched against +different regions could converge onto the *same* fallback region during a +regional outage, defeating the hedge's purpose and inflating RU. + +**Implementation requirement.** The retry layer must: + +1. Always honor `ExcludeRegions` from the operation options when computing + the next endpoint after a retry trigger. +2. Treat "all eligible regions excluded" as a terminal condition — return + the last seen response/error rather than looping infinitely. +3. Not bypass `ExcludeRegions` for any retry trigger class (PPAF write + retry, PPCB markdown failback, transport-layer 503, throttling, etc.). + +This is a **cross-cutting invariant** that any new retry trigger added +after Phase 1 must respect. + +### 8.5 File Layout + +New files: +``` +src/driver/pipeline/ + hedging_diagnostics.rs # HedgeDiagnostics struct +``` + +Modified files: +``` +src/driver/pipeline/operation_pipeline.rs + # - evaluate_transport_result() may return OperationAction::Hedge + # - STAGE 7 dispatch arm calls execute_hedged() + # - execute_hedged() implementation (§6.4) +src/driver/pipeline/transport_pipeline.rs + # - ExecutionContext::Hedging marker on the secondary's transport_request +src/options/operation_options.rs # Add availability_strategy field +src/options/mod.rs # Export AvailabilityStrategy, HedgingStrategy +src/diagnostics/mod.rs # HedgeDiagnostics type +``` + +There is no `hedging.rs` module: the hedge code is a peer of the +rest of the pipeline dispatch in `operation_pipeline.rs`. + +--- + +## 9. Interaction with Existing Systems + +### 9.1 PPAF / PPCB + +Hedging and partition-level failover are **complementary**: + +| System | Handles | Trigger | +|--------|---------|---------| +| Hedging | Latency | Timer (threshold exceeded) | +| PPAF | Write failures (single-master) | 403/3 from service | +| PPCB | Read/write failures | Failure count threshold | + +**No interference:** Each hedged pipeline invocation has its own +`OperationRetryState`. Partition-level effects (`LocationEffect::MarkPartitionUnavailable`) +are applied via the shared `LocationStateStore`, which is lock-free and handles +concurrent CAS operations correctly. Multiple hedges marking the same partition +unavailable is idempotent (the CAS loop merges failure counts). + +### 9.2 Session Consistency + +**Challenge:** Session tokens are per-region. A hedged request to Region B may not +have the session token captured from a prior request that went to Region A. + +**Resolution:** The `SessionManager` is shared across all hedges. Each pipeline +invocation: +1. Reads the latest session token before sending (STAGE 3). +2. Captures the response session token after receiving (post-STAGE 4). + +Because hedges run in parallel, a hedge to Region B may use a session token from +Region A. If that fails with 404/1002, the pipeline's session retry logic handles +it internally — this is indistinguishable from normal session retry behavior. + +**Concurrent capture from competing hedges.** Even after `execute_hedged()` +cancels losing hedges and returns the winner, a losing hedge's transport +future may already have received a response in flight (cancellation is +best-effort — see §14.2). If both the winner and a losing hedge call back +into `SessionManager` to record their captured tokens, the order in which +the writes land is non-deterministic. The hedging design relies on +**`SessionManager` updates being commutative under max-LSN merge**: each +write merges the incoming token with the stored token by taking the higher +LSN per-partition. Under that contract, late writes from a losing hedge are +safe (they can only advance the stored LSN, never regress it) and the next +operation always observes a token at least as fresh as the winner's. + +This is a precondition on the existing `SessionManager` API — if it ever +moves to a last-write-wins model, `execute_hedged()` must instead +suppress session capture on hedges that observed cancellation before +STAGE 4 completed (or capture into a per-hedge buffer that only the winner +flushes). This invariant must be covered by the unit tests listed in §15.1. + +### 9.3 Throughput Control + +Each hedged request independently checks the throughput control group +budget. Hedging **does** increase RU consumption when the alternate +hedge actually executes transport. With the single-alternate model +(§6), the maximum RU multiplier introduced by hedging is **2×** — one +primary + one alternate. + +The throughput control snapshot is acquired per-attempt in the +operation pipeline (STAGE 3), so the alternate sees the latest budget. + +**Pathological interaction under TC saturation.** When the throughput +control group is saturated, both the primary and the alternate will be +throttled to 429 by the local TC gate before reaching the network. +`execute_hedged()` classifies 429 as transient (see §7.2), drains both +responses, and returns the most recent 429 — i.e., under TC saturation +hedging is at best a no-op and at worst adds 2× TC pressure plus the +threshold-timer latency on top. + +Because hedging is on by default (§5.2), operators on TC-saturated +accounts should explicitly opt out via `AvailabilityStrategy::Disabled` +on the driver or via `AZURE_COSMOS_HEDGING_DISABLED=true` at deploy +time. + +**Mitigations the implementation must adopt:** + +1. **Sizing guidance (operator-facing docs).** State explicitly that + the maximum RU multiplier introduced by hedging is **2×**, and TC + group budgets should be sized with that headroom in mind. +2. **Short-circuit on local TC 429.** If the primary returns a + TC-gate 429 *before* reaching transport (i.e., the throttle is + local rather than service-side), `execute_hedged()` SHOULD treat + that as a "do not fan out" signal — the alternate will hit the + same gate. Distinguish this from a service-side 429 (which is + genuinely region-local and benefits from hedging) via the response + source field. +3. **Optional: exempt the alternate from TC accounting.** + Speculative-hedge RU is not user-attributable; a losing alternate's + RU is wasted by definition. A future option MAY skip TC accounting + for the alternate hedge. Out of scope for Phase 1; tracked as a + follow-up. + +### 9.4 End-to-End Deadline + +If `EndToEndOperationLatencyPolicy` is configured, **both the primary +and the alternate share the same deadline**. The deadline is computed +once at the start of `execute_hedged()` (after the threshold timer +fires — see §6.5 #3 zero-overhead-happy-path) and threaded into +both the primary's and the secondary's `OperationRetryState` via +pipeline invocation. + +Implication: the alternate has less time budget than the primary. If +the deadline is 5s and the threshold is 3s, the alternate has only ~2s +to complete. + +### 9.5 Hedging-win feedback into PPCB + +A repeated win by the **alternate region** is signal: the primary +partition is consistently slow on this access pattern. The +`execute_hedged()` records each alternate-win via a callback into PPCB so +the circuit breaker can transition the primary partition to a degraded +state after a configurable number of consecutive wins. + +**Mechanism (sketch — pending PPCB owner sign-off):** + +```rust +// Called by `execute_hedged()` immediately after an alternate-region win, +// before returning the response to the caller. +fn record_hedge_win( + location_store: &LocationStateStore, + partition: PartitionKeyRangeId, + primary_region: &Region, +) { + location_store.record_consecutive_hedge_win(partition, primary_region); + // PPCB's internal threshold (e.g., 5 consecutive wins) triggers + // a transition to `Unhealthy` and the existing PPCB probe machinery + // takes over. A primary-region win on the same partition resets + // the counter. +} +``` + +**Invariants:** + +1. **Counter is per (partition, primary_region) pair.** A hedge-win + on partition P with primary region A does not affect partition Q. +2. **Primary-region wins reset the counter.** Any direct primary win + on the same partition clears the consecutive-hedge-win counter so + transient cross-region latency spikes do not accumulate. +3. **PPCB owns the threshold and the state transition.** The hedging + `execute_hedged()` only emits the signal; whether N hedge-wins trip the + breaker, and what state the partition transitions to, lives in the + PPCB module. +4. **Updates are lock-free / CAS-based.** Matches the existing + `LocationStateStore` contract (§9.1). + +**Status:** This contract crosses module boundaries; the exact PPCB +state transition and threshold constant are out of scope for this +spec and will be co-designed with the PPCB owner before Phase 1 ships. +The `execute_hedged()`-side callsite is the load-bearing commitment here — +any PPCB-side implementation that consumes `record_consecutive_hedge_win()` +satisfies this contract. + +### 9.6 Hub-Region-Processing-Only Header + +The driver emits the `x-ms-cosmos-hub-region-processing-only: True` +request header on retries triggered by a `404 / 1002 +(READ_SESSION_NOT_AVAILABLE)` response, scoped to **single-master +data-plane** operations. The header is specified in +[`HUB_REGION_PROCESSING_HEADER_SPEC.md`](https://github.com/Azure/azure-sdk-for-rust/blob/75c9c291528652355d62d2cd70af04b59a61e6d3/sdk/cosmos/azure_data_cosmos/docs/HUB_REGION_PROCESSING_HEADER_SPEC.md) +and implemented in [Rust PR #4389][pr-4389] (parity baseline with +[.NET PR #5447](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5447)). + +#### 9.6.1 The hedging-specific correctness gap + +The Rust latch lives on `OperationRetryState` +(`components.rs::OperationRetryState::hub_region_processing_only`) +and is set in `retry_evaluation.rs::build_session_retry_state` when +all four conditions hold: + +1. `is_dataplane` +2. `!can_use_multiple_write_locations` (single-master account) +3. `session_token_retry_count == 0` (first 1002 within the operation) +4. `!hub_region_processing_only` (idempotency) + +It is consumed in `operation_pipeline.rs::apply_hub_region_header` on +every subsequent transport attempt of the same operation. + +Per §8.2, **the alternate hedge has its own `OperationRetryState`**. +Without additional coordination, this means the primary and the +alternate would independently observe their own first 1002, each +pay the full hub-discovery latency, and re-issue the next attempt +with the header set. The header's purpose — *bound the discovery +cycle to a single 1002 round-trip per operation* — is defeated for +whichever side has not yet observed 1002. + +This is the same gap .NET v3 had after its first hub-region header +PR ([#5447](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5447)) +and **explicitly fixed** in +[PR #5815 — *Read Consistency Strategy: Adds hub region header for +LastCommittedWriteRegion strategy*][dotnet-pr-5815], in the section +*"Hedging request with hub region header"*: + +> When `CrossRegionHedgingAvailabilityStrategy` is active, the primary +> request may discover the hub region mid-flight … Hedged requests are +> clones of the original and run with their own `ClientRetryPolicy` +> instance, so they would normally repeat the entire hub discovery +> cycle independently. To avoid this redundant retry overhead, we +> introduce a `CrossRegionAvailabilityContext` — a lightweight shared +> object with a volatile `bool ShouldAddHubRegionProcessingOnlyHeader` +> flag. This context is injected into `RequestMessage.Properties` +> before the clone loop in `CrossRegionHedgingAvailabilityStrategy`. +> Since `Clone()` performs a shallow dictionary copy, all clones +> (primary + hedges) share the same `CrossRegionAvailabilityContext` +> reference. When the primary's `ClientRetryPolicy` sets the hub flag +> after 2× 404/1002, it also sets the flag on the shared context. +> Each hedge's `ClientRetryPolicy.OnBeforeSendRequest` reads this +> shared flag on every attempt and attaches the +> `x-ms-cosmos-hub-region-processing-only` header immediately — +> without needing to go through its own 404/1002 discovery. + +The Rust hedge path MUST adopt the equivalent design. + +#### 9.6.2 Required design — `Arc` shared latch + +Construct a single `Arc` in `execute_hedged()` after +the threshold elapses (so the happy path stays alloc-free — §6.5 #3) +**before the alternate hedge is spawned** (i.e. after the threshold +timer fires — keeps the §6.5 invariant #3 zero-overhead happy path +intact), and thread it into the primary's pipeline params as well. +Concretely: + +```rust +// In execute_hedged(), right after the threshold timer fires and +// before building the secondary transport_request: +let shared_hub_region_latch: Arc = Arc::new(AtomicBool::new(false)); + +// Both primary and alternate carry the same Arc. +let primary_retry_state = OperationRetryState::initial(/* … */) + .with_shared_hub_region_latch(shared_hub_region_latch.clone()); +let alternate_retry_state = OperationRetryState::initial(/* … */) + .with_shared_hub_region_latch(shared_hub_region_latch.clone()); +``` + +> **Happy-path note.** When the primary returns a final result before +> the threshold timer fires, no `Arc` is constructed and +> no latch threading happens — the per-state field added by +> [#4389][pr-4389] remains the only latch, exactly as today. This is +> required by §6.5 invariant #3 (zero-overhead happy path). + +This requires a small extension to `OperationRetryState`: + +```rust +pub struct OperationRetryState { + // … existing fields … + + /// Per-operation hub-region-processing-only latch. + /// Sticky for the lifetime of this `OperationRetryState`. + pub hub_region_processing_only: bool, + + /// Cross-hedge shared latch. `Some(_)` only when this operation is + /// running inside `execute_hedged()` past the threshold — + /// `None` on the non-hedged code path and on the zero-overhead + /// happy path, so today's allocator behavior is preserved. + /// + /// Mirrors .NET v3's `CrossRegionAvailabilityContext` injected + /// into `RequestMessage.Properties` before the clone loop + /// (azure-cosmos-dotnet-v3 PR #5815). + pub shared_hub_region_latch: Option>, +} +``` + +The two existing call sites are then extended: + +- **`build_session_retry_state` (latch-set side).** When the four + trigger conditions fire and the new state sets + `hub_region_processing_only = true`, also publish the result on + the shared latch if present: + + ```rust + if let Some(shared) = &retry_state.shared_hub_region_latch { + shared.store(true, Ordering::Release); + } + ``` + + `Release` is sufficient — the only thing being published is the + bool itself; no further state hangs off it. + +- **`apply_hub_region_header` (header-emission side).** Emit the + header when *either* the per-state latch is set or the shared + latch is set: + + ```rust + let emit = retry_state.hub_region_processing_only + || retry_state + .shared_hub_region_latch + .as_ref() + .map(|shared| shared.load(Ordering::Acquire)) + .unwrap_or(false); + if emit { + transport_request.headers.insert( + HeaderName::from_static(request_header_names::HUB_REGION_PROCESSING_ONLY), + HeaderValue::from_static("True"), + ); + } + ``` + +This preserves the §5/§7/§8 invariants of +`HUB_REGION_PROCESSING_HEADER_SPEC.md` (account-level scope, +data-plane scope, idempotency / sticky semantics) on a per-hedge +basis while also propagating the discovery from one side of the +race to the other as soon as it happens. + +#### 9.6.3 Eligibility — when the shared latch is actually wired + +The shared latch is populated only when all of the following are true +at the point the alternate hedge is about to spawn inside +`execute_hedged()`: + +| Condition | Why | +|---|---| +| Operation is data-plane (`is_dataplane`) | Mirrors the §1.5 scope of `HUB_REGION_PROCESSING_HEADER_SPEC.md`. | +| Account is single-master (`!can_use_multiple_write_locations`) | Mirrors AC-4 of `HUB_REGION_PROCESSING_HEADER_SPEC.md`; multi-master accounts have a separate recovery path and the header is never emitted. | +| Hedging actually fans out (threshold elapsed → secondary spawned) | When `execute_hedged()` returns from the happy path (§6.4 — primary wins before the threshold), there is no second pipeline to propagate to. | + +When any condition fails, `shared_hub_region_latch` is `None` and the +existing per-state behavior from [#4389][pr-4389] is preserved +bit-for-bit. + +#### 9.6.4 Interaction with §8.4 (Local-only retries inside a hedge) + +The §8.4 local-only-retry contract is unaffected by the shared latch: +the latch governs only which request header is emitted, not the +endpoint resolution. `ExcludeRegions` continues to pin each hedge to +its own region across retries; the shared latch merely ensures every +hedge's retries — within their pinned region — also carry the +hub-region hint once any hedge has observed 1002. No new retry +trigger paths or region-fallback edges are introduced. + +#### 9.6.5 Concurrency notes + +- `AtomicBool` with `Release` / `Acquire` ordering is sufficient — + the bool is the only thing being shared and there is no dependent + state. `Relaxed` would also be functionally correct (single-flag + race with a monotonic 0 → 1 transition) but `Release` / `Acquire` + is preferred for reader/code-author clarity and costs nothing on + every architecture the Rust SDK targets. +- The latch is monotonic 0 → 1 and never reset within an operation — + matches the "sticky" semantics of the per-state latch in + `components.rs`. +- The `Arc` is scoped to one outer `execute_hedged()` call, so + it is dropped when `execute_hedged()` returns (no global state, no + leak across operations). +- A losing hedge whose transport already responded after + cancellation (cf. §14.2) may still observe and CAS-set the shared + latch — this is benign: `execute_hedged()` has already returned a + winner, and the next observer of the dropped `Arc` is no one. + +--- + +## 10. Diagnostics & Observability + +### 10.1 HedgeDiagnostics + +> **Divergence from .NET diagnostic shape:** .NET attaches diagnostics in two +> different shapes depending on which loop produces the winner. +> - **Fast path** (winner emerges during the spawn-and-race loop): only +> `HedgeConfig` is attached when the **primary** wins (`requestNumber == 0`); +> when a hedge wins, `HedgeContext = hedgeRegions.Take(requestNumber + 1)` +> and `ResponseRegion` are also attached +> ([source](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Routing/AvailabilityStrategy/CrossRegionHedgingAvailabilityStrategy.cs#L198-L224)). +> - **Drain path** (winner emerges from the post-loop drain after all hedges +> were launched): all three fields are always attached, with +> `HedgeContext = hedgeRegions` (the full list). +> +> The Rust impl simplifies this by **always** attaching the full +> `HedgeDiagnostics` (config, regions launched, winner region, was_hedge flag) +> whenever a hedging strategy was active for the operation — even when the +> primary wins immediately. This is strictly more informative and avoids the +> bookkeeping required to mirror .NET's two-shape behavior. + +**Attachment contract.** `DiagnosticsContext::hedge_diagnostics` is +`Some(_)` if and only if a hedging strategy was **resolved and active** +for the operation — i.e. `should_hedge()` returned `true` and the +`execute_hedged()` was entered. It is `None` in all of the following cases: + +- No `AvailabilityStrategy` was resolved (no client/operation/driver-default + strategy, or the user set `AvailabilityStrategy::Disabled`). +- A strategy was resolved but `should_hedge()` returned `false` (e.g. + fewer than 2 applicable preferred regions, or the operation's + `ResourceType` is outside the phase-allowed set — see §5.1). +- The strategy resolved but `execute_hedged()` short-circuited before + spawning the primary (e.g. cancellation observed at entry). + +**Field semantics when the primary wins before the first hedge fires:** + +| Field | Value | +|---|---| +| `strategy_config` | The active strategy config (always populated) | +| `regions_contacted` | `vec![regions[0]]` (just the primary) | +| `response_region` | `regions[0]` | +| `total_requests_launched` | `1` | +| `was_hedge` | `false` | + +This lets callers distinguish *"hedging was active and the primary won +amongst the launched requests"* from *"hedging was active but no hedge +ever fired because the primary returned within the threshold window"*. + +```rust +/// Diagnostic information about a hedging execution, attached to the winning +/// response. +#[derive(Clone, Debug)] +pub struct HedgeDiagnostics { + /// The hedging strategy configuration that was active. + pub strategy_config: HedgingStrategyConfig, + /// Regions that had requests launched (up to and including the winner). + /// + /// With the single-alternate model (§6) this is either + /// `vec![primary]` (primary won before the threshold timer fired) + /// or `vec![primary, alternate]` (the alternate hedge was spawned). + pub regions_contacted: Vec, + /// The target region of the winning response. + pub response_region: Region, + /// How many hedge requests were launched (including primary). + /// Either `1` (no hedge fired) or `2` (alternate spawned). + pub total_requests_launched: usize, + /// Whether the primary or the alternate hedge won. + pub was_hedge: bool, +} + +#[derive(Clone, Debug)] +pub struct HedgingStrategyConfig { + /// The configured threshold before the alternate-region hedge fires. + pub threshold: HedgeThreshold, +} +``` + +### 10.2 DiagnosticsContext Integration + +The secondary's `transport_request` is built with +`ExecutionContext::Hedging` (TPS §3.4) so that the on-wire request +carries an explicit "this is a hedged secondary attempt" marker. +This marker: + +- Lets the transport layer attribute logs / spans / metric labels + to the correct attempt class. +- Is the input that distinguishes a secondary attempt from a normal + retry inside the per-attempt `transport_pipeline.rs` (TPS §5). +- Is set by `execute_hedged()` exactly once per operation (§6.4), + on the secondary only. + +```rust +// In DiagnosticsContext (existing) +pub struct DiagnosticsContext { + // ... existing fields ... + + /// Hedging diagnostics, present only if hedging was active. + pub hedge_diagnostics: Option, +} +``` + +### 10.3 Tracing + +```rust +// At hedging entry +tracing::debug!( + threshold = ?strategy.threshold().get(), + regions = ?regions, + "hedging enabled, launching primary request" +); + +// When the threshold elapses and the alternate is spawned +tracing::debug!( + target_region = %regions[1], + elapsed = ?start.elapsed(), + "launching alternate-region hedge" +); + +// When a winner is found +tracing::debug!( + region = %winner_region, + elapsed = ?start.elapsed(), + was_hedge, + "hedging winner selected, cancelling loser" +); +``` + +### 10.4 Reserved Telemetry / Metrics Surface + +Neither .NET v3 nor the current Rust driver emits quantitative metrics for +hedging beyond the structured per-response `HedgeDiagnostics`. Phase 1 will +ship without metrics, but the spec **reserves** the following surface so +that later phases (or a separate observability PR) can add them without +breaking changes. + +**Reserved `tracing` event names** (under target `cosmos.hedge`): + +| Event | Level | Fields | Emitted when | +|---|---|---|---| +| `cosmos.hedge.enabled_for_operation` | DEBUG | `threshold_ms`, `region_count` | `evaluate_transport_result` decides to hedge a specific operation | +| `cosmos.hedge.alternate_spawned` | DEBUG | `target_region`, `elapsed_ms` | The threshold elapsed and the alternate hedge was spawned | +| `cosmos.hedge.canceled` | DEBUG | `which` (`primary` / `alternate`), `target_region`, `reason` (`winner_found` / `deadline` / `app_canceled`) | A losing pipeline is canceled | +| `cosmos.hedge.won` | INFO | `winner_region`, `elapsed_ms`, `was_hedge` | A response is selected as final | +| `cosmos.hedge.both_transient` | WARN | `last_status_code` | Both primary and alternate returned transient responses | +| `cosmos.hedge.recorded_alternate_win` | DEBUG | `primary_region`, `partition` | `execute_hedged()` recorded an alternate-region win for PPCB feedback (§9.5) | + +**Reserved metric names** (intentionally namespaced; not emitted in +Phase 1, awaiting an `azure_core` metrics surface): + +| Metric | Type | Labels | Description | +|---|---|---|---| +| `cosmos.hedge.operations_total` | counter | `result` (`primary_won` / `alternate_won` / `both_transient` / `disabled`) | Hedging-eligible operations grouped by outcome | +| `cosmos.hedge.alternate_spawned_total` | counter | | Total alternate hedges spawned (i.e., operations where the threshold elapsed) | +| `cosmos.hedge.first_response_latency_ms` | histogram | `was_hedge` (bool) | Latency from `execute_hedged()` entry to the winning response | +| `cosmos.hedge.canceled_total` | counter | `reason` (`winner_found` / `deadline` / `app_canceled`) | Pipelines canceled before completion | +| `cosmos.hedge.ru_charge_winner` | histogram | `was_hedge` | RU of the winning response; this is the caller-visible RU charge | +| `cosmos.hedge.ru_charge_total` | histogram | `winner_region` | Total RU consumed across primary + alternate, including the loser; operator-facing only | +| `cosmos.hedge.consecutive_alternate_wins` | gauge | `partition`, `primary_region` | Current PPCB-feedback counter value for a (partition, primary-region) pair (§9.5) | + +Notes: + +- RU consumed by losing hedges is **not** reported to the caller. The + external per-operation RU contract remains the winning response's RU + charge, while any aggregate hedge cost is surfaced via the separate + operator metric `cosmos.hedge.ru_charge_total`. See §17 Q3 for the + resolved external-contract decision. +- Histogram bucket layout intentionally unspecified — defer to whichever + metrics provider `azure_core` settles on. +- Event/metric names follow OpenTelemetry conventions: dot-separated, + lower-snake-case, namespaced under `cosmos.hedge`. + +--- + +## 11. Options API Design + +### 11.1 Client-Level Configuration + +```rust +// DriverOptions (client-level) +let threshold = HedgeThreshold::new(Duration::from_millis(500)) + .expect("500ms is non-zero"); +let driver = CosmosDriverRuntimeBuilder::new() + .build(endpoint, credential, DriverOptionsBuilder::new() + .with_availability_strategy(AvailabilityStrategy::Hedging( + HedgingStrategy::new(threshold), + )) + .build() + ).await?; +``` + +### 11.2 Per-Operation Override + +```rust +// Disable hedging for a specific operation +let options = OperationOptionsBuilder::new() + .with_availability_strategy(AvailabilityStrategy::Disabled) + .build(); + +// Or use a tighter threshold for a single read +let tight = HedgeThreshold::new(Duration::from_millis(200)) + .expect("200ms is non-zero"); +let options = OperationOptionsBuilder::new() + .with_availability_strategy(AvailabilityStrategy::Hedging( + HedgingStrategy::new(tight), + )) + .build(); +``` + +### 11.3 Layered Resolution + +The existing `OperationOptionsView` layered resolution applies: + +``` +operation > account > runtime > environment +``` + +If the operation sets `AvailabilityStrategy::Disabled`, it overrides a client-level +hedging strategy. + +#### 11.3.1 Availability-strategy resolution priority + +The driver picks the effective strategy in the following priority order +(highest first): + +| Priority | Source | Notes | +|:---:|---|---| +| 1 | Operation `availability_strategy` (incl. `Disabled`) | Per-request override | +| 2 | Client / runtime `availability_strategy` | Applies to all requests | +| 3 | Environment variables (§4.4) | Deploy-time intent; `AZURE_COSMOS_HEDGING_DISABLED` short-circuits to `Disabled`; `AZURE_COSMOS_HEDGING_THRESHOLD_MS` overrides the default threshold but only if no code-level strategy is set | +| 4 | **Driver default** (§5.2) | Default-on for accounts with ≥ 2 applicable preferred regions; threshold = `min(1000ms, request_timeout / 2)`; independent of PPAF/PPCB | +| 5 | None | Hedging off (single-region account or insufficient region config) | + +The resolved strategy is consumed by `evaluate_transport_result` +(TPS §3.4), which calls `should_hedge()` (§5.1) and (when eligible) +returns `OperationAction::Hedge { secondary_routing }` carrying the +chosen threshold and the secondary `RoutingDecision`. The pipeline +does the resolution lookup once per per-attempt iteration; there is +no separate orchestrator-side resolution step. + +A user-configured `AvailabilityStrategy::Disabled` at any layer suppresses every +lower layer (including the driver default and env-var-derived strategy) — +explicit opt-out always wins. + +The env var `AZURE_COSMOS_HEDGING_DISABLED=true` is equivalent to setting +`AvailabilityStrategy::Disabled` at the runtime layer (priority 3). It +overrides priorities 4 and 5 but is itself overridden by code-level +`Hedging(..)` at priorities 1 or 2. Operators who want to globally +disable hedging at deploy time without touching code should use this +env var. + +--- + +## 12. Cancellation & Resource Cleanup + +### 12.1 Structural Cancellation (No Token) + +Cancellation in the hedge path is structural: `execute_hedged()` +owns both attempts via `tokio::select!`, and the loser's future is +dropped when the winner returns. Dropping the future runs the +standard `Drop` chain through the transport pipeline and cancels the +in-flight HTTP/AMQP request at the next `await` point (TPS §5.1). + +```rust +// No CancellationToken in the hedge path. +tokio::select! { + biased; + r = &mut primary_fut, if !primary_done => { /* primary won */ } + r = &mut secondary_fut, if !secondary_done => { /* secondary won */ } + _ = app_cancel_signal(deadline) => { /* deadline tripped */ } +} +// On return, the un-polled future is dropped → in-flight request cancelled. +``` + +This is the same model TPS §4.2 uses for non-hedged retries and is +the reason the spec does not require `tokio_util` as a dependency +(see §17 Q10). + +### 12.2 Pipeline Cooperation + +The transport pipeline (TPS §5) already honors: + +1. **Per-attempt deadline** — every request carries an `Instant` + deadline, checked before send and at every layered `await`. +2. **Future-drop cancellation** — every `await` inside the transport + stack is cancellation-safe; dropping the surrounding future at + any point releases all owned resources (connection, body buffer, + diagnostics row). + +`execute_hedged()` relies on both: the per-attempt deadline lets it +observe application-level cancellation (the deadline is set from the +caller's overall budget), and the drop-the-future model lets it +cancel the loser without any additional plumbing. + +### 12.3 No Task Spawning + +`execute_hedged()` does **not** use `tokio::spawn` or `FuturesUnordered` +(see §6.4.2). The primary and alternate futures are pinned to the +`execute_hedged()`'s stack frame and polled directly by `tokio::select!`. +This means: + +- No `'static + Send` constraint on captured values. +- No `JoinHandle` to drain, no orphaned-task risk. +- Dropping `execute_hedged()`'s future (e.g., on `tokio::time::timeout` + expiry one layer up) drops both pipelines via standard structured + concurrency \u2014 the in-flight transport `Drop`s emit the same + cancellation signal a `CancellationToken` would. + +Panics inside a pipeline propagate naturally back through the awaited +future rather than being caught by a `JoinHandle`. This matches the +behavior of every other awaited operation in the driver. + +--- + +## 13. Write Hedging (Removed) + +> **Removed from scope.** Earlier drafts of this spec described a +> multi-master write-hedging mode gated by an +> `enable_multi_write_region_hedge` flag, mirroring Java v4's behavior. +> That mode is now an explicit Non-Goal (see §1). Rationale: +> +> - Write hedging on multi-master amplifies 409 Conflict / 412 +> Precondition Failed surface area because the same write can be in +> flight to multiple regions simultaneously via async replication. +> - Adoption in Java v4 has been near-zero; the operational surprise +> of a "successful" hedge that produces a 409 on the loser-region +> replay is consistently called out as a footgun. +> - Single-master writes have never been hedged in any SDK; PPAF +> handles write failover for single-master. +> +> If service-side idempotency keys are added later, a separate +> proposal can revisit write hedging on that primitive. Until then, +> writes go through the existing operation pipeline unchanged — no +> hedging fan-out, no MM-write configuration knob. + +This section is retained as a numbered placeholder so cross-references +to §13 in earlier reviews still resolve; the prior subsection +numbering (§13.1 / §13.2 / §13.3) is intentionally not preserved. + +--- + +## 14. Error Handling & Edge Cases + +### 14.1 Both Pipelines Return Transient Errors + +If both the primary and the alternate return transient errors (e.g., 503 +on both regions), `execute_hedged()` returns the **last** response +received. The retry logic within each pipeline invocation will have +already attempted retries before surfacing the error. + +### 14.2 Primary Succeeds After the Alternate Hedge Launched + +If the primary returns a final result 1ms after the alternate hedge is +launched, the alternate is cancelled. The alternate's transport request +may or may not have been sent (depends on timing). Cancellation is +best-effort — an in-flight HTTP request cannot be aborted mid-stream, +but the response will be discarded. + +> **Divergence from .NET on application-cancel diagnostics.** When the +> *application* cancellation token fires mid-race, .NET awaits the +> most-recently-completed task with no timeout (relying on it being +> already completed) before re-raising. The Rust hedge path instead +> harvests the alternate pipeline within a bounded `HARVEST_WINDOW = 50ms` +> window (see §6.4 / §6.5 invariant #7). This bounds user-visible +> cancel latency under a stuck transport future at the cost of +> occasionally returning slightly less-rich diagnostics than .NET would. +> Documented as best-effort: "diagnostics-on-cancel are attached when +> the alternate has produced a result within 50ms of cancellation." + +### 14.3 Deadline Interplay + +If the end-to-end deadline is shorter than the hedging threshold, hedging has no +effect — the primary request will hit the deadline before any hedge fires. + +``` +deadline = 200ms, threshold = 500ms +→ Primary fires at t=0 +→ Hedge would fire at t=500ms, but deadline hit at t=200ms +→ Only primary result returned (or deadline error) +``` + +**Guideline:** `threshold` should be significantly less than `end_to_end_timeout / 2` +to leave time for the hedge to complete. + +### 14.4 Region List Changes During Hedging + +The applicable-region snapshot is captured at the start of +`execute_hedged()`. If account metadata refreshes during +execution (e.g., a `RefreshAccountProperties` effect), the +`ExcludeRegions` set on the already-launched alternate hedge is +unchanged. + +Per the §8.4 contract, `ExcludeRegions` is a **hard constraint** inside +the alternate hedge: `resolve_endpoint()` does *not* fall back to an +excluded region even if the metadata refresh has marked the only allowed +region unavailable. If the alternate ends up with no eligible endpoint, +the retry layer returns the terminal "all eligible regions excluded" +condition (§8.4 item 2) as the result. + +**How `execute_hedged()` handles that terminal condition.** The "all +eligible regions excluded" result is classified as transient (§7.2) and +does *not* short-circuit `execute_hedged()`. The primary continues racing, +and if the primary produces a final result it wins. If the primary is +also transient, §14.1 applies. + +--- + +## 15. Test Plan + +### 15.1 Unit Tests + +| Test | Validates | +|------|-----------| +| `should_hedge_read_multi_region` | Reads eligible on multi-region account with ≥ 2 applicable preferred regions | +| `should_hedge_read_single_region` | Reads NOT eligible on single-region account | +| `should_hedge_excluded_to_one_region` | Reads NOT eligible when `ExcludeRegions` leaves < 2 applicable read endpoints | +| `should_hedge_no_preferred_regions` | NOT eligible when application-preferred-region list is empty | +| `should_hedge_write_never` | Writes (Create / Replace / Upsert / Delete / Patch) NEVER hedged regardless of topology | +| `should_hedge_non_document` | Non-Document `ResourceType`s excluded in Phase 1 | +| `should_hedge_disabled_override` | Per-operation `AvailabilityStrategy::Disabled` overrides client-level hedging | +| `should_hedge_env_disabled` | `AZURE_COSMOS_HEDGING_DISABLED=true` suppresses driver default + env-var threshold | +| `is_final_result_success` | 200 → final | +| `is_final_result_conflict` | 409 → final | +| `is_final_result_503` | 503 → transient | +| `is_final_result_404_0` | 404/0 → final | +| `is_final_result_404_1002` | 404/1002 → transient | +| `is_final_result_429` | 429 → transient | +| `hedge_threshold_rejects_zero` | `HedgeThreshold::new(Duration::ZERO)` returns `None` (matches the §4.1 newtype contract) | +| `hedge_threshold_accepts_positive` | `HedgeThreshold::new(Duration::from_millis(1))` is `Some(_)` | +| `alternate_region_pin_excludes_primary` | Alternate hedge's `ExcludeRegions` contains the primary region | +| `alternate_region_pin_unions_user_excludes` | When the user supplied `ExcludeRegions = {X}`, the alternate hedge's set is `{X} ∪ (all_regions \ regions[1])` | +| `exclude_regions_honored_by_every_retry_trigger` | For each retry trigger class — PPAF write retry, PPCB markdown failback, transport-layer 503, throttling 429, session-token 1002 — fault-inject the trigger inside the alternate hedge and assert the retry attempt does **not** route to a region listed in the hedge's `ExcludeRegions`. Encodes the §8.4 cross-cutting invariant. | +| `app_cancel_preserves_hedge_diagnostics` | Cancel the application token while both pipelines are racing; assert the returned error carries `HedgeDiagnostics` from the most-advanced pipeline (covers §6.5 invariant #7). | +| `record_hedge_win_increments_ppcb_counter` | An alternate-region win calls `record_consecutive_hedge_win` exactly once on the primary partition (§9.5). | +| `primary_win_resets_hedge_win_counter` | A direct primary-region win clears the consecutive-hedge-win counter on that partition. | +| `zero_overhead_happy_path_no_allocs` | When the primary returns before the threshold timer fires, `execute_hedged()` allocates no per-hedge state (no `CancellationToken`, no cloned `OperationOptions`, no `ExcludeRegions` recompute). Backed by `dhat-rs` allocation count. | +| `shared_hub_region_latch_initialized_when_eligible` | `execute_hedged()` invoked on a data-plane / single-master operation; the threshold elapses and a secondary is spawned. Assert both the primary's and the secondary's `OperationRetryState.shared_hub_region_latch` are `Some(_)` and point to the same `Arc` instance (encodes §9.6.2 / §9.6.3). | +| `shared_hub_region_latch_none_on_zero_overhead_happy_path` | Primary returns before the threshold; assert no `Arc` was ever constructed and the per-state latch remains the only mechanism — preserves §6.5 invariant #3 and the [#4389][pr-4389] baseline allocator behavior (§9.6.2). | +| `shared_hub_region_latch_none_on_multi_master_or_metadata` | Multi-master *or* metadata pipeline; assert `shared_hub_region_latch` is `None` even when the alternate spawns, matching `HUB_REGION_PROCESSING_HEADER_SPEC.md` §5 account-level / §1.5 data-plane gates (§9.6.3). | +| `shared_hub_region_latch_propagates_first_1002_across_hedges` | Drive 1002 through `build_session_retry_state` on the primary; assert (a) the primary's per-state `hub_region_processing_only` is `true`, (b) the shared `Arc` is `true`, (c) on the next transport attempt the alternate — whose per-state latch is still `false` — has `apply_hub_region_header` emit the header. Rust counterpart of .NET PR #5815's `CrossRegionAvailabilityContext_PropagatesHubHeaderFlagToHedgedRequests` test. | +| `shared_hub_region_latch_no_1002_emits_no_header` | Neither side observes 1002; assert no transport attempt calls `apply_hub_region_header` with the header set, regardless of `shared_hub_region_latch` presence. | + +### 15.2 Integration Tests (Fault Injection) + +| Test | Setup | Validates | +|------|-------|-----------| +| `hedging_read_primary_slow` | 2s delay on Region A reads, threshold 200ms | Alternate Region B wins; diagnostics show `was_hedge=true`, `total_requests_launched=2` | +| `hedging_read_primary_fast` | No faults | Primary wins before threshold; `hedge_diagnostics=Some(_)` with `was_hedge=false` and `total_requests_launched=1` | +| `hedging_read_primary_503` | 503 on Region A reads | Alternate Region B wins with success | +| `hedging_read_both_regions_slow` | 2s delay on both regions | Whichever responds first wins (graceful degradation) | +| `hedging_write_not_hedged` | 2s delay on writes on a multi-master account | NO alternate hedge fires; write returns after the delay | +| `hedging_disabled_per_operation` | Client hedging on; operation `Disabled` | No alternate hedge; normal path | +| `hedging_respects_deadline` | threshold > deadline | No alternate fires; deadline error | +| `hedging_with_ppcb_existing_failures` | Region A primary has prior PPCB failures | Hedging still fires; PPCB and hedging compose without interference | +| `hedging_cancels_loser` | Delay on Region A | Region B wins; verify Region A transport task observed cancellation (hit_count ≤ 1) | +| `hedging_failback_to_primary` | Region A initially slow, then fast | First few reads hedged; subsequent reads complete on primary before the threshold | +| `hedging_exclude_regions_under_503_retry` | Alternate hedge gets a 503 (triggers transport retry) while a third region is healthy and excluded by that hedge's `ExcludeRegions` | Alternate hedge's retry stays pinned to its region (does NOT fall back to the third region) — fault-injection counterpart to the §8.4 invariant unit test. | +| `hedging_alternate_wins_trip_ppcb` | Force N consecutive alternate-region wins on the same partition | PPCB transitions the primary partition to `Unhealthy` after the configured threshold (§9.5). | +| `hedging_hub_region_header_propagates_across_hedges` | 2-region single-master data-plane account; fault-inject `404/1002` on the primary's first attempt against Region A, healthy 200 on the alternate against Region B after the threshold | Primary's retry against Region A emits `x-ms-cosmos-hub-region-processing-only: True` (per-state latch) **and** the alternate against Region B emits the same header on every attempt — without itself ever observing a 1002 (per the shared `Arc` from §9.6). Encodes the cross-hedge propagation invariant under fault injection; counterpart of .NET PR #5815's emulator-level coverage. | + +### 15.3 Multi-Region Live Tests + +Gated by `test_category = "multi_region"`: + +| Test | Account Type | Validates | +|------|-------------|-----------| +| `hedging_read_cross_region` | 2-region SM | Read hedged to satellite when primary slow | +| `hedging_ppcb_feedback_cross_region` | 2-region SM with primary partition under load | Repeated alternate wins trip PPCB; subsequent reads route directly to the alternate without hedging until PPCB probes the primary back to `Healthy` | + +--- + +## 16. Implementation Phases + +The phased rollout introduced in §1 ("Operation-type scope (phased)") +maps onto the implementation milestones below. Each phase is auditable +against the §1 Goals. + +| §1 Goal | Phase that closes it | +|---|---| +| **G1. Reduce tail latency** (p99/p99.9 bounded by `threshold + RTT`) | Phase 1 (point reads). Phase 2 widens to feed-style operations + metadata. | +| **G2. Transparent to application** (single `CosmosResponse`; opt-in diagnostics) | Phase 1 (`HedgeDiagnostics`, `DiagnosticsContext` integration). | +| **G3. Configurable** (single `threshold` knob at client and per-operation levels; explicit opt-out) | Phase 1. | +| **G4. Complementary to failover** (composes with PPAF/PPCB; feeds PPCB) | Phase 1 (lock-free `LocationStateStore` interaction §9.1 + PPCB feedback callsite §9.5). | +| **G5. Resource-safe** (≤ 2 concurrent pipelines, loser cancelled promptly) | Phase 1 (single-`select!` `execute_hedged()` §6.4 + structural drop-the-future cancellation §12). | +| **G6. Zero-overhead happy path** (no per-hedge state when primary wins early) | Phase 1 (gated by `zero_overhead_happy_path_no_allocs` test §15.1). | + +§1 Non-Goals (single-region hedging, write hedging, multi-region +fan-out > 1 alternate, automatic threshold tuning, PPAF coupling) +remain out of scope for every phase below. + +### Phase 1: Point-read Hedging + PPCB Feedback (MVP) + +**Operation rows from §1 covered (Phase 1 column):** +- Document point reads (`GetItem`). + +Writes are excluded by spec rule (§1 Non-Goals, §5.1 row 4). Feed-style +operations (Query / ReadMany / ChangeFeed) and metadata operations are +deferred to Phase 2 because they require additional coordination — see +that section. + +**Scope:** + +- `HedgeThreshold`, `HedgingStrategy`, `AvailabilityStrategy` types (§4). +- `should_hedge()` covering point reads (§5.1; phase-allowed + `ResourceType` = `{Document}` with `OperationType = Read`). +- `is_final_result()` (§7.1). +- `execute_hedged()` (§6.4) extending the `OperationAction::Hedge` + arm of `operation_pipeline.rs` STAGE 7, with: + - zero-overhead happy path (§6.5 invariant #3), + - single alternate-region hedge (§6.5 invariant #1), + - app-cancel diagnostics preservation (§6.5 invariant #7). +- `HedgeDiagnostics` + `DiagnosticsContext` integration (§10). +- Integration into `cosmos_driver.rs` (§8.1) — final integration point + to be coordinated with the in-pipeline shape in + `TRANSPORT_PIPELINE_SPEC.md` §4.2 (§3.4). +- Cooperative cancellation via `CancellationToken` (§12); the loser + pipeline observes cancellation at every pipeline `select!` point + → satisfies **G5**. +- **Default-on activation** (§5.2) independent of PPAF/PPCB — driver + default threshold `min(1000ms, request_timeout / 2)`, applied when + the account has ≥ 2 applicable preferred regions and no user + strategy is set. +- **Per-operation override surface** (satisfies **G3**): + `OperationOptions::availability_strategy` accepting + `Some(AvailabilityStrategy::Hedging(..))`, + `Some(AvailabilityStrategy::Disabled)`, or `None`; layered + resolution per §11.3 / §11.3.1. +- **Environment variable opt-out** (§4.4 / §11.3.1): + `AZURE_COSMOS_HEDGING_DISABLED` and + `AZURE_COSMOS_HEDGING_THRESHOLD_MS`. +- **PPCB feedback callsite** (§9.5): `record_consecutive_hedge_win` + invoked on every alternate-region win; the PPCB-side state + transition is co-designed with the PPCB module owner before + Phase 1 ships (tracked as an in-flight dependency, not blocking the + `execute_hedged()` design). +- **Hub-region-processing-only header cross-hedge propagation** + (§9.6): extend `OperationRetryState` with + `shared_hub_region_latch: Option>`; construct the + `Arc` in `execute_hedged()` after the threshold + elapses (preserves §6.5 invariant #3); update + `build_session_retry_state` to publish to the shared latch on first + 1002 (`Release`) and `apply_hub_region_header` to read-OR + per-state and shared latches (`Acquire`). Mirrors + [.NET v3 PR #5815][dotnet-pr-5815]'s `CrossRegionAvailabilityContext` + fix. +- Unit + fault-injection tests per §15 including the `§8.4` + cross-cutting `ExcludeRegions` invariant, the `§6.5` invariant #3 + zero-overhead allocation test, the `§9.5` PPCB-feedback test, and + the five `§9.6` shared-latch unit tests plus the + `hedging_hub_region_header_propagates_across_hedges` fault-injection + test. + +**§1 Goals closed at end of Phase 1:** G2, G3, G4, G5, G6 in full; +G1 for point reads only. + +**Out of scope this phase (deferred to Phase 2 / Future per §1 table):** +Feed-style operations (Query / ReadMany / ChangeFeed), metadata +operations, stored procedure execution, adaptive threshold tuning. + +**Deliverables:** + +- New files: `hedging.rs`, `hedging_diagnostics.rs` (see §8.5). +- Modified: `cosmos_driver.rs`, `operation_options.rs`, `mod.rs`, + `diagnostics/mod.rs`, plus a `LocationStateStore`-side + `record_consecutive_hedge_win` API contributed under the PPCB module. +- Extended: `components.rs::OperationRetryState` (new + `shared_hub_region_latch: Option>` field), + `retry_evaluation.rs::build_session_retry_state` (shared-latch + publish), `operation_pipeline.rs::apply_hub_region_header` + (per-state OR shared-latch emit) — per §9.6. + +### Phase 2: Feed-style operations + Metadata + +**Operation rows from §1 covered (Phase 2 column):** + +- `QueryItems` — hedged **per page**. +- `ReadMany` — hedged **per page**. +- Change feed (`ReadFeed`) — hedged **per page**. +- Metadata operations: Database / Container / Offer / Throughput + **reads only**. + +**Scope (deferred — design pass required before scheduling):** + +- Extend `should_hedge()`'s phase-allowed set to add the feed-style + operations and the metadata read `ResourceType`s. +- **Per-page semantics for feeds.** Each page request is an + independent hedge fan-out; the winning page's continuation token is + forwarded to the next page. Final integration must align with the + `FeedRange` abstraction being designed by the feed-operations spec + (cross-coordination required — see also Ashley's spec); the + per-page hedge boundary is conditional on that spec landing. +- **Metadata cache invalidation.** Hedged metadata reads must not + produce stale-cache races when one region returns an older view + than another; decide whether to prefer the latest `_etag` / + resource id or the fastest response. +- **Diagnostics caveat for multi-stage operations.** Query / ReadMany / + ChangeFeed contact regions *before* the hedge dispatch starts + (query plan fetches, partition-key-range cache loads, + identity-batching pre-flights, metadata-cache priming). + `HedgeDiagnostics::regions_contacted` covers only the regions the + hedge path itself fanned out to; pre-hedge contacts show up in + the surrounding `DiagnosticsContext` (existing per-attempt region + trail). Operators must consult both surfaces to distinguish + hedge-driven contacts from setup-driven contacts — keeps **G2** + intact under multi-stage operations. + +**§1 Goals advanced:** G1 widens to feed-style operations and +metadata reads. G2 extended with pre-hedge / hedge contact +disambiguation. + +### Future: Stored Procedure Execution + Adaptive Thresholds + +**Operation rows from §1 covered (Future column):** + +- Stored procedure execution (`ExecuteJavaScript`) — **🟡 candidate** + pending a separate design proposal (server-side state mutation, + idempotency, body cloning of script payloads interact with hedging + in non-obvious ways). + +**Out-of-spec extensions** (explicit §1 Non-Goals — recorded here so +the future-work landing zone is well known): + +- Latency histogram tracking per-region. +- Auto-tuning threshold based on observed p50 / p90 latency. +- Exponential backoff on the threshold after repeated alternate-wins. +- Write hedging on idempotency-key-aware service paths (if those + primitives ever exist on the wire). + +These items intentionally do **not** advance any §1 Goal; adding any +of them constitutes a new goal and requires a spec amendment. + +--- + +## 17. Open Questions + +1. **Should hedging be enabled by default?** — **Resolved.** Yes, on + by default for accounts with ≥ 2 applicable preferred regions, + independent of PPAF and PPCB (§5.2). Rationale: the Rust driver is + greenfield and has no backward-compatibility constraint that + forced .NET v3 and Java v4 to gate hedging on PPAF. Opt-out is via + `AvailabilityStrategy::Disabled` (per-op or per-client) or + `AZURE_COSMOS_HEDGING_DISABLED=true` at deploy time. + +2. **Interaction with `EndToEndOperationLatencyPolicy`** — + **Resolved.** Primary and alternate share the deadline (§9.4). + The alternate inherits the *remaining* budget after the threshold, + not its own full timeout. + +3. **RU accounting** — **Resolved.** Caller-visible per-operation RU + charge is the **winning** response's RU only. Aggregate hedge cost + (winner + alternate) is surfaced via the operator-facing + `cosmos.hedge.ru_charge_total` metric (§10.4). Intentional + divergence from .NET v3, which folds sub-request RU into + `RequestCharge`. + +4. **Composition with the `x-ms-cosmos-hub-region-processing-only` + header from [PR #4389][pr-4389]** — **Resolved.** The per-state + `hub_region_processing_only: bool` latch on `OperationRetryState` + is per-hedge by construction (§8.2), which would defeat the + header's *one round-trip per operation* guarantee under hedging. + `execute_hedged()` constructs a single `Arc` shared + between primary and alternate when the threshold elapses; both + sides read-OR the per-state and shared latches at + `apply_hub_region_header`. Full design in §9.6. Mirrors + [.NET v3 PR #5815][dotnet-pr-5815]'s `CrossRegionAvailabilityContext` + fix. + +5. **Race with background PPCB failback** — **Resolved.** Primary and + alternate are independent pipeline invocations with independent + retry states; the shared `LocationStateStore` uses CAS-based + updates that are safe under concurrency (§9.1). + +6. **Max concurrent hedges cap** — **Resolved.** Capped at one + alternate region (≤ 2 concurrent pipelines) by spec rule (§6.5 + invariant #1). This is a stronger guarantee than .NET v3 and + Java v4, which fan out across all preferred regions; the cap is + chosen to bound RU multiplier at 2× and avoid the diminishing + returns of progressive fan-out. + +7. **PPCB threshold for consecutive-secondary-win trip** — **Open.** + `execute_hedged()` emits the `record_consecutive_hedge_win` + signal on every secondary win (§9.5), but the threshold ("after + N consecutive wins, mark partition `Unhealthy`") and the state + transition itself live in the PPCB module. Pending PPCB-owner + sign-off. + +8. **Reconciliation with `TRANSPORT_PIPELINE_SPEC.md` §4.2** — + **Resolved.** This spec adopts the TPS in-pipeline shape: + `OperationAction::Hedge { secondary_routing }` returned by + `evaluate_transport_result` and dispatched by STAGE 7 calling + `execute_hedged()`. See §3.4 for the adoption statement and the + five preserved invariants. The `OperationAction::Hedge` arm of + `operation_pipeline.rs` is the only entry into `execute_hedged()`. + +9. **FeedRange integration for Phase 2** — **Open.** Per-page hedging + for Query / ReadMany / ChangeFeed must align with the `FeedRange` + abstraction being designed by the feed-operation spec. Not a + Phase 1 blocker; tracked in §16 Phase 2 scope. + +10. **`tokio_util` dependency** — **Resolved.** Not needed under the + TPS shape. Cancellation in the hedge path is structural: + `tokio::select!` owns both attempts and dropping the loser's + future cancels the in-flight transport via the standard `Drop` + chain (§8.3, §12.1). No `CancellationToken` and therefore no + `tokio_util` dependency is required. + +11. **Threshold-policy disagreement with TPS §4.2** — **Open.** This + spec specifies a static driver default of + `min(1000ms, request_timeout / 2)` (§5.2), while TPS §4.2 + specifies a dynamic P99-based threshold clamped to 50–4000 ms. + Both shapes are compatible with the + `OperationAction::Hedge { secondary_routing }` enum (the + threshold value is computed by the evaluator and passed into + `execute_hedged()`), so the disagreement is a pure policy + question, not a structural one. Pending cross-team alignment on + which policy ships in v1. + +--- + +## Appendix A: .NET SDK Source References + +- [`AvailabilityStrategy.cs`](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Routing/AvailabilityStrategy/AvailabilityStrategy.cs) — Public factory methods +- [`CrossRegionHedgingAvailabilityStrategy.cs`](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Routing/AvailabilityStrategy/CrossRegionHedgingAvailabilityStrategy.cs) — Core implementation (410 lines) +- [`DisabledAvailabilityStrategy.cs`](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Routing/AvailabilityStrategy/DisabledAvailabilityStrategy.cs) — Sentinel for per-request disable +- [`AvailabilityStrategyInternal.cs`](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Routing/AvailabilityStrategy/AvailabilityStrategyInternal.cs) — Internal abstract base + +## Appendix B: Glossary + +| Term | Definition | +|------|-----------| +| Hedging | Sending the same request to a primary region and (after a threshold) one alternate region; first non-transient response wins | +| Threshold | Time before the alternate-region hedge fires | +| Alternate region | The single fallback region targeted by the hedge — `applicable_read_endpoints[1]` after `ExcludeRegions` filtering | +| Final result | A response that is definitively non-transient (success or permanent error) — see §7.1 | +| Transient result | A response that might succeed in another region (5xx, timeout, 404/1002, 429, 403, 410) — see §7.2 | +| PPAF | Per-Partition Automatic Failover (write failover on single-master). Independent of hedging in this driver. | +| PPCB | Per-Partition Circuit Breaker (read/write failover on failure threshold). Receives signal from hedging on repeated alternate-region wins (§9.5). | +| MM | Multi-master (multi-write-region) account | +| SM | Single-master account |