From c2c2308d144d01420a1f21bdd913017a74b2a508 Mon Sep 17 00:00:00 2001 From: Ananth Mudumba Date: Fri, 19 Jun 2026 20:18:00 -0700 Subject: [PATCH 1/5] docs(cosmos): native driver distribution design + ADRs Add a design doc and a numbered ADR set describing how the native Cosmos driver (azurecosmosdriver cdylib/staticlib + C header) is distributed to language SDKs. Core model: separate provenance from distribution. One Rust build produces all signed platform binaries as an internal-only hand-off artifact (ADR 0001/0008); distribution is per-language native packages on each language's existing internal + external feed (ADR 0009) -- NuGet NativeAssets + meta for .NET (0002), cgo prebuilt header+lib via the Go feed (0003), JAR for Java (future). ABI handshake (0004), bytes-in/bytes-out copy-out (0005), opt-in transport (0006), and platform matrix (0007) round out the set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../docs/distribution/adr/0000-index.md | 19 +++ .../adr/0001-build-once-internal-handoff.md | 23 +++ .../adr/0002-dotnet-nuget-nativeassets.md | 21 +++ .../distribution/adr/0003-go-cgo-prebuilt.md | 21 +++ .../adr/0004-abi-version-handshake.md | 20 +++ .../adr/0005-binding-owns-marshalling.md | 20 +++ .../distribution/adr/0006-native-is-opt-in.md | 20 +++ .../distribution/adr/0007-platform-matrix.md | 20 +++ .../adr/0008-build-and-signing-pipeline.md | 20 +++ .../0009-per-language-feed-distribution.md | 23 +++ .../docs/distribution/distribution-design.md | 134 ++++++++++++++++++ 11 files changed, 341 insertions(+) create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-dotnet-nuget-nativeassets.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-go-cgo-prebuilt.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-abi-version-handshake.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-binding-owns-marshalling.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-native-is-opt-in.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-platform-matrix.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-build-and-signing-pipeline.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-per-language-feed-distribution.md create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md new file mode 100644 index 0000000000..a316289c25 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md @@ -0,0 +1,19 @@ +# Architecture Decision Records — native driver distribution + +ADRs (Architecture Decision Records) capture **what we decided** and a brief **why**, in a minimal, easy-to-reference form. They are **numbered and immutable**: once accepted, an ADR is not edited; a later ADR may **supersede** it. Detailed discussion and alternatives live in the design doc ([`../distribution-design.md`](../distribution-design.md)), not here — an ADR is the "regression test" that keeps us doing the same thing everywhere. + +**Format (template):** Context (2–4 sentences) · Decision (1–3 bullets) · Consequences (2–4 bullets) · Alternatives considered (1 line each) · Status. + +| # | Title | Status | +|---|-------|--------| +| [0001](0001-build-once-internal-handoff.md) | One build → internal-only hand-off artifact; no neutral consumer bundle | Accepted | +| [0002](0002-dotnet-nuget-nativeassets.md) | .NET consumes via per-RID NuGet NativeAssets + meta-package | Accepted | +| [0003](0003-go-cgo-prebuilt.md) | Go consumes via cgo against a prebuilt header + lib from the Go feed | Accepted | +| [0004](0004-abi-version-handshake.md) | Native lib exports an ABI version; hosts check it before use | Accepted | +| [0005](0005-binding-owns-marshalling.md) | Each language binding owns marshalling and buffer copy-out | Accepted | +| [0006](0006-native-is-opt-in.md) | Native transport is opt-in until GA, then default-with-fallback | Accepted | +| [0007](0007-platform-matrix.md) | A defined platform matrix; unsupported platforms error clearly | Accepted | +| [0008](0008-build-and-signing-pipeline.md) | One build, sign binaries once, fan-out; jobs never rebuild | Accepted | +| [0009](0009-per-language-feed-distribution.md) | Distribute as per-language packages on each language's existing feeds | Accepted | + +> These are **proposed** for the design review. "Accepted" is provisional until the review signs off. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md new file mode 100644 index 0000000000..9b72c63af7 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md @@ -0,0 +1,23 @@ +# ADR 0001 — One build → internal-only hand-off artifact; no neutral consumer bundle + +**Status:** Accepted (proposed for review) + +## Context +The native driver must reach .NET, Go, and later Java, each with a different package format and its own feed. Two concerns were previously conflated: **provenance** (there must be exactly one build + binary-signing per release, or the languages drift onto different driver builds) and **distribution** (how a consumer pulls the bytes). A neutral consumer-facing bundle would force a Go user to download a package containing DLLs/JARs they cannot use, and would require standing up new consumer feed infrastructure. + +## Decision +- For each release, **one** Rust build produces all platform binaries (cdylib + staticlib), the cbindgen C header, an `ABI_VERSION`, and checksums, and **signs the binaries**. +- These are published as an **internal-only hand-off artifact** (e.g. an Azure Artifacts Universal Package or a pipeline artifact) that is the **single source of truth for provenance**. +- This artifact is **not consumer-facing and not language-shaped.** Consumers never download it; per-language publish jobs consume it (see ADR 0008, ADR 0009). +- The Rust crate declares `crate-type = ["cdylib", "staticlib"]` so both link forms come from one build. + +## Consequences +- Single provenance + SBOM anchor without forcing any cross-language or cross-platform download on consumers. +- No new **consumer** feed to operate; only an internal hand-off artifact. +- All per-language jobs must consume this artifact rather than rebuilding (enforced by ADR 0008). + +## Alternatives considered +- Neutral consumer-facing bundle/feed — rejected: forces irrelevant bytes on consumers; new infra. +- Per-language independent builds — rejected: drift risk, duplicated signing. + +> Supersedes the earlier "canonical artifact bundle as a distributed product" framing. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-dotnet-nuget-nativeassets.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-dotnet-nuget-nativeassets.md new file mode 100644 index 0000000000..fbbc00020c --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-dotnet-nuget-nativeassets.md @@ -0,0 +1,21 @@ +# ADR 0002 — .NET consumes via per-RID NuGet NativeAssets + meta-package + +**Status:** Accepted (proposed for review) + +## Context +.NET consumers expect NuGet. Native libraries are platform-specific, so a single package carrying every platform forces every consumer to download all of them. Mature native-backed SDKs (SkiaSharp, Microsoft.Data.SqlClient.SNI) solve this with a split package layout. + +## Decision +- Ship per-RID **`Microsoft.Azure.Cosmos.NativeAssets.`** packages, each carrying one platform's dynamic lib under `runtimes//native/`. +- Front them with a thin **meta-package** whose `runtime.json` resolves the consumer's RID to the right per-RID package. +- `Microsoft.Azure.Cosmos` takes an **opt-in** dependency on the meta-package (ADR 0006). +- Publish to **nuget.org** and the **internal azure-sdk NuGet feed** (per ADR 0009). + +## Consequences +- A consumer downloads only its platform's binary. +- More package IDs to version and keep in lockstep (the meta `runtime.json` pins per-RID versions). +- Reuses Azure's already-reserved `Microsoft.Azure.*` prefix — no new org-level construct. + +## Alternatives considered +- Single "fat" NativeAssets package — kept only as an **interim** (Phase 1–2) for speed, not GA. +- Embed natives directly in `Microsoft.Azure.Cosmos` — rejected: bloats the flagship for pure-managed users. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-go-cgo-prebuilt.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-go-cgo-prebuilt.md new file mode 100644 index 0000000000..d9b7cece4f --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-go-cgo-prebuilt.md @@ -0,0 +1,21 @@ +# ADR 0003 — Go consumes via cgo against a prebuilt header + lib from the Go feed + +**Status:** Accepted (proposed for review) + +## Context +Go has no NuGet. Go links C libraries through cgo, which needs a C header and a library available at `go build` time. The Go SDK already implements the completion-queue receive loop, `cgo.Handle` correlation, and buffer copy-out; the only distribution question is how header+lib+ABI version reach the Go build. + +## Decision +- Go consumes the prebuilt **`include/` header and `lib/` library via cgo** (`#cgo CFLAGS -I…` to parse the header into `C.*` symbols; `#cgo LDFLAGS -L… -lazurecosmosdriver` to link), **not NuGet**. +- Prefer the **static `.a`** for a self-contained Go binary; dynamic linking is supported as an option. +- The header + lib are delivered through the **azure-sdk-for-go feed** — an Azure Artifacts Universal Package fetched at build, or a vendored "binaries" Go module with per-OS build tags (delivery shape is open Q3). Either way it derives from the ADR 0001 hand-off artifact. + +## Consequences +- Go reuses the exact same signed binaries as .NET — no Go-specific build of the driver. +- cgo + static lib means `CGO_ENABLED=1` and a C toolchain on the Go build host; cross-compilation needs a cross C toolchain. +- Everything resolves at `go build` — no runtime resolver / `runtime.json` / RID probing. The same `ABI_VERSION` feeds Go's handshake (ADR 0004). + +## Alternatives considered +- Wrap the lib in NuGet for Go — rejected: Go can't consume NuGet. +- A neutral consumer bundle Go downloads — rejected (ADR 0001/0009): pulls irrelevant formats. +- Pure-Go reimplementation of the driver — rejected: defeats the shared-core goal. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-abi-version-handshake.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-abi-version-handshake.md new file mode 100644 index 0000000000..9ee9d61900 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-abi-version-handshake.md @@ -0,0 +1,20 @@ +# ADR 0004 — Native lib exports an ABI version; hosts check it before use + +**Status:** Accepted (proposed for review) + +## Context +The native library and each language SDK ship on independent cadences. A managed/native version mismatch at the byte level (the Application Binary Interface — ABI) corrupts memory or crashes deep in a call, with no clear cause. + +## Decision +- The native library exports **`cosmos_abi_version() -> u32`**, and ships the same value in the hand-off's `ABI_VERSION`. +- Every language host reads it **at load, before the first real call**, and fails fast with a versioned message when it is below the host's `MinSupported`. +- A breaking C-ABI change is a **major bump**, coordinated with SDK releases; each SDK pins a compatible range. + +## Consequences +- Mismatches surface as a clear, actionable error instead of a crash. +- Adds a tiny, one-time check at load per process. +- Creates an explicit compatibility contract across all three languages. + +## Alternatives considered +- Rely on package version pins only — rejected: private deployment / manual overrides bypass the package graph. +- No handshake — rejected: silent corruption is the worst failure mode here. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-binding-owns-marshalling.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-binding-owns-marshalling.md new file mode 100644 index 0000000000..41ae95b7d7 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-binding-owns-marshalling.md @@ -0,0 +1,20 @@ +# ADR 0005 — Each language binding owns marshalling and buffer copy-out + +**Status:** Accepted (proposed for review) + +## Context +The driver core is schema-agnostic: the C-ABI passes request/response bodies as raw bytes (`const uint8_t*` + length), never parsed JSON ([`NATIVE_WRAPPER_SPEC.md`](../../NATIVE_WRAPPER_SPEC.md)). Pushing serialization into the wrapper would re-introduce the parse/re-serialize/re-parse waste the old `azure_data_cosmos_native` crate had. + +## Decision +- The ABI stays **bytes-in / bytes-out**; the wrapper does no JSON parsing. +- Each language binding **owns its own marshalling** (string encoding, structs) and **copies response buffers out of native memory** into host memory, then frees the native buffer. +- Rust owns the buffer until the host copies it out; ownership transfer is explicit per the ABI spec. + +## Consequences +- The native binaries are identical for every language; differences live in each binding. +- No double-encoding cost across the FFI boundary. +- Each language must implement copy-out correctly (lifetime + free) — a per-binding responsibility. + +## Alternatives considered +- Serialize to JSON strings in the wrapper — rejected: redundant parsing, hides driver concepts. +- Zero-copy borrow of native buffers into host memory — rejected for now: lifetime hazards across GC'd hosts; copy-out is the safe default. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-native-is-opt-in.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-native-is-opt-in.md new file mode 100644 index 0000000000..d7d38c57ca --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-native-is-opt-in.md @@ -0,0 +1,20 @@ +# ADR 0006 — Native transport is opt-in until GA, then default-with-fallback + +**Status:** Accepted (proposed for review) + +## Context +`Microsoft.Azure.Cosmos` (pure managed) and the Go SDK run today on any platform with no native dependency. Introducing a platform-specific native library must not silently break that portability or force a migration on a version bump. + +## Decision +- Native transport is **opt-in** per SDK (a client option / build flag / env switch) until it is GA on every supported platform. +- Only after GA may a given SDK make native the **default**, and only with a **fallback** to the managed/pure path on an unsupported platform or a load failure. +- No consumer is ever force-migrated to native by a routine version bump. + +## Consequences +- The "runs anywhere" guarantee is preserved throughout the rollout. +- The native path can mature behind a flag with real dogfooding before it is the default. +- Each SDK needs a clean fallback path, not just an error, once native is the default. + +## Alternatives considered +- Native-on by default from day one — rejected: breaks portability before the platform matrix is proven. +- Separate "native-enabled" package flavor — kept as a possible packaging option, not mandated here. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-platform-matrix.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-platform-matrix.md new file mode 100644 index 0000000000..3d01806e63 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-platform-matrix.md @@ -0,0 +1,20 @@ +# ADR 0007 — A defined platform matrix; unsupported platforms error clearly + +**Status:** Accepted (proposed for review) + +## Context +A native library must be built per platform (OS + architecture + libc). The support surface must be bounded and explicit so build, signing, and testing are tractable, and so consumers get a clear answer on an unsupported platform. + +## Decision +- The GA matrix is: `win-x64`, `win-arm64`, `linux-x64` (glibc, low floor), `linux-musl-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. +- `linux-musl-x64` is a **separate target** from `linux-x64` (a glibc build will not load on musl). +- `wasm` is out of scope (no FFI story). An unsupported platform **fails with an actionable error** naming the supported set, never a silent or cryptic failure. + +## Consequences +- Bounded, predictable build/sign/test surface. +- Clear consumer experience on unsupported platforms. +- Adding a platform later is an additive ADR + matrix row, not a redesign. + +## Alternatives considered +- Build only the most common platforms and let others fail at link/load — rejected: poor experience, no clear message. +- Include `win-x86` / mobile now — deferred (open Q): no demand yet. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-build-and-signing-pipeline.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-build-and-signing-pipeline.md new file mode 100644 index 0000000000..bb8fcc7616 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-build-and-signing-pipeline.md @@ -0,0 +1,20 @@ +# ADR 0008 — One build, sign binaries once, fan-out; jobs never rebuild + +**Status:** Accepted (proposed for review) + +## Context +The binary is built in `azure-sdk-for-rust`, but the language packages live in other repos and publish to other feeds. The bytes that define the ABI must be signed at their source, and per-language packaging must not be able to alter, re-sign, or independently rebuild them — otherwise the languages drift onto different driver builds (the failure ADR 0001 exists to prevent). + +## Decision +- A single pipeline next to the Rust build produces the per-platform binaries (ADR 0001), **signs each binary once** (Authenticode on Windows; codesign + notarization on macOS), checksums them, and publishes the internal hand-off artifact. +- Per-language publish jobs **consume the already-signed hand-off** and emit NuGet / Go-consumable / JAR packages; they **never rebuild or re-sign the native binary** (they may sign their own package wrapper, e.g. the `.nupkg`/`.jar`, in that language's existing ESRP flow). +- Build-once is enforced by discipline: all language jobs consume one hand-off from one Rust build. + +## Consequences +- The ABI-defining bytes are signed once, at the source, identically for all languages. +- Per-language jobs are simple repackagers; they can live in or near each SDK repo. +- Requires the hand-off artifact (ADR 0001) and an SBOM / component-governance owner (open Q7). + +## Alternatives considered +- Each language rebuilds + signs its own copy — rejected: drift and duplicated trust roots. +- Sign only the language packages, not the native binary — rejected: leaves the actual loaded bytes unsigned. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-per-language-feed-distribution.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-per-language-feed-distribution.md new file mode 100644 index 0000000000..4678329de0 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-per-language-feed-distribution.md @@ -0,0 +1,23 @@ +# ADR 0009 — Distribute as per-language packages on each language's existing feeds + +**Status:** Accepted (proposed for review) + +## Context +Each language already has mature feed infrastructure with its own ACLs, signing (ESRP), and governance: .NET on nuget.org + an internal azure-sdk NuGet feed, Go on the azure-sdk-for-go feed, Java on the azure-sdk-for-java Maven feed + Maven Central. A neutral, cross-language consumer bundle would force consumers to download formats they cannot use (a Go user pulling DLLs/JARs) and would require new consumer feed infrastructure. + +## Decision +- The native driver is distributed as **a normal dependency in each language's native package format on that language's existing internal + external feed** — not as a neutral cross-language bundle and not on any new consumer feed. + - .NET → NuGet NativeAssets + meta-package on nuget.org + internal NuGet feed (ADR 0002). + - Go → cgo-consumable header + lib via the azure-sdk-for-go feed (ADR 0003). + - Java → JAR on the azure-sdk-for-java Maven feed + Maven Central (future; not finalized). +- A consumer only ever downloads its own language's package format. + +## Consequences +- Reuses each SDK's mature feed, ACL, signing, and governance — **no new consumer feed to build**. +- Idiomatic: matches how the Azure SDKs already ship dependencies. +- Drift protection now depends on pipeline discipline (ADR 0008) rather than a single shared consumer artifact. + +## Alternatives considered +- Single neutral consumer bundle/feed carrying all formats — rejected: forces irrelevant bytes on consumers; new infra; not idiomatic. + +> Together with ADR 0001, this replaces the earlier "consumers download a canonical bundle" model: provenance is an internal hand-off (0001); distribution is per-language feeds (0009). diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md new file mode 100644 index 0000000000..533578db0e --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md @@ -0,0 +1,134 @@ +# Design: distributing the native Cosmos driver to language SDKs + +> **Status:** Draft for review · **Author:** Cosmos Rust SDK team · **Type:** Design discussion (decisions are recorded separately as ADRs — see [`adr/`](adr/0000-index.md)) +> +> This is the *discussion* doc: context, alternatives, and the "why". The **decisions** live as short, immutable records in [`adr/`](adr/0000-index.md). If the two disagree, the ADR wins. + +## 1. Purpose + +The crate `azure_data_cosmos_driver_native` compiles to a native library — `azurecosmosdriver.{dll,so,dylib}` (and a static `.a`) plus a cbindgen-generated C header — wrapping the schema-agnostic driver core ([`NATIVE_WRAPPER_SPEC.md`](../NATIVE_WRAPPER_SPEC.md)). It is the single reuse point for **every non-Rust language SDK**. This doc defines how that compiled artifact reaches those SDKs: + +- **.NET** (`Microsoft.Azure.Cosmos`) — first consumer, via NuGet. +- **Go** (`azcosmos`) — near-term, via cgo (the Foreign Function Interface — FFI — mechanism Go uses to link C libraries). **Not** NuGet. +- **Java** — future, likely a JAR with bundled natives. Consumption model **not finalized; not urgent.** Called out so the design stays multi-language. + +Non-goal: the C-ABI surface itself — that is owned by `NATIVE_WRAPPER_SPEC.md`. This doc moves the *compiled bytes*; it does not define the functions. + +## 2. Core principle: separate provenance from distribution + +An earlier draft fused two roles into one consumer-facing "bundle". They are different concerns and must be decoupled: + +- **Provenance (build-once):** there must be exactly **one** build + binary-signing of the machine code per release, or .NET, Go, and Java silently drift onto different driver builds. This lives **internal to CI** and is never consumer-facing. +- **Distribution:** how a consumer physically pulls the bytes. This should be **each language's existing, idiomatic feed** — not a new neutral feed, and not a cross-language bundle. + +The whole design follows from keeping these apart: **one internal build artifact → fan-out to per-language packages on per-language feeds.** + +## 3. Goals and constraints + +| # | Goal | +|---|------| +| G1 | **Build-once provenance:** one signed set of platform binaries per release is the source of truth; no language rebuilds the driver. | +| G2 | **Idiomatic distribution:** each language consumes the driver as a normal dependency on its own feed (NuGet / Go module / Maven) — a Go consumer never sees a DLL or JAR. | +| G3 | A managed/native **version mismatch fails fast and clearly**, never as memory corruption. | +| G4 | Existing pure-managed / pure-Go consumers are **never force-migrated**; native is opt-in until proven. | +| G5 | **No new consumer-facing feed infrastructure** — reuse each SDK's mature feed, ACL, signing, and governance. | + +Hard constraint: the binary is built in `azure-sdk-for-rust`, but each language SDK lives in its own repo (`azure-cosmos-dotnet-v3`, `azure-sdk-for-go`, `azure-sdk-for-java`). Distribution crosses that repo boundary. + +## 4. Decisions at a glance + +Recorded as ADRs; this is the index with one-line rationale. See [`adr/`](adr/0000-index.md) for the authoritative form. + +| ADR | Decision | One-line why | +|---|---|---| +| [0001](adr/0001-build-once-internal-handoff.md) | One build produces all platform binaries + header + ABI version as an **internal-only hand-off artifact**; no neutral consumer bundle/feed. | Single provenance without forcing a cross-language download. | +| [0002](adr/0002-dotnet-nuget-nativeassets.md) | .NET consumes via **per-RID NuGet NativeAssets + meta-package** on nuget.org + the internal NuGet feed. | Matches SkiaSharp / SqlClient.SNI; download only your RID. | +| [0003](adr/0003-go-cgo-prebuilt.md) | Go consumes via **cgo against a prebuilt header + lib** delivered through the azure-sdk-for-go feed (Universal Package / vendored module). | Go has no NuGet; cgo links a C header + lib at build time. | +| [0004](adr/0004-abi-version-handshake.md) | The native lib exports **`cosmos_abi_version()`**; every host checks it before first use. | Independent cadences drift; fail fast, not corrupt. | +| [0005](adr/0005-binding-owns-marshalling.md) | Each binding **owns its marshalling and buffer copy-out**; the ABI stays bytes-in/bytes-out. | Keeps the ABI schema-agnostic. | +| [0006](adr/0006-native-is-opt-in.md) | Native transport is **opt-in** until GA on all platforms, then may default with fallback. | Preserve the "runs anywhere" guarantee. | +| [0007](adr/0007-platform-matrix.md) | A defined **platform matrix**; unsupported platforms fail with an actionable error. | Bounded support surface. | +| [0008](adr/0008-build-and-signing-pipeline.md) | **One build, sign the binaries once, fan-out** to per-language publish jobs that must consume the hand-off and never rebuild. | Provenance + trust enforced by pipeline discipline. | +| [0009](adr/0009-per-language-feed-distribution.md) | Distribution is **per-language native packages into each language's existing internal + external feed**; no neutral consumer feed. | Idiomatic; reuses mature per-language infra; download-only-your-format. | + +## 5. The model + +``` + Rust build — ONE run, all platforms (azure-sdk-for-rust CI) + produces .dll/.so/.dylib/.a + azurecosmosdriver.h + ABI_VERSION + and SIGNS THE BINARIES (Authenticode / codesign + notarize) + │ + ▼ + INTERNAL HAND-OFF ARTIFACT (Azure Artifacts Universal Package + or pipeline artifact) — provenance + SBOM anchor. + NOT consumer-facing. NOT language-shaped. ← ADR 0001 + │ + ┌─────────────────┼──────────────────┐ + .NET │ Go │ Java │ ← each job pulls the SAME signed binaries + job ▼ job ▼ job ▼ (never rebuilds — ADR 0008) + NuGet NativeAssets Go module / JAR w/ natives + → nuget.org + Universal Package → azure-sdk-for-java + internal NuGet → azure-sdk-for-go Maven feed + Central + │ feed │ + .NET consumers │ Java consumers + (only .nupkg) Go consumers (only .jar) + (only Go artifact) ← ADR 0009 +``` + +Consumer distribution is **per-language feeds**; the "bundle" is demoted from the distributed product to an **internal CI hand-off** no consumer touches. This removes any need for a new neutral consumer feed and ensures a consumer downloads only its own package format. + +## 6. Per-consumer link model + +### 6.1 .NET — NuGet (ADR 0002) +Per-RID **NativeAssets** packages (`Microsoft.Azure.Cosmos.NativeAssets.`) each carry one platform's dynamic lib under `runtimes//native/`, fronted by a thin **meta-package** whose `runtime.json` resolves the consumer's RID (Runtime IDentifier — the `-` platform string) to the right per-RID package. The managed package takes an **opt-in** dependency on the meta-package. Published to nuget.org **and** the internal azure-sdk NuGet feed. At runtime a custom resolver (`NativeLibrary.SetDllImportResolver`) loads the lib, runs the ABI handshake, and errors clearly on unsupported RIDs; P/Invoke uses source-generated `LibraryImport`. + +### 6.2 Go — cgo prebuilt (ADR 0003) +Go cannot consume NuGet. cgo links a **C header + library** at `go build` time: a `#cgo CFLAGS: -I` makes `#include "azurecosmosdriver.h"` resolve to the prebuilt header (cgo auto-generates the callable `C.*` symbols — no hand-written signatures), and `#cgo LDFLAGS: -L -lazurecosmosdriver` links the lib (static `.a` by default for a self-contained binary). The header + lib are delivered through the **azure-sdk-for-go feed** as an Azure Artifacts Universal Package fetched at build, or a vendored "binaries" Go module with per-OS build tags (open Q3). The Go layer already owns the completion-queue receive loop, `cgo.Handle` correlation, and copy-out of response buffers. **No runtime resolver / runtime.json / RID probing** — everything resolves at `go build`. + +### 6.3 Java — JAR (future, not finalized) +Likely a JAR bundling per-OS dynamic libs extracted and loaded via `System.load`, with JNI or Panama (Project Panama / FFM) as the FFI, published to the azure-sdk-for-java Maven feed + Maven Central. **Deliberately unspecified** — flagged so the build-once hand-off (ADR 0001) stays general enough to repackage into a JAR later without redesign. + +## 7. Platform matrix (ADR 0007) + +| Target / RID | Notes | +|---|---| +| `win-x64`, `win-arm64` | Authenticode-signed | +| `linux-x64` (glibc) | low glibc floor for portability | +| `linux-musl-x64` (Alpine) | distinct target — glibc build won't load on musl | +| `linux-arm64` | cross-compiled | +| `osx-x64`, `osx-arm64` | codesigned + notarized | + +Out: `wasm` (no FFI story). Open: `win-x86`, `linux-musl-arm64`, mobile. + +## 8. Build, signing, and CI (ADR 0008) + +One pipeline next to the Rust build produces the per-platform binaries, **signs each binary once** (Authenticode / codesign + notarize), checksums them, and publishes the **internal hand-off artifact**. Per-language publish jobs consume that artifact and emit NuGet / Go-consumable / JAR packages, signing only their **package wrapper** in that language's existing ESRP flow — they **never rebuild or re-sign the native binary**. Build-once is enforced by discipline: all language jobs consume one hand-off from one Rust build. Supply chain: SBOM / component governance for the Rust crate graph is new surface for the consuming orgs — owner TBD (Q7). + +## 9. ABI and versioning (ADR 0004) + +The lib exports `cosmos_abi_version() -> u32`; each host reads it at load and fails fast on `< MinSupported` with a versioned message. The native binaries carry their own SemVer; each language package pins a **compatible range**. A breaking C-ABI change = major bump + coordinated SDK releases. + +## 10. Compatibility posture (ADR 0006) + +Native is **opt-in** per SDK (client option / build flag) until GA on every supported platform. Only then may a given SDK make native the default, and only with a **fallback** to the managed/pure path on unsupported platforms or load failure. No consumer is force-migrated by a version bump. + +## 11. Rollout + +| Phase | Distribution | Consumers | Selection | +|---|---|---|---| +| 0 — dev loop | local artifact copy | us | always | +| 1 — internal | hand-off artifact; .NET fat NativeAssets on internal feed; Go via internal Universal Package | internal dogfood | opt-in | +| 2 — public prerelease | .NET `-preview` on nuget.org; Go module published; (Java TBD) | early adopters | opt-in | +| 3 — GA | per-RID NuGet + meta on nuget.org; Go module GA on azure-sdk-for-go feed | all | opt-in | +| 4 — native-default (later) | same | all | default + fallback | + +## 12. Open questions + +1. **Internal hand-off shape** — Azure Artifacts Universal Package vs raw pipeline artifact for the one-build → three-jobs hand-off? +2. **Packaging ownership** — central fan-out (in azure-sdk-for-rust publishing into three language feeds) vs each SDK repo pulls the hand-off and publishes itself? (Security-boundary question.) +3. **Go delivery** — Universal Package fetched at build vs a vendored "binaries" Go module? +4. **Static vs dynamic default for Go** — self-contained `.a` vs shared `.so`? +5. **Version mapping** — one native version fanned out to three feeds simultaneously, or independent per-language cadence off a pinned native version? +6. **NuGet naming** — `Microsoft.Azure.Cosmos.NativeAssets.*` (reserved prefix) confirmed; meta-package name? +7. **Supply chain** — SBOM / component-governance owner for the Rust crate graph. +8. **Java** — when does it need this, and does JNI vs Panama change any hand-off requirement (e.g. a Panama-friendly header)? From c44b29392c7af04e1addf9089b3be4eaf93df2d9 Mon Sep 17 00:00:00 2001 From: Ananth Mudumba Date: Tue, 23 Jun 2026 14:57:28 -0700 Subject: [PATCH 2/5] docs(cosmos): consolidate distribution ADRs pre-publish Since these ADRs are unpublished (no PR, no accepted record), the immutability/supersession discipline does not yet apply, so the pre-publication editing history was scrubbed for a clean first cut: - Drop ghost "supersedes the earlier canonical-bundle" scaffolding from 0001 and 0002 and the design doc; no reviewer ever saw that draft, so referencing it is noise. - Reorder per-language-feed distribution from 0009 to 0002 so the keystone distribution decision sits next to the build-once provenance decision (0001); renumber the former 0002-0008 to 0003-0009. Provenance (0001) and distribution (0002) stay as two decoupled records so either can be superseded independently later. - Re-point all in-text and link cross-references to the new numbers; reorder the index and design-doc decision tables to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../docs/distribution/adr/0000-index.md | 16 ++++----- .../adr/0001-build-once-internal-handoff.md | 7 ++-- ...=> 0002-per-language-feed-distribution.md} | 10 +++--- ...s.md => 0003-dotnet-nuget-nativeassets.md} | 6 ++-- ...go-prebuilt.md => 0004-go-cgo-prebuilt.md} | 6 ++-- ...shake.md => 0005-abi-version-handshake.md} | 2 +- ...ng.md => 0006-binding-owns-marshalling.md} | 2 +- ...-is-opt-in.md => 0007-native-is-opt-in.md} | 2 +- ...form-matrix.md => 0008-platform-matrix.md} | 2 +- ....md => 0009-build-and-signing-pipeline.md} | 2 +- .../docs/distribution/distribution-design.md | 36 +++++++++---------- 11 files changed, 45 insertions(+), 46 deletions(-) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0009-per-language-feed-distribution.md => 0002-per-language-feed-distribution.md} (80%) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0002-dotnet-nuget-nativeassets.md => 0003-dotnet-nuget-nativeassets.md} (91%) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0003-go-cgo-prebuilt.md => 0004-go-cgo-prebuilt.md} (93%) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0004-abi-version-handshake.md => 0005-abi-version-handshake.md} (94%) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0005-binding-owns-marshalling.md => 0006-binding-owns-marshalling.md} (95%) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0006-native-is-opt-in.md => 0007-native-is-opt-in.md} (94%) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0007-platform-matrix.md => 0008-platform-matrix.md} (94%) rename sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/{0008-build-and-signing-pipeline.md => 0009-build-and-signing-pipeline.md} (96%) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md index a316289c25..16b5ff0115 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md @@ -7,13 +7,13 @@ ADRs (Architecture Decision Records) capture **what we decided** and a brief **w | # | Title | Status | |---|-------|--------| | [0001](0001-build-once-internal-handoff.md) | One build → internal-only hand-off artifact; no neutral consumer bundle | Accepted | -| [0002](0002-dotnet-nuget-nativeassets.md) | .NET consumes via per-RID NuGet NativeAssets + meta-package | Accepted | -| [0003](0003-go-cgo-prebuilt.md) | Go consumes via cgo against a prebuilt header + lib from the Go feed | Accepted | -| [0004](0004-abi-version-handshake.md) | Native lib exports an ABI version; hosts check it before use | Accepted | -| [0005](0005-binding-owns-marshalling.md) | Each language binding owns marshalling and buffer copy-out | Accepted | -| [0006](0006-native-is-opt-in.md) | Native transport is opt-in until GA, then default-with-fallback | Accepted | -| [0007](0007-platform-matrix.md) | A defined platform matrix; unsupported platforms error clearly | Accepted | -| [0008](0008-build-and-signing-pipeline.md) | One build, sign binaries once, fan-out; jobs never rebuild | Accepted | -| [0009](0009-per-language-feed-distribution.md) | Distribute as per-language packages on each language's existing feeds | Accepted | +| [0002](0002-per-language-feed-distribution.md) | Distribute as per-language packages on each language's existing feeds | Accepted | +| [0003](0003-dotnet-nuget-nativeassets.md) | .NET consumes via per-RID NuGet NativeAssets + meta-package | Accepted | +| [0004](0004-go-cgo-prebuilt.md) | Go consumes via cgo against a prebuilt header + lib from the Go feed | Accepted | +| [0005](0005-abi-version-handshake.md) | Native lib exports an ABI version; hosts check it before use | Accepted | +| [0006](0006-binding-owns-marshalling.md) | Each language binding owns marshalling and buffer copy-out | Accepted | +| [0007](0007-native-is-opt-in.md) | Native transport is opt-in until GA, then default-with-fallback | Accepted | +| [0008](0008-platform-matrix.md) | A defined platform matrix; unsupported platforms error clearly | Accepted | +| [0009](0009-build-and-signing-pipeline.md) | One build, sign binaries once, fan-out; jobs never rebuild | Accepted | > These are **proposed** for the design review. "Accepted" is provisional until the review signs off. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md index 9b72c63af7..9f2920bf17 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md @@ -3,21 +3,20 @@ **Status:** Accepted (proposed for review) ## Context -The native driver must reach .NET, Go, and later Java, each with a different package format and its own feed. Two concerns were previously conflated: **provenance** (there must be exactly one build + binary-signing per release, or the languages drift onto different driver builds) and **distribution** (how a consumer pulls the bytes). A neutral consumer-facing bundle would force a Go user to download a package containing DLLs/JARs they cannot use, and would require standing up new consumer feed infrastructure. +The native driver must reach .NET, Go, and later Java, each with a different package format and its own feed. Two concerns are easily conflated: **provenance** (there must be exactly one build + binary-signing per release, or the languages drift onto different driver builds) and **distribution** (how a consumer pulls the bytes). A neutral consumer-facing bundle would force a Go user to download a package containing DLLs/JARs they cannot use, and would require standing up new consumer feed infrastructure. ## Decision - For each release, **one** Rust build produces all platform binaries (cdylib + staticlib), the cbindgen C header, an `ABI_VERSION`, and checksums, and **signs the binaries**. - These are published as an **internal-only hand-off artifact** (e.g. an Azure Artifacts Universal Package or a pipeline artifact) that is the **single source of truth for provenance**. -- This artifact is **not consumer-facing and not language-shaped.** Consumers never download it; per-language publish jobs consume it (see ADR 0008, ADR 0009). +- This artifact is **not consumer-facing and not language-shaped.** Consumers never download it; per-language publish jobs consume it (distribution is ADR 0002; the pipeline that produces and fans it out is ADR 0009). - The Rust crate declares `crate-type = ["cdylib", "staticlib"]` so both link forms come from one build. ## Consequences - Single provenance + SBOM anchor without forcing any cross-language or cross-platform download on consumers. - No new **consumer** feed to operate; only an internal hand-off artifact. -- All per-language jobs must consume this artifact rather than rebuilding (enforced by ADR 0008). +- All per-language jobs must consume this artifact rather than rebuilding (enforced by ADR 0009). ## Alternatives considered - Neutral consumer-facing bundle/feed — rejected: forces irrelevant bytes on consumers; new infra. - Per-language independent builds — rejected: drift risk, duplicated signing. -> Supersedes the earlier "canonical artifact bundle as a distributed product" framing. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-per-language-feed-distribution.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-per-language-feed-distribution.md similarity index 80% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-per-language-feed-distribution.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-per-language-feed-distribution.md index 4678329de0..f34eb34a8c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-per-language-feed-distribution.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-per-language-feed-distribution.md @@ -1,4 +1,4 @@ -# ADR 0009 — Distribute as per-language packages on each language's existing feeds +# ADR 0002 — Distribute as per-language packages on each language's existing feeds **Status:** Accepted (proposed for review) @@ -7,17 +7,17 @@ Each language already has mature feed infrastructure with its own ACLs, signing ## Decision - The native driver is distributed as **a normal dependency in each language's native package format on that language's existing internal + external feed** — not as a neutral cross-language bundle and not on any new consumer feed. - - .NET → NuGet NativeAssets + meta-package on nuget.org + internal NuGet feed (ADR 0002). - - Go → cgo-consumable header + lib via the azure-sdk-for-go feed (ADR 0003). + - .NET → NuGet NativeAssets + meta-package on nuget.org + internal NuGet feed (ADR 0003). + - Go → cgo-consumable header + lib via the azure-sdk-for-go feed (ADR 0004). - Java → JAR on the azure-sdk-for-java Maven feed + Maven Central (future; not finalized). - A consumer only ever downloads its own language's package format. ## Consequences - Reuses each SDK's mature feed, ACL, signing, and governance — **no new consumer feed to build**. - Idiomatic: matches how the Azure SDKs already ship dependencies. -- Drift protection now depends on pipeline discipline (ADR 0008) rather than a single shared consumer artifact. +- Drift protection now depends on pipeline discipline (ADR 0009) rather than a single shared consumer artifact. ## Alternatives considered - Single neutral consumer bundle/feed carrying all formats — rejected: forces irrelevant bytes on consumers; new infra; not idiomatic. -> Together with ADR 0001, this replaces the earlier "consumers download a canonical bundle" model: provenance is an internal hand-off (0001); distribution is per-language feeds (0009). +> Pairs with ADR 0001: provenance is the internal hand-off (0001); distribution is per-language feeds (this ADR). The two are intentionally decoupled. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-dotnet-nuget-nativeassets.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-dotnet-nuget-nativeassets.md similarity index 91% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-dotnet-nuget-nativeassets.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-dotnet-nuget-nativeassets.md index fbbc00020c..2302f0737c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-dotnet-nuget-nativeassets.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-dotnet-nuget-nativeassets.md @@ -1,4 +1,4 @@ -# ADR 0002 — .NET consumes via per-RID NuGet NativeAssets + meta-package +# ADR 0003 — .NET consumes via per-RID NuGet NativeAssets + meta-package **Status:** Accepted (proposed for review) @@ -8,8 +8,8 @@ ## Decision - Ship per-RID **`Microsoft.Azure.Cosmos.NativeAssets.`** packages, each carrying one platform's dynamic lib under `runtimes//native/`. - Front them with a thin **meta-package** whose `runtime.json` resolves the consumer's RID to the right per-RID package. -- `Microsoft.Azure.Cosmos` takes an **opt-in** dependency on the meta-package (ADR 0006). -- Publish to **nuget.org** and the **internal azure-sdk NuGet feed** (per ADR 0009). +- `Microsoft.Azure.Cosmos` takes an **opt-in** dependency on the meta-package (ADR 0007). +- Publish to **nuget.org** and the **internal azure-sdk NuGet feed** (per ADR 0002). ## Consequences - A consumer downloads only its platform's binary. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-go-cgo-prebuilt.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-go-cgo-prebuilt.md similarity index 93% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-go-cgo-prebuilt.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-go-cgo-prebuilt.md index d9b7cece4f..dd6f1f0381 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-go-cgo-prebuilt.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-go-cgo-prebuilt.md @@ -1,4 +1,4 @@ -# ADR 0003 — Go consumes via cgo against a prebuilt header + lib from the Go feed +# ADR 0004 — Go consumes via cgo against a prebuilt header + lib from the Go feed **Status:** Accepted (proposed for review) @@ -13,9 +13,9 @@ Go has no NuGet. Go links C libraries through cgo, which needs a C header and a ## Consequences - Go reuses the exact same signed binaries as .NET — no Go-specific build of the driver. - cgo + static lib means `CGO_ENABLED=1` and a C toolchain on the Go build host; cross-compilation needs a cross C toolchain. -- Everything resolves at `go build` — no runtime resolver / `runtime.json` / RID probing. The same `ABI_VERSION` feeds Go's handshake (ADR 0004). +- Everything resolves at `go build` — no runtime resolver / `runtime.json` / RID probing. The same `ABI_VERSION` feeds Go's handshake (ADR 0005). ## Alternatives considered - Wrap the lib in NuGet for Go — rejected: Go can't consume NuGet. -- A neutral consumer bundle Go downloads — rejected (ADR 0001/0009): pulls irrelevant formats. +- A neutral consumer bundle Go downloads — rejected (ADR 0001/0002): pulls irrelevant formats. - Pure-Go reimplementation of the driver — rejected: defeats the shared-core goal. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-abi-version-handshake.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-abi-version-handshake.md similarity index 94% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-abi-version-handshake.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-abi-version-handshake.md index 9ee9d61900..543b3add3f 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-abi-version-handshake.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-abi-version-handshake.md @@ -1,4 +1,4 @@ -# ADR 0004 — Native lib exports an ABI version; hosts check it before use +# ADR 0005 — Native lib exports an ABI version; hosts check it before use **Status:** Accepted (proposed for review) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-binding-owns-marshalling.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-binding-owns-marshalling.md similarity index 95% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-binding-owns-marshalling.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-binding-owns-marshalling.md index 41ae95b7d7..8f4b8e3e6a 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-binding-owns-marshalling.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-binding-owns-marshalling.md @@ -1,4 +1,4 @@ -# ADR 0005 — Each language binding owns marshalling and buffer copy-out +# ADR 0006 — Each language binding owns marshalling and buffer copy-out **Status:** Accepted (proposed for review) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-native-is-opt-in.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-native-is-opt-in.md similarity index 94% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-native-is-opt-in.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-native-is-opt-in.md index d7d38c57ca..0070dd2ea7 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-native-is-opt-in.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-native-is-opt-in.md @@ -1,4 +1,4 @@ -# ADR 0006 — Native transport is opt-in until GA, then default-with-fallback +# ADR 0007 — Native transport is opt-in until GA, then default-with-fallback **Status:** Accepted (proposed for review) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-platform-matrix.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-platform-matrix.md similarity index 94% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-platform-matrix.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-platform-matrix.md index 3d01806e63..0dd9a84c72 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-platform-matrix.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-platform-matrix.md @@ -1,4 +1,4 @@ -# ADR 0007 — A defined platform matrix; unsupported platforms error clearly +# ADR 0008 — A defined platform matrix; unsupported platforms error clearly **Status:** Accepted (proposed for review) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-build-and-signing-pipeline.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-build-and-signing-pipeline.md similarity index 96% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-build-and-signing-pipeline.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-build-and-signing-pipeline.md index bb8fcc7616..7fa8ca7aa4 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-build-and-signing-pipeline.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-build-and-signing-pipeline.md @@ -1,4 +1,4 @@ -# ADR 0008 — One build, sign binaries once, fan-out; jobs never rebuild +# ADR 0009 — One build, sign binaries once, fan-out; jobs never rebuild **Status:** Accepted (proposed for review) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md index 533578db0e..ae55eaf555 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md @@ -16,7 +16,7 @@ Non-goal: the C-ABI surface itself — that is owned by `NATIVE_WRAPPER_SPEC.md` ## 2. Core principle: separate provenance from distribution -An earlier draft fused two roles into one consumer-facing "bundle". They are different concerns and must be decoupled: +Provenance and distribution are two different concerns, and the design keeps them decoupled: - **Provenance (build-once):** there must be exactly **one** build + binary-signing of the machine code per release, or .NET, Go, and Java silently drift onto different driver builds. This lives **internal to CI** and is never consumer-facing. - **Distribution:** how a consumer physically pulls the bytes. This should be **each language's existing, idiomatic feed** — not a new neutral feed, and not a cross-language bundle. @@ -42,14 +42,14 @@ Recorded as ADRs; this is the index with one-line rationale. See [`adr/`](adr/00 | ADR | Decision | One-line why | |---|---|---| | [0001](adr/0001-build-once-internal-handoff.md) | One build produces all platform binaries + header + ABI version as an **internal-only hand-off artifact**; no neutral consumer bundle/feed. | Single provenance without forcing a cross-language download. | -| [0002](adr/0002-dotnet-nuget-nativeassets.md) | .NET consumes via **per-RID NuGet NativeAssets + meta-package** on nuget.org + the internal NuGet feed. | Matches SkiaSharp / SqlClient.SNI; download only your RID. | -| [0003](adr/0003-go-cgo-prebuilt.md) | Go consumes via **cgo against a prebuilt header + lib** delivered through the azure-sdk-for-go feed (Universal Package / vendored module). | Go has no NuGet; cgo links a C header + lib at build time. | -| [0004](adr/0004-abi-version-handshake.md) | The native lib exports **`cosmos_abi_version()`**; every host checks it before first use. | Independent cadences drift; fail fast, not corrupt. | -| [0005](adr/0005-binding-owns-marshalling.md) | Each binding **owns its marshalling and buffer copy-out**; the ABI stays bytes-in/bytes-out. | Keeps the ABI schema-agnostic. | -| [0006](adr/0006-native-is-opt-in.md) | Native transport is **opt-in** until GA on all platforms, then may default with fallback. | Preserve the "runs anywhere" guarantee. | -| [0007](adr/0007-platform-matrix.md) | A defined **platform matrix**; unsupported platforms fail with an actionable error. | Bounded support surface. | -| [0008](adr/0008-build-and-signing-pipeline.md) | **One build, sign the binaries once, fan-out** to per-language publish jobs that must consume the hand-off and never rebuild. | Provenance + trust enforced by pipeline discipline. | -| [0009](adr/0009-per-language-feed-distribution.md) | Distribution is **per-language native packages into each language's existing internal + external feed**; no neutral consumer feed. | Idiomatic; reuses mature per-language infra; download-only-your-format. | +| [0002](adr/0002-per-language-feed-distribution.md) | Distribution is **per-language native packages into each language's existing internal + external feed**; no neutral consumer feed. | Idiomatic; reuses mature per-language infra; download-only-your-format. | +| [0003](adr/0003-dotnet-nuget-nativeassets.md) | .NET consumes via **per-RID NuGet NativeAssets + meta-package** on nuget.org + the internal NuGet feed. | Matches SkiaSharp / SqlClient.SNI; download only your RID. | +| [0004](adr/0004-go-cgo-prebuilt.md) | Go consumes via **cgo against a prebuilt header + lib** delivered through the azure-sdk-for-go feed (Universal Package / vendored module). | Go has no NuGet; cgo links a C header + lib at build time. | +| [0005](adr/0005-abi-version-handshake.md) | The native lib exports **`cosmos_abi_version()`**; every host checks it before first use. | Independent cadences drift; fail fast, not corrupt. | +| [0006](adr/0006-binding-owns-marshalling.md) | Each binding **owns its marshalling and buffer copy-out**; the ABI stays bytes-in/bytes-out. | Keeps the ABI schema-agnostic. | +| [0007](adr/0007-native-is-opt-in.md) | Native transport is **opt-in** until GA on all platforms, then may default with fallback. | Preserve the "runs anywhere" guarantee. | +| [0008](adr/0008-platform-matrix.md) | A defined **platform matrix**; unsupported platforms fail with an actionable error. | Bounded support surface. | +| [0009](adr/0009-build-and-signing-pipeline.md) | **One build, sign the binaries once, fan-out** to per-language publish jobs that must consume the hand-off and never rebuild. | Provenance + trust enforced by pipeline discipline. | ## 5. The model @@ -65,30 +65,30 @@ Recorded as ADRs; this is the index with one-line rationale. See [`adr/`](adr/00 │ ┌─────────────────┼──────────────────┐ .NET │ Go │ Java │ ← each job pulls the SAME signed binaries - job ▼ job ▼ job ▼ (never rebuilds — ADR 0008) + job ▼ job ▼ job ▼ (never rebuilds — ADR 0009) NuGet NativeAssets Go module / JAR w/ natives → nuget.org + Universal Package → azure-sdk-for-java internal NuGet → azure-sdk-for-go Maven feed + Central │ feed │ .NET consumers │ Java consumers (only .nupkg) Go consumers (only .jar) - (only Go artifact) ← ADR 0009 + (only Go artifact) ← ADR 0002 ``` -Consumer distribution is **per-language feeds**; the "bundle" is demoted from the distributed product to an **internal CI hand-off** no consumer touches. This removes any need for a new neutral consumer feed and ensures a consumer downloads only its own package format. +Consumer distribution is **per-language feeds**; the build-once "bundle" is an **internal CI hand-off**, not the distributed product, so no consumer touches it. This removes any need for a new neutral consumer feed and ensures a consumer downloads only its own package format. ## 6. Per-consumer link model -### 6.1 .NET — NuGet (ADR 0002) +### 6.1 .NET — NuGet (ADR 0003) Per-RID **NativeAssets** packages (`Microsoft.Azure.Cosmos.NativeAssets.`) each carry one platform's dynamic lib under `runtimes//native/`, fronted by a thin **meta-package** whose `runtime.json` resolves the consumer's RID (Runtime IDentifier — the `-` platform string) to the right per-RID package. The managed package takes an **opt-in** dependency on the meta-package. Published to nuget.org **and** the internal azure-sdk NuGet feed. At runtime a custom resolver (`NativeLibrary.SetDllImportResolver`) loads the lib, runs the ABI handshake, and errors clearly on unsupported RIDs; P/Invoke uses source-generated `LibraryImport`. -### 6.2 Go — cgo prebuilt (ADR 0003) +### 6.2 Go — cgo prebuilt (ADR 0004) Go cannot consume NuGet. cgo links a **C header + library** at `go build` time: a `#cgo CFLAGS: -I` makes `#include "azurecosmosdriver.h"` resolve to the prebuilt header (cgo auto-generates the callable `C.*` symbols — no hand-written signatures), and `#cgo LDFLAGS: -L -lazurecosmosdriver` links the lib (static `.a` by default for a self-contained binary). The header + lib are delivered through the **azure-sdk-for-go feed** as an Azure Artifacts Universal Package fetched at build, or a vendored "binaries" Go module with per-OS build tags (open Q3). The Go layer already owns the completion-queue receive loop, `cgo.Handle` correlation, and copy-out of response buffers. **No runtime resolver / runtime.json / RID probing** — everything resolves at `go build`. ### 6.3 Java — JAR (future, not finalized) Likely a JAR bundling per-OS dynamic libs extracted and loaded via `System.load`, with JNI or Panama (Project Panama / FFM) as the FFI, published to the azure-sdk-for-java Maven feed + Maven Central. **Deliberately unspecified** — flagged so the build-once hand-off (ADR 0001) stays general enough to repackage into a JAR later without redesign. -## 7. Platform matrix (ADR 0007) +## 7. Platform matrix (ADR 0008) | Target / RID | Notes | |---|---| @@ -100,15 +100,15 @@ Likely a JAR bundling per-OS dynamic libs extracted and loaded via `System.load` Out: `wasm` (no FFI story). Open: `win-x86`, `linux-musl-arm64`, mobile. -## 8. Build, signing, and CI (ADR 0008) +## 8. Build, signing, and CI (ADR 0009) One pipeline next to the Rust build produces the per-platform binaries, **signs each binary once** (Authenticode / codesign + notarize), checksums them, and publishes the **internal hand-off artifact**. Per-language publish jobs consume that artifact and emit NuGet / Go-consumable / JAR packages, signing only their **package wrapper** in that language's existing ESRP flow — they **never rebuild or re-sign the native binary**. Build-once is enforced by discipline: all language jobs consume one hand-off from one Rust build. Supply chain: SBOM / component governance for the Rust crate graph is new surface for the consuming orgs — owner TBD (Q7). -## 9. ABI and versioning (ADR 0004) +## 9. ABI and versioning (ADR 0005) The lib exports `cosmos_abi_version() -> u32`; each host reads it at load and fails fast on `< MinSupported` with a versioned message. The native binaries carry their own SemVer; each language package pins a **compatible range**. A breaking C-ABI change = major bump + coordinated SDK releases. -## 10. Compatibility posture (ADR 0006) +## 10. Compatibility posture (ADR 0007) Native is **opt-in** per SDK (client option / build flag) until GA on every supported platform. Only then may a given SDK make native the default, and only with a **fallback** to the managed/pure path on unsupported platforms or load failure. No consumer is force-migrated by a version bump. From 0bd4c69c91c30362de46626ac291017c4460fe2d Mon Sep 17 00:00:00 2001 From: Ananth Mudumba Date: Tue, 23 Jun 2026 15:01:10 -0700 Subject: [PATCH 3/5] docs(cosmos): rename distribution dir to native-distribution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The generic "distribution" name was ambiguous; "native-distribution" makes clear these docs cover distributing the native (FFI) driver artifact to the language SDKs. Pure directory rename — all internal links are relative and unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../docs/{distribution => native-distribution}/adr/0000-index.md | 0 .../adr/0001-build-once-internal-handoff.md | 0 .../adr/0002-per-language-feed-distribution.md | 0 .../adr/0003-dotnet-nuget-nativeassets.md | 0 .../adr/0004-go-cgo-prebuilt.md | 0 .../adr/0005-abi-version-handshake.md | 0 .../adr/0006-binding-owns-marshalling.md | 0 .../adr/0007-native-is-opt-in.md | 0 .../adr/0008-platform-matrix.md | 0 .../adr/0009-build-and-signing-pipeline.md | 0 .../{distribution => native-distribution}/distribution-design.md | 0 11 files changed, 0 insertions(+), 0 deletions(-) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0000-index.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0001-build-once-internal-handoff.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0002-per-language-feed-distribution.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0003-dotnet-nuget-nativeassets.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0004-go-cgo-prebuilt.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0005-abi-version-handshake.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0006-binding-owns-marshalling.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0007-native-is-opt-in.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0008-platform-matrix.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/adr/0009-build-and-signing-pipeline.md (100%) rename sdk/cosmos/azure_data_cosmos_driver/docs/{distribution => native-distribution}/distribution-design.md (100%) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0000-index.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0001-build-once-internal-handoff.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0001-build-once-internal-handoff.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0001-build-once-internal-handoff.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-per-language-feed-distribution.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0002-per-language-feed-distribution.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0002-per-language-feed-distribution.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0002-per-language-feed-distribution.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-dotnet-nuget-nativeassets.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0003-dotnet-nuget-nativeassets.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0003-dotnet-nuget-nativeassets.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0003-dotnet-nuget-nativeassets.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-go-cgo-prebuilt.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0004-go-cgo-prebuilt.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0004-go-cgo-prebuilt.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0004-go-cgo-prebuilt.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-abi-version-handshake.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0005-abi-version-handshake.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0005-abi-version-handshake.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0005-abi-version-handshake.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-binding-owns-marshalling.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0006-binding-owns-marshalling.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0006-binding-owns-marshalling.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0006-binding-owns-marshalling.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-native-is-opt-in.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0007-native-is-opt-in.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0007-native-is-opt-in.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0007-native-is-opt-in.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-platform-matrix.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0008-platform-matrix.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0008-platform-matrix.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0008-platform-matrix.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-build-and-signing-pipeline.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0009-build-and-signing-pipeline.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/adr/0009-build-and-signing-pipeline.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0009-build-and-signing-pipeline.md diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md similarity index 100% rename from sdk/cosmos/azure_data_cosmos_driver/docs/distribution/distribution-design.md rename to sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md From 7e726413c556fa53f24089c9dfc34f99d4357758 Mon Sep 17 00:00:00 2001 From: Ananth Mudumba Date: Tue, 23 Jun 2026 15:32:30 -0700 Subject: [PATCH 4/5] docs(cosmos): address distribution design review findings Resolve the 9 review findings on PR #4651: - #1 CI: drop broken NATIVE_WRAPPER_SPEC.md links (lives in #4461); add cSpell terms (cbindgen, Authenticode, ESRP, manylinux, etc.) - #2 ADR 0001/0009: require RID-keyed hand-off + both .so/.a + C-only header - #3 ADR 0009: enforce build-once via checksum+signature verification - #4 ADR 0005: u32 is a monotonic ABI revision; add MaxSupported (reject too-new) - #5 promote Q5 to new ADR 0010 (single version fan-out) - #6 ADR 0000 template: Status under the title - #7 reciprocal Pairs-with note in ADR 0001 - #8 ADR 0008: name glibc floor 2.17 (manylinux2014) - #9 ADR 0007: silent-fallback for benign causes vs fail-loud for integrity/ABI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .vscode/cspell.json | 17 +++++ .../native-distribution/adr/0000-index.md | 3 +- .../adr/0001-build-once-internal-handoff.md | 3 + .../adr/0003-dotnet-nuget-nativeassets.md | 4 +- .../adr/0005-abi-version-handshake.md | 7 +- .../adr/0006-binding-owns-marshalling.md | 2 +- .../adr/0007-native-is-opt-in.md | 23 ++++--- .../adr/0008-platform-matrix.md | 2 +- .../adr/0009-build-and-signing-pipeline.md | 4 +- .../adr/0010-native-version-fanout.md | 19 ++++++ .../distribution-design.md | 68 +++++++------------ 11 files changed, 88 insertions(+), 64 deletions(-) create mode 100644 sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0010-native-version-fanout.md diff --git a/.vscode/cspell.json b/.vscode/cspell.json index 7427d86590..23c8aabd9c 100644 --- a/.vscode/cspell.json +++ b/.vscode/cspell.json @@ -27,14 +27,21 @@ "agentic", "amqp", "asyncoperation", + "authenticode", + "azcosmos", "azsdk", "azurecli", + "azurecosmosdriver", "bugbug", "callsite", "callsites", + "cbindgen", + "cflags", "checkpointstore", "clippy", "codeowners", + "codesign", + "codesigned", "contoso", "cplusplus", "cpptools", @@ -47,10 +54,13 @@ "devicecode", "docsrs", "doctest", + "dogfood", + "dogfooding", "dotenv", "downcasted", "downcasting", "entra", + "esrp", "etag", "eventhub", "eventhubs", @@ -62,11 +72,15 @@ "iothub", "keyvault", "lastexitcode", + "lazurecosmosdriver", + "ldflags", + "libc", "linuxpool", "linuxvmimage", "lldb", "macpool", "macvmimage", + "manylinux", "markdownlintrc", "msrc", "msrv", @@ -82,6 +96,8 @@ "posix", "pullrequest", "pwsh", + "reimplementation", + "repackagers", "reqwest", "resourcemanager", "rngs", @@ -93,6 +109,7 @@ "schannel", "seekable", "servicebus", + "skia", "spector", "splitmix", "startswith", diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md index 16b5ff0115..dba8a34fd8 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md @@ -2,7 +2,7 @@ ADRs (Architecture Decision Records) capture **what we decided** and a brief **why**, in a minimal, easy-to-reference form. They are **numbered and immutable**: once accepted, an ADR is not edited; a later ADR may **supersede** it. Detailed discussion and alternatives live in the design doc ([`../distribution-design.md`](../distribution-design.md)), not here — an ADR is the "regression test" that keeps us doing the same thing everywhere. -**Format (template):** Context (2–4 sentences) · Decision (1–3 bullets) · Consequences (2–4 bullets) · Alternatives considered (1 line each) · Status. +**Format (template):** Status (immediately under the title) · Context (2–4 sentences) · Decision (1–3 bullets) · Consequences (2–4 bullets) · Alternatives considered (1 line each). | # | Title | Status | |---|-------|--------| @@ -15,5 +15,6 @@ ADRs (Architecture Decision Records) capture **what we decided** and a brief **w | [0007](0007-native-is-opt-in.md) | Native transport is opt-in until GA, then default-with-fallback | Accepted | | [0008](0008-platform-matrix.md) | A defined platform matrix; unsupported platforms error clearly | Accepted | | [0009](0009-build-and-signing-pipeline.md) | One build, sign binaries once, fan-out; jobs never rebuild | Accepted | +| [0010](0010-native-version-fanout.md) | One native version fanned out to all feeds simultaneously; each SDK pins a compatible range | Proposed | > These are **proposed** for the design review. "Accepted" is provisional until the review signs off. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0001-build-once-internal-handoff.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0001-build-once-internal-handoff.md index 9f2920bf17..f7d9ad82b2 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0001-build-once-internal-handoff.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0001-build-once-internal-handoff.md @@ -9,6 +9,7 @@ The native driver must reach .NET, Go, and later Java, each with a different pac - For each release, **one** Rust build produces all platform binaries (cdylib + staticlib), the cbindgen C header, an `ABI_VERSION`, and checksums, and **signs the binaries**. - These are published as an **internal-only hand-off artifact** (e.g. an Azure Artifacts Universal Package or a pipeline artifact) that is the **single source of truth for provenance**. - This artifact is **not consumer-facing and not language-shaped.** Consumers never download it; per-language publish jobs consume it (distribution is ADR 0002; the pipeline that produces and fans it out is ADR 0009). +- The hand-off is **RID-keyed** (one subtree per `-`/libc target) and carries **both link forms per RID** — the dynamic `.dll`/`.so`/`.dylib` *and* the static `.a` — plus a single **C-only header** (no C++ constructs). This layout is a hard requirement so any language, including a future Java JAR (ADR 0002), can repackage straight from the hand-off with **no redesign and no rebuild**. - The Rust crate declares `crate-type = ["cdylib", "staticlib"]` so both link forms come from one build. ## Consequences @@ -20,3 +21,5 @@ The native driver must reach .NET, Go, and later Java, each with a different pac - Neutral consumer-facing bundle/feed — rejected: forces irrelevant bytes on consumers; new infra. - Per-language independent builds — rejected: drift risk, duplicated signing. +> Pairs with ADR 0002: provenance is this internal hand-off (0001); distribution is per-language feeds (0002). The two are intentionally decoupled. + diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0003-dotnet-nuget-nativeassets.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0003-dotnet-nuget-nativeassets.md index 2302f0737c..8573484bbc 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0003-dotnet-nuget-nativeassets.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0003-dotnet-nuget-nativeassets.md @@ -8,7 +8,7 @@ ## Decision - Ship per-RID **`Microsoft.Azure.Cosmos.NativeAssets.`** packages, each carrying one platform's dynamic lib under `runtimes//native/`. - Front them with a thin **meta-package** whose `runtime.json` resolves the consumer's RID to the right per-RID package. -- `Microsoft.Azure.Cosmos` takes an **opt-in** dependency on the meta-package (ADR 0007). +- The **native-transport major** of `Microsoft.Azure.Cosmos` takes a **direct dependency** on the meta-package (ADR 0007); the per-RID native then restores transitively like any NuGet dependency. There is no opt-in toggle — taking that major *is* taking native. (A `` always restores transitively, so an "opt-in dependency" would be a contradiction.) - Publish to **nuget.org** and the **internal azure-sdk NuGet feed** (per ADR 0002). ## Consequences @@ -18,4 +18,4 @@ ## Alternatives considered - Single "fat" NativeAssets package — kept only as an **interim** (Phase 1–2) for speed, not GA. -- Embed natives directly in `Microsoft.Azure.Cosmos` — rejected: bloats the flagship for pure-managed users. +- Embed every platform's native directly in the `Microsoft.Azure.Cosmos` package — rejected: forces every consumer to download all platforms' binaries (the per-RID split exists precisely to avoid that). Note this is *not* about sparing "pure-managed users" — in the native major (ADR 0007) there are none — it is about not shipping six platforms to every consumer. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0005-abi-version-handshake.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0005-abi-version-handshake.md index 543b3add3f..1a9e141d15 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0005-abi-version-handshake.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0005-abi-version-handshake.md @@ -6,12 +6,13 @@ The native library and each language SDK ship on independent cadences. A managed/native version mismatch at the byte level (the Application Binary Interface — ABI) corrupts memory or crashes deep in a call, with no clear cause. ## Decision -- The native library exports **`cosmos_abi_version() -> u32`**, and ships the same value in the hand-off's `ABI_VERSION`. -- Every language host reads it **at load, before the first real call**, and fails fast with a versioned message when it is below the host's `MinSupported`. -- A breaking C-ABI change is a **major bump**, coordinated with SDK releases; each SDK pins a compatible range. +- The native library exports **`cosmos_abi_version() -> u32`** — a **monotonically increasing integer ABI revision, not a packed SemVer** — and ships the same value in the hand-off's `ABI_VERSION`. +- Every language host reads it **at load, before the first real call**, and accepts only the **single ABI revision it was built against** (`Expected`): any mismatch — the native revision being **lower (too old)** *or* **higher (too new / unknown)** — fails fast with a versioned message, so a different native is never silently accepted. There is **no accepted range**: because every release is one coordinated native version fanned out to all SDKs (ADR 0010), each host targets exactly one ABI revision, which keeps the compatibility test matrix at a single point instead of a range. +- A breaking C-ABI change is a **major bump**, coordinated with SDK releases; each SDK pins the **exact** native version it targets (ADR 0010), not a range. ## Consequences - Mismatches surface as a clear, actionable error instead of a crash. +- Both too-old and too-new natives are rejected explicitly; a host never runs against an ABI revision it does not understand. - Adds a tiny, one-time check at load per process. - Creates an explicit compatibility contract across all three languages. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0006-binding-owns-marshalling.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0006-binding-owns-marshalling.md index 8f4b8e3e6a..a170e160d3 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0006-binding-owns-marshalling.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0006-binding-owns-marshalling.md @@ -3,7 +3,7 @@ **Status:** Accepted (proposed for review) ## Context -The driver core is schema-agnostic: the C-ABI passes request/response bodies as raw bytes (`const uint8_t*` + length), never parsed JSON ([`NATIVE_WRAPPER_SPEC.md`](../../NATIVE_WRAPPER_SPEC.md)). Pushing serialization into the wrapper would re-introduce the parse/re-serialize/re-parse waste the old `azure_data_cosmos_native` crate had. +The driver core is schema-agnostic: the C-ABI passes request/response bodies as raw bytes (`const uint8_t*` + length), never parsed JSON (see `NATIVE_WRAPPER_SPEC.md`, introduced in #4461). Pushing serialization into the wrapper would re-introduce the parse/re-serialize/re-parse waste the old `azure_data_cosmos_native` crate had. ## Decision - The ABI stays **bytes-in / bytes-out**; the wrapper does no JSON parsing. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0007-native-is-opt-in.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0007-native-is-opt-in.md index 0070dd2ea7..e9410b8e01 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0007-native-is-opt-in.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0007-native-is-opt-in.md @@ -1,20 +1,23 @@ -# ADR 0007 — Native transport is opt-in until GA, then default-with-fallback +# ADR 0007 — Native transport ships in a new SDK major; no parallel managed transport **Status:** Accepted (proposed for review) ## Context -`Microsoft.Azure.Cosmos` (pure managed) and the Go SDK run today on any platform with no native dependency. Introducing a platform-specific native library must not silently break that portability or force a migration on a version bump. +`Microsoft.Azure.Cosmos` (pure managed) and the Go SDK run today on any platform with no native dependency. Introducing a platform-specific native library changes the portability and trust profile of the SDK. The question is how consumers move onto native **without being force-migrated mid-stream**, and whether a managed and a native transport are maintained side by side. The switch to the Rust driver is a breaking change in how the SDK talks to the service — it lands at a **major version**, not as a per-call toggle. ## Decision -- Native transport is **opt-in** per SDK (a client option / build flag / env switch) until it is GA on every supported platform. -- Only after GA may a given SDK make native the **default**, and only with a **fallback** to the managed/pure path on an unsupported platform or a load failure. -- No consumer is ever force-migrated to native by a routine version bump. +- Native transport is introduced in a **new SDK major version**. Adopting that major **is** adopting native: there is no per-client / per-call opt-in switch, and no managed transport kept in parallel *inside the same major*. +- Existing majors keep their current managed / pure transport unchanged. No consumer is **force-migrated by a routine (minor/patch) bump** — they move to native deliberately, by choosing to take the new major. +- The supported surface is bounded by the platform matrix (ADR 0008). On an unsupported platform/RID the native major fails with a **clear, actionable error**; it does **not** silently fall back to a managed path. If a pure-managed / pure-Go audience must be served long-term, that is a **separate, explicitly limited package**, not a hidden fallback inside the flagship. +- **Integrity and contract failures are fail-loud:** a signature/checksum failure, a corrupt library, or an ABI mismatch (ADR 0005) on a library that *is* present surfaces a clear error and aborts — never a silent downgrade that could mask tampering or misconfiguration. ## Consequences -- The "runs anywhere" guarantee is preserved throughout the rollout. -- The native path can mature behind a flag with real dogfooding before it is the default. -- Each SDK needs a clean fallback path, not just an error, once native is the default. +- "Which transport am I on" has one answer per major: the native major is native, full stop — no per-deployment drift between managed and native, and no doubled transport matrix to test and support. +- The native path matures across the **prerelease line of the new major** (dogfood → preview → GA) rather than behind a runtime flag in an existing major. +- Migration is a deliberate major-version decision that goes through the usual major-version compatibility review — not a surprise on upgrade. +- Platforms outside the matrix are a clear error in the native major; serving them is a separate-package decision, not an in-SDK fallback. ## Alternatives considered -- Native-on by default from day one — rejected: breaks portability before the platform matrix is proven. -- Separate "native-enabled" package flavor — kept as a possible packaging option, not mandated here. +- **Per-client / per-call opt-in toggle inside one major (native *or* managed)** — rejected: keeps two transports alive in parallel, doubles the test/support matrix, and does not match how the switch actually happens — moving to the Rust driver is a major-version event. +- **Silent runtime fallback to managed on unsupported platforms** — rejected: hides capability gaps and muddies the trust model; an explicit error (or a separate limited package) is clearer. +- **Native-on by default within an existing major** — rejected: that is a force-migration by upgrade. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0008-platform-matrix.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0008-platform-matrix.md index 0dd9a84c72..79519b2b8d 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0008-platform-matrix.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0008-platform-matrix.md @@ -6,7 +6,7 @@ A native library must be built per platform (OS + architecture + libc). The support surface must be bounded and explicit so build, signing, and testing are tractable, and so consumers get a clear answer on an unsupported platform. ## Decision -- The GA matrix is: `win-x64`, `win-arm64`, `linux-x64` (glibc, low floor), `linux-musl-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. +- The GA matrix is: `win-x64`, `win-arm64`, `linux-x64` (glibc, **floor 2.17 — the manylinux2014 baseline**), `linux-musl-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. - `linux-musl-x64` is a **separate target** from `linux-x64` (a glibc build will not load on musl). - `wasm` is out of scope (no FFI story). An unsupported platform **fails with an actionable error** naming the supported set, never a silent or cryptic failure. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0009-build-and-signing-pipeline.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0009-build-and-signing-pipeline.md index 7fa8ca7aa4..f670e8219e 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0009-build-and-signing-pipeline.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0009-build-and-signing-pipeline.md @@ -8,11 +8,13 @@ The binary is built in `azure-sdk-for-rust`, but the language packages live in o ## Decision - A single pipeline next to the Rust build produces the per-platform binaries (ADR 0001), **signs each binary once** (Authenticode on Windows; codesign + notarization on macOS), checksums them, and publishes the internal hand-off artifact. - Per-language publish jobs **consume the already-signed hand-off** and emit NuGet / Go-consumable / JAR packages; they **never rebuild or re-sign the native binary** (they may sign their own package wrapper, e.g. the `.nupkg`/`.jar`, in that language's existing ESRP flow). -- Build-once is enforced by discipline: all language jobs consume one hand-off from one Rust build. +- Build-once is **enforced by verification, not just discipline**: before repackaging, every per-language publish job **verifies the hand-off's checksums and signatures** against the values published by the build job, and **fails the publish** on any mismatch — so a job physically cannot ship a rebuilt, altered, or unsigned native binary. ## Consequences - The ABI-defining bytes are signed once, at the source, identically for all languages. - Per-language jobs are simple repackagers; they can live in or near each SDK repo. +- The hand-off they fan out is RID-keyed and carries both link forms (`.so`/`.dll`/`.dylib` + `.a`) plus a C-only header (ADR 0001), so the one pipeline feeds .NET, Go, and a future Java JAR with no per-language rebuild. +- A tampered or accidentally-rebuilt binary is caught at publish time by the checksum/signature gate, not discovered in production. - Requires the hand-off artifact (ADR 0001) and an SBOM / component-governance owner (open Q7). ## Alternatives considered diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0010-native-version-fanout.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0010-native-version-fanout.md new file mode 100644 index 0000000000..4f3a559dc4 --- /dev/null +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0010-native-version-fanout.md @@ -0,0 +1,19 @@ +# ADR 0010 — One native version fanned out to all feeds simultaneously; each SDK pins that exact version + +**Status:** Proposed (for design review) + +## Context +The native driver ships on its own SemVer (ADR 0005), while .NET, Go, and Java each release on their own cadence. A foundational question underlies the whole distribution model: when a new native version is built, is it published to every language feed **at once** off the single hand-off (ADR 0001), or may each language pull a pinned native version and publish on its **own** schedule? This decides whether "which driver am I running" has one answer across languages or N answers, and it shapes the fan-out pipeline (ADR 0009) — so it belongs in an ADR, not an open question. + +## Decision +- A given native release (one build + signing, ADR 0001) is **fanned out to all language feeds simultaneously** from that single hand-off — .NET, Go, and (later) Java publish the *same* native version together. +- Each language SDK **pins the exact native version** from that fan-out and may cut its *own* managed/SDK releases independently between native releases; what it must not do is float onto a *different* native build. +- The native SemVer is the **one source of truth** for "which driver"; per-language package versions map to it but never fork it. Hosts enforce this at load by accepting only that exact ABI revision (ADR 0005) — there is no compatibility *range*, which keeps the test matrix a single point. + +## Consequences +- No cross-language version skew: a fix in the native driver lands everywhere in one coordinated fan-out, not at N independent times. +- The fan-out pipeline (ADR 0009) publishes a release as an all-or-nothing set; a partial fan-out (some feeds updated, some not) is an explicit failure state to guard. +- Languages keep cadence freedom for their *own* surface area while staying lockstep on the native bytes. + +## Alternatives considered +- Fully independent per-language cadence off pinned native versions — rejected for GA: maximizes skew and makes "which driver is this customer on" unanswerable across languages; revisit only if a language's release cadence makes lockstep impractical. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md index ae55eaf555..752ea1b01c 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md @@ -2,11 +2,11 @@ > **Status:** Draft for review · **Author:** Cosmos Rust SDK team · **Type:** Design discussion (decisions are recorded separately as ADRs — see [`adr/`](adr/0000-index.md)) > -> This is the *discussion* doc: context, alternatives, and the "why". The **decisions** live as short, immutable records in [`adr/`](adr/0000-index.md). If the two disagree, the ADR wins. +> This is the *discussion* doc: context, alternatives, and the "why". The **decisions** live as short, immutable records in [`adr/`](adr/0000-index.md) and are **not restated here** — this body keeps only what the ADRs don't carry (the narrative, the model diagram, rollout, and open questions). If the two disagree, the ADR wins. ## 1. Purpose -The crate `azure_data_cosmos_driver_native` compiles to a native library — `azurecosmosdriver.{dll,so,dylib}` (and a static `.a`) plus a cbindgen-generated C header — wrapping the schema-agnostic driver core ([`NATIVE_WRAPPER_SPEC.md`](../NATIVE_WRAPPER_SPEC.md)). It is the single reuse point for **every non-Rust language SDK**. This doc defines how that compiled artifact reaches those SDKs: +The crate `azure_data_cosmos_driver_native` compiles to a native library — `azurecosmosdriver.{dll,so,dylib}` (and a static `.a`) plus a cbindgen-generated C header — wrapping the schema-agnostic driver core (see `NATIVE_WRAPPER_SPEC.md`, introduced in #4461). It is the single reuse point for **every non-Rust language SDK**. This doc defines how that compiled artifact reaches those SDKs: - **.NET** (`Microsoft.Azure.Cosmos`) — first consumer, via NuGet. - **Go** (`azcosmos`) — near-term, via cgo (the Foreign Function Interface — FFI — mechanism Go uses to link C libraries). **Not** NuGet. @@ -30,7 +30,7 @@ The whole design follows from keeping these apart: **one internal build artifact | G1 | **Build-once provenance:** one signed set of platform binaries per release is the source of truth; no language rebuilds the driver. | | G2 | **Idiomatic distribution:** each language consumes the driver as a normal dependency on its own feed (NuGet / Go module / Maven) — a Go consumer never sees a DLL or JAR. | | G3 | A managed/native **version mismatch fails fast and clearly**, never as memory corruption. | -| G4 | Existing pure-managed / pure-Go consumers are **never force-migrated**; native is opt-in until proven. | +| G4 | Existing consumers are **never force-migrated by a routine (minor/patch) bump**; native arrives in a **new SDK major** (ADR 0007), adopted deliberately. | | G5 | **No new consumer-facing feed infrastructure** — reuse each SDK's mature feed, ACL, signing, and governance. | Hard constraint: the binary is built in `azure-sdk-for-rust`, but each language SDK lives in its own repo (`azure-cosmos-dotnet-v3`, `azure-sdk-for-go`, `azure-sdk-for-java`). Distribution crosses that repo boundary. @@ -47,9 +47,10 @@ Recorded as ADRs; this is the index with one-line rationale. See [`adr/`](adr/00 | [0004](adr/0004-go-cgo-prebuilt.md) | Go consumes via **cgo against a prebuilt header + lib** delivered through the azure-sdk-for-go feed (Universal Package / vendored module). | Go has no NuGet; cgo links a C header + lib at build time. | | [0005](adr/0005-abi-version-handshake.md) | The native lib exports **`cosmos_abi_version()`**; every host checks it before first use. | Independent cadences drift; fail fast, not corrupt. | | [0006](adr/0006-binding-owns-marshalling.md) | Each binding **owns its marshalling and buffer copy-out**; the ABI stays bytes-in/bytes-out. | Keeps the ABI schema-agnostic. | -| [0007](adr/0007-native-is-opt-in.md) | Native transport is **opt-in** until GA on all platforms, then may default with fallback. | Preserve the "runs anywhere" guarantee. | +| [0007](adr/0007-native-is-opt-in.md) | Native transport ships in a **new SDK major**; no parallel managed transport, no silent fallback. | A major-version cutover, not a per-call opt-in. | | [0008](adr/0008-platform-matrix.md) | A defined **platform matrix**; unsupported platforms fail with an actionable error. | Bounded support surface. | | [0009](adr/0009-build-and-signing-pipeline.md) | **One build, sign the binaries once, fan-out** to per-language publish jobs that must consume the hand-off and never rebuild. | Provenance + trust enforced by pipeline discipline. | +| [0010](adr/0010-native-version-fanout.md) | **One native version fanned out to all feeds simultaneously**; each SDK pins that **exact** version. | One source of truth for "which driver"; no cross-language version skew. | ## 5. The model @@ -77,58 +78,35 @@ Recorded as ADRs; this is the index with one-line rationale. See [`adr/`](adr/00 Consumer distribution is **per-language feeds**; the build-once "bundle" is an **internal CI hand-off**, not the distributed product, so no consumer touches it. This removes any need for a new neutral consumer feed and ensures a consumer downloads only its own package format. -## 6. Per-consumer link model +## 6. Per-consumer and pipeline specifics → ADRs -### 6.1 .NET — NuGet (ADR 0003) -Per-RID **NativeAssets** packages (`Microsoft.Azure.Cosmos.NativeAssets.`) each carry one platform's dynamic lib under `runtimes//native/`, fronted by a thin **meta-package** whose `runtime.json` resolves the consumer's RID (Runtime IDentifier — the `-` platform string) to the right per-RID package. The managed package takes an **opt-in** dependency on the meta-package. Published to nuget.org **and** the internal azure-sdk NuGet feed. At runtime a custom resolver (`NativeLibrary.SetDllImportResolver`) loads the lib, runs the ABI handshake, and errors clearly on unsupported RIDs; P/Invoke uses source-generated `LibraryImport`. +The per-language link model, platform matrix, build/signing pipeline, ABI handshake, and compatibility posture are all **decisions** — so they live in the ADRs rather than being restated here (restating them is exactly what lets the doc and the ADRs drift). The at-a-glance table in [§4](#4-decisions-at-a-glance) links each one; in brief: -### 6.2 Go — cgo prebuilt (ADR 0004) -Go cannot consume NuGet. cgo links a **C header + library** at `go build` time: a `#cgo CFLAGS: -I` makes `#include "azurecosmosdriver.h"` resolve to the prebuilt header (cgo auto-generates the callable `C.*` symbols — no hand-written signatures), and `#cgo LDFLAGS: -L -lazurecosmosdriver` links the lib (static `.a` by default for a self-contained binary). The header + lib are delivered through the **azure-sdk-for-go feed** as an Azure Artifacts Universal Package fetched at build, or a vendored "binaries" Go module with per-OS build tags (open Q3). The Go layer already owns the completion-queue receive loop, `cgo.Handle` correlation, and copy-out of response buffers. **No runtime resolver / runtime.json / RID probing** — everything resolves at `go build`. +- **.NET / Go consumption** — [ADR 0003](adr/0003-dotnet-nuget-nativeassets.md) (per-RID NuGet NativeAssets + meta-package), [ADR 0004](adr/0004-go-cgo-prebuilt.md) (cgo against a prebuilt header + lib). +- **Java** — deliberately **not finalized** (likely a JAR with bundled natives); tracked in [§8](#8-open-questions) so the build-once hand-off ([ADR 0001](adr/0001-build-once-internal-handoff.md)) stays general enough to repackage into a JAR later without redesign. +- **Platform matrix** — [ADR 0008](adr/0008-platform-matrix.md). +- **ABI handshake & versioning** — [ADR 0005](adr/0005-abi-version-handshake.md) and [ADR 0010](adr/0010-native-version-fanout.md). +- **Build, signing & CI** — [ADR 0009](adr/0009-build-and-signing-pipeline.md). +- **Compatibility posture (major-version cutover, no parallel managed transport)** — [ADR 0007](adr/0007-native-is-opt-in.md). -### 6.3 Java — JAR (future, not finalized) -Likely a JAR bundling per-OS dynamic libs extracted and loaded via `System.load`, with JNI or Panama (Project Panama / FFM) as the FFI, published to the azure-sdk-for-java Maven feed + Maven Central. **Deliberately unspecified** — flagged so the build-once hand-off (ADR 0001) stays general enough to repackage into a JAR later without redesign. +## 7. Rollout -## 7. Platform matrix (ADR 0008) - -| Target / RID | Notes | -|---|---| -| `win-x64`, `win-arm64` | Authenticode-signed | -| `linux-x64` (glibc) | low glibc floor for portability | -| `linux-musl-x64` (Alpine) | distinct target — glibc build won't load on musl | -| `linux-arm64` | cross-compiled | -| `osx-x64`, `osx-arm64` | codesigned + notarized | - -Out: `wasm` (no FFI story). Open: `win-x86`, `linux-musl-arm64`, mobile. - -## 8. Build, signing, and CI (ADR 0009) - -One pipeline next to the Rust build produces the per-platform binaries, **signs each binary once** (Authenticode / codesign + notarize), checksums them, and publishes the **internal hand-off artifact**. Per-language publish jobs consume that artifact and emit NuGet / Go-consumable / JAR packages, signing only their **package wrapper** in that language's existing ESRP flow — they **never rebuild or re-sign the native binary**. Build-once is enforced by discipline: all language jobs consume one hand-off from one Rust build. Supply chain: SBOM / component governance for the Rust crate graph is new surface for the consuming orgs — owner TBD (Q7). - -## 9. ABI and versioning (ADR 0005) - -The lib exports `cosmos_abi_version() -> u32`; each host reads it at load and fails fast on `< MinSupported` with a versioned message. The native binaries carry their own SemVer; each language package pins a **compatible range**. A breaking C-ABI change = major bump + coordinated SDK releases. - -## 10. Compatibility posture (ADR 0007) - -Native is **opt-in** per SDK (client option / build flag) until GA on every supported platform. Only then may a given SDK make native the default, and only with a **fallback** to the managed/pure path on unsupported platforms or load failure. No consumer is force-migrated by a version bump. - -## 11. Rollout - -| Phase | Distribution | Consumers | Selection | +| Phase | Distribution | Consumers | Adoption | |---|---|---|---| -| 0 — dev loop | local artifact copy | us | always | -| 1 — internal | hand-off artifact; .NET fat NativeAssets on internal feed; Go via internal Universal Package | internal dogfood | opt-in | -| 2 — public prerelease | .NET `-preview` on nuget.org; Go module published; (Java TBD) | early adopters | opt-in | -| 3 — GA | per-RID NuGet + meta on nuget.org; Go module GA on azure-sdk-for-go feed | all | opt-in | -| 4 — native-default (later) | same | all | default + fallback | +| 0 — dev loop | local artifact copy | us | local build | +| 1 — internal | hand-off artifact; .NET fat NativeAssets on internal feed; Go via internal Universal Package | internal dogfood | native major, prerelease | +| 2 — public prerelease | .NET `-preview` on nuget.org; Go module published; (Java TBD) | early adopters | native major, `-preview` | +| 3 — GA | per-RID NuGet + meta on nuget.org; Go module GA on azure-sdk-for-go feed | all | native major, GA (native-only) | + +The native transport is the defining change of the new major; there is no separate "switch native on by default later" phase, because within that major native is the only transport (ADR 0007). -## 12. Open questions +## 8. Open questions 1. **Internal hand-off shape** — Azure Artifacts Universal Package vs raw pipeline artifact for the one-build → three-jobs hand-off? 2. **Packaging ownership** — central fan-out (in azure-sdk-for-rust publishing into three language feeds) vs each SDK repo pulls the hand-off and publishes itself? (Security-boundary question.) 3. **Go delivery** — Universal Package fetched at build vs a vendored "binaries" Go module? 4. **Static vs dynamic default for Go** — self-contained `.a` vs shared `.so`? -5. **Version mapping** — one native version fanned out to three feeds simultaneously, or independent per-language cadence off a pinned native version? +5. **Version mapping** — *Resolved by [ADR 0010](adr/0010-native-version-fanout.md):* one native version is fanned out to all feeds simultaneously and each SDK pins that **exact** version (hosts accept only that one ABI revision, ADR 0005), rather than an independent per-language cadence off a pinned range. 6. **NuGet naming** — `Microsoft.Azure.Cosmos.NativeAssets.*` (reserved prefix) confirmed; meta-package name? 7. **Supply chain** — SBOM / component-governance owner for the Rust crate graph. 8. **Java** — when does it need this, and does JNI vs Panama change any hand-off requirement (e.g. a Panama-friendly header)? From 1ece734b2d931afec222fc338b315940821d80f5 Mon Sep 17 00:00:00 2001 From: ananth7592 <223556219+Copilot@users.noreply.github.com> Date: Thu, 9 Jul 2026 13:03:50 -0700 Subject: [PATCH 5/5] docs(cosmos): mark Go publish strategy WIP; firm up cgo-packaged-binary, discuss pure-Go-port alternative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Go distribution (ADR 0004) is marked WIP on its delivery shape while the core model is stated firmly: Go links a PACKAGED native binary via cgo (CGO_ENABLED=1) through the C.* FFI stubs from the cbindgen header. Explicitly rules out a pure-Go shim that downloads the native library at build/run time — the binary is packaged and linked, never fetched by a stub. Also expands the pure-Go-reimplementation alternative with the findings from the time-boxed spike (a working CGO=0, zero-dependency vertical slice validated behavior-for-behavior against the real Rust driver via a differential harness), and records why it stays rejected for distribution (abandons build-once single provenance). Adds a narrative 'Alternative considered — a pure-Go port' section and tightens the Go-delivery open question to encode the no-shim constraint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../docs/native-distribution/adr/0000-index.md | 2 +- .../native-distribution/adr/0004-go-cgo-prebuilt.md | 12 ++++++++---- .../docs/native-distribution/distribution-design.md | 12 ++++++++++-- 3 files changed, 19 insertions(+), 7 deletions(-) diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md index dba8a34fd8..a9954cf351 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0000-index.md @@ -9,7 +9,7 @@ ADRs (Architecture Decision Records) capture **what we decided** and a brief **w | [0001](0001-build-once-internal-handoff.md) | One build → internal-only hand-off artifact; no neutral consumer bundle | Accepted | | [0002](0002-per-language-feed-distribution.md) | Distribute as per-language packages on each language's existing feeds | Accepted | | [0003](0003-dotnet-nuget-nativeassets.md) | .NET consumes via per-RID NuGet NativeAssets + meta-package | Accepted | -| [0004](0004-go-cgo-prebuilt.md) | Go consumes via cgo against a prebuilt header + lib from the Go feed | Accepted | +| [0004](0004-go-cgo-prebuilt.md) | Go consumes via cgo against a prebuilt header + lib from the Go feed | Proposed (delivery shape WIP) | | [0005](0005-abi-version-handshake.md) | Native lib exports an ABI version; hosts check it before use | Accepted | | [0006](0006-binding-owns-marshalling.md) | Each language binding owns marshalling and buffer copy-out | Accepted | | [0007](0007-native-is-opt-in.md) | Native transport is opt-in until GA, then default-with-fallback | Accepted | diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0004-go-cgo-prebuilt.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0004-go-cgo-prebuilt.md index dd6f1f0381..40dc0cbf3a 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0004-go-cgo-prebuilt.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/adr/0004-go-cgo-prebuilt.md @@ -1,14 +1,17 @@ # ADR 0004 — Go consumes via cgo against a prebuilt header + lib from the Go feed -**Status:** Accepted (proposed for review) +**Status:** Proposed for review — core model firm, **delivery shape WIP** + +> **WIP.** The *decision* — Go links a **packaged** prebuilt native library via cgo (`CGO_ENABLED=1`) using the `C.*` FFI stubs from the cbindgen header — is firm. Still being settled: the **delivery shape** (Universal Package vs vendored binaries module, Q3) and the static-vs-dynamic default (Q4). Explicitly **out of scope**: a pure-Go shim that downloads the native lib — the binary is packaged and linked, never fetched by a stub. The pure-Go-*port* alternative (re-implementing the driver in Go) was spiked and is discussed under *Alternatives considered* + the design doc. ## Context Go has no NuGet. Go links C libraries through cgo, which needs a C header and a library available at `go build` time. The Go SDK already implements the completion-queue receive loop, `cgo.Handle` correlation, and buffer copy-out; the only distribution question is how header+lib+ABI version reach the Go build. ## Decision -- Go consumes the prebuilt **`include/` header and `lib/` library via cgo** (`#cgo CFLAGS -I…` to parse the header into `C.*` symbols; `#cgo LDFLAGS -L… -lazurecosmosdriver` to link), **not NuGet**. +- Go consumes the prebuilt **`include/` header and `lib/` library via cgo** with **`CGO_ENABLED=1`**: `#cgo CFLAGS -I…` parses the header into the `C.*` **FFI stubs**, and `#cgo LDFLAGS -L… -lazurecosmosdriver` links the library. **Not** NuGet, and **not** a pure-Go build. +- The native library is **packaged inside the delivered artifact** and linked at `go build`. It is **not** a pure-Go shim that downloads the library at build/run time — customers link a real binary through the FFI stubs; they are not handed a downloader stub. - Prefer the **static `.a`** for a self-contained Go binary; dynamic linking is supported as an option. -- The header + lib are delivered through the **azure-sdk-for-go feed** — an Azure Artifacts Universal Package fetched at build, or a vendored "binaries" Go module with per-OS build tags (delivery shape is open Q3). Either way it derives from the ADR 0001 hand-off artifact. +- The header + lib are delivered through the **azure-sdk-for-go feed** — an Azure Artifacts Universal Package fetched at build, or a vendored "binaries" Go module with per-OS build tags (delivery *shape* is open Q3; in every shape the binary is **packaged**, not fetched by a shim). Either way it derives from the ADR 0001 hand-off artifact. ## Consequences - Go reuses the exact same signed binaries as .NET — no Go-specific build of the driver. @@ -18,4 +21,5 @@ Go has no NuGet. Go links C libraries through cgo, which needs a C header and a ## Alternatives considered - Wrap the lib in NuGet for Go — rejected: Go can't consume NuGet. - A neutral consumer bundle Go downloads — rejected (ADR 0001/0002): pulls irrelevant formats. -- Pure-Go reimplementation of the driver — rejected: defeats the shared-core goal. +- A pure-Go **shim module** that downloads the native lib at build/run time — rejected: customers can't be handed a downloader stub; the binary must be packaged and linked via cgo. +- **Pure-Go reimplementation of the driver — rejected (for distribution).** A time-boxed spike built a working `CGO_ENABLED=0`, zero-dependency vertical slice (point read/create + retry, routing, and multi-region/transport failover) and validated it behavior-for-behavior against the real Rust driver with a differential harness. It still defeats build-once single provenance (ADR 0001): every language would re-port and re-verify, making "the driver" N hand-maintained translations that chase every Rust change. Valuable as a parity oracle / risk probe, not a shipping vehicle — see the design doc's *Alternative considered — a pure-Go port* section. diff --git a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md index 752ea1b01c..3b3f7f5e92 100644 --- a/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md +++ b/sdk/cosmos/azure_data_cosmos_driver/docs/native-distribution/distribution-design.md @@ -9,7 +9,7 @@ The crate `azure_data_cosmos_driver_native` compiles to a native library — `azurecosmosdriver.{dll,so,dylib}` (and a static `.a`) plus a cbindgen-generated C header — wrapping the schema-agnostic driver core (see `NATIVE_WRAPPER_SPEC.md`, introduced in #4461). It is the single reuse point for **every non-Rust language SDK**. This doc defines how that compiled artifact reaches those SDKs: - **.NET** (`Microsoft.Azure.Cosmos`) — first consumer, via NuGet. -- **Go** (`azcosmos`) — near-term, via cgo (the Foreign Function Interface — FFI — mechanism Go uses to link C libraries). **Not** NuGet. +- **Go** (`azcosmos`) — near-term, via cgo (the Foreign Function Interface — FFI — mechanism Go uses to link C libraries). **Not** NuGet. *(Publish strategy is **WIP** — see the callout in [§6](#6-per-consumer-and-pipeline-specifics--adrs) and [ADR 0004](adr/0004-go-cgo-prebuilt.md).)* - **Java** — future, likely a JAR with bundled natives. Consumption model **not finalized; not urgent.** Called out so the design stays multi-language. Non-goal: the C-ABI surface itself — that is owned by `NATIVE_WRAPPER_SPEC.md`. This doc moves the *compiled bytes*; it does not define the functions. @@ -82,6 +82,8 @@ Consumer distribution is **per-language feeds**; the build-once "bundle" is an * The per-language link model, platform matrix, build/signing pipeline, ABI handshake, and compatibility posture are all **decisions** — so they live in the ADRs rather than being restated here (restating them is exactly what lets the doc and the ADRs drift). The at-a-glance table in [§4](#4-decisions-at-a-glance) links each one; in brief: +> **⚠️ Go publish strategy — WIP.** The direction is firm: Go consumes a **packaged native binary via cgo** (`CGO_ENABLED=1`), linking the **FFI stubs generated from the cbindgen C header** — the driver ships **inside** the consumed artifact and is linked at `go build`. What is explicitly **ruled out** is a "pure-Go shim" that downloads the native library at build or run time: customers link a real binary through the FFI stubs, they are never handed a downloader stub. Still being pressure-tested: the *delivery shape* ([§8](#8-open-questions) Q3), the static-vs-dynamic default (Q4), and the pure-Go-port alternative (below). Recorded firmly in [ADR 0004](adr/0004-go-cgo-prebuilt.md). + - **.NET / Go consumption** — [ADR 0003](adr/0003-dotnet-nuget-nativeassets.md) (per-RID NuGet NativeAssets + meta-package), [ADR 0004](adr/0004-go-cgo-prebuilt.md) (cgo against a prebuilt header + lib). - **Java** — deliberately **not finalized** (likely a JAR with bundled natives); tracked in [§8](#8-open-questions) so the build-once hand-off ([ADR 0001](adr/0001-build-once-internal-handoff.md)) stays general enough to repackage into a JAR later without redesign. - **Platform matrix** — [ADR 0008](adr/0008-platform-matrix.md). @@ -89,6 +91,12 @@ The per-language link model, platform matrix, build/signing pipeline, ABI handsh - **Build, signing & CI** — [ADR 0009](adr/0009-build-and-signing-pipeline.md). - **Compatibility posture (major-version cutover, no parallel managed transport)** — [ADR 0007](adr/0007-native-is-opt-in.md). +### Alternative considered — a pure-Go port (spike) + +A different answer to "how does Go get the driver" is to **not consume the native binary at all** and instead **re-port the Rust driver into pure Go** — no cgo, no native artifact. A time-boxed spike built a working vertical slice this way (point read/create plus the retry, routing, and multi-region + transport-failure failover paths) with **`CGO_ENABLED=0` and zero third-party dependencies**, and validated it **behavior-for-behavior against the real Rust driver** with a differential harness: the same scenarios run through both implementations against one shared in-memory emulator, asserting the caller-visible outcomes match. + +This does **not** change the design, and the native + cgo model stands, because a pure-Go port **abandons build-once single provenance** ([ADR 0001](adr/0001-build-once-internal-handoff.md)): every language would re-implement and re-verify the driver, so "the driver" becomes N parallel, hand-maintained translations that must chase every change to the Rust core — precisely the drift this design exists to prevent. The spike's real value is as a **parity oracle and risk probe** for the port surface, not as a shipping vehicle; accordingly the pure-Go option stays *rejected for distribution* in [ADR 0004](adr/0004-go-cgo-prebuilt.md). + ## 7. Rollout | Phase | Distribution | Consumers | Adoption | @@ -104,7 +112,7 @@ The native transport is the defining change of the new major; there is no separa 1. **Internal hand-off shape** — Azure Artifacts Universal Package vs raw pipeline artifact for the one-build → three-jobs hand-off? 2. **Packaging ownership** — central fan-out (in azure-sdk-for-rust publishing into three language feeds) vs each SDK repo pulls the hand-off and publishes itself? (Security-boundary question.) -3. **Go delivery** — Universal Package fetched at build vs a vendored "binaries" Go module? +3. **Go delivery shape** — Universal Package fetched at build vs a vendored "binaries" Go module. *Not open: whichever shape wins, the native library is **packaged inside** the delivered artifact and linked via cgo — never a pure-Go shim that downloads the library separately.* 4. **Static vs dynamic default for Go** — self-contained `.a` vs shared `.so`? 5. **Version mapping** — *Resolved by [ADR 0010](adr/0010-native-version-fanout.md):* one native version is fanned out to all feeds simultaneously and each SDK pins that **exact** version (hosts accept only that one ABI revision, ADR 0005), rather than an independent per-language cadence off a pinned range. 6. **NuGet naming** — `Microsoft.Azure.Cosmos.NativeAssets.*` (reserved prefix) confirmed; meta-package name?