Add ChangeFeed AllVersionsAndDeletes mode by yumnahussain · Pull Request #4706 · Azure/azure-sdk-for-rust

yumnahussain · 2026-07-07T18:39:45Z

Adds the AllVersionsAndDeletes change feed mode.

What it adds

Driver — CosmosOperation::change_feed_all_versions_and_deletes factory and an all_versions_and_deletes header flag that emit A-IM: Full-Fidelity Feed (instead of Incremental Feed). Everything else on the wire is identical to LatestVersion. This mode always routes through Gateway 1.0 so the header is never dropped by a Gateway 2.0 downgrade.
SDK — ChangeFeedMode::AllVersionsAndDeletes. query_change_feed stays the single entry point and dispatches on the mode to pick the factory. In this mode the service returns every intermediate version plus deletes, so the ChangeFeedItem<T> envelope's previous() and metadata() (operation type, LSN, commit timestamp, previous-image LSN, TTL-expiry flag) are populated.
Start gating — Beginning and PointInTime are not supported for this mode; intermediate versions and deletes only exist within the container's retention window. The request is issued and the service returns 400 BadRequest. Use Now or resume from a continuation token.
Continuation token — the feed mode is now recorded in the token and validated on resume. Resuming a LatestVersion token as AllVersionsAndDeletes (or vice versa) is rejected with a 400. Only the marker is stored, so LatestVersion and query tokens keep their exact existing payload and legacy tokens are treated as LatestVersion.

Known limitations

Now is re-evaluated per range on first poll and is not converted to a concrete PointInTime; a range never polled before a checkpoint can miss changes between the original start and resume. Lossless per-range Now is a follow-up.

Tests

Iterator create/replace/delete envelope decoding; continuation-token mode encode/accept/reject unit tests; in-memory emulator end-to-end coverage (latest-version, envelope decoding, Beginning/PointInTime rejection); live emulator create/replace/delete, split/fan-out, and PointInTime rejection tests.

Two nits from the deep PR review of #4706: - cosmos_operation.rs: the ChangeFeedStartFrom::Now doc claimed AllVersionsAndDeletes *must* resolve Now to a concrete start before persisting, which contradicts the shipped, deliberately-documented per-range Now limitation in query_change_feed. Reworded to match. - change_feed.rs: all_versions_and_deletes_rejects_beginning only asserted is_err(). It now asserts the client-side rejection carries a 400 BadRequest, so a regression to a different status (or one that lets the request reach the emulator) fails the test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The change feed page iterator now deserializes each item into the caller's type directly instead of tolerantly unwrapping the wire-format `current` field. Callers bind `T = ChangeFeedItem<YourDoc>` on `query_change_feed` and read the post-change document via `current()`, so the whole wire envelope is preserved and passed up to the customer rather than stripped by the SDK. `ChangeFeedItem<T>` models the full envelope: `current`, `previous`, and `metadata` (with `ChangeFeedMetadata` / `ChangeFeedOperationType`). All fields beyond `current` are optional, so `LatestVersion` items (`{ current }` only) deserialize unchanged while the type is also ready for full-fidelity reads that populate `previous` / `metadata`. - Add `models::ChangeFeedItem<T>`, `ChangeFeedMetadata`, `ChangeFeedOperationType` with unit tests covering create/replace/delete envelopes and the metadata-absent LatestVersion shape. - Make `ChangeFeedPageIterator` parse-only (drop the unwrap helpers). - Update `query_change_feed` and iterator rustdoc/examples to the envelope contract. - Migrate the emulator and split change-feed integration tests to `ChangeFeedItem<MockItem>`. - CHANGELOG: note the envelope contract and the new types. This is a breaking change to the unreleased change feed API from #4621. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The real Cosmos emulator's LatestVersion change feed returns a metadata object carrying positional fields (lsn/crts) but no operationType. A required operation_type made ChangeFeedItem<T> fail to deserialize those responses (missing field \operationType\), which surfaced as six failing emulator_tests::cosmos_change_feed::* tests on the windows_stable CI job. Make operation_type Option<ChangeFeedOperationType> with #[serde(default)] so every envelope field is optional and a single type tolerantly serves both the LatestVersion and full-fidelity wire shapes. Accessors now return Option; add a regression unit test for the partial-metadata shape. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add an iterator-level test proving the parse-only iterator hands the whole envelope to the caller, including a partial \metadata\ object (lsn/crts, no operationType) that mirrors the real LatestVersion wire shape. Previously only the envelope type itself was tested with metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Stacks on the change-feed wire-envelope work (#4723) and reuses its shared `ChangeFeedItem<T>` type. Adds only the full-fidelity delta: - driver: `CosmosOperation::change_feed_all_versions_and_deletes` factory and a `full_fidelity_feed` header flag emitting `A-IM: Full-Fidelity Feed` in place of `Incremental Feed`. - driver in-memory emulator: parse the `a-im` header and synthesize a `create` envelope for full-fidelity reads. - SDK: `ChangeFeedMode::AllVersionsAndDeletes` and mode dispatch in `query_change_feed` selecting the factory; reject `ChangeFeedStartFrom::Beginning` (400 BadRequest) since intermediate versions and deletes only exist within the retention window. - Tests: iterator create/replace/delete envelope decoding plus emulator end-to-end coverage (latest-version, full-fidelity, Beginning rejection). CHANGELOG entries for both crates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The vnext Cosmos emulator does not honor the change feed wire-format header and returns bare documents instead of the `{ "current": ... }` envelope. The parse-only iterator deserialized those directly into `ChangeFeedItem<T>`, leaving `current` empty and panicking the emulator change feed tests. Give `ChangeFeedItem<T>` a tolerant `Deserialize`: an object carrying any reserved envelope key (`current`/`previous`/`metadata`) is parsed as the envelope; otherwise the whole document is mapped onto `current` so no data is lost on non-enveloping backends. Real Cosmos still envelopes, so `previous`/`metadata` are preserved there. Add unit tests for the flat shape and a delete envelope (which carries no `current`), and note the behavior in the changelog. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add the `ChangeFeedPolicy` model type and `ContainerProperties::with_change_feed_policy()` so containers can configure a full-fidelity retention window (`retentionDuration`, in minutes), which the service requires to serve `ChangeFeedMode::AllVersionsAndDeletes` reads. Add integration coverage for AVAD: - Emulator tests (emulator-only; the vnext Linux emulator does not support full fidelity): a create/replace/delete of one document surfaces as three envelopes with chained `previousImageLsn`, and creates fan out once each across multiple physical partitions. - A live split test that captures a `Now` token, forces a real partition split, then verifies post-split creates and deletes resume exactly once with no baseline replay. - Unit tests for `ChangeFeedPolicy` serialization and round-tripping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

yumnahussain · 2026-07-09T22:44:24Z

@copilot resolve the merge conflicts in this pull request

Co-authored-by: yumnahussain <161499581+yumnahussain@users.noreply.github.com>

Copilot · 2026-07-09T22:50:28Z

@copilot resolve the merge conflicts in this pull request

Resolved. I merged yumnahussain/cosmos-changefeed-envelope into this branch and fixed the conflict in sdk/cosmos/azure_data_cosmos/CHANGELOG.md in commit 5e3c326 (with a follow-up changelog readability tweak in c69739e).

yumnahussain · 2026-07-13T17:47:22Z

@copilot resolve the merge conflicts in this pull request

…nvelope' into yumnahussain/yumnahussain-cosmos-changefeed-avad # Conflicts: # sdk/cosmos/azure_data_cosmos/tests/in_memory_emulator_tests/mod.rs # sdk/cosmos/azure_data_cosmos_driver/src/in_memory_emulator/operations.rs Co-authored-by: yumnahussain <161499581+yumnahussain@users.noreply.github.com>

Copilot · 2026-07-13T17:57:28Z

@copilot resolve the merge conflicts in this pull request

Resolved the new merge conflicts and merged the updated stacked base branch into this PR in commit 4e30d25.

yumnahussain · 2026-07-27T18:20:26Z

/azp run rust - cosmos - weekly

azure-pipelines · 2026-07-27T18:21:01Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Bump the Cosmos account resource to the 2025-05-01-preview API version and set the top-level enableAllVersionsAndDeletesChangeFeed property, gated on enableContinuousBackup. This is required for the AVAD full-fidelity change-feed live tests, which were returning HTTP 400 at container creation because the account had no full-fidelity feature enabled. Only continuous-backup accounts opt in; periodic multi-write accounts stay unchanged (AVAD requires continuous backup and is incompatible with multiple write regions). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

yumnahussain · 2026-07-27T22:43:36Z

/azp run rust - cosmos - weekly

azure-pipelines · 2026-07-27T22:43:56Z

Azure Pipelines: Successfully started running 1 pipeline(s).

The enableAllVersionsAndDeletesChangeFeed account property cannot be set during account creation ('All Versions and Deletes ChangeFeed cannot be enabled during account creation', BadRequest), so setting it in the create template broke the Deploy test resources step on every continuous-backup job. Revert the API-version bump and the property to restore a working deploy. Enabling AVAD requires a post-create update, which needs a separate approach. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The AllVersionsAndDeletes (full-fidelity) change feed tests were gated on test_category "emulator", which is shared by the Docker emulator job and the live SingleWrite jobs, so they ran against non-AVAD accounts and returned 400. AVAD also cannot be enabled at account-create time. Add a dedicated Eventual + continuous-backup job that enables the feature after deployment and re-gate the three tests to run only there: - live-platform-matrix.json: add ubuntu-only "Eventual Avad" job (testCategory 'avad', continuous backup). - test-resources.bicep: emit AZURE_COSMOS_ENABLE_AVAD and AZURE_COSMOS_ACCOUNT_RESOURCE_ID so the post script can enable AVAD; no create-time flag. - test-resources-pre.ps1: register the AllVersionsAndDeletesChangeFeed subscription feature gate for the avad job. - test-resources-post.ps1: PATCH the account to enable AVAD and poll until active; no-op for other jobs. - cosmos_change_feed.rs: re-gate the three AVAD tests from 'emulator' to 'avad' so they stop 400ing in SingleWrite jobs. - build.rs: register "avad" as a valid test_category cfg. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

yumnahussain · 2026-07-27T23:48:39Z

/azp run rust - cosmos - weekly

azure-pipelines · 2026-07-27T23:48:58Z

Azure Pipelines: Successfully started running 1 pipeline(s).

This reverts commit 3f60cce. The AVAD-account provisioning approach was flawed: the pre-deployment script registered a Microsoft.DocumentDB feature ('AllVersionsAndDeletesChangeFeed') that does not exist, so it failed with NotFound and aborted the EventualAvad job's deploy. A split-target AVAD test was also left gated on 'split', so it kept failing against the non-AVAD Split account. Revert to the prior known state and plan a clean approach for a fixed AVAD-enabled account. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

sdk/cosmos/azure_data_cosmos_driver/src/driver/dataflow/planner.rs:270

A continuation captured before the first poll is still lossy here. At snapshot time no service request has established an ETag, so the token has an empty active_tokens set; priming only after resume sends If-None-Match: * at resume time and excludes every version/delete written in the gap. This cannot provide the lossless guarantee claimed by this comment and the new public docs. Prevent exporting/resuming an unprimed AVAD token, or eagerly establish and persist all per-range positions before a token can be captured.

    // versions/deletes in the gap. Priming is needed whenever no range has yet
    // recorded a continuation: on a fresh start, and also on resume from a
    // checkpoint taken *before* the first page was pulled (an empty token set).

sdk/cosmos/azure_data_cosmos/src/models/container_properties.rs:483

as_secs() truncates sub-second durations, so a nonzero value such as Duration::from_nanos(1) serializes as 0 and disables full-fidelity retention, contrary to this API's documented round-up behavior. Round the full duration, including nanoseconds, before converting to minutes.

    let minutes = value
        .map(|retention| retention.as_secs().div_ceil(60).min(i32::MAX as u64) as i32)
        .unwrap_or(0);

yumnahussain · 2026-07-28T17:48:09Z

@copilot resolve the merge conflicts in this pull request

Resolved conflicts in: - container_client.rs: merged BinaryEncodingOptions/OperationOptions imports from main - in_memory_emulator_tests/mod.rs: added binary_round_trip module from main - cosmos_headers.rs: kept both AVAD a-im tests and binary serialization format tests Co-authored-by: yumnahussain <161499581+yumnahussain@users.noreply.github.com>

Co-authored-by: yumnahussain <161499581+yumnahussain@users.noreply.github.com>

Copilot · 2026-07-28T17:56:04Z

@copilot resolve the merge conflicts in this pull request

Resolved the merge conflicts in 768c737:

container_client.rs: combined imports from both branches — BinaryEncodingOptions/OperationOptions from main alongside ChangeFeedMode from this branch
in_memory_emulator_tests/mod.rs: added the binary_round_trip module from main
cosmos_headers.rs: kept all tests from both sides — the AVAD a-im header tests from this branch and the binary serialization format header tests from main

Copilot

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

sdk/cosmos/azure_data_cosmos_driver/src/driver/dataflow/unordered_merge.rs:175

Clearing this flag before prime_children succeeds makes a failed priming pass non-recoverable. Some earlier children may already hold ETags while later children remain unpinned; that partial plan can still be snapshotted after the error, and because its token set is non-empty the resumed planner will skip priming, allowing the unpinned ranges to restart from resume-time Now and lose changes. Track per-child priming progress and only mark priming complete after every live range is pinned.

        if self.prime_on_first_drain {
            self.prime_on_first_drain = false;
            self.prime_children(context).await?;

sdk/cosmos/azure_data_cosmos_driver/src/driver/dataflow/unordered_merge.rs:123

This check disappears in release builds. Request::handle_change_feed_response deliberately leaves a child in its prior state when the response has no ETag, but priming then advances to the next child and reports success; a later checkpoint omits this range and it resumes from Now, violating the lossless guarantee. Treat a missing ETag as a returned service-contract error instead of a debug-only assertion.

                        debug_assert!(
                            response.headers().etag.is_some(),
                            "priming poll returned a page without an ETag; the range \
                             cannot record its start position and would resume from `Now`"
                        );

sdk/cosmos/azure_data_cosmos/src/clients/container_client.rs:941

This unconditional lossless-resume claim contradicts the PR's documented limitation and ChangeFeedStartFrom::Now's driver docs. A caller can capture a continuation before polling the first page; that token has no per-range positions, so resume evaluates Now later and misses changes written in the gap. Limit the guarantee to tokens captured after the first page has completed and document the pre-first-page gap.

    /// * When starting from [`ChangeFeedStartFrom::Now`], every range is pinned
    ///   to its concrete starting position before the first page is returned, so
    ///   a range that is never served before a checkpoint still resumes from its
    ///   true start rather than resume-time. No intermediate versions or deletes
    ///   are dropped across a resume. `Now` is deliberately **not** rewritten to

sdk/cosmos/azure_data_cosmos/src/models/container_properties.rs:483

A non-zero subsecond Duration serializes as 0: as_secs() truncates it before div_ceil, and durations such as 60.5 seconds also round down to one minute. This contradicts the public guarantee that every non-zero duration rounds up and can silently disable full-fidelity retention for subsecond inputs. Include subsec_nanos() when deciding whether to add the partial minute.

    let minutes = value
        .map(|retention| retention.as_secs().div_ceil(60).min(i32::MAX as u64) as i32)
        .unwrap_or(0);

github-actions Bot added the Cosmos The azure_cosmos crate label Jul 7, 2026

github-project-automation Bot added this to CosmosDB Rust SDK and Driver Jul 7, 2026

github-project-automation Bot moved this to Todo in CosmosDB Rust SDK and Driver Jul 7, 2026

yumnahussain force-pushed the yumnahussain/yumnahussain-cosmos-changefeed-avad branch from c2445af to 93a13de Compare July 7, 2026 19:06

Base automatically changed from yumnahussain/cosmos-changefeed-pull-api to main July 7, 2026 21:17

yumnahussain force-pushed the yumnahussain/yumnahussain-cosmos-changefeed-avad branch from 8a6b740 to 7ad4819 Compare July 8, 2026 22:19

yumnahussain and others added 6 commits July 8, 2026 15:52

Merge branch 'main' into yumnahussain/cosmos-changefeed-envelope

4fac43f

Potential fix for pull request finding

86abbd8

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

2e3e090

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

yumnahussain force-pushed the yumnahussain/yumnahussain-cosmos-changefeed-avad branch from 7ad4819 to d9e57bc Compare July 9, 2026 06:28

yumnahussain force-pushed the yumnahussain/yumnahussain-cosmos-changefeed-avad branch from d9e57bc to 2032bc1 Compare July 9, 2026 06:30

yumnahussain changed the base branch from main to yumnahussain/cosmos-changefeed-envelope July 9, 2026 06:30

yumnahussain and others added 2 commits July 9, 2026 08:18

Copilot started work on behalf of yumnahussain July 9, 2026 22:44 View session

Copilot AI and others added 2 commits July 9, 2026 22:48

Resolve stacked PR merge conflict

5e3c326

Co-authored-by: yumnahussain <161499581+yumnahussain@users.noreply.github.com>

Split long changelog entry

c69739e

Co-authored-by: yumnahussain <161499581+yumnahussain@users.noreply.github.com>

Copilot finished work on behalf of yumnahussain July 9, 2026 22:50

Merge branch 'main' into yumnahussain/cosmos-changefeed-envelope

e8cf6c0

Copilot started work on behalf of yumnahussain July 13, 2026 17:51 View session

This comment was marked as spam.

Sign in to view

analogrelay approved these changes Jul 27, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 27, 2026 22:39

Copilot started reviewing on behalf of yumnahussain July 27, 2026 22:39 View session

This comment was marked as spam.

Sign in to view

Copilot AI review requested due to automatic review settings July 27, 2026 23:18

Copilot started reviewing on behalf of yumnahussain July 27, 2026 23:19 View session

This comment was marked as spam.

Sign in to view

Copilot AI review requested due to automatic review settings July 27, 2026 23:47

Copilot started reviewing on behalf of yumnahussain July 27, 2026 23:47 View session

This comment was marked as spam.

Sign in to view

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Copilot AI and others added 2 commits July 28, 2026 17:55

Merge main into AVAD change feed branch

4663401

Co-authored-by: yumnahussain <161499581+yumnahussain@users.noreply.github.com>

Copilot AI reviewed Jul 28, 2026

View reviewed changes

tvaron3 approved these changes Jul 28, 2026

View reviewed changes

NaluTripician approved these changes Jul 28, 2026

View reviewed changes

NaluTripician mentioned this pull request Jul 28, 2026

Enable continuous backup for AllVersionsAndDeletes tests #4901

Open

Uh oh!

Conversation

yumnahussain commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it adds

Known limitations

Tests

Uh oh!

yumnahussain commented Jul 9, 2026

Uh oh!

Copilot AI commented Jul 9, 2026

Uh oh!

yumnahussain commented Jul 13, 2026

Uh oh!

Copilot AI commented Jul 13, 2026

Uh oh!

yumnahussain commented Jul 27, 2026

Uh oh!

azure-pipelines Bot commented Jul 27, 2026

Uh oh!

This comment was marked as spam.

Uh oh!

yumnahussain commented Jul 27, 2026

Uh oh!

azure-pipelines Bot commented Jul 27, 2026

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

yumnahussain commented Jul 27, 2026

Uh oh!

azure-pipelines Bot commented Jul 27, 2026

Uh oh!

This comment was marked as spam.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

yumnahussain commented Jul 28, 2026

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yumnahussain commented Jul 7, 2026 •

edited

Loading