Add ChangeFeed AllVersionsAndDeletes mode

[yumnahussain]

Adds the AllVersionsAndDeletes change feed mode.

What it adds

• Driver — CosmosOperation::change_feed_all_versions_and_deletes factory and an all_versions_and_deletes header flag that emit A-IM: Full-Fidelity Feed (instead of Incremental Feed). Everything else on the wire is identical to LatestVersion. This mode always routes through Gateway 1.0 so the header is never dropped by a Gateway 2.0 downgrade.
• SDK — ChangeFeedMode::AllVersionsAndDeletes. query_change_feed stays the single entry point and dispatches on the mode to pick the factory. In this mode the service returns every intermediate version plus deletes, so the ChangeFeedItem<T> envelope's previous() and metadata() (operation type, LSN, commit timestamp, previous-image LSN, TTL-expiry flag) are populated.
• Start gating — Beginning and PointInTime are not supported for this mode; intermediate versions and deletes only exist within the container's retention window. The request is issued and the service returns 400 BadRequest. Use Now or resume from a continuation token.
• Continuation token — the feed mode is now recorded in the token and validated on resume. Resuming a LatestVersion token as AllVersionsAndDeletes (or vice versa) is rejected with a 400. Only the marker is stored, so LatestVersion and query tokens keep their exact existing payload and legacy tokens are treated as LatestVersion.

Known limitations

• Now is re-evaluated per range on first poll and is not converted to a concrete PointInTime; a range never polled before a checkpoint can miss changes between the original start and resume. Lossless per-range Now is a follow-up.

Tests

Iterator create/replace/delete envelope decoding; continuation-token mode encode/accept/reject unit tests; in-memory emulator end-to-end coverage (latest-version, envelope decoding, Beginning/PointInTime rejection); live emulator create/replace/delete, split/fan-out, and PointInTime rejection tests.

[yumnahussain]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Resolved. I merged yumnahussain/cosmos-changefeed-envelope into this branch and fixed the conflict in sdk/cosmos/azure_data_cosmos/CHANGELOG.md in commit 5e3c326 (with a follow-up changelog readability tweak in c69739e).

[yumnahussain]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Resolved the new merge conflicts and merged the updated stacked base branch into this PR in commit 4e30d25.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 13 comments.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

sdk/cosmos/azure_data_cosmos/src/models/container_properties.rs:470

• This constructor can silently disable the mode it promises to enable: Duration::from_secs(59) becomes retentionDuration: 0, while 90 seconds is shortened to 60 seconds. Since the wire format only supports whole minutes, make this boundary fallible and reject zero or non-whole-minute durations instead of changing the caller's requested policy.

    pub fn all_versions_and_deletes(retention: Duration) -> Self {
        Self {
            retention_duration_minutes: (retention.as_secs() / 60).min(i32::MAX as u64) as i32,

sdk/cosmos/azure_data_cosmos/src/clients/container_client.rs:897

• This contradicts the PR description, which tells users to use an in-retention PointInTime for this mode. The implementation and new emulator tests reject that start instead. Please resolve which contract is intended and align the code, tests, changelog, API docs, and PR description so reviewers and users are not given opposite behavior.

    /// * Only [`ChangeFeedStartFrom::Now`] or resuming from a continuation token
    ///   is supported. [`ChangeFeedStartFrom::Beginning`] and
    ///   [`ChangeFeedStartFrom::PointInTime`] are **not** supported, because
    ///   intermediate versions and deletes are only retained within the
    ///   container's retention / continuous-backup window. The service gates
    ///   this and rejects such a request with `400 Bad Request`.

sdk/cosmos/azure_data_cosmos/tests/split_tests/cosmos_change_feed_split.rs:435

• These lines are not rustfmt-formatted, so the repository's mandatory formatting check will fail. Run cargo fmt -p azure_data_cosmos over the complete change; the same formatter will also wrap the other newly added long expressions in this test.

got_creates.sort();
expected_creates.sort();

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

sdk/cosmos/azure_data_cosmos/tests/emulator_tests/cosmos_change_feed.rs:895

• A single next() only polls one physical range: UnorderedMerge::next_page advances one child per page (unordered_merge.rs:24-26,78-104). Since this test deliberately creates 2+ ranges, writes routed to unpolled ranges establish their Now position only after the writes and can be omitted, making the fan-out test time out/fail. Poll once per current feed range before writing.

            // Prime the per-range `Now` positions before writing.
            let primed = iterator
                .next()
                .await
                .expect("change feed stream always yields a page")?;

sdk/cosmos/azure_data_cosmos/src/models/container_properties.rs:446

• The stated 1-hour minimum is inaccurate: the official Azure Cosmos DB .NET ChangeFeedPolicy documentation configures a 5-minute full-fidelity retention window, and the wire setting has minute granularity. This public documentation would incorrectly discourage supported shorter windows; avoid hard-coding an unverified range since validation is intentionally delegated to the service.

/// The retention window has minute granularity. On the service it must fall
/// within the supported range (currently 1 hour to 30 days) when this mode
/// is enabled; this type does not enforce that range, leaving the service as the
/// source of truth.

sdk/cosmos/azure_data_cosmos/tests/split_tests/cosmos_change_feed_split.rs:435

• These added statements are not rustfmt-indented, so the repository's mandatory cargo fmt --check will fail. Run cargo fmt -p azure_data_cosmos before merging.

got_creates.sort();
expected_creates.sort();

[NaluTripician]

Local deep review of the AllVersionsAndDeletes change feed mode. Overall this is a high-quality, unusually well-tested PR — the two dangerous wire paths (Gateway 2.0 A-IM downgrade protection, and continuation-token mode binding) are both handled correctly and tested. No blocking wire/correctness bugs.

The inline notes below are, in priority order: one High (lossy Now resume undercuts AVAD's core delete guarantee), three Medium (client-side fail-fast, sub-minute retention truncation, preview-gating), two Low, and a fmt nit. The top ask before "done" is the lossy Now resume.

[NaluTripician]

This Now re-evaluation is the one thing I'd want to resolve (or surface much louder) before this ships stable.

For LatestVersion, re-evaluating Now per range at first-poll is benign — the feed still converges to latest state under at-least-once delivery. For AllVersionsAndDeletes it's permanent data loss: intermediate versions and deletes only exist inside the retention window, so a range that's never polled before a checkpoint resumes from resume-time and silently drops every delete/version in that gap. It's reachable on the normal fan-out path (UnorderedMerge round-robins pages, so a full_container feed checkpointed after a few pages leaves some ranges holding an unresolved Now). Missing deletes is exactly the guarantee AVAD exists to provide, so an audit/cache-invalidation consumer can silently miss deletions with no error.

.NET/Java pin Now to a concrete per-range continuation (LSN/etag) at first poll so full-fidelity resume is lossless. Options, roughly in order of preference:

1. For AVAD, resolve Now to a concrete per-range start before persisting the token.
2. If that stays a follow-up, add a test that documents the current lossy behavior (a regression anchor for the future fix) and repeat this caveat directly on the AllVersionsAndDeletes enum variant, not just here.

[yumnahussain]

Fixed. Fresh AllVersionsAndDeletes feeds now pin every range to a concrete starting ETag before the first page is returned (prime-on-first-drain in UnorderedMerge, gated by the planner on full_fidelity_feed && !is_resume). The first poll of each range is a start-from-Now 304 that still carries an ETag, so each range records its true start even if it is never served before a checkpoint. The snapshot then holds a per-range token for every range, so resume can't fall back to a resume-time Now and drop the versions/deletes in the gap. Now keeps its semantics (it isn't rewritten to a concrete PointInTime). Covered by driver priming unit tests plus an integration test (all_versions_and_deletes_pins_every_range_before_checkpoint) asserting both ranges are pinned after a single served page and both resume from their pinned ETags; a companion test pins the LatestVersion no-prime path.

[NaluTripician]

Deferring Beginning/PointInTime rejection entirely to the service means query_change_feed(...).await? succeeds and the 400 only surfaces on the first pages.next().await — often far from the call site that made the mistake. .NET/Java reject this combination client-side at build time with an immediate, offline, self-describing error.

Suggest failing fast here when mode == AllVersionsAndDeletes and start is Beginning/PointInTime with no continuation token, returning a client-side error ("use Now or resume from a continuation token"). It's cheap, matches the peer SDKs, and doesn't depend on the server (or the emulator stand-in) gating identically in every case. If server-authoritative gating is a deliberate choice, worth a one-line note on why.

[tvaron3]

java and .net do this on client but it also means that once support for these is added it requires an sdk upgrade. Point in time for avad is something already being worked on by the service.

[yumnahussain]

Leaving start-position validation server-authoritative here rather than adding a client-side fail-fast. tvaron asked earlier in this review to drop the client-side guard, and the service/emulator already rejects unsupported starts (Beginning/PointInTime under AVAD) with 400. Duplicating that check on the client risks drifting from the service contract as retention/continuous-backup windows evolve. I've kept the why note in code next to the start handling so the choice is explicit.

[NaluTripician]

Sub-minute durations silently truncate to 0 here — and per this struct's own docs 0 disables all-versions-and-deletes. So ChangeFeedPolicy::all_versions_and_deletes(Duration::from_secs(30)) returns a policy that disables the very mode the method name promises to enable, with no error. A caller who confuses seconds/minutes (or passes from_secs(90) thinking "1.5 min") gets silent truncation, possibly below the service minimum or to a disabling value. The change_feed_policy_truncates_sub_minute_retention test currently enshrines the 90s→1min behavior.

Prefer rounding up to a 1-minute floor for any non-zero duration, or rejecting a duration that truncates to 0 (or below the documented service minimum), rather than silently producing a disabling policy.

[yumnahussain]

Fixed. all_versions_and_deletes(Duration) now rounds any non-zero duration up to the 1-minute floor (.as_secs().div_ceil(60)) instead of truncating sub-minute values to 0, which previously disabled the mode. Test renamed to change_feed_policy_rounds_sub_minute_retention_up (30s->1min, 90s->2min, 0s->0).

[NaluTripician]

Minor: exposing retention_duration_minutes as a raw public i32 alongside the typed retention_duration() accessor is a bit of a footgun. A caller can bypass the constructor and assign a negative/out-of-range value directly; retention_duration() papers over negatives with .max(0), but serialization still sends the raw negative to the service. #[non_exhaustive] blocks external struct-literal construction (good), but the field stays directly assignable on an existing value. Consider a private field with a getter/with_* setter, or clamp/validate on serialize.

[yumnahussain]

Fixed. retention_duration_minutes is now private with a retention_duration_minutes() getter, and serialization clamps negatives to 0 (serialize_with) so an invalid value can never reach the wire. The public constructor and retention_duration() are unchanged. Added change_feed_policy_clamps_negative_retention_on_serialize.

[NaluTripician]

Design question: should AllVersionsAndDeletes ship behind a preview feature (like preview_dtx in this same CHANGELOG) rather than going straight onto the stable surface? It's landing with a documented data-loss gap (the lossy Now resume) and a documented emulator gap (no delete/pre-image replay). Once this variant is stable, its lossy-Now semantics effectively become a compatibility contract. Preview-gating until lossless per-range Now lands would avoid users building on a mode that can silently miss deletes.

[yumnahussain]

Keeping AllVersionsAndDeletes on the stable surface (no preview flag). The main concern behind gating was the lossy-Now resume gap, which is now closed by the per-range pinning fix above — a resumed feed no longer drops versions/deletes. The mode is also encoded in the continuation token, so a token can't be silently resumed in the wrong mode. Happy to revisit if you'd still prefer a preview gate.

[NaluTripician]

Nice thorough envelope test. One small gap: it asserts operation_type, lsn, previous_image_lsn, and time_to_live_expired, but not conflict_resolution_timestamp (crts) — even though crts is part of the AVAD metadata contract and the in-memory emulator synthesizes it. Covered indirectly by the existing create-envelope test, so minor, but asserting crts here would fully lock down the AVAD metadata shape.

[yumnahussain]

Fixed. Added crts to the create entry of the envelope in deserializes_all_versions_and_deletes_page and assert metadata().conflict_resolution_timestamp() deserializes to the expected Duration.

[NaluTripician]

Nit: these two lines (got_creates.sort(); / expected_creates.sort();) are at column 0 while the surrounding block is indented — cargo fmt --check should flag this. Worth confirming the fmt CI gate is actually green on this file.

[yumnahussain]

Fixed. Re-indented the two sort() lines to match the surrounding block. (Note: cargo fmt --check doesn't flag these, so it slipped past the gate — corrected manually.)

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

sdk/cosmos/azure_data_cosmos/src/models/container_properties.rs:489

• Duration::as_secs() truncates sub-second values, so a nonzero retention such as Duration::from_nanos(1) serializes as 0 and disables AllVersionsAndDeletes, contrary to this method's stated round-up contract. Include subsec_nanos() when deciding whether a partial minute remains.

            retention_duration_minutes: retention.as_secs().div_ceil(60).min(i32::MAX as u64)
                as i32,