Skip to content

Commit f79b879

Browse files
committed
fix: skip .lte() dedup check on Cosmos DB to prevent data loss
On Cosmos DB, wallTime has second precision (increment 0). After a restart, the .lte() check at line 1064 compares the new event timestamp against startAfter (from the last checkpoint). If events arrive within the same wall-clock second as the last checkpoint, their timestamps equal startAfter and .lte() returns true — silently dropping the events. On standard MongoDB, clusterTime has monotonically increasing increments within a second, so new events always have strictly greater timestamps. The .lte() check is a client-side dedup guard only needed for the legacy startAtOperationTime path. When resumeAfter is used (which includes Cosmos DB), the server guarantees no duplicates. The resume test still has an intermittent failure (~30% flake rate) on Cosmos DB, which appears to be a separate timing issue in the test utility getClientCheckpoint — not a production bug.
1 parent 83be468 commit f79b879

2 files changed

Lines changed: 23 additions & 1 deletion

File tree

modules/module-mongodb/src/replication/ChangeStream.ts

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1061,7 +1061,22 @@ export class ChangeStream {
10611061
this.touch();
10621062

10631063
try {
1064-
if (startAfter != null && this.getEventTimestamp(originalChangeDocument).lte(startAfter)) {
1064+
// Skip events at or before the resume point to prevent duplicate processing.
1065+
// This guard is only needed for the legacy startAtOperationTime path where
1066+
// the server may redeliver events at the boundary. When resumeAfter is used
1067+
// (which includes Cosmos DB), the server guarantees no duplicates.
1068+
// On Cosmos DB, this check is harmful: wallTime has second precision
1069+
// (increment 0), so events within the same second as the last checkpoint
1070+
// would be incorrectly dropped — causing silent data loss after restart.
1071+
//
1072+
// WARNING: No dedicated test covers the isCosmosDb guard here. The bug
1073+
// requires data events to arrive within the same wall-clock second as the
1074+
// last checkpoint after a restart — a timing condition that's difficult to
1075+
// reproduce reliably in tests. The "resume after restart" integration test
1076+
// exercises this path but is flaky due to a separate getClientCheckpoint
1077+
// polling issue. If refactoring this code, verify manually that events
1078+
// are not dropped on Cosmos DB after restart.
1079+
if (!this.isCosmosDb && startAfter != null && this.getEventTimestamp(originalChangeDocument).lte(startAfter)) {
10651080
continue;
10661081
}
10671082
} catch {

modules/module-mongodb/test/src/cosmosdb_mode.test.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,13 @@ bucket_definitions:
216216

217217
context2.startStreaming();
218218

219+
// Wait for the stream to initialize and process the initial checkpoint.
220+
// On Cosmos DB, wall-clock LSNs have second precision — if the insert
221+
// happens in the same second as the last checkpoint, getClientCheckpoint
222+
// can resolve before the data event is committed. A brief delay ensures
223+
// the next insert falls in a new second.
224+
await setTimeout(1100);
225+
219226
const collection2 = db2.collection('test_data');
220227
const result2 = await collection2.insertOne({ description: 'after_restart' });
221228
const id2 = result2.insertedId;

0 commit comments

Comments
 (0)