Skip to content

Commit 71ef5ef

Browse files
TheovanKraayTheovanKraay
authored
Merge pull request #39 from AzureCosmosDB/copilot/add-idempotency-guidance-rule-9-1
Add Change Feed idempotency guidance to Rule 9.1
2 parents 8f8a4e0 + 31c1302 commit 71ef5ef

2 files changed

Lines changed: 195 additions & 3 deletions

File tree

‎skills/cosmosdb-best-practices/AGENTS.md‎

Lines changed: 97 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7799,16 +7799,112 @@ var query = ordersByStatusContainer.GetItemQueryIterator<OrderStatusView>(
77997799
| Better scalability | Eventual consistency (slight delay) |
78007800
| Lower RU cost per query | RU cost for writes to both containers |
78017801

7802+
**⚠️ Change Feed delivers events at-least-once.** Your handler MUST be idempotent — processing the same event twice must produce the same result. Never use `counter += 1` or `get() + 1` patterns in Change Feed handlers, as event replay will silently double-count.
7803+
7804+
**Incorrect — non-idempotent handler (counter drift on replay):**
7805+
7806+
```java
7807+
// ❌ WRONG — at-least-once replay doubles counts
7808+
private void handleChanges(List<JsonNode> changes, ChangeFeedProcessorContext context) {
7809+
for (JsonNode node : changes) {
7810+
GameScore score = objectMapper.treeToValue(node, GameScore.class);
7811+
PlayerProfile profile = playerRepository.findById(score.getPlayerId()).orElseGet(PlayerProfile::new);
7812+
profile.setTotalGamesPlayed(profile.getTotalGamesPlayed() + 1); // NON-IDEMPOTENT
7813+
profile.setTotalScore(profile.getTotalScore() + score.getScore()); // NON-IDEMPOTENT
7814+
playerRepository.save(profile);
7815+
}
7816+
}
7817+
```
7818+
7819+
```csharp
7820+
// ❌ WRONG — same problem in .NET
7821+
async Task HandleChangesAsync(IReadOnlyCollection<GameScore> changes, CancellationToken ct)
7822+
{
7823+
foreach (var score in changes)
7824+
{
7825+
var profile = await GetProfileAsync(score.PlayerId);
7826+
profile.TotalGamesPlayed += 1; // NON-IDEMPOTENT
7827+
profile.TotalScore += score.Score; // NON-IDEMPOTENT
7828+
await SaveProfileAsync(profile);
7829+
}
7830+
}
7831+
```
7832+
7833+
**Correct — idempotent alternatives:**
7834+
7835+
Use one of these patterns to ensure safe replay:
7836+
7837+
**1. Replace pattern — write absolute values, not deltas:**
7838+
7839+
```java
7840+
// ✅ CORRECT — replace with absolute value from the event
7841+
private void handleChanges(List<JsonNode> changes, ChangeFeedProcessorContext context) {
7842+
for (JsonNode node : changes) {
7843+
GameScore score = objectMapper.treeToValue(node, GameScore.class);
7844+
PlayerProfile profile = playerRepository.findById(score.getPlayerId()).orElseGet(PlayerProfile::new);
7845+
// Idempotent: same event replayed produces same result
7846+
profile.setHighScore(Math.max(profile.getHighScore(), score.getScore()));
7847+
playerRepository.save(profile);
7848+
}
7849+
}
7850+
```
7851+
7852+
**2. Conditional write — use ETags to detect duplicate processing:**
7853+
7854+
```csharp
7855+
// ✅ CORRECT — ETag prevents duplicate processing
7856+
async Task HandleChangesAsync(IReadOnlyCollection<GameScore> changes, CancellationToken ct)
7857+
{
7858+
foreach (var score in changes)
7859+
{
7860+
var response = await container.ReadItemAsync<PlayerProfile>(
7861+
score.PlayerId, new PartitionKey(score.PlayerId));
7862+
var profile = response.Resource;
7863+
profile.HighScore = Math.Max(profile.HighScore, score.Score);
7864+
await container.ReplaceItemAsync(profile, profile.Id,
7865+
new PartitionKey(profile.Id),
7866+
new ItemRequestOptions { IfMatchEtag = response.ETag });
7867+
}
7868+
}
7869+
```
7870+
7871+
**3. Mark-and-rebuild — flag affected records and recalculate from source of truth:**
7872+
7873+
```python
7874+
# ✅ CORRECT — mark dirty and rebuild from source data
7875+
async def handle_changes(changes):
7876+
for change in changes:
7877+
player_id = change["playerId"]
7878+
# Mark the profile as needing recalculation
7879+
await profiles_container.patch_item(
7880+
item=player_id,
7881+
partition_key=player_id,
7882+
patch_operations=[
7883+
{"op": "set", "path": "/needsRecalc", "value": True}
7884+
]
7885+
)
7886+
# Separate process recalculates from source of truth
7887+
```
7888+
7889+
| Idempotent Pattern | When to Use | Trade-off |
7890+
|--------------------|-------------|-----------|
7891+
| Replace (absolute value) | High scores, latest status, max/min values | Only works for non-cumulative data |
7892+
| Conditional write (ETag) | Any update where you can detect duplicates | Extra read + possible retry on conflict |
7893+
| Mark-and-rebuild | Counters, aggregations, cumulative totals | Higher latency, requires rebuild process |
7894+
78027895
**Key Points:**
7896+
- **Change Feed delivers at-least-once** — handlers MUST be idempotent
78037897
- Change Feed provides reliable, ordered event stream of all document changes
78047898
- Materialized views trade storage cost for query efficiency
78057899
- Updates are eventually consistent (typically <1 second delay)
78067900
- Use lease container to track processor progress (enables resume after failures)
7901+
- Never use `counter += 1`, `total += value`, or `get() + 1` patterns in Change Feed handlers
78077902
- Consider Azure Functions with Cosmos DB trigger for serverless implementation
7808-
- Consider Global Secondary Index (GSI) implementation as alternative for automatic sync between containers with different partition keys.
7903+
- Consider Global Secondary Index (GSI) implementation as alternative for automatic sync between containers with different partition keys
78097904

78107905
Reference(s):
78117906
[Change feed in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/change-feed)
7907+
[Change feed design patterns in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/nosql/change-feed-design-patterns)
78127908
[Global Secondary Indexes (GSI) in Azure Cosmos DB](https://learn.microsoft.com/en-us/azure/cosmos-db/global-secondary-indexes)
78137909

78147910
### 9.2 Use count-based or cached rank approaches instead of full partition scans for ranking

‎skills/cosmosdb-best-practices/rules/pattern-change-feed-materialized-views.md‎

Lines changed: 98 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: Use Change Feed for cross-partition query optimization with materialized views
33
impact: HIGH
44
impactDescription: eliminates cross-partition query overhead for admin/analytics scenarios
5-
tags: pattern, change-feed, materialized-views, cross-partition, query-optimization
5+
tags: pattern, change-feed, materialized-views, cross-partition, query-optimization, idempotency, at-least-once
66
---
77

88
## Use Change Feed for Materialized Views or Global Secondary Index
@@ -186,14 +186,110 @@ var query = ordersByStatusContainer.GetItemQueryIterator<OrderStatusView>(
186186
| Better scalability | Eventual consistency (slight delay) |
187187
| Lower RU cost per query | RU cost for writes to both containers |
188188

189+
**⚠️ Change Feed delivers events at-least-once.** Your handler MUST be idempotent — processing the same event twice must produce the same result. Never use `counter += 1` or `get() + 1` patterns in Change Feed handlers, as event replay will silently double-count.
190+
191+
**Incorrect — non-idempotent handler (counter drift on replay):**
192+
193+
```java
194+
// ❌ WRONG — at-least-once replay doubles counts
195+
private void handleChanges(List<JsonNode> changes, ChangeFeedProcessorContext context) {
196+
for (JsonNode node : changes) {
197+
GameScore score = objectMapper.treeToValue(node, GameScore.class);
198+
PlayerProfile profile = playerRepository.findById(score.getPlayerId()).orElseGet(PlayerProfile::new);
199+
profile.setTotalGamesPlayed(profile.getTotalGamesPlayed() + 1); // NON-IDEMPOTENT
200+
profile.setTotalScore(profile.getTotalScore() + score.getScore()); // NON-IDEMPOTENT
201+
playerRepository.save(profile);
202+
}
203+
}
204+
```
205+
206+
```csharp
207+
// ❌ WRONG — same problem in .NET
208+
async Task HandleChangesAsync(IReadOnlyCollection<GameScore> changes, CancellationToken ct)
209+
{
210+
foreach (var score in changes)
211+
{
212+
var profile = await GetProfileAsync(score.PlayerId);
213+
profile.TotalGamesPlayed += 1; // NON-IDEMPOTENT
214+
profile.TotalScore += score.Score; // NON-IDEMPOTENT
215+
await SaveProfileAsync(profile);
216+
}
217+
}
218+
```
219+
220+
**Correct — idempotent alternatives:**
221+
222+
Use one of these patterns to ensure safe replay:
223+
224+
**1. Replace pattern — write absolute values, not deltas:**
225+
226+
```java
227+
// ✅ CORRECT — replace with absolute value from the event
228+
private void handleChanges(List<JsonNode> changes, ChangeFeedProcessorContext context) {
229+
for (JsonNode node : changes) {
230+
GameScore score = objectMapper.treeToValue(node, GameScore.class);
231+
PlayerProfile profile = playerRepository.findById(score.getPlayerId()).orElseGet(PlayerProfile::new);
232+
// Idempotent: same event replayed produces same result
233+
profile.setHighScore(Math.max(profile.getHighScore(), score.getScore()));
234+
playerRepository.save(profile);
235+
}
236+
}
237+
```
238+
239+
**2. Conditional write — use ETags to detect duplicate processing:**
240+
241+
```csharp
242+
// ✅ CORRECT — ETag prevents duplicate processing
243+
async Task HandleChangesAsync(IReadOnlyCollection<GameScore> changes, CancellationToken ct)
244+
{
245+
foreach (var score in changes)
246+
{
247+
var response = await container.ReadItemAsync<PlayerProfile>(
248+
score.PlayerId, new PartitionKey(score.PlayerId));
249+
var profile = response.Resource;
250+
profile.HighScore = Math.Max(profile.HighScore, score.Score);
251+
await container.ReplaceItemAsync(profile, profile.Id,
252+
new PartitionKey(profile.Id),
253+
new ItemRequestOptions { IfMatchEtag = response.ETag });
254+
}
255+
}
256+
```
257+
258+
**3. Mark-and-rebuild — flag affected records and recalculate from source of truth:**
259+
260+
```python
261+
# ✅ CORRECT — mark dirty and rebuild from source data
262+
async def handle_changes(changes):
263+
for change in changes:
264+
player_id = change["playerId"]
265+
# Mark the profile as needing recalculation
266+
await profiles_container.patch_item(
267+
item=player_id,
268+
partition_key=player_id,
269+
patch_operations=[
270+
{"op": "set", "path": "/needsRecalc", "value": True}
271+
]
272+
)
273+
# Separate process recalculates from source of truth
274+
```
275+
276+
| Idempotent Pattern | When to Use | Trade-off |
277+
|--------------------|-------------|-----------|
278+
| Replace (absolute value) | High scores, latest status, max/min values | Only works for non-cumulative data |
279+
| Conditional write (ETag) | Any update where you can detect duplicates | Extra read + possible retry on conflict |
280+
| Mark-and-rebuild | Counters, aggregations, cumulative totals | Higher latency, requires rebuild process |
281+
189282
**Key Points:**
283+
- **Change Feed delivers at-least-once** — handlers MUST be idempotent
190284
- Change Feed provides reliable, ordered event stream of all document changes
191285
- Materialized views trade storage cost for query efficiency
192286
- Updates are eventually consistent (typically <1 second delay)
193287
- Use lease container to track processor progress (enables resume after failures)
288+
- Never use `counter += 1`, `total += value`, or `get() + 1` patterns in Change Feed handlers
194289
- Consider Azure Functions with Cosmos DB trigger for serverless implementation
195-
- Consider Global Secondary Index (GSI) implementation as alternative for automatic sync between containers with different partition keys.
290+
- Consider Global Secondary Index (GSI) implementation as alternative for automatic sync between containers with different partition keys
196291

197292
Reference(s):
198293
[Change feed in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/change-feed)
294+
[Change feed design patterns in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/nosql/change-feed-design-patterns)
199295
[Global Secondary Indexes (GSI) in Azure Cosmos DB](https://learn.microsoft.com/en-us/azure/cosmos-db/global-secondary-indexes)

0 commit comments

Comments
 (0)