Skip to content

Commit efde457

Browse files
dwdoughertydwdougherty
authored
DEV: (cmds+) updates for streams idempotent production (#2637)
* DOC-6151: (DEV) update XADD for idempotent production (commit 1) * Flesh out content * Apply review suggestions
1 parent 84236f6 commit efde457

9 files changed

Lines changed: 576 additions & 81 deletions

File tree

‎content/commands/xadd.md‎

Lines changed: 84 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -4,46 +4,76 @@ acl_categories:
44
- '@stream'
55
- '@fast'
66
arguments:
7-
- display_text: key
8-
key_spec_index: 0
7+
- key_spec_index: 0
98
name: key
109
type: key
11-
- display_text: nomkstream
12-
name: nomkstream
10+
- name: nomkstream
1311
optional: true
1412
since: 6.2.0
1513
token: NOMKSTREAM
1614
type: pure-token
1715
- arguments:
16+
- name: keepref
17+
token: KEEPREF
18+
type: pure-token
19+
- name: delref
20+
token: DELREF
21+
type: pure-token
22+
- name: acked
23+
token: ACKED
24+
type: pure-token
25+
name: condition
26+
optional: true
27+
type: oneof
28+
- arguments:
29+
- arguments:
30+
- name: idmpauto-token
31+
token: IDMPAUTO
32+
type: pure-token
33+
- display_text: producer-id
34+
name: pid
35+
type: string
36+
name: idmpauto-with-pid
37+
type: block
1838
- arguments:
19-
- display_text: maxlen
20-
name: maxlen
39+
- name: idmp-token
40+
token: IDMP
41+
type: pure-token
42+
- display_text: producer-id
43+
name: pid
44+
type: string
45+
- display_text: idempotent-id
46+
name: iid
47+
type: string
48+
name: idmp-with-pid-iid
49+
type: block
50+
name: idmp
51+
optional: true
52+
type: oneof
53+
- arguments:
54+
- arguments:
55+
- name: maxlen
2156
token: MAXLEN
2257
type: pure-token
23-
- display_text: minid
24-
name: minid
58+
- name: minid
2559
since: 6.2.0
2660
token: MINID
2761
type: pure-token
2862
name: strategy
2963
type: oneof
3064
- arguments:
31-
- display_text: equal
32-
name: equal
65+
- name: equal
3366
token: '='
3467
type: pure-token
35-
- display_text: approximately
36-
name: approximately
68+
- name: approximately
3769
token: '~'
3870
type: pure-token
3971
name: operator
4072
optional: true
4173
type: oneof
42-
- display_text: threshold
43-
name: threshold
74+
- name: threshold
4475
type: string
45-
- display_text: count
46-
name: count
76+
- name: count
4777
optional: true
4878
since: 6.2.0
4979
token: LIMIT
@@ -52,21 +82,17 @@ arguments:
5282
optional: true
5383
type: block
5484
- arguments:
55-
- display_text: auto-id
56-
name: auto-id
85+
- name: auto-id
5786
token: '*'
5887
type: pure-token
59-
- display_text: id
60-
name: id
88+
- name: id
6189
type: string
6290
name: id-selector
6391
type: oneof
6492
- arguments:
65-
- display_text: field
66-
name: field
93+
- name: field
6794
type: string
68-
- display_text: value
69-
name: value
95+
- name: value
7096
type: string
7197
multiple: true
7298
name: data
@@ -116,8 +142,10 @@ linkTitle: XADD
116142
railroad_diagram: /images/railroad/xadd.svg
117143
since: 5.0.0
118144
summary: Appends a new message to a stream. Creates the key if it doesn't exist.
119-
syntax_fmt: "XADD key [NOMKSTREAM] [KEEPREF | DELREF | ACKED] [<MAXLEN | MINID> [= | ~] threshold\n [LIMIT\_\
120-
count]] <* | id> field value [field value ...]"
145+
syntax_fmt: "XADD key [NOMKSTREAM] [KEEPREF | DELREF | ACKED]\n \
146+
\ [IDMPAUTO producer-id | IDMP producer-id idempotent-id]\n \
147+
\ [<MAXLEN | MINID> [= | ~] threshold [LIMIT\_count]] <* | id>\n \
148+
\ field value [field value ...]"
121149
title: XADD
122150
---
123151

@@ -153,6 +181,24 @@ One or more field-value pairs that make up the stream entry. You must provide at
153181
Prevents the creation of a new stream if the key does not exist. Available since Redis 6.2.0.
154182
</details>
155183

184+
<details open>
185+
<summary><code>IDMPAUTO producer-id | IDMP producer-id idempotent-id</code></summary>
186+
187+
Enables idempotent message processing (at-most-once production) to prevent duplicate entries. Available since Redis 8.6.
188+
189+
- `IDMPAUTO producer-id`: Automatically generates a unique idempotent ID (iid) for the specified producer-id. Redis tracks this iid to prevent duplicate messages from the same producer-id.
190+
- `IDMP producer-id idempotent-id`: Uses the specified idempotent-id for the given producer-id. If this producer-id/idempotent-id combination was already used, the command returns the ID of the existing entry instead of creating a duplicate.
191+
192+
The producer-id identifies the source of the message, while the idempotent-id ensures uniqueness within that producer-id's message stream. Redis maintains an internal map of recent producer-id/idempotent-id combinations to detect and prevent duplicates.
193+
194+
Both modes can only be specified when the entry ID is `*` (auto-generated).
195+
196+
Use [`XCFGSET`]({{< relref "/commands/xcfgset" >}}) to configure how long idempotent IDs are retained (`IDMP-DURATION`) and the maximum number tracked per producer (`IDMP-MAXSIZE`).
197+
198+
See [Idempotent message processing]({{< relref "/develop/data-types/streams/idempotency" >}}) for more information.
199+
200+
</details>
201+
156202
<details open>
157203
<summary><code>KEEPREF | DELREF | ACKED</code></summary>
158204

@@ -267,6 +313,16 @@ XLEN mystream
267313
XRANGE mystream - +
268314
{{% /redis-cli %}}
269315

316+
### Idempotent message processing examples
317+
318+
{{% redis-cli %}}
319+
XADD mystream IDMP producer1 msg1 * field value
320+
XADD mystream IDMP producer1 msg1 * field different_value
321+
XADD mystream IDMPAUTO producer2 * field value
322+
XADD mystream IDMPAUTO producer2 * field value
323+
XCFGSET mystream IDMP-DURATION 300 IDMP-MAXSIZE 1000
324+
{{% /redis-cli %}}
325+
270326
## Redis Enterprise and Redis Cloud compatibility
271327

272328
| Redis<br />Enterprise | Redis<br />Cloud | <span style="min-width: 9em; display: table-cell">Notes</span> |
@@ -280,13 +336,13 @@ XRANGE mystream - +
280336
tab2="RESP3" >}}
281337

282338
One of the following:
283-
* [Bulk string reply](../../develop/reference/protocol-spec#bulk-strings): The ID of the added entry. The ID is the one automatically generated if an asterisk (`*`) is passed as the _id_ argument, otherwise the command just returns the same ID specified by the user during insertion.
339+
* [Bulk string reply](../../develop/reference/protocol-spec#bulk-strings): The ID of the added entry. The ID is the one automatically generated if an asterisk (`*`) is passed as the _id_ argument, otherwise the command just returns the same ID specified by the user during insertion. When using IDMP and a duplicate is detected, returns the ID of the existing entry.
284340
* [Nil reply](../../develop/reference/protocol-spec#bulk-strings): if the NOMKSTREAM option is given and the key doesn't exist.
285341

286342
-tab-sep-
287343

288344
One of the following:
289-
* [Bulk string reply](../../develop/reference/protocol-spec#bulk-strings): The ID of the added entry. The ID is the one automatically generated if an asterisk (`*`) is passed as the _id_ argument, otherwise the command just returns the same ID specified by the user during insertion.
345+
* [Bulk string reply](../../develop/reference/protocol-spec#bulk-strings): The ID of the added entry. The ID is the one automatically generated if an asterisk (`*`) is passed as the _id_ argument, otherwise the command just returns the same ID specified by the user during insertion. When using IDMP and a duplicate is detected, returns the ID of the existing entry.
290346
* [Null reply](../../develop/reference/protocol-spec#nulls): if the NOMKSTREAM option is given and the key doesn't exist.
291347

292348
{{< /multitabs >}}

‎content/commands/xcfgset.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
---
2+
acl_categories:
3+
- STREAM
4+
arguments:
5+
- key_spec_index: 0
6+
name: key
7+
type: key
8+
- arguments:
9+
- name: duration-token
10+
token: IDMP-DURATION
11+
type: pure-token
12+
- name: duration
13+
type: integer
14+
name: duration-block
15+
optional: true
16+
type: block
17+
- arguments:
18+
- name: maxsize-token
19+
token: IDMP-MAXSIZE
20+
type: pure-token
21+
- name: maxsize
22+
type: integer
23+
name: maxsize-block
24+
optional: true
25+
type: block
26+
arity: -2
27+
categories:
28+
- docs
29+
- develop
30+
- stack
31+
- oss
32+
- rs
33+
- rc
34+
- oss
35+
- kubernetes
36+
- clients
37+
command_flags:
38+
- WRITE
39+
- FAST
40+
complexity: O(1)
41+
description: Sets the IDMP configuration parameters for a stream.
42+
function: xcfgsetCommand
43+
group: stream
44+
hidden: false
45+
key_specs:
46+
- begin_search:
47+
index:
48+
pos: 1
49+
find_keys:
50+
range:
51+
lastkey: 0
52+
limit: 0
53+
step: 1
54+
flags:
55+
- RW
56+
- UPDATE
57+
linkTitle: XCFGSET
58+
reply_schema:
59+
const: OK
60+
since: 8.6.0
61+
summary: Sets the IDMP configuration parameters for a stream.
62+
syntax_fmt: XCFGSET key [IDMP-DURATION duration] [IDMP-MAXSIZE maxsize]
63+
title: XCFGSET
64+
---
65+
Sets the IDMP (Idempotent Message Processing) configuration parameters for a stream. This command configures how long idempotent IDs are retained and the maximum number of idempotent IDs tracked per producer.
66+
67+
## Required arguments
68+
69+
<details open><summary><code>key</code></summary>
70+
71+
The name of the stream key. The stream must already exist.
72+
73+
</details>
74+
75+
## Optional arguments
76+
77+
<details open><summary><code>IDMP-DURATION duration</code></summary>
78+
79+
Sets the duration in seconds that each idempotent ID (iid) is kept in the stream's IDMP map. Valid range: 1-86,400 seconds. Default: 100 seconds.
80+
81+
When an idempotent ID expires, it can be reused for new messages. This provides an operational guarantee that Redis will not forget an idempotency ID before the duration elapses (unless capacity is reached).
82+
83+
</details>
84+
85+
<details open><summary><code>IDMP-MAXSIZE maxsize</code></summary>
86+
87+
Sets the maximum number of most recent idempotent IDs kept for each producer in the stream's IDMP map. Valid range: 1-10,000 entries. Default: 100 entries.
88+
89+
When the capacity is reached, the oldest idempotent IDs for that producer are evicted regardless of remaining duration. This prevents unbounded memory growth.
90+
91+
</details>
92+
93+
## Behavior
94+
95+
- Calling `XCFGSET` clears all existing producer IDMP maps for the stream.
96+
- At least one of `IDMP-DURATION` or `IDMP-MAXSIZE` must be specified.
97+
- The stream must exist before calling this command.
98+
- Configuration changes apply immediately to all future IDMP operations.
99+
100+
## Examples
101+
102+
```redis-cli
103+
XADD mystream * field value
104+
XCFGSET mystream IDMP-DURATION 300
105+
XCFGSET mystream IDMP-MAXSIZE 1000
106+
XCFGSET mystream IDMP-DURATION 600 IDMP-MAXSIZE 500
107+
```
108+
109+
## Return information
110+
111+
{{< multitabs id="return-info"
112+
tab1="RESP2"
113+
tab2="RESP3" >}}
114+
115+
[Simple string reply](../../develop/reference/protocol-spec#simple-strings): `OK` if the configuration was set successfully.
116+
117+
-tab-sep-
118+
119+
[Simple string reply](../../develop/reference/protocol-spec#simple-strings): `OK` if the configuration was set successfully.
120+
121+
{{< /multitabs >}}
122+
123+
## Error conditions
124+
125+
The command returns an error in the following cases:
126+
127+
- **WRONGTYPE**: The key exists but is not a stream
128+
- **ERR no such key**: The stream does not exist
129+
- **ERR syntax error**: Invalid command syntax or missing required arguments
130+
- **ERR invalid duration**: Duration value is outside the valid range (1-86,400)
131+
- **ERR invalid maxsize**: Maxsize value is outside the valid range (1-10,000)

0 commit comments

Comments
 (0)