Skip to content

Commit 2681414

Browse files
committed
fix: test issue
1 parent 6920fca commit 2681414

7 files changed

Lines changed: 413 additions & 59 deletions

File tree

build.gradle

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ ext {
8585

8686
subprojects {
8787
group = "com.github.sonus21"
88-
version = "4.0.0-RC3"
88+
version = "4.0.0-LC"
8989

9090
dependencies {
9191
// https://mvnrepository.com/artifact/org.springframework/spring-messaging

docs/_config.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
# You can create any custom variable you would like, and they will be accessible
1515
# in the templates via {{ site.myvariable }}.
1616
title: Rqueue
17-
description: Library for Async Processing using Redis
17+
description: Library for Async Processing using Redis or NATS JetStream
1818
baseurl: "/rqueue" # the subpath of your site, e.g. /blog
1919
url: "https://sonus21.github.io/rqueue" # the base hostname & protocol for your site, e.g. http://example.com
2020

Lines changed: 379 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,379 @@
1+
---
2+
layout: default
3+
title: NATS Configuration
4+
parent: Configuration
5+
nav_order: 3
6+
---
7+
8+
# NATS Configuration
9+
{: .no_toc }
10+
11+
<details open markdown="block">
12+
<summary>Table of contents</summary>
13+
{: .text-delta }
14+
1. TOC
15+
{:toc}
16+
</details>
17+
18+
---
19+
20+
Rqueue supports **NATS JetStream** as a drop-in replacement for Redis. All listener,
21+
producer, and middleware APIs work identically — the only changes required are the
22+
dependency and two properties.
23+
24+
{: .warning }
25+
The NATS backend does not support delayed enqueue, scheduled messages, or periodic/cron
26+
jobs. Calls to `enqueueIn`, `enqueueAt`, and `enqueuePeriodic` throw
27+
`UnsupportedOperationException` at runtime. Use the Redis backend for workloads that
28+
need scheduling.
29+
30+
---
31+
32+
## Quick Setup
33+
34+
### 1. Add the dependency
35+
36+
Add `rqueue-nats` alongside `rqueue-spring-boot-starter`:
37+
38+
**Gradle**
39+
```groovy
40+
implementation 'com.github.sonus21:rqueue-spring-boot-starter:4.0.0-RELEASE'
41+
implementation 'com.github.sonus21:rqueue-nats:4.0.0-RELEASE'
42+
```
43+
44+
**Maven**
45+
```xml
46+
<dependency>
47+
<groupId>com.github.sonus21</groupId>
48+
<artifactId>rqueue-spring-boot-starter</artifactId>
49+
<version>4.0.0-RELEASE</version>
50+
</dependency>
51+
<dependency>
52+
<groupId>com.github.sonus21</groupId>
53+
<artifactId>rqueue-nats</artifactId>
54+
<version>4.0.0-RELEASE</version>
55+
</dependency>
56+
```
57+
58+
### 2. Configure `application.properties`
59+
60+
```properties
61+
rqueue.backend=nats
62+
rqueue.nats.connection.url=nats://localhost:4222
63+
```
64+
65+
No `RedisConnectionFactory` bean is needed. All Rqueue listener, producer, and
66+
middleware annotations work without any code changes.
67+
68+
### 3. Start NATS with JetStream enabled
69+
70+
```sh
71+
# native binary
72+
nats-server -js
73+
74+
# Docker
75+
docker run -p 4222:4222 nats:latest -js
76+
```
77+
78+
At startup, Rqueue's `NatsStreamValidator` and `NatsKvBucketValidator` provision all
79+
required streams and KV buckets automatically.
80+
81+
---
82+
83+
## Connection Properties
84+
85+
All connection properties are under the `rqueue.nats.connection` prefix.
86+
87+
| Property | Type | Default | Description |
88+
|---|---|---|---|
89+
| `url` | `String` | `nats://localhost:4222` | Single NATS server URL. |
90+
| `username` | `String` || Username for username/password authentication. |
91+
| `password` | `String` || Password for username/password authentication. |
92+
| `token` | `String` || Token for token-based authentication. |
93+
| `credentials-path` | `String` || Path to a `.creds` file for NKey/JWT authentication. |
94+
| `tls` | `boolean` | `false` | Enable TLS for the connection. |
95+
| `connection-name` | `String` || Logical name visible in NATS server monitoring. |
96+
| `connect-timeout` | `Duration` | (client default) | Maximum time to wait for initial connection. |
97+
| `reconnect-wait` | `Duration` | (client default) | Time to wait between reconnect attempts. |
98+
| `max-reconnects` | `int` | `-1` (unlimited) | Maximum reconnect attempts. |
99+
| `ping-interval` | `Duration` | (client default) | Interval between server pings. |
100+
101+
### Authentication examples
102+
103+
**Token authentication**
104+
```properties
105+
rqueue.nats.connection.token=s3cr3t
106+
```
107+
108+
**Username / password**
109+
```properties
110+
rqueue.nats.connection.username=rqueue
111+
rqueue.nats.connection.password=s3cr3t
112+
```
113+
114+
**NKey / JWT credentials file**
115+
```properties
116+
rqueue.nats.connection.credentials-path=/etc/nats/rqueue.creds
117+
```
118+
119+
### Connection resilience
120+
121+
```properties
122+
# Retry for up to 10 minutes (120 attempts × 5 s wait)
123+
rqueue.nats.connection.max-reconnects=120
124+
rqueue.nats.connection.reconnect-wait=5s
125+
```
126+
127+
Set `max-reconnects=-1` (the default) for unlimited retries in production — NATS
128+
reconnects silently to any cluster node without dropping in-flight messages.
129+
130+
---
131+
132+
## Stream Properties
133+
134+
Each registered queue maps to one or more JetStream streams. The defaults below apply
135+
to every stream Rqueue creates. All properties are under `rqueue.nats.stream`.
136+
137+
| Property | Type | Default | Description |
138+
|---|---|---|---|
139+
| `replicas` | `int` | `1` | Number of stream replicas. Must not exceed the number of JetStream-enabled servers in the cluster. |
140+
| `storage` | `String` | `FILE` | Storage backend: `FILE` (durable) or `MEMORY` (faster, lost on restart). |
141+
| `retention` | `String` | `LIMITS` | Retention policy: `LIMITS`, `INTEREST`, or `WORK_QUEUE`. |
142+
| `max-age` | `Duration` | `14d` | Maximum age of messages before automatic removal. |
143+
| `max-bytes` | `long` | `-1` (unlimited) | Maximum total stream size in bytes. |
144+
| `max-messages` | `long` | `-1` (unlimited) | Maximum number of messages in the stream. |
145+
| `discard-policy` | `String` | `OLD` | What to discard when limits are hit: `OLD` (oldest messages) or `NEW` (reject new publishes). |
146+
| `duplicate-window` | `Duration` | `2m` | Server-side dedup window for the `Nats-Msg-Id` header. |
147+
148+
{: .note }
149+
`duplicate-window` must be less than or equal to `max-age`. Set it to cover the
150+
maximum time a publisher might retry the same message ID (e.g. after a crash recovery).
151+
152+
### Retention policy guide
153+
154+
| Value | When to use |
155+
|---|---|
156+
| `LIMITS` (default) | General-purpose queues. Messages are kept until age/size limits are hit. |
157+
| `INTEREST` | Fan-out / pub-sub patterns. Messages are removed once every active consumer has acked. |
158+
| `WORK_QUEUE` | Lowest storage overhead. Message is removed as soon as any consumer acks it. Use for non-fan-out queues where exactly-once delivery per message is the goal. |
159+
160+
### Three-replica production setup
161+
162+
```properties
163+
rqueue.nats.stream.replicas=3
164+
rqueue.nats.stream.storage=FILE
165+
rqueue.nats.stream.max-age=7d
166+
rqueue.nats.stream.duplicate-window=5m
167+
```
168+
169+
---
170+
171+
## Consumer Properties
172+
173+
Consumer properties control the durable pull consumers Rqueue creates for each
174+
`(queue, consumerName)` pair. All properties are under `rqueue.nats.consumer`.
175+
176+
| Property | Type | Default | Description |
177+
|---|---|---|---|
178+
| `ack-wait` | `Duration` | `30s` | Time the server waits for an ack before redelivering. Must be longer than your slowest message handler. |
179+
| `max-deliver` | `long` | `3` | Delivery attempts before a message is forwarded to the DLQ. |
180+
| `max-ack-pending` | `long` | `1000` | Maximum unacked messages a consumer can hold before the server stops delivering. |
181+
| `fetch-wait` | `Duration` | `2s` | How long `pop()` blocks waiting for messages before returning empty. |
182+
183+
{: .note }
184+
`ack-wait` is the most important consumer setting. If a message handler takes longer
185+
than `ack-wait`, the server redelivers the message to another consumer, causing
186+
duplicate processing. Set it to at least 2× your 99th-percentile handler latency.
187+
188+
### Tuning for slow handlers
189+
190+
```properties
191+
# Handlers can take up to 5 minutes
192+
rqueue.nats.consumer.ack-wait=6m
193+
# Give each message 5 attempts before DLQ
194+
rqueue.nats.consumer.max-deliver=5
195+
```
196+
197+
### Tuning for high-throughput queues
198+
199+
```properties
200+
# Allow more unacked messages in-flight
201+
rqueue.nats.consumer.max-ack-pending=5000
202+
# Reduce idle wait to pick up bursts faster
203+
rqueue.nats.consumer.fetch-wait=500ms
204+
```
205+
206+
---
207+
208+
## Naming Properties
209+
210+
Naming properties control how stream and subject names are derived from queue names.
211+
All properties are under `rqueue.nats.naming`.
212+
213+
| Property | Type | Default | Description |
214+
|---|---|---|---|
215+
| `stream-prefix` | `String` | `rqueue-js-` | Prefix for every JetStream stream name. |
216+
| `subject-prefix` | `String` | `rqueue.js.` | Prefix for every JetStream subject. |
217+
| `dlq-suffix` | `String` | `-dlq` | Suffix appended to stream and subject names for DLQ streams. |
218+
219+
For a queue named `orders` with priority sub-queues `high` and `low` and a DLQ, the
220+
default naming produces:
221+
222+
| Purpose | Stream name | Subject |
223+
|---|---|---|
224+
| Main queue | `rqueue-js-orders` | `rqueue.js.orders` |
225+
| Priority: high | `rqueue-js-orders-high` | `rqueue.js.orders.high` |
226+
| Priority: low | `rqueue-js-orders-low` | `rqueue.js.orders.low` |
227+
| Dead-letter queue | `rqueue-js-orders-dlq` | `rqueue.js.orders.dlq` |
228+
229+
{: .note }
230+
Change the prefixes before the first deployment. Renaming them afterward requires
231+
manually migrating or recreating all streams.
232+
233+
---
234+
235+
## Auto-Provisioning
236+
237+
### Streams (`rqueue.nats.auto-create-streams`)
238+
239+
When `true` (default), `NatsStreamValidator` creates every required stream at startup,
240+
immediately after all `@RqueueListener` methods are registered and before message
241+
pollers start. This means the hot publish/pop path never pays a `getStreamInfo`
242+
round-trip to confirm stream existence.
243+
244+
Set to `false` for accounts where credentials lack `add_stream` permission. The
245+
validator will instead check that every required stream exists and abort boot with a
246+
clear `IllegalStateException` listing all missing streams — a deterministic startup
247+
failure rather than a `stream not found` error on first enqueue.
248+
249+
### DLQ streams (`rqueue.nats.auto-create-dlq-stream`)
250+
251+
When `true`, a dead-letter stream is automatically created for every queue whose
252+
`@RqueueListener` declares a `deadLetterQueue`. Default is `false` — enable it when
253+
you want the DLQ stream provisioned alongside the main stream without pre-creating it
254+
manually.
255+
256+
### Consumers (`rqueue.nats.auto-create-consumers`)
257+
258+
When `true` (default), durable pull consumers are created lazily on the first `pop`
259+
call for each `(stream, consumerName)` pair and the subscription is cached in-process.
260+
There is no per-pop round-trip after warm-up.
261+
262+
Set to `false` to fail-fast on missing consumers instead of creating them.
263+
264+
### KV buckets (`rqueue.nats.auto-create-kv-buckets`)
265+
266+
Rqueue uses six shared KV buckets for state that Redis stores in keys, hashes, and
267+
sorted sets:
268+
269+
| Bucket | Purpose | TTL |
270+
|---|---|---|
271+
| `rqueue-queue-config` | Per-queue configuration and DLQ wiring | None (persists) |
272+
| `rqueue-jobs` | Job execution history per message ID | `rqueue.message.durability` (default 7 days) |
273+
| `rqueue-locks` | Distributed locks for scheduler leadership | Set per lock acquisition |
274+
| `rqueue-message-metadata` | Per-message delivery status and retry count | None |
275+
| `rqueue-workers` | Worker process info (host, PID, last-seen) | `rqueue.worker.registry.worker.ttl` (default 300 s) |
276+
| `rqueue-worker-heartbeats` | Per-(queue, worker) heartbeats | `rqueue.worker.registry.queue.ttl` (default 3600 s) |
277+
278+
When `auto-create-kv-buckets=true` (default), each store lazily creates its bucket on
279+
first use. When set to `false`, `NatsKvBucketValidator` walks every bucket and aborts
280+
boot listing any that are missing.
281+
282+
---
283+
284+
## Locked-Down JetStream Accounts
285+
286+
For deployments where application credentials cannot call `add_stream` or `kv_create`
287+
at runtime, disable all auto-provisioning and pre-create every resource before
288+
starting the application:
289+
290+
```properties
291+
rqueue.nats.auto-create-streams=false
292+
rqueue.nats.auto-create-consumers=false
293+
rqueue.nats.auto-create-dlq-stream=false
294+
rqueue.nats.auto-create-kv-buckets=false
295+
```
296+
297+
### Pre-creating streams
298+
299+
For a queue `orders` with priorities `high` / `low` and a DLQ:
300+
301+
```sh
302+
nats stream add rqueue-js-orders \
303+
--subjects "rqueue.js.orders" \
304+
--storage file --replicas 3 --retention limits
305+
306+
nats stream add rqueue-js-orders-high \
307+
--subjects "rqueue.js.orders.high" \
308+
--storage file --replicas 3 --retention limits
309+
310+
nats stream add rqueue-js-orders-low \
311+
--subjects "rqueue.js.orders.low" \
312+
--storage file --replicas 3 --retention limits
313+
314+
nats stream add rqueue-js-orders-dlq \
315+
--subjects "rqueue.js.orders.dlq" \
316+
--storage file --replicas 3 --retention limits
317+
```
318+
319+
### Pre-creating KV buckets
320+
321+
Match TTL values to your `rqueue.worker.registry.*` settings:
322+
323+
```sh
324+
# Persistent state — no TTL
325+
nats kv add rqueue-queue-config --replicas=3 --storage=file
326+
nats kv add rqueue-message-metadata --replicas=3 --storage=file
327+
328+
# Job history — match rqueue.message.durability (default 7 days)
329+
nats kv add rqueue-jobs --replicas=3 --storage=file --ttl=7d
330+
331+
# Locks — cover your longest expected lock hold
332+
nats kv add rqueue-locks --replicas=3 --storage=file --ttl=10m
333+
334+
# Worker registry — match rqueue.worker.registry.worker.ttl (default 300 s)
335+
nats kv add rqueue-workers --replicas=3 --storage=file --ttl=5m
336+
337+
# Queue heartbeats — match rqueue.worker.registry.queue.ttl (default 3600 s)
338+
nats kv add rqueue-worker-heartbeats --replicas=3 --storage=file --ttl=1h
339+
```
340+
341+
{: .warning }
342+
KV bucket TTLs are immutable after creation. To change a TTL, delete the bucket and
343+
recreate it. Do not delete `rqueue-queue-config` without backing it up first — it
344+
stores all registered queue configurations.
345+
346+
---
347+
348+
## Inspecting Runtime State
349+
350+
Use the `nats` CLI to inspect what Rqueue has created:
351+
352+
```sh
353+
# List all Rqueue streams
354+
nats stream ls | grep rqueue-js-
355+
356+
# Show message counts per queue
357+
nats stream info rqueue-js-orders
358+
359+
# List KV buckets
360+
nats kv ls | grep rqueue-
361+
362+
# Inspect queue configuration
363+
nats kv get rqueue-queue-config orders
364+
```
365+
366+
---
367+
368+
## Limitations
369+
370+
| Feature | Redis backend | NATS backend |
371+
|---|---|---|
372+
| `enqueueIn` (delayed) | Supported | Not supported (throws `UnsupportedOperationException`) |
373+
| `enqueueAt` (scheduled) | Supported | Not supported |
374+
| `enqueuePeriodic` (cron) | Supported | Not supported |
375+
| `priorityGroup` weighting | Full support | Boot warning; weighting not honored |
376+
| Elastic `concurrency` (min < max) | Supported | Falls back to `max` |
377+
| `@RqueueHandler(primary)` | Supported | Ignored with boot warning |
378+
| Dashboard charts and message browse | Full support | Queue sizes only; charts and message browse unavailable |
379+
| Reactive listener container | Supported | Enqueue side only |

0 commit comments

Comments
 (0)