|
| 1 | +--- |
| 2 | +layout: default |
| 3 | +title: NATS Configuration |
| 4 | +parent: Configuration |
| 5 | +nav_order: 3 |
| 6 | +--- |
| 7 | + |
| 8 | +# NATS Configuration |
| 9 | +{: .no_toc } |
| 10 | + |
| 11 | +<details open markdown="block"> |
| 12 | + <summary>Table of contents</summary> |
| 13 | + {: .text-delta } |
| 14 | +1. TOC |
| 15 | +{:toc} |
| 16 | +</details> |
| 17 | + |
| 18 | +--- |
| 19 | + |
| 20 | +Rqueue supports **NATS JetStream** as a drop-in replacement for Redis. All listener, |
| 21 | +producer, and middleware APIs work identically — the only changes required are the |
| 22 | +dependency and two properties. |
| 23 | + |
| 24 | +{: .warning } |
| 25 | +The NATS backend does not support delayed enqueue, scheduled messages, or periodic/cron |
| 26 | +jobs. Calls to `enqueueIn`, `enqueueAt`, and `enqueuePeriodic` throw |
| 27 | +`UnsupportedOperationException` at runtime. Use the Redis backend for workloads that |
| 28 | +need scheduling. |
| 29 | + |
| 30 | +--- |
| 31 | + |
| 32 | +## Quick Setup |
| 33 | + |
| 34 | +### 1. Add the dependency |
| 35 | + |
| 36 | +Add `rqueue-nats` alongside `rqueue-spring-boot-starter`: |
| 37 | + |
| 38 | +**Gradle** |
| 39 | +```groovy |
| 40 | +implementation 'com.github.sonus21:rqueue-spring-boot-starter:4.0.0-RELEASE' |
| 41 | +implementation 'com.github.sonus21:rqueue-nats:4.0.0-RELEASE' |
| 42 | +``` |
| 43 | + |
| 44 | +**Maven** |
| 45 | +```xml |
| 46 | +<dependency> |
| 47 | + <groupId>com.github.sonus21</groupId> |
| 48 | + <artifactId>rqueue-spring-boot-starter</artifactId> |
| 49 | + <version>4.0.0-RELEASE</version> |
| 50 | +</dependency> |
| 51 | +<dependency> |
| 52 | + <groupId>com.github.sonus21</groupId> |
| 53 | + <artifactId>rqueue-nats</artifactId> |
| 54 | + <version>4.0.0-RELEASE</version> |
| 55 | +</dependency> |
| 56 | +``` |
| 57 | + |
| 58 | +### 2. Configure `application.properties` |
| 59 | + |
| 60 | +```properties |
| 61 | +rqueue.backend=nats |
| 62 | +rqueue.nats.connection.url=nats://localhost:4222 |
| 63 | +``` |
| 64 | + |
| 65 | +No `RedisConnectionFactory` bean is needed. All Rqueue listener, producer, and |
| 66 | +middleware annotations work without any code changes. |
| 67 | + |
| 68 | +### 3. Start NATS with JetStream enabled |
| 69 | + |
| 70 | +```sh |
| 71 | +# native binary |
| 72 | +nats-server -js |
| 73 | + |
| 74 | +# Docker |
| 75 | +docker run -p 4222:4222 nats:latest -js |
| 76 | +``` |
| 77 | + |
| 78 | +At startup, Rqueue's `NatsStreamValidator` and `NatsKvBucketValidator` provision all |
| 79 | +required streams and KV buckets automatically. |
| 80 | + |
| 81 | +--- |
| 82 | + |
| 83 | +## Connection Properties |
| 84 | + |
| 85 | +All connection properties are under the `rqueue.nats.connection` prefix. |
| 86 | + |
| 87 | +| Property | Type | Default | Description | |
| 88 | +|---|---|---|---| |
| 89 | +| `url` | `String` | `nats://localhost:4222` | Single NATS server URL. | |
| 90 | +| `username` | `String` | — | Username for username/password authentication. | |
| 91 | +| `password` | `String` | — | Password for username/password authentication. | |
| 92 | +| `token` | `String` | — | Token for token-based authentication. | |
| 93 | +| `credentials-path` | `String` | — | Path to a `.creds` file for NKey/JWT authentication. | |
| 94 | +| `tls` | `boolean` | `false` | Enable TLS for the connection. | |
| 95 | +| `connection-name` | `String` | — | Logical name visible in NATS server monitoring. | |
| 96 | +| `connect-timeout` | `Duration` | (client default) | Maximum time to wait for initial connection. | |
| 97 | +| `reconnect-wait` | `Duration` | (client default) | Time to wait between reconnect attempts. | |
| 98 | +| `max-reconnects` | `int` | `-1` (unlimited) | Maximum reconnect attempts. | |
| 99 | +| `ping-interval` | `Duration` | (client default) | Interval between server pings. | |
| 100 | + |
| 101 | +### Authentication examples |
| 102 | + |
| 103 | +**Token authentication** |
| 104 | +```properties |
| 105 | +rqueue.nats.connection.token=s3cr3t |
| 106 | +``` |
| 107 | + |
| 108 | +**Username / password** |
| 109 | +```properties |
| 110 | +rqueue.nats.connection.username=rqueue |
| 111 | +rqueue.nats.connection.password=s3cr3t |
| 112 | +``` |
| 113 | + |
| 114 | +**NKey / JWT credentials file** |
| 115 | +```properties |
| 116 | +rqueue.nats.connection.credentials-path=/etc/nats/rqueue.creds |
| 117 | +``` |
| 118 | + |
| 119 | +### Connection resilience |
| 120 | + |
| 121 | +```properties |
| 122 | +# Retry for up to 10 minutes (120 attempts × 5 s wait) |
| 123 | +rqueue.nats.connection.max-reconnects=120 |
| 124 | +rqueue.nats.connection.reconnect-wait=5s |
| 125 | +``` |
| 126 | + |
| 127 | +Set `max-reconnects=-1` (the default) for unlimited retries in production — NATS |
| 128 | +reconnects silently to any cluster node without dropping in-flight messages. |
| 129 | + |
| 130 | +--- |
| 131 | + |
| 132 | +## Stream Properties |
| 133 | + |
| 134 | +Each registered queue maps to one or more JetStream streams. The defaults below apply |
| 135 | +to every stream Rqueue creates. All properties are under `rqueue.nats.stream`. |
| 136 | + |
| 137 | +| Property | Type | Default | Description | |
| 138 | +|---|---|---|---| |
| 139 | +| `replicas` | `int` | `1` | Number of stream replicas. Must not exceed the number of JetStream-enabled servers in the cluster. | |
| 140 | +| `storage` | `String` | `FILE` | Storage backend: `FILE` (durable) or `MEMORY` (faster, lost on restart). | |
| 141 | +| `retention` | `String` | `LIMITS` | Retention policy: `LIMITS`, `INTEREST`, or `WORK_QUEUE`. | |
| 142 | +| `max-age` | `Duration` | `14d` | Maximum age of messages before automatic removal. | |
| 143 | +| `max-bytes` | `long` | `-1` (unlimited) | Maximum total stream size in bytes. | |
| 144 | +| `max-messages` | `long` | `-1` (unlimited) | Maximum number of messages in the stream. | |
| 145 | +| `discard-policy` | `String` | `OLD` | What to discard when limits are hit: `OLD` (oldest messages) or `NEW` (reject new publishes). | |
| 146 | +| `duplicate-window` | `Duration` | `2m` | Server-side dedup window for the `Nats-Msg-Id` header. | |
| 147 | + |
| 148 | +{: .note } |
| 149 | +`duplicate-window` must be less than or equal to `max-age`. Set it to cover the |
| 150 | +maximum time a publisher might retry the same message ID (e.g. after a crash recovery). |
| 151 | + |
| 152 | +### Retention policy guide |
| 153 | + |
| 154 | +| Value | When to use | |
| 155 | +|---|---| |
| 156 | +| `LIMITS` (default) | General-purpose queues. Messages are kept until age/size limits are hit. | |
| 157 | +| `INTEREST` | Fan-out / pub-sub patterns. Messages are removed once every active consumer has acked. | |
| 158 | +| `WORK_QUEUE` | Lowest storage overhead. Message is removed as soon as any consumer acks it. Use for non-fan-out queues where exactly-once delivery per message is the goal. | |
| 159 | + |
| 160 | +### Three-replica production setup |
| 161 | + |
| 162 | +```properties |
| 163 | +rqueue.nats.stream.replicas=3 |
| 164 | +rqueue.nats.stream.storage=FILE |
| 165 | +rqueue.nats.stream.max-age=7d |
| 166 | +rqueue.nats.stream.duplicate-window=5m |
| 167 | +``` |
| 168 | + |
| 169 | +--- |
| 170 | + |
| 171 | +## Consumer Properties |
| 172 | + |
| 173 | +Consumer properties control the durable pull consumers Rqueue creates for each |
| 174 | +`(queue, consumerName)` pair. All properties are under `rqueue.nats.consumer`. |
| 175 | + |
| 176 | +| Property | Type | Default | Description | |
| 177 | +|---|---|---|---| |
| 178 | +| `ack-wait` | `Duration` | `30s` | Time the server waits for an ack before redelivering. Must be longer than your slowest message handler. | |
| 179 | +| `max-deliver` | `long` | `3` | Delivery attempts before a message is forwarded to the DLQ. | |
| 180 | +| `max-ack-pending` | `long` | `1000` | Maximum unacked messages a consumer can hold before the server stops delivering. | |
| 181 | +| `fetch-wait` | `Duration` | `2s` | How long `pop()` blocks waiting for messages before returning empty. | |
| 182 | + |
| 183 | +{: .note } |
| 184 | +`ack-wait` is the most important consumer setting. If a message handler takes longer |
| 185 | +than `ack-wait`, the server redelivers the message to another consumer, causing |
| 186 | +duplicate processing. Set it to at least 2× your 99th-percentile handler latency. |
| 187 | + |
| 188 | +### Tuning for slow handlers |
| 189 | + |
| 190 | +```properties |
| 191 | +# Handlers can take up to 5 minutes |
| 192 | +rqueue.nats.consumer.ack-wait=6m |
| 193 | +# Give each message 5 attempts before DLQ |
| 194 | +rqueue.nats.consumer.max-deliver=5 |
| 195 | +``` |
| 196 | + |
| 197 | +### Tuning for high-throughput queues |
| 198 | + |
| 199 | +```properties |
| 200 | +# Allow more unacked messages in-flight |
| 201 | +rqueue.nats.consumer.max-ack-pending=5000 |
| 202 | +# Reduce idle wait to pick up bursts faster |
| 203 | +rqueue.nats.consumer.fetch-wait=500ms |
| 204 | +``` |
| 205 | + |
| 206 | +--- |
| 207 | + |
| 208 | +## Naming Properties |
| 209 | + |
| 210 | +Naming properties control how stream and subject names are derived from queue names. |
| 211 | +All properties are under `rqueue.nats.naming`. |
| 212 | + |
| 213 | +| Property | Type | Default | Description | |
| 214 | +|---|---|---|---| |
| 215 | +| `stream-prefix` | `String` | `rqueue-js-` | Prefix for every JetStream stream name. | |
| 216 | +| `subject-prefix` | `String` | `rqueue.js.` | Prefix for every JetStream subject. | |
| 217 | +| `dlq-suffix` | `String` | `-dlq` | Suffix appended to stream and subject names for DLQ streams. | |
| 218 | + |
| 219 | +For a queue named `orders` with priority sub-queues `high` and `low` and a DLQ, the |
| 220 | +default naming produces: |
| 221 | + |
| 222 | +| Purpose | Stream name | Subject | |
| 223 | +|---|---|---| |
| 224 | +| Main queue | `rqueue-js-orders` | `rqueue.js.orders` | |
| 225 | +| Priority: high | `rqueue-js-orders-high` | `rqueue.js.orders.high` | |
| 226 | +| Priority: low | `rqueue-js-orders-low` | `rqueue.js.orders.low` | |
| 227 | +| Dead-letter queue | `rqueue-js-orders-dlq` | `rqueue.js.orders.dlq` | |
| 228 | + |
| 229 | +{: .note } |
| 230 | +Change the prefixes before the first deployment. Renaming them afterward requires |
| 231 | +manually migrating or recreating all streams. |
| 232 | + |
| 233 | +--- |
| 234 | + |
| 235 | +## Auto-Provisioning |
| 236 | + |
| 237 | +### Streams (`rqueue.nats.auto-create-streams`) |
| 238 | + |
| 239 | +When `true` (default), `NatsStreamValidator` creates every required stream at startup, |
| 240 | +immediately after all `@RqueueListener` methods are registered and before message |
| 241 | +pollers start. This means the hot publish/pop path never pays a `getStreamInfo` |
| 242 | +round-trip to confirm stream existence. |
| 243 | + |
| 244 | +Set to `false` for accounts where credentials lack `add_stream` permission. The |
| 245 | +validator will instead check that every required stream exists and abort boot with a |
| 246 | +clear `IllegalStateException` listing all missing streams — a deterministic startup |
| 247 | +failure rather than a `stream not found` error on first enqueue. |
| 248 | + |
| 249 | +### DLQ streams (`rqueue.nats.auto-create-dlq-stream`) |
| 250 | + |
| 251 | +When `true`, a dead-letter stream is automatically created for every queue whose |
| 252 | +`@RqueueListener` declares a `deadLetterQueue`. Default is `false` — enable it when |
| 253 | +you want the DLQ stream provisioned alongside the main stream without pre-creating it |
| 254 | +manually. |
| 255 | + |
| 256 | +### Consumers (`rqueue.nats.auto-create-consumers`) |
| 257 | + |
| 258 | +When `true` (default), durable pull consumers are created lazily on the first `pop` |
| 259 | +call for each `(stream, consumerName)` pair and the subscription is cached in-process. |
| 260 | +There is no per-pop round-trip after warm-up. |
| 261 | + |
| 262 | +Set to `false` to fail-fast on missing consumers instead of creating them. |
| 263 | + |
| 264 | +### KV buckets (`rqueue.nats.auto-create-kv-buckets`) |
| 265 | + |
| 266 | +Rqueue uses six shared KV buckets for state that Redis stores in keys, hashes, and |
| 267 | +sorted sets: |
| 268 | + |
| 269 | +| Bucket | Purpose | TTL | |
| 270 | +|---|---|---| |
| 271 | +| `rqueue-queue-config` | Per-queue configuration and DLQ wiring | None (persists) | |
| 272 | +| `rqueue-jobs` | Job execution history per message ID | `rqueue.message.durability` (default 7 days) | |
| 273 | +| `rqueue-locks` | Distributed locks for scheduler leadership | Set per lock acquisition | |
| 274 | +| `rqueue-message-metadata` | Per-message delivery status and retry count | None | |
| 275 | +| `rqueue-workers` | Worker process info (host, PID, last-seen) | `rqueue.worker.registry.worker.ttl` (default 300 s) | |
| 276 | +| `rqueue-worker-heartbeats` | Per-(queue, worker) heartbeats | `rqueue.worker.registry.queue.ttl` (default 3600 s) | |
| 277 | + |
| 278 | +When `auto-create-kv-buckets=true` (default), each store lazily creates its bucket on |
| 279 | +first use. When set to `false`, `NatsKvBucketValidator` walks every bucket and aborts |
| 280 | +boot listing any that are missing. |
| 281 | + |
| 282 | +--- |
| 283 | + |
| 284 | +## Locked-Down JetStream Accounts |
| 285 | + |
| 286 | +For deployments where application credentials cannot call `add_stream` or `kv_create` |
| 287 | +at runtime, disable all auto-provisioning and pre-create every resource before |
| 288 | +starting the application: |
| 289 | + |
| 290 | +```properties |
| 291 | +rqueue.nats.auto-create-streams=false |
| 292 | +rqueue.nats.auto-create-consumers=false |
| 293 | +rqueue.nats.auto-create-dlq-stream=false |
| 294 | +rqueue.nats.auto-create-kv-buckets=false |
| 295 | +``` |
| 296 | + |
| 297 | +### Pre-creating streams |
| 298 | + |
| 299 | +For a queue `orders` with priorities `high` / `low` and a DLQ: |
| 300 | + |
| 301 | +```sh |
| 302 | +nats stream add rqueue-js-orders \ |
| 303 | + --subjects "rqueue.js.orders" \ |
| 304 | + --storage file --replicas 3 --retention limits |
| 305 | + |
| 306 | +nats stream add rqueue-js-orders-high \ |
| 307 | + --subjects "rqueue.js.orders.high" \ |
| 308 | + --storage file --replicas 3 --retention limits |
| 309 | + |
| 310 | +nats stream add rqueue-js-orders-low \ |
| 311 | + --subjects "rqueue.js.orders.low" \ |
| 312 | + --storage file --replicas 3 --retention limits |
| 313 | + |
| 314 | +nats stream add rqueue-js-orders-dlq \ |
| 315 | + --subjects "rqueue.js.orders.dlq" \ |
| 316 | + --storage file --replicas 3 --retention limits |
| 317 | +``` |
| 318 | + |
| 319 | +### Pre-creating KV buckets |
| 320 | + |
| 321 | +Match TTL values to your `rqueue.worker.registry.*` settings: |
| 322 | + |
| 323 | +```sh |
| 324 | +# Persistent state — no TTL |
| 325 | +nats kv add rqueue-queue-config --replicas=3 --storage=file |
| 326 | +nats kv add rqueue-message-metadata --replicas=3 --storage=file |
| 327 | + |
| 328 | +# Job history — match rqueue.message.durability (default 7 days) |
| 329 | +nats kv add rqueue-jobs --replicas=3 --storage=file --ttl=7d |
| 330 | + |
| 331 | +# Locks — cover your longest expected lock hold |
| 332 | +nats kv add rqueue-locks --replicas=3 --storage=file --ttl=10m |
| 333 | + |
| 334 | +# Worker registry — match rqueue.worker.registry.worker.ttl (default 300 s) |
| 335 | +nats kv add rqueue-workers --replicas=3 --storage=file --ttl=5m |
| 336 | + |
| 337 | +# Queue heartbeats — match rqueue.worker.registry.queue.ttl (default 3600 s) |
| 338 | +nats kv add rqueue-worker-heartbeats --replicas=3 --storage=file --ttl=1h |
| 339 | +``` |
| 340 | + |
| 341 | +{: .warning } |
| 342 | +KV bucket TTLs are immutable after creation. To change a TTL, delete the bucket and |
| 343 | +recreate it. Do not delete `rqueue-queue-config` without backing it up first — it |
| 344 | +stores all registered queue configurations. |
| 345 | + |
| 346 | +--- |
| 347 | + |
| 348 | +## Inspecting Runtime State |
| 349 | + |
| 350 | +Use the `nats` CLI to inspect what Rqueue has created: |
| 351 | + |
| 352 | +```sh |
| 353 | +# List all Rqueue streams |
| 354 | +nats stream ls | grep rqueue-js- |
| 355 | + |
| 356 | +# Show message counts per queue |
| 357 | +nats stream info rqueue-js-orders |
| 358 | + |
| 359 | +# List KV buckets |
| 360 | +nats kv ls | grep rqueue- |
| 361 | + |
| 362 | +# Inspect queue configuration |
| 363 | +nats kv get rqueue-queue-config orders |
| 364 | +``` |
| 365 | + |
| 366 | +--- |
| 367 | + |
| 368 | +## Limitations |
| 369 | + |
| 370 | +| Feature | Redis backend | NATS backend | |
| 371 | +|---|---|---| |
| 372 | +| `enqueueIn` (delayed) | Supported | Not supported (throws `UnsupportedOperationException`) | |
| 373 | +| `enqueueAt` (scheduled) | Supported | Not supported | |
| 374 | +| `enqueuePeriodic` (cron) | Supported | Not supported | |
| 375 | +| `priorityGroup` weighting | Full support | Boot warning; weighting not honored | |
| 376 | +| Elastic `concurrency` (min < max) | Supported | Falls back to `max` | |
| 377 | +| `@RqueueHandler(primary)` | Supported | Ignored with boot warning | |
| 378 | +| Dashboard charts and message browse | Full support | Queue sizes only; charts and message browse unavailable | |
| 379 | +| Reactive listener container | Supported | Enqueue side only | |
0 commit comments