From 004d4dfb18a19d1ac9cfa0318e516ee46b994412 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 24 Apr 2025 11:50:31 +0200 Subject: [PATCH 01/60] task queue --- node.js/{outbox.md => task-queue.md} | 103 ++++++++++++++------------- 1 file changed, 52 insertions(+), 51 deletions(-) rename node.js/{outbox.md => task-queue.md} (62%) diff --git a/node.js/outbox.md b/node.js/task-queue.md similarity index 62% rename from node.js/outbox.md rename to node.js/task-queue.md index 551370541f..be2d20b7a6 100644 --- a/node.js/outbox.md +++ b/node.js/task-queue.md @@ -1,64 +1,66 @@ --- synopsis: > - Learn details about the outbox feature. + Learn details about the task-queue feature. # layout: node-js status: released --- -# Outboxing with `cds.outboxed` +# Queuing with `cds.queued` [[toc]] ## Overview -Often, remote operations should be delayed until the main transaction succeeded. Otherwise, the remote operations are also triggered in case of a rollback. -To enable this, an outbox can be used to defer remote operations until the success of the current transaction. +The _task queue_ feature allows you to safely store requests to the database before they are processed. It includes a built-in retry mechanism to handle failures reliably. -Every CAP service can be _outboxed_ that means event dispatching becomes _asynchronous_. +A common use case is the outbox pattern, where remote operations are deferred until the main transaction has been successfully committed. +This prevents accidental execution of remote calls in case the transaction is rolled back. +Every CAP service can be _queued_ that means event dispatching becomes _asynchronous_. -## Outboxing a Service +## Queueing a Service -### cds.outboxed(srv) {.method} -Programmatically, you can get the outboxed service as follows: +### cds.queued(srv) {.method} + +Programmatically, you can get the queued service as follows: ```js const srv = await cds.connect.to('yourService') -const outboxed = cds.outboxed(srv) +const queued = cds.queued(srv) -await outboxed.emit('someEvent', { some: 'message' }) // asynchronous -await outboxed.send('someEvent', { some: 'message' }) // synchronous +await queued.emit('someEvent', { some: 'message' }) // asynchronous +await queued.send('someEvent', { some: 'message' }) // asynchronous ``` ::: tip -You still need to `await` these operations. Then the messages are stored in the database, together with the main transaction. +You still need to `await` these operations since messages are stored in the database, in the current transaction. ::: -The `cds.outboxed` function can also be called with optional configuration options. +The `cds.queued` function can also be called with optional configuration options. ```js -const outboxed = cds.outboxed(srv, { kind: 'persistent-outbox' }) +const queued = cds.queued(srv, { kind: 'persistent-queue' }) ``` -> The persistent outbox can only be used if it's enabled globally with `cds.requires.outbox = true` because it requires a dedicated database table. +> The persistent queue can only be used if it's enabled globally with `cds.requires.queue = true` because it requires a dedicated database table. ::: warning One-time configuration -Once you outboxed a service, you cannot override its outbox configuration options again. +Once you queued a service, you cannot override its configuration options again. ::: -### cds.unboxed(srv) {.method} +### cds.unqueued(srv) {.method} -Use this on an outboxed service to get back the original service: +Use this on an unqueued service to get back the original service: ```js -const unboxed = cds.unboxed(srv) +const unqueued = cds.unqueued(srv) ``` -This is useful if your service is outboxed per configuration. +This is useful if your service is outboxed (i.e., queued) per configuration. ### Per Configuration @@ -76,26 +78,26 @@ You can also configure services to be outboxed by default: } ``` -::: tip Outboxed by default -Some services are outboxed by default, these services include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. +::: tip Queued by default +Some services are queued by default, these services include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. ::: -For transactional safety, you're encouraged to enable the [persistent outbox](#persistent-outbox). +For transactional safety, you're encouraged to enable the [persistent queue](#persistent-queue). -## Persistent Outbox (Default) {#persistent-outbox} +## Persistent Queue (Default) {#persistent-outbox} You can enable it globally for all outboxed services with: ```json { "requires": { - "outbox": true + "queue": true } } ``` -Using the persistent outbox, the to-be-emitted message is stored in a database table first. The same database transaction is used +Using the persistent queue, the to-be-emitted message is stored in a database table first. The same database transaction is used as for other operations, therefore transactional consistency is guaranteed. You can use the following configuration options: @@ -103,8 +105,8 @@ You can use the following configuration options: ```json { "requires": { - "outbox": { - "kind": "persistent-outbox", + "queue": { + "kind": "persistent-queue", "maxAttempts": 20, "chunkSize": 100, "storeLastError": true, @@ -118,7 +120,7 @@ The optional parameters are: - `maxAttempts` (default `20`): The number of unsuccessful emits until the message is ignored. It will still remain in the database table. - `chunkSize` (default `100`): The number of messages which are read from the database table in one go. -- `storeLastError` (default `true`): Specifies if error information of the last failed emit should be stored in the outbox table. +- `storeLastError` (default `true`): Specifies if error information of the last failed emit should be stored in the tasks table. - `parallel` (default `true`): Specifies if messages are sent in parallel (faster but the order isn't guaranteed). @@ -139,12 +141,12 @@ The respective message is then updated and the `attempts` field is set to `maxAt ::: -Your database model is automatically extended by the entity `cds.outbox.Messages`: +Your database model is automatically extended by the entity `cds.queued.Tasks`: ```cds -namespace cds.outbox; +namespace cds.queued; -entity Messages { +entity Tasks { key ID : UUID; timestamp : Timestamp; target : String; @@ -156,20 +158,19 @@ entity Messages { } ``` -In your CDS model, you can refer to the entity `cds.outbox.Messages` using the path `@sap/cds/srv/outbox`, +In your CDS model, you can refer to the entity `cds.queued.Tasks` using the path `@sap/cds/srv/queue`, for example to expose it in a service. #### Known Limitations - If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. -- The service that handles the outboxed event must not use user roles and attributes as they are not stored. However, the user ID is stored to recreate the correct context. -- The service that handles the outboxed event must not perform any database modifications, because a global database transaction is used when dispatching the events. The outbox must only be used for services which communicate with external systems. +- The service that handles the queued event must not use user roles and attributes as they are not stored. However, the user ID is stored to recreate the correct context. ### Managing the Dead Letter Queue -You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.outbox.Messages` as well as bound actions to either revive or delete the respective message. +You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.queued.Tasks` as well as bound actions to either revive or delete the respective message. ::: tip Please see [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general Outbox guide. @@ -179,13 +180,13 @@ Please see [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) i ::: code-group ```cds [srv/outbox-dead-letter-queue-service.cds] -using from '@sap/cds/srv/outbox'; +using from '@sap/cds/srv/queue'; @requires: 'internal-user' service OutboxDeadLetterQueueService { @readonly - entity DeadOutboxMessages as projection on cds.outbox.Messages + entity DeadOutboxMessages as projection on cds.queued.Tasks actions { action revive(); action delete(); @@ -212,15 +213,15 @@ Finally, entries in the dead letter queue can either be _revived_ by resetting t ::: -## In-Memory Outbox +## In-Memory Queue -You can enable it globally for all outboxed services with: +You can enable it globally for all queued services with: ```json { "requires": { - "outbox": { - "kind": "in-memory-outbox" + "queue": { + "kind": "in-memory-queue" } } } @@ -252,28 +253,28 @@ To disable deferred emitting for a particular service, you can set the `outbox` ## Troubleshooting -### Delete Entries in the Outbox Table +### Delete Entries in the Tasks Table -To manually delete entries in the table `cds.outbox.Messages`, you can either -expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.outbox.Messages` +To manually delete entries in the table `cds.queued.Tasks`, you can either +expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.queued.Tasks` entity: ```js const db = await cds.connect.to('db') -const { Messages } = db.entities('cds.outbox') -await DELETE.from(Messages) +const { Tasks } = db.entities('cds.queued') +await DELETE.from(Tasks) ``` -### Outbox Table Not Found +### Tasks Table Not Found -If the outbox table is not found on the database, this can be caused by insufficient configuration data in _package.json_. +If the tasks table is not found on the database, this can be caused by insufficient configuration data in _package.json_. -In case you have overwritten `requires.db.model` there, make sure to add the outbox model path `@sap/cds/srv/outbox`: +In case you have overwritten `requires.db.model` there, make sure to add the queue model path `@sap/cds/srv/queue`: ```jsonc "requires": { "db": { ... - "model": [..., "@sap/cds/srv/outbox"] + "model": [..., "@sap/cds/srv/queue"] } } ``` From de9bbfad33898b5ad5fbf58cc897e8a38ab3132f Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 24 Apr 2025 11:57:39 +0200 Subject: [PATCH 02/60] . --- guides/deployment/custom-builds.md | 2 +- node.js/messaging.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/guides/deployment/custom-builds.md b/guides/deployment/custom-builds.md index 19b519fe0a..22dae0c6b4 100644 --- a/guides/deployment/custom-builds.md +++ b/guides/deployment/custom-builds.md @@ -22,7 +22,7 @@ The CDS model folders and files used by `cds build` are determined as follows: - Known root folders, by [default](../../get-started/#project-structure) the folders _db/, srv/, app/_, can also be configured by [_folders.db, folders.srv, folders.app_](../../get-started/#project-structure). - The _src_ folder configured for the individual build task. - If [Feature toggles](../extensibility/feature-toggles#enable-feature-toggles) are enabled: subfolders of the folder _fts_. -- CDS Model folders and files defined by the [required services](../../node.js/cds-env#services) of your project. This also includes models used by required built-in CDS services, like [persistent outbox](../../node.js/outbox#persistent-outbox) or [MTX related services](../multitenancy/mtxs#mtx-services-reference). +- CDS Model folders and files defined by the [required services](../../node.js/cds-env#services) of your project. This also includes models used by required built-in CDS services, like [persistent queue](../../node.js/queue#persistent-queue) or [MTX related services](../multitenancy/mtxs#mtx-services-reference). Feature toggle folders and required built-in service models will also be added if user defined models have already been configured as [_model_ option](#build-task-properties) in your build tasks. diff --git a/node.js/messaging.md b/node.js/messaging.md index 5580a8a6e3..e5ad4abc81 100644 --- a/node.js/messaging.md +++ b/node.js/messaging.md @@ -127,7 +127,7 @@ this.after(['CREATE', 'UPDATE', 'DELETE'], 'Reviews', async (_, req) => { ``` ::: tip The messages are sent once the transaction is successful. -Per default, a persistent outbox is used. See [Messaging - Outbox](./outbox) for more information. +Per default, a persistent queue is used. See [Messaging - Queue](./queue) for more information. ::: ## Receiving Events From aa84a7c86db78b97243ccb1b586c4226ef274e4b Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 24 Apr 2025 12:03:07 +0200 Subject: [PATCH 03/60] . --- node.js/_menu.md | 2 +- node.js/{task-queue.md => queue.md} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename node.js/{task-queue.md => queue.md} (100%) diff --git a/node.js/_menu.md b/node.js/_menu.md index eec9275676..609d8aae34 100644 --- a/node.js/_menu.md +++ b/node.js/_menu.md @@ -57,7 +57,7 @@ # [cds. utils](cds-utils) # [cds. test()](cds-test) # [cds. plugins](cds-plugins) -# [cds. outboxed()](outbox) +# [cds. queued()](queue) # [TypeScript](typescript) # [Fiori Support](fiori) # [Best Practices](best-practices) diff --git a/node.js/task-queue.md b/node.js/queue.md similarity index 100% rename from node.js/task-queue.md rename to node.js/queue.md From 0575afee4cd2fa4e675325420f309e1dcb0e6250 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 24 Apr 2025 13:02:59 +0200 Subject: [PATCH 04/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index be2d20b7a6..cf05cb7bba 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -5,7 +5,7 @@ synopsis: > status: released --- -# Queuing with `cds.queued` +# Queueing with `cds.queued` [[toc]] From a7e0c68f05a31ee272fb3ee57b0528f37db63d8d Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 24 Apr 2025 13:09:12 +0200 Subject: [PATCH 05/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index cf05cb7bba..755cb6ea00 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -12,7 +12,7 @@ status: released ## Overview -The _task queue_ feature allows you to safely store requests to the database before they are processed. It includes a built-in retry mechanism to handle failures reliably. +The _task queue_ feature allows you to defer request processing. A common use case is the outbox pattern, where remote operations are deferred until the main transaction has been successfully committed. This prevents accidental execution of remote calls in case the transaction is rolled back. From 0a94949c715e450573d37355875f4e3dda3184d8 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 24 Apr 2025 13:10:01 +0200 Subject: [PATCH 06/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 755cb6ea00..60b05de6b9 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -36,7 +36,7 @@ await queued.send('someEvent', { some: 'message' }) // asynchronous ``` ::: tip -You still need to `await` these operations since messages are stored in the database, in the current transaction. +You still need to `await` these operations. In case of a persistent queue, messages are stored in the database, within the current transaction. ::: The `cds.queued` function can also be called with optional configuration options. From 5087453dd543a7eb7a4a957078541ee7e2c3749a Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 24 Apr 2025 13:11:25 +0200 Subject: [PATCH 07/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 60b05de6b9..c02eed99db 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -85,7 +85,7 @@ Some services are queued by default, these services include [`cds.MessagingServi For transactional safety, you're encouraged to enable the [persistent queue](#persistent-queue). -## Persistent Queue (Default) {#persistent-outbox} +## Persistent Queue (Default) {#persistent-queue} You can enable it globally for all outboxed services with: From 1a493af6aa7ded650dc95348ba89eee2273f41a7 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 25 Apr 2025 14:48:16 +0200 Subject: [PATCH 08/60] . --- node.js/queue.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index c02eed99db..23f7b8adbb 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -5,7 +5,7 @@ synopsis: > status: released --- -# Queueing with `cds.queued` +# Queueing with `cds.core` [[toc]] @@ -23,13 +23,13 @@ Every CAP service can be _queued_ that means event dispatching becomes _asynchro ## Queueing a Service -### cds.queued(srv) {.method} +### cds.core(srv) {.method} Programmatically, you can get the queued service as follows: ```js const srv = await cds.connect.to('yourService') -const queued = cds.queued(srv) +const queued = cds.core(srv) await queued.emit('someEvent', { some: 'message' }) // asynchronous await queued.send('someEvent', { some: 'message' }) // asynchronous @@ -39,10 +39,10 @@ await queued.send('someEvent', { some: 'message' }) // asynchronous You still need to `await` these operations. In case of a persistent queue, messages are stored in the database, within the current transaction. ::: -The `cds.queued` function can also be called with optional configuration options. +The `cds.core` function can also be called with optional configuration options. ```js -const queued = cds.queued(srv, { kind: 'persistent-queue' }) +const queued = cds.core(srv, { kind: 'persistent-queue' }) ``` > The persistent queue can only be used if it's enabled globally with `cds.requires.queue = true` because it requires a dedicated database table. @@ -141,10 +141,10 @@ The respective message is then updated and the `attempts` field is set to `maxAt ::: -Your database model is automatically extended by the entity `cds.queued.Tasks`: +Your database model is automatically extended by the entity `cds.core.Tasks`: ```cds -namespace cds.queued; +namespace cds.core; entity Tasks { key ID : UUID; @@ -158,7 +158,7 @@ entity Tasks { } ``` -In your CDS model, you can refer to the entity `cds.queued.Tasks` using the path `@sap/cds/srv/queue`, +In your CDS model, you can refer to the entity `cds.core.Tasks` using the path `@sap/cds/srv/queue`, for example to expose it in a service. @@ -170,7 +170,7 @@ for example to expose it in a service. ### Managing the Dead Letter Queue -You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.queued.Tasks` as well as bound actions to either revive or delete the respective message. +You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.core.Tasks` as well as bound actions to either revive or delete the respective message. ::: tip Please see [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general Outbox guide. @@ -186,7 +186,7 @@ using from '@sap/cds/srv/queue'; service OutboxDeadLetterQueueService { @readonly - entity DeadOutboxMessages as projection on cds.queued.Tasks + entity DeadOutboxMessages as projection on cds.core.Tasks actions { action revive(); action delete(); @@ -255,13 +255,13 @@ To disable deferred emitting for a particular service, you can set the `outbox` ### Delete Entries in the Tasks Table -To manually delete entries in the table `cds.queued.Tasks`, you can either -expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.queued.Tasks` +To manually delete entries in the table `cds.core.Tasks`, you can either +expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.core.Tasks` entity: ```js const db = await cds.connect.to('db') -const { Tasks } = db.entities('cds.queued') +const { Tasks } = db.entities('cds.core') await DELETE.from(Tasks) ``` From 40ea457f92658bf78e0e07ec81d7a33a651772bd Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 25 Apr 2025 15:42:15 +0200 Subject: [PATCH 09/60] enabled by default --- node.js/queue.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 23f7b8adbb..6398a7b86b 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -87,12 +87,14 @@ For transactional safety, you're encouraged to enable the [persistent queue](#pe ## Persistent Queue (Default) {#persistent-queue} -You can enable it globally for all outboxed services with: +The persistent queue is enabled by default. + +You can disable it globally with: ```json { "requires": { - "queue": true + "queue": false } } ``` From e0c15a3c1ed70ae0e13f10897e22b0aba90764bf Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:36:09 +0200 Subject: [PATCH 10/60] cds.core.Queued.Messages --- node.js/queue.md | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 6398a7b86b..d7b8372b12 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -5,7 +5,7 @@ synopsis: > status: released --- -# Queueing with `cds.core` +# Queueing with `cds.queued` [[toc]] @@ -23,13 +23,13 @@ Every CAP service can be _queued_ that means event dispatching becomes _asynchro ## Queueing a Service -### cds.core(srv) {.method} +### cds.queued(srv) {.method} Programmatically, you can get the queued service as follows: ```js const srv = await cds.connect.to('yourService') -const queued = cds.core(srv) +const queued = cds.queued(srv) await queued.emit('someEvent', { some: 'message' }) // asynchronous await queued.send('someEvent', { some: 'message' }) // asynchronous @@ -39,10 +39,10 @@ await queued.send('someEvent', { some: 'message' }) // asynchronous You still need to `await` these operations. In case of a persistent queue, messages are stored in the database, within the current transaction. ::: -The `cds.core` function can also be called with optional configuration options. +The `cds.queued` function can also be called with optional configuration options. ```js -const queued = cds.core(srv, { kind: 'persistent-queue' }) +const queued = cds.queued(srv, { kind: 'persistent-queue' }) ``` > The persistent queue can only be used if it's enabled globally with `cds.requires.queue = true` because it requires a dedicated database table. @@ -143,24 +143,23 @@ The respective message is then updated and the `attempts` field is set to `maxAt ::: -Your database model is automatically extended by the entity `cds.core.Tasks`: +Your database model is automatically extended by the entity `cds.core.Queued.Messages`: ```cds namespace cds.core; -entity Tasks { +entity Queued.Messages { key ID : UUID; timestamp : Timestamp; target : String; msg : LargeString; attempts : Integer default 0; - partition : Integer default 0; lastError : LargeString; lastAttemptTimestamp : Timestamp @cds.on.update: $now; } ``` -In your CDS model, you can refer to the entity `cds.core.Tasks` using the path `@sap/cds/srv/queue`, +In your CDS model, you can refer to the entity `cds.core.Queued.Messages` using the path `@sap/cds/srv/queue`, for example to expose it in a service. @@ -172,7 +171,7 @@ for example to expose it in a service. ### Managing the Dead Letter Queue -You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.core.Tasks` as well as bound actions to either revive or delete the respective message. +You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.core.Queued.Messages` as well as bound actions to either revive or delete the respective message. ::: tip Please see [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general Outbox guide. @@ -188,7 +187,7 @@ using from '@sap/cds/srv/queue'; service OutboxDeadLetterQueueService { @readonly - entity DeadOutboxMessages as projection on cds.core.Tasks + entity DeadOutboxMessages as projection on cds.core.Queued.Messages actions { action revive(); action delete(); @@ -257,14 +256,13 @@ To disable deferred emitting for a particular service, you can set the `outbox` ### Delete Entries in the Tasks Table -To manually delete entries in the table `cds.core.Tasks`, you can either -expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.core.Tasks` +To manually delete entries in the table `cds.core.Queued.Messages`, you can either +expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.core.Queued.Messages` entity: ```js const db = await cds.connect.to('db') -const { Tasks } = db.entities('cds.core') -await DELETE.from(Tasks) +await DELETE.from('cds.core.Queued.Messages') ``` ### Tasks Table Not Found From aa72384eea882d4007a7e5946d1d82fbccd8dc30 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:43:53 +0200 Subject: [PATCH 11/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index d7b8372b12..72a02c8d9d 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -12,7 +12,7 @@ status: released ## Overview -The _task queue_ feature allows you to defer request processing. +The _task queue_ feature allows you to defer message processing. A common use case is the outbox pattern, where remote operations are deferred until the main transaction has been successfully committed. This prevents accidental execution of remote calls in case the transaction is rolled back. From 12ecfbf4b69eef92aacb454090ba807ec5a5fd24 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:44:07 +0200 Subject: [PATCH 12/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 72a02c8d9d..75d336feee 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -12,7 +12,7 @@ status: released ## Overview -The _task queue_ feature allows you to defer message processing. +The _task queue_ feature allows you to defer event processing. A common use case is the outbox pattern, where remote operations are deferred until the main transaction has been successfully committed. This prevents accidental execution of remote calls in case the transaction is rolled back. From ebe9f77f60e4fc53891ca7486dfc09d7abfb4444 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:45:13 +0200 Subject: [PATCH 13/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 75d336feee..d6cffcb0d8 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -36,7 +36,7 @@ await queued.send('someEvent', { some: 'message' }) // asynchronous ``` ::: tip -You still need to `await` these operations. In case of a persistent queue, messages are stored in the database, within the current transaction. +You still need to `await` these operations. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. ::: The `cds.queued` function can also be called with optional configuration options. From 95415271b0575bb8f0e1fe6f3facaa34615fcb90 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:46:13 +0200 Subject: [PATCH 14/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index d6cffcb0d8..5586e82427 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -45,7 +45,7 @@ The `cds.queued` function can also be called with optional configuration options const queued = cds.queued(srv, { kind: 'persistent-queue' }) ``` -> The persistent queue can only be used if it's enabled globally with `cds.requires.queue = true` because it requires a dedicated database table. +> The persistent queue can only be used if it's not disabled globally by `cds.requires.queue = false` because it requires a dedicated database table. ::: warning One-time configuration Once you queued a service, you cannot override its configuration options again. From 7c78c3a589665c80567a60bd466674fef8c71023 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:46:56 +0200 Subject: [PATCH 15/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 5586e82427..792efa50c6 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -54,7 +54,7 @@ Once you queued a service, you cannot override its configuration options again. ### cds.unqueued(srv) {.method} -Use this on an unqueued service to get back the original service: +Use this on a queued service to get back the original service: ```js const unqueued = cds.unqueued(srv) From 6c0b7dc120c9f3dd74f53d8bbaada039eadd40ca Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:47:15 +0200 Subject: [PATCH 16/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 792efa50c6..5230d6c350 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -79,7 +79,7 @@ You can also configure services to be outboxed by default: ``` ::: tip Queued by default -Some services are queued by default, these services include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. +Some services are outboxed by default, these services include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. ::: For transactional safety, you're encouraged to enable the [persistent queue](#persistent-queue). From 2b0a4064bf91e8af2c005b4449ef68a80d9e26e2 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 10:47:41 +0200 Subject: [PATCH 17/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 5230d6c350..7e72f5bf77 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -82,7 +82,7 @@ You can also configure services to be outboxed by default: Some services are outboxed by default, these services include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. ::: -For transactional safety, you're encouraged to enable the [persistent queue](#persistent-queue). +For transactional safety, you're encouraged to use the [persistent queue](#persistent-queue) which is enabled by default. ## Persistent Queue (Default) {#persistent-queue} From e3a08ffc30e6d3b8434ff28481a4c8e0bb784f52 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 11:47:58 +0200 Subject: [PATCH 18/60] apis --- node.js/queue.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 7e72f5bf77..2467318f94 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -133,13 +133,14 @@ After a maximum number of attempts, the message is ignored for processing and re therefore also acts as a dead letter queue. See [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), to learn about how to handle such messages. -There is only one active message processor per service, tenant, and app instance. +There is only one active message processor per service, tenant, app instance and message. Hence, there won't be duplicate emits except in the unlikely case of an app crash right after the emit and before the deletion of the message entry. ::: tip Some errors during the emit are identified as unrecoverable, for example in [SAP Event Mesh](../guides/messaging/event-mesh) if the used topic is forbidden. The respective message is then updated and the `attempts` field is set to `maxAttempts` to prevent further processing. [Programming errors](./best-practices#error-types) crash the server instance and must be fixed. +To mark your own errors as unrecoverable, you can set `unrecoverable = true` on the error object. ::: @@ -213,6 +214,33 @@ Finally, entries in the dead letter queue can either be _revived_ by resetting t <<< ./assets/dead-letter-queue-2.js#snippet{10-12,14-16} [srv/outbox-dead-letter-queue-service.js] ::: +#### Additional APIs + +To manually trigger the message processing, for example if your server is restarted, you can use the `flush` method. + +```js +const srv = await cds.connect.to('yourService') +cds.queued(srv).flush() // for current tenant +cds.queued(srv).flush(tenant) // for provided tenant +``` + +Once a message has been successfully processed, it will trigger the `/#succeeded` handlers. + +```js +srv.after('someEvent/#succeeded', (data, req) => { + // `data` is the result of the event processor + console.log('Message successfully processed:', data) +}) +``` + +Similarly, you can use the ` { + // `data` is the error from the event processor + console.log('Message could not be processed:', data) +}) +``` ## In-Memory Queue From 1f615da0f6378fdda532b3e4cc695522e89c61f6 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 11:48:08 +0200 Subject: [PATCH 19/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 2467318f94..d2711c7d57 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -233,7 +233,7 @@ srv.after('someEvent/#succeeded', (data, req) => { }) ``` -Similarly, you can use the `/#failed` event to handle failed messages (once the maximum retry count is reached). ```js srv.after('someEvent/#failed', (data, req) => { From 61ab993cb19de99add2a6be047d5a0ed368ed9cc Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Wed, 30 Apr 2025 16:10:42 +0200 Subject: [PATCH 20/60] use outboxed --- node.js/queue.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index d2711c7d57..430c55a317 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -72,7 +72,7 @@ You can also configure services to be outboxed by default: "requires": { "yourService": { "kind": "odata", - "outbox": true + "outboxed": true } } } @@ -267,14 +267,14 @@ The message is lost if its emit fails, there is no retry mechanism. ## Immediate Emit -To disable deferred emitting for a particular service, you can set the `outbox` option of your service to `false`: +To disable deferred emitting for a particular service, you can set the `outboxed` option of your service to `false`: ```json { "requires": { "messaging": { "kind": "enterprise-messaging", - "outbox": false + "outboxed": false } } } From 8717614e2560f4244fc79c4806f726b2fea9cae9 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Mon, 5 May 2025 14:12:18 +0200 Subject: [PATCH 21/60] reduced chunkSize --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 430c55a317..3abf2ca351 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -110,7 +110,7 @@ You can use the following configuration options: "queue": { "kind": "persistent-queue", "maxAttempts": 20, - "chunkSize": 100, + "chunkSize": 10, "storeLastError": true, "parallel": true } From de58e52c182b6b853cf5d0aa5d712673fc26f409 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Mon, 5 May 2025 14:12:27 +0200 Subject: [PATCH 22/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 3abf2ca351..02b1d27820 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -121,7 +121,7 @@ You can use the following configuration options: The optional parameters are: - `maxAttempts` (default `20`): The number of unsuccessful emits until the message is ignored. It will still remain in the database table. -- `chunkSize` (default `100`): The number of messages which are read from the database table in one go. +- `chunkSize` (default `10`): The number of messages which are read from the database table in one go. - `storeLastError` (default `true`): Specifies if error information of the last failed emit should be stored in the tasks table. - `parallel` (default `true`): Specifies if messages are sent in parallel (faster but the order isn't guaranteed). From d74d1ff7beb68f341666c98b662937b44af7424b Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Mon, 5 May 2025 15:45:45 +0200 Subject: [PATCH 23/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 02b1d27820..3a0087a4e1 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -121,7 +121,7 @@ You can use the following configuration options: The optional parameters are: - `maxAttempts` (default `20`): The number of unsuccessful emits until the message is ignored. It will still remain in the database table. -- `chunkSize` (default `10`): The number of messages which are read from the database table in one go. +- `chunkSize` (default `10`): The number of messages which are read from the database table in one go. Only applies for `parallel != false`. - `storeLastError` (default `true`): Specifies if error information of the last failed emit should be stored in the tasks table. - `parallel` (default `true`): Specifies if messages are sent in parallel (faster but the order isn't guaranteed). From cbbabb543fe19cc5c054ba2dfb1486ff054f308c Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 09:40:55 +0200 Subject: [PATCH 24/60] revert name --- node.js/queue.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 3a0087a4e1..f1f7184cdf 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -144,7 +144,7 @@ To mark your own errors as unrecoverable, you can set `unrecoverable = true` on ::: -Your database model is automatically extended by the entity `cds.core.Queued.Messages`: +Your database model is automatically extended by the entity `cds.outbox.Messages`: ```cds namespace cds.core; @@ -160,7 +160,7 @@ entity Queued.Messages { } ``` -In your CDS model, you can refer to the entity `cds.core.Queued.Messages` using the path `@sap/cds/srv/queue`, +In your CDS model, you can refer to the entity `cds.outbox.Messages` using the path `@sap/cds/srv/outbox`, for example to expose it in a service. @@ -172,7 +172,7 @@ for example to expose it in a service. ### Managing the Dead Letter Queue -You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.core.Queued.Messages` as well as bound actions to either revive or delete the respective message. +You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.outbox.Messages` as well as bound actions to either revive or delete the respective message. ::: tip Please see [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general Outbox guide. @@ -182,13 +182,13 @@ Please see [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) i ::: code-group ```cds [srv/outbox-dead-letter-queue-service.cds] -using from '@sap/cds/srv/queue'; +using from '@sap/cds/srv/outbox'; @requires: 'internal-user' service OutboxDeadLetterQueueService { @readonly - entity DeadOutboxMessages as projection on cds.core.Queued.Messages + entity DeadOutboxMessages as projection on cds.outbox.Messages actions { action revive(); action delete(); @@ -284,25 +284,25 @@ To disable deferred emitting for a particular service, you can set the `outboxed ### Delete Entries in the Tasks Table -To manually delete entries in the table `cds.core.Queued.Messages`, you can either -expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.core.Queued.Messages` +To manually delete entries in the table `cds.outbox.Messages`, you can either +expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.outbox.Messages` entity: ```js const db = await cds.connect.to('db') -await DELETE.from('cds.core.Queued.Messages') +await DELETE.from('cds.outbox.Messages') ``` ### Tasks Table Not Found If the tasks table is not found on the database, this can be caused by insufficient configuration data in _package.json_. -In case you have overwritten `requires.db.model` there, make sure to add the queue model path `@sap/cds/srv/queue`: +In case you have overwritten `requires.db.model` there, make sure to add the queue model path `@sap/cds/srv/outbox`: ```jsonc "requires": { "db": { ... - "model": [..., "@sap/cds/srv/queue"] + "model": [..., "@sap/cds/srv/outbox"] } } ``` From cead605c20cb18fd00fe75094b04754133343593 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 09:42:09 +0200 Subject: [PATCH 25/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index f1f7184cdf..d286420630 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -42,7 +42,7 @@ You still need to `await` these operations. In case of a persistent queue, which The `cds.queued` function can also be called with optional configuration options. ```js -const queued = cds.queued(srv, { kind: 'persistent-queue' }) +const queued = cds.queued(srv, { maxAttempts: 5 }) ``` > The persistent queue can only be used if it's not disabled globally by `cds.requires.queue = false` because it requires a dedicated database table. From ec923c28cd8f91a2918edcb6b5308ea4085bff9d Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 09:44:26 +0200 Subject: [PATCH 26/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index d286420630..6dbf5b8067 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -78,7 +78,7 @@ You can also configure services to be outboxed by default: } ``` -::: tip Queued by default +::: tip Outboxed by default Some services are outboxed by default, these services include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. ::: From 2dfdf324684090f1c80e4c5cb9e435a55aeed7c2 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 09:57:27 +0200 Subject: [PATCH 27/60] . --- node.js/queue.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 6dbf5b8067..3e9907b319 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -99,8 +99,7 @@ You can disable it globally with: } ``` -Using the persistent queue, the to-be-emitted message is stored in a database table first. The same database transaction is used -as for other operations, therefore transactional consistency is guaranteed. +Using the persistent queue, the to-be-emitted message is stored in a database table within the current transaction, therefore transactional consistency is guaranteed. You can use the following configuration options: @@ -112,7 +111,9 @@ You can use the following configuration options: "maxAttempts": 20, "chunkSize": 10, "storeLastError": true, - "parallel": true + "parallel": true, + "timeout": "1h", + "legacyLocking": true } } } @@ -124,6 +125,8 @@ The optional parameters are: - `chunkSize` (default `10`): The number of messages which are read from the database table in one go. Only applies for `parallel != false`. - `storeLastError` (default `true`): Specifies if error information of the last failed emit should be stored in the tasks table. - `parallel` (default `true`): Specifies if messages are sent in parallel (faster but the order isn't guaranteed). +- `timeout` (default `"1h"`): The time after which a message with `status = "processing"` can be processed again. Only for `legacyLocking = false`. +- `legacyLocking` (default `true`): If set to `false`, database locks are only used to set the status of the message to `processing` to prevent long-kept database locks. This is recommended but incompatible for parallel usage with `@sap/cds^8` instances. Once the transaction succeeds, the messages are read from the database table and emitted. From c92d57afb9f4a252765fb5b00b39f66325524d4f Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 10:10:26 +0200 Subject: [PATCH 28/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 3e9907b319..1b51f90fc4 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -150,7 +150,7 @@ To mark your own errors as unrecoverable, you can set `unrecoverable = true` on Your database model is automatically extended by the entity `cds.outbox.Messages`: ```cds -namespace cds.core; +namespace cds.outbox; entity Queued.Messages { key ID : UUID; From 4b78fc0ff4993e7778d7a0f8c4bed28039d33fa6 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 10:10:53 +0200 Subject: [PATCH 29/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 1b51f90fc4..53af0ae8f1 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -152,7 +152,7 @@ Your database model is automatically extended by the entity `cds.outbox.Messages ```cds namespace cds.outbox; -entity Queued.Messages { +entity Messages { key ID : UUID; timestamp : Timestamp; target : String; From 390b7cdca43e99a9d0c9e10c1098357da38f29d4 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:12:32 +0200 Subject: [PATCH 30/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 53af0ae8f1..46c83b4f6e 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -17,7 +17,7 @@ The _task queue_ feature allows you to defer event processing. A common use case is the outbox pattern, where remote operations are deferred until the main transaction has been successfully committed. This prevents accidental execution of remote calls in case the transaction is rolled back. -Every CAP service can be _queued_ that means event dispatching becomes _asynchronous_. +Every CAP service can be _queued_, meaning that event dispatching becomes _asynchronous_. ## Queueing a Service From fe8bb854540427e28db77f6ac8bbd3452cd4b6a0 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:14:29 +0200 Subject: [PATCH 31/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 46c83b4f6e..4d51d0113d 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -36,7 +36,7 @@ await queued.send('someEvent', { some: 'message' }) // asynchronous ``` ::: tip -You still need to `await` these operations. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. +You still need to `await` these operations becase they are asynchronous. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. ::: The `cds.queued` function can also be called with optional configuration options. From b27ee41a8b1df467e7693da15cd30301a9f27701 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:15:38 +0200 Subject: [PATCH 32/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 4d51d0113d..5c6148cae1 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -79,7 +79,7 @@ You can also configure services to be outboxed by default: ``` ::: tip Outboxed by default -Some services are outboxed by default, these services include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. +Some services are outboxed by default; these include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. ::: For transactional safety, you're encouraged to use the [persistent queue](#persistent-queue) which is enabled by default. From c0bbf8fc6555cfd4e7c7945a3cb39fbf4c644ca8 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:17:19 +0200 Subject: [PATCH 33/60] . --- node.js/queue.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 5c6148cae1..23bb42e342 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -131,13 +131,13 @@ The optional parameters are: Once the transaction succeeds, the messages are read from the database table and emitted. If an emit was successful, the respective message is deleted from the database table. -If not, there will be retries after (exponentially growing) waiting times. +If not, the system retries the message after exponentially increasing delays. After a maximum number of attempts, the message is ignored for processing and remains in the database table which therefore also acts as a dead letter queue. See [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), to learn about how to handle such messages. There is only one active message processor per service, tenant, app instance and message. -Hence, there won't be duplicate emits except in the unlikely case of an app crash right after the emit and before the deletion of the message entry. +This ensures no duplicate emits, except in the unlikely case of an app crash right after the emit and before the message is deleted. ::: tip Some errors during the emit are identified as unrecoverable, for example in [SAP Event Mesh](../guides/messaging/event-mesh) if the used topic is forbidden. From 698d3b0997fe455d7f9988dcc9a944dda9a083f7 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:17:55 +0200 Subject: [PATCH 34/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 23bb42e342..fdd1a62b61 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -264,7 +264,7 @@ This is similar to the following code if done manually: cds.context.on('succeeded', () => this.emit(msg)) ``` ::: warning No retry mechanism -The message is lost if its emit fails, there is no retry mechanism. +The message is lost if the emit fails. There is no retry mechanism. ::: From c0521df3353d94204f1755a2b901af1306e824bc Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:18:13 +0200 Subject: [PATCH 35/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index fdd1a62b61..300980946f 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -178,7 +178,7 @@ for example to expose it in a service. You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.outbox.Messages` as well as bound actions to either revive or delete the respective message. ::: tip -Please see [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general Outbox guide. +See [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general Outbox guide. ::: #### 1. Define the Service From 245f3c8ee1fb0d6fddeb1695d27b17c3343f46af Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:22:22 +0200 Subject: [PATCH 36/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 300980946f..d0ff198a58 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -258,7 +258,7 @@ You can enable it globally for all queued services with: } } ``` -Messages are emitted when the current transaction is successful. Until then, messages are only kept in memory. +Messages are emitted only when the current transaction is successful. Until then, messages are only kept in memory. This is similar to the following code if done manually: ```js cds.context.on('succeeded', () => this.emit(msg)) From ab289bb4662be0debb482675941dfe89a78fc8e4 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:22:36 +0200 Subject: [PATCH 37/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index d0ff198a58..1ffcb9f022 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -258,7 +258,7 @@ You can enable it globally for all queued services with: } } ``` -Messages are emitted only when the current transaction is successful. Until then, messages are only kept in memory. +Messages are emitted only when the current transaction is successfully committed. Until then, messages are only kept in memory. This is similar to the following code if done manually: ```js cds.context.on('succeeded', () => this.emit(msg)) From 49a57bff6daeeb44cd919472f04836579864716c Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:27:28 +0200 Subject: [PATCH 38/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 1ffcb9f022..01dcdc5d86 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -258,7 +258,7 @@ You can enable it globally for all queued services with: } } ``` -Messages are emitted only when the current transaction is successfully committed. Until then, messages are only kept in memory. +Messages are emitted only after the current transaction is successfully committed. Until then, messages are only kept in memory. This is similar to the following code if done manually: ```js cds.context.on('succeeded', () => this.emit(msg)) From 0f9961e6ed72d8523d615676c3d9bb140913c40f Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:28:01 +0200 Subject: [PATCH 39/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 01dcdc5d86..8e3f86d3bb 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -321,4 +321,4 @@ Add the model path accordingly: } ``` -Note that model configuration isn't required for CAP projects using the [standard project layout](../get-started/#project-structure) that contain the folders `db`, `srv`, and `app`. In this case, you can delete the entire `model` configuration. +Note that model configuration isn't required for CAP projects using the [standard project layout](../get-started/#project-structure) with `db`, `srv`, and `app` folders. In this case, you can delete the entire `model` configuration. From e17b4c0042db3e88860be0b9781ba0f568265913 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:50:44 +0200 Subject: [PATCH 40/60] . --- node.js/queue.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 8e3f86d3bb..aeb76ef356 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -65,23 +65,22 @@ This is useful if your service is outboxed (i.e., queued) per configuration. ### Per Configuration -You can also configure services to be outboxed by default: +Some services are outboxed by default; these include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. +You can configure the outbox behavior by specifying the `outboxed` option in your service configuration. ```json { "requires": { "yourService": { "kind": "odata", - "outboxed": true + "outboxed": { + "maxAttempts": 5 + } } } } ``` -::: tip Outboxed by default -Some services are outboxed by default; these include [`cds.MessagingService`](messaging) and `cds.AuditLogService`. -::: - For transactional safety, you're encouraged to use the [persistent queue](#persistent-queue) which is enabled by default. From 89dda8274e35b27daba852d8952f62b3410c6f51 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:52:07 +0200 Subject: [PATCH 41/60] . --- node.js/queue.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index aeb76ef356..0b6ed7a22b 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -128,8 +128,8 @@ The optional parameters are: - `legacyLocking` (default `true`): If set to `false`, database locks are only used to set the status of the message to `processing` to prevent long-kept database locks. This is recommended but incompatible for parallel usage with `@sap/cds^8` instances. -Once the transaction succeeds, the messages are read from the database table and emitted. -If an emit was successful, the respective message is deleted from the database table. +Once the transaction succeeds, the messages are read from the database table and dispatched. +If it was successful, the respective message is deleted from the database table. If not, the system retries the message after exponentially increasing delays. After a maximum number of attempts, the message is ignored for processing and remains in the database table which therefore also acts as a dead letter queue. From 4cab9c5e5db0f2777a0756457a1a2d129e6b8b60 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 13:53:42 +0200 Subject: [PATCH 42/60] . --- node.js/queue.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/node.js/queue.md b/node.js/queue.md index 0b6ed7a22b..8449a9689b 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -157,8 +157,10 @@ entity Messages { target : String; msg : LargeString; attempts : Integer default 0; + partition : Integer default 0; lastError : LargeString; lastAttemptTimestamp : Timestamp @cds.on.update: $now; + status : String(23); } ``` From 9934a4d1fae632d50cb91c7169604ecc959fc282 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:11:51 +0200 Subject: [PATCH 43/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 8449a9689b..d47646c002 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -170,7 +170,7 @@ for example to expose it in a service. #### Known Limitations -- If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. +- If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. It can be triggered manually using the `flush` method. - The service that handles the queued event must not use user roles and attributes as they are not stored. However, the user ID is stored to recreate the correct context. From 9f03a1df9a7b7f087ee4ef1a0dc787de5dcbe143 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:14:33 +0200 Subject: [PATCH 44/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index d47646c002..f1705fb139 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -179,7 +179,7 @@ for example to expose it in a service. You can manage the dead letter queue by implementing a service that exposes a read-only projection on entity `cds.outbox.Messages` as well as bound actions to either revive or delete the respective message. ::: tip -See [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general Outbox guide. +See [Outbox Dead Letter Queue](../java/outbox#outbox-dead-letter-queue) in the CAP Java documentation for additional considerations while we work on a general outbox guide. ::: #### 1. Define the Service From a924f8a78db434e21a633db8674f0db3752e21e6 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:16:52 +0200 Subject: [PATCH 45/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index f1705fb139..cd01da7175 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -204,7 +204,7 @@ service OutboxDeadLetterQueueService { #### 2. Filter for Dead Entries -As `maxAttempts` is configurable, its value can not be added as a static filter to projection `DeadOutboxMessages`, but must be considered programmatically. +As `maxAttempts` is configurable, its value cannot be added as a static filter to projection `DeadOutboxMessages`, but must be considered programmatically. ::: code-group <<< ./assets/dead-letter-queue-1.js#snippet{5-8} [srv/outbox-dead-letter-queue-service.js] From 8444e0ab6b81cb8d9b36870925334744601c31a7 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:19:25 +0200 Subject: [PATCH 46/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index cd01da7175..00530de36b 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -218,7 +218,7 @@ Finally, entries in the dead letter queue can either be _revived_ by resetting t <<< ./assets/dead-letter-queue-2.js#snippet{10-12,14-16} [srv/outbox-dead-letter-queue-service.js] ::: -#### Additional APIs +### Additional APIs To manually trigger the message processing, for example if your server is restarted, you can use the `flush` method. From 8723b0505bf1050d64e3b6823a71e9a3f7af83f9 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:20:18 +0200 Subject: [PATCH 47/60] . --- node.js/queue.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 00530de36b..35889cc45c 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -224,8 +224,7 @@ To manually trigger the message processing, for example if your server is restar ```js const srv = await cds.connect.to('yourService') -cds.queued(srv).flush() // for current tenant -cds.queued(srv).flush(tenant) // for provided tenant +cds.queued(srv).flush() ``` Once a message has been successfully processed, it will trigger the `/#succeeded` handlers. From a26599091680dca1c0f71e5c0e384d1584b28d21 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:22:05 +0200 Subject: [PATCH 48/60] . --- node.js/queue.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/node.js/queue.md b/node.js/queue.md index 35889cc45c..dfcd132ed2 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -245,6 +245,10 @@ srv.after('someEvent/#failed', (data, req) => { }) ``` +::: tip +Event handlers have to be registered for these specific events. They are excluded from the `*` wildcard handler. +::: + ## In-Memory Queue You can enable it globally for all queued services with: From 9d98371c704076395ad8a5a369efe062baa88cb0 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:24:36 +0200 Subject: [PATCH 49/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index dfcd132ed2..1ed32fa936 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -246,7 +246,7 @@ srv.after('someEvent/#failed', (data, req) => { ``` ::: tip -Event handlers have to be registered for these specific events. They are excluded from the `*` wildcard handler. +Event handlers have to be registered for these specific events. The `*` wildcard handler will not be called for these. ::: ## In-Memory Queue From 1c4a3c8a6c3b3d6bef35f42872024d35f417c5f3 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 16 May 2025 14:25:05 +0200 Subject: [PATCH 50/60] . --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 1ed32fa936..de4634da7f 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -251,7 +251,7 @@ Event handlers have to be registered for these specific events. The `*` wildcard ## In-Memory Queue -You can enable it globally for all queued services with: +You can enable the in-memory queue globally with: ```json { From e97d6437ec88975ac693adf49ba6e6af5e979284 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 22 May 2025 16:11:14 +0200 Subject: [PATCH 51/60] merge --- guides/deployment/custom-builds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/deployment/custom-builds.md b/guides/deployment/custom-builds.md index 7bbdecd977..0fb76a9dcc 100644 --- a/guides/deployment/custom-builds.md +++ b/guides/deployment/custom-builds.md @@ -22,7 +22,7 @@ The CDS model folders and files used by `cds build` are determined as follows: - Known root folders, by [default](../../get-started/#project-structure) the folders _db/, srv/, app/_, can also be configured by [`folders.db`, `folders.srv`, `folders.app`](../../get-started/#project-structure). - The _src_ folder configured for the individual build task. - If [feature toggles](../extensibility/feature-toggles#enable-feature-toggles) are enabled: subfolders of the folder _fts_. -- CDS Model folders and files defined by the [required services](../../node.js/cds-env#services) of your project. This also includes models used by required built-in CDS services, like [persistent outbox](../../node.js/outbox#persistent-outbox) or [MTX related services](../multitenancy/mtxs#mtx-services-reference). +- CDS Model folders and files defined by the [required services](../../node.js/cds-env#services) of your project. This also includes models used by required built-in CDS services, like [persistent queue](../../node.js/queue#persistent-queue) or [MTX related services](../multitenancy/mtxs#mtx-services-reference). Feature toggle folders and required built-in service models will also be added if user defined models have already been configured as [_model_ option](#build-task-properties) in your build tasks. From df7e58fbc5900c261fde6ff27dcae5eef76ef793 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Thu, 22 May 2025 16:20:23 +0200 Subject: [PATCH 52/60] . --- node.js/messaging.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/node.js/messaging.md b/node.js/messaging.md index e5ad4abc81..8615996f35 100644 --- a/node.js/messaging.md +++ b/node.js/messaging.md @@ -160,6 +160,24 @@ messaging.on('*', async msg => { /*...*/ }) In general, messages do not contain user information but operate with a technical user. As a consequence, the user of the message processing context (`cds.context.user`) is set to [`cds.User.privileged`](/node.js/authentication#privileged-user) and, hence, any necessary authorization checks must be done in custom handlers. ::: +### Inbox + +You can store received messages in an inbox before they're processed. Under the hood, it uses the [task queue](./queue) for reliable asynchronous processing. +Enable it by setting the `inboxed` option to `true`, for example: + +```js +{ + cds: { + requires: { + messaging: { + kind: 'enterprise-messaging', + inboxed: true + } + } + } +} +``` + ## CloudEvents Protocol [CloudEvents](https://cloudevents.io/) is a commonly used specification for describing event data. From 1680be3f9ddd260fd8bb20cd0f1023079c0d3dd2 Mon Sep 17 00:00:00 2001 From: "Dr. David A. Kunz" Date: Fri, 23 May 2025 11:18:32 +0200 Subject: [PATCH 53/60] spelling --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index de4634da7f..0cee57be81 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -36,7 +36,7 @@ await queued.send('someEvent', { some: 'message' }) // asynchronous ``` ::: tip -You still need to `await` these operations becase they are asynchronous. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. +You still need to `await` these operations because they are asynchronous. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. ::: The `cds.queued` function can also be called with optional configuration options. From f8308c5120896702da52810915f29cf2e6d6ee85 Mon Sep 17 00:00:00 2001 From: Rene Jeglinsky Date: Fri, 23 May 2025 16:21:33 +0200 Subject: [PATCH 54/60] fix link --- guides/messaging/task-queues.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/messaging/task-queues.md b/guides/messaging/task-queues.md index 568bfe9120..6642ab178c 100644 --- a/guides/messaging/task-queues.md +++ b/guides/messaging/task-queues.md @@ -26,7 +26,7 @@ This guide will grow with the functionality. Regarding the _outbox_, please see the following existing documentation: - [Transactional Outbox](../../java/outbox) in CAP Java -- [Outboxing with `cds.outboxed`](../../node.js/outbox) in CAP Node.js +- [Outboxing with `cds.queued`](../../node.js/queue) in CAP Node.js From 813e2dd4d61d1654d5e913a19d7631f58dd2a000 Mon Sep 17 00:00:00 2001 From: Rene Jeglinsky Date: Fri, 23 May 2025 16:22:57 +0200 Subject: [PATCH 55/60] fix link --- guides/messaging/task-queues.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/messaging/task-queues.md b/guides/messaging/task-queues.md index 6642ab178c..49953114bc 100644 --- a/guides/messaging/task-queues.md +++ b/guides/messaging/task-queues.md @@ -42,7 +42,7 @@ Simply configure your messaging service for Node.js as cds.requires.mess With the inbox, all messages are acknowledged towards the message broker regardless of whether they can be processed or not. Hence, failures need to be managed via the dead letter queue built on `cds.outbox.Messages`. -[Learn more about the dead letter queue in Node.js.](../../node.js/outbox#managing-the-dead-letter-queue){.learn-more} +[Learn more about the dead letter queue in Node.js.](../../node.js/queue#managing-the-dead-letter-queue){.learn-more} [Learn more about the dead letter queue in Java.](../../java/outbox#outbox-dead-letter-queue){.learn-more} Inboxing is especially beneficial in case the message broker does not allow to trigger redelivery and/ or "fix" the message payload. From 0828b87d93c5b174d9eaa43a37557659b32a3c67 Mon Sep 17 00:00:00 2001 From: Rene Jeglinsky Date: Mon, 26 May 2025 10:24:48 +0200 Subject: [PATCH 56/60] details note for config --- node.js/queue.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 0cee57be81..6b0a7c9c20 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -23,7 +23,7 @@ Every CAP service can be _queued_, meaning that event dispatching becomes _async ## Queueing a Service -### cds.queued(srv) {.method} +### cds. queued (srv) {.method} Programmatically, you can get the queued service as follows: @@ -52,7 +52,7 @@ Once you queued a service, you cannot override its configuration options again. ::: -### cds.unqueued(srv) {.method} +### cds. unqueued (srv) {.method} Use this on a queued service to get back the original service: @@ -100,7 +100,7 @@ You can disable it globally with: Using the persistent queue, the to-be-emitted message is stored in a database table within the current transaction, therefore transactional consistency is guaranteed. -You can use the following configuration options: +::: details You can use the following configuration options: ```json { @@ -127,6 +127,7 @@ The optional parameters are: - `timeout` (default `"1h"`): The time after which a message with `status = "processing"` can be processed again. Only for `legacyLocking = false`. - `legacyLocking` (default `true`): If set to `false`, database locks are only used to set the status of the message to `processing` to prevent long-kept database locks. This is recommended but incompatible for parallel usage with `@sap/cds^8` instances. +::: Once the transaction succeeds, the messages are read from the database table and dispatched. If it was successful, the respective message is deleted from the database table. From cc998c3c472506fd6f9164b9827e936072d0b57d Mon Sep 17 00:00:00 2001 From: Rene Jeglinsky Date: Mon, 26 May 2025 11:26:37 +0200 Subject: [PATCH 57/60] edit note titles --- node.js/queue.md | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 6b0a7c9c20..172530a555 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -5,7 +5,7 @@ synopsis: > status: released --- -# Queueing with `cds.queued` +# Queuing with `cds.queued` [[toc]] @@ -20,7 +20,7 @@ This prevents accidental execution of remote calls in case the transaction is ro Every CAP service can be _queued_, meaning that event dispatching becomes _asynchronous_. -## Queueing a Service +## Queuing a Service ### cds. queued (srv) {.method} @@ -35,8 +35,8 @@ await queued.emit('someEvent', { some: 'message' }) // asynchronous await queued.send('someEvent', { some: 'message' }) // asynchronous ``` -::: tip -You still need to `await` these operations because they are asynchronous. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. +::: tip `await` needed +You still need to `await` these operations because they're asynchronous. In case of a persistent queue, which is the default, messages are stored in the database, within the current transaction. ::: The `cds.queued` function can also be called with optional configuration options. @@ -54,13 +54,13 @@ Once you queued a service, you cannot override its configuration options again. ### cds. unqueued (srv) {.method} -Use this on a queued service to get back the original service: +Use this on a queued service to get back to the original service: ```js const unqueued = cds.unqueued(srv) ``` -This is useful if your service is outboxed (i.e., queued) per configuration. +This is useful if your service is outboxed (that is, queued) per configuration. ### Per Configuration @@ -120,9 +120,9 @@ Using the persistent queue, the to-be-emitted message is stored in a database ta The optional parameters are: -- `maxAttempts` (default `20`): The number of unsuccessful emits until the message is ignored. It will still remain in the database table. +- `maxAttempts` (default `20`): The number of unsuccessful emits until the message is ignored. It will remain in the database table. - `chunkSize` (default `10`): The number of messages which are read from the database table in one go. Only applies for `parallel != false`. -- `storeLastError` (default `true`): Specifies if error information of the last failed emit should be stored in the tasks table. +- `storeLastError` (default `true`): Specifies if error information of the last failed emit is stored in the tasks table. - `parallel` (default `true`): Specifies if messages are sent in parallel (faster but the order isn't guaranteed). - `timeout` (default `"1h"`): The time after which a message with `status = "processing"` can be processed again. Only for `legacyLocking = false`. - `legacyLocking` (default `true`): If set to `false`, database locks are only used to set the status of the message to `processing` to prevent long-kept database locks. This is recommended but incompatible for parallel usage with `@sap/cds^8` instances. @@ -136,10 +136,10 @@ After a maximum number of attempts, the message is ignored for processing and re therefore also acts as a dead letter queue. See [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), to learn about how to handle such messages. -There is only one active message processor per service, tenant, app instance and message. -This ensures no duplicate emits, except in the unlikely case of an app crash right after the emit and before the message is deleted. +There's only one active message processor per service, tenant, app instance, and message. +This ensures that no duplicate emits happen, except in the unlikely case of an app crash right after the emit and before the message is deleted. -::: tip +::: tip Unrecoverable errors Some errors during the emit are identified as unrecoverable, for example in [SAP Event Mesh](../guides/messaging/event-mesh) if the used topic is forbidden. The respective message is then updated and the `attempts` field is set to `maxAttempts` to prevent further processing. [Programming errors](./best-practices#error-types) crash the server instance and must be fixed. @@ -172,8 +172,9 @@ for example to expose it in a service. #### Known Limitations - If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. It can be triggered manually using the `flush` method. -- The service that handles the queued event must not use user roles and attributes as they are not stored. However, the user ID is stored to recreate the correct context. +- The service that handles the queued event must not use user roles and attributes as they are not stored. However, the user ID is stored to re-create the correct context. +### Disable Persistent Queue ### Managing the Dead Letter Queue @@ -228,7 +229,7 @@ const srv = await cds.connect.to('yourService') cds.queued(srv).flush() ``` -Once a message has been successfully processed, it will trigger the `/#succeeded` handlers. +Once a message has been successfully processed, it triggers the `/#succeeded` handlers. ```js srv.after('someEvent/#succeeded', (data, req) => { @@ -246,8 +247,8 @@ srv.after('someEvent/#failed', (data, req) => { }) ``` -::: tip -Event handlers have to be registered for these specific events. The `*` wildcard handler will not be called for these. +::: tip Register on specific events +Event handlers have to be registered for these specific events. The `*` wildcard handler is not called for these. ::: ## In-Memory Queue @@ -269,7 +270,7 @@ This is similar to the following code if done manually: cds.context.on('succeeded', () => this.emit(msg)) ``` ::: warning No retry mechanism -The message is lost if the emit fails. There is no retry mechanism. +The message is lost if the emit fails. There's no retry mechanism. ::: From a77e0225413837101d12f7251296640cddf265fc Mon Sep 17 00:00:00 2001 From: sjvans <30337871+sjvans@users.noreply.github.com> Date: Tue, 27 May 2025 22:55:45 +0200 Subject: [PATCH 58/60] cds 9: task queue - review (#1878) Co-authored-by: Dr. David A. Kunz --- node.js/queue.md | 121 ++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 48 deletions(-) diff --git a/node.js/queue.md b/node.js/queue.md index 172530a555..05de8daa8e 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -1,15 +1,15 @@ --- synopsis: > - Learn details about the task-queue feature. -# layout: node-js + Learn details about the task queue feature. status: released --- -# Queuing with `cds.queued` +# Queueing with `cds.queued` [[toc]] + ## Overview The _task queue_ feature allows you to defer event processing. @@ -17,22 +17,30 @@ The _task queue_ feature allows you to defer event processing. A common use case is the outbox pattern, where remote operations are deferred until the main transaction has been successfully committed. This prevents accidental execution of remote calls in case the transaction is rolled back. -Every CAP service can be _queued_, meaning that event dispatching becomes _asynchronous_. +Every non-database CAP service can be _queued_, meaning that event dispatching becomes _asynchronous_. +::: tip +The _task queue_ feature can be disabled globally via cds.requires.queue = false. +::: -## Queuing a Service + +## Queueing a Service ### cds. queued (srv) {.method} +```tsx +function cds.queued ( srv: Service, options? ) => QueuedService +``` + Programmatically, you can get the queued service as follows: ```js const srv = await cds.connect.to('yourService') -const queued = cds.queued(srv) +const qd_srv = cds.queued(srv) -await queued.emit('someEvent', { some: 'message' }) // asynchronous -await queued.send('someEvent', { some: 'message' }) // asynchronous +await qd_srv.emit('someEvent', { some: 'message' }) // asynchronous +await qd_srv.send('someEvent', { some: 'message' }) // asynchronous ``` ::: tip `await` needed @@ -42,26 +50,34 @@ You still need to `await` these operations because they're asynchronous. In case The `cds.queued` function can also be called with optional configuration options. ```js -const queued = cds.queued(srv, { maxAttempts: 5 }) +const qd_srv = cds.queued(srv, { maxAttempts: 5 }) ``` -> The persistent queue can only be used if it's not disabled globally by `cds.requires.queue = false` because it requires a dedicated database table. +> The persistent queue can only be used if it is not disabled globally via `cds.requires.queue = false`, as it requires a dedicated database table. ::: warning One-time configuration Once you queued a service, you cannot override its configuration options again. ::: +For backwards compatibility, `cds.outboxed(srv)` works as a synonym. + ### cds. unqueued (srv) {.method} +```tsx +function cds.unqueued ( srv: QueuedService ) => Service +``` + Use this on a queued service to get back to the original service: ```js -const unqueued = cds.unqueued(srv) +const srv = cds.unqueued(qd_srv) ``` This is useful if your service is outboxed (that is, queued) per configuration. +For backwards compatibility, `cds.unboxed(srv)` works as a synonym. + ### Per Configuration @@ -81,22 +97,13 @@ You can configure the outbox behavior by specifying the `outboxed` option in you } ``` -For transactional safety, you're encouraged to use the [persistent queue](#persistent-queue) which is enabled by default. +For transactional safety, you're encouraged to use the [persistent queue](#persistent-queue), which is enabled by default. -## Persistent Queue (Default) {#persistent-queue} - -The persistent queue is enabled by default. -You can disable it globally with: +## Persistent Queue (Default) {#persistent-queue} -```json -{ - "requires": { - "queue": false - } -} -``` +The persistent queue is the default configuration. Using the persistent queue, the to-be-emitted message is stored in a database table within the current transaction, therefore transactional consistency is guaranteed. @@ -108,11 +115,11 @@ Using the persistent queue, the to-be-emitted message is stored in a database ta "queue": { "kind": "persistent-queue", "maxAttempts": 20, + "parallel": true, "chunkSize": 10, "storeLastError": true, - "parallel": true, - "timeout": "1h", - "legacyLocking": true + "legacyLocking": true, + "timeout": "1h" } } } @@ -120,24 +127,24 @@ Using the persistent queue, the to-be-emitted message is stored in a database ta The optional parameters are: -- `maxAttempts` (default `20`): The number of unsuccessful emits until the message is ignored. It will remain in the database table. -- `chunkSize` (default `10`): The number of messages which are read from the database table in one go. Only applies for `parallel != false`. -- `storeLastError` (default `true`): Specifies if error information of the last failed emit is stored in the tasks table. -- `parallel` (default `true`): Specifies if messages are sent in parallel (faster but the order isn't guaranteed). -- `timeout` (default `"1h"`): The time after which a message with `status = "processing"` can be processed again. Only for `legacyLocking = false`. -- `legacyLocking` (default `true`): If set to `false`, database locks are only used to set the status of the message to `processing` to prevent long-kept database locks. This is recommended but incompatible for parallel usage with `@sap/cds^8` instances. +- `maxAttempts` (default `20`): The number of unsuccessful emits until the message is considered unprocessable. The message will remain in the database table! +- `parallel` (default `true`): Specifies if messages are sent in parallel (faster, but the order isn't guaranteed). +- `chunkSize` (default `10`): The number of messages that are read from the database table in one go. Only applies for `parallel !== false`. +- `storeLastError` (default `true`): Specifies whether error information of the last failed emit is stored in the tasks table. +- `legacyLocking` (default `true`): If set to `false`, database locks are only used to set the status of the message to `processing` to prevent long-kept database locks. Although this is the recommended approach, it is incompatible with task runners still on `@sap/cds^8`. +- `timeout` (default `"1h"`): The time after which a message with `status === "processing"` is considered to be abandoned and eligable to be processed again. Only for `legacyLocking === false`. ::: Once the transaction succeeds, the messages are read from the database table and dispatched. -If it was successful, the respective message is deleted from the database table. -If not, the system retries the message after exponentially increasing delays. -After a maximum number of attempts, the message is ignored for processing and remains in the database table which +If processing was successful, the respective message is deleted from the database table. +If processing failed, the system retries the message after exponentially increasing delays. +After a maximum number of attempts, the message is ignored for processing and remains in the database, which therefore also acts as a dead letter queue. See [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), to learn about how to handle such messages. -There's only one active message processor per service, tenant, app instance, and message. -This ensures that no duplicate emits happen, except in the unlikely case of an app crash right after the emit and before the message is deleted. +There is only one active message processor per service, tenant, app instance, and message. +This ensures that no duplicate emits happen, except in the highly unlikely case of an app crash right after successful processing but before the message could be deleted. ::: tip Unrecoverable errors Some errors during the emit are identified as unrecoverable, for example in [SAP Event Mesh](../guides/messaging/event-mesh) if the used topic is forbidden. @@ -165,16 +172,14 @@ entity Messages { } ``` -In your CDS model, you can refer to the entity `cds.outbox.Messages` using the path `@sap/cds/srv/outbox`, -for example to expose it in a service. +In your CDS model, you can refer to the entity `cds.outbox.Messages` using the path `@sap/cds/srv/outbox`, for example to expose it in a service (cf. [Managing the Dead Letter Queue](#managing-the-dead-letter-queue)). -#### Known Limitations +### Known Limitations - If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. It can be triggered manually using the `flush` method. -- The service that handles the queued event must not use user roles and attributes as they are not stored. However, the user ID is stored to re-create the correct context. +- The service that handles the queued event must not rely on user roles and attributes, as they are not stored with the message. In other words, asynchroneous task are always processed in a priviledged mode. However, the user ID is stored to re-create the correct context. -### Disable Persistent Queue ### Managing the Dead Letter Queue @@ -220,7 +225,17 @@ Finally, entries in the dead letter queue can either be _revived_ by resetting t <<< ./assets/dead-letter-queue-2.js#snippet{10-12,14-16} [srv/outbox-dead-letter-queue-service.js] ::: -### Additional APIs + +### Additional APIs + +You can use the `schedule` method as a shortcut for `cds.queued(srv).send()`, with optional scheduling options `after` and `every`: + +```js +await srv.schedule('someEvent', { some: 'message' }) +await srv.schedule('someEvent', { some: 'message' }).after('1h') // after one hour +await srv.schedule('someEvent', { some: 'message' }).every('1h') // every hour after each processing +``` + To manually trigger the message processing, for example if your server is restarted, you can use the `flush` method. @@ -251,6 +266,8 @@ srv.after('someEvent/#failed', (data, req) => { Event handlers have to be registered for these specific events. The `*` wildcard handler is not called for these. ::: + + ## In-Memory Queue You can enable the in-memory queue globally with: @@ -264,19 +281,23 @@ You can enable the in-memory queue globally with: } } ``` + Messages are emitted only after the current transaction is successfully committed. Until then, messages are only kept in memory. This is similar to the following code if done manually: + ```js cds.context.on('succeeded', () => this.emit(msg)) ``` + ::: warning No retry mechanism The message is lost if the emit fails. There's no retry mechanism. ::: + ## Immediate Emit -To disable deferred emitting for a particular service, you can set the `outboxed` option of your service to `false`: +To disable deferred emitting for a particular service only, you can set the `outboxed` option of that service to `false`: ```json { @@ -289,9 +310,12 @@ To disable deferred emitting for a particular service, you can set the `outboxed } ``` + + ## Troubleshooting -### Delete Entries in the Tasks Table + +### Delete Entries in the Messages Table To manually delete entries in the table `cds.outbox.Messages`, you can either expose it in a service, see [Managing the Dead Letter Queue](#managing-the-dead-letter-queue), or programmatically modify it using the `cds.outbox.Messages` @@ -302,11 +326,12 @@ const db = await cds.connect.to('db') await DELETE.from('cds.outbox.Messages') ``` -### Tasks Table Not Found -If the tasks table is not found on the database, this can be caused by insufficient configuration data in _package.json_. +### Messages Table Not Found + +If the messages table is not found on the database, this can be caused by insufficient configuration data in _package.json_. -In case you have overwritten `requires.db.model` there, make sure to add the queue model path `@sap/cds/srv/outbox`: +In case you have overwritten `requires.db.model` there, make sure to add the outbox model path `@sap/cds/srv/outbox`: ```jsonc "requires": { From 5c5f922eee9f2a2c8c6803ff31eb1d166a4b4e91 Mon Sep 17 00:00:00 2001 From: sjvans <30337871+sjvans@users.noreply.github.com> Date: Tue, 27 May 2025 23:11:35 +0200 Subject: [PATCH 59/60] typo Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index 05de8daa8e..f1fca43c4a 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -178,7 +178,7 @@ In your CDS model, you can refer to the entity `cds.outbox.Messages` using the p ### Known Limitations - If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. It can be triggered manually using the `flush` method. -- The service that handles the queued event must not rely on user roles and attributes, as they are not stored with the message. In other words, asynchroneous task are always processed in a priviledged mode. However, the user ID is stored to re-create the correct context. +- The service that handles the queued event must not rely on user roles and attributes, as they are not stored with the message. In other words, asynchronous task are always processed in a priviledged mode. However, the user ID is stored to re-create the correct context. ### Managing the Dead Letter Queue From 2c67c6a6cf9ab1ad8574b2a3f9a311e0b35a3ffc Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 27 May 2025 23:12:19 +0200 Subject: [PATCH 60/60] typo --- node.js/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node.js/queue.md b/node.js/queue.md index f1fca43c4a..305ae2c56e 100644 --- a/node.js/queue.md +++ b/node.js/queue.md @@ -178,7 +178,7 @@ In your CDS model, you can refer to the entity `cds.outbox.Messages` using the p ### Known Limitations - If the app crashes, another emit for the respective tenant and service is necessary to restart the message processing. It can be triggered manually using the `flush` method. -- The service that handles the queued event must not rely on user roles and attributes, as they are not stored with the message. In other words, asynchronous task are always processed in a priviledged mode. However, the user ID is stored to re-create the correct context. +- The service that handles the queued event must not rely on user roles and attributes, as they are not stored with the message. In other words, asynchronous task are always processed in a privileged mode. However, the user ID is stored to re-create the correct context. ### Managing the Dead Letter Queue