Skip to content

Commit 571559d

Browse files
dahliadahlia
authored
Merge pull request #599 from dahlia/mysql
Add `MysqlMessageQueue` to `@fedify/mysql`
2 parents de6972c + 6123b62 commit 571559d

14 files changed

Lines changed: 1831 additions & 13 deletions

File tree

‎CHANGES.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,14 @@ To be released.
6565

6666
### @fedify/mysql
6767

68+
- Added `MysqlMessageQueue` class to the `@fedify/mysql` package, a
69+
MySQL/MariaDB-backed `MessageQueue` implementation. It uses periodic
70+
polling (`SELECT … FOR UPDATE SKIP LOCKED`) to deliver messages and
71+
MySQL advisory locks (`GET_LOCK`/`RELEASE_LOCK`) for ordering-key
72+
serialization. Supports delayed delivery, ordering keys,
73+
`enqueueMany()`, and concurrent workers. Requires MySQL 8.0+ or
74+
MariaDB 10.6+. [[#586], [#599]]
75+
6876
- Added `@fedify/mysql` package, a MySQL/MariaDB-backed `KvStore`
6977
implementation. It provides `MysqlKvStore`, which stores key–value
7078
pairs in a MySQL table using the [`mysql2`] driver. Supports TTL,
@@ -73,7 +81,9 @@ To be released.
7381

7482
[`mysql2`]: https://www.npmjs.com/package/mysql2
7583
[#585]: https://github.com/fedify-dev/fedify/issues/585
84+
[#586]: https://github.com/fedify-dev/fedify/issues/586
7685
[#597]: https://github.com/fedify-dev/fedify/pull/597
86+
[#599]: https://github.com/fedify-dev/fedify/pull/599
7787

7888

7989
Version 2.0.3

‎docs/manual/deploy.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,9 @@ Development
168168
Production
169169
: Consider [`PostgresKvStore`](./kv.md#postgreskvstore) and
170170
[`PostgresMessageQueue`](./mq.md#postgresmessagequeue) if you already use
171-
PostgreSQL, or [`RedisKvStore`](./kv.md#rediskvstore) and
171+
PostgreSQL, [`MysqlKvStore`](./kv.md#mysqlkvstore) and
172+
[`MysqlMessageQueue`](./mq.md#mysqlmessagequeue) if you already use
173+
MySQL or MariaDB, or [`RedisKvStore`](./kv.md#rediskvstore) and
172174
[`RedisMessageQueue`](./mq.md#redismessagequeue) for dedicated caching
173175
infrastructure. There is also [`AmqpMessageQueue`](./mq.md#amqpmessagequeue)
174176
for RabbitMQ users.

‎docs/manual/federation.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -115,8 +115,10 @@ runtime).
115115

116116
As separate packages, [`@fedify/redis`] provides [`RedisMessageQueue`] class,
117117
which is a Redis-backed implementation for production use,
118-
and [`@fedify/postgres`] provides [`PostgresMessageQueue`] class, which is a
119-
PostgreSQL-backed implementation for production use, and [`@fedify/amqp`]
118+
[`@fedify/postgres`] provides [`PostgresMessageQueue`] class, which is a
119+
PostgreSQL-backed implementation for production use,
120+
[`@fedify/mysql`] provides [`MysqlMessageQueue`] class, which is a
121+
MySQL/MariaDB-backed implementation for production use, and [`@fedify/amqp`]
120122
provides [`AmqpMessageQueue`] class, which is an AMQP broker-backed
121123
implementation for production use.
122124

@@ -187,6 +189,7 @@ Further details are explained in the [*Message queue* section](./mq.md).
187189
188190
[`RedisMessageQueue`]: https://jsr.io/@fedify/redis/doc/mq/~/RedisMessageQueue
189191
[`PostgresMessageQueue`]: https://jsr.io/@fedify/postgres/doc/mq/~/PostgresMessageQueue
192+
[`MysqlMessageQueue`]: https://jsr.io/@fedify/mysql/doc/mq/~/MysqlMessageQueue
190193
[`@fedify/amqp`]: https://github.com/fedify-dev/fedify/tree/main/packages/amqp
191194
[`AmqpMessageQueue`]: https://jsr.io/@fedify/amqp/doc/mq/~/AmqpMessageQueue
192195

‎docs/manual/inbox.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -349,8 +349,9 @@ const federation = createFederation({
349349
> The `InProcessMessageQueue` is a simple in-memory message queue that is
350350
> suitable for development and testing. For production use, you should
351351
> consider using a more robust message queue, such as [`DenoKvMessageQueue`]
352-
> from [`@fedify/denokv`] package or [`RedisMessageQueue`] from
353-
> [`@fedify/redis`] package.
352+
> from [`@fedify/denokv`] package, [`RedisMessageQueue`] from
353+
> [`@fedify/redis`] package, or [`MysqlMessageQueue`] from
354+
> [`@fedify/mysql`] package.
354355
>
355356
> For more information, see the [*Message queue* section](./mq.md).
356357
@@ -389,6 +390,8 @@ duplicate retry mechanisms and leverages the backend's optimized retry features.
389390
[`@fedify/denokv`]: https://github.com/fedify-dev/fedify/tree/main/packages/denokv
390391
[`RedisMessageQueue`]: https://jsr.io/@fedify/redis/doc/mq/~/RedisMessageQueue
391392
[`@fedify/redis`]: https://github.com/fedify-dev/fedify/tree/main/packages/redis
393+
[`MysqlMessageQueue`]: https://jsr.io/@fedify/mysql/doc/mq/~/MysqlMessageQueue
394+
[`@fedify/mysql`]: https://github.com/fedify-dev/fedify/tree/main/packages/mysql
392395

393396

394397
Activity idempotency

‎docs/manual/mq.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,88 @@ const federation = createFederation({
293293
[`AmqpMessageQueue`]: https://jsr.io/@fedify/amqp/doc/mq/~/AmqpMessageQueue
294294
[RabbitMQ]: https://www.rabbitmq.com/
295295

296+
### [`MysqlMessageQueue`]
297+
298+
*This API is available since Fedify 2.1.0.*
299+
300+
To use [`MysqlMessageQueue`], you need to install the *@fedify/mysql* package
301+
first:
302+
303+
::: code-group
304+
305+
~~~~ bash [Deno]
306+
deno add jsr:@fedify/mysql
307+
~~~~
308+
309+
~~~~ bash [npm]
310+
npm add @fedify/mysql mysql2
311+
~~~~
312+
313+
~~~~ bash [pnpm]
314+
pnpm add @fedify/mysql mysql2
315+
~~~~
316+
317+
~~~~ bash [Yarn]
318+
yarn add @fedify/mysql mysql2
319+
~~~~
320+
321+
~~~~ bash [Bun]
322+
bun add @fedify/mysql mysql2
323+
~~~~
324+
325+
:::
326+
327+
[`MysqlMessageQueue`] is a message queue implementation that uses a MySQL or
328+
MariaDB database as the backend. Since MySQL and MariaDB do not provide a
329+
`LISTEN`/`NOTIFY` mechanism, it uses **polling** to discover new messages.
330+
The polling interval is configurable and defaults to 1 second to minimize
331+
latency; this is shorter than the default for PostgreSQL-backed queues.
332+
333+
Concurrent workers are safely supported via `SELECT … FOR UPDATE SKIP LOCKED`
334+
and MySQL advisory locks (`GET_LOCK`/`RELEASE_LOCK`).
335+
336+
> [!NOTE]
337+
> `MysqlMessageQueue` requires MySQL 8.0+ or MariaDB 10.6+ for
338+
> `SELECT … FOR UPDATE SKIP LOCKED` support.
339+
340+
> [!NOTE]
341+
> Because `MysqlMessageQueue` uses polling rather than a push-based
342+
> notification system, there is an inherent latency between when a message
343+
> is enqueued and when it is delivered. With the default 1-second poll
344+
> interval, messages may take up to 1 second to be picked up. You can
345+
> lower the `pollInterval` option to reduce this latency at the cost of
346+
> additional database load.
347+
348+
Best for
349+
: Production use in systems that already use MySQL or MariaDB.
350+
351+
Pros
352+
: Persistent, supports multiple workers, minimal additional infrastructure
353+
for MySQL/MariaDB users.
354+
355+
Cons
356+
: Polling-based delivery (up to `pollInterval` latency); requires
357+
MySQL 8.0+ or MariaDB 10.6+.
358+
359+
~~~~ typescript twoslash
360+
import type { KvStore } from "@fedify/fedify";
361+
// ---cut-before---
362+
import { createFederation } from "@fedify/fedify";
363+
import { MysqlMessageQueue } from "@fedify/mysql";
364+
import mysql from "mysql2/promise";
365+
366+
const pool = mysql.createPool("mysql://user:pass@localhost/db");
367+
const federation = createFederation<void>({
368+
// ---cut-start---
369+
kv: null as unknown as KvStore,
370+
// ---cut-end---
371+
queue: new MysqlMessageQueue(pool), // [!code highlight]
372+
// ... other options
373+
});
374+
~~~~
375+
376+
[`MysqlMessageQueue`]: https://jsr.io/@fedify/mysql/doc/mq/~/MysqlMessageQueue
377+
296378
### `SqliteMessageQueue`
297379

298380
*This API is available since Fedify 2.0.0.*
@@ -767,6 +849,9 @@ The following implementations do not yet support native retry:
767849
[`PostgresMessageQueue`]
768850
: Native retry support planned for future release.
769851

852+
[`MysqlMessageQueue`]
853+
: No native retry support (`~MessageQueue.nativeRetrial` is `false`).
854+
770855
[`AmqpMessageQueue`]
771856
: Native retry support planned for future release.
772857

@@ -835,6 +920,7 @@ The following implementations support ordering keys:
835920
| [`DenoKvMessageQueue`] | Yes |
836921
| [`RedisMessageQueue`] | Yes |
837922
| [`PostgresMessageQueue`] | Yes |
923+
| [`MysqlMessageQueue`] | Yes |
838924
| [`AmqpMessageQueue`] | Yes[^1] |
839925
| [`SqliteMessageQueue`] | Yes |
840926
| `WorkersMessageQueue` | Yes[^2] |

‎docs/manual/relay.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,8 @@ Configuration options
154154
~~~~
155155

156156
> [!NOTE]
157-
> For production, use [`RedisMessageQueue`] or [`PostgresMessageQueue`].
157+
> For production, use [`RedisMessageQueue`], [`PostgresMessageQueue`],
158+
> or [`MysqlMessageQueue`].
158159

159160
`subscriptionHandler` (required)
160161
: Callback to approve or reject subscription requests. See
@@ -177,6 +178,7 @@ Configuration options
177178

178179
[`RedisMessageQueue`]: https://jsr.io/@fedify/redis/doc/mq/~/RedisMessageQueue
179180
[`PostgresMessageQueue`]: https://jsr.io/@fedify/postgres/doc/mq/~/PostgresMessageQueue
181+
[`MysqlMessageQueue`]: https://jsr.io/@fedify/mysql/doc/mq/~/MysqlMessageQueue
180182

181183

182184
Relay types

‎docs/manual/send.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -390,8 +390,9 @@ const federation = createFederation({
390390
> The `InProcessMessageQueue` is a simple in-memory message queue that is
391391
> suitable for development and testing. For production use, you should
392392
> consider using a more robust message queue, such as [`DenoKvMessageQueue`]
393-
> from [`@fedify/denokv`] package or [`RedisMessageQueue`] from
394-
> [`@fedify/redis`] package.
393+
> from [`@fedify/denokv`] package, [`RedisMessageQueue`] from
394+
> [`@fedify/redis`] package, or [`MysqlMessageQueue`] from
395+
> [`@fedify/mysql`] package.
395396
>
396397
> For further information, see the [*Message queue* section](./mq.md).
397398
@@ -414,6 +415,8 @@ an error and does not retry the delivery.
414415
[`@fedify/denokv`]: https://github.com/fedify-dev/fedify/tree/main/packages/denokv
415416
[`RedisMessageQueue`]: https://jsr.io/@fedify/redis/doc/mq/~/RedisMessageQueue
416417
[`@fedify/redis`]: https://github.com/fedify-dev/fedify/tree/main/packages/redis
418+
[`MysqlMessageQueue`]: https://jsr.io/@fedify/mysql/doc/mq/~/MysqlMessageQueue
419+
[`@fedify/mysql`]: https://github.com/fedify-dev/fedify/tree/main/packages/mysql
417420

418421

419422
Optimizing activity delivery for large audiences

‎packages/mysql/README.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,20 +6,22 @@
66
[![JSR][JSR badge]][JSR]
77
[![npm][npm badge]][npm]
88

9-
This package provides [Fedify]'s [`KvStore`] implementation for
10-
MySQL/MariaDB:
9+
This package provides [Fedify]'s [`KvStore`] and [`MessageQueue`]
10+
implementations for MySQL/MariaDB:
1111

1212
- [`MysqlKvStore`]
13+
- [`MysqlMessageQueue`]
1314

1415
~~~~ typescript
1516
import { createFederation } from "@fedify/fedify";
16-
import { MysqlKvStore } from "@fedify/mysql";
17+
import { MysqlKvStore, MysqlMessageQueue } from "@fedify/mysql";
1718
import mysql from "mysql2/promise";
1819

1920
const pool = mysql.createPool("mysql://user:password@localhost/dbname");
2021

2122
const federation = createFederation({
2223
kv: new MysqlKvStore(pool),
24+
queue: new MysqlMessageQueue(pool),
2325
});
2426
~~~~
2527

@@ -29,7 +31,9 @@ const federation = createFederation({
2931
[npm]: https://www.npmjs.com/package/@fedify/mysql
3032
[Fedify]: https://fedify.dev/
3133
[`KvStore`]: https://jsr.io/@fedify/fedify/doc/federation/~/KvStore
34+
[`MessageQueue`]: https://jsr.io/@fedify/fedify/doc/federation/~/MessageQueue
3235
[`MysqlKvStore`]: https://jsr.io/@fedify/mysql/doc/~/MysqlKvStore
36+
[`MysqlMessageQueue`]: https://jsr.io/@fedify/mysql/doc/mq/~/MysqlMessageQueue
3337

3438

3539
Installation

‎packages/mysql/deno.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@
44
"license": "MIT",
55
"exports": {
66
".": "./src/mod.ts",
7-
"./kv": "./src/kv.ts"
7+
"./kv": "./src/kv.ts",
8+
"./mq": "./src/mq.ts"
89
},
910
"exclude": [
1011
"dist",

‎packages/mysql/package.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,16 @@
5151
"require": "./dist/kv.cjs",
5252
"default": "./dist/kv.js"
5353
},
54+
"./mq": {
55+
"types": {
56+
"import": "./dist/mq.d.ts",
57+
"require": "./dist/mq.d.cts",
58+
"default": "./dist/mq.d.ts"
59+
},
60+
"import": "./dist/mq.js",
61+
"require": "./dist/mq.cjs",
62+
"default": "./dist/mq.js"
63+
},
5464
"./package.json": "./package.json"
5565
},
5666
"files": [

0 commit comments

Comments
 (0)