Parallel subscription processing

[DavidBadura]

The subscription engine now processes one subscription at a time instead of one shared stream across all of them. Each subscription is claimed individually via FOR UPDATE SKIP LOCKED, read from its own position and event filter, and committed in its own short transaction. Several subscription:run workers can now run in true parallel, each grabs the next free subscription, and a slow or failing one no longer holds up the rest.

[github-actions]

Hello 👋

here is the most recent benchmark result:

SplitStreamBench
================

+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                         | time (kde mode)                                     | memory                                     |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                 | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad10000Events () | 2.645ms (±0.00%)   | 2.682ms (±0.00%)   | -1.41%    | 35.769mb        | 36.288mb   | -1.43%      |
| benchSave10000Events () | 631.167ms (±0.00%) | 508.057ms (±0.00%) | +24.23%   | 35.904mb        | 36.423mb   | -1.43%      |
+-------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SimpleSetupStreamStoreBench
===========================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 981.000μs (±0.00%) | 951.300μs (±0.00%) | +3.12%    | 35.141mb        | 35.141mb   | 0.00%       |
| benchLoad10000Events ()                | 55.117ms (±0.00%)  | 55.062ms (±0.00%)  | +0.10%    | 35.141mb        | 35.141mb   | 0.00%       |
| benchSave1Event ()                     | 982.900μs (±0.00%) | 1.001ms (±0.00%)   | -1.76%    | 35.141mb        | 35.141mb   | 0.00%       |
| benchSave10000Events ()                | 284.264ms (±0.00%) | 241.306ms (±0.00%) | +17.80%   | 35.141mb        | 35.141mb   | 0.00%       |
| benchSave10000Aggregates ()            | 12.144s (±0.00%)   | 10.864s (±0.00%)   | +11.78%   | 35.141mb        | 35.141mb   | 0.00%       |
| benchSave10000AggregatesTransaction () | 3.819s (±0.00%)    | 3.816s (±0.00%)    | +0.07%    | 35.141mb        | 35.141mb   | 0.00%       |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SubscriptionEngineBatchBench
============================

+---------------------------+--------------------+-------------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                                    | memory                                     |
+---------------------------+--------------------+-------------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>     | Tag: base         | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+--------------------+-------------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 102.565ms (±0.00%) | 82.830ms (±0.00%) | +23.83%   | 35.720mb        | 35.720mb   | -0.00%      |
+---------------------------+--------------------+-------------------+-----------+-----------------+------------+-------------+

NoopSubscriptionEngineBench
===========================

+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                                   | memory                                     |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>    | Tag: base         | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 90.053ms (±0.00%) | 82.402ms (±0.00%) | +9.28%    | 48.296mb        | 48.296mb   | -0.00%      |
+---------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+

SubscriptionEngineBench
=======================

+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
|                           | time (kde mode)                               | memory                                     |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
| subject                   | Tag: <current>  | Tag: base       | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+
| benchHandle10000Events () | 2.498s (±0.00%) | 2.522s (±0.00%) | -0.95%    | 48.296mb        | 48.296mb   | -0.00%      |
+---------------------------+-----------------+-----------------+-----------+-----------------+------------+-------------+

CommandToQueryBench
===================

+----------------+-------------------+------------------+-----------+-----------------+------------+-------------+
|                | time (kde mode)                                  | memory                                     |
+----------------+-------------------+------------------+-----------+-----------------+------------+-------------+
| subject        | Tag: <current>    | Tag: base        | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------+-------------------+------------------+-----------+-----------------+------------+-------------+
| benchCreate () | 24.972ms (±0.00%) | 9.146ms (±0.00%) | +173.03%  | 5.464mb         | 5.062mb    | +7.95%      |
| benchUpdate () | 4.419ms (±0.00%)  | 5.146ms (±0.00%) | -14.14%   | 5.037mb         | 5.052mb    | -0.29%      |
| benchBoth ()   | 7.078ms (±0.00%)  | 6.905ms (±0.00%) | +2.50%    | 5.870mb         | 5.751mb    | +2.07%      |
+----------------+-------------------+------------------+-----------+-----------------+------------+-------------+

PersonalDataBench
=================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 960.100μs (±0.00%) | 968.500μs (±0.00%) | -0.87%    | 35.805mb        | 35.805mb   | 0.00%       |
| benchLoad10000Events ()                | 80.977ms (±0.00%)  | 80.479ms (±0.00%)  | +0.62%    | 35.805mb        | 35.805mb   | 0.00%       |
| benchSave1Event ()                     | 1.786ms (±0.00%)   | 1.778ms (±0.00%)   | +0.46%    | 35.805mb        | 35.805mb   | 0.00%       |
| benchSave10000Events ()                | 265.665ms (±0.00%) | 277.585ms (±0.00%) | -4.29%    | 35.807mb        | 35.807mb   | 0.00%       |
| benchSave10000Aggregates ()            | 13.527s (±0.00%)   | 15.640s (±0.00%)   | -13.51%   | 35.805mb        | 35.805mb   | 0.00%       |
| benchSave10000AggregatesTransaction () | 7.024s (±0.00%)    | 7.050s (±0.00%)    | -0.37%    | 36.045mb        | 36.045mb   | 0.00%       |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

SnapshotsBench
==============

+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                   | memory                                     |
+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>    | Tag: base         | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+
| benchLoad10000EventsMissingSnapshot () | 56.363ms (±0.00%) | 55.562ms (±0.00%) | +1.44%    | 35.146mb        | 35.146mb   | 0.00%       |
| benchLoad10000Events ()                | 1.094ms (±0.00%)  | 1.093ms (±0.00%)  | +0.06%    | 35.145mb        | 35.145mb   | 0.00%       |
+----------------------------------------+-------------------+-------------------+-----------+-----------------+------------+-------------+

SimpleSetupTaggableStoreBench
=============================

+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
|                                        | time (kde mode)                                     | memory                                     |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| subject                                | Tag: <current>     | Tag: base          | time-diff | Tag: <current>  | Tag: base  | memory-diff |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+
| benchLoad1Event ()                     | 968.800μs (±0.00%) | 1.032ms (±0.00%)   | -6.11%    | 36.306mb        | 36.306mb   | 0.00%       |
| benchLoad10000Events ()                | 57.361ms (±0.00%)  | 57.280ms (±0.00%)  | +0.14%    | 36.306mb        | 36.306mb   | 0.00%       |
| benchSave1Event ()                     | 2.004ms (±0.00%)   | 1.128ms (±0.00%)   | +77.61%   | 36.306mb        | 36.306mb   | 0.00%       |
| benchSave10000Events ()                | 272.192ms (±0.00%) | 300.754ms (±0.00%) | -9.50%    | 36.306mb        | 36.306mb   | 0.00%       |
| benchSave10000Aggregates ()            | 11.167s (±0.00%)   | 11.801s (±0.00%)   | -5.38%    | 36.307mb        | 36.307mb   | 0.00%       |
| benchSave10000AggregatesTransaction () | 3.843s (±0.00%)    | 3.845s (±0.00%)    | -0.05%    | 36.307mb        | 36.307mb   | 0.00%       |
| benchAppend1Event ()                   | 10.301ms (±0.00%)  | 1.310ms (±0.00%)   | +686.24%  | 36.306mb        | 36.306mb   | 0.00%       |
| benchAppend100Events ()                | 5.924ms (±0.00%)   | 5.873ms (±0.00%)   | +0.86%    | 36.306mb        | 36.306mb   | 0.00%       |
+----------------------------------------+--------------------+--------------------+-----------+-----------------+------------+-------------+

This comment gets update everytime a new commit comes in!