Tuneable worker pools

changelog/68532.added.md

@@ -0,0 +1,6 @@
+Added tunable worker pools: partition the master's MWorkers into named pools
+and route specific commands (for example `_auth`) to dedicated pools so a
+slow workload cannot starve time-critical traffic. Controlled by the new
+`worker_pools` and `worker_pools_enabled` master settings; see the "Tunable
+Worker Pools" topic guide for details. Existing `worker_threads`
+configurations remain fully backward compatible.

doc/ref/configuration/master.rst

@@ -2491,30 +2491,117 @@ limit is to search the internet for something like this:
 
 Default: ``5``
 
-The number of threads to start for receiving commands and replies from minions.
-If minions are stalling on replies because you have many minions, raise the
-worker_threads value.
+The number of MWorker processes to start for receiving commands and replies
+from minions.  If minions are stalling on replies because you have many
+minions, raise the ``worker_threads`` value.
 
 Worker threads should not be put below 3 when using the peer system, but can
 drop down to 1 worker otherwise.
 
 Standards for busy environments:
 
 * Use one worker thread per 200 minions.
-* The value of worker_threads should not exceed 1½ times the available CPU cores.
+* The value of ``worker_threads`` should not exceed 1½ times the available CPU
+  cores.
 
 .. note::
     When the master daemon starts, it is expected behaviour to see
-    multiple salt-master processes, even if 'worker_threads' is set to '1'. At
-    a minimum, a controlling process will start along with a Publisher, an
-    EventPublisher, and a number of MWorker processes will be started. The
-    number of MWorker processes is tuneable by the 'worker_threads'
-    configuration value while the others are not.
+    multiple salt-master processes, even if ``worker_threads`` is set to
+    ``1``. At a minimum, a controlling process will start along with a
+    Publisher, an EventPublisher, and a number of MWorker processes will be
+    started. The number of MWorker processes is tuneable by the
+    ``worker_threads`` configuration value while the others are not.
 
 .. code-block:: yaml
 
     worker_threads: 5
 
+.. note::
+    ``worker_threads`` only controls the size of the single default worker
+    pool used by the legacy code path.  For finer-grained routing — for
+    example to give ``_auth`` its own dedicated MWorkers — see
+    :conf_master:`worker_pools`, :conf_master:`worker_pools_enabled`, and the
+    :ref:`tunable worker pools <tunable-worker-pools>` topic guide.  When
+    ``worker_pools`` is unset the master automatically builds a single
+    catchall pool sized by ``worker_threads``, so existing configurations
+    behave exactly as before.
+
+.. conf_master:: worker_pools_enabled
+
+``worker_pools_enabled``
+------------------------
+
+.. versionadded:: 3008.0
+
+Default: ``True``
+
+Master-level switch for the :ref:`tunable worker pools <tunable-worker-pools>`
+feature.  When ``True`` (the default) the master uses
+:conf_master:`worker_pools` (or, if that is unset, a single catchall pool
+sized by :conf_master:`worker_threads`) to route requests to per-pool
+MWorkers.  When ``False`` the master falls back to the legacy single-queue
+MWorker model.
+
+The default value preserves the historical behavior when no other pool
+settings are provided, so upgrading does not require any configuration
+changes.  Set this to ``False`` only if you need to disable pooled routing
+entirely — for example to debug a transport issue.
+
+.. code-block:: yaml
+
+    worker_pools_enabled: True
+
+.. conf_master:: worker_pools
+
+``worker_pools``
+----------------
+
+.. versionadded:: 3008.0
+
+Default: ``{}`` (an implicit single catchall pool sized by
+:conf_master:`worker_threads`)
+
+Defines the MWorker pools the master should start and the commands each pool
+should service.  When unset, the master builds a single pool named
+``default`` with ``worker_count`` equal to :conf_master:`worker_threads` and
+a catchall that receives every command — equivalent to the pre-3008.0
+behavior.
+
+Each key under ``worker_pools`` names a pool.  The value is a dictionary
+with two required fields:
+
+``worker_count``
+    Integer ``>= 1``.  The number of MWorker processes to start for the
+    pool.
+
+``commands``
+    List of command strings.  Each string must be either an exact command
+    name (for example ``_auth`` or ``_return``) or the single catchall
+    entry ``"*"``.
+
+A command may be mapped to at most one pool.  Exactly one pool must use
+the ``"*"`` catchall so that every command has a routing destination;
+payloads whose ``cmd`` is not matched by an explicit mapping are sent to
+that pool.
+
+The master refuses to start if the configuration is invalid — for example
+if two pools claim the same command, if no pool (or more than one pool)
+uses the ``"*"`` catchall, or if a pool has no ``commands``.  See
+:ref:`tunable worker pools <tunable-worker-pools>` for a full walkthrough
+of the validation rules and recommended layouts.
+
+.. code-block:: yaml
+
+    worker_pools:
+      auth:
+        worker_count: 2
+        commands:
+          - _auth
+      default:
+        worker_count: 8
+        commands:
+          - "*"
+
 .. conf_master:: pub_hwm
 
 ``pub_hwm``

doc/topics/performance/index.rst

@@ -11,3 +11,4 @@ for Salt.
     :maxdepth: 1
 
     pki_index
+    worker_pools