diff --git a/docs/percona-server-system-variables.md b/docs/percona-server-system-variables.md index 6ad67da45e4..b4063ed2202 100644 --- a/docs/percona-server-system-variables.md +++ b/docs/percona-server-system-variables.md @@ -83,5 +83,10 @@ | Innodb_oldest_view_low_limit_trx_id | Numeric | Global | | Innodb_purge_trx_id | Numeric | Global | | Innodb_purge_undo_no | Numeric | Global | +| Threadpool_average_hp_queue_wait_us | String | Global | +| Threadpool_average_queue_wait_us | String | Global | | Threadpool_idle_threads | Numeric | Global | +| Threadpool_requests_starved_in_queue | Numeric | Global | +| Threadpool_requests_waiting_in_hp_queue | Numeric | Global | +| Threadpool_requests_waiting_in_queue | Numeric | Global | | Threadpool_threads | Numeric | Global | diff --git a/docs/threadpool.md b/docs/threadpool.md index a1b26121b97..7447d2dd5f1 100644 --- a/docs/threadpool.md +++ b/docs/threadpool.md @@ -60,21 +60,21 @@ The thread pool adds the connection to the high-priority queue and decrements th * Has a non-zero number of high-priority tickets -Otherwise, the variable adds the connection to the low-priority queue with the initial value. +Otherwise, the variable adds the connection to the normal-priority queue with the initial value. -Each time, the thread pool checks the high-priority queue for the next connection. When the high-priority queue is empty, the thread pool picks connections from the low-priority queue. The default behavior is to put events from already started transactions into the high-priority queue. +Each time, the thread pool checks the high-priority queue for the next connection. When the high-priority queue is empty, the thread pool picks connections from the normal-priority queue. The default behavior is to put events from already started transactions into the high-priority queue. -If the value equals `0`, all connections are put into the low-priority queue. If the value exceeds zero, each connection could be put into a high-priority queue. +If the value equals `0`, all connections are put into the normal-priority queue. If the value exceeds zero, each connection could be put into a high-priority queue. -The [thread_pool_high_prio_mode](#thread_pool_high_prio_mode) variable prioritizes all statements for a connection or assigns connections to the low-priority queue. To implement this new [thread_pool_high_prio_mode](#thread_pool_high_prio_mode) variable +The [thread_pool_high_prio_mode](#thread_pool_high_prio_mode) variable prioritizes all statements for a connection or assigns connections to the normal-priority queue. To implement this new [thread_pool_high_prio_mode](#thread_pool_high_prio_mode) variable -## Low-priority queue throttling +## Normal-priority queue throttling -One case that can limit thread pool performance and even lead to deadlocks under high concurrency is when thread groups are oversubscribed due to active threads reaching the oversubscribe limit. Still, all/most worker threads are waiting on locks currently held by a transaction from another connection that is not currently in the thread pool. +Thread pool performance can degrade, or even deadlock, under high concurrency when a thread group becomes oversubscribed — that is, when the number of active worker threads in the group reaches [`thread_pool_oversubscribe`](#thread_pool_oversubscribe) — and most of those threads are simultaneously waiting on locks held by a transaction whose connection has not yet been picked up by the thread pool. -In this case, the oversubscribe limit does not account for those threads in the pool that marked themselves inactive. As a result, the number of threads (both active and waiting) in the pool grows until it hits the [`thread_pool_max_threads`](#thread_pool_max_threads) value. If the connection executing the transaction holding the lock has managed to enter the thread pool by then, we get a large (depending on the [`thread_pool_max_threads`](#thread_pool_max_threads) value) number of concurrently running threads and, thus, suboptimal performance. Otherwise, we get a deadlock as no more threads can be created to process those transaction(s) and release the lock(s). +The oversubscribe limit does not account for threads that have marked themselves inactive while waiting. As a result, the total number of threads in the pool (both active and waiting) continues to grow until the pool reaches [`thread_pool_max_threads`](#thread_pool_max_threads). If the connection executing the transaction that holds the lock enters the thread pool before `thread_pool_max_threads` is reached, the pool ends up running a large number of threads concurrently (proportional to [`thread_pool_max_threads`](#thread_pool_max_threads)), resulting in suboptimal performance. Otherwise, the pool deadlocks, because no more threads can be created to process the blocking transactions and release their locks. -Such situations are prevented by throttling the low-priority queue when the total number of worker threads (both active and waiting ones) reaches the oversubscribe limit. If there are too many worker threads, do not start new transactions; create new threads until queued events from the already-started transactions are processed. +To prevent this scenario, the thread pool throttles the normal-priority queue when the total number of worker threads (both active and waiting) reaches the [`thread_pool_oversubscribe`](#thread_pool_oversubscribe) limit. While the pool is throttled, new transactions are not started and no new threads are created; the pool processes only queued events from already-started transactions, allowing their locks to be released so the pool can drain. ## Handling long network waits @@ -94,26 +94,13 @@ Specific workloads (large result sets, BLOBs, slow clients) can wait longer on n | Default | one-thread-per-connection | -This variable defines how the server handles threads for connections from the client. +Defines how the server dispatches statements from client connections to execution threads. The default, `one-thread-per-connection`, dedicates a single thread to each connection for the connection's entire lifetime. To activate the thread pool, set `thread_handling` to `pool-of-threads` in `my.cnf` and restart the server; all connections are then processed through the shared pool controlled by [`thread_pool_size`](#thread_pool_size) and the other `thread_pool_*` variables. The `no-threads` value serves all connections from a single thread and is intended for debugging only. Because `thread_handling` is not dynamic, changing the value requires a server restart. As a rule of thumb, the thread pool benefits workloads with very high connection counts (typically 20,000 or more) and short CPU-bound queries; for smaller connection counts, `one-thread-per-connection` usually performs better. -| Values | Description | -|---------------------------|--------------------------------------------------------| -| one-thread-per-connection | One thread handles all requests for a connection | -| pool-of-threads | A thread pool handles requests for all connections | -| no-threads | A single thread for all connections for debugging mode | - -### `thread_pool_idle_timeout` - -| Option | Description | -| -------------- | ------------------ | -| Command-line: | Yes | -| Config file: | Yes | -| Scope: | Global | -| Dynamic: | Yes | -| Data type: | Numeric | -| Default value: | 60 (seconds) | - -This variable can limit the time an idle thread should wait before exiting. +| Value | Description | +|---------------------------|--------------------------------------------------------------| +| one-thread-per-connection | One thread handles all requests for a connection. | +| pool-of-threads | A thread pool handles requests for all connections. | +| no-threads | A single thread serves all connections (debugging mode only).| ### `thread_pool_high_prio_mode` @@ -121,11 +108,11 @@ This variable provides more fine-grained control over high-priority scheduling g The following values are allowed: -* `transactions` (the default). In this mode, only statements from already started transactions may go into the high-priority queue depending on the number of high-priority tickets currently available in a connection (see thread_pool_high_prio_tickets). - -* `statements`. In this mode, all individual statements go into the high-priority queue, regardless of the transactional state and the number of available high-priority tickets. Use this value to prioritize `AUTOCOMMIT` transactions or other statements, such as administrative ones. Setting this value globally essentially disables high-priority scheduling. All connections use the high-priority queue. - -* `none`. This mode disables the priority queue for a connection. Certain types of connections, such as monitoring, are insensitive to execution latency and do not allocate the server resources that would impact the performance of other connections. These types of connections do not require high-priority scheduling. Setting this value globally essentially disables high-priority scheduling. All connections use the low-priority queue. +| Value | Description | +|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `transactions` | Default. Only statements from already started transactions may go into the high-priority queue, depending on the number of high-priority tickets currently available in the connection (see [`thread_pool_high_prio_tickets`](#thread_pool_high_prio_tickets)). | +| `statements` | All individual statements go into the high-priority queue, regardless of transactional state or available tickets. Use the `statements` value to prioritize `AUTOCOMMIT` transactions or administrative statements. Setting `thread_pool_high_prio_mode` to `statements` globally effectively disables high-priority scheduling, because all connections use the high-priority queue. | +| `none` | Disables the priority queue for a connection. Useful for connections that are insensitive to execution latency, such as monitoring. Setting `thread_pool_high_prio_mode` to `none` globally effectively disables high-priority scheduling, because all connections use the normal-priority queue. | ### `thread_pool_high_prio_tickets` @@ -140,6 +127,19 @@ The following values are allowed: This variable controls the high-priority queue policy. Assigns the selected number of tickets to each new connection to enter the high-priority queue. Setting this variable to `0` disables the high-priority queue. +### `thread_pool_idle_timeout` + +| Option | Description | +| -------------- | ------------------ | +| Command-line: | Yes | +| Config file: | Yes | +| Scope: | Global | +| Dynamic: | Yes | +| Data type: | Numeric | +| Default value: | 60 (seconds) | + +Defines the number of seconds a worker thread that has no work remains in the thread pool before exiting. When an idle thread's wait exceeds `thread_pool_idle_timeout`, the thread is removed from the pool, freeing memory and OS resources; if load returns, a new worker thread is created (subject to [`thread_pool_oversubscribe`](#thread_pool_oversubscribe) and [`thread_pool_max_threads`](#thread_pool_max_threads)). A lower value reclaims resources faster under bursty or intermittent workloads but increases thread-creation overhead when activity resumes; a higher value keeps threads warm and reduces creation churn at the cost of additional idle memory. Observe [`Threadpool_idle_threads`](#threadpool_idle_threads) and [`Threadpool_threads`](#threadpool_threads) to see how many threads are being retained in the pool at the current setting. + ### `thread_pool_max_threads` | Option | Description | @@ -164,7 +164,7 @@ This variable can limit the maximum number of threads in the pool. When the limi | Data type: | Numeric | | Default value: | 3 | -Determines the number of threads run simultaneously. A value lower than `3` could cause sleep and wake-up actions. +Defines the maximum number of worker threads that may be active simultaneously within a single thread group. When a worker thread in a group blocks (on I/O, a lock, or is otherwise stalled — see [`thread_pool_stall_limit`](#thread_pool_stall_limit)), the group may create additional worker threads up to the `thread_pool_oversubscribe` limit so that queued requests can continue to be processed. A higher value improves responsiveness under workloads that frequently block, at the cost of more context switching and contention; a lower value keeps concurrency tighter to the CPU, but values below the default of `3` can cause frequent thread sleep and wake-up cycles. When the total number of worker threads in a group (both active and waiting) reaches the `thread_pool_oversubscribe` limit, the thread pool throttles the normal-priority queue (see [Normal-priority queue throttling](#normal-priority-queue-throttling)) to prevent deadlocks and runaway thread creation. Tune alongside [`thread_pool_size`](#thread_pool_size) and observe [`Threadpool_threads`](#threadpool_threads) together with the queue-wait metrics to evaluate the effect. ### `thread_pool_size` @@ -177,7 +177,7 @@ Determines the number of threads run simultaneously. A value lower than `3` coul | Data type: | Numeric | | Default value: | Number of processors | -Defines the number of threads that can use the CPU simultaneously. +Defines the number of thread groups in the thread pool. Each thread group maintains its own high-priority and normal-priority queues and handles the connections assigned to the group, using one active worker thread at a time; additional worker threads may be created within the group up to the [`thread_pool_oversubscribe`](#thread_pool_oversubscribe) limit. New connections are distributed across the thread groups in round-robin order, so `thread_pool_size` effectively sets the target concurrency of the thread pool. As a starting point, use a value equal to the number of available CPU cores and tune from there based on observed [`Threadpool_idle_threads`](#threadpool_idle_threads), [`Threadpool_threads`](#threadpool_threads), and the queue-wait metrics [`Threadpool_average_queue_wait_us`](#threadpool_average_queue_wait_us) and [`Threadpool_average_hp_queue_wait_us`](#threadpool_average_hp_queue_wait_us). ### `thread_pool_stall_limit` @@ -194,6 +194,36 @@ Defines the number of milliseconds before a running thread is considered stalled ## Status variables +### `Threadpool_average_hp_queue_wait_us` + +| Option | Description | +| -------------- | ------------------ | +| Scope: | Global | +| Data type: | String | + +This status variable reports aggregated wait-time statistics, in microseconds, for requests waiting in the high-priority queue. The value is a formatted string, for example: + +```text +avg: 0.000, min: 0.000, max: 0.000, dev: 0.000, cnt: 0 +``` + +Each sample is a single wait-time measurement, in microseconds, taken when a request is dequeued by a worker thread. Every dequeued request contributes exactly one sample. The `cnt` field shows the number of samples collected since server start; `avg`, `min`, `max`, and `dev` (standard deviation) are computed across all samples. Requests that the listener thread picks up immediately, without waiting, are included as zero-valued samples. + +### `Threadpool_average_queue_wait_us` + +| Option | Description | +| -------------- | ------------------ | +| Scope: | Global | +| Data type: | String | + +This status variable reports aggregated wait-time statistics, in microseconds, for requests waiting in the normal-priority queue. The value is a formatted string, for example: + +```text +avg: 590.000, min: 470.000, max: 736.000, dev: 110.266, cnt: 5 +``` + +Each sample is a single wait-time measurement, in microseconds, taken when a request is dequeued by a worker thread. Every dequeued request contributes exactly one sample. The `cnt` field shows the number of samples collected since server start; `avg`, `min`, `max`, and `dev` (standard deviation) are computed across all samples. Requests that the listener thread picks up immediately, without waiting, are included as zero-valued samples. + ### `Threadpool_idle_threads` | Option | Description | @@ -201,7 +231,34 @@ Defines the number of milliseconds before a running thread is considered stalled | Scope: | Global | | Data type: | Numeric | -This status variable shows the number of idle threads in the pool. +This status variable shows the number of idle threads in the pool — worker threads that are part of the pool but are not currently executing a request. Together with [`Threadpool_threads`](#threadpool_threads), `Threadpool_idle_threads` indicates how busy the pool is: when the value is close to `Threadpool_threads`, the pool is underutilized; when the value is near zero, the pool is saturated. Threads that remain idle longer than [`thread_pool_idle_timeout`](#thread_pool_idle_timeout) seconds may exit the pool. + +### `Threadpool_requests_starved_in_queue` + +| Option | Description | +| -------------- | ------------------ | +| Scope: | Global | +| Data type: | Numeric | + +This status variable shows the number of requests in the normal-priority queue that are being starved by requests in the high-priority queue. A non-zero value indicates that worker threads are busy processing high-priority traffic and normal-priority requests are not being picked up, which can be used as an indicator of thread pool saturation. + +### `Threadpool_requests_waiting_in_hp_queue` + +| Option | Description | +| -------------- | ------------------ | +| Scope: | Global | +| Data type: | Numeric | + +This status variable shows the number of requests currently waiting in the thread pool's high-priority queue. Which requests enter the high-priority queue is governed by [`thread_pool_high_prio_mode`](#thread_pool_high_prio_mode) and [`thread_pool_high_prio_tickets`](#thread_pool_high_prio_tickets). A sustained or growing value indicates that high-priority traffic is arriving faster than worker threads can drain the queue. Observe `Threadpool_requests_waiting_in_hp_queue` alongside [`Threadpool_average_hp_queue_wait_us`](#threadpool_average_hp_queue_wait_us) to gauge the effect on latency, and alongside [`Threadpool_requests_starved_in_queue`](#threadpool_requests_starved_in_queue) to detect when high-priority traffic is starving the normal-priority queue. + +### `Threadpool_requests_waiting_in_queue` + +| Option | Description | +| -------------- | ------------------ | +| Scope: | Global | +| Data type: | Numeric | + +This status variable shows the number of requests currently waiting in the thread pool's normal-priority queue. Requests enter the normal-priority queue when they do not meet the criteria for the high-priority queue set by [`thread_pool_high_prio_mode`](#thread_pool_high_prio_mode) and [`thread_pool_high_prio_tickets`](#thread_pool_high_prio_tickets). A sustained or growing value indicates either that normal-priority traffic is arriving faster than worker threads can drain the queue, or that high-priority traffic is monopolizing the worker threads. Observe `Threadpool_requests_waiting_in_queue` alongside [`Threadpool_average_queue_wait_us`](#threadpool_average_queue_wait_us) to gauge the effect on latency, and alongside [`Threadpool_requests_starved_in_queue`](#threadpool_requests_starved_in_queue) to detect starvation by the high-priority queue. ### `Threadpool_threads` @@ -210,4 +267,4 @@ This status variable shows the number of idle threads in the pool. | Scope: | Global | | Data type: | Numeric | -This status variable shows the number of threads in the pool. +This status variable shows the total number of worker threads currently in the pool, including both idle threads and threads actively executing requests. The pool grows dynamically, up to [`thread_pool_max_threads`](#thread_pool_max_threads), as load increases and threads stall (see [`thread_pool_stall_limit`](#thread_pool_stall_limit)); threads that remain idle longer than [`thread_pool_idle_timeout`](#thread_pool_idle_timeout) seconds may exit. The target concurrency of the pool is set by [`thread_pool_size`](#thread_pool_size) and [`thread_pool_oversubscribe`](#thread_pool_oversubscribe). Compare `Threadpool_threads` with [`Threadpool_idle_threads`](#threadpool_idle_threads) to determine how many threads are actively running requests: busy threads = `Threadpool_threads` − `Threadpool_idle_threads`.