Skip to content

Commit 4aed3a0

Browse files
eeldalyeeldaly
authored
Add Query Resource Based Eviction (#7488)
* Add Query Resource Based Eviction Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * update changelog Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * update guarantees Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * lint Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * lint atomic Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * lint modernize Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * use configs.evictionConfig instead of copy Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * Panic on eivctor creation failures Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * Add support for evicting multiple queries per cycle Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * Doc gen Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * lint Signed-off-by: Essam Eldaly <eeldaly@amazon.com> * Add log. Remove unneeded err Signed-off-by: Essam Eldaly <eeldaly@amazon.com> --------- Signed-off-by: Essam Eldaly <eeldaly@amazon.com> Signed-off-by: Essam Eldaly <60357054+eeldaly@users.noreply.github.com>
1 parent e67fdbc commit 4aed3a0

19 files changed

Lines changed: 1897 additions & 32 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
* [FEATURE] Memberlist: Add `-memberlist.cluster-label` and `-memberlist.cluster-label-verification-disabled` to prevent accidental cross-cluster gossip joins and support rolling label rollout. #7385
1212
* [FEATURE] Querier: Add timeout classification to classify query timeouts as 4XX (user error) or 5XX (system error) based on phase timing. When enabled, queries that spend most of their time in PromQL evaluation return `422 Unprocessable Entity` instead of `503 Service Unavailable`. #7374
1313
* [FEATURE] Querier: Implement Resource Based Throttling in Querier. #7442
14+
* [FEATURE] Querier: Add resource-based query eviction that automatically cancels the heaviest running query when CPU or heap utilization exceeds configured thresholds. #7488
1415
* [ENHANCEMENT] Upgrade prometheus alertmanager version to v0.32.1. #7462
1516
* [ENHANCEMENT] Tenant Federation: Avoid purging the regex resolver LRU cache on user-sync ticks when the set of known users has not changed. #7489
1617
* [ENHANCEMENT] Memberlist: Add `-memberlist.packet-read-timeout`, `-memberlist.max-packet-size`, and `-memberlist.max-concurrent-connections` flags to bound inbound gossip TCP connections, preventing slow-read, OOM, and connection-flood attacks on the gossip port. #7518

‎docs/blocks-storage/querier.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,48 @@ querier:
330330
# type. 0 to disable.
331331
# CLI flag: -querier.query-protection.rejection.threshold.heap-utilization
332332
[heap_utilization: <float> | default = 0]
333+
334+
eviction:
335+
threshold:
336+
# EXPERIMENTAL: Max CPU utilization that this instance can reach before
337+
# evicting the heaviest running query (across all tenants) in
338+
# percentage, between 0 and 1. monitored_resources config must include
339+
# the resource type. 0 to disable.
340+
# CLI flag: -querier.query-protection.eviction.threshold.cpu-utilization
341+
[cpu_utilization: <float> | default = 0]
342+
343+
# EXPERIMENTAL: Max heap utilization that this instance can reach before
344+
# evicting the heaviest running query (across all tenants) in
345+
# percentage, between 0 and 1. monitored_resources config must include
346+
# the resource type. 0 to disable.
347+
# CLI flag: -querier.query-protection.eviction.threshold.heap-utilization
348+
[heap_utilization: <float> | default = 0]
349+
350+
# EXPERIMENTAL: How frequently the evictor checks system resource
351+
# utilization.
352+
# CLI flag: -querier.query-protection.eviction.check-interval
353+
[check_interval: <duration> | default = 1s]
354+
355+
# EXPERIMENTAL: Number of check intervals to wait after an eviction before
356+
# evicting again.
357+
# CLI flag: -querier.query-protection.eviction.cooldown-period
358+
[cooldown_period: <int> | default = 3]
359+
360+
# EXPERIMENTAL: The query metric used to determine the heaviest query for
361+
# eviction. Supported values: fetched_samples, fetched_series,
362+
# fetched_chunks, fetched_chunk_bytes.
363+
# CLI flag: -querier.query-protection.eviction.eviction-metric
364+
[eviction_metric: <string> | default = "fetched_samples"]
365+
366+
# EXPERIMENTAL: Minimum time a query must be running before it becomes
367+
# eligible for eviction. Queries younger than this are ignored.
368+
# CLI flag: -querier.query-protection.eviction.min-query-age
369+
[min_query_age: <duration> | default = 10s]
370+
371+
# EXPERIMENTAL: Maximum number of queries to evict in a single check cycle
372+
# when resource thresholds are breached.
373+
# CLI flag: -querier.query-protection.eviction.max-evictions-per-cycle
374+
[max_evictions_per_cycle: <int> | default = 1]
333375
```
334376
335377
### `blocks_storage_config`

‎docs/blocks-storage/store-gateway.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,48 @@ store_gateway:
372372
# CLI flag: -store-gateway.query-protection.rejection.threshold.heap-utilization
373373
[heap_utilization: <float> | default = 0]
374374

375+
eviction:
376+
threshold:
377+
# EXPERIMENTAL: Max CPU utilization that this instance can reach before
378+
# evicting the heaviest running query (across all tenants) in
379+
# percentage, between 0 and 1. monitored_resources config must include
380+
# the resource type. 0 to disable.
381+
# CLI flag: -store-gateway.query-protection.eviction.threshold.cpu-utilization
382+
[cpu_utilization: <float> | default = 0]
383+
384+
# EXPERIMENTAL: Max heap utilization that this instance can reach before
385+
# evicting the heaviest running query (across all tenants) in
386+
# percentage, between 0 and 1. monitored_resources config must include
387+
# the resource type. 0 to disable.
388+
# CLI flag: -store-gateway.query-protection.eviction.threshold.heap-utilization
389+
[heap_utilization: <float> | default = 0]
390+
391+
# EXPERIMENTAL: How frequently the evictor checks system resource
392+
# utilization.
393+
# CLI flag: -store-gateway.query-protection.eviction.check-interval
394+
[check_interval: <duration> | default = 1s]
395+
396+
# EXPERIMENTAL: Number of check intervals to wait after an eviction before
397+
# evicting again.
398+
# CLI flag: -store-gateway.query-protection.eviction.cooldown-period
399+
[cooldown_period: <int> | default = 3]
400+
401+
# EXPERIMENTAL: The query metric used to determine the heaviest query for
402+
# eviction. Supported values: fetched_samples, fetched_series,
403+
# fetched_chunks, fetched_chunk_bytes.
404+
# CLI flag: -store-gateway.query-protection.eviction.eviction-metric
405+
[eviction_metric: <string> | default = "fetched_samples"]
406+
407+
# EXPERIMENTAL: Minimum time a query must be running before it becomes
408+
# eligible for eviction. Queries younger than this are ignored.
409+
# CLI flag: -store-gateway.query-protection.eviction.min-query-age
410+
[min_query_age: <duration> | default = 10s]
411+
412+
# EXPERIMENTAL: Maximum number of queries to evict in a single check cycle
413+
# when resource thresholds are breached.
414+
# CLI flag: -store-gateway.query-protection.eviction.max-evictions-per-cycle
415+
[max_evictions_per_cycle: <int> | default = 1]
416+
375417
hedged_request:
376418
# If true, hedged requests are applied to object store calls. It can help
377419
# with reducing tail latency.

‎docs/configuration/config-file-reference.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3914,6 +3914,48 @@ query_protection:
39143914
# disable.
39153915
# CLI flag: -ingester.query-protection.rejection.threshold.heap-utilization
39163916
[heap_utilization: <float> | default = 0]
3917+
3918+
eviction:
3919+
threshold:
3920+
# EXPERIMENTAL: Max CPU utilization that this instance can reach before
3921+
# evicting the heaviest running query (across all tenants) in percentage,
3922+
# between 0 and 1. monitored_resources config must include the resource
3923+
# type. 0 to disable.
3924+
# CLI flag: -ingester.query-protection.eviction.threshold.cpu-utilization
3925+
[cpu_utilization: <float> | default = 0]
3926+
3927+
# EXPERIMENTAL: Max heap utilization that this instance can reach before
3928+
# evicting the heaviest running query (across all tenants) in percentage,
3929+
# between 0 and 1. monitored_resources config must include the resource
3930+
# type. 0 to disable.
3931+
# CLI flag: -ingester.query-protection.eviction.threshold.heap-utilization
3932+
[heap_utilization: <float> | default = 0]
3933+
3934+
# EXPERIMENTAL: How frequently the evictor checks system resource
3935+
# utilization.
3936+
# CLI flag: -ingester.query-protection.eviction.check-interval
3937+
[check_interval: <duration> | default = 1s]
3938+
3939+
# EXPERIMENTAL: Number of check intervals to wait after an eviction before
3940+
# evicting again.
3941+
# CLI flag: -ingester.query-protection.eviction.cooldown-period
3942+
[cooldown_period: <int> | default = 3]
3943+
3944+
# EXPERIMENTAL: The query metric used to determine the heaviest query for
3945+
# eviction. Supported values: fetched_samples, fetched_series,
3946+
# fetched_chunks, fetched_chunk_bytes.
3947+
# CLI flag: -ingester.query-protection.eviction.eviction-metric
3948+
[eviction_metric: <string> | default = "fetched_samples"]
3949+
3950+
# EXPERIMENTAL: Minimum time a query must be running before it becomes
3951+
# eligible for eviction. Queries younger than this are ignored.
3952+
# CLI flag: -ingester.query-protection.eviction.min-query-age
3953+
[min_query_age: <duration> | default = 10s]
3954+
3955+
# EXPERIMENTAL: Maximum number of queries to evict in a single check cycle
3956+
# when resource thresholds are breached.
3957+
# CLI flag: -ingester.query-protection.eviction.max-evictions-per-cycle
3958+
[max_evictions_per_cycle: <int> | default = 1]
39173959
```
39183960

39193961
### `ingester_client_config`
@@ -5081,6 +5123,48 @@ query_protection:
50815123
# disable.
50825124
# CLI flag: -querier.query-protection.rejection.threshold.heap-utilization
50835125
[heap_utilization: <float> | default = 0]
5126+
5127+
eviction:
5128+
threshold:
5129+
# EXPERIMENTAL: Max CPU utilization that this instance can reach before
5130+
# evicting the heaviest running query (across all tenants) in percentage,
5131+
# between 0 and 1. monitored_resources config must include the resource
5132+
# type. 0 to disable.
5133+
# CLI flag: -querier.query-protection.eviction.threshold.cpu-utilization
5134+
[cpu_utilization: <float> | default = 0]
5135+
5136+
# EXPERIMENTAL: Max heap utilization that this instance can reach before
5137+
# evicting the heaviest running query (across all tenants) in percentage,
5138+
# between 0 and 1. monitored_resources config must include the resource
5139+
# type. 0 to disable.
5140+
# CLI flag: -querier.query-protection.eviction.threshold.heap-utilization
5141+
[heap_utilization: <float> | default = 0]
5142+
5143+
# EXPERIMENTAL: How frequently the evictor checks system resource
5144+
# utilization.
5145+
# CLI flag: -querier.query-protection.eviction.check-interval
5146+
[check_interval: <duration> | default = 1s]
5147+
5148+
# EXPERIMENTAL: Number of check intervals to wait after an eviction before
5149+
# evicting again.
5150+
# CLI flag: -querier.query-protection.eviction.cooldown-period
5151+
[cooldown_period: <int> | default = 3]
5152+
5153+
# EXPERIMENTAL: The query metric used to determine the heaviest query for
5154+
# eviction. Supported values: fetched_samples, fetched_series,
5155+
# fetched_chunks, fetched_chunk_bytes.
5156+
# CLI flag: -querier.query-protection.eviction.eviction-metric
5157+
[eviction_metric: <string> | default = "fetched_samples"]
5158+
5159+
# EXPERIMENTAL: Minimum time a query must be running before it becomes
5160+
# eligible for eviction. Queries younger than this are ignored.
5161+
# CLI flag: -querier.query-protection.eviction.min-query-age
5162+
[min_query_age: <duration> | default = 10s]
5163+
5164+
# EXPERIMENTAL: Maximum number of queries to evict in a single check cycle
5165+
# when resource thresholds are breached.
5166+
# CLI flag: -querier.query-protection.eviction.max-evictions-per-cycle
5167+
[max_evictions_per_cycle: <int> | default = 1]
50845168
```
50855169
50865170
### `query_frontend_config`
@@ -6850,6 +6934,48 @@ query_protection:
68506934
# CLI flag: -store-gateway.query-protection.rejection.threshold.heap-utilization
68516935
[heap_utilization: <float> | default = 0]
68526936
6937+
eviction:
6938+
threshold:
6939+
# EXPERIMENTAL: Max CPU utilization that this instance can reach before
6940+
# evicting the heaviest running query (across all tenants) in percentage,
6941+
# between 0 and 1. monitored_resources config must include the resource
6942+
# type. 0 to disable.
6943+
# CLI flag: -store-gateway.query-protection.eviction.threshold.cpu-utilization
6944+
[cpu_utilization: <float> | default = 0]
6945+
6946+
# EXPERIMENTAL: Max heap utilization that this instance can reach before
6947+
# evicting the heaviest running query (across all tenants) in percentage,
6948+
# between 0 and 1. monitored_resources config must include the resource
6949+
# type. 0 to disable.
6950+
# CLI flag: -store-gateway.query-protection.eviction.threshold.heap-utilization
6951+
[heap_utilization: <float> | default = 0]
6952+
6953+
# EXPERIMENTAL: How frequently the evictor checks system resource
6954+
# utilization.
6955+
# CLI flag: -store-gateway.query-protection.eviction.check-interval
6956+
[check_interval: <duration> | default = 1s]
6957+
6958+
# EXPERIMENTAL: Number of check intervals to wait after an eviction before
6959+
# evicting again.
6960+
# CLI flag: -store-gateway.query-protection.eviction.cooldown-period
6961+
[cooldown_period: <int> | default = 3]
6962+
6963+
# EXPERIMENTAL: The query metric used to determine the heaviest query for
6964+
# eviction. Supported values: fetched_samples, fetched_series,
6965+
# fetched_chunks, fetched_chunk_bytes.
6966+
# CLI flag: -store-gateway.query-protection.eviction.eviction-metric
6967+
[eviction_metric: <string> | default = "fetched_samples"]
6968+
6969+
# EXPERIMENTAL: Minimum time a query must be running before it becomes
6970+
# eligible for eviction. Queries younger than this are ignored.
6971+
# CLI flag: -store-gateway.query-protection.eviction.min-query-age
6972+
[min_query_age: <duration> | default = 10s]
6973+
6974+
# EXPERIMENTAL: Maximum number of queries to evict in a single check cycle
6975+
# when resource thresholds are breached.
6976+
# CLI flag: -store-gateway.query-protection.eviction.max-evictions-per-cycle
6977+
[max_evictions_per_cycle: <int> | default = 1]
6978+
68536979
hedged_request:
68546980
# If true, hedged requests are applied to object store calls. It can help with
68556981
# reducing tail latency.

‎docs/configuration/v1-guarantees.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,13 @@ Currently experimental features are:
130130
- `-validation.max-label-cardinality-for-unoptimized-regex` (int) - maximum label cardinality
131131
- `-validation.max-total-label-value-length-for-unoptimized-regex` (int) - maximum total length of all label values in bytes
132132
- HATracker: `-distributor.ha-tracker.enable-startup-sync` (bool) - If enabled, fetches all tracked keys on startup to populate the local cache.
133+
- Querier: Resource-based query eviction
134+
- `-querier.query-protection.eviction.threshold.cpu-utilization` (float)
135+
- `-querier.query-protection.eviction.threshold.heap-utilization` (float)
136+
- `-querier.query-protection.eviction.check-interval` (duration)
137+
- `-querier.query-protection.eviction.cooldown-period` (int)
138+
- `-querier.query-protection.eviction.eviction-metric` (string)
139+
- `-querier.query-protection.eviction.min-query-age` (duration)
133140
- Ingester: Active Series Tracker
134141
- Per-tenant `active_series_trackers` configuration in runtime config overrides
135142
- Counts active series matching PromQL label matchers and exposes `cortex_ingester_active_series_per_tracker` metric

0 commit comments

Comments
 (0)