Skip to content

fix: size replay remote-write queue for backfill bursts #1840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zackpollard merged 1 commit into from
Jul 21, 2026
Merged

fix: size replay remote-write queue for backfill bursts #1840

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions kubernetes/apps/monitoring/victoria-metrics/replay/job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,17 @@ spec:
- -replay.disableProgressBar=true
# The default (1000) batches 1000 evaluation points per query_range
# request; against ~475k raw series that asks vmsingle for ~8GiB in
# one query and gets a 422, aborting the replay. 100 keeps the
# worst request well inside the ~5GiB concurrent-query budget at
# the cost of a longer (multi-hour) replay.
- -replay.maxDatapointsPerQuery=100
# one query and gets a 422, aborting the replay. Base rules also
# emit one sample per client_ip per 5m bucket, so a batch is a
# multi-million-sample remote-write burst: 50 points halves the
# burst and the queue below absorbs it.
- -replay.maxDatapointsPerQuery=50
# Survive transient 422s from concurrent dashboard/rule load; a
# rule that exhausts its retries aborts the whole replay.
- -replay.ruleRetryAttempts=10
- -remoteWrite.maxQueueSize=4000000
- -remoteWrite.maxBatchSize=10000
- -remoteWrite.concurrency=4
volumeMounts:
- name: rules
mountPath: /rules
Expand All @@ -63,6 +67,9 @@ spec:
- -replay.disableProgressBar=true
- -replay.maxDatapointsPerQuery=100
- -replay.ruleRetryAttempts=10
- -remoteWrite.maxQueueSize=4000000
- -remoteWrite.maxBatchSize=10000
- -remoteWrite.concurrency=4
volumeMounts:
- name: rules
mountPath: /rules
Expand Down
Loading