Stop flapping alerts#1582
Open
dstathis wants to merge 2 commits into
Open
Conversation
dstathis
requested review from
carlcsaposs-canonical,
dragomirp,
juju-charm-bot,
marceloneppel and
taurus-forever
and removed request for
a team
June 10, 2026 09:21
taurus-forever
left a comment
taurus-forever
left a comment
Contributor
There was a problem hiding this comment.
@dstathis thank you for the PR! Can you please share the root cause of the issue?
Are the problem in the real outage/lack of patroni leader OR alerts are really fake/noisy?
In the first case, we should fix the stability, in the second fix alerts.
I believe 5 minutes downtime is unacceptable slow/long for 99.999% uptime target.
Anyway, defaults here defined by @7annaba3l business use cases.
AFAIK, COS have a way to fine-tune defaults per deployment,
while charm should be safe with it's default => provide maximum uptime,
otherwise immediately alert about the outage.
Author
|
I am not really sure if there is a real problem. I haven't noticed any issues with related apps. There do seem to be some error logs but I don't really know how to interpret them. |
Contributor
|
Well... is it some unstable networking environment? The The message Once K8s etcd/DCS is reachable again => the new Primary elected => all restored. We will consider to bump 0m slightly, as it might be too agressive (@7annaba3l to provide a good value), but 5m is too high for sure. We should trace your DCS instabilities and either fix the origin OR fine tune timeouts to avoid the real Primary (writes) outages. It will fix alerts instead of muting them. WDYT? @dstathis |
Increased alert duration for PatroniHasNoLeader from 0m to 5m. Signed-off-by: Dylan Stephano-Shachter <dylan.stephano-shachter@canonical.com>
Signed-off-by: Dylan Stephano-Shachter <dylan.stephano-shachter@canonical.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
The Postgresql charm constantly has flapping alerts. I see these three regularly.
Solution
Only fire if the condition is detected for 5 minutes or more.
Checklist