notifications: consume state change events from runtime updates

[yhabteab]

Currently, all the relevant events are consumed from the respective history pipelines, and were not as is the case for histories, HA aware and thus were running on both Icinga DB HA instances. This meant that both instances were consuming the same events, but since Icinga Notifications had a logic to detect such superfluous events, it was not a problem. Also, the history streams are lacking some information, in order to be able to determine whether a checkable is in downtime, flapping state, or is acknowledged, which is relevant for each state change event. So, the first attempt in overcoming this was to add the missing flags to the state change history stream (basically, patching Icinga 2 and requiring a specific version of it just for this), but fortunately, @nilmerg came up with a much better solution, which I hadn't thought of, which is to consume the events directly from the runtime updates stream, which already contains all the relevant flags. Apart from that, the runtime updates are also already HA aware, so just like the other runtime updates consumers, the Icinga Notifications component will also pause consuming events on the passive instance.

Now, this change alone wouldn't have been that big of a deal, because my initial plan was to just hook into the existing runtime state updates pipeline, but then after some discussion with Johannes, he suggested to completely decouple the Notifications component from the existing runtime updates pipeline. This means that instead of making the Notifications component dependent on the success rate and performance of the database queries of the runtime updates pipeline, it will have its own dedicated Redis message reader. This way, both the Notifications component and the existing runtime updates pipeline can operate independently (can move their own stream offsets without affecting each other) while still consuming the same events from the same Redis stream. Obviously, this also means that they can't send XDel commands for each message they consume independently, because that would cause the other component to miss messages, so instead, they will both just acknowledge the messages they consume, and a separate collector goroutine will be responsible for tracking the acknowledged messages and sending the XDel commands to Redis in batches. Note that is not a Redis consumer group, but rather a Go channel based implementation, that mimics some of the consumer group concepts.

Last but not least, in order to always have the latest check output of a checkable, there is also a new goroutine that periodically (every 5 minutes) syncs the check output of all checkables that are currently in a non-OK state from the icinga:{host,service}:state Redis hashes to Icinga Notifications. This is necessary because a checkable's check can, for instance, be in CRITICAL state for a long time, while its output can change with each check result. And since that won't trigger any state change events, thus, not queued to the runtime updates stream, there's no way for Icinga Notifications to know about the updated output. EDIT that part of the code is no longer part of this PR.

All in all, with this PR, all the code related to consuming history events from the history pipelines has become obsolete, and isn't used anymore. However, since we are planning to implement a way to send non-state notifications (e.g. for downtime start/end, acknowledgement ste, etc.) in the future, which will eventually require consuming the respective history events, I didn't remove the code yet, instead, I just temporarily disabled it with a reference to respective Icinga Notifications issue.

Requires

• Miscellaneous changes for Notifications icinga-go-library#212
• Let events affect incidents in multiple ways icinga-notifications#442

resolves #1127

[Copilot]

Pull request overview

This PR reworks the Icinga Notifications integration to consume state-change–relevant data from the runtime updates stream (HA-aware and richer flags), adds a periodic check-output sync for long-running non-OK states, and extends runtime-updates syncing to support an additional independent consumer plus coordinated Redis stream deletion.

Changes:

• Switch Notifications event generation to use v1.HostState / v1.ServiceState (runtime updates) instead of state history for state-change semantics and flags.
• Add a periodic “check output sync” loop that re-submits non-OK hard states to keep incident messages current.
• Update runtime updates sync to support an optional parallel on-upsert callback (Notifications) and shared deletion via “all consumers ack” coordination.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/notifications/notifications.go	Adds runtime-update based state event building, retry/backoff submission, and periodic check-output syncing.
pkg/notifications/fetch.go	Introduces generic Redis-hash streaming helper and tracks is_volatile internally for event logic.
pkg/icingadb/runtime_updates.go	Adds RU options (parallel + onUpsert callback), multiple independent XREAD consumers, and coordinated XDEL.
cmd/icingadb/main.go	Wires Notifications submission into runtime updates and starts periodic check-output sync after initial state sync.
go.mod / go.sum	Bumps icinga-go-library and updates go-redis dependency.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yhabteab]

Rebased and dropped all the timer aka check output syncer related code from this PR as discussed offline. I'll open another PR for that once this PR is done.

[yhabteab]

Fixed some bugs caused by the newest changes in the referenced IGL PR.

[oxzi]

First thing first: This is a big change and it looks solid at first glance. I went over the code multiple times and tried only to highlight relevant things. However, due to the complexity, please take some of my comments with a grain of salt.

[yhabteab]

Addressed all your requested changes unless otherwise commented to the respective review comments.

[oxzi]

⛵