apollo_dashboard,apollo_deployments: add top-level intervalSec to Alerts; use it in alert builder

[Itay-Tsabary-Starkware]

Move the evaluation interval from a per-alert field to a top-level field
on the Alerts struct. The interval is group-scoped in Grafana, so a single
value for all alerts is the correct model.

Also restructure alert_builder.py to upload/dump per rule-group instead of
per individual alert, using the top-level intervalSec as the group interval.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com

---

Note

Medium Risk
Changes the alert JSON schema and the provisioning/deployment flow (grouped updates and per-group CRDs), which could alter alert evaluation cadence or cause missing/duplicated rules if grouping or intervals are misapplied.

Overview
Updates the generated dev_grafana_alerts.json schema to include top-level groups (with per-group evaluation intervals) and moves alerts into these rule groups (evaluation_rate_high/default/low), along with various severity/observer applicability tweaks driven by the regrouping.

Replaces AlertGroup with EvaluationRate in the Rust alert definitions, adds deterministic sorting, and changes Alerts serialization to emit the new groups + alerts structure.

Refactors alert_builder.py and the sequencer Grafana construct to provision alerts per rule group (one group file/CRD per group) and to set the rule-group interval once, rather than per alert.

^{Reviewed by Cursor Bugbot for commit 96c5d81. Bugbot is set up for automated code reviews on this repo. Configure here.}

[reviewable-StarkWare]

This change is 

[Itay-Tsabary-Starkware]

• apollo_dashboard: remove per-alert intervalSec field #13450
• apollo_dashboard,apollo_deployments: add top-level intervalSec to Alerts; use it in alert builder #13449 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Artifacts upload workflows:

• Started at 2026-04-06T07:21:54Z
• Started at 2026-04-13T15:00:01Z
• Started at 2026-04-14T06:54:14Z
• Started at 2026-04-14T07:29:24Z

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 79dc8e9. Configure here.}

[matanl-starkware]

@matanl-starkware reviewed 19 files and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on Itay-Tsabary-Starkware).

---

crates/apollo_dashboard/resources/dev_grafana_alerts.json line 41 at r4 (raw file):

      ],
      "for": "30s",
      "intervalSec": 30,

Why is this still here?

Code quote:

"intervalSec": 30,

---

crates/apollo_dashboard/src/alerts.rs line 223 at r4 (raw file):

            Self::Low => "evaluation_rate_low",
        }
    }

Why is this required in addition to the serde/rename instructions above?
Consider keeping only one of them.

Code quote:

    pub(crate) fn name(&self) -> &'static str {
        match self {
            Self::High => "evaluation_rate_high",
            Self::Default => "evaluation_rate_default",
            Self::Low => "evaluation_rate_low",
        }
    }

[matanl-starkware]

@matanl-starkware made 1 comment and resolved 1 discussion.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on Itay-Tsabary-Starkware).

---

crates/apollo_dashboard/resources/dev_grafana_alerts.json line 41 at r4 (raw file):

Previously, matanl-starkware (Matan Lior) wrote…

Why is this still here?

Removed in following PR 13450

[cursor]

PR Summary

Medium Risk
Changes the on-disk alert JSON schema and the provisioning pipeline (Grafana upload + Kubernetes CRDs) from per-alert intervals to rule-group based intervals, which can alter alert evaluation cadence and deployment behavior if any integration expects the old format.

Overview
Refactors apollo_dashboard alert serialization to introduce explicit Grafana rule groups via a new EvaluationRate enum (default/high/low), emit a top-level groups section, and stably sort alerts by group then name for deterministic JSON output.

Updates the monitoring deployment tooling to build/upload/dump alerts as rule groups instead of individual rules: alert_builder.py now batches alerts by ruleGroup and updates each group in Grafana in one call, and the sequencer grafana.py construct now creates one SharedGrafanaAlertRuleGroup CRD per group using the group interval. The generated dev_grafana_alerts.json is reorganized accordingly (new groups plus many ruleGroup/severity/observer applicability tweaks).

^{Reviewed by Cursor Bugbot for commit d04135f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[matanl-starkware]

@matanl-starkware reviewed 4 files and all commit messages, and resolved 1 discussion.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on Itay-Tsabary-Starkware).

[Itay-Tsabary-Starkware]

@Itay-Tsabary-Starkware reviewed 21 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on Itay-Tsabary-Starkware).