Add per-tenant alert generator URL template for customizable alert source links (#7302)

* Add per-tenant Grafana Explore URL format for alert GeneratorURL

Add support for tenants to configure alert GeneratorURL to use Grafana
Explore format instead of the default Prometheus /graph format. This is
controlled by three new per-tenant settings: ruler_alert_generator_url_format,
ruler_grafana_datasource_uid, and ruler_grafana_org_id.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Fix gofmt alignment in ruler and validation packages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Replace Grafana-specific config with generic Go template for alert generator URLs

Replace the 3 Grafana-specific per-tenant config fields
(ruler_alert_generator_url_format, ruler_grafana_datasource_uid,
ruler_grafana_org_id) with a single generic field:
ruler_alert_generator_url_template.

This field accepts a Go text/template string with .ExternalURL and
.Expression variables, plus built-in functions like urlquery. Users
can construct any URL format (Grafana, Perses, etc.) without Cortex
needing to understand specific UI formats.

The ruler_external_url per-tenant override and SendAlerts signature
(func(expr string) string) are kept unchanged.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Add getting-started example for per-tenant alert generator URL templates

Add per-tenant Alertmanager datasources (tenant-a, tenant-b) to Grafana
provisioning so alerts are visible in Grafana's alerting UI.

Add runtime-config.yaml with per-tenant overrides:
- tenant-a: Grafana Explore URL template with full pane JSON
- tenant-b: Perses explore URL template with PrometheusTimeSeriesQuery

Update Perses from v0.49 to v0.53.1 and enable the explorer feature
(frontend.explorer.enable: true). Rename project from "default" to
"cortex" to match template URLs.

Add Step 7 to the getting-started guide with instructions for:
- Configuring per-tenant alert generator URL templates
- Loading alertmanager configs and demo alert rules
- Viewing alerts in Grafana at /alerting/groups?groupBy=alertname
- Verifying generator URLs via the API

Also configure ruler.alertmanager_url and ruler.external_url, and set
an explicit UID on the Grafana Cortex datasource for use in templates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Add per-tenant Grafana Explore URL format for alert GeneratorURL

Add support for tenants to configure alert GeneratorURL to use Grafana
Explore format instead of the default Prometheus /graph format. This is
controlled by three new per-tenant settings: ruler_alert_generator_url_format,
ruler_grafana_datasource_uid, and ruler_grafana_org_id.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Fix gofmt alignment in ruler and validation packages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Replace Grafana-specific config with generic Go template for alert generator URLs

Replace the 3 Grafana-specific per-tenant config fields
(ruler_alert_generator_url_format, ruler_grafana_datasource_uid,
ruler_grafana_org_id) with a single generic field:
ruler_alert_generator_url_template.

This field accepts a Go text/template string with .ExternalURL and
.Expression variables, plus built-in functions like urlquery. Users
can construct any URL format (Grafana, Perses, etc.) without Cortex
needing to understand specific UI formats.

The ruler_external_url per-tenant override and SendAlerts signature
(func(expr string) string) are kept unchanged.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Add getting-started example for per-tenant alert generator URL templates

Add per-tenant Alertmanager datasources (tenant-a, tenant-b) to Grafana
provisioning so alerts are visible in Grafana's alerting UI.

Add runtime-config.yaml with per-tenant overrides:
- tenant-a: Grafana Explore URL template with full pane JSON
- tenant-b: Perses explore URL template with PrometheusTimeSeriesQuery

Update Perses from v0.49 to v0.53.1 and enable the explorer feature
(frontend.explorer.enable: true). Rename project from "default" to
"cortex" to match template URLs.

Add Step 7 to the getting-started guide with instructions for:
- Configuring per-tenant alert generator URL templates
- Loading alertmanager configs and demo alert rules
- Viewing alerts in Grafana at /alerting/groups?groupBy=alertname
- Verifying generator URLs via the API

Also configure ruler.alertmanager_url and ruler.external_url, and set
an explicit UID on the Grafana Cortex datasource for use in templates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* fix linting

Signed-off-by: Charlie Le <charlie_le@apple.com>

* Fix per-tenant ExternalURL, datasource UID, and add CHANGELOG

- Set ExternalURL on rules.ManagerOptions to the per-tenant override so
  {{ $externalURL }} in alert annotation/label templates reflects the
  tenant's ruler_external_url, not just the global config.
- Fix Grafana datasource: add explicit uid: tenant-a so the template
  reference "datasource":"tenant-a" resolves correctly.
- Fix runtime-config.yaml template to reference "datasource":"tenant-a"
  instead of "datasource":"cortex".
- Add text/template security comment explaining the intentional choice
  over html/template.
- Add CHANGELOG entry for the ruler_alert_generator_url_template feature.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Add URL validation to generator URL template output

Validate that the output of executeGeneratorURLTemplate produces a safe
URL by checking:
- Scheme must be http or https (blocks javascript: and data: URIs)
- Host must be present
- Fragment must not contain HTML characters < or > (blocks script injection)

Add test cases covering javascript URI, data URI, fragment injection,
missing host, and valid fragment scenarios.

Signed-off-by: Friedrich Gonzalez <charlie_le@apple.com>
Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>

* Re-resolve per-tenant external URL on every alert send

The NotifyFunc closure was capturing externalURLStr once at manager
creation time, so runtime config changes to ruler_external_url would
not take effect until the ruler was restarted. Move the resolution
into a helper that re-reads from overrides on each call.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

* Cache parsed generator URL template to avoid re-parsing on every alert

The generator URL template is parsed from the runtime config string on
every alert send. Cache the parsed template and only re-parse when the
template string changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>

---------

Signed-off-by: Charlie Le <charlie_le@apple.com>
Signed-off-by: Friedrich Gonzalez <charlie_le@apple.com>
Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>

Author: 3 people

CHANGELOG.md

@@ -1,6 +1,7 @@
 # Changelog
 
 ## master / unreleased
+* [FEATURE] Ruler: Add per-tenant `ruler_alert_generator_url_template` runtime config option to customize alert generator URLs using Go templates. Supports Grafana Explore, Perses, and other UIs. #7302
 * [FEATURE] Distributor: Add experimental `-distributor.enable-start-timestamp` flag for Prometheus Remote Write 2.0. When enabled, `StartTimestamp (ST)` is ingested. #7371
 * [FEATURE] Memberlist: Add `-memberlist.cluster-label` and `-memberlist.cluster-label-verification-disabled` to prevent accidental cross-cluster gossip joins and support rolling label rollout. #7385
 * [FEATURE] Querier: Add timeout classification to classify query timeouts as 4XX (user error) or 5XX (system error) based on phase timing. When enabled, queries that spend most of their time in PromQL evaluation return `422 Unprocessable Entity` instead of `503 Service Unavailable`. #7374

docs/configuration/config-file-reference.md

@@ -4388,6 +4388,16 @@ query_rejection:
 # external labels for alerting rules
 [ruler_external_labels: <map of string (labelName) to string (labelValue)> | default = []]
 
+# Per-tenant external URL for the ruler. If set, it overrides the global
+# -ruler.external.url for this tenant's alert notifications.
+[ruler_external_url: <string> | default = ""]
+
+# Go text/template for alert generator URLs. Available variables: .ExternalURL
+# (resolved external URL) and .Expression (PromQL expression). Built-in
+# functions like urlquery are available. If empty, uses default Prometheus
+# /graph format.
+[ruler_alert_generator_url_template: <string> | default = ""]
+
 # Enable to allow rules to be evaluated with data from a single zone, if other
 # zones are not available.
 [rules_partial_data: <boolean> | default = false]

docs/getting-started/.env

@@ -2,4 +2,4 @@ CORTEX_VERSION=v1.20.1
 GRAFANA_VERSION=10.4.2
 PROMETHEUS_VERSION=v3.2.1
 SEAWEEDFS_VERSION=3.67
-PERSES_VERSION=v0.49-distroless-debug
+PERSES_VERSION=v0.53.1-distroless-debug

docs/getting-started/cortex-config.yaml

@@ -82,6 +82,14 @@ frontend_worker:
 # https://cortexmetrics.io/docs/configuration/configuration-file/#ruler_config
 ruler:
   enable_api: true
+  external_url: http://localhost:9009
+  alertmanager_url: http://localhost:9009/alertmanager
+
+# Per-tenant runtime configuration (hot-reloaded without restart).
+# This file configures per-tenant overrides such as custom alert generator
+# URL templates for Grafana, Perses, or any metrics explorer.
+runtime_config:
+  file: /config/runtime-config.yaml
 
 # https://cortexmetrics.io/docs/configuration/configuration-file/#ruler_storage_config
 ruler_storage:

docs/getting-started/docker-compose.yaml

@@ -17,6 +17,7 @@ services:
       - -config.file=/config/cortex-config.yaml
     volumes:
       - ./cortex-config.yaml:/config/cortex-config.yaml:ro
+      - ./runtime-config.yaml:/config/runtime-config.yaml:ro
     ports:
       - "9009:9009"
     healthcheck:
@@ -47,6 +48,8 @@ services:
     volumes:
       - ./perses/config.yaml:/etc/perses/config/config.yaml:ro
       - ./perses/datasource.yaml:/etc/perses/resources/datasource.yaml:ro
+      - ./perses/datasource-tenant-a.yaml:/etc/perses/resources/datasource-tenant-a.yaml:ro
+      - ./perses/datasource-tenant-b.yaml:/etc/perses/resources/datasource-tenant-b.yaml:ro
       - ./perses/project.yaml:/etc/perses/resources/project.yaml:ro
       - ./perses/dashboards/cortex-writes.yaml:/etc/perses/resources/cortex-writes.yaml:ro
   prometheus:

docs/getting-started/grafana-datasource-docker.yaml

@@ -5,6 +5,7 @@ apiVersion: 1
 datasources:
   - name: Cortex
     type: prometheus
+    uid: cortex
     access: proxy
     orgId: 1
     url: http://cortex:9009/api/prom
@@ -22,6 +23,7 @@ datasources:
     isDefault: true
   - name: Tenant A
     type: prometheus
+    uid: tenant-a
     access: proxy
     orgId: 1
     url: http://cortex:9009/api/prom
@@ -71,3 +73,25 @@ datasources:
     secureJsonData:
       httpHeaderValue1: cortex
     version: 1
+  - orgId: 1
+    name: Tenant A Alertmanager
+    type: alertmanager
+    access: proxy
+    url: http://cortex:9009/
+    jsonData:
+      httpHeaderName1: X-Scope-OrgID
+      implementation: cortex
+    secureJsonData:
+      httpHeaderValue1: tenant-a
+    version: 1
+  - orgId: 1
+    name: Tenant B Alertmanager
+    type: alertmanager
+    access: proxy
+    url: http://cortex:9009/
+    jsonData:
+      httpHeaderName1: X-Scope-OrgID
+      implementation: cortex
+    secureJsonData:
+      httpHeaderValue1: tenant-b
+    version: 1

docs/getting-started/perses/config.yaml

@@ -8,14 +8,19 @@ security:
 database:
   file:
     extension: yaml
-    folder: /perses
+    folder: /tmp/perses-data
 
 schemas:
   datasources_path: /etc/perses/cue/schemas/datasources
   interval: 5m
   panels_path: /etc/perses/cue/schemas/panels
   queries_path: /etc/perses/cue/schemas/queries
   variables_path: /etc/perses/cue/schemas/variables
+
+frontend:
+  explorer:
+    enable: true
+
 provisioning:
   folders:
     - /etc/perses/resources

docs/getting-started/perses/dashboards/cortex-writes.yaml

@@ -4,7 +4,7 @@ metadata:
     createdAt: 2025-03-24T19:15:47.468680767Z
     updatedAt: 2025-03-24T19:43:53.000136362Z
     version: 12
-    project: default
+    project: cortex
 spec:
     display:
         name: Cortex / Writes

docs/getting-started/perses/datasource-tenant-a.yaml

@@ -0,0 +1,14 @@
+kind: GlobalDatasource
+metadata:
+  name: TenantA
+spec:
+  default: false
+  plugin:
+    kind: PrometheusDatasource
+    spec:
+      proxy:
+        kind: HTTPProxy
+        spec:
+          url: http://cortex:9009/api/prom
+          headers:
+            X-Scope-OrgID: tenant-a

docs/getting-started/perses/datasource-tenant-b.yaml

@@ -0,0 +1,14 @@
+kind: GlobalDatasource
+metadata:
+  name: TenantB
+spec:
+  default: false
+  plugin:
+    kind: PrometheusDatasource
+    spec:
+      proxy:
+        kind: HTTPProxy
+        spec:
+          url: http://cortex:9009/api/prom
+          headers:
+            X-Scope-OrgID: tenant-b