Can we define alerts and recording rules in the fallback config?

[shibumi]

Hi everyone,

I am working on a setup right now where we install several clusters with several Mimir and Grafana stacks.

We want to ensure that we deploy the same alerting rules and recording rules on each of these Mimir instances.

Is there any good way to import Prometheus alerts and recording rules to all these Mimir instances? Can we just write everything into the fallback config and Mimir loads this for the Anonymous Tenant? (We do not use tenants).

Would this work?

I am aware that the mimirtool exist, but we want to avoid using it and really want something that we deploy into the cluster.

It is very sad that Custom Resource Definitions (such as PrometheusRule) are ignored by Mimir and clicking through dozens of Grafanas for configuring alerts is also not really an option.

[cookesan]

No, I would not put recording or alerting rule groups in the fallback/runtime config.

In Cortex, the "fallback config" knob in the generated config reference is under Alertmanager:

alertmanager:
  fallback_config_file: ...

That is Alertmanager configuration fallback, not ruler rule-group loading. Runtime config examples are for overrides and other runtime settings; they are not Prometheus rule files.

Rules belong to the ruler path: configure ruler / ruler_storage, then load Prometheus rule groups into the ruler backend/API. The getting-started example has a separate rules.yaml with normal Prometheus-style:

groups:
  - name: example
    rules:
      - record: up:count
        expr: count(up)

So the deployable Kubernetes shape is usually: package the rule YAML separately and provision it into the ruler storage/API as part of your release process. If you want CRD-based PrometheusRule reconciliation, that needs an operator/controller to translate those CRDs into the ruler API/storage; the fallback config will not make Cortex/Mimir read them as anonymous-tenant rules.

[shibumi]

How would I load the Prometheus rules into the ruler backend? AFAIK, this is currently only possible via mimirtool is this correct? Or is it also possible to just drop the rules as YAML files in the GCS or S3 bucket and ruler would load it from there automatically?

I am asking, because we are very interested in a GitOps-like workflow and this API only approach seems infeasible for us. It would require additional infrastructure reading files and uploading it to the alertmanager/ruler backends.

[cookesan]

For the S3/GCS ruler storage backends, I would not drop raw Prometheus YAML files directly into the bucket. The Ruler API is the supported write path there.

The API endpoint is POST /api/v1/rules/{namespace} with Content-Type: application/yaml and the tenant header (X-Scope-OrgID when auth is enabled). It accepts a single Prometheus rule group YAML body and stores it through the configured RuleStore. In the object-store implementation, that becomes Cortex's internal rule-group object under the ruler bucket prefix, not a plain rules.yaml file that Cortex parses from the bucket.

The exception is ruler_storage.backend: local: the sharded-ruler docs say local reads Prometheus YAML from the filesystem layout <rules dir>/<tenant id>/rules1.yaml, but that backend is read-only and does not support create/delete through the API. It can be GitOps-friendly if you mount the same generated files into every ruler pod, but it is not the S3/GCS object-store mode.

So for GitOps with S3/GCS, the stable pattern is still a reconciler or CI job that reads Git YAML and calls the Ruler API. That can be mimirtool, cortextool if it fits your version, or a small curl/script wrapper; it does not have to be heavy infrastructure, but something needs to translate Git files into the tenant/namespace/group API writes. Alertmanager config is similar: the documented write path is POST /api/v1/alerts with the YAML body, not direct raw-file placement in the backend.

[shibumi]

This seems to be a huge downgrade compared to the traditional Prometheus-Operator + Thanos stack, where you would just create CRDs in the cluster and the operator would automatically create the necessary rules.

Do you know if there are any plans to support Prometheus CRDs in the future? At least for alerting?

We have decided to alert with Grafana instead now.