feat(ingester): Add active series tracker for pattern-based monitoring (#7476)

Add a new active series tracker feature that counts active series by
configurable label matchers (including regex) and exposes the counts as
Prometheus metrics. This is designed for internal monitoring without
enforcing any limits.

Key changes:
- Add ActiveSeriesTrackersConfig type in validation package with PromQL
  matcher syntax support
- Add ActiveSeriesTrackers field to Limits struct for per-tenant config
  with default fallback
- Add ActiveForMatchers() method to ActiveSeries for counting matching
  series across all stripes
- Add cortex_ingester_active_series_per_tracker gauge metric
- Integrate into updateActiveSeries() periodic tick
- Matchers are validated and compiled during config unmarshalling
- Runtime hot-reloadable via existing runtime config overrides

Configuration example:
  overrides:
    tenant-123:
      active_series_trackers:
        - name: api_metrics
          matchers: '{__name__=~"api_.*"}'

Metric emitted:
  cortex_ingester_active_series_per_tracker{user="tenant", name="api_metrics"} 42

Signed-off-by: Ben Ye <benye@amazon.com>

Author: yeya24

CHANGELOG.md

@@ -4,6 +4,7 @@
 * [CHANGE] Querier: Make query time range configurations per-tenant: `query_ingesters_within`, `query_store_after`, and `shuffle_sharding_ingesters_lookback_period`. Uses `model.Duration` instead of `time.Duration` to support serialization but has minimum unit of 1ms (nanoseconds/microseconds not supported). #7160
 * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446
 * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481
+* [FEATURE] Ingester: Add experimental active series tracker that counts active series by configurable label matchers (including regex) per tenant and exposes `cortex_ingester_active_series_per_tracker` metric. Configured via `active_series_trackers` in runtime config overrides. #7476
 * [FEATURE] Ruler: Add per-tenant `ruler_alert_generator_url_template` runtime config option to customize alert generator URLs using Go templates. Supports Grafana Explore, Perses, and other UIs. #7302
 * [FEATURE] Distributor: Add experimental `-distributor.enable-start-timestamp` flag for Prometheus Remote Write 2.0. When enabled, `StartTimestamp (ST)` is ingested. #7371
 * [FEATURE] Memberlist: Add `-memberlist.cluster-label` and `-memberlist.cluster-label-verification-disabled` to prevent accidental cross-cluster gossip joins and support rolling label rollout. #7385

docs/configuration/config-file-reference.md

@@ -4164,6 +4164,10 @@ The `limits_config` configures default and per-tenant limits imposed by Cortex s
 # [max_series]
 [limits_per_label_set: <list of LimitsPerLabelSet> | default = []]
 
+# List of active series tracker configurations. Each tracker counts active
+# series matching its matchers and exposes the count as a metric.
+[active_series_trackers: <list of ActiveSeriesTrackerConfig> | default = []]
+
 # [EXPERIMENTAL] True to enable native histogram.
 # CLI flag: -blocks-storage.tsdb.enable-native-histograms
 [enable_native_histograms: <boolean> | default = false]
@@ -6892,6 +6896,17 @@ limits:
 [label_set: <map of string (labelName) to string (labelValue)> | default = []]
 ```
 
+### `ActiveSeriesTrackerConfig`
+
+```yaml
+# Name of the tracker, used as a label value in the emitted metric.
+[name: <string> | default = ""]
+
+# PromQL series selector (e.g. {__name__=~"api_.*"}). All matchers must match
+# for a series to be counted.
+[matchers: <string> | default = ""]
+```
+
 ### `PriorityDef`
 
 ```yaml

docs/configuration/v1-guarantees.md

@@ -130,3 +130,6 @@ Currently experimental features are:
     - `-validation.max-label-cardinality-for-unoptimized-regex` (int) - maximum label cardinality
     - `-validation.max-total-label-value-length-for-unoptimized-regex` (int) - maximum total length of all label values in bytes
 - HATracker: `-distributor.ha-tracker.enable-startup-sync` (bool) - If enabled, fetches all tracked keys on startup to populate the local cache.
+- Ingester: Active Series Tracker
+  - Per-tenant `active_series_trackers` configuration in runtime config overrides
+  - Counts active series matching PromQL label matchers and exposes `cortex_ingester_active_series_per_tracker` metric

docs/proposals/active-series-tracker.md

@@ -0,0 +1,71 @@
+# Active Series Tracker
+
+## Problem
+
+AMP needs to monitor active series counts by configurable patterns (e.g., all series with `__name__=~"api_.*"`) for internal observability. The existing `LimitsPerLabelSet` feature is unsuitable because:
+
+1. **No regex matching** — only supports exact `label=value` matching.
+2. **Default partition side-effects** — adding labelset buckets reduces the default partition count.
+3. **Coupled to limit enforcement** — designed for enforcing series limits, not pure monitoring.
+
+## Requirements
+
+- Track active series counts by configurable label matchers (including regex).
+- Expose counts as Prometheus metrics on the ingester (internal only, not vended to customers).
+- Configuration supports **per-tenant overrides** with a **default** fallback (same pattern as all other Limits fields).
+- **Runtime hot-reloadable** via the existing runtime config file mechanism.
+- **No limit enforcement** — purely observational.
+- **No default partition** — unmatched series are simply not tracked.
+- A series can match multiple tracker entries simultaneously.
+
+## Design
+
+### Configuration
+
+Tracker config lives in the `Limits` struct, following the same per-tenant override pattern as `LimitsPerLabelSet`:
+
+```yaml
+# Default trackers (applied to all tenants without overrides)
+limits:
+  active_series_trackers:
+    - name: api_metrics
+      matchers: '{__name__=~"api_.*"}'
+
+# Per-tenant overrides via runtime config
+overrides:
+  tenant-123:
+    active_series_trackers:
+      - name: api_metrics
+        matchers: '{__name__=~"api_.*"}'
+      - name: system_metrics
+        matchers: '{__name__=~"node_.*|process_.*"}'
+```
+
+The `matchers` field uses standard PromQL matcher syntax parsed via `parser.ParseMetricSelector`.
+
+### Runtime Reload
+
+Tracker config is part of `Limits`, which is reloaded via the runtime config manager every `runtime-config.reload-period` (default 10s). Matchers are parsed and validated during YAML/JSON unmarshalling. Invalid configs are rejected (existing config stays active).
+
+### Metrics
+
+A new gauge metric emitted per ingester:
+
+```
+cortex_ingester_active_series_per_tracker{user="<tenant>", name="<tracker_name>"} <count>
+```
+
+### Matching Logic
+
+On each active series metrics update tick (default 1min), for each tenant:
+1. Read the tenant's tracker config via `i.limits.ActiveSeriesTrackers(userID)`
+2. For each tracker, count active series whose labels satisfy all matchers
+3. Emit the gauge metric
+
+A series can match multiple trackers. Tenants without configured trackers emit no tracker metrics.
+
+### Performance Considerations
+
+- Matching runs once per update period (default 1min), not on every sample ingestion.
+- The number of trackers is expected to be small (< 10).
+- Compiled matchers are cached in the parsed Limits and only recompiled on config change.

integration/active_series_tracker_test.go

@@ -0,0 +1,160 @@
+//go:build requires_docker
+
+package integration
+
+import (
+	"fmt"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/prometheus/prometheus/model/labels"
+	"github.com/prometheus/prometheus/prompb"
+	"github.com/stretchr/testify/require"
+	"gopkg.in/yaml.v3"
+
+	"github.com/cortexproject/cortex/integration/e2e"
+	e2edb "github.com/cortexproject/cortex/integration/e2e/db"
+	"github.com/cortexproject/cortex/integration/e2ecortex"
+)
+
+func TestActiveSeriesTrackerPerTenant(t *testing.T) {
+	s, err := e2e.NewScenario(networkName)
+	require.NoError(t, err)
+	defer s.Close()
+
+	// Write runtime config with per-tenant active series trackers.
+	runtimeConfig := map[string]interface{}{
+		"overrides": map[string]interface{}{
+			"user-1": map[string]interface{}{
+				"active_series_trackers": []map[string]string{
+					{"name": "api_metrics", "matchers": `{__name__=~"api_.*"}`},
+					{"name": "node_metrics", "matchers": `{__name__=~"node_.*"}`},
+				},
+			},
+		},
+	}
+	runtimeCfgYAML, err := yaml.Marshal(runtimeConfig)
+	require.NoError(t, err)
+	require.NoError(t, writeFileToSharedDir(s, runtimeConfigFile, runtimeCfgYAML))
+
+	flags := BlocksStorageFlags()
+	flags["-distributor.shard-by-all-labels"] = "true"
+	flags["-ingester.active-series-metrics-enabled"] = "true"
+	flags["-ingester.active-series-metrics-update-period"] = "2s"
+	flags["-ingester.active-series-metrics-idle-timeout"] = "5m"
+	flags["-runtime-config.file"] = filepath.Join(e2e.ContainerSharedDir, runtimeConfigFile)
+	flags["-runtime-config.reload-period"] = "1s"
+	flags["-alertmanager.web.external-url"] = "http://localhost/alertmanager"
+	flags["-alertmanager-storage.backend"] = "local"
+	flags["-alertmanager-storage.local.path"] = filepath.Join(e2e.ContainerSharedDir, "alertmanager_configs")
+
+	require.NoError(t, writeFileToSharedDir(s, "alertmanager_configs", []byte{}))
+
+	consul := e2edb.NewConsul()
+	minio := e2edb.NewMinio(9000, flags["-blocks-storage.s3.bucket-name"])
+	require.NoError(t, s.StartAndWaitReady(consul, minio))
+
+	flags["-ring.store"] = "consul"
+	flags["-consul.hostname"] = consul.NetworkHTTPEndpoint()
+
+	cortex := e2ecortex.NewSingleBinary("cortex-1", flags, "")
+	require.NoError(t, s.StartAndWaitReady(cortex))
+
+	// Wait until the ring is ready.
+	require.NoError(t, cortex.WaitSumMetrics(e2e.Equals(float64(512)), "cortex_ring_tokens_total"))
+
+	c, err := e2ecortex.NewClient(cortex.HTTPEndpoint(), cortex.HTTPEndpoint(), "", "", "user-1")
+	require.NoError(t, err)
+
+	now := time.Now()
+	for _, name := range []string{"api_requests_total", "api_errors_total", "node_cpu_seconds", "process_memory_bytes"} {
+		series, _ := generateSeries(name, now, prompb.Label{Name: "job", Value: "test"})
+		res, err := c.Push(series)
+		require.NoError(t, err)
+		require.Equal(t, 200, res.StatusCode, fmt.Sprintf("push %s failed", name))
+	}
+
+	// user-1 has trackers: api_metrics (matches 2), node_metrics (matches 1).
+	require.NoError(t, cortex.WaitSumMetricsWithOptions(
+		e2e.Equals(2),
+		[]string{"cortex_ingester_active_series_per_tracker"},
+		e2e.WithLabelMatchers(
+			labels.MustNewMatcher(labels.MatchEqual, "user", "user-1"),
+			labels.MustNewMatcher(labels.MatchEqual, "name", "api_metrics"),
+		),
+		e2e.WaitMissingMetrics,
+	))
+
+	require.NoError(t, cortex.WaitSumMetricsWithOptions(
+		e2e.Equals(1),
+		[]string{"cortex_ingester_active_series_per_tracker"},
+		e2e.WithLabelMatchers(
+			labels.MustNewMatcher(labels.MatchEqual, "user", "user-1"),
+			labels.MustNewMatcher(labels.MatchEqual, "name", "node_metrics"),
+		),
+		e2e.WaitMissingMetrics,
+	))
+
+	// user-2 has no trackers configured — should have no tracker metrics.
+	c2, err := e2ecortex.NewClient(cortex.HTTPEndpoint(), cortex.HTTPEndpoint(), "", "", "user-2")
+	require.NoError(t, err)
+
+	series2, _ := generateSeries("api_requests_total", now, prompb.Label{Name: "job", Value: "test"})
+	res, err := c2.Push(series2)
+	require.NoError(t, err)
+	require.Equal(t, 200, res.StatusCode)
+
+	// Wait for user-2 active series to be counted.
+	require.NoError(t, cortex.WaitSumMetricsWithOptions(
+		e2e.Equals(1),
+		[]string{"cortex_ingester_active_series"},
+		e2e.WithLabelMatchers(labels.MustNewMatcher(labels.MatchEqual, "user", "user-2")),
+		e2e.WaitMissingMetrics,
+	))
+
+	// user-2 should have no tracker metrics.
+	sum, err := cortex.SumMetrics(
+		[]string{"cortex_ingester_active_series_per_tracker"},
+		e2e.WithLabelMatchers(labels.MustNewMatcher(labels.MatchEqual, "user", "user-2")),
+		e2e.SkipMissingMetrics,
+	)
+	require.NoError(t, err)
+	require.Equal(t, 0.0, sum[0])
+
+	// Now update runtime config: remove node_metrics tracker for user-1.
+	runtimeConfig2 := map[string]interface{}{
+		"overrides": map[string]interface{}{
+			"user-1": map[string]interface{}{
+				"active_series_trackers": []map[string]string{
+					{"name": "api_metrics", "matchers": `{__name__=~"api_.*"}`},
+				},
+			},
+		},
+	}
+	runtimeCfgYAML2, err := yaml.Marshal(runtimeConfig2)
+	require.NoError(t, err)
+	require.NoError(t, writeFileToSharedDir(s, runtimeConfigFile, runtimeCfgYAML2))
+
+	// Wait for the stale node_metrics tracker metric to be removed.
+	require.NoError(t, cortex.WaitSumMetricsWithOptions(
+		e2e.Equals(0),
+		[]string{"cortex_ingester_active_series_per_tracker"},
+		e2e.WithLabelMatchers(
+			labels.MustNewMatcher(labels.MatchEqual, "user", "user-1"),
+			labels.MustNewMatcher(labels.MatchEqual, "name", "node_metrics"),
+		),
+		e2e.SkipMissingMetrics,
+	))
+
+	// api_metrics tracker should still work.
+	require.NoError(t, cortex.WaitSumMetricsWithOptions(
+		e2e.Equals(2),
+		[]string{"cortex_ingester_active_series_per_tracker"},
+		e2e.WithLabelMatchers(
+			labels.MustNewMatcher(labels.MatchEqual, "user", "user-1"),
+			labels.MustNewMatcher(labels.MatchEqual, "name", "api_metrics"),
+		),
+		e2e.WaitMissingMetrics,
+	))
+}

pkg/ingester/active_series.go

@@ -248,3 +248,13 @@ func (s *activeSeriesStripe) getActiveNativeHistogram() int {
 
 	return s.activeNativeHistogram
 }
+
+// matchesAll returns true if the labels satisfy all given matchers.
+func matchesAll(lbs labels.Labels, matchers []*labels.Matcher) bool {
+	for _, m := range matchers {
+		if !m.Matches(lbs.Get(m.Name)) {
+			return false
+		}
+	}
+	return true
+}