Skip to content

Commit d3037df

Browse files
authored
[anomalydetection] Unify scorer, make it a correlator (4) (#52492)
### What does this PR do? Unifies the anomaly scorer into a single `anomalyScorer` struct implementing the `Correlator` interface in a single file. We can now benchmark the scorer. This removes the helper reporting logs, it's merged into this struct. This contains the main score logic and also the subscribe logic which is used to create events (high severity = event). > [!NOTE] > We use "the severity becomes high" and "the severity isn't high anymore" as start / end of correlation. This will be reworked in #52493 to directly emit event at the start. The config can now enable/disable logs and reporter events (telemetry always enabled if component enabled). New config keys for output: ```yaml anomaly_detection: anomaly_scorer: enabled: true output: logs: true correlation_events: false ``` #### Diagram ```mermaid flowchart LR IN["ProcessAnomaly(a)"] --> PEND["pending map"] PEND --> ADV subgraph ADV["Advance(t) — mu held"] EWMA["merge → evict → bucket → EWMA"] end ADV -->|"[]secEWMA\nmu released"| SUBS subgraph SUBS["Per-subscription state machines — subsMu"] FSM["Low / Medium / High FSM\n+ cooldown per sub"] FSM -->|"transition"| CB["Listener.OnSeverityTransition(evt)"] end CB -->|"self-sub\n(internal watcher)"| IW["gauges · logs · episode tracking"] IW --> AC["ActiveCorrelations()"] ADV --> SC["ScoreState() / LastScore()"] SUBS --> SB["Subscribe(cfg) → func()\nexposes via SubscribeScorer on Component"] ``` ### Motivation Nice score for kafka scenario, no false positives (using bocpd+tukey_biweight+holt_residual): <img width="531" height="304" alt="Screenshot 2026-06-19 at 11 20 44" src="https://github.com/user-attachments/assets/ace05855-359a-4064-b42c-994b6baf5a1e" /> ### Describe how you validated your changes ### Additional Notes Co-authored-by: celian.raimbault <celian.raimbault@datadoghq.com>
1 parent 4644313 commit d3037df

19 files changed

Lines changed: 1024 additions & 790 deletions

File tree

comp/anomalydetection/observer/AGENTS.md

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ and the testbench use the same engine.
4242
| `impl/log_pattern_extractor.go` | Log → virtual metrics via pattern clustering |
4343
| `impl/log_metrics_extractor.go` | Log → virtual metrics via regex extraction |
4444
| `impl/anomaly_correlator_time_cluster.go` | Default time-proximity correlator |
45+
| `impl/anomaly_scorer.go` | Unified EWMA anomaly scorer (Correlator + standalone replay) |
4546
| `impl/patterns/` | Tokenizer + clusterer used by log pattern extractor |
4647

4748
### Component catalog (defaults)
@@ -58,8 +59,27 @@ Registered in `impl/component_catalog.go`. Enabled by default unless noted:
5859
| Detector | `cusum`, `scanmw`, `scanwelch`, `holt_residual`, `tukey_biweight` | off |
5960
| Correlator | `time_cluster` | on |
6061
| Correlator | `cross_signal`, `passthrough` | off |
62+
| Correlator | `anomaly_scorer` | off |
63+
64+
Toggle detectors/correlators/extractors via `anomaly_detection.detectors.<name>.enabled` in datadog.yaml.
65+
66+
The `anomaly_scorer` correlator has a **dedicated config namespace** under `anomaly_detection.anomaly_scorer.*` (not `detectors.*`) with an `output` sub-section controlling logs and correlation events:
67+
68+
```yaml
69+
anomaly_detection:
70+
anomaly_scorer:
71+
enabled: true
72+
alpha: 0.3
73+
window_secs: 30
74+
low_threshold: 0.030
75+
high_threshold: 0.060
76+
output:
77+
logs: true
78+
correlation_events: false
79+
cooldown_secs: 300
80+
```
6181
62-
Toggle via `anomaly_detection.detectors.<name>.enabled` in datadog.yaml.
82+
The scorer is also available standalone (without the engine) via `NewAnomalyScorer` in `impl/` for testbench replay.
6383

6484
## Key Design Decisions
6585

comp/anomalydetection/observer/def/types.go

Lines changed: 30 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -309,8 +309,8 @@ type Correlator interface {
309309
Reset()
310310
}
311311

312-
// ScorerConfig holds the tunable parameters for the anomaly scoring pipeline.
313-
type ScorerConfig struct {
312+
// AnomalyScorerConfig holds the tunable parameters for the anomaly scoring pipeline.
313+
type AnomalyScorerConfig struct {
314314
// Alpha is the EWMA smoothing factor (0 < α ≤ 1). Lower = smoother.
315315
Alpha float64 `json:"alpha"`
316316
// SaturationK is the saturation constant k: saturation = 1−exp(−n/k).
@@ -332,16 +332,16 @@ type ScorerConfig struct {
332332
// specific detector names. Each entry is [low, medium, high, xhigh] thresholds.
333333
// Detectors not in this map default to level 2 (Medium) regardless of their score.
334334
DetectorThresholds map[string][4]float64 `json:"detector_thresholds,omitempty"`
335-
// MaxBuckets overrides the number of ScoreBucket entries retained in
335+
// MaxBuckets overrides the number of AnomalyScoreBucket entries retained in
336336
// ScoreState(). 0 (default) means "cap at WindowSecs", which is the
337337
// correct behaviour for the live agent. Set to a large positive value
338338
// (e.g. math.MaxInt64) to keep an unlimited history for offline replay.
339339
MaxBuckets int64 `json:"max_buckets,omitempty"`
340340
}
341341

342-
// ScoreBucket is the per-second telemetry unit emitted by the scorer.
342+
// AnomalyScoreBucket is the per-second telemetry unit emitted by the scorer.
343343
// One bucket is produced for every 1-second tick, even if it has no anomalies.
344-
type ScoreBucket struct {
344+
type AnomalyScoreBucket struct {
345345
// Second is the Unix timestamp (floor) for this bucket.
346346
Second int64 `json:"second"`
347347
// Bins[L] is the number of deduplicated anomalies at level L (0=VeryLow … 4=XHigh).
@@ -363,17 +363,17 @@ const (
363363
SeverityHigh SeverityLevel = 2
364364
)
365365

366-
// ScorerEventDirection describes whether a severity transition is an
366+
// AnomalyScorerEventDirection describes whether a severity transition is an
367367
// escalation or de-escalation.
368-
type ScorerEventDirection int
368+
type AnomalyScorerEventDirection int
369369

370370
const (
371-
// ScorerEventBoth delivers transitions in either direction (default zero value).
372-
ScorerEventBoth ScorerEventDirection = 0
373-
// ScorerEventEscalation delivers only transitions where ToLevel > FromLevel.
374-
ScorerEventEscalation ScorerEventDirection = 1
375-
// ScorerEventDeescalation delivers only transitions where ToLevel < FromLevel.
376-
ScorerEventDeescalation ScorerEventDirection = 2
371+
// AnomalyScorerEventBoth delivers transitions in either direction (default zero value).
372+
AnomalyScorerEventBoth AnomalyScorerEventDirection = 0
373+
// AnomalyScorerEventEscalation delivers only transitions where ToLevel > FromLevel.
374+
AnomalyScorerEventEscalation AnomalyScorerEventDirection = 1
375+
// AnomalyScorerEventDeescalation delivers only transitions where ToLevel < FromLevel.
376+
AnomalyScorerEventDeescalation AnomalyScorerEventDirection = 2
377377
)
378378

379379
// SeverityEvent records a severity state-machine transition.
@@ -384,72 +384,48 @@ type SeverityEvent struct {
384384
FromLevel SeverityLevel `json:"from_level"`
385385
// ToLevel is the state after the transition.
386386
ToLevel SeverityLevel `json:"to_level"`
387-
// Direction is ScorerEventEscalation when ToLevel > FromLevel, and
388-
// ScorerEventDeescalation when ToLevel < FromLevel.
389-
Direction ScorerEventDirection `json:"direction"`
387+
// Direction is AnomalyScorerEventEscalation when ToLevel > FromLevel, and
388+
// AnomalyScorerEventDeescalation when ToLevel < FromLevel.
389+
Direction AnomalyScorerEventDirection `json:"direction"`
390390
}
391391

392-
// ScorerListener receives severity state-machine transitions from the scorer.
393-
type ScorerListener interface {
392+
// AnomalyScorerListener receives severity state-machine transitions from the scorer.
393+
type AnomalyScorerListener interface {
394394
OnSeverityTransition(event SeverityEvent)
395395
}
396396

397-
// ScorerEventFilter selects which SeverityEvents are delivered to a listener.
397+
// AnomalyScorerEventFilter selects which SeverityEvents are delivered to a listener.
398398
// All conditions are ANDed; a nil or empty slice means "any value".
399-
// The zero value ScorerEventFilter{} matches every transition.
400-
type ScorerEventFilter struct {
399+
// The zero value AnomalyScorerEventFilter{} matches every transition.
400+
type AnomalyScorerEventFilter struct {
401401
// FromLevels restricts to events whose FromLevel is in the set.
402402
FromLevels []SeverityLevel
403403
// ToLevels restricts to events whose ToLevel is in the set.
404404
ToLevels []SeverityLevel
405405
// Direction restricts by escalation or de-escalation.
406-
Direction ScorerEventDirection
406+
Direction AnomalyScorerEventDirection
407407
}
408408

409-
// AnomalyScorerConfiguration is the single object passed to SubscribeScorer
409+
// AnomalyScorerConfiguration is the single object passed to Subscribe
410410
// when registering a listener. It bundles who to call (Listener), which
411411
// transitions to deliver (Filter), and per-subscription state-machine tuning.
412412
type AnomalyScorerConfiguration struct {
413413
// Listener is called for each matching severity transition. Required.
414-
Listener ScorerListener
414+
Listener AnomalyScorerListener
415415
// Filter controls which transitions are delivered.
416-
// Zero value ScorerEventFilter{} delivers all transitions.
417-
Filter ScorerEventFilter
416+
// Zero value AnomalyScorerEventFilter{} delivers all transitions.
417+
Filter AnomalyScorerEventFilter
418418
// CooldownSecs is the minimum number of seconds that must elapse after a
419419
// delivered transition before a downward (de-escalation) transition can be
420420
// delivered again.
421421
// Zero means no cooldown (every matching transition is delivered).
422422
CooldownSecs int64
423423
}
424424

425-
// ScoreState is the accumulated telemetry snapshot from the scorer.
426-
type ScoreState struct {
427-
Buckets []ScoreBucket `json:"buckets"`
428-
Config ScorerConfig `json:"config"`
429-
}
430-
431-
// AnomalyScorer computes a smoothed anomaly intensity signal (EWMA) from the
432-
// stream of anomalies produced by the detection pipeline. It mirrors the
433-
// Correlator lifecycle: ProcessAnomaly → Advance (once per second tick) → ScoreState → Reset.
434-
type AnomalyScorer interface {
435-
// Name returns the scorer name for debugging.
436-
Name() string
437-
// ProcessAnomaly feeds a raw anomaly into the scorer's current-second buffer.
438-
ProcessAnomaly(a Anomaly)
439-
// Advance finalises the bucket at dataTime (unix seconds) and runs the
440-
// EWMA + state-machine update for that second. Callers must invoke this
441-
// after each 1-second detection cycle.
442-
Advance(dataTime int64)
443-
// LastScore returns the most recently computed EWMA score. Returns 0
444-
// before the first Advance call.
445-
LastScore() float64
446-
// ScoreState returns the accumulated telemetry (all buckets + events so far).
447-
ScoreState() ScoreState
448-
// Reset clears all internal state for reanalysis.
449-
Reset()
450-
// Subscribe registers a listener to receive severity transitions matching
451-
// cfg.Filter. Returns an unsubscribe function. Safe to call concurrently.
452-
Subscribe(cfg AnomalyScorerConfiguration) func()
425+
// AnomalyScoreState is the accumulated telemetry snapshot from the scorer.
426+
type AnomalyScoreState struct {
427+
Buckets []AnomalyScoreBucket `json:"buckets"`
428+
Config AnomalyScorerConfig `json:"config"`
453429
}
454430

455431
// Reporter receives reports and displays or delivers them.

comp/anomalydetection/observer/impl/BUILD.bazel

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ go_library(
99
"anomaly_correlator_time_cluster.go",
1010
"anomaly_processor_correlator.go",
1111
"anomaly_scorer.go",
12-
"anomaly_scorer_helper.go",
1312
"component_catalog.go",
1413
"context_provider.go",
1514
"correlation_identity.go",
@@ -49,13 +48,10 @@ go_library(
4948
"//comp/anomalydetection/recorder/def",
5049
"//comp/anomalydetection/reporter/def",
5150
"//comp/core/config",
52-
"//comp/core/hostname/hostnameinterface/def",
5351
"//comp/core/log/def",
5452
"//comp/core/telemetry/def",
5553
"//comp/core/telemetry/impl/noops",
5654
"//comp/def",
57-
"//comp/forwarder/eventplatform/def",
58-
"//pkg/logs/message",
5955
"//pkg/util/log",
6056
"//pkg/util/option",
6157
],

0 commit comments

Comments
 (0)