Fork test: PR #1129 - feat(anomaly): add LLM-generated AI explanations for has_no_row_anomalies

demos/dqx_row_anomaly_detection_demo.py

@@ -425,7 +425,7 @@ def inject_anomalies_and_dq_issues(
 # MAGIC %md
 # MAGIC ---
 # MAGIC
-# MAGIC ## Section 5: (Optional) Review Results to understand why some records are anomalous
+# MAGIC ## Section 5a: (Optional) Review Results to understand why some records are anomalous
 # MAGIC
 # MAGIC You’ll see flagged anomalies, severity percentiles, and top contributors.
 # MAGIC
@@ -485,6 +485,63 @@ def inject_anomalies_and_dq_issues(
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC ---
+# MAGIC ## Section 5b: AI explanations
+# MAGIC
+# MAGIC AI explanations are **on by default** — the checks in Section 4 already produced
+# MAGIC `_dq_info[0].anomaly.ai_explanation` (a plain-language `narrative`, `business_impact`,
+# MAGIC `action`, and the deterministic `top_features` pattern). The LLM call runs **inside Spark**
+# MAGIC via the SQL `ai_query` function against a Databricks Model Serving endpoint — no extra
+# MAGIC dependency, and rows are grouped so the model is called **once per group** (capped by
+# MAGIC `max_groups`). This requires **Databricks serverless compute or Databricks Runtime 15.4 LTS
+# MAGIC or above** (where `ai_query` is available). If `ai_query` is unavailable or no endpoint is
+# MAGIC reachable, explanations are skipped with a warning and scoring still completes.
+# MAGIC
+# MAGIC This cell just shows how to override the endpoint or turn explanations off — none of these
+# MAGIC kwargs are required.
+
+# COMMAND ----------
+# DBTITLE 1,Apply checks (AI explanations are on by default)
+
+checks_with_ai = [
+    DQDatasetRule(
+        check_func=has_no_row_anomalies,
+        check_func_kwargs={
+            "model_name": model_name_auto,
+            "registry_table": registry_table,
+            # All optional — explanations + contributions are on by default:
+            # "enable_ai_explanation": False,                       # turn explanations off
+            # "ai_explanation_llm_model_config": {"model_name": "databricks-claude-sonnet-4-5"},  # override endpoint
+            # "redact_columns": ["region"],                         # keep sensitive names out of the prompt
+            # "max_groups": 500,                                    # cap on LLM calls per run
+        },
+    )
+]
+
+df_ai = dq_engine.apply_checks(df_new, checks_with_ai)
+
+anomaly = F.col("_dq_info").getItem(0).getField("anomaly")
+explanation = anomaly.getField("ai_explanation")
+print("🤖 Top anomalies with AI explanations:\n")
+display(
+    df_ai.filter(anomaly.getField("is_anomaly") == True)
+    .orderBy(anomaly.getField("severity_percentile").desc())
+    .select(
+        "transaction_id",
+        "amount",
+        "quantity",
+        F.round(anomaly.getField("severity_percentile"), 1).alias("severity_percentile"),
+        explanation.getField("top_features").alias("top_features"),
+        explanation.getField("narrative").alias("why_flagged"),
+        explanation.getField("business_impact").alias("business_impact"),
+        explanation.getField("action").alias("suggested_action"),
+    )
+    .limit(10)
+)
+
+# COMMAND ----------
+
 # MAGIC %md
 # MAGIC ---
 # MAGIC ## Section 6: (Optional) Threshold Tradeoffs
@@ -662,7 +719,7 @@ def inject_anomalies_and_dq_issues(
         check_func_kwargs={
             "model_name": model_name_manual,
             "threshold": 95.0,
-            "enable_contributions": True,  # default is False
+            "enable_contributions": True,  # on by default; shown here for clarity
             "registry_table": registry_table
         }
     )

docs/dqx/docs/guide/row_anomaly_detection/index.mdx

@@ -245,7 +245,7 @@ However, when you run a row anomaly detection check, DQX also adds a `_dq_info`
 
 The info column is an array of structs, with one element per anomaly detection check that was applied.
 
-Feature contributions are disabled by default for faster scoring. You can enable with `enable_contributions` parameter:
+Feature contributions are on by default and are computed only for anomalous rows (severity at or above the threshold) — non-anomalous rows carry a `null` contributions map, so the SHAP cost scales with the number of anomalies, not the table size. Set `enable_contributions=False` to skip them entirely for the fastest scoring. Severity percentiles are always computed for every row, so you can still count would-be anomalies at other thresholds after scoring; but if you want contributions (and AI explanations) for rows below your enforcement threshold, score with the lower exploration threshold and filter afterwards:
 
 ```python
 DQDatasetRule(
@@ -254,7 +254,7 @@ DQDatasetRule(
   check_func_kwargs={
     "model_name": "catalog.schema.orders_monitor",  # fully qualified name
     "registry_table": "catalog.schema.dqx_anomaly_models",   # fully qualified name
-    "enable_contributions": True,
+    # "enable_contributions": False,  # optional: skip SHAP (also disables AI explanations)
   }
 )
 ```
@@ -292,7 +292,7 @@ For full parameter and schema details, see [Row Anomaly Detection in Quality Che
 1. **Feature engineering** (automatic): DQX detects column types and creates features (numerical standardized, categorical one-hot or frequency-encoded, temporal expanded to hour/day/month/weekend).
 2. **Smart sampling and training**: DQX samples your data (default 30%, capped at 1M rows), trains an ensemble of Isolation Forest models, and captures baseline statistics for drift detection.
 3. **Model registry**: Models and metadata live in MLflow and a Delta table; segmented models use deterministic names (for example `__seg_region=US_tier=gold`).
-4. **Scoring and explanation**: Raw scores are normalized to a 0–100 severity percentile. Set `enable_contributions=True` for SHAP contributions (which features drove each score); default is `False` for faster scoring.
+4. **Scoring and explanation**: Raw scores are normalized to a 0–100 severity percentile. SHAP contributions (which features drove each score) are on by default and computed only for anomalous rows; set `enable_contributions=False` to skip them for the fastest scoring.
 5. **Auto-discovery of columns and segments**: When you call `train()` without `columns` or `segment_by`, DQX automatically discovers both. It selects numeric columns with enough variance as features and may **auto-segment** when it finds suitable segment columns: categorical or low-cardinality columns with 2–50 distinct values, low null rate, and enough rows per segment (for example region, product category). If your auto-trained model is segmented, that is expected. To force a single global model, pass `segment_by=[]` (or omit segment columns from the data used for discovery) or set `columns` and `segment_by` explicitly.
 
 ### Why Isolation Forest?
@@ -322,11 +322,58 @@ Each element is a **struct** with a shared “wide” schema. Currently, the onl
 | `threshold` | double | Severity percentile threshold used (e.g. 95.0). |
 | `model` | string | Full model name (e.g. Unity Catalog name). |
 | `segment` | map<string, string> | Segment key-value pairs for segmented models; `null` for global models. |
-| `contributions` | map<string, double> | Per-feature contribution percentages (0–100). Present when `enable_contributions=True`; `null` by default. |
+| `contributions` | map<string, double> | Per-feature contribution percentages (0–100). On by default (`enable_contributions=True`); populated only for anomalous rows — `null` for non-anomalous rows or if you set it `False`. |
 | `confidence_std` | double | Ensemble score standard deviation. Present when `enable_confidence_std=True`; `null` otherwise. |
+| `ai_explanation` | struct | LLM-generated explanation for the row's `(segment, pattern)` group. On by default (`enable_ai_explanation=True`); `null` for non-anomalous rows, when disabled, or when no serving endpoint is reachable. See [AI explanations](#ai-explanations) below. |
+
+The nested `ai_explanation` struct (populated when AI explanations are on — the default):
+
+| Field | Type | Description |
+|--------|------|-------------|
+| `narrative` | string | Plain-language description of why the group was flagged. |
+| `business_impact` | string | Likely downstream impact if the rows are processed unchanged. |
+| `action` | string | What an analyst should investigate. |
+| `top_features` | string | Deterministic top-2 contributing features (e.g. `amount+quantity`) — the group's pattern key. |
+| `group_size` | long | Number of anomalous rows in this `(segment, pattern)` group. |
+| `group_avg_severity` | double | Mean `severity_percentile` across the group. |
 
 **Access in PySpark:** use `F.element_at(F.col("_dq_info"), 1)` for the first element (1-based), then `.getField("anomaly").getField("severity_percentile")` etc. Alternatively `F.col("_dq_info").getItem(0)` for 0-based index (see [Troubleshooting](/docs/guide/row_anomaly_detection/troubleshooting) for Spark Connect–friendly patterns).
 
+### AI explanations
+
+AI explanations are **on by default**: each anomalous row gets a plain-language, LLM-generated explanation in `_dq_info[0].anomaly.ai_explanation` answering *why was this flagged, what's the impact, and what should I do* — without anyone reading raw SHAP percentages. Set `enable_ai_explanation=False` to turn it off (or `enable_contributions=False`, which disables explanations too, since they use the SHAP contributions as input). The explanation is AI-generated from the anomaly signal (feature names + SHAP + severity); it is **not** grounded in your catalog's table/column descriptions, so treat the business-impact and action as a starting point, not authoritative.
+
+The call runs inside Spark via the SQL `ai_query` function against a Databricks Model Serving endpoint, so it needs no extra setup and scales with the cluster. This requires **Databricks serverless compute or Databricks Runtime 15.4 LTS or above** (where `ai_query` is available); on older runtimes explanations are skipped with a warning and scoring still completes. Similar anomalous rows are grouped together and the model is called **once per group** rather than once per row, so cost stays predictable on large datasets.
+
+Rows are grouped by their top two contributing features, so occasionally two different kinds of anomaly that share the same top two features land in the same group and get one shared explanation. That's intentional — it keeps the number of AI calls (and the cost) low, at the price of a slightly more general explanation for those rows.
+
+```python
+from databricks.labs.dqx.rule import DQDatasetRule
+from databricks.labs.dqx.anomaly.check_funcs import has_no_row_anomalies
+
+# Contributions + AI explanations are on by default — this is all you need:
+checks = [
+  DQDatasetRule(
+    criticality="error",
+    check_func=has_no_row_anomalies,
+    check_func_kwargs={
+      "model_name": "catalog.schema.orders_monitor",
+      "registry_table": "catalog.schema.dqx_anomaly_models",
+      "threshold": 95.0,
+      # Optional: override the serving endpoint (defaults to databricks-claude-sonnet-4-5)
+      # "ai_explanation_llm_model_config": {"model_name": "databricks-claude-sonnet-4-5"},
+      # Optional: turn explanations off -> "enable_ai_explanation": False
+    },
+  )
+]
+```
+
+<Admonition type="note" title="Cost and privacy">
+* Explanations are **on by default** and call a Model Serving endpoint, so they add per-run LLM cost. `max_groups` (default 500) caps how many groups the model is called for per run. For segmented models the cap is shared across segments (at least one call each), so if you set `max_groups` lower than the number of segments you still get one call per segment — a warning is logged when that happens. Set `enable_ai_explanation=False` to turn explanations off.
+* No serving endpoint? If the configured endpoint isn't reachable (e.g. Foundation Model APIs aren't enabled in the workspace), explanations are skipped with a warning and scoring still completes — nothing breaks.
+* `redact_columns` keeps the listed feature and segment names out of the prompt. Segment **values** for non-redacted keys are sent verbatim — avoid segmenting on sensitive columns, or list them in `redact_columns`.
+</Admonition>
+
 ## Practical examples (non-technical)
 
 - A sudden surge of high-value orders in a region that usually has low spend.