Merge branch 'aarthy/fixes' into 'enterprise'

feat(monitors): allow filtering by anomaly types and show more history

See merge request dkinternal/testgen/dataops-testgen!410

Author: ci bot

README.md

@@ -1,5 +1,5 @@
 # DataOps Data Quality TestGen
-![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green) [![Latest Version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2Ftags%2F&query=results%5B0%5D.name&label=latest%20version&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Docker Pulls](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2F&query=pull_count&style=flat&label=docker%20pulls&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Documentation](https://img.shields.io/badge/docs-On%20datakitchen.io-06A04A?style=flat)](https://docs.datakitchen.io/articles/#!dataops-testgen-help/dataops-testgen-help) [![Static Badge](https://img.shields.io/badge/Slack-Join%20Discussion-blue?style=flat&logo=slack)](https://data-observability-slack.datakitchen.io/join)
+![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green) [![Latest Version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2Ftags%2F&query=results%5B0%5D.name&label=latest%20version&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Docker Pulls](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2F&query=pull_count&style=flat&label=docker%20pulls&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Documentation](https://img.shields.io/badge/docs-On%20datakitchen.io-06A04A?style=flat)](https://docs.datakitchen.io/articles/dataops-testgen-help/dataops-testgen-help) [![Static Badge](https://img.shields.io/badge/Slack-Join%20Discussion-blue?style=flat&logo=slack)](https://data-observability-slack.datakitchen.io/join)
 
 *<p style="text-align: center;">DataOps Data Quality TestGen, or "TestGen" for short, can help you find data issues so you can alert your users and notify your suppliers. It does this by delivering simple, fast data quality test generation and execution by data profiling, new dataset screening and hygiene review, algorithmic generation of data quality validation tests, ongoing production testing of new data refreshes, and continuous anomaly monitoring of datasets. TestGen is part of DataKitchen's Open Source Data Observability.</p>*
 
@@ -110,7 +110,7 @@ Within the virtual environment, install the TestGen package with pip.
 pip install dataops-testgen
 ```
 
-Verify that the [_testgen_ command line](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-commands-and-details) works.
+Verify that the [_testgen_ command line](https://docs.datakitchen.io/articles/dataops-testgen-help/testgen-commands-and-details) works.
 ```shell
 testgen --help
 ```
@@ -187,7 +187,7 @@ python3 dk-installer.py tg delete-demo
 
 ### Upgrade to latest version
 
-New releases of TestGen are announced on the `#releases` channel on [Data Observability Slack](https://data-observability-slack.datakitchen.io/join), and release notes can be found on the [DataKitchen documentation portal](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-release-notes/a/h1_1691719522). Use the following command to upgrade to the latest released version.
+New releases of TestGen are announced on the `#releases` channel on [Data Observability Slack](https://data-observability-slack.datakitchen.io/join), and release notes can be found on the [DataKitchen documentation portal](https://docs.datakitchen.io/articles/dataops-testgen-help/testgen-release-notes/a/h1_1691719522). Use the following command to upgrade to the latest released version.
 
  ```shell
  python3 dk-installer.py tg upgrade
@@ -203,7 +203,7 @@ python3 dk-installer.py tg delete
 
 ### Access the _testgen_ CLI
 
-The [_testgen_ command line](https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-commands-and-details) can be accessed within the running container.
+The [_testgen_ command line](https://docs.datakitchen.io/articles/dataops-testgen-help/testgen-commands-and-details) can be accessed within the running container.
 
 ```shell
 docker compose exec engine bash
@@ -232,7 +232,7 @@ We recommend you start by going through the [Data Observability Overview Demo](h
 For support requests, [join the Data Observability Slack](https://data-observability-slack.datakitchen.io/join) 👋 and post on the `#support` channel.
 
 ### Connect to your database
-Follow [these instructions](https://docs.datakitchen.io/articles/#!dataops-testgen-help/connect-your-database) to improve the quality of data in your database.
+Follow [these instructions](https://docs.datakitchen.io/articles/dataops-testgen-help/connect-your-database) to improve the quality of data in your database.
 
 ### Community
 Talk and learn with other data practitioners who are building with DataKitchen. Share knowledge, get help, and contribute to our open-source project.

pyproject.toml

@@ -99,8 +99,8 @@ tg-patch-streamlit = "testgen.ui.scripts.patch_streamlit:patch"
 [project.urls]
 "Source Code" = "https://github.com/DataKitchen/dataops-testgen"
 "Bug Tracker" = "https://github.com/DataKitchen/dataops-testgen/issues"
-"Documentation" = "https://docs.datakitchen.io/articles/#!dataops-testgen-help/dataops-testgen-help"
-"Release Notes" = "https://docs.datakitchen.io/articles/#!dataops-testgen-help/testgen-release-notes"
+"Documentation" = "https://docs.datakitchen.io/articles/dataops-testgen-help/dataops-testgen-help"
+"Release Notes" = "https://docs.datakitchen.io/articles/dataops-testgen-help/testgen-release-notes"
 "Slack" = "https://data-observability-slack.datakitchen.io/join"
 "Homepage" = "https://example.com"
 

testgen/commands/test_thresholds_prediction.py

@@ -3,6 +3,7 @@
 from typing import ClassVar
 
 import pandas as pd
+from scipy import stats
 
 from testgen.common.database.database_service import (
     execute_db_queries,
@@ -29,13 +30,14 @@ class TestThresholdsPrediction:
         "prediction",
     )
     num_forecast = 10
+    t_distribution_threshold = 20
     z_score_map: ClassVar = {
-        ("lower_tolerance", PredictSensitivity.low): -2.0,    # 2.5th percentile
-        ("lower_tolerance", PredictSensitivity.medium): -1.5, # 7th percentile
-        ("lower_tolerance", PredictSensitivity.high): -1.0,   # 16th percentile
-        ("upper_tolerance", PredictSensitivity.high): 1.0,    # 84th percentile
-        ("upper_tolerance", PredictSensitivity.medium): 1.5,  # 93rd percentile
-        ("upper_tolerance", PredictSensitivity.low): 2.0,     # 97.5th percentile
+        ("lower_tolerance", PredictSensitivity.low): -3.0,    # 0.13th percentile
+        ("lower_tolerance", PredictSensitivity.medium): -2.5, # 0.62nd percentile
+        ("lower_tolerance", PredictSensitivity.high): -2.0,   # 2.3rd percentile
+        ("upper_tolerance", PredictSensitivity.high): 2.0,    # 97.7th percentile
+        ("upper_tolerance", PredictSensitivity.medium): 2.5,  # 99.4th percentile
+        ("upper_tolerance", PredictSensitivity.low): 3.0,     # 99.87th percentile
     }
 
     def __init__(self, test_suite: TestSuite, run_date: datetime):
@@ -71,9 +73,15 @@ def run(self) -> None:
                             ] if self.test_suite.predict_holiday_codes else None,
                         )
 
+                        num_points = len(history)
                         for key, z_score in self.z_score_map.items():
+                            if num_points < self.t_distribution_threshold:
+                                percentile = stats.norm.cdf(z_score)
+                                multiplier = stats.t.ppf(percentile, df=num_points - 1)
+                            else:
+                                multiplier = z_score
                             column = f"{key[0]}|{key[1].value}"
-                            forecast[column] = forecast["mean"] + (z_score * forecast["se"])
+                            forecast[column] = forecast["mean"] + (multiplier * forecast["se"])
 
                         next_date = forecast.index[0]
                         sensitivity = self.test_suite.predict_sensitivity or PredictSensitivity.medium

testgen/common/notifications/notifications.py

@@ -393,7 +393,7 @@ def get_body_template(self) -> str:
         </tr>
         <tr class="footer">
           <td>
-            <a href="https://docs.datakitchen.io/articles/#!dataops-testgen-help/introduction-to-dataops-testgen"
+            <a href="https://docs.datakitchen.io/articles/dataops-testgen-help/introduction-to-dataops-testgen"
                target="_blank">TestGen Help</a>
           </td>
           <td align="right">

testgen/template/data_chars/data_chars_update.sql

@@ -17,20 +17,37 @@ WITH new_chars AS (
       schema_name,
       table_name,
       run_date
+),
+updated_records AS (
+   UPDATE data_table_chars
+   SET approx_record_ct = n.approx_record_ct,
+      record_ct = n.record_ct,
+      column_ct = n.column_ct,
+      last_refresh_date = n.run_date,
+      drop_date = NULL
+   FROM new_chars n
+      INNER JOIN data_table_chars d ON (
+         n.table_groups_id = d.table_groups_id
+         AND n.schema_name = d.schema_name
+         AND n.table_name = d.table_name
+      )
+   WHERE data_table_chars.table_id = d.table_id
+   RETURNING data_table_chars.*, d.drop_date as old_drop_date
+)
+INSERT INTO data_structure_log (
+   table_groups_id,
+   table_id,
+   table_name,
+   change_date,
+   change
 )
-UPDATE data_table_chars
-SET approx_record_ct = n.approx_record_ct,
-   record_ct = n.record_ct,
-   column_ct = n.column_ct,
-   last_refresh_date = n.run_date,
-   drop_date = NULL
-FROM new_chars n
-   INNER JOIN data_table_chars d ON (
-      n.table_groups_id = d.table_groups_id
-      AND n.schema_name = d.schema_name
-      AND n.table_name = d.table_name
-   )
-WHERE data_table_chars.table_id = d.table_id;
+SELECT u.table_groups_id,
+   u.table_id,
+   u.table_name,
+   u.last_refresh_date,
+   'A'
+   FROM updated_records u
+   WHERE u.old_drop_date IS NOT NULL;
 
 -- Add new records
 WITH new_chars AS (
@@ -170,7 +187,7 @@ update_chars AS (
       )
    WHERE data_column_chars.table_id = d.table_id
       AND data_column_chars.column_name = d.column_name
-   RETURNING data_column_chars.*, d.db_data_type as old_data_type
+   RETURNING data_column_chars.*, d.db_data_type as old_data_type, d.drop_date as old_drop_date, n.run_date as run_date
 )
 INSERT INTO data_structure_log (
    table_groups_id,
@@ -193,7 +210,20 @@ SELECT u.table_groups_id,
    u.old_data_type,
    u.db_data_type
    FROM update_chars u
-   WHERE u.old_data_type <> u.db_data_type;
+   WHERE u.old_data_type <> u.db_data_type
+      AND u.old_drop_date IS NULL
+UNION ALL
+SELECT u.table_groups_id,
+   u.table_id,
+   u.column_id,
+   u.table_name,
+   u.column_name,
+   u.run_date,
+   'A',
+   NULL,
+   u.db_data_type
+   FROM update_chars u
+   WHERE u.old_drop_date IS NOT NULL;
 
 
 -- Add new records

testgen/ui/components/frontend/js/components/help_menu.js

@@ -23,7 +23,7 @@ import { Icon } from './icon.js';
 
 const { a, div, span } = van.tags;
 
-const baseHelpUrl = 'https://docs.datakitchen.io/articles/#!dataops-testgen-help/';
+const baseHelpUrl = 'https://docs.datakitchen.io/articles/dataops-testgen-help/';
 const releaseNotesTopic = 'testgen-release-notes';
 const upgradeTopic = 'upgrade-testgen';
 

testgen/ui/components/frontend/js/components/monitor_anomalies_summary.js

@@ -21,8 +21,13 @@
  * @property {number} lookback_end
  * @property {string?} project_code
  * @property {string?} table_group_id
+ *
+ * @typedef SummaryOptions
+ * @type {object}
+ * @property {function(string)?} onTagClick
+ * @property {object?} activeTypes
  */
-import { emitEvent } from '../utils.js';
+import { emitEvent, getValue, loadStylesheet } from '../utils.js';
 import { formatDuration, humanReadableDuration } from '../display_utils.js';
 import { withTooltip } from './tooltip.js';
 import van from '../van.min.js';
@@ -31,49 +36,63 @@ const { a, div, i, span } = van.tags;
 
 /**
  * @param {MonitorSummary} summary
- * @param {any?} topLabel
+ * @param {string?} label
+ * @param {SummaryOptions?} options
  */
-const AnomaliesSummary = (summary, label = 'Anomalies') => {
+const AnomaliesSummary = (summary, label = 'Anomalies', options = {}) => {
+    loadStylesheet('anomalies-summary', summaryStylesheet);
+
     if (!summary.lookback) {
         return span({class: 'text-secondary mt-3 mb-2'}, 'No monitor runs yet');
     }
 
-    const SummaryTag = (label, value, hasErrors, isTraining, isPending) => div(
-        {class: 'flex-row fx-gap-1'},
-        div(
-            {class: `flex-row fx-justify-center anomaly-tag ${value > 0 ? 'has-anomalies' : hasErrors ? 'has-errors' : isTraining ? 'is-training' : isPending ? 'is-pending' : ''}`},
-            value > 0
-                ? value
-                : hasErrors
-                    ? withTooltip(
-                        i({class: 'material-symbols-rounded'}, 'warning'),
-                        {text: 'Execution error', position: 'top-right'},
-                    )
-                    : isTraining
+    const SummaryTag = (typeKey, tagLabel, value, hasErrors, isTraining, isPending) => {
+        const isClickable = !!options.onTagClick;
+        const isActive = van.derive(() => (getValue(options.activeTypes) ?? []).includes(typeKey));
+
+        return div(
+            {
+                class: () => `flex-row fx-gap-1 p-1 border-radius-1 summary-tag ${isClickable ? 'clickable' : ''} ${isActive.val ? 'active' : ''}`,
+                onclick: isClickable ? (event) => {
+                    event.stopPropagation();
+                    options.onTagClick(typeKey);
+                } : undefined,
+            },
+            div(
+                {class: `flex-row fx-justify-center anomaly-tag ${value > 0 ? 'has-anomalies' : hasErrors ? 'has-errors' : isTraining ? 'is-training' : isPending ? 'is-pending' : ''}`},
+                value > 0
+                    ? value
+                    : hasErrors
                         ? withTooltip(
-                            i({class: 'material-symbols-rounded'}, 'more_horiz'),
-                            {text: 'Training model', position: 'top-right'},
+                            i({class: 'material-symbols-rounded'}, 'warning'),
+                            {text: 'Execution error', position: 'top-right'},
                         )
-                        : isPending
+                        : isTraining
                             ? withTooltip(
-                                span({class: 'pl-2 pr-2', style: 'position: relative;'}, '-'),
-                                {text: 'No results yet or not configured'},
+                                i({class: 'material-symbols-rounded'}, 'more_horiz'),
+                                {text: 'Training model', position: 'top-right'},
                             )
-                            : i({class: 'material-symbols-rounded'}, 'check'),
-        ),
-        span({}, label),
-    );
+                            : isPending
+                                ? withTooltip(
+                                    span({class: 'pl-2 pr-2', style: 'position: relative;'}, '-'),
+                                    {text: 'No results yet or not configured'},
+                                )
+                                : i({class: 'material-symbols-rounded'}, 'check'),
+            ),
+            span({}, tagLabel),
+        );
+    };
 
     const numRuns = summary.lookback === 1 ? 'run' : `${summary.lookback} runs`;
     const duration = humanReadableDuration(formatDuration(summary.lookback_start, new Date()), true)
     const labelElement = span({class: 'text-small text-secondary'}, `${label} in last ${numRuns} (${duration})`);
 
     const contentElement = div(
         {class: 'flex-row fx-gap-5'},
-        SummaryTag('Freshness', summary.freshness_anomalies, summary.freshness_has_errors, summary.freshness_is_training, summary.freshness_is_pending),
-        SummaryTag('Volume', summary.volume_anomalies, summary.volume_has_errors, summary.volume_is_training, summary.volume_is_pending),
-        SummaryTag('Schema', summary.schema_anomalies, summary.schema_has_errors, false, summary.schema_is_pending),
-        SummaryTag('Metrics', summary.metric_anomalies, summary.metric_has_errors, summary.metric_is_training, summary.metric_is_pending),
+        SummaryTag('freshness', 'Freshness', summary.freshness_anomalies, summary.freshness_has_errors, summary.freshness_is_training, summary.freshness_is_pending),
+        SummaryTag('volume', 'Volume', summary.volume_anomalies, summary.volume_has_errors, summary.volume_is_training, summary.volume_is_pending),
+        SummaryTag('schema', 'Schema', summary.schema_anomalies, summary.schema_has_errors, false, summary.schema_is_pending),
+        SummaryTag('metrics', 'Metrics', summary.metric_anomalies, summary.metric_has_errors, summary.metric_is_training, summary.metric_is_pending),
     );
 
     if (summary.project_code && summary.table_group_id) {
@@ -96,4 +115,12 @@ const AnomaliesSummary = (summary, label = 'Anomalies') => {
     return div({class: 'flex-column fx-gap-2'}, labelElement, contentElement);
 };
 
+const summaryStylesheet = new CSSStyleSheet();
+summaryStylesheet.replace(`
+.summary-tag.clickable:hover,
+.summary-tag.active {
+    background: var(--select-hover-background);
+}
+`);
+
 export { AnomaliesSummary };

testgen/ui/components/frontend/js/components/monitor_settings_form.js

@@ -286,9 +286,9 @@ const PredictionForm = (
                 name: 'predict_sensitivity',
                 label: 'Sensitivity',
                 options: [
-                    { label: 'Low', value: 'low', help: 'Fewer alerts. Flag values outside 2 standard deviations of predicted value.' },
-                    { label: 'Medium', value: 'medium', help: 'Balanced. Flag values outside 1.5 standard deviations of predicted value.' },
-                    { label: 'High', value: 'high', help: 'More alerts. Flag values outside 1 standard deviation of predicted value.' },
+                    { label: 'Low', value: 'low', help: 'Fewer alerts. Flag values outside 3 standard deviations of predicted value.' },
+                    { label: 'Medium', value: 'medium', help: 'Balanced. Flag values outside 2.5 standard deviations of predicted value.' },
+                    { label: 'High', value: 'high', help: 'More alerts. Flag values outside 2 standard deviations of predicted value.' },
                 ],
                 value: predictSensitivity,
                 onChange: (value) => predictSensitivity.val = value,