Re-enable ClickHouse in CLI tests (#2095)

* Re-enable ClickHouse in CLI tests

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Fix prettier formatting for warehouse-type array

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Add clickhouse to Seed e2e dbt project step

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Fix get_elementary_database_and_schema to use actual elementary schema

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Revert "Fix get_elementary_database_and_schema to use actual elementary schema"

This reverts commit 1fc1952.

* Fix ClickHouse column alias handling in populate_test_alerts macro

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Use .get() for all field accesses in populate_test_alerts to handle ClickHouse column names

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Revert "Use .get() for all field accesses in populate_test_alerts to handle ClickHouse column names"

This reverts commit 5a7efb8.

* Revert "Fix ClickHouse column alias handling in populate_test_alerts macro"

This reverts commit a3bef9c.

* Add debug logging to identify Undefined field in populate_test_alerts

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Update debug logging to use 'is defined' checks for accurate Undefined detection

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Add more detailed debug logging to find nested Undefined values

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Fix ClickHouse column alias handling with .get() fallbacks

ClickHouse returns columns with table prefix (e.g., 'failed_tests.database_name'
instead of 'database_name'). Use .get() with fallbacks to handle both naming
conventions for the affected fields:
- database_name
- schema_name
- tags
- test_params
- severity
- status
- result_rows

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Fix ClickHouse column name ambiguity with explicit SQL aliases

ClickHouse returns columns with table prefix when column names are ambiguous
across joined tables. The affected columns (database_name, schema_name, tags,
test_params, severity, status, result_rows) exist in both failed_tests and
tests tables.

Fix: Add explicit 'as column_name' aliases in the SQL query to force
ClickHouse to use the alias name instead of the qualified column name.
This is cleaner than using .get() fallbacks in the macro.

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* Remove DBT_EDR_DEBUG env var (debugging complete)

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Itamar Hartstein <haritamar@gmail.com>

Author: devin-ai-integration[bot]

.github/workflows/test-all-warehouses.yml

@@ -33,7 +33,15 @@ jobs:
       matrix:
         dbt-version: ${{ inputs.dbt-version && fromJSON(format('["{0}"]', inputs.dbt-version)) || fromJSON('[null]') }}
         warehouse-type:
-          [postgres, snowflake, bigquery, redshift, databricks_catalog, athena]
+          [
+            postgres,
+            snowflake,
+            bigquery,
+            redshift,
+            databricks_catalog,
+            athena,
+            clickhouse,
+          ]
     uses: ./.github/workflows/test-warehouse.yml
     with:
       warehouse-type: ${{ matrix.warehouse-type }}

.github/workflows/test-warehouse.yml

@@ -14,6 +14,7 @@ on:
           - databricks_catalog
           - spark
           - athena
+          - clickhouse
       elementary-ref:
         type: string
         required: false
@@ -103,10 +104,10 @@ jobs:
         working-directory: ${{ env.E2E_DBT_PROJECT_DIR }}
         run: docker compose up -d postgres
 
-      # - name: Start Clickhouse
-      #   if: inputs.warehouse-type == 'clickhouse'
-      #   working-directory: ${{ env.E2E_DBT_PROJECT_DIR }}
-      #   run: docker compose up -d clickhouse
+      - name: Start Clickhouse
+        if: inputs.warehouse-type == 'clickhouse'
+        working-directory: ${{ env.E2E_DBT_PROJECT_DIR }}
+        run: docker compose up -d clickhouse
 
       - name: Setup Python
         uses: actions/setup-python@v5
@@ -161,7 +162,7 @@ jobs:
 
       - name: Seed e2e dbt project
         working-directory: ${{ env.E2E_DBT_PROJECT_DIR }}
-        if: inputs.warehouse-type == 'postgres' || inputs.generate-data
+        if: inputs.warehouse-type == 'postgres' || inputs.warehouse-type == 'clickhouse' || inputs.generate-data
         run: |
           python generate_data.py
           dbt seed -f --target "${{ inputs.warehouse-type }}"

elementary/monitor/dbt_project/macros/alerts/population/test_alerts.sql

@@ -4,6 +4,7 @@
     {% set test_alerts = [] %}
     {% set raw_test_alerts_agate = run_query(elementary_cli.populate_test_alerts_query(days_back)) %}
     {% set raw_test_alerts = elementary.agate_to_dicts(raw_test_alerts_agate) %}
+    
     {% for raw_test_alert in raw_test_alerts %}
         {% set test_type = raw_test_alert.alert_type %}
         {% set status = raw_test_alert.status | lower %}
@@ -138,25 +139,26 @@
         failed_tests.test_execution_id,
         failed_tests.test_unique_id,
         failed_tests.model_unique_id,
-        failed_tests.database_name,
+        {# Explicit aliases for columns that exist in both failed_tests and tests tables to avoid ClickHouse column name ambiguity #}
+        failed_tests.database_name as database_name,
         failed_tests.detected_at,
         {{ elementary.edr_current_timestamp() }} as created_at,
-        failed_tests.schema_name,
+        failed_tests.schema_name as schema_name,
         failed_tests.table_name,
         failed_tests.column_name,
         failed_tests.test_type as alert_type,
         failed_tests.sub_type,
         failed_tests.test_results_description as alert_description,
         failed_tests.owners,
-        failed_tests.tags,
+        failed_tests.tags as tags,
         failed_tests.test_results_query as alert_results_query,
         failed_tests.other,
         failed_tests.test_name,
         failed_tests.test_short_name,
-        failed_tests.test_params,
-        failed_tests.severity,
-        failed_tests.status,
-        failed_tests.result_rows,
+        failed_tests.test_params as test_params,
+        failed_tests.severity as severity,
+        failed_tests.status as status,
+        failed_tests.result_rows as result_rows,
         tests.meta as test_meta,
         tests.description as test_description,
         artifacts_meta.meta as model_meta,