Update the list of supported ClickHouse versions to the latest and relevant metrics (DataDog#21294)

* added system_metrics generator and generated queries

* updated default hatch env matrix, added keeper repo

* update docker compose files, switched from zookeeper to clickhouse-keeper

* updated base conftest for new docker compose

* added generator for system events

* updated generator, added tests template

* added generated data

* updated metadata.csv

* updated tests

* added async metrics parsing

* added support of plain and json logs by default

* fixed linters

* exported new dashboard

* added 25.8 and removed 25.5 versions metrics

* fixed max desc length for metadata file

* added metrics based on system.errors

* added metrics stat counter

* added system errors check to the agent and fixed e2e

* explicit query match mapping

* added explicit agent check before the next assert

* switch system.errors metrics type to monotonic_gauge and extended metadata.csv descriptions with that info

* added changelog entry

* updated conf.yaml.example file

* increased conditions waiting time on the env startup

* split compose files for mounted logs testing and the regular run

* fixed metrics_metadata

* fixed UNKNOWN_IDENTIFIER metrics name

* updated conf.example with new ddev

* fixed license-headers

* small fixes on dashboard panels

* update tests with brand new CH client

* linter fix in conftest

* removed unfinished test

* template typo fixes + renamed metrics prefixes and the package name

* removed 25.6 and 25.7 versions

* simplified docker setup to 1 CH and 1 Keeper nodes

* updated docker setup for legacy CH versions

* renamed hatch generator script

* rolled back old metrics + added version specific ones

* fix format

* CI sync

* added changelog file

* fixed license-headers validator

* renamed changelog file

* renamed metrics on the dashboard

* fixed some leftovers after merge

* updated flush_interval_milliseconds for newer CH versions for DBM integration tests

* fixed DBM tests on supported CH versions

* fixed formatter

* updated license-headers

* returned previous manifest metric-check name

* removed all quantiles

* removed `ddsource` assertion from the tests

* return ddsource back to fqt_events as it was before

* added cache for query manager

* added TLS setting after merge conflicts

* fixed dbm_integration tests after the merge

* added 23.2 and 23.8 to the legacy versions

* onboarded new config settings

* fixed LogTrace event metric

* introduced generic errors.raised metric for all the errors

* set version gating for system errors to 21.3 to have remote column

* added test for system errors query, fixed dbm query check

* fixed linter

Author: ekopanev

.github/workflows/test-all.yml

@@ -736,6 +736,7 @@ jobs:
         - py3.13-23.8
         - py3.13-24.8
         - py3.13-25.3
+        - py3.13-25.8
       fail-fast: false
   j4131274:
     uses: ./.github/workflows/test-target.yml

clickhouse/README.md

@@ -106,7 +106,7 @@ See [service_checks.json][9] for a list of service checks provided by this integ
 Need help? Contact [Datadog support][10].
 
 
-[1]: https://clickhouse.yandex
+[1]: https://clickhouse.com/
 [2]: https://docs.datadoghq.com/agent/kubernetes/integrations/
 [3]: /account/settings/agent/latest
 [4]: https://github.com/DataDog/integrations-core/blob/master/clickhouse/datadog_checks/clickhouse/data/conf.yaml.example

clickhouse/assets/configuration/spec.yaml

@@ -120,6 +120,16 @@ files:
       value:
         type: boolean
         example: false
+    - name: use_legacy_queries
+      description: Enable the legacy query set for collecting metrics. Set to `false` to disable.
+      value:
+        type: boolean
+        example: true
+    - name: use_advanced_queries
+      description: Enable the advanced query set for collecting metrics. Set to `false` to disable.
+      value:
+        type: boolean
+        example: true
     - name: query_samples
       description: Configure collection of database samples snapshots from system.processes
       options:
@@ -405,16 +415,42 @@ files:
   - template: logs
     example:
     - source: clickhouse
+      service: clickhouse
+      type: file
+      path:  /var/log/clickhouse-server/clickhouse-server.log
+      tags:
+        - format:plain
+      log_processing_rules:
+        - type: multi_line
+          name: new_log_start_with_date
+          pattern: '\d{4}\.\d{2}\.\d{2}'
+    - source: clickhouse
+      service: clickhouse
+      type: file
+      path: /var/log/clickhouse-server/clickhouse-server.err.log
+      tags:
+        - format:plain
+      log_processing_rules:
+        - type: multi_line
+          name: new_log_start_with_date
+          pattern: '\d{4}\.\d{2}\.\d{2}'
+    - source: clickhouse
+      service: clickhouse
       type: file
       path:  /var/log/clickhouse-server/clickhouse-server.log
+      tags:
+        - format:json
       log_processing_rules:
         - type: multi_line
-          name: log_start_with_date
-          pattern: \d{4}\.\d{2}\.\d{2}
+          name: new_log_start_with_json
+          pattern: '\{'
     - source: clickhouse
+      service: clickhouse
       type: file
       path: /var/log/clickhouse-server/clickhouse-server.err.log
+      tags:
+        - format:json
       log_processing_rules:
         - type: multi_line
-          name: log_start_with_date
-          pattern: \d{4}\.\d{2}\.\d{2}
+          name: new_log_start_with_json
+          pattern: '\{'