feat: enhance ClickHouse Cloud integration with multi-service support and detailed configuration options

Author: pythonicrahul

CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2026-04-03
+
+### Added
+- Custom Datadog Agent check for ClickHouse Cloud log collection
+- Query log collection from `system.query_log` (completed queries and exceptions)
+- Text log collection from `system.text_log` (Error, Warning, Fatal levels)
+- OpenMetrics configuration for ClickHouse Cloud Prometheus endpoint
+- Cursor-based pagination with duplicate-delivery-over-loss semantics
+- Configurable batch size, slow-query threshold, backfill window, and query timeout
+- Input validation with bounds checking on all numeric config parameters
+- HTTP retry logic (2 retries with backoff on 502/503/504)
+- Configurable `cluster_name` for the Datadog `service` field
+- `ddsource: clickhouse` on all log entries for Datadog pipeline compatibility
+- Comprehensive test suite (53 tests)
+- CI workflow with ruff lint and pytest across Python 3.10/3.11/3.12
+- Dependabot configuration for pip and GitHub Actions

README.md

@@ -66,6 +66,7 @@ instances:
   - service_id: "<your-service-uuid>"
     key_id: "<your-api-key-id>"
     key_secret: "<your-api-key-secret>"
+    cluster_name: "<your-cluster-name>"     # appears as "service" in Datadog Logs
 
     collect_query_logs: true
     collect_text_logs: true
@@ -86,6 +87,35 @@ All three credential fields are required. Everything else has sensible defaults.
 | `initial_backfill_minutes` | 60 | 1 -- 1440 | How far back to look on first run (avoids flooding Datadog with history) |
 | `query_timeout_seconds` | 30 | 5 -- 300 | HTTP timeout for each ClickHouse Cloud API call |
 
+### Monitoring multiple ClickHouse Cloud services
+
+Add one entry per service under `instances` in the log check config. Each instance runs independently with its own cursor, credentials, and tags:
+
+```yaml
+instances:
+  # Production cluster
+  - service_id: "abc-123-prod"
+    key_id: "<prod-key-id>"
+    key_secret: "<prod-key-secret>"
+    cluster_name: "prod-analytics"
+    tags:
+      - "env:production"
+      - "clickhouse_cluster:prod-analytics"
+
+  # Staging cluster
+  - service_id: "def-456-staging"
+    key_id: "<staging-key-id>"
+    key_secret: "<staging-key-secret>"
+    cluster_name: "staging-analytics"
+    collect_text_logs: false          # only query logs for staging
+    log_batch_size: 500
+    tags:
+      - "env:staging"
+      - "clickhouse_cluster:staging-analytics"
+```
+
+For metrics, add a separate entry per service in the OpenMetrics config as well (each with its own org/endpoint). Use `cluster_name` and tags to distinguish services in Datadog dashboards and log facets.
+
 ### 4. Configure metrics
 
 Edit `/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml`:

checks/clickhouse_cloud.py

@@ -7,6 +7,8 @@
 
 from __future__ import annotations
 
+__version__ = "1.0.0"
+
 import json
 import time
 from collections.abc import Callable

conf.d/clickhouse_cloud.d/conf.yaml.example

@@ -1,30 +1,88 @@
+## All options defined here are available to all instances.
+#
 init_config:
 
 instances:
-  - service_id: "<your-service-uuid>"          # required – ClickHouse Cloud service UUID
-    key_id: "<your-api-key-id>"                 # required – Cloud API key ID
-    key_secret: "<your-api-key-secret>"         # required – Cloud API key secret
 
-    # Cluster / service identity
-    cluster_name: "<your-cluster-name>"         # used as the "service" field in Datadog Logs
-                                                # (default: "clickhouse")
+    ## @param service_id - string - required
+    ## ClickHouse Cloud service UUID.
+    ## Found in the ClickHouse Cloud console under your service's connection details.
+    #
+  - service_id: "<your-service-uuid>"
 
-    # Log collection toggles
-    collect_query_logs: true                    # default: true
-    collect_text_logs: true                     # default: true
+    ## @param key_id - string - required
+    ## ClickHouse Cloud API key ID.
+    ## Create one at https://clickhouse.cloud/settings/api-keys with query access.
+    #
+    key_id: "<your-api-key-id>"
 
-    # Tuning
-    log_batch_size: 1000                        # rows per check run (1–10000)
-    slow_query_threshold_ms: 5000              # marks query logs as warning (0–3600000)
-    initial_backfill_minutes: 60               # lookback window on first run (1–1440)
-    query_timeout_seconds: 30                  # HTTP timeout per query (5–300)
+    ## @param key_secret - string - required
+    ## ClickHouse Cloud API key secret (paired with key_id above).
+    #
+    key_secret: "<your-api-key-secret>"
 
-    # Applied to all logs shipped
-    tags:
-      - "env:prod"
-      - "clickhouse_cluster:<your-cluster-name>"
+    ## @param cluster_name - string - optional - default: clickhouse
+    ## Friendly name for this ClickHouse Cloud service.
+    ## Used as the "service" field on every Datadog log entry, making it easy
+    ## to filter logs by cluster in the Datadog Logs Explorer.
+    #
+    # cluster_name: clickhouse
 
+    ## @param collect_query_logs - boolean - optional - default: true
+    ## Enable collection of query logs from system.query_log.
+    ## Captures completed queries and exceptions with duration, memory, and row stats.
+    #
+    # collect_query_logs: true
+
+    ## @param collect_text_logs - boolean - optional - default: true
+    ## Enable collection of server logs from system.text_log.
+    ## Captures Error, Warning, and Fatal level server-side log entries.
+    #
+    # collect_text_logs: true
+
+    ## @param log_batch_size - integer - optional - default: 1000
+    ## Maximum number of log rows fetched per check run, per table.
+    ## Range: 1–10000. Higher values reduce API calls but increase memory usage.
+    #
+    # log_batch_size: 1000
+
+    ## @param slow_query_threshold_ms - integer - optional - default: 5000
+    ## Queries slower than this threshold (in milliseconds) are tagged with
+    ## status "warning" instead of "info" in Datadog Logs.
+    ## Range: 0–3600000. Set to 0 to disable slow-query highlighting.
+    #
+    # slow_query_threshold_ms: 5000
+
+    ## @param initial_backfill_minutes - integer - optional - default: 60
+    ## How far back (in minutes) to look on the very first check run when
+    ## no cursor exists yet. Subsequent runs resume from the last cursor.
+    ## Range: 1–1440 (up to 24 hours).
+    #
+    # initial_backfill_minutes: 60
+
+    ## @param query_timeout_seconds - integer - optional - default: 30
+    ## HTTP timeout in seconds for each ClickHouse Cloud Query API request.
+    ## Range: 5–300. Increase if you see timeout errors with large batch sizes.
+    #
+    # query_timeout_seconds: 30
+
+    ## @param tags - list of strings - optional
+    ## List of tags to attach to every log entry emitted by this check.
+    ##
+    ## Learn more about tagging: https://docs.datadoghq.com/tagging/
+    #
+    # tags:
+    #   - "env:prod"
+    #   - "clickhouse_cluster:<your-cluster-name>"
+
+## Log section (required for send_log() to work)
+##
+## The Datadog Agent's log launcher needs this block to open a file handle
+## for the integration. Without it, send_log() silently fails with:
+##   "Failed to write log to file, file is nil for integration ID: clickhouse_cloud"
+##
+## Do not remove or comment out this section.
+#
 logs:
   - type: integration
     source: clickhouse
-    

integrations-extras

@@ -0,0 +1 @@
+Subproject commit 683881dc728a7e8a595209c907dec18c0d2aa08a