feat: ClickHouse sink for query history, aggregated metrics and session snapshots

[Sanikadze]

Summary

Adds an optional ClickHouse sink to the master role that streams query history, aggregated metrics and session snapshots into a ClickHouse cluster for long-term, ad-hoc SQL analysis. Complements the existing JSON archiver (which keeps working unchanged) and the Prometheus/VictoriaMetrics path.

The sink is opt-in via clickhouse.enabled: true in yagpcc.yaml. When disabled the orchestrator is a no-op and yagpcc behaves exactly as before.

Motivation

The JSON archiver and Prometheus metrics cover the live-monitoring and short-term audit use-cases, but neither lets operators run ad-hoc SQL queries against historical query data (top-N slowest, by user/database/resgroup, plan diffs over time, etc.). ClickHouse is the natural fit for this workload — columnar storage, fast aggregation, TTL-based retention.

What it does

The sink persists three first-class tables (plus a bookkeeping table) under the yagpcc database in ClickHouse:

Table	Purpose
yagpcc.query_events	One row per query status change (SUBMIT/START/DONE/QUERY_DONE/ERROR/CANCELLING/CANCELED/END). Holds the full plan tree as JSON, per-segment metrics as Array(Tuple(...)) and aggregated SystemStat/Instrumentation totals.
yagpcc.aggregated_metrics	SummingMergeTree rollups keyed by (user, database, query_id, plan_id, bucket_time); resource_group is carried as a non-key column. Mirrors the in-memory AggregatedStorage flush.
yagpcc.session_snapshots	Periodic snapshot of pg_stat_activity (default every 10s).
yagpcc._yagpcc_meta	Schema-version ledger. Used by both auto-migrate and verify-schema modes.

The DDL lives in internal/sink/clickhouse/migrations/0001_init.up.sql and is embedded into the binary via //go:embed. The retention TTL is a text/template placeholder rendered from clickhouse.retention_days at startup.

Architecture

internal/sink/clickhouse/
├── client.go         NewClient + TLS config + Ping
├── writer.go         ClickhouseWriter orchestrator (lifecycle, Submit, FlushAggregates)
├── tables.go         QueryEventWriter, AggregatedWriter, SessionSnapshotWriter
├── mapping.go        proto/storage to CH row converters
├── buffer.go         thread-safe ring buffer (drop_oldest | block)
├── migrations.go     ParseMigrations, RenderTemplate, ApplyMigrations
├── schema.go         VerifySchema, DumpSchema, DumpMigration
├── metrics.go        Prometheus collectors + per-table hook factories
├── migrations/       0001_init.{up,down}.sql (embedded)
└── testdata/         fixtures for unit tests

Boundaries are designed to keep the sink isolated from yagpcc's hot-path storage:

• All proto/storage → ClickHouse row conversion logic is concentrated in mapping.go.
• The orchestrator exposes a small ClickhouseSink interface to internal/master/background.go (Submit + FlushAggregates) so the master can swap a fake in tests.
• Per-table writers depend on a batchPreparer interface (a subset of driver.Conn) so unit tests use hand-rolled fakes instead of standing up a real server.

Data flow

yagp-hooks-collector --UDS--> yagpcc segments --gRPC pull--> master/background.go
                                                                  |
                              +-----------------------------------+
                              v                                   v
               ArchiveOrAggregate (existing)            AggregatedStorage cycle hook
                              |                                   |
                              v                                   v
                  ClickhouseWriter.Submit              ClickhouseWriter.FlushAggregates
                              |                                   |
                              v                                   v
              QueryEventWriter.Write              AggregatedWriter.FlushBuckets
              (filter + Buffer.Append)                            |
                              |                                   v
                              v                       INSERT yagpcc.aggregated_metrics
                   periodic Flush(ctx)
                              |
                              v
                INSERT yagpcc.query_events             SessionSnapshotWriter (own ticker)
                                                                  |
                                                                  v
                                                INSERT yagpcc.session_snapshots

archiver.go and statwriter.go keep working unchanged; the JSON archive becomes a manual disaster-recovery source rather than the only persistence path. Both can be enabled at the same time, only one, or neither.

Configuration

Minimal master config:

role: master
# ... existing options ...

clickhouse:
  enabled: true
  addrs:
    - "ch01.example.com:9440"
  database: yagpcc
  user: yagpcc_writer
  # password is read from env YAGPCC_CH_PASSWORD (never put it in yaml)
  schema_management: auto       # auto | verify_only | dump_only
  retention_days: 30
  batch_size: 10000
  flush_interval: 10s
  buffer_max_rows: 100000
  on_buffer_overflow: drop_oldest  # drop_oldest | block
  async_insert: true
  min_duration_ms: 60000        # filter short queries (default: 100)
  session_snapshot_interval_sec: 10
  sinks:                        # per-table toggles, all enabled by default
    query_events: true
    aggregated_metrics: true
    session_snapshots: true
  tls:
    enabled: true
    ca_file: /etc/yagpcc/certs/ch_ca.crt

When enabled: false, the sink is constructed as a no-op and adds no overhead.

Note: the config also accepts sinks.plan_nodes (default false) — it is reserved for a future per-plan-node table and is a no-op in this PR.

CLI flags

• --dump-schema — print cumulative CH DDL to stdout and exit (no connection required)
• --dump-migration --from=N --to=M — print SQL to migrate between schema versions
• --migrate-only — connect, apply pending migrations, exit
• --verify-schema — connect, verify schema version matches binary expectations, exit

Schema management

Mode	Behaviour
auto (default)	Apply pending migrations on startup. Idempotent.
verify_only	Check _yagpcc_meta version, fatal on mismatch.
dump_only	Print DDL to stdout, exit. For CI integration.

The bookkeeping table yagpcc._yagpcc_meta tracks version, applied_at, direction='up'|'down' for each migration.

Observability

New Prometheus metrics:

• yagpcc_ch_inserts_total{table, status} — counter of INSERT batches
• yagpcc_ch_dropped_rows_total{table, reason} — counter of dropped rows (buffer_full, filter, mapping_error, insert_error)
• yagpcc_ch_buffer_size{table} — gauge of current buffer occupancy
• yagpcc_ch_batch_duration_seconds{table} — histogram of batch flush latency
• yagpcc_ch_schema_mismatch — gauge, 1 when the deployed schema version disagrees with the binary
• yagpcc_ch_unreachable — gauge, 1 while ClickHouse is unreachable (suitable for alerting)

Backpressure

The ring buffer offers two policies:

• drop_oldest (default) — drop oldest rows when buffer is full, increment yagpcc_ch_dropped_rows_total{reason="buffer_full"}. Producer never blocks.
• block — block the producer until the writer drains. NOT recommended in production: a stuck CH can stall the master.

Graceful shutdown

ctx.Done() triggers a final flush guarded by a 30s timeout so a stuck CH cannot block master shutdown. The flusher goroutine and the per-table writers are drained in order, then the connection is closed.

Testing

• Unit tests — every component has a corresponding *_test.go with hand-rolled fakes for batchPreparer / driver.Conn. Coverage in internal/sink/clickhouse/ is high (writer, tables, migrations, schema, buffer, mapping, client, metrics all have dedicated suites).
• Integration test — internal/sink/clickhouse/integration_test.go uses testcontainers-go to spin up a real ClickHouse container, apply migrations, write rows, verify reads. Gated by -tags=integration to keep go test ./... fast; CI runs it as a dedicated clickhouse-integration job.
• End-to-end test plan — see Test plan section below.

Files changed

Category	Files
New package internal/sink/clickhouse/	8 .go + 10 _test.go + 2 .sql migrations
CLI / entrypoint	cmd/server/schema_cli.go + test, cmd/server/main.go (schema-CLI short-circuit + dump_only startup mode)
Config	internal/config/clickhouse.go + test, internal/config/config.go (validation), cmd/server/yagpcc_master.yaml (example)
Wiring	internal/master/background.go (CH sink interface, Submit/Flush), internal/master/archiver.go (submit on archive), internal/storage/aggregated_storage.go (cycle hook)
App lifecycle	internal/app/app.go (master-only CH writer construction)
CI	.github/workflows/test.yaml (new clickhouse-integration job)
Docs	docs/clickhouse-sink-architecture.md, docs/vm-vs-ch.md, README, architecture(+ru), PROJECT_CONTEXT.md
RAT exclude	.rat-excludes (embedded SQL migrations)

46 files changed, ~10200 insertions.

Backwards compatibility

• Default clickhouse.enabled: false keeps existing deployments unchanged.
• JSON archiver path is untouched; both paths can coexist.
• No new required config fields; all CH options have sensible defaults.
• No proto changes; gRPC API is unaffected.

Test plan

• make build succeeds
• make unittest passes (excluding integration tag)
• make lint passes (golangci-lint)
• make package produces a deb that installs cleanly
• Apache RAT audit workflow passes (license headers + .rat-excludes)
• Deploy on dev stand with clickhouse.enabled: false — yagpcc behaves identically to current main
• Deploy on dev stand with clickhouse.enabled: true, run load — yagpcc.query_events rows appear, yagpcc_ch_inserts_total{table="query_events"} grows
• Verify graceful shutdown — systemctl stop yagpcc flushes buffers within 30s, no WARN clickhouse sink close error in logs
• Verify --dump-schema outputs valid CH DDL
• Verify --migrate-only applies migrations from a clean DB and exits with code 0
• Verify --verify-schema exits with code 0 when versions match, code 1 when they don't

Known dependencies

• github.com/ClickHouse/clickhouse-go/v2 (direct, native protocol client)
• github.com/testcontainers/testcontainers-go (test-only)
• Transitive: opentelemetry, segmentio/asm, shopspring/decimal — pulled in by clickhouse-go

Related

• docs/clickhouse-sink-architecture.md — full architecture (this PR adds it)
• docs/vm-vs-ch.md — when to use VictoriaMetrics vs ClickHouse (this PR adds it)

[leborchuk]

Need to be rewritten as was suggested in f74a5f5