|
| 1 | +# ClickHouse Sink Connector |
| 2 | + |
| 3 | +The ClickHouse sink connector consumes messages from Iggy topics and inserts them into ClickHouse tables. Supports three insert formats: `json_each_row` (default), `row_binary`, and `string` passthrough. |
| 4 | + |
| 5 | +## Features |
| 6 | + |
| 7 | +- **Multiple Insert Formats**: Insert as `JSONEachRow`, `RowBinaryWithDefaults`, or raw string passthrough (CSV/TSV/JSON) |
| 8 | +- **Schema Validation**: In `row_binary` mode, the table schema is fetched and validated at startup |
| 9 | +- **Automatic Retries**: Configurable retry count and delay for transient errors |
| 10 | +- **Batch Processing**: Insert messages in configurable batches via the stream configuration |
| 11 | + |
| 12 | +## Configuration |
| 13 | + |
| 14 | +```toml |
| 15 | +type = "sink" |
| 16 | +key = "clickhouse" |
| 17 | +enabled = true |
| 18 | +version = 0 |
| 19 | +name = "ClickHouse sink" |
| 20 | +path = "target/release/libiggy_connector_clickhouse_sink" |
| 21 | + |
| 22 | +[[streams]] |
| 23 | +stream = "example_stream" |
| 24 | +topics = ["example_topic"] |
| 25 | +schema = "json" |
| 26 | +batch_length = 1000 |
| 27 | +poll_interval = "5ms" |
| 28 | +consumer_group = "clickhouse_sink_connector" |
| 29 | + |
| 30 | +[plugin_config] |
| 31 | +url = "http://localhost:8123" |
| 32 | +database = "default" |
| 33 | +username = "default" |
| 34 | +password = "" |
| 35 | +table = "events" |
| 36 | +insert_format = "json_each_row" |
| 37 | +timeout_seconds = 30 |
| 38 | +max_retries = 3 |
| 39 | +retry_delay = 1 # seconds |
| 40 | +verbose_logging = false |
| 41 | +``` |
| 42 | + |
| 43 | +## Configuration Options |
| 44 | + |
| 45 | +| Option | Type | Default | Description | |
| 46 | +| ------ | ---- | ------- | ----------- | |
| 47 | +| `url` | string | required | ClickHouse HTTP endpoint | |
| 48 | +| `table` | string | required | Target table name | |
| 49 | +| `database` | string | `"default"` | ClickHouse database | |
| 50 | +| `username` | string | `"default"` | ClickHouse username | |
| 51 | +| `password` | string | `""` | ClickHouse password | |
| 52 | +| `insert_format` | string | `"json_each_row"` | Insert format: `json_each_row`, `row_binary`, or `string` | |
| 53 | +| `string_format` | string | `"json_each_row"` | ClickHouse format for `string` mode: `json_each_row`, `csv`, or `tsv` | |
| 54 | +| `timeout_seconds` | u64 | `30` | HTTP request timeout | |
| 55 | +| `max_retries` | u32 | `3` | Max retry attempts on transient errors | |
| 56 | +| `retry_delay` | u64 | `1` | Delay between retries, in seconds | |
| 57 | +| `verbose_logging` | bool | `false` | Log inserts at info level instead of debug | |
| 58 | + |
| 59 | +> **TODO:** `database` and `table` values are interpolated directly into SQL. Currently only |
| 60 | +> single quotes are escaped; backslashes pass through unchanged, which can misparse string |
| 61 | +> literals if a value ends with `\`. A future improvement should validate both fields against a |
| 62 | +> strict allowlist (`^[A-Za-z_][A-Za-z0-9_]*$`) at config load and escape backslashes in SQL |
| 63 | +> string literals. Deferred because these sinks run in operator-controlled environments where |
| 64 | +> config values are trusted. |
| 65 | +
|
| 66 | +## Insert Formats |
| 67 | + |
| 68 | +### `json_each_row` (Default) |
| 69 | + |
| 70 | +Accepts messages with a `Payload::Json` payload. Each message is sent as a JSON object on its own line using ClickHouse's `JSONEachRow` format. ClickHouse handles type coercion from the JSON values to the column types, so the table can have any schema. |
| 71 | + |
| 72 | +```toml |
| 73 | +[plugin_config] |
| 74 | +url = "http://localhost:8123" |
| 75 | +table = "events" |
| 76 | +insert_format = "json_each_row" |
| 77 | +``` |
| 78 | + |
| 79 | +### `row_binary` |
| 80 | + |
| 81 | +Accepts messages with a `Payload::Json` payload. At startup the connector fetches the table schema from `system.columns` and validates that all column types are supported. Messages are then serialised to ClickHouse's `RowBinaryWithDefaults` binary format, which is more efficient than JSON for large volumes. |
| 82 | + |
| 83 | +Requires ClickHouse 23.7 or newer, when `RowBinaryWithDefaults` was introduced. Older servers reject the format; use `json_each_row` instead. |
| 84 | + |
| 85 | +The table must already exist. Columns with an ordinary `DEFAULT` expression can be omitted from the message — the connector emits a `0x01` prefix byte to signal that the default should be used. `MATERIALIZED`, `ALIAS`, and `EPHEMERAL` columns are not insertable and are dropped from the schema entirely. |
| 86 | + |
| 87 | +The schema is captured once at startup and never refreshed. Do not `ALTER TABLE` the target while the connector runs. See [Schema changes while running](#schema-changes-while-running). |
| 88 | + |
| 89 | +**Supported types:** the 8/16/32/64-bit integer and float primitives (`Int8`-`Int64`, `UInt8`-`UInt64`, `Float32`, `Float64`), `String`, `FixedString(n)`, `Bool`/`Boolean`, `UUID`, `Date`, `Date32`, `DateTime`, `DateTime64(p)`, `Decimal` (precision 1-38; `Decimal256` is not supported), `IPv4`, `IPv6`, `Enum8`, `Enum16`, and the composites `Nullable(T)`, `Array(T)`, `Map(K, V)`, `Tuple(...)`. `LowCardinality(T)` is transparently unwrapped to its inner type `T` (RowBinary serialises it identically). |
| 90 | + |
| 91 | +**Unsupported types** (cause startup to fail): the 128/256-bit wide integers (`Int128`, `UInt128`, `Int256`, `UInt256`), `Variant`, `JSON` (native column type), and geo types. |
| 92 | + |
| 93 | +```toml |
| 94 | +[plugin_config] |
| 95 | +url = "http://localhost:8123" |
| 96 | +table = "events" |
| 97 | +insert_format = "row_binary" |
| 98 | +``` |
| 99 | + |
| 100 | +### `string` |
| 101 | + |
| 102 | +Accepts messages with a `Payload::Text` payload and passes them through to ClickHouse without modification. Use `string_format` to tell ClickHouse which format to expect. |
| 103 | + |
| 104 | +```toml |
| 105 | +[plugin_config] |
| 106 | +url = "http://localhost:8123" |
| 107 | +table = "events" |
| 108 | +insert_format = "string" |
| 109 | +string_format = "csv" # or "tsv" or "json_each_row" |
| 110 | +``` |
| 111 | + |
| 112 | +## Example Configs |
| 113 | + |
| 114 | +### JSON Events |
| 115 | + |
| 116 | +```toml |
| 117 | +[[streams]] |
| 118 | +stream = "events" |
| 119 | +topics = ["user_events"] |
| 120 | +schema = "json" |
| 121 | +batch_length = 500 |
| 122 | +poll_interval = "10ms" |
| 123 | +consumer_group = "clickhouse_sink" |
| 124 | + |
| 125 | +[plugin_config] |
| 126 | +url = "http://localhost:8123" |
| 127 | +database = "analytics" |
| 128 | +table = "user_events" |
| 129 | +insert_format = "json_each_row" |
| 130 | +``` |
| 131 | + |
| 132 | +### High-Throughput with RowBinary |
| 133 | + |
| 134 | +```toml |
| 135 | +[[streams]] |
| 136 | +stream = "metrics" |
| 137 | +topics = ["app_metrics"] |
| 138 | +schema = "json" |
| 139 | +batch_length = 5000 |
| 140 | +poll_interval = "5ms" |
| 141 | +consumer_group = "clickhouse_sink" |
| 142 | + |
| 143 | +[plugin_config] |
| 144 | +url = "http://localhost:8123" |
| 145 | +database = "telemetry" |
| 146 | +table = "metrics" |
| 147 | +insert_format = "row_binary" |
| 148 | +max_retries = 5 |
| 149 | +retry_delay = 1 # seconds |
| 150 | +verbose_logging = true |
| 151 | +``` |
| 152 | + |
| 153 | +### CSV Passthrough |
| 154 | + |
| 155 | +```toml |
| 156 | +[[streams]] |
| 157 | +stream = "exports" |
| 158 | +topics = ["csv_data"] |
| 159 | +schema = "text" |
| 160 | +batch_length = 1000 |
| 161 | +poll_interval = "50ms" |
| 162 | +consumer_group = "clickhouse_sink" |
| 163 | + |
| 164 | +[plugin_config] |
| 165 | +url = "http://localhost:8123" |
| 166 | +table = "raw_imports" |
| 167 | +insert_format = "string" |
| 168 | +string_format = "csv" |
| 169 | +``` |
| 170 | + |
| 171 | +## Reliability |
| 172 | + |
| 173 | +The connector retries failed inserts up to `max_retries` times, starting from `retry_delay`. Retryable HTTP errors and network/timeout errors both back off exponentially with full jitter, so instances spread their retries instead of hammering a recovering server in lockstep. Non-retryable errors fail immediately. The startup ping and schema fetch use the same jittered backoff. |
| 174 | + |
| 175 | +On shutdown the connector logs the total number of messages processed. |
| 176 | + |
| 177 | +### Bad rows in a batch |
| 178 | + |
| 179 | +A message whose payload type does not match the chosen format (for example a text payload in JSON mode) is always skipped with a warning. The rest of the batch is still sent. |
| 180 | + |
| 181 | +The `rowbinary` format has one extra case. It turns each row into binary and writes it straight into the batch buffer, so a row that cannot be converted (a value that does not fit the target column) cannot be skipped cleanly — a half-written row would corrupt the rows after it. In that case the **whole batch fails** on the first bad row and is retried as a unit per the rules above. |
| 182 | + |
| 183 | +If a single malformed row keeps failing, every retry of that batch will fail too. Fix or remove the bad message at the source, or switch to the `json` / `string` format, which skip bad rows instead of failing the batch. |
| 184 | + |
| 185 | +### Schema changes while running |
| 186 | + |
| 187 | +In `row_binary` mode the table schema is fetched once at startup and cached for the lifetime of the connector. The insert stream is **positional**: rows are written as a bare `RowBinaryWithDefaults` body in the column order captured at startup, and ClickHouse decodes them against the table's current column order. |
| 188 | + |
| 189 | +An `ALTER TABLE` that runs while the connector is live breaks that assumption. Adding, dropping, or reordering a column shifts the byte layout by one or more columns. Depending on how the shifted bytes decode, ClickHouse either rejects the batch as malformed or, worse, stores it silently with values landing in the wrong columns. Nothing detects this at runtime, so the corruption is easy to miss. |
| 190 | + |
| 191 | +Only `row_binary` is affected. The `json_each_row` and `string` (`json_each_row` string) formats are self-describing and map values by field name, so they tolerate schema changes. |
| 192 | + |
| 193 | +Until the hardening below lands, treat the `row_binary` schema as fixed for the connector's lifetime: **restart the connector after any `ALTER TABLE`** on the target table so it re-fetches the schema. |
| 194 | + |
| 195 | +Two planned fixes remove the restriction: |
| 196 | + |
| 197 | +1. **Explicit column list in the INSERT.** Emitting `INSERT INTO db.table (col1, col2, ...) FORMAT RowBinaryWithDefaults` binds the stream to column *names* instead of table position. ClickHouse then routes each value by name, applies `DEFAULT` for columns the connector omits, and returns a hard error (instead of silently corrupting rows) when a named column has been dropped or renamed. This makes added and reordered columns safe and turns the remaining drift into a visible failure. |
| 198 | +2. **Refresh the schema on a failed insert.** When an insert fails with a data error, re-fetch the schema from `system.columns` and rebuild the column list before the batch is retried, letting the connector recover from a drop or rename on its own rather than failing every retry against a stale snapshot. |
| 199 | + |
| 200 | +### Delivery semantics: at-least-once |
| 201 | + |
| 202 | +This connector provides **at-least-once** delivery — not exactly-once. Retries resend the full batch body without an `insert_deduplication_token`, so if the server applied a batch but the acknowledgement was lost in transit (network drop, timeout), the retry will insert the same rows again. |
| 203 | + |
| 204 | +**Affected table engines:** |
| 205 | + |
| 206 | +- `MergeTree` — no deduplication at all; duplicate rows will be stored. |
| 207 | +- `ReplicatedMergeTree` — has implicit block-level deduplication based on the data checksum (controlled by `replicated_deduplication_window`, default 100 blocks), which will suppress duplicates in the common retry case as long as the window has not been exceeded. |
| 208 | + |
| 209 | +If your workload cannot tolerate duplicate rows, either: |
| 210 | + |
| 211 | +1. Use a `ReplicatedMergeTree` table and keep `max_retries` low enough that retries stay within the deduplication window, or |
| 212 | +2. Use a `CollapsingMergeTree` / `ReplacingMergeTree` and apply deduplication at query time, or |
| 213 | +3. Accept duplicates at write time and deduplicate with `DISTINCT` or `GROUP BY` in your queries. |
| 214 | + |
| 215 | +## Testing |
| 216 | + |
| 217 | +Integration tests against a live ClickHouse container cover only the end-to-end path from Iggy to ClickHouse: messages produced to a stream land as rows in the target table. They are intentionally **not exhaustive** over the wire format. They do not, for example, round-trip a `UUID` column, a `DEFAULT` / `MATERIALIZED` / `ALIAS` column, or the `RowBinaryWithDefaults` `0x01` use-default path against the real server. |
| 218 | + |
| 219 | +Those cases are instead pinned by unit tests that assert exact output bytes for every supported type, including `UUID` word order, the `0x01` use-default prefix byte, and the schema parser dropping `MATERIALIZED` / `ALIAS` / `EPHEMERAL` columns while flagging `DEFAULT` ones. |
| 220 | + |
| 221 | +Byte-exact unit tests plus a minimal e2e path is a deliberate trade. Containerized integration tests are costly to run, and the residual risk they would cover, a real server disagreeing with our model of the wire format, is low: the `RowBinaryWithDefaults` format is stable and unlikely to change under us. |
0 commit comments