Skip to content

Commit 99871b3

Browse files
committed
chore(clickhouse): bring the AI_EVENTS satellite under declarative HCL
Model the AI_EVENTS role in posthog/clickhouse/hcl/ (the third managed role after OPS and LOGS) and wire it into both the offline guard and the live gate. The ai_events column set is identical across every table. Factor it into two abstract bases in roles/ai_events/shared and extend them: - _ai_events_columns (columns only) -> the distributed ai_events reader (local); - _ai_events_data (extends _ai_events_columns, adds order_by/partition/ttl + indexes) -> the data tables sharded_ai_events (local) and ai_events (US/EU), which then only set their own engine (zoo_path differs). Env layering follows the actual topology: - local/hobby run the MSK ingest variant (kafka_ai_events_json + ai_events_json_mv) with sharded_ai_events + a distributed ai_events reader; - US/EU run WarpStream (kafka_ai_events_json_ws + ai_events_json_ws_mv, MSK dropped by migration 0248) writing into a single ai_events data table (US and EU identical; the earlier per-EU sharded_ai_events has been dropped in prod); - person / person_distinct_id2 are env-uniform Distributed shims (0240), also in roles/ai_events/shared; dev has only the top-level shared objects. - nodes: local/dev/prod-us/prod-eu ai_events lines. - dump-live.sh / check-live.sh: add ai_events (:9100) so the convergence gate introspects and diffs it too. - check.sh: skip validation of the person / cross-cluster Distributed proxies. Goldens are generated from the layers (abstract bases resolve away, so the flat schema is unchanged). Verified against a live multinode ai_events node: dump-live + check-live report no drift for local/ai_events; prod goldens are authored from the captured prod dumps.
1 parent ceafe76 commit 99871b3

15 files changed

Lines changed: 7520 additions & 6 deletions

posthog/clickhouse/hcl/check-live.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ ENV="${VERIFY_LIVE_ENV:-local}"
2626
WARN="${VERIFY_LIVE_WARN:-0}"
2727
DUMPDIR="${1:-${LIVE_DUMP_DIR:?dump dir required (pass as arg1 or set LIVE_DUMP_DIR); run dump-live.sh first}}"
2828

29-
ROLES=(ops logs)
29+
ROLES=(ops logs ai_events)
3030

3131
# Object-name globs the gate ignores, parsed from exclude.hcl (the quoted glob
3232
# strings) — the same list dump-live.sh feeds hclexp -exclude, applied here to
@@ -86,5 +86,5 @@ if [ "$rc" -ne 0 ] && [ "$WARN" = "1" ]; then
8686
echo "check-live: drift detected (warn mode — not failing). Reconcile before enforcing."
8787
exit 0
8888
fi
89-
[ "$rc" -eq 0 ] && echo "check-live: OPS/LOGS live schema matches the HCL golden"
89+
[ "$rc" -eq 0 ] && echo "check-live: live schema matches the HCL golden for all managed roles"
9090
exit $rc

posthog/clickhouse/hcl/check.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ MANIFEST="$HCL/nodes"
2626
# - events_main / events_recent are Distributed proxies to the main events cluster
2727
# - query_log_archive / writable_query_log_archive are Distributed over the OPS-only
2828
# sharded_query_log_archive, which is absent from non-OPS (shared-only) compositions
29-
SKIP='custom_metrics,custom_metrics_backups,custom_metrics_dictionaries,custom_metrics_part_counts,custom_metrics_replication_queue,custom_metrics_server_crash,custom_metrics_table_sizes,custom_metrics_test,ops_query_log_archive_mv,events_main,events_recent,query_log_archive,writable_query_log_archive'
29+
SKIP='custom_metrics,custom_metrics_backups,custom_metrics_dictionaries,custom_metrics_part_counts,custom_metrics_replication_queue,custom_metrics_server_crash,custom_metrics_table_sizes,custom_metrics_test,ops_query_log_archive_mv,events_main,events_recent,query_log_archive,writable_query_log_archive,person,person_distinct_id2'
3030

3131
rc=0
3232
while read -r env role layers; do

posthog/clickhouse/hcl/dump-live.sh

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,9 @@ chmod 0755 "$OUTDIR" 2>/dev/null || true
3535
# Match the published ports in docker-compose.multinode-clickhouse.yml.
3636
# role default-host default-port default-db
3737
ROLES=(
38-
"ops localhost 9300 posthog"
39-
"logs localhost 9500 posthog"
38+
"ops localhost 9300 posthog"
39+
"logs localhost 9500 posthog"
40+
"ai_events localhost 9100 posthog"
4041
)
4142

4243
# Pin to the same chschema build as bin/hclexp; override via repo variable.

0 commit comments

Comments
 (0)