Skip to content

Commit c25f1ba

Browse files
committed
fix(weave): shard call_parts by id so call_start/call_end co-locate
call_parts was using the default rand() sharding key, so call_start and call_end for the same call could land on different shards. Once split, the partial-state rows can never merge in calls_merged_local (OPTIMIZE runs per-shard), and queries that filter on an aggregated column see inconsistent state. Concretely: call_end doesn't carry parent_id, so its row defaults to NULL. Filters like trace_roots_only (`parent_id IS NULL`) then match the call_end row of every child call as if it were a root, inflating counts. Shard by `id` instead of `wf_clickhouse_calls_shard_key()` (which defaults to trace_id): trace_id is Nullable on call_end so sipHash64 returns Nullable, which ClickHouse rejects as a sharding expression (TYPE_MISMATCH). `id` is non-null on every call_part row and uniquely identifies a call, so all parts of one call land together. calls_merged Distributed table is intentionally left rand(): the only writes come from the MV which fires on the local source/target pair and never goes through the Distributed wrapper.
1 parent dbe0478 commit c25f1ba

1 file changed

Lines changed: 7 additions & 2 deletions

File tree

weave/trace_server/clickhouse_trace_server_migrator.py

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,8 +162,13 @@ def _is_transient_ch_error(exc: BaseException) -> bool:
162162
# Valid values: "trace_id" (default), "id", "project_id"
163163
ID_SHARDED_TABLES: dict[str, str] = {
164164
"calls_complete": wf_clickhouse_calls_shard_key(),
165-
# Keep one trace/turn on one shard. This matches the default calls sharding
166-
# key and keeps trace detail reads local.
165+
# call_parts (call_start/call_end rows) must shard by `id`, not `trace_id`:
166+
# call_end rows don't carry trace_id (only call_start does), so sharding by
167+
# trace_id sends them via rand() and they land on a different shard than
168+
# the matching call_start. Without co-location the partial states never
169+
# merge, and queries that filter on aggregated columns (e.g. parent_id IS
170+
# NULL for trace_roots_only) match the call_end row of every child call.
171+
"call_parts": "id",
167172
"spans": "trace_id",
168173
"messages": "trace_id",
169174
# Keep each agent aggregate on one shard. Shard versions by the same key so

0 commit comments

Comments
 (0)