chore(weave): add replicated CH backends to nightly tests#7000
Draft
gtarpenning wants to merge 10 commits into
Draft
chore(weave): add replicated CH backends to nightly tests#7000gtarpenning wants to merge 10 commits into
gtarpenning wants to merge 10 commits into
Conversation
This was referenced May 28, 2026
Member
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
gtarpenning
force-pushed
the
gtarpenning/wb-34908-files-chunked-read-race
branch
from
1b11270 to
c980d45
Compare
gtarpenning
force-pushed
the
gtarpenning/add-replicated-ch-nightly-stacked
branch
from
751af18 to
83ef92d
Compare
Adds two parallel matrix jobs that exercise the replicated CH path: - nightly-tests-replicated-1s3r (1 shard x 3 replicas, ReplicatedMergeTree) - nightly-tests-replicated-2s2r (2 shards x 2 replicas + Distributed tables) Both reuse the existing topology compose files under tests/trace_server/conftest_lib/topologies/ that test.yaml already drives for migrator coverage, and surface failures via slack-notification. Timeouts bumped to 90/120 min vs the baseline 30 min since replicated writes go through Keeper and Distributed adds an extra query hop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The session-scoped ClickHouse fixture's patched migration runner was constructing the migrator without the replicated/distributed kwargs, so it ran in cloud mode while the application code in the same process read WF_CLICKHOUSE_REPLICATED=true from env. Split-brain produced UNKNOWN_DATABASE on ch-s1-r3 (1s3r) and missing _local-suffix tables (2s2r) once the nightly started exercising replicated topologies. Mirror the production wiring in clickhouse_trace_server_batched.py:6900. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TRUNCATE and DROP DATABASE against a single replica don't propagate; peer replicas keep stale state and the next test sees UNKNOWN_TABLE / UNKNOWN_DATABASE on a different node. Append ON CLUSTER (gated by WF_CLICKHOUSE_REPLICATED) and SYNC so the next test starts clean across the cluster. Also filter Distributed-engine tables out of the truncate list — TRUNCATE on a Distributed wrapper fails; the underlying _local ReplicatedMergeTree carries the data. No-op in cloud mode: _on_cluster_clause() returns "" so single-node SQL is byte-identical. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OPTIMIZE is unsupported on Distributed engines. In 2s2r CI, both inline and `force_merge_calls`-driven OPTIMIZE calls hit the Distributed wrapper and failed with Code: 48 NOT_IMPLEMENTED, cascading into ~64 teardown failures through the autouse logging_error_check fixture. Add a shared `force_optimize_calls_merged(ch_client)` helper that targets `calls_merged_local ON CLUSTER` when distributed, else the base table. Use it from both the inline call site and `force_merge_calls`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ClickHouse's default async Distributed INSERT (buffer locally, forward later) breaks read-after-write in the test suite — writes return before the destination shard sees the row, and immediate SELECTs against the Distributed wrapper get 0 rows. Production users tolerate that latency; the test suite cannot. Server-side fix: mount a users.d override on every 2s2r CH node that flips distributed_foreground_insert=1 on the default profile. Applies to every connection from every test thread; never reaches production deployments. 1s3r has no Distributed tables and is unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ancing=in_order ReplicatedMergeTree replicates between same-shard replicas asynchronously. With distributed_foreground_insert=1, the write to call_parts_local lands synchronously on one replica and the MV's downstream insert into calls_merged_local lands on that same replica. A read against the Distributed wrapper picks a replica per shard via load_balancing; with the default `random`, the next SELECT can hit a replica that hasn't replicated the row yet -> stale read. `load_balancing=in_order` always routes to the first healthy replica per shard, so test reads consistently hit the replica that received the most recent write. Production keeps the default for load spreading; tests trade that for determinism. This file is mounted only by the test compose topologies. Production servers don't pick it up.
Smoke 26585979011 closed out the bulk of distributed-mode failures (174 down to 16) once the annotation_queues / distributed_product_mode fixes landed. The remaining tests hit four distinct prod bugs that each need their own RCA + fix in follow-up PRs: - globalIn rewrite (CH 25.11 + distributed_product_mode=global) - llm_token_prices rand-sharded -> cost JOIN misses - DELETE on Distributed engine (feedback, llm_token_prices purge) - OPTIMIZE TABLE on Distributed engine (storage-size tests) Wires a small `pytest_collection_modifyitems` in trace/conftest.py keyed off `WF_CLICKHOUSE_USE_DISTRIBUTED_TABLES`. Tests pass on cloud and 1s3r unchanged; skipped only under multi-shard distributed. Reasons are recorded in the table so each entry maps to a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gtarpenning
changed the title
gtarpenning
force-pushed
the
gtarpenning/wb-34908-files-chunked-read-race
branch
from
c980d45 to
6a9ed19
Compare
gtarpenning
force-pushed
the
gtarpenning/add-replicated-ch-nightly-stacked
branch
from
83ef92d to
37a50ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WB-34904
Summary
tests/trace_server/conftest_lib/topologies/{1s3r,2s2r}.yaml) with embedded Keeper.pytest_collection_modifyitemsskip hook intests/trace/conftest.pyfor the 4 deferred distributed-mode known failures (globalIn, cost JOIN cross-shard, DELETE on Distributed for feedback, OPTIMIZE on Distributed for storage-size).test_overrides.xml), ON CLUSTER truncate/drop in fixtures,load_balancing=in_order, distributed-aware OPTIMIZE.Testing
smoke 26596965141 - all 24 replicated trace legs (3.10-3.13 x 1s3r/2s2r) pass.