feat(31): shared invariants module in quickwit-dst#6246
Merged
Conversation
4 tasks
fulmicoton-dd
approved these changes
Mar 31, 2026
g-talbot
force-pushed
the
gtt/phase-31-pg-metastore
branch
from
c8836d6 to
eccf657
Compare
g-talbot
force-pushed
the
gtt/phase-31-execute
branch
2 times, most recently
from
3eb3316 to
c4418dd
Compare
g-talbot
force-pushed
the
gtt/phase-31-pg-metastore
branch
2 times, most recently
from
6eeaecd to
f2113e5
Compare
g-talbot
force-pushed
the
gtt/phase-31-execute
branch
2 times, most recently
from
aa5ff45 to
142ddd6
Compare
g-talbot
force-pushed
the
gtt/phase-31-pg-metastore
branch
from
f2113e5 to
0d561b4
Compare
g-talbot
force-pushed
the
gtt/phase-31-execute
branch
from
142ddd6 to
972bd20
Compare
g-talbot
force-pushed
the
gtt/phase-31-pg-metastore
branch
from
0d561b4 to
8ba9201
Compare
g-talbot
force-pushed
the
gtt/phase-31-execute
branch
from
972bd20 to
73c90c4
Compare
g-talbot
force-pushed
the
gtt/phase-31-pg-metastore
branch
from
8ba9201 to
0761d11
Compare
g-talbot
force-pushed
the
gtt/phase-31-execute
branch
from
73c90c4 to
44bd6bb
Compare
g-talbot
force-pushed
the
gtt/phase-31-pg-metastore
branch
from
0761d11 to
b551b13
Compare
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…arquet-engine Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Matthew Kim <matthew.kim@datadoghq.com>
Co-authored-by: Matthew Kim <matthew.kim@datadoghq.com>
Resolve merge conflicts by taking main's versions of otel_metrics.rs and arrow_metrics.rs (the PR didn't modify these files — conflicts came from the base branch divergence). Kept PR's table_config module export in quickwit-parquet-engine/src/lib.rs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pre-existing splits were serialized before the parquet_file field was added, so their JSON doesn't contain it. Adding #[serde(default)] makes deserialization fall back to empty string for old splits. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the commit timeout fires and the accumulator contains only zero-column batches, union_fields is empty and concat_batches fails with "must either specify a row count or at least one column". Now flush_internal treats empty union_fields the same as empty pending_batches — resets state and returns None. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Resolve Cargo.lock/Cargo.toml merge conflicts - P1 (sort column lookup): Already addressed by sort fields tag_ prefix fix — sort field names now match Parquet column names - P2 (window_start at epoch 0): Remove time_range.start_secs > 0 guard so window_start is computed for all batches when window_duration > 0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Resolve writer.rs conflict: keep META-07 self-describing roundtrip test - P1 (create_timestamp serde): Add #[serde(default)] to StoredMetricsSplit.create_timestamp for backward-compatible reads of pre-existing file-backed index JSON - P1 (compaction window overlap): No change needed — Bound::Included vs Bound::Excluded already handles half-open interval semantics correctly, and the edge case (zero duration) is impossible - fields.rs: No change — Matt noted it resolves with wide schema rebase Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Resolve postgres.rs conflict: keep check_invariant! macros, add window_duration_secs consistency check - Group setup_dogstatsd_exporter + setup_invariant_recorder into single setup_metrics() function (fulmicoton-dd review) - Rename `id` to `invariant_id` in invariant_recorder (fulmicoton-dd review) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
g-talbot
added a commit
that referenced
this pull request
Apr 9, 2026
UPSTREAM-CANDIDATES.md incorrectly stated TLA+ specs and Stateright models don't exist. They do (contributed in #6246): ParquetDataModel.tla, SortSchema.tla, TimeWindowedCompaction.tla, plus quickwit-dst invariants and Stateright model tests. Updated to accurately reflect that the remaining aspirational piece is the simulation infrastructure (SimClock, FaultInjector, etc.). Also removed the /sesh-mode aspirational entry — it's actively being used and the underlying specs/models are real. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
invariants/module toquickwit-dst— the single source of truth for invariant definitions shared across the verification pyramid (TLA+ specs, stateright models, production code)InvariantIdenum catalogs all 20 invariants (SS-1..5, TW-1..3, CS-1..3, MC-1..4, DM-1..5) with Display and descriptionswindow_start_secs(),is_valid_window_duration(),compare_with_null_ordering()check_invariant!macro always evaluates the condition (debug and release), implementing both Layer 3 (Prevention —debug_assert!panic in debug) and Layer 4 (Production — pluggableInvariantRecorderfor Datadog metrics)set_invariant_recorder(fn)wires up statsd at process startup; no recorder = no-op (singleOnceLockload)staterightis now an optional dependency behindmodel-checkingfeature — production crates pull only the invariant definitions, not the model checkerquickwit-parquet-engineuses shared functions andcheck_invariant!macro instead of inlinedebug_assert!Stacks on
gtt/phase-31-pg-metastore(PR #6245).Test plan
cargo test -p quickwit-dst— 11 invariant unit tests + 2 doctests passcargo test -p quickwit-dst --features model-checking— all 24 tests pass (stateright models use shared functions)cargo test -p quickwit-parquet-engine— all 153 tests passcargo test -p quickwit-metastore --all-features— all 187 tests pass (including postgres)cargo test -p quickwit-indexing --all-features— 234 pass (8 pre-existing Pulsar/GCP PubSub flakes skipped)cargo test -p quickwit-integration-tests --all-features— all 46 passcargo clippy -p quickwit-dst --all-features --tests— cleancargo clippy -p quickwit-parquet-engine --all-features --tests— clean🤖 Generated with Claude Code