Skip to content

Commit 5329a59

Browse files
committed
Merge remote-tracking branch 'origin/main' into matthew.kim/parquet-janitor
2 parents 1d08467 + 1c43154 commit 5329a59

210 files changed

Lines changed: 18343 additions & 3029 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/skills/sesh-mode/SKILL.md

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
---
2+
description: "Verification-first development workflow with TLA+ specs, DST, and formal methods. Use when you want Claude to follow the rigorous plan→spec→test→implement sequence."
3+
user-invocable: true
4+
---
5+
6+
# Sesh Mode — Verification-First Development
7+
8+
Activate this mode when working on features that touch state machines, protocols, or critical data paths. This adds formal verification requirements on top of the base CLAUDE.md.
9+
10+
## Before Writing Any Code
11+
12+
**MUST** follow this sequence before implementation:
13+
14+
1. **Define the plan**: What are you doing and why? What invariants must hold?
15+
2. **Check ADR/roadmap**: `docs/internals/adr/README.md` → find relevant supplement
16+
3. **Read the spec**: If touching state machines or protocols, read `docs/internals/specs/tla/*.tla`
17+
4. **Write tests first**: DST tests define correctness, write them before code
18+
5. **Only then**: Start implementation
19+
20+
## Three Engineering Pillars
21+
22+
Every code change **MUST** respect all three:
23+
24+
| Pillar | Location | Purpose |
25+
|--------|----------|---------|
26+
| **Code Quality** | [CODE_STYLE.md](../../../CODE_STYLE.md) + CLAUDE.md | Coding standards & reliability |
27+
| **Formal Specs** | `docs/internals/specs/tla/`, `stateright_*.rs` | Protocol correctness |
28+
| **DST** | DST crate (when created) | Fault tolerance |
29+
30+
**Priority**: Safety > Performance > Developer Experience
31+
32+
## The Verification Pyramid
33+
34+
All verification layers share the same invariants:
35+
36+
```
37+
TLA+ Specs (docs/internals/specs/tla/*.tla)
38+
│ mirrors
39+
Shared Invariants (invariants/) ← SINGLE SOURCE
40+
│ used by
41+
┌───────────────┼───────────────┐
42+
▼ ▼ ▼
43+
Stateright DST Tests Production Metrics
44+
(exhaustive) (simulation) (Observability)
45+
```
46+
47+
## Testing Through Production Path
48+
49+
**MUST NOT** claim a feature works unless tested through the actual network stack.
50+
51+
```bash
52+
# 1. Start quickwit
53+
cargo run -p quickwit-cli -- run --config ../config/quickwit.yaml
54+
55+
# 2. Ingest via OTLP
56+
# (send logs/traces to localhost:4317)
57+
58+
# 3. Query via REST API
59+
curl http://localhost:7280/api/v1/<index>/search -d '{"query": "*"}'
60+
```
61+
62+
**Bypasses to AVOID**: Testing indexing pipeline without the HTTP/gRPC server, testing search without the REST API layer.
63+
64+
## DST (Deterministic Simulation Testing)
65+
66+
- DST tests define correctness for stateful components
67+
- Write DST tests before implementation for new state machines
68+
- Shared invariants are the single source of truth across all verification layers
69+
70+
## Architecture Evolution
71+
72+
Quickwit tracks architectural change through three lenses. See `docs/internals/adr/EVOLUTION.md` for the full process.
73+
74+
```
75+
Architecture Evolution
76+
77+
┌────────────────────┼────────────────────┐
78+
▼ ▼ ▼
79+
Characteristics Gaps Deviations
80+
(Proactive) (Reactive) (Pragmatic)
81+
"What we need" "What we learned" "What we accepted"
82+
```
83+
84+
| Lens | Location | When to Use |
85+
|------|----------|-------------|
86+
| **Characteristics** | `docs/internals/adr/` | Track cloud-native requirements |
87+
| **Gaps** | `docs/internals/adr/gaps/` | Design limitation from incident/production |
88+
| **Deviations** | `docs/internals/adr/deviations/` | Intentional divergence from ADR intent |
89+
90+
**Before implementing, check for**:
91+
- Open gaps (design limitations to be aware of)
92+
- Deviations (intentional divergence from ADRs)
93+
- Characteristic status (what's implemented vs planned)
94+
95+
## Additional Commit Checklist (on top of CLAUDE.md MUST items)
96+
97+
These are expected unless justified:
98+
- [ ] Functions under 70 lines
99+
- [ ] Explanatory variables for complex expressions
100+
- [ ] Documentation explains "why"
101+
- [ ] ADR/roadmap updated if applicable
102+
- [ ] DST test for new state transitions
103+
- [ ] Integration test for new API endpoints
104+
- [ ] Tests through production path (HTTP/gRPC)
105+
106+
## Reference Documentation
107+
108+
| Topic | Location |
109+
|-------|----------|
110+
| Verification & DST | [docs/internals/VERIFICATION.md](../../../docs/internals/VERIFICATION.md) |
111+
| Verification philosophy | [docs/internals/VERIFICATION_STACK.md](../../../docs/internals/VERIFICATION_STACK.md) |
112+
| Simulation workflow | [docs/internals/SIMULATION_FIRST_WORKFLOW.md](../../../docs/internals/SIMULATION_FIRST_WORKFLOW.md) |
113+
| Benchmarking | [docs/internals/BENCHMARKING.md](../../../docs/internals/BENCHMARKING.md) |
114+
| Rust style patterns | [docs/internals/RUST_STYLE.md](../../../docs/internals/RUST_STYLE.md) |
115+
| ADR index | [docs/internals/adr/README.md](../../../docs/internals/adr/README.md) |
116+
| Architecture evolution | [docs/internals/adr/EVOLUTION.md](../../../docs/internals/adr/EVOLUTION.md) |
117+
| Compaction architecture | [docs/internals/compaction-architecture.md](../../../docs/internals/compaction-architecture.md) |
118+
| Tantivy + Parquet design | [docs/internals/tantivy-parquet-architecture.md](../../../docs/internals/tantivy-parquet-architecture.md) |
119+
| Locality compaction | [docs/internals/locality-compaction/](../../../docs/internals/locality-compaction/) |

.github/workflows/ci.yml

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ jobs:
7676
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v.6.2.0
7777
with:
7878
python-version: '3.11'
79-
- uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
79+
- uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
8080
id: modified
8181
with:
8282
filters: |
@@ -88,11 +88,11 @@ jobs:
8888
- .github/workflows/ci.yml
8989
- name: Setup stable Rust Toolchain
9090
if: steps.modified.outputs.rust_src == 'true'
91-
uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9 # master
91+
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9 # master
9292
with:
9393
toolchain: stable
9494
- name: Setup cache
95-
uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2.8.2
95+
uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
9696
if: steps.modified.outputs.rust_src == 'true'
9797
with:
9898
workspaces: "./quickwit -> target"
@@ -130,7 +130,7 @@ jobs:
130130
actions: write
131131
steps:
132132
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
133-
- uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
133+
- uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
134134
id: modified
135135
with:
136136
filters: |
@@ -146,18 +146,18 @@ jobs:
146146
sudo apt-get -y install protobuf-compiler libcurl4-openssl-dev
147147
- name: Setup nightly Rust Toolchain (for rustfmt)
148148
if: steps.modified.outputs.rust_src == 'true'
149-
uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9 # master
149+
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9 # master
150150
with:
151151
toolchain: nightly
152152
components: rustfmt
153153
- name: Setup stable Rust Toolchain
154154
if: steps.modified.outputs.rust_src == 'true'
155-
uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9 # master
155+
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9 # master
156156
with:
157157
toolchain: stable
158158
- name: Setup cache
159159
if: steps.modified.outputs.rust_src == 'true'
160-
uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2.8.2
160+
uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
161161
with:
162162
workspaces: "./quickwit -> target"
163163
shared-key: "quickwit-cargo"
@@ -206,12 +206,12 @@ jobs:
206206
steps:
207207
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
208208
- name: Install Rust toolchain
209-
uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9 # master
209+
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9 # master
210210
with:
211211
toolchain: stable
212212

213213
- name: Cache cargo tools
214-
uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1
214+
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
215215
with:
216216
path: ~/.cargo/bin
217217
key: ${{ runner.os }}-cargo-tools-${{ hashFiles('**/Cargo.lock') }}

.github/workflows/coverage.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,12 +117,13 @@ jobs:
117117
sudo apt update
118118
sudo apt install libsasl2-dev
119119
sudo apt install libsasl2-2
120+
sudo apt install libcurl4-openssl-dev
120121
121122
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v.6.2.0
122123
with:
123124
python-version: '3.11'
124125

125-
- uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1
126+
- uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
126127
with:
127128
path: |
128129
~/.cargo/git

.github/workflows/ui-ci.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,11 +75,11 @@ jobs:
7575
cache: "yarn"
7676
cache-dependency-path: quickwit/quickwit-ui/yarn.lock
7777
- name: Setup stable Rust Toolchain
78-
uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9 # master
78+
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9 # master
7979
with:
8080
toolchain: stable
8181
- name: Setup Rust cache
82-
uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2.8.2
82+
uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
8383
with:
8484
workspaces: "./quickwit -> target"
8585
shared-key: "quickwit-cargo"

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,6 @@ qwdata
2929

3030
# Generated by prost/tonic build
3131
*_descriptor.bin
32+
33+
# GSD planning artifacts (local only)
34+
.planning

0 commit comments

Comments
 (0)