Skip to content

Commit 82fa2be

Browse files
committed
Merge branch 'main' into nadav/merge-feature-split-merges
2 parents bdb04be + 515d752 commit 82fa2be

198 files changed

Lines changed: 8397 additions & 2235 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,6 @@ states/
3838

3939
# CI-downloaded Lambda artifacts (used by Docker builds; not committed)
4040
quickwit/lambda-artifact/
41+
42+
# OpenSSL CA serial file generated during TLS test setup
43+
**/ca.srl

LICENSE-3rdparty.csv

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,7 @@ cobs,https://github.com/jamesmunns/cobs.rs,MIT OR Apache-2.0,"Allen Welkie <>, J
165165
codespan-reporting,https://github.com/brendanzab/codespan,Apache-2.0,Brendan Zabarauskas <bjzaba@yahoo.com.au>
166166
colorchoice,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The colorchoice Authors
167167
colored,https://github.com/mackwic/colored,MPL-2.0,Thomas Wickham <mackwic@gmail.com>
168+
combine,https://github.com/Marwes/combine,MIT,Markus Westerlind <marwes91@gmail.com>
168169
comfy-table,https://github.com/nukesor/comfy-table,MIT,Arne Beer <contact@arne.beer>
169170
community-id,https://github.com/traceflight/rs-community-id,MIT OR Apache-2.0,Julian Wang <traceflight@outlook.com>
170171
compression-codecs,https://github.com/Nullus157/async-compression,MIT OR Apache-2.0,"Wim Looman <wim@nemo157.com>, Allen Bui <fairingrey@gmail.com>"
@@ -192,7 +193,6 @@ crc-catalog,https://github.com/akhilles/crc-catalog,MIT OR Apache-2.0,Akhil Vela
192193
crc-fast,https://github.com/awesomized/crc-fast-rust,MIT OR Apache-2.0,Don MacAskill
193194
crc32fast,https://github.com/srijs/rust-crc32fast,MIT OR Apache-2.0,"Sam Rijs <srijs@airpost.net>, Alex Crichton <alex@alexcrichton.com>"
194195
criterion-plot,https://github.com/criterion-rs/criterion.rs,Apache-2.0 OR MIT,"Jorge Aparicio <japaricious@gmail.com>, Brook Heisler <brookheisler@gmail.com>"
195-
critical-section,https://github.com/rust-embedded/critical-section,MIT OR Apache-2.0,The critical-section Authors
196196
cron,https://github.com/zslayton/cron,MIT OR Apache-2.0,Zack Slayton <zack.slayton@gmail.com>
197197
crossbeam-channel,https://github.com/crossbeam-rs/crossbeam,MIT OR Apache-2.0,The crossbeam-channel Authors
198198
crossbeam-deque,https://github.com/crossbeam-rs/crossbeam,MIT OR Apache-2.0,The crossbeam-deque Authors
@@ -419,6 +419,10 @@ jiff,https://github.com/BurntSushi/jiff,Unlicense OR MIT,Andrew Gallant <jamslam
419419
jiff-static,https://github.com/BurntSushi/jiff,Unlicense OR MIT,Andrew Gallant <jamslam@gmail.com>
420420
jiff-tzdb,https://github.com/BurntSushi/jiff,Unlicense OR MIT,Andrew Gallant <jamslam@gmail.com>
421421
jiff-tzdb-platform,https://github.com/BurntSushi/jiff,Unlicense OR MIT,Andrew Gallant <jamslam@gmail.com>
422+
jni,https://github.com/jni-rs/jni-rs,MIT OR Apache-2.0,jni team
423+
jni-macros,https://github.com/jni-rs/jni-rs,MIT OR Apache-2.0,The jni-macros Authors
424+
jni-sys,https://github.com/jni-rs/jni-sys,MIT OR Apache-2.0,"Steven Fackler <sfackler@gmail.com>, Robert Bragg <robert@sixbynine.org>"
425+
jni-sys-macros,https://github.com/jni-rs/jni-sys,MIT OR Apache-2.0,Robert Bragg <robert@sixbynine.org>
422426
jobserver,https://github.com/rust-lang/jobserver-rs,MIT OR Apache-2.0,Alex Crichton <alex@alexcrichton.com>
423427
js-sys,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys,MIT OR Apache-2.0,The wasm-bindgen Developers
424428
json_comments,https://github.com/tmccombs/json-comments-rs,Apache-2.0,Thayne McCombs <astrothayne@gmail.com>
@@ -470,8 +474,8 @@ measure_time,https://github.com/PSeitz/rust_measure_time,MIT,Pascal Seitz <pasca
470474
memchr,https://github.com/BurntSushi/memchr,Unlicense OR MIT,"Andrew Gallant <jamslam@gmail.com>, bluss"
471475
memmap2,https://github.com/RazrFalcon/memmap2-rs,MIT OR Apache-2.0,"Dan Burkert <dan@danburkert.com>, Yevhenii Reizner <razrfalcon@gmail.com>, The Contributors"
472476
metrics,https://github.com/metrics-rs/metrics,MIT,Toby Lawrence <toby@nuclearfurnace.com>
473-
metrics-exporter-otel,https://github.com/palindrom615/metrics,MIT,Whoemoon Jang <palindrom615@gmail.com>
474477
metrics-exporter-prometheus,https://github.com/metrics-rs/metrics,MIT AND Apache-2.0,Toby Lawrence <toby@nuclearfurnace.com>
478+
metrics-opentelemetry,https://github.com/DoumanAsh/metrics-opentelemetry,BSL-1.0,The metrics-opentelemetry Authors
475479
metrics-util,https://github.com/metrics-rs/metrics,MIT,Toby Lawrence <toby@nuclearfurnace.com>
476480
mime,https://github.com/hyperium/mime,MIT OR Apache-2.0,Sean McArthur <sean@seanmonstar.com>
477481
mime_guess,https://github.com/abonander/mime_guess,MIT,Austin Bonander <austin.bonander@gmail.com>
@@ -706,6 +710,8 @@ rustls,https://github.com/rustls/rustls,Apache-2.0 OR ISC OR MIT,The rustls Auth
706710
rustls-native-certs,https://github.com/rustls/rustls-native-certs,Apache-2.0 OR ISC OR MIT,The rustls-native-certs Authors
707711
rustls-pemfile,https://github.com/rustls/pemfile,Apache-2.0 OR ISC OR MIT,The rustls-pemfile Authors
708712
rustls-pki-types,https://github.com/rustls/pki-types,MIT OR Apache-2.0,The rustls-pki-types Authors
713+
rustls-platform-verifier,https://github.com/rustls/rustls-platform-verifier,MIT OR Apache-2.0,The rustls-platform-verifier Authors
714+
rustls-platform-verifier-android,https://github.com/rustls/rustls-platform-verifier,MIT OR Apache-2.0,The rustls-platform-verifier-android Authors
709715
rustls-webpki,https://github.com/rustls/webpki,ISC,The rustls-webpki Authors
710716
rustop,https://chiselapp.com/user/fifr/repository/rustop,MIT,Frank Fischer <frank-fischer@shadow-soft.de>
711717
rustversion,https://github.com/dtolnay/rustversion,MIT OR Apache-2.0,David Tolnay <dtolnay@gmail.com>
@@ -765,6 +771,7 @@ signal-hook,https://github.com/vorner/signal-hook,Apache-2.0 OR MIT,"Michal 'vor
765771
signal-hook-registry,https://github.com/vorner/signal-hook,MIT OR Apache-2.0,"Michal 'vorner' Vaner <vorner@vorner.cz>, Masaki Hara <ackie.h.gmai@gmail.com>"
766772
signature,https://github.com/RustCrypto/traits/tree/master/signature,Apache-2.0 OR MIT,RustCrypto Developers
767773
simd-adler32,https://github.com/mcountryman/simd-adler32,MIT,Marvin Countryman <me@maar.vin>
774+
simd_cesu8,https://github.com/seancroach/simd_cesu8,Apache-2.0 OR MIT,Sean C. Roach <me@seancroach.dev>
768775
simdutf8,https://github.com/rusticstuff/simdutf8,MIT OR Apache-2.0,Hans Kratz <hans@appfour.com>
769776
simple_asn1,https://github.com/acw/simple_asn1,ISC,Adam Wick <awick@uhsure.com>
770777
siphasher,https://github.com/jedisct1/rust-siphash,MIT OR Apache-2.0,Frank Denis <github@pureftpd.org>
@@ -862,6 +869,7 @@ tonic-health,https://github.com/hyperium/tonic,MIT,James Nugent <james@jen20.com
862869
tonic-prost,https://github.com/hyperium/tonic,MIT,Lucio Franco <luciofranco14@gmail.com>
863870
tonic-prost-build,https://github.com/hyperium/tonic,MIT,Lucio Franco <luciofranco14@gmail.com>
864871
tonic-reflection,https://github.com/hyperium/tonic,MIT,"James Nugent <james@jen20.com>, Samani G. Gikandi <samani@gojulas.com>"
872+
tonic-types,https://github.com/hyperium/tonic,MIT,"Lucio Franco <luciofranco14@gmail.com>, Rafael Lemos <flemos.rafael.dev@gmail.com>"
865873
tower,https://github.com/tower-rs/tower,MIT,Tower Maintainers <team@tower-rs.com>
866874
tower-http,https://github.com/tower-rs/tower-http,MIT,Tower Maintainers <team@tower-rs.com>
867875
tower-layer,https://github.com/tower-rs/tower,MIT,Tower Maintainers <team@tower-rs.com>
@@ -939,6 +947,7 @@ wasmparser,https://github.com/bytecodealliance/wasm-tools/tree/main/crates/wasmp
939947
wasmtimer,https://github.com/whizsid/wasmtimer-rs,MIT,"WhizSid <whizsid@aol.com>, Pierre Krieger <pierre.krieger1708@gmail.com>"
940948
web-sys,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/web-sys,MIT OR Apache-2.0,The wasm-bindgen Developers
941949
web-time,https://github.com/daxpedda/web-time,MIT OR Apache-2.0,The web-time Authors
950+
webpki-root-certs,https://github.com/rustls/webpki-roots,CDLA-Permissive-2.0,The webpki-root-certs Authors
942951
webpki-roots,https://github.com/rustls/webpki-roots,CDLA-Permissive-2.0,The webpki-roots Authors
943952
whoami,https://github.com/ardaku/whoami,Apache-2.0 OR BSL-1.0 OR MIT,The whoami Authors
944953
winapi,https://github.com/retep998/winapi-rs,MIT OR Apache-2.0,Peter Atashian <retep998@gmail.com>

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
</p>
1616

1717
<h2 align="center">
18-
Cloud-native search engine for observability (logs, traces, and soon metrics!). An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
18+
Open-source search engine for observability (logs, traces, and soon metrics!).
1919
</h2>
2020

2121
<h4 align="center">

config/quickwit.yaml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,41 @@ version: 0.8
4040
# extra_headers:
4141
# x-header-1: header-value-1
4242
# x-header-2: header-value-2
43+
# tls:
44+
# cert_path: /path/to/server.crt
45+
# key_path: /path/to/server.key
46+
# # mTLS: require clients to present a certificate signed by this CA.
47+
# ca_path: /path/to/ca.crt
48+
# verify_client_cert: true
49+
# # How often cert_path/key_path are polled for changes and hot-reloaded; an
50+
# # immediate reload can also be triggered with SIGHUP. Defaults to 5m.
51+
# cert_reload_interval: 5m
52+
#
53+
# Optional plaintext health-check server. Disabled unless `listen_port` is set (or the
54+
# `QW_HEALTH_LISTEN_PORT` environment variable). It serves only `/health/livez` and
55+
# `/health/readyz`, letting liveness/readiness probes reach the node even when the REST API above
56+
# is put behind mTLS. The same routes also stay on the main REST server. Bind it to a
57+
# cluster-internal interface, as it performs no TLS termination or client authentication.
58+
#
59+
# health:
60+
# listen_port: 7282
4361
#
4462
# grpc:
4563
# max_message_size: 10 MiB
64+
# tls:
65+
# cert_path: /path/to/server.crt
66+
# key_path: /path/to/server.key
67+
# # Trust root for peers this node *connects to* (every node is also a gRPC client to
68+
# # its peers). With verify_client_cert, also validates peers connecting to this node.
69+
# ca_path: /path/to/ca.crt
70+
# # mTLS: require peers to present a certificate signed by ca_path. Each node reuses
71+
# # cert_path/key_path above as its own client identity toward peers.
72+
# verify_client_cert: true
73+
# # Hostname checked against the peer certificate's SAN when connecting to peers.
74+
# expected_name: quickwit.local
75+
# # How often cert_path/key_path are polled for changes and hot-reloaded; an
76+
# # immediate reload can also be triggered with SIGHUP. Defaults to 5m.
77+
# cert_reload_interval: 5m
4678
#
4779
# IP address advertised by the node, i.e. the IP address that peer nodes should use to connect to the node for RPCs.
4880
# The environment variable `QW_ADVERTISE_ADDRESS` can also be used to override this value.

docs/configuration/node-config.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ This section contains the REST API configuration options.
4242
| `listen_port` | The port on which the REST API listens for HTTP traffic. | `QW_REST_LISTEN_PORT` | `7280` |
4343
| `cors_allow_origins` | Configure the CORS origins which are allowed to access the API. [Read more](#configuring-cors-cross-origin-resource-sharing) | |
4444
| `extra_headers` | List of header names and values | | |
45+
| `tls` | Enables HTTPS for the REST API. [Read more](#tls-configuration) | | |
4546

4647
### Configuring CORS (Cross-origin resource sharing)
4748

@@ -74,6 +75,7 @@ This section contains the configuration options for gRPC services and clients us
7475
| Property | Description | Env variable | Default value |
7576
| --- | --- | --- | --- |
7677
| `max_message_size` | The maximum size (in bytes) of messages exchanged by internal gRPC clients and services. | | `20 MiB` |
78+
| `tls` | Enables TLS for gRPC services and clients. [Read more](#tls-configuration) | | |
7779

7880
Example of a gRPC configuration:
7981

@@ -87,6 +89,85 @@ We advise changing the default value of 20 MiB only if you encounter the followi
8789
`Error, message length too large: found 24732228 bytes, the limit is: 20971520 bytes.` In that case, increase `max_message_size` by increments of 10 MiB until the issue disappears. This is a temporary fix: the next version of Quickwit will rely exclusively on gRPC streaming endpoints and handle messages of any length.
8890
:::
8991

92+
## Health check configuration
93+
94+
This section configures an optional, **plaintext (no TLS)** HTTP server that exposes only the health endpoints `/health/livez` (liveness) and `/health/readyz` (readiness). Its purpose is to let liveness/readiness probes (for example from Kubernetes or a load balancer) reach the node even when the main REST API is put behind [TLS or mTLS](#tls-configuration), which a simple HTTP probe cannot negotiate.
95+
96+
The health server is **disabled by default**. It starts only when `listen_port` is set (or the `QW_HEALTH_LISTEN_PORT` environment variable is provided). The same `/health/*` endpoints always remain available on the main REST API as well.
97+
98+
| Property | Description | Env variable | Default value |
99+
| --- | --- | --- | --- |
100+
| `listen_port` | The port on which the plaintext health server listens for HTTP traffic. When unset, the health server is disabled. | `QW_HEALTH_LISTEN_PORT` | _(disabled)_ |
101+
102+
Pick a free port that is not already used by the REST or gRPC servers.
103+
104+
Example of a health check configuration:
105+
106+
```yaml
107+
health:
108+
listen_port: 7282
109+
```
110+
111+
:::warning
112+
This server performs no TLS termination and no client authentication. Bind it to a cluster-internal interface (via `listen_address`) and do not expose it publicly.
113+
:::
114+
115+
## TLS configuration
116+
117+
Both the REST API (`rest.tls`) and the internal gRPC services (`grpc.tls`) can be secured with TLS, optionally with mutual TLS (mTLS). The two sections share the same properties:
118+
119+
:::tip
120+
When the REST API is behind mTLS, simple HTTP health probes can no longer reach it. Enable the plaintext [health check server](#health-check-configuration) to keep liveness/readiness probes working.
121+
:::
122+
123+
| Property | Description | Default value |
124+
| --- | --- | --- |
125+
| `cert_path` | Path to the PEM-encoded X.509 certificate (or chain) presented by the server. Setting this enables TLS. | |
126+
| `key_path` | Path to the PEM-encoded private key matching `cert_path`. | |
127+
| `ca_path` | Path to a PEM file holding the trusted CA certificate(s). Used by the server to validate client certificates when `verify_client_cert` is enabled, and by the gRPC client to validate peer certificates. Multiple CA certificates may be concatenated in the same file: all of them are trusted (see [CA rotation](#ca-rotation)). | |
128+
| `verify_client_cert` | If `true`, require clients (REST) or peers (gRPC) to present a certificate signed by `ca_path`, i.e. enforce mutual TLS. | `false` |
129+
| `expected_name` | gRPC only. The hostname the gRPC client checks against the peer certificate's Subject Alternative Name (SAN). Defaults to the peer's address. | |
130+
| `cert_reload_interval` | How often `cert_path` and `key_path` are polled for on-disk changes and hot-reloaded, without restarting the process. An immediate reload can also be triggered by sending `SIGHUP` to the process. | `5m` |
131+
132+
Certificates are hot-reloaded: when `cert_path`/`key_path` change on disk, new connections pick up the new certificate within `cert_reload_interval` (or immediately on `SIGHUP`), while in-flight connections keep the certificate they negotiated. A new certificate is only applied if it parses and matches its key; otherwise the previous certificate is kept. Note that the CA trust roots (`ca_path`) are **not** hot-reloaded — rotating them still requires a restart.
133+
134+
### CA rotation
135+
136+
Because `ca_path` accepts multiple CA certificates concatenated in a single PEM file, you can rotate the CA without downtime by temporarily trusting both the old and the new CA:
137+
138+
1. Append the **new** CA certificate to the `ca_path` file, so it contains both the old and the new CA.
139+
2. Roll-restart every node. Each node now trusts certificates signed by either CA, while peers may still present certificates signed by the old CA.
140+
3. Re-issue every node's `cert_path`/`key_path` with certificates signed by the new CA (these are hot-reloaded, no restart needed).
141+
4. Remove the **old** CA certificate from the `ca_path` file, leaving only the new CA.
142+
5. Roll-restart every node again to drop trust in the old CA.
143+
144+
Since the CA file is read once at startup, both restarts are required to pick up the changes to `ca_path`.
145+
146+
Example of a REST configuration with mTLS:
147+
148+
```yaml
149+
rest:
150+
tls:
151+
cert_path: /path/to/server.crt
152+
key_path: /path/to/server.key
153+
ca_path: /path/to/ca.crt
154+
verify_client_cert: true
155+
cert_reload_interval: 5m
156+
```
157+
158+
Example of a gRPC configuration with mTLS:
159+
160+
```yaml
161+
grpc:
162+
tls:
163+
cert_path: /path/to/server.crt
164+
key_path: /path/to/server.key
165+
ca_path: /path/to/ca.crt
166+
expected_name: quickwit.local
167+
verify_client_cert: true
168+
cert_reload_interval: 5m
169+
```
170+
90171
## Storage configuration
91172

92173
Please refer to the dedicated [storage configuration](storage-config) page to learn more about configuring Quickwit for various storage providers.

docs/configuration/ports-config.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ The ports used are computed relative to the `rest.listen_port` port, as follows.
1919

2020
It is not possible for the moment to configure these ports independently.
2121

22+
Optionally, an additional plaintext health check server can be enabled on its own port by setting `health.listen_port` (see the [health check configuration](node-config.md#health-check-configuration)). It is disabled by default. The port `${rest.listen_port} + 2` (7282 by default) is a natural choice, but any free port that does not collide with the three above works.
23+
2224

2325
In order to form a cluster, you will also need to define a `peer_seeds` parameter.
2426
The following addresses are valid peer seed addresses:

0 commit comments

Comments
 (0)