Add searcher metastore read replica setting

Author: shuheiktgw

config/quickwit.yaml

@@ -105,10 +105,8 @@ version: 0.8
 #
 # Optional PostgreSQL read replica URI. Nodes started with the
 # `metastore_read_replica` service connect to it over a read-only connection and
-# serve stale-tolerant read-only metastore requests (e.g. from searchers).
-# When set, nodes that issue stale-tolerant read-only metastore requests require
-# a `metastore_read_replica` node at startup and do not fall back to the primary
-# metastore. Defaults to unset.
+# serve stale-tolerant read-only metastore requests. Searchers use those nodes
+# only when `searcher.use_metastore_read_replica` is enabled. Defaults to unset.
 #
 # metastore_read_replica_uri: postgres://username:password@read-replica-host:port/db
 #
@@ -177,6 +175,11 @@ indexer:
 # https://quickwit.io/docs/configuration/node-config#searcher-configuration
 #
 # searcher:
+#   # If true, routes read-only metastore requests from searchers, including
+#   # DataFusion when enabled, to nodes running the `metastore_read_replica`
+#   # service. Searchers require at least one `metastore_read_replica` node at
+#   # startup and do not fall back to the primary metastore.
+#   use_metastore_read_replica: false
 #   fast_field_cache_capacity: 1G
 #   split_footer_cache_capacity: 500M
 #   partial_request_cache_capacity: 64M

docs/configuration/node-config.md

@@ -22,15 +22,15 @@ A commented example is available here: [quickwit.yaml](https://github.com/quickw
 | `version` | Config file version. `0.7` is the only available value with a retro compatibility on `0.5` and `0.4`. | | |
 | `cluster_id` | Unique identifier of the cluster the node will be joining. Clusters sharing the same network should use distinct cluster IDs.| `QW_CLUSTER_ID` | `quickwit-default-cluster` |
 | `node_id` | Unique identifier of the node. It must be distinct from the node IDs of its cluster peers. Defaults to the instance's short hostname if not set. | `QW_NODE_ID` | short hostname |
-| `enabled_services` | Enabled services (control_plane, indexer, janitor, metastore, searcher) | `QW_ENABLED_SERVICES` | all services |
+| `enabled_services` | Enabled services (control_plane, indexer, janitor, metastore, metastore_read_replica, searcher) | `QW_ENABLED_SERVICES` | all services except metastore_read_replica |
 | `listen_address` | The IP address or hostname that Quickwit service binds to for starting REST and GRPC server and connecting this node to other nodes. By default, Quickwit binds itself to 127.0.0.1 (localhost). This default is not valid when trying to form a cluster. | `QW_LISTEN_ADDRESS` | `127.0.0.1` |
 | `advertise_address` | IP address advertised by the node, i.e. the IP address that peer nodes should use to connect to the node for RPCs. | `QW_ADVERTISE_ADDRESS` | `listen_address` |
 | `gossip_listen_port` | The port which to listen for the Gossip cluster membership service (UDP). | `QW_GOSSIP_LISTEN_PORT` | `rest.listen_port` |
 | `grpc_listen_port` | The port on which gRPC services listen for traffic. | `QW_GRPC_LISTEN_PORT` | `rest.listen_port + 1` |
 | `peer_seeds` | List of IP addresses or hostnames used to bootstrap the cluster and discover the complete set of nodes. This list may contain the current node address and does not need to be exhaustive. If the list of peer seeds contains a host name, Quickwit will resolve it by querying the DNS every minute. On kubernetes for instance, it is a good practise to set it to a [headless service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services). | `QW_PEER_SEEDS` | |
 | `data_dir` | Path to directory where data (tmp data, splits kept for caching purpose) is persisted. This is mostly used in indexing. | `QW_DATA_DIR` | `./qwdata` |
 | `metastore_uri` | Metastore URI. Can be a local directory or `s3://my-bucket/indexes` or `postgres://username:password@localhost:5432/metastore`. [Learn more about the metastore configuration](metastore-config.md). | `QW_METASTORE_URI` | `{data_dir}/indexes` |
-| `metastore_read_replica_uri` | Optional PostgreSQL read replica URI. Nodes running the `metastore_read_replica` service connect to it over a read-only connection and serve stale-tolerant read-only metastore requests (e.g. from searchers). When set, nodes that issue those requests require a `metastore_read_replica` node at startup and do not fall back to the primary metastore. | `QW_METASTORE_READ_REPLICA_URI` | |
+| `metastore_read_replica_uri` | Optional PostgreSQL read replica URI. Nodes running the `metastore_read_replica` service connect to it over a read-only connection and serve stale-tolerant read-only metastore requests. Searchers use those nodes only when `searcher.use_metastore_read_replica` is enabled. | `QW_METASTORE_READ_REPLICA_URI` | |
 | `default_index_root_uri` | Default index root URI that defines the location where index data (splits) is stored. The index URI is built following the scheme: `{default_index_root_uri}/{index-id}` | `QW_DEFAULT_INDEX_ROOT_URI` | `{data_dir}/indexes` |
 | environment variable only | Log level of Quickwit. Can be a direct log level, or a comma separated list of `module_name=level` | `RUST_LOG` | `info` |
 
@@ -286,6 +286,7 @@ This section contains the configuration options for a Searcher.
 | `max_num_concurrent_split_searches` | Maximum number of concurrent split search requests running on a Searcher. | `100` |
 | `split_cache` | Searcher split cache configuration options defined in the section below. Cache disabled if unspecified. | |
 | `request_timeout_secs` | The time before a search request is cancelled. This should match the timeout of the stack calling into quickwit if there is one set.  | `30` |
+| `use_metastore_read_replica` | If true, routes read-only metastore requests from searchers, including DataFusion when enabled, to nodes running the `metastore_read_replica` service. Searchers require at least one `metastore_read_replica` node at startup and do not fall back to the primary metastore. | `false` |
 
 ### Searcher split cache configuration
 
@@ -302,6 +303,7 @@ Example:
 
 ```yaml
 searcher:
+  use_metastore_read_replica: false
   fast_field_cache_capacity: 1G
   split_footer_cache_capacity: 500M
   partial_request_cache_capacity: 64M

quickwit/quickwit-config/resources/tests/node_config/quickwit.json

@@ -65,6 +65,7 @@
         "replication_factor": 2
     },
     "searcher": {
+        "use_metastore_read_replica": true,
         "aggregation_memory_limit": "1G",
         "aggregation_bucket_limit": 500000,
         "fast_field_cache_capacity": "10G",

quickwit/quickwit-config/resources/tests/node_config/quickwit.toml

@@ -55,6 +55,7 @@ parquet_merge_use_streaming_engine = true
 replication_factor = 2
 
 [searcher]
+use_metastore_read_replica = true
 aggregation_memory_limit = "1G"
 aggregation_bucket_limit = 500_000
 fast_field_cache_capacity = "10G"

quickwit/quickwit-config/resources/tests/node_config/quickwit.yaml

@@ -59,6 +59,7 @@ ingest_api:
   replication_factor: 2
 
 searcher:
+  use_metastore_read_replica: true
   aggregation_memory_limit: 1G
   aggregation_bucket_limit: 500000
   fast_field_cache_capacity: 10G

quickwit/quickwit-config/src/node_config/mod.rs

@@ -343,6 +343,9 @@ pub struct SearcherConfig {
     #[serde(default)]
     #[serde(skip_serializing_if = "Option::is_none")]
     pub storage_timeout_policy: Option<StorageTimeoutPolicy>,
+    /// Routes read-only metastore requests from searchers, including DataFusion when enabled, to
+    /// nodes running the `metastore_read_replica` service.
+    pub use_metastore_read_replica: bool,
     pub warmup_memory_budget: ByteSize,
     pub warmup_single_split_initial_allocation: ByteSize,
     /// Lambda configuration for serverless leaf search execution.
@@ -566,6 +569,7 @@ impl Default for SearcherConfig {
             request_timeout_secs: Self::default_request_timeout_secs(),
             leaf_request_timeout_secs: Self::default_request_timeout_secs(),
             storage_timeout_policy: None,
+            use_metastore_read_replica: false,
             warmup_memory_budget: ByteSize::gb(100),
             warmup_single_split_initial_allocation: ByteSize::mb(300),
             lambda: None,

quickwit/quickwit-config/src/node_config/serialize.rs

@@ -794,6 +794,7 @@ mod tests {
                     timeout_millis: 2_000,
                     max_num_retries: 2
                 }),
+                use_metastore_read_replica: true,
                 warmup_memory_budget: ByteSize::gb(100),
                 warmup_single_split_initial_allocation: ByteSize::mb(300),
                 lambda: Some(LambdaConfig {

quickwit/quickwit-serve/src/lib.rs

@@ -728,10 +728,13 @@ pub async fn serve_quickwit(
     };
 
     // Searchers and the DataFusion analytics path use the primary metastore by default. When
-    // `metastore_read_replica_uri` is configured, they require `metastore_read_replica` nodes and
-    // do not fall back to the primary if none are available.
+    // `searcher.use_metastore_read_replica` is enabled, searcher nodes require
+    // `metastore_read_replica` nodes and do not fall back to the primary if none are available.
+    let use_metastore_read_replica_for_search = node_config
+        .is_service_enabled(QuickwitService::Searcher)
+        && node_config.searcher_config.use_metastore_read_replica;
     let search_metastore_client: MetastoreReadServiceClient =
-        if node_config.metastore_read_replica_uri.is_some() {
+        if use_metastore_read_replica_for_search {
             let read_replica_metastore = build_metastore_client(
                 &cluster,
                 QuickwitService::MetastoreReadReplica,