Document the metastore read replica and test read-only resolution

- Document `metastore_read_replica_uri` in the node config reference and
  the example `quickwit.yaml`.
- Add a PostgreSQL-gated test that resolves a read-only metastore against
  a real database and verifies it serves read RPCs.

Author: shuheiktgw

config/quickwit.yaml

@@ -103,6 +103,13 @@ version: 0.8
 # metastore_uri: s3://your-bucket/indexes
 # metastore_uri: postgres://username:password@host:port/db
 #
+# Optional PostgreSQL read replica URI. Nodes started with the
+# `metastore_read_replica` service connect to it over a read-only connection and
+# serve stale-tolerant read-only metastore requests (e.g. from searchers). When
+# unset, those requests are served by the primary metastore. Defaults to unset.
+#
+# metastore_read_replica_uri: postgres://username:password@read-replica-host:port/db
+#
 # When using a file-backed metastore, the state of the metastore will be cached forever.
 # If you are indexing and searching from different processes, it is possible to periodically
 # refresh the state of the metastore on the searcher using the `polling_interval` hashtag.

docs/configuration/node-config.md

@@ -30,6 +30,7 @@ A commented example is available here: [quickwit.yaml](https://github.com/quickw
 | `peer_seeds` | List of IP addresses or hostnames used to bootstrap the cluster and discover the complete set of nodes. This list may contain the current node address and does not need to be exhaustive. If the list of peer seeds contains a host name, Quickwit will resolve it by querying the DNS every minute. On kubernetes for instance, it is a good practise to set it to a [headless service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services). | `QW_PEER_SEEDS` | |
 | `data_dir` | Path to directory where data (tmp data, splits kept for caching purpose) is persisted. This is mostly used in indexing. | `QW_DATA_DIR` | `./qwdata` |
 | `metastore_uri` | Metastore URI. Can be a local directory or `s3://my-bucket/indexes` or `postgres://username:password@localhost:5432/metastore`. [Learn more about the metastore configuration](metastore-config.md). | `QW_METASTORE_URI` | `{data_dir}/indexes` |
+| `metastore_read_replica_uri` | Optional PostgreSQL read replica URI. Nodes running the `metastore_read_replica` service connect to it over a read-only connection and serve stale-tolerant read-only metastore requests (e.g. from searchers). | `QW_METASTORE_READ_REPLICA_URI` | |
 | `default_index_root_uri` | Default index root URI that defines the location where index data (splits) is stored. The index URI is built following the scheme: `{default_index_root_uri}/{index-id}` | `QW_DEFAULT_INDEX_ROOT_URI` | `{data_dir}/indexes` |
 | environment variable only | Log level of Quickwit. Can be a direct log level, or a comma separated list of `module_name=level` | `RUST_LOG` | `info` |
 

quickwit/quickwit-metastore/src/metastore_resolver.rs

@@ -240,4 +240,31 @@ mod tests {
             metastore_resolver.resolve(&postgres_uri).await.unwrap();
         }
     }
+
+    #[cfg(feature = "postgres")]
+    #[tokio::test]
+    async fn test_metastore_resolver_resolve_read_only_postgres() {
+        use std::env;
+
+        use quickwit_proto::metastore::{ListIndexesMetadataRequest, MetastoreService};
+
+        let metastore_resolver = MetastoreResolver::unconfigured();
+        let test_database_url = env::var("QW_TEST_DATABASE_URL").unwrap_or_else(|_| {
+            "postgres://quickwit-dev:quickwit-dev@localhost/quickwit-metastore-dev".to_string()
+        });
+        let postgres_uri = Uri::from_str(&test_database_url).unwrap();
+        // The read replica skips migrations, so ensure the schema exists by resolving the
+        // read-write metastore first.
+        metastore_resolver.resolve(&postgres_uri).await.unwrap();
+
+        let read_only_metastore = metastore_resolver
+            .resolve_read_only(&postgres_uri)
+            .await
+            .unwrap();
+        // A read-only connection must still serve read RPCs.
+        read_only_metastore
+            .list_indexes_metadata(ListIndexesMetadataRequest::all())
+            .await
+            .unwrap();
+    }
 }