Route search-path metastore reads to the read replica

Add `ReadReplicaRoutingMetastore`, a `MetastoreService` wrapper that
routes the stale-tolerant reads issued by the search and analytics paths
(`index_metadata`, `indexes_metadata`, `list_indexes_metadata`,
`list_splits`, `list_metrics_splits`, `list_sketch_splits`) to read
replica nodes when any are connected, and everything else (writes and
non-hot-path reads) to the primary.

- Routing is decided per request from the read replica balance channel's
  live connection set, so it degrades to the primary when no replica is
  deployed. The check is a synchronous `watch` read with no borrow held
  across an await.
- Wire it into the searcher service and the DataFusion session builder, so
  all search (REST, Elasticsearch, gRPC) and metrics analytics benefit,
  while REST admin handlers keep read-your-writes against the primary.

Author: shuheiktgw

quickwit/quickwit-metastore/src/lib.rs

@@ -42,6 +42,7 @@ pub use metastore::file_backed::FileBackedMetastore;
 pub(crate) use metastore::index_metadata::serialize::{IndexMetadataV0_8, VersionedIndexMetadata};
 #[cfg(feature = "postgres")]
 pub use metastore::postgres::PostgresqlMetastore;
+pub use metastore::read_replica_routing_metastore::ReadReplicaRoutingMetastore;
 pub use metastore::{
     AddSourceRequestExt, CreateIndexRequestExt, CreateIndexResponseExt, IndexMetadata,
     IndexMetadataResponseExt, IndexesMetadataResponseExt, ListIndexesMetadataResponseExt,

quickwit/quickwit-metastore/src/metastore/mod.rs

@@ -18,6 +18,7 @@ pub(crate) mod index_metadata;
 pub mod postgres;
 
 pub mod control_plane_metastore;
+pub mod read_replica_routing_metastore;
 
 use std::cmp::Ordering;
 use std::ops::{Bound, RangeInclusive};