feat(secrets): configurable credential chain and writer

Turning on the Postgres secrets backend used to be all-or-nothing: setting
[secrets] dropped vault from the read chain and ran a mandatory import. Now
the backend stores and the write target are operator config, so a site can
read from postgres, vault, or both -- in whatever order it wants -- and
choose where new writes land.

Two new [secrets] fields drive it, both naming a CredentialBackend
(postgres or vault):

- stores -- the backend read order, first match wins. The env/file local
  overrides are always tried first (when their [credentials.*] section is
  enabled); this list just orders the backends behind them. Defaults to
  ["vault"], today's behavior. Order is the operator's choice; for example,
  to roll Postgres in gradually: ["vault"] -> ["postgres", "vault"] ->
  ["postgres"].
- writer -- vault (default) or postgres. Flip it to send new writes to the
  journal.

import_from stays fully independent -- importing from vault is orthogonal to
where reads and writes flow, and it is now gated visibly at the call site.
An empty or duplicate stores list fails the boot; a writer whose backend is
not in stores is allowed but warns about the read-after-write gap (e.g. a
deliberate postgres shadow-write).

Tests added!

This supports #2811

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

Author: chet

crates/api-core/src/cfg/file.rs

@@ -730,9 +730,9 @@ pub struct CarbideConfig {
     #[serde(default)]
     pub tracing: TracingConfig,
 
-    /// Secrets backend configuration. When present, credentials live
-    /// encrypted in Postgres and vault leaves the credential chain
-    /// entirely; when absent, vault remains the credential store.
+    /// Secrets backend configuration. When present, the credential reader
+    /// chain and write target are operator-configured (defaulting to the same
+    /// env -> file -> vault behavior as when it is absent); see `SecretsConfig`.
     pub secrets: Option<SecretsConfig>,
 }
 
@@ -846,24 +846,26 @@ pub enum BgpLeafSessionPassword {
     SiteWide,
 }
 
-/// Configures the Postgres secrets backend. When this section is present,
-/// credentials live encrypted in Postgres and vault is not in the
-/// credential chain at all -- the one-time import either completes before
-/// the process serves traffic, or the process does not start. Vault keeps
-/// serving PKI certificates either way.
+/// Configures the Postgres secrets backend and how credentials flow. When
+/// this section is present the reader chain and the write target come from
+/// `stores` / `writer` below; their defaults keep today's behavior
+/// (env -> file -> vault, writes to vault), so adding `[secrets]` does not
+/// change credential routing on its own. Operators choose which backend
+/// stores to read, in what order, and which store takes writes, by editing
+/// `stores` and `writer`. Vault keeps serving PKI certificates regardless of
+/// the chain.
 ///
-/// Enabling this on an existing site has two prerequisites that live
-/// outside this process:
+/// Two prerequisites live outside this process and matter once writes move
+/// to Postgres (`writer = "postgres"`) or vault leaves `stores`:
 ///
 /// - Services that read credentials from vault through their own chains
-///   (`bmc-proxy`, `dsx-exchange-consumer`) keep reading vault and will
-///   not see anything carbide-api writes to Postgres afterwards. They must
-///   be migrated or fed another way before credentials here change.
-/// - During a rolling upgrade, replicas still running the vault config
-///   keep writing rotated credentials to vault, where they are stranded
-///   once the import has completed. Keep autonomous credential writers
-///   (site-explorer credential rotation) disabled until the whole fleet
-///   runs this config.
+///   (`bmc-proxy`, `dsx-exchange-consumer`) will not see anything carbide-api
+///   writes to Postgres. They must be pointed at the same store, or fed
+///   another way, before the credentials they read change.
+/// - During a rolling upgrade, replicas still on an older config keep writing
+///   rotated credentials to their own writer. Keep autonomous credential
+///   writers (site-explorer credential rotation) disabled until the whole
+///   fleet runs a consistent config.
 #[derive(Clone, Debug, Deserialize, Serialize)]
 #[serde(deny_unknown_fields)]
 pub struct SecretsConfig {
@@ -884,9 +886,37 @@ pub struct SecretsConfig {
     /// ```
     pub routing: std::collections::HashMap<String, String>,
 
+    /// The credential *store* read order, highest priority first (first match
+    /// wins). The local-override readers (env, file) are always tried ahead of
+    /// these, when their `[credentials.*]` section is enabled; this list only
+    /// orders the stores behind them. Order is the operator's choice -- list
+    /// the stores you want, in the priority you want. Defaults to `["vault"]`
+    /// -- with the local overrides, that is the env -> file -> vault chain.
+    ///
+    /// For example, to roll Postgres in gradually, walk this list:
+    ///
+    /// 1. `["vault"]` -- Postgres configured but not yet read.
+    /// 2. `["postgres", "vault"]` -- Postgres in front, vault as the safety net
+    ///    for anything Postgres misses.
+    /// 3. `["postgres"]` -- vault no longer read.
+    ///
+    /// An empty list, or a store named twice, fails the boot.
+    #[serde(default = "default_secret_stores")]
+    pub stores: Vec<CredentialBackend>,
+
+    /// Where new credential writes go. Defaults to `vault`; set to `postgres`
+    /// to send new writes to the journal. Independent of `stores`: e.g.
+    /// `writer = "postgres"` while `postgres` is not in `stores` (reads still
+    /// served by vault) is a valid shadow-write -- it confirms writes land
+    /// before reads start trusting Postgres -- and only logs a warning.
+    #[serde(default)]
+    pub writer: CredentialBackend,
+
     /// A source backend to import secrets from at startup. Unset means a
     /// fresh site with nothing to import; unsupported values fail config
-    /// parsing rather than silently skipping the import.
+    /// parsing rather than silently skipping the import. Independent of
+    /// `stores`/`writer` -- importing from vault is orthogonal to where
+    /// reads and writes flow.
     pub import_from: Option<ImportSource>,
 
     /// How to treat secrets that already exist in Postgres during import.
@@ -902,6 +932,27 @@ pub enum ImportSource {
     Vault,
 }
 
+/// A credential backend -- postgres or vault. Listed in `[secrets].stores` to
+/// order the backends behind the always-first local overrides (env, file;
+/// first match wins, see `ChainedCredentialReader`), and named by
+/// `[secrets].writer` to choose where new writes go.
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum CredentialBackend {
+    /// The Postgres secrets journal.
+    Postgres,
+    /// Vault/OpenBao KV. The default write target (today's behavior).
+    #[default]
+    Vault,
+}
+
+/// The default backend-store order (just vault). With the always-first env/file
+/// local overrides, this is the env -> file -> vault chain, so adding
+/// `[secrets]` changes nothing until an operator edits it.
+fn default_secret_stores() -> Vec<CredentialBackend> {
+    vec![CredentialBackend::Vault]
+}
+
 /// Configures the KMS backends that wrap DEKs. Several named providers can
 /// be defined: the active one wraps DEKs for new writes, and every provider
 /// answers unwraps for the kek_ids it has.
@@ -4156,6 +4207,11 @@ firmware_url = "https://firmware.example.com/fw-b.bin"
             secrets.import_approach,
             crate::secrets::ImportApproach::MissingOnly
         );
+
+        // stores/writer were omitted above, so they default to vault-only
+        // (env/file are prepended separately) writing to vault.
+        assert_eq!(secrets.stores, vec![CredentialBackend::Vault]);
+        assert_eq!(secrets.writer, CredentialBackend::Vault);
     }
 
     // Verifies that a typo'd import source fails config parsing instead of
@@ -4186,6 +4242,96 @@ firmware_url = "https://firmware.example.com/fw-b.bin"
         assert!(toml::from_str::<Wrapper>(toml_str).is_err());
     }
 
+    // Verifies the stores list and writer parse from their enum values --
+    // one with Postgres in front of vault (writes to Postgres) and a
+    // postgres-only one (vault not read, writes to Postgres).
+    #[test]
+    fn secrets_config_parses_stores_and_writer() {
+        #[derive(Deserialize)]
+        struct Wrapper {
+            secrets: SecretsConfig,
+        }
+
+        let pg_first = r#"
+            [secrets]
+            stores = ["postgres", "vault"]
+            writer = "postgres"
+
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+        let secrets = toml::from_str::<Wrapper>(pg_first)
+            .expect("parse pg-first")
+            .secrets;
+        assert_eq!(
+            secrets.stores,
+            vec![CredentialBackend::Postgres, CredentialBackend::Vault]
+        );
+        assert_eq!(secrets.writer, CredentialBackend::Postgres);
+
+        // Postgres-only reads, writes to postgres too. (The
+        // writer-defaults-to-vault case is covered by the deserialize test
+        // above, with vault still in stores -- pairing a postgres-only chain
+        // with a vault writer is the read-after-write gap run.rs warns about.)
+        let postgres_only = r#"
+            [secrets]
+            stores = ["postgres"]
+            writer = "postgres"
+
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+        let secrets = toml::from_str::<Wrapper>(postgres_only)
+            .expect("parse postgres-only")
+            .secrets;
+        assert_eq!(secrets.stores, vec![CredentialBackend::Postgres]);
+        assert_eq!(secrets.writer, CredentialBackend::Postgres);
+    }
+
+    // Verifies a typo'd store or writer value fails parsing rather than
+    // silently dropping a backend from the chain.
+    #[test]
+    fn secrets_config_rejects_unknown_backend() {
+        #[derive(Deserialize)]
+        struct Wrapper {
+            #[expect(dead_code)]
+            secrets: SecretsConfig,
+        }
+
+        let base_kms = r#"
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+
+        let bad_store = format!("[secrets]\nstores = [\"postgrez\"]\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&bad_store).is_err());
+
+        // env/file are local overrides, not backend stores -- they belong in
+        // [credentials.*], not [secrets].stores, so they're rejected here.
+        let env_as_store = format!("[secrets]\nstores = [\"env\"]\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&env_as_store).is_err());
+
+        let bad_writer = format!("[secrets]\nwriter = \"valt\"\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&bad_writer).is_err());
+    }
+
     // Verifies that a misspelled optional key in [secrets] -- here
     // `import_fom` for `import_from` -- fails to parse instead of leaving
     // the import silently disabled. Without deny_unknown_fields, the typo'd