feat(secrets): configurable credential chain and writer

Turning on the Postgres secrets backend used to be all-or-nothing: setting
[secrets] dropped vault from the read chain and ran a mandatory import. Now
the backend stores and the write target are operator config, so a site can
read from postgres, vault, or both -- in whatever order it wants -- and
choose where new writes land.

Two new [secrets] fields drive it, both naming a CredentialBackend
(postgres or vault):

- stores -- the backend read order, first match wins. The env/file local
  overrides are always tried first (when their [credentials.*] section is
  enabled); this list just orders the backends behind them. Defaults to
  ["vault"], today's behavior. Order is the operator's choice; for example,
  to roll Postgres in gradually: ["vault"] -> ["postgres", "vault"] ->
  ["postgres"].
- writer -- vault (default) or postgres. Flip it to send new writes to the
  journal.

import_from stays fully independent -- importing from vault is orthogonal to
where reads and writes flow, and it is now gated visibly at the call site.
An empty or duplicate stores list fails the boot. Store order is the
operator's choice; when the writer's backend isn't the highest-priority
store, a write can be shadowed on read for any path a higher-priority store
also holds, so that warns -- a deliberate shadow-write stays valid, it is
not rejected.

Tests added!

This supports #2811

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

Author: chet

crates/api-core/src/cfg/file.rs

@@ -730,9 +730,9 @@ pub struct CarbideConfig {
     #[serde(default)]
     pub tracing: TracingConfig,
 
-    /// Secrets backend configuration. When present, credentials live
-    /// encrypted in Postgres and vault leaves the credential chain
-    /// entirely; when absent, vault remains the credential store.
+    /// Secrets backend configuration. When present, the credential reader
+    /// chain and write target are operator-configured (defaulting to the same
+    /// env -> file -> vault behavior as when it is absent); see `SecretsConfig`.
     pub secrets: Option<SecretsConfig>,
 }
 
@@ -846,24 +846,26 @@ pub enum BgpLeafSessionPassword {
     SiteWide,
 }
 
-/// Configures the Postgres secrets backend. When this section is present,
-/// credentials live encrypted in Postgres and vault is not in the
-/// credential chain at all -- the one-time import either completes before
-/// the process serves traffic, or the process does not start. Vault keeps
-/// serving PKI certificates either way.
+/// Configures the Postgres secrets backend and how credentials flow. When
+/// this section is present the reader chain and the write target come from
+/// `stores` / `writer` below; their defaults keep today's behavior
+/// (env -> file -> vault, writes to vault), so adding `[secrets]` does not
+/// change credential routing on its own. Operators choose which backend
+/// stores to read, in what order, and which store takes writes, by editing
+/// `stores` and `writer`. Vault keeps serving PKI certificates regardless of
+/// the chain.
 ///
-/// Enabling this on an existing site has two prerequisites that live
-/// outside this process:
+/// Two prerequisites live outside this process and matter once writes move
+/// to Postgres (`writer = "postgres"`) or vault leaves `stores`:
 ///
 /// - Services that read credentials from vault through their own chains
-///   (`bmc-proxy`, `dsx-exchange-consumer`) keep reading vault and will
-///   not see anything carbide-api writes to Postgres afterwards. They must
-///   be migrated or fed another way before credentials here change.
-/// - During a rolling upgrade, replicas still running the vault config
-///   keep writing rotated credentials to vault, where they are stranded
-///   once the import has completed. Keep autonomous credential writers
-///   (site-explorer credential rotation) disabled until the whole fleet
-///   runs this config.
+///   (`bmc-proxy`, `dsx-exchange-consumer`) will not see anything carbide-api
+///   writes to Postgres. They must be pointed at the same store, or fed
+///   another way, before the credentials they read change.
+/// - During a rolling upgrade, replicas still on an older config keep writing
+///   rotated credentials to their own writer. Keep autonomous credential
+///   writers (site-explorer credential rotation) disabled until the whole
+///   fleet runs a consistent config.
 #[derive(Clone, Debug, Deserialize, Serialize)]
 #[serde(deny_unknown_fields)]
 pub struct SecretsConfig {
@@ -884,9 +886,37 @@ pub struct SecretsConfig {
     /// ```
     pub routing: std::collections::HashMap<String, String>,
 
+    /// The credential *store* read order, highest priority first (first match
+    /// wins). The local-override readers (env, file) are always tried ahead of
+    /// these, when their `[credentials.*]` section is enabled; this list only
+    /// orders the stores behind them. Order is the operator's choice -- list
+    /// the stores you want, in the priority you want. Defaults to `["vault"]`
+    /// -- with the local overrides, that is the env -> file -> vault chain.
+    ///
+    /// For example, to roll Postgres in gradually, walk this list:
+    ///
+    /// 1. `["vault"]` -- Postgres configured but not yet read.
+    /// 2. `["postgres", "vault"]` -- Postgres in front, vault as the safety net
+    ///    for anything Postgres misses.
+    /// 3. `["postgres"]` -- vault no longer read.
+    ///
+    /// An empty list, or a store named twice, fails the boot.
+    #[serde(default = "default_secret_stores")]
+    pub stores: Vec<CredentialBackend>,
+
+    /// Where new credential writes go. Defaults to `vault`; set to `postgres`
+    /// to send new writes to the journal. Independent of `stores`: e.g.
+    /// `writer = "postgres"` while `postgres` is not in `stores` (reads still
+    /// served by vault) is a valid shadow-write -- it confirms writes land
+    /// before reads start trusting Postgres -- and only logs a warning.
+    #[serde(default)]
+    pub writer: CredentialBackend,
+
     /// A source backend to import secrets from at startup. Unset means a
     /// fresh site with nothing to import; unsupported values fail config
-    /// parsing rather than silently skipping the import.
+    /// parsing rather than silently skipping the import. Independent of
+    /// `stores`/`writer` -- importing from vault is orthogonal to where
+    /// reads and writes flow.
     pub import_from: Option<ImportSource>,
 
     /// How to treat secrets that already exist in Postgres during import.
@@ -902,6 +932,27 @@ pub enum ImportSource {
     Vault,
 }
 
+/// A credential backend -- postgres or vault. Listed in `[secrets].stores` to
+/// order the backends behind the always-first local overrides (env, file;
+/// first match wins, see `ChainedCredentialReader`), and named by
+/// `[secrets].writer` to choose where new writes go.
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum CredentialBackend {
+    /// The Postgres secrets journal.
+    Postgres,
+    /// Vault/OpenBao KV. The default write target (today's behavior).
+    #[default]
+    Vault,
+}
+
+/// The default backend-store order (just vault). With the always-first env/file
+/// local overrides, this is the env -> file -> vault chain, so adding
+/// `[secrets]` changes nothing until an operator edits it.
+fn default_secret_stores() -> Vec<CredentialBackend> {
+    vec![CredentialBackend::Vault]
+}
+
 /// Configures the KMS backends that wrap DEKs. Several named providers can
 /// be defined: the active one wraps DEKs for new writes, and every provider
 /// answers unwraps for the kek_ids it has.
@@ -4156,6 +4207,11 @@ firmware_url = "https://firmware.example.com/fw-b.bin"
             secrets.import_approach,
             crate::secrets::ImportApproach::MissingOnly
         );
+
+        // stores/writer were omitted above, so they default to vault-only
+        // (env/file are prepended separately) writing to vault.
+        assert_eq!(secrets.stores, vec![CredentialBackend::Vault]);
+        assert_eq!(secrets.writer, CredentialBackend::Vault);
     }
 
     // Verifies that a typo'd import source fails config parsing instead of
@@ -4186,6 +4242,96 @@ firmware_url = "https://firmware.example.com/fw-b.bin"
         assert!(toml::from_str::<Wrapper>(toml_str).is_err());
     }
 
+    // Verifies the stores list and writer parse from their enum values --
+    // one with Postgres in front of vault (writes to Postgres) and a
+    // postgres-only one (vault not read, writes to Postgres).
+    #[test]
+    fn secrets_config_parses_stores_and_writer() {
+        #[derive(Deserialize)]
+        struct Wrapper {
+            secrets: SecretsConfig,
+        }
+
+        let pg_first = r#"
+            [secrets]
+            stores = ["postgres", "vault"]
+            writer = "postgres"
+
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+        let secrets = toml::from_str::<Wrapper>(pg_first)
+            .expect("parse pg-first")
+            .secrets;
+        assert_eq!(
+            secrets.stores,
+            vec![CredentialBackend::Postgres, CredentialBackend::Vault]
+        );
+        assert_eq!(secrets.writer, CredentialBackend::Postgres);
+
+        // Postgres-only reads, writes to postgres too. (The
+        // writer-defaults-to-vault case is covered by the deserialize test
+        // above, with vault still in stores -- pairing a postgres-only chain
+        // with a vault writer is the read-after-write gap run.rs warns about.)
+        let postgres_only = r#"
+            [secrets]
+            stores = ["postgres"]
+            writer = "postgres"
+
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+        let secrets = toml::from_str::<Wrapper>(postgres_only)
+            .expect("parse postgres-only")
+            .secrets;
+        assert_eq!(secrets.stores, vec![CredentialBackend::Postgres]);
+        assert_eq!(secrets.writer, CredentialBackend::Postgres);
+    }
+
+    // Verifies a typo'd store or writer value fails parsing rather than
+    // silently dropping a backend from the chain.
+    #[test]
+    fn secrets_config_rejects_unknown_backend() {
+        #[derive(Deserialize)]
+        struct Wrapper {
+            #[expect(dead_code)]
+            secrets: SecretsConfig,
+        }
+
+        let base_kms = r#"
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+
+        let bad_store = format!("[secrets]\nstores = [\"postgrez\"]\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&bad_store).is_err());
+
+        // env/file are local overrides, not backend stores -- they belong in
+        // [credentials.*], not [secrets].stores, so they're rejected here.
+        let env_as_store = format!("[secrets]\nstores = [\"env\"]\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&env_as_store).is_err());
+
+        let bad_writer = format!("[secrets]\nwriter = \"valt\"\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&bad_writer).is_err());
+    }
+
     // Verifies that a misspelled optional key in [secrets] -- here
     // `import_fom` for `import_from` -- fails to parse instead of leaving
     // the import silently disabled. Without deny_unknown_fields, the typo'd