feat(secrets): configurable credential chain and writer

Turning on the Postgres secrets backend used to be all-or-nothing: setting
[secrets] dropped vault from the read chain and ran a mandatory import. Now
the backend stores are operator config, so a site can roll Postgres in
gradually -- Postgres in front of vault, vault as a safety net, writes
flipped over when ready, vault dropped once fully migrated.

Two new [secrets] fields drive it:

- stores -- the backend-store read order (postgres, vault), first match
  wins. The env/file local overrides are always tried first (when their
  [credentials.*] section is enabled); this list just orders the stores
  behind them. Defaults to ["vault"], today's behavior. The rollout is
  editing this list: ["vault"] -> ["postgres", "vault"] -> ["postgres"].
- writer -- vault (default) or postgres. Flip it to send new writes to the
  journal.

import_from stays fully independent -- importing from vault is orthogonal to
where reads and writes flow, and it is now gated visibly at the call site. A
small pure assemble_chain prepends the local overrides, orders the stores
from config, and picks the writer, so the wiring is unit-testable without a
database; run.rs just hands it the constructed backends. An empty or
duplicate stores list fails the boot, as does putting vault ahead of
postgres; a writer whose backend is not in stores is allowed but warns
about the read-after-write gap (e.g. a deliberate postgres shadow-write).

Tests added!

This supports #2811

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

Author: chet

crates/api-core/src/cfg/file.rs

@@ -730,9 +730,9 @@ pub struct CarbideConfig {
     #[serde(default)]
     pub tracing: TracingConfig,
 
-    /// Secrets backend configuration. When present, credentials live
-    /// encrypted in Postgres and vault leaves the credential chain
-    /// entirely; when absent, vault remains the credential store.
+    /// Secrets backend configuration. When present, the credential reader
+    /// chain and write target are operator-configured (defaulting to the same
+    /// env -> file -> vault behavior as when it is absent); see `SecretsConfig`.
     pub secrets: Option<SecretsConfig>,
 }
 
@@ -846,24 +846,25 @@ pub enum BgpLeafSessionPassword {
     SiteWide,
 }
 
-/// Configures the Postgres secrets backend. When this section is present,
-/// credentials live encrypted in Postgres and vault is not in the
-/// credential chain at all -- the one-time import either completes before
-/// the process serves traffic, or the process does not start. Vault keeps
-/// serving PKI certificates either way.
+/// Configures the Postgres secrets backend and how credentials flow. When
+/// this section is present the reader chain and the write target come from
+/// `stores` / `writer` below; their defaults keep today's behavior
+/// (env -> file -> vault, writes to vault), so adding `[secrets]` does not
+/// change credential routing on its own. An operator rolls Postgres in
+/// gradually by editing `stores` and flipping `writer`. Vault keeps serving
+/// PKI certificates regardless of the chain.
 ///
-/// Enabling this on an existing site has two prerequisites that live
-/// outside this process:
+/// Two prerequisites live outside this process and matter once writes move
+/// to Postgres (`writer = "postgres"`) or vault leaves `stores`:
 ///
 /// - Services that read credentials from vault through their own chains
-///   (`bmc-proxy`, `dsx-exchange-consumer`) keep reading vault and will
-///   not see anything carbide-api writes to Postgres afterwards. They must
-///   be migrated or fed another way before credentials here change.
-/// - During a rolling upgrade, replicas still running the vault config
-///   keep writing rotated credentials to vault, where they are stranded
-///   once the import has completed. Keep autonomous credential writers
-///   (site-explorer credential rotation) disabled until the whole fleet
-///   runs this config.
+///   (`bmc-proxy`, `dsx-exchange-consumer`) will not see anything carbide-api
+///   writes to Postgres. They must be migrated or fed another way before the
+///   credentials they read change.
+/// - During a rolling upgrade, replicas still on an older config keep writing
+///   rotated credentials to their own writer. Keep autonomous credential
+///   writers (site-explorer credential rotation) disabled until the whole
+///   fleet runs a consistent config.
 #[derive(Clone, Debug, Deserialize, Serialize)]
 #[serde(deny_unknown_fields)]
 pub struct SecretsConfig {
@@ -884,9 +885,36 @@ pub struct SecretsConfig {
     /// ```
     pub routing: std::collections::HashMap<String, String>,
 
+    /// The credential *store* read order, highest priority first (first match
+    /// wins). The local-override readers (env, file) are always tried ahead of
+    /// these, when their `[credentials.*]` section is enabled; this list only
+    /// orders the stores behind them. Defaults to `["vault"]` -- with the local
+    /// overrides, that is the pre-Postgres env -> file -> vault chain. Roll
+    /// Postgres in gradually by editing this list:
+    ///
+    /// 1. `["vault"]` -- Postgres configured but not yet read.
+    /// 2. `["postgres", "vault"]` -- Postgres in front, vault as the safety net
+    ///    for anything Postgres misses.
+    /// 3. `["postgres"]` -- fully migrated, vault dropped.
+    ///
+    /// Postgres must come before vault; an empty list, or a store listed twice,
+    /// fails the boot.
+    #[serde(default = "default_secret_stores")]
+    pub stores: Vec<StoreKind>,
+
+    /// Where new credential writes go. Defaults to `vault`; set to `postgres`
+    /// to send new writes to the journal. Independent of `stores`: e.g.
+    /// `writer = "postgres"` while `postgres` is not in `stores` (reads still
+    /// served by vault) is a valid shadow-write -- it confirms writes land
+    /// before reads start trusting Postgres -- and only logs a warning.
+    #[serde(default)]
+    pub writer: WriterKind,
+
     /// A source backend to import secrets from at startup. Unset means a
     /// fresh site with nothing to import; unsupported values fail config
-    /// parsing rather than silently skipping the import.
+    /// parsing rather than silently skipping the import. Independent of
+    /// `stores`/`writer` -- importing from vault is orthogonal to where
+    /// reads and writes flow.
     pub import_from: Option<ImportSource>,
 
     /// How to treat secrets that already exist in Postgres during import.
@@ -902,6 +930,36 @@ pub enum ImportSource {
     Vault,
 }
 
+/// A credential *store* backend, named in `[secrets].stores` to order the
+/// stores behind the always-first local overrides (env, file). First match in
+/// the chain wins (see `ChainedCredentialReader`).
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum StoreKind {
+    /// The Postgres secrets journal.
+    Postgres,
+    /// Vault/OpenBao KV.
+    Vault,
+}
+
+/// The single backend new credential writes go to.
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum WriterKind {
+    /// Writes go to Vault/OpenBao KV (today's behavior).
+    #[default]
+    Vault,
+    /// Writes go to the Postgres secrets journal.
+    Postgres,
+}
+
+/// The default backend-store order (just vault). With the always-first env/file
+/// local overrides, this is the pre-Postgres chain env -> file -> vault, so
+/// adding `[secrets]` changes nothing until an operator edits it.
+fn default_secret_stores() -> Vec<StoreKind> {
+    vec![StoreKind::Vault]
+}
+
 /// Configures the KMS backends that wrap DEKs. Several named providers can
 /// be defined: the active one wraps DEKs for new writes, and every provider
 /// answers unwraps for the kek_ids it has.
@@ -4156,6 +4214,11 @@ firmware_url = "https://firmware.example.com/fw-b.bin"
             secrets.import_approach,
             crate::secrets::ImportApproach::MissingOnly
         );
+
+        // stores/writer were omitted above, so they default to vault-only
+        // (env/file are prepended separately) writing to vault.
+        assert_eq!(secrets.stores, vec![StoreKind::Vault]);
+        assert_eq!(secrets.writer, WriterKind::Vault);
     }
 
     // Verifies that a typo'd import source fails config parsing instead of
@@ -4186,6 +4249,91 @@ firmware_url = "https://firmware.example.com/fw-b.bin"
         assert!(toml::from_str::<Wrapper>(toml_str).is_err());
     }
 
+    // Verifies the stores list and writer parse from their enum values --
+    // a mid-rollout config (Postgres in front of vault, writes to Postgres)
+    // and a fully-migrated one (vault dropped, writes to Postgres).
+    #[test]
+    fn secrets_config_parses_stores_and_writer() {
+        #[derive(Deserialize)]
+        struct Wrapper {
+            secrets: SecretsConfig,
+        }
+
+        let mid = r#"
+            [secrets]
+            stores = ["postgres", "vault"]
+            writer = "postgres"
+
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+        let secrets = toml::from_str::<Wrapper>(mid).expect("parse mid").secrets;
+        assert_eq!(secrets.stores, vec![StoreKind::Postgres, StoreKind::Vault]);
+        assert_eq!(secrets.writer, WriterKind::Postgres);
+
+        // Fully migrated: postgres-only reads, writes to postgres too. (The
+        // writer-defaults-to-vault case is covered by the deserialize test
+        // above, with vault still in stores -- pairing a postgres-only chain
+        // with a vault writer is the read-after-write gap run.rs warns about.)
+        let migrated = r#"
+            [secrets]
+            stores = ["postgres"]
+            writer = "postgres"
+
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+        let secrets = toml::from_str::<Wrapper>(migrated)
+            .expect("parse migrated")
+            .secrets;
+        assert_eq!(secrets.stores, vec![StoreKind::Postgres]);
+        assert_eq!(secrets.writer, WriterKind::Postgres);
+    }
+
+    // Verifies a typo'd reader or writer value fails parsing rather than
+    // silently dropping a backend from the chain.
+    #[test]
+    fn secrets_config_rejects_unknown_reader_or_writer_kind() {
+        #[derive(Deserialize)]
+        struct Wrapper {
+            #[expect(dead_code)]
+            secrets: SecretsConfig,
+        }
+
+        let base_kms = r#"
+            [secrets.kms]
+            active = "local"
+            [secrets.kms.providers.local]
+            type = "integrated"
+            keys.default-key = { env = "K" }
+            [secrets.routing]
+            "/" = "default-key"
+        "#;
+
+        let bad_reader = format!("[secrets]\nstores = [\"postgrez\"]\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&bad_reader).is_err());
+
+        // env/file are local overrides, not store readers -- they belong in
+        // [credentials.*], not [secrets].stores, so they're rejected here.
+        let env_as_reader = format!("[secrets]\nstores = [\"env\"]\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&env_as_reader).is_err());
+
+        let bad_writer = format!("[secrets]\nwriter = \"valt\"\n{base_kms}");
+        assert!(toml::from_str::<Wrapper>(&bad_writer).is_err());
+    }
+
     // Verifies that a misspelled optional key in [secrets] -- here
     // `import_fom` for `import_from` -- fails to parse instead of leaving
     // the import silently disabled. Without deny_unknown_fields, the typo'd