feat(config): add secret management for KMS config files#932
Conversation
Manuthor
left a comment
There was a problem hiding this comment.
Thanks for this!
Only a partial review.
Could you rebase please?
f7fd394 to
e03d7e0
Compare
|
We should also support our own KMS server for storing this kind of config files secrets and support another |
d3e4c7e to
6411945
Compare
8e5fcb6 to
98791ca
Compare
This reverts commit 44dd245.
…all workflows" This reverts commit 70f9979.
There was a problem hiding this comment.
Pull request overview
This PR adds startup-time secret management for KMS TOML configuration, including environment interpolation, optional secrets-file merging, and feature-gated external secret URI backends for Vault, AWS SSM, Azure Key Vault, and Cosmian KMS.
Changes:
- Adds config loading stages for env-var interpolation, secrets-file deep merge, and secret URI resolution.
- Introduces secret backend implementations and integration test scripts for Vault/AWS/Azure.
- Updates packaging/Nix hashes, CI matrix entries, and configuration templates/documentation.
Reviewed changes
Copilot reviewed 21 out of 25 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
crate/server/src/config/command_line/clap_config.rs |
Adds env interpolation, secrets-file loading/merging, and backend resolution during config load. |
crate/server/src/config/command_line/secret_backends.rs |
Adds Vault, AWS SSM, Azure KV, and Cosmian KMS secret URI backends plus tests. |
crate/server/src/config/command_line/mod.rs |
Registers the new secret backend module. |
crate/server/Cargo.toml |
Adds secret backend feature flags and dependencies; reorganizes manifest sections. |
Cargo.lock |
Updates dependency lockfile for the new/changed dependency graph. |
pkg/kms.toml |
Documents secrets file, env interpolation, and URI backend schemes. |
pkg/secrets.toml |
Adds a template for storing sensitive config values separately. |
.github/workflows/test_all.yml |
Adds secret backend test types and CI credentials wiring. |
.github/scripts/nix.sh |
Adds dispatch and environment handling for secret backend test scripts. |
.github/scripts/test/test_secret_vault.sh |
Adds Vault integration test script. |
.github/scripts/test/test_secret_aws.sh |
Adds AWS SSM integration test script. |
.github/scripts/test/test_secret_azure.sh |
Adds Azure Key Vault integration test script. |
shell.nix |
Adds optional AWS CLI and Docker client tooling for secret tests. |
nix/expected-hashes/server.vendor.static.sha256 |
Updates expected server static vendor hash. |
nix/expected-hashes/server.vendor.dynamic.sha256 |
Updates expected server dynamic vendor hash. |
nix/expected-hashes/cli.vendor.linux.sha256 |
Updates expected CLI Linux vendor hash. |
nix/expected-hashes/cli.vendor.dynamic.darwin.sha256 |
Updates expected CLI Darwin dynamic vendor hash. |
nix/expected-hashes/cli.vendor.static.darwin.sha256 |
Touches expected CLI Darwin static vendor hash. |
nix/expected-hashes/ui.vendor.fips.sha256 |
Updates expected UI FIPS vendor hash. |
nix/expected-hashes/ui.vendor.non-fips.sha256 |
Updates expected UI non-FIPS vendor hash. |
nix/expected-hashes/ui.pnpm.linux.sha256 |
Touches expected UI pnpm Linux hash. |
nix/expected-hashes/ui.pnpm.darwin.sha256 |
Touches expected UI pnpm Darwin hash. |
deny.toml |
Formatting-only changes. |
CHANGELOG/.gitignore |
Removes the placeholder file for the changelog directory. |
| /// | ||
| /// For each occurrence of `${VAR_NAME}`, the value of the environment variable `VAR_NAME` | ||
| /// is substituted. Returns an error message if a referenced variable is not set. | ||
| fn interpolate_env_vars(content: &str) -> Result<String, String> { |
There was a problem hiding this comment.
I do not understand the logic of the 2 new functions and their implementation seem to be fragile to future changes.
Moreover the existing function default_config_with_comments is unnecessary complicated. Could you resorb this complexity by solving this issue #951?
Closes #882
What
Add secret management support to KMS TOML configuration files so that sensitive values (passwords, tokens, API keys) are never stored in clear text.
Changes
Phase 1 — Local secret handling
${VAR_NAME}and${VAR_NAME:-default}syntax resolved at startup in all config valuessecrets_filekey in the main config orCOSMIAN_KMS_SECRETS_CONFenv var; deep-merged at startup before deserializationpkg/secrets.tomltemplate with documented examplesPhase 2 — External secret backends (optional feature flags)
URI schemes resolved at startup:
vault://<mount>/<path>[#<field>]secret-vaultaws-ssm://<region>/<param-path>secret-awsazure-kv://<vault>/secrets/<name>secret-azureEach backend spawns a dedicated OS thread with its own Tokio runtime to avoid nested-runtime panics at startup.
Testing