From 336cecbdef8651666be00a4dca93fbcd76fd5c5b Mon Sep 17 00:00:00 2001 From: Daniel Gerlag Date: Thu, 21 May 2026 16:16:59 -0700 Subject: [PATCH 1/2] secret stores Signed-off-by: Daniel Gerlag --- .../how-to-guides/configuration/_index.md | 9 + .../configure-drasi-server/_index.md | 49 +++ .../configure-secret-stores/_index.md | 314 ++++++++++++++++++ .../reference/configuration/_index.md | 82 ++++- 4 files changed, 451 insertions(+), 3 deletions(-) create mode 100644 docs/content/drasi-server/how-to-guides/configuration/configure-secret-stores/_index.md diff --git a/docs/content/drasi-server/how-to-guides/configuration/_index.md b/docs/content/drasi-server/how-to-guides/configuration/_index.md index e290a41f..29348a14 100644 --- a/docs/content/drasi-server/how-to-guides/configuration/_index.md +++ b/docs/content/drasi-server/how-to-guides/configuration/_index.md @@ -55,4 +55,13 @@ description: "Step-by-step instructions for configuring Drasi Server" + +
+
+
+

Configure Secret Stores

+

Resolve passwords and tokens from external secret stores

+
+
+ diff --git a/docs/content/drasi-server/how-to-guides/configuration/configure-drasi-server/_index.md b/docs/content/drasi-server/how-to-guides/configuration/configure-drasi-server/_index.md index db31a021..51c94b6f 100644 --- a/docs/content/drasi-server/how-to-guides/configuration/configure-drasi-server/_index.md +++ b/docs/content/drasi-server/how-to-guides/configuration/configure-drasi-server/_index.md @@ -22,6 +22,8 @@ related: url: "/drasi-server/how-to-guides/configuration/configure-reactions/" - title: "Configure Bootstrap Providers" url: "/drasi-server/how-to-guides/configuration/configure-bootstrap-providers/" + - title: "Configure Secret Stores" + url: "/drasi-server/how-to-guides/configuration/configure-secret-stores/" - title: "Install with Docker" url: "/drasi-server/how-to-guides/installation/install-with-docker/" reference: @@ -99,6 +101,11 @@ stateStore: kind: redb path: ./data/state.redb +# Secret store for resolving secret references (optional) +# secretStore: +# kind: file +# path: ./secrets.json + # Performance tuning (optional) # (if omitted, DrasiLib defaults are used) defaultPriorityQueueCapacity: 10000 @@ -146,6 +153,7 @@ Top-level settings in `server.yaml`: | `persistConfig` | boolean | `true` | Persist API changes back to the config file (if writable) | | `persistIndex` | boolean | `false` | Enable persistent indexes using RocksDB (stored under `./data//index`) | | `stateStore` | object | None | Persist plugin state across restarts (see below) | +| `secretStore` | object | None | Secret store provider for resolving secret references (see below) | | `defaultPriorityQueueCapacity` | integer | None | Default event queue capacity for queries/reactions (if set, overrides DrasiLib defaults) | | `defaultDispatchBufferCapacity` | integer | None | Default dispatch buffer capacity for sources/queries (if set, overrides DrasiLib defaults) | | `sources` | array | `[]` | Source plugin instances (see: Configure Sources) | @@ -189,6 +197,43 @@ stateStore: If `stateStore` is not configured, an in-memory store is used and plugin state is lost on restart. +## Secret Store Configuration + +The secret store enables you to keep sensitive values (passwords, API keys, tokens) out of your configuration file. Instead of embedding secrets as plaintext or relying solely on environment variables, you reference named secrets that are resolved at runtime from an external store. + +Configure a secret store by adding a `secretStore` field at the top level: + +```yaml +secretStore: + kind: file + path: ./secrets.json +``` + +Then use **secret envelopes** in source or reaction configuration fields: + +```yaml +sources: + - kind: postgres + id: my-db + password: + kind: Secret + name: DB_PASSWORD # resolved from the secret store at runtime +``` + +Three providers are available: + +| Provider | `kind` | Use case | +|----------|--------|----------| +| File | `file` | Development/testing — reads from a JSON file | +| OS Keyring | `keyring` | Local development — uses OS credential manager | +| Azure Key Vault | `azure-keyvault` | Production on Azure — resolves from Key Vault | + +For full provider configuration details, see [Configure Secret Stores]({{< relref "configure-secret-stores" >}}). + +{{% alert title="Bootstrap constraint" color="info" %}} +The `secretStore` configuration itself cannot use secret references (circular dependency). Use literal values or environment variables for the secret store's own fields. +{{% /alert %}} + ## Performance Tuning These settings control queue/buffer sizing in DrasiLib. They are most useful for high-throughput workloads or when you want to set consistent defaults across multiple queries/sources. @@ -547,6 +592,10 @@ DB_PASSWORD=secret123 For production, set environment variables through your deployment platform (Docker, systemd, etc.). +{{% alert title="Tip: Secret stores" color="info" %}} +For stronger secret management, consider using a [secret store]({{< relref "configure-secret-stores" >}}) instead of (or alongside) environment variables. Secret stores keep credentials in a dedicated vault and prevent them from appearing in environment variable dumps or process listings. +{{% /alert %}} + ### Separating Concerns with Multiple Config Files For complex deployments, consider organizing configs by environment: diff --git a/docs/content/drasi-server/how-to-guides/configuration/configure-secret-stores/_index.md b/docs/content/drasi-server/how-to-guides/configuration/configure-secret-stores/_index.md new file mode 100644 index 00000000..b6a9eb2a --- /dev/null +++ b/docs/content/drasi-server/how-to-guides/configuration/configure-secret-stores/_index.md @@ -0,0 +1,314 @@ +--- +type: "docs" +title: "Configure Secret Stores" +linkTitle: "Configure Secret Stores" +weight: 60 +description: "Resolve passwords and tokens from external secret stores instead of hardcoding them in configuration" +related: + concepts: + - title: "Sources" + url: "/concepts/sources/" + howto: + - title: "Configure Drasi Server" + url: "/drasi-server/how-to-guides/configuration/configure-drasi-server/" + - title: "Configure Sources" + url: "/drasi-server/how-to-guides/configuration/configure-sources/" + - title: "Configure Reactions" + url: "/drasi-server/how-to-guides/configuration/configure-reactions/" + reference: + - title: "Configuration Reference" + url: "/drasi-server/reference/configuration/" +--- + +Secret stores let you keep sensitive values (passwords, API keys, tokens) out of your configuration files. Instead of providing a literal value or relying solely on environment variables, you reference a named secret that is resolved at runtime from an external store. + +## Why use a secret store? + +| Approach | Trade-offs | +|----------|-----------| +| Hardcoded values | Secrets in plaintext on disk — risky if config is committed to version control | +| Environment variables (`${VAR}`) | Better, but still visible in process listings and `.env` files | +| **Secret store** | Secrets stay in a dedicated vault; config files contain only references | + +Secret stores are especially useful in production environments where credentials should never appear in plaintext configuration files or environment variable dumps. + +## How it works + +1. You configure a **secret store provider** at the top level of your server config (or per-instance in multi-instance mode). +2. In any Source or Reaction configuration field that would normally take a string value, you use a **Secret envelope** instead. +3. At startup, Drasi Server resolves each secret reference by calling the configured secret store provider. + +### Secret envelope syntax + +Anywhere a scalar string value is expected, you can substitute a secret envelope: + +```yaml +# Instead of a literal value: +password: my-secret-password + +# Use a secret reference: +password: + kind: Secret + name: DB_PASSWORD +``` + +The `name` field identifies the secret in the configured store. Its meaning depends on the provider: +- **File provider:** the JSON key in the secrets file +- **Keyring provider:** the entry name in the OS credential store +- **Azure Key Vault provider:** the Key Vault secret name + +## Configuring a secret store + +Add a `secretStore` field at the top level of your server configuration: + +```yaml +host: 0.0.0.0 +port: 8080 + +secretStore: + kind: file + path: ./secrets.json + +sources: + - kind: postgres + id: my-db + host: localhost + password: + kind: Secret + name: DB_PASSWORD + # ... +``` + +In multi-instance mode, `secretStore` can be set per instance: + +```yaml +instances: + - id: production + secretStore: + kind: azure-keyvault + vaultUrl: https://my-vault.vault.azure.net/ + authMethod: managed_identity + sources: + # ... +``` + +## Available providers + +### File + +Reads secrets from a flat JSON file on disk. Best for **development and testing**. + +```yaml +secretStore: + kind: file + path: ./secrets.json +``` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `kind` | string | Yes | Must be `file` | +| `path` | string | Yes | Path to a JSON file containing key-value secret pairs | + +The secrets file is a simple JSON object mapping secret names to string values: + +```json +{ + "DB_PASSWORD": "my-secret-password", + "API_KEY": "sk-abc123" +} +``` + +### Keyring (OS credential store) + +Uses the operating system's native credential manager. Best for **local development** without files on disk. + +- **macOS:** Keychain +- **Linux:** Secret Service (GNOME Keyring / KDE Wallet) +- **Windows:** Credential Manager + +```yaml +secretStore: + kind: keyring +``` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `kind` | string | Yes | Must be `keyring` | + +Secret names map directly to keyring entry names. Store secrets using your OS tools: + +```bash +# macOS example +security add-generic-password -a drasi -s DB_PASSWORD -w "my-secret" + +# Linux example (using secret-tool) +secret-tool store --label="DB_PASSWORD" service drasi username DB_PASSWORD +``` + +### Azure Key Vault + +Resolves secrets from Azure Key Vault using Azure Identity credentials. Best for **production on Azure**. + +```yaml +secretStore: + kind: azure-keyvault + vaultUrl: https://my-vault.vault.azure.net/ + authMethod: developer_tools +``` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `kind` | string | Yes | Must be `azure-keyvault` | +| `vaultUrl` | string | Yes | Full URL to your Azure Key Vault (e.g., `https://my-vault.vault.azure.net/`) | +| `authMethod` | string | Yes | Authentication method (see table below) | +| `clientId` | string | Conditional | Required for `managed_identity_user_assigned` and `client_secret` | +| `tenantId` | string | Conditional | Required for `client_secret` | +| `clientSecret` | string | Conditional | Required for `client_secret` | + +#### Authentication methods + +| `authMethod` | Use case | Requirements | +|---|---|---| +| `developer_tools` | Local development | `az login` session, VS Code Azure extension, or IntelliJ Azure toolkit | +| `managed_identity` | Azure VMs, App Service, ACI | System-assigned managed identity attached to compute resource | +| `managed_identity_user_assigned` | Shared identity across resources | `clientId` field with the user-assigned identity's client ID | +| `workload_identity` | AKS with federated identity | `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_FEDERATED_TOKEN_FILE` env vars (set by AKS) | +| `client_secret` | Service principals (CI/CD, non-Azure hosts) | `tenantId`, `clientId`, `clientSecret` fields | + +#### Azure Key Vault example + +```yaml +secretStore: + kind: azure-keyvault + vaultUrl: https://drasi-prod-kv.vault.azure.net/ + authMethod: managed_identity + +sources: + - kind: postgres + id: orders-db + host: orders-db.postgres.database.azure.com + user: drasi_user + password: + kind: Secret + name: ORDERS-DB-PASSWORD # Key Vault secret name (hyphens, not underscores) + database: orders + # ... +``` + +{{% alert title="Key Vault naming" color="info" %}} +Azure Key Vault secret names can only contain alphanumeric characters and hyphens. Use hyphens instead of underscores (e.g., `DB-PASSWORD` not `DB_PASSWORD`). +{{% /alert %}} + +## Bootstrap constraint + +A secret store's own configuration **cannot** use secret references. This avoids a circular dependency — the secret store must be initialized before it can resolve secrets for other components. + +Use literal values or environment variables for the secret store's own config: + +```yaml +# ✅ Correct — env var for the secret store's own config +secretStore: + kind: azure-keyvault + vaultUrl: ${VAULT_URL} + authMethod: client_secret + tenantId: ${AZURE_TENANT_ID} + clientId: ${AZURE_CLIENT_ID} + clientSecret: ${AZURE_CLIENT_SECRET} + +# ❌ Incorrect — cannot use Secret envelope in secretStore config +secretStore: + kind: azure-keyvault + vaultUrl: https://my-vault.vault.azure.net/ + authMethod: client_secret + clientSecret: + kind: Secret # This won't work! + name: my-client-secret +``` + +## Complete example + +This example shows a PostgreSQL source with the file-based secret store resolving the database password: + +```yaml +id: postgres-secrets-demo +host: 0.0.0.0 +port: 8080 +logLevel: info + +# Secret store: read secrets from a JSON file +secretStore: + kind: file + path: ./secrets.json + +# PostgreSQL source with secret reference for password +sources: + - kind: postgres + id: pg-sensors + autoStart: true + host: localhost + port: 5432 + user: postgres + password: + kind: Secret + name: DB_PASSWORD + database: drasi_demo + slotName: drasi_slot + publicationName: drasi_pub + tables: + - sensors + bootstrapProvider: + kind: postgres + +# Query +queries: + - id: high-temp + query: | + MATCH (s:Sensor) + WHERE s.temperature > 75 + RETURN s.id, s.name, s.location, s.temperature + queryLanguage: Cypher + sources: + - sourceId: pg-sensors + autoStart: true + +# Reaction +reactions: + - kind: log + id: log-temps + queries: [high-temp] + autoStart: true +``` + +With `secrets.json`: + +```json +{ + "DB_PASSWORD": "Drasi@Pass123" +} +``` + +## Secret store vs environment variables + +You can mix both approaches. Environment variables still work for non-sensitive configuration, and secret stores handle credentials: + +```yaml +sources: + - kind: postgres + id: orders-db + host: ${DB_HOST:-localhost} # env var for host (not sensitive) + port: ${DB_PORT:-5432} # env var for port (not sensitive) + user: ${DB_USER:-drasi} # env var for user (low sensitivity) + password: + kind: Secret + name: DB_PASSWORD # secret store for password (high sensitivity) + database: ${DB_NAME:-orders} +``` + +## Troubleshooting + +| Error | Cause | Solution | +|-------|-------|----------| +| Secret store plugin not found | Plugin `.so`/`.dylib` not in `plugins/` directory | Build the secret store plugin and copy it to the server's plugin directory | +| Secret 'X' not found | Named secret doesn't exist in the store | Verify the secret name matches exactly (case-sensitive) | +| Azure auth failed | Invalid credentials or insufficient permissions | Check auth method config; ensure identity has "Key Vault Secrets User" role | +| Circular dependency error | Secret envelope used in `secretStore` config | Use literal values or env vars for the secret store's own configuration | diff --git a/docs/content/drasi-server/reference/configuration/_index.md b/docs/content/drasi-server/reference/configuration/_index.md index 9c2a58c4..8f5eb4ab 100644 --- a/docs/content/drasi-server/reference/configuration/_index.md +++ b/docs/content/drasi-server/reference/configuration/_index.md @@ -48,6 +48,7 @@ It does **not** document the per-component configuration for Sources, Queries, R | `persistConfig` | boolean | `true` | Save API changes to config file | | `persistIndex` | boolean | `false` | Use persistent RocksDB indexes | | `stateStore` | object | None | Plugin state persistence | +| `secretStore` | object | None | Secret store provider for resolving secret references | | `defaultPriorityQueueCapacity` | integer | None | Default queue size | | `defaultDispatchBufferCapacity` | integer | None | Default buffer size | | `sources` | array | `[]` | Source configurations | @@ -178,6 +179,7 @@ persistConfig: true # Single-instance mode instance settings (used when instances: [] is empty) persistIndex: false stateStore: null +secretStore: null defaultPriorityQueueCapacity: null defaultDispatchBufferCapacity: null @@ -216,6 +218,7 @@ When `instances` is empty (the default), Drasi Server runs **one** DrasiLib inst |---|---:|---:|---| | `persistIndex` | boolean | `false` | Enable persistent query indexing using RocksDB (stored under `./data//index`). | | `stateStore` | object | none | Optional state store for plugin runtime state persistence (see below). | +| `secretStore` | object | none | Optional secret store provider for resolving `Secret` envelope references in source/reaction configs (see below). | | `defaultPriorityQueueCapacity` | integer | none | If set, overrides the DrasiLib default priority queue capacity for queries/reactions. | | `defaultDispatchBufferCapacity` | integer | none | If set, overrides the DrasiLib default dispatch buffer capacity for sources/queries. | | `sources` | array | `[]` | Source plugin instances (see: Configure Sources). | @@ -241,13 +244,85 @@ stateStore: | `kind` | string | Yes | Must be `redb`. | | `path` | string | Yes | Path to the REDB file. Supports env interpolation (e.g. `${STATE_STORE_PATH:-./data/state.redb}`). | +## Secret store configuration + +The secret store allows source and reaction configurations to reference named secrets instead of embedding sensitive values directly. Secrets are resolved at runtime when plugins are initialized. + +If `secretStore` is omitted, secret envelope references (`kind: Secret`) cannot be used in component configurations. + +### Secret envelope syntax + +In any source or reaction config field that accepts a string, you can use a secret reference: + +```yaml +# Literal value +password: my-password + +# Secret reference (resolved at runtime) +password: + kind: Secret + name: DB_PASSWORD +``` + +### file (JSON file) + +Reads secrets from a flat JSON file. Best for development and testing. + +```yaml +secretStore: + kind: file + path: ./secrets.json +``` + +| Field | Type | Required | Description | +|---|---:|:---:|---| +| `kind` | string | Yes | Must be `file`. | +| `path` | string | Yes | Path to a JSON file with key-value secret pairs. Supports env interpolation. | + +### keyring (OS credential store) + +Uses the operating system's native credential manager (macOS Keychain, Linux Secret Service, Windows Credential Manager). + +```yaml +secretStore: + kind: keyring +``` + +| Field | Type | Required | Description | +|---|---:|:---:|---| +| `kind` | string | Yes | Must be `keyring`. | + +### azure-keyvault (Azure Key Vault) + +Resolves secrets from Azure Key Vault using Azure Identity credentials. + +```yaml +secretStore: + kind: azure-keyvault + vaultUrl: https://my-vault.vault.azure.net/ + authMethod: managed_identity +``` + +| Field | Type | Required | Description | +|---|---:|:---:|---| +| `kind` | string | Yes | Must be `azure-keyvault`. | +| `vaultUrl` | string | Yes | Full URL to the Azure Key Vault. | +| `authMethod` | string | Yes | One of: `developer_tools`, `managed_identity`, `managed_identity_user_assigned`, `workload_identity`, `client_secret`. | +| `clientId` | string | Conditional | Required for `managed_identity_user_assigned` and `client_secret`. | +| `tenantId` | string | Conditional | Required for `client_secret`. | +| `clientSecret` | string | Conditional | Required for `client_secret`. | + +{{% alert title="Bootstrap constraint" color="warning" %}} +The `secretStore` configuration itself **cannot** use secret envelope references (circular dependency). Use literal values or environment variables for the secret store's own fields. +{{% /alert %}} + ## Multi-instance mode (instances) When `instances` is **non-empty**, Drasi Server runs **one DrasiLib instance per entry** and uses **per-instance** component lists and instance settings. -{{< alert title="Important" color="warning" >}} -When `instances` is set, the following **top-level** fields are ignored for runtime behavior: `persistIndex`, `stateStore`, `defaultPriorityQueueCapacity`, `defaultDispatchBufferCapacity`, `sources`, `queries`, `reactions`. -{{< /alert >}} +{{% alert title="Important" color="warning" %}} +When `instances` is set, the following **top-level** fields are ignored for runtime behavior: `persistIndex`, `stateStore`, `secretStore`, `defaultPriorityQueueCapacity`, `defaultDispatchBufferCapacity`, `sources`, `queries`, `reactions`. +{{% /alert %}} ### Instance fields @@ -256,6 +331,7 @@ When `instances` is set, the following **top-level** fields are ignored for runt | `id` | string | Auto-generated UUID | Unique instance id (used in API routes and for persistence paths). | | `persistIndex` | boolean | `false` | Enable persistent indexing for this instance. | | `stateStore` | object | none | Optional per-instance state store. | +| `secretStore` | object | none | Optional per-instance secret store provider. | | `defaultPriorityQueueCapacity` | integer | none | Optional per-instance default. | | `defaultDispatchBufferCapacity` | integer | none | Optional per-instance default. | | `sources` | array | `[]` | Sources for this instance. | From fba633b575629ccbe68075e3ca41e3d5f7bc2d91 Mon Sep 17 00:00:00 2001 From: Daniel Gerlag Date: Thu, 21 May 2026 16:21:06 -0700 Subject: [PATCH 2/2] fix: add spelling dictionary entries for secret store docs Add words used in secret store documentation to the custom spellcheck dictionary: ACI, Hardcoded, IntelliJ, KDE, Keychain, Keyring, keyring, VMs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/config/en-custom.txt | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.github/config/en-custom.txt b/.github/config/en-custom.txt index c5a43319..56a204ae 100644 --- a/.github/config/en-custom.txt +++ b/.github/config/en-custom.txt @@ -1642,3 +1642,11 @@ Shopify ns Handlebars Stripe +ACI +Hardcoded +IntelliJ +KDE +Keychain +Keyring +keyring +VMs