Implement project secrets (#2854)

* Implement secrets management API

* Test secrets management API

* test: add test cases for delete secrets endpoint

* Check secrets exist before deleting

* Add dstack secret CLI command

* Interpolate env with secrets

* Interpolate registry_auth with secrets

* Validate secrets

* Rebase migration

* Document secrets

Author: r4victor

docs/docs/concepts/secrets.md

@@ -0,0 +1,125 @@
+# Secrets
+
+Secrets allow centralized management of sensitive values such as API keys and credentials. They are project-scoped, managed by project admins, and can be referenced in run configurations to pass sensitive values to runs in a secure manner.
+
+!!! info "Secrets encryption"
+    By default, secrets are stored in plaintext in the DB.
+    Configure [server encryption](../guides/server-deployment.md#encryption) to store secrets encrypted.
+
+## Manage secrets
+
+### Set
+
+Use the `dstack secret set` command to create a new secret:
+
+<div class="termy">
+
+```shell
+$ dstack secret set my_secret some_secret_value
+OK
+```
+
+</div>
+
+The same command can be used to update an existing secret:
+
+<div class="termy">
+
+```shell
+$ dstack secret set my_secret another_secret_value
+OK
+```
+
+</div>
+
+### List
+
+Use the `dstack secret list` command to list all secrets set in a project:
+
+<div class="termy">
+
+```shell
+$ dstack secret
+ NAME       VALUE  
+ hf_token   ****** 
+ my_secret  ******
+
+```
+
+</div>
+
+### Get
+
+The `dstack secret list` does not show secret values. To see a secret value, use the `dstack secret get` command:
+
+<div class="termy">
+
+```shell
+$ dstack secret get my_secret
+ NAME       VALUE             
+ my_secret  some_secret_value 
+
+```
+
+</div>
+
+### Delete
+
+Secrets can be deleted using the `dstack secret delete` command:
+
+<div class="termy">
+
+```shell
+$ dstack secret delete my_secret
+Delete the secret my_secret? [y/n]: y
+OK
+```
+
+</div>
+
+## Use secrets
+
+You can use the `${{ secrets.<secret_name> }}` syntax to reference secrets in run configurations. Currently, secrets interpolation is supported in `env` and `registry_auth` properties.
+
+### `env`
+
+Suppose you need to pass a sensitive environment variable to a run such as `HF_TOKEN`. You'd first create a secret holding the environment variable value:
+
+<div class="termy">
+
+```shell
+$ dstack secret set hf_token {hf_token_value}
+OK
+```
+
+</div>
+
+and then reference the secret in `env`:
+
+<div editor-title=".dstack.yml"> 
+
+```yaml
+type: service
+env:
+  - HF_TOKEN=${{ secrets.hf_token }}
+commands:
+  ...
+```
+
+</div>
+
+### `registry_auth`
+
+If you need to pull a private Docker image, you can store registry credentials as secrets and reference them in `registry_auth`:
+
+<div editor-title=".dstack.yml"> 
+
+```yaml
+type: service
+image: nvcr.io/nim/deepseek-ai/deepseek-r1-distill-llama-8b
+registry_auth:
+  username: $oauthtoken
+  password: ${{ secrets.ngc_api_key }}
+```
+
+</div>

docs/docs/reference/cli/dstack/secret.md

@@ -0,0 +1,61 @@
+# dstack secret
+
+The `dstack secret` commands allow managing [Secrets](../../../concepts/secrets.md).
+
+## dstack secret set
+
+The `dstack secret set` command creates a new secret or updates an existing one.
+
+##### Usage
+
+<div class="termy">
+
+```shell
+$ dstack secret set --help
+#GENERATE#
+```
+
+</div>
+
+## dstack secret list
+
+The `dstack secret list` command lists all secrets set in a project.
+##### Usage
+
+<div class="termy">
+
+```shell
+$ dstack secret list --help
+#GENERATE#
+```
+
+</div>
+
+## dstack secret get
+
+The `dstack secret get` command show the value of a specified secret.
+##### Usage
+
+<div class="termy">
+
+```shell
+$ dstack secret get --help
+#GENERATE#
+```
+
+</div>
+
+## dstack secret delete
+
+The `dstack secret delete` command deletes the specified secret.
+
+##### Usage
+
+<div class="termy">
+
+```shell
+$ dstack secret delete --help
+#GENERATE#
+```
+
+</div>

mkdocs.yml

@@ -221,6 +221,7 @@ nav:
           - Fleets: docs/concepts/fleets.md
           - Volumes: docs/concepts/volumes.md
           - Repos: docs/concepts/repos.md
+          - Secrets: docs/concepts/secrets.md
           - Projects: docs/concepts/projects.md
           - Gateways: docs/concepts/gateways.md
       - Guides:
@@ -254,6 +255,7 @@ nav:
             - dstack offer: docs/reference/cli/dstack/offer.md
             - dstack volume: docs/reference/cli/dstack/volume.md
             - dstack gateway: docs/reference/cli/dstack/gateway.md
+            - dstack secret: docs/reference/cli/dstack/secret.md
         - API:
             - Python API: docs/reference/api/python/index.md
             - REST API: docs/reference/api/rest/index.md

src/dstack/_internal/cli/commands/secrets.py

@@ -0,0 +1,92 @@
+import argparse
+
+from dstack._internal.cli.commands import APIBaseCommand
+from dstack._internal.cli.services.completion import SecretNameCompleter
+from dstack._internal.cli.utils.common import (
+    confirm_ask,
+    console,
+)
+from dstack._internal.cli.utils.secrets import print_secrets_table
+
+
+class SecretCommand(APIBaseCommand):
+    NAME = "secret"
+    DESCRIPTION = "Manage secrets"
+
+    def _register(self):
+        super()._register()
+        self._parser.set_defaults(subfunc=self._list)
+        subparsers = self._parser.add_subparsers(dest="action")
+
+        list_parser = subparsers.add_parser(
+            "list", help="List secrets", formatter_class=self._parser.formatter_class
+        )
+        list_parser.set_defaults(subfunc=self._list)
+
+        get_parser = subparsers.add_parser(
+            "get", help="Get secret value", formatter_class=self._parser.formatter_class
+        )
+        get_parser.add_argument(
+            "name",
+            help="The name of the secret",
+        ).completer = SecretNameCompleter()
+        get_parser.set_defaults(subfunc=self._get)
+
+        set_parser = subparsers.add_parser(
+            "set", help="Set secret", formatter_class=self._parser.formatter_class
+        )
+        set_parser.add_argument(
+            "name",
+            help="The name of the secret",
+        )
+        set_parser.add_argument(
+            "value",
+            help="The value of the secret",
+        )
+        set_parser.set_defaults(subfunc=self._set)
+
+        delete_parser = subparsers.add_parser(
+            "delete",
+            help="Delete secrets",
+            formatter_class=self._parser.formatter_class,
+        )
+        delete_parser.add_argument(
+            "name",
+            help="The name of the secret",
+        ).completer = SecretNameCompleter()
+        delete_parser.add_argument(
+            "-y", "--yes", help="Don't ask for confirmation", action="store_true"
+        )
+        delete_parser.set_defaults(subfunc=self._delete)
+
+    def _command(self, args: argparse.Namespace):
+        super()._command(args)
+        args.subfunc(args)
+
+    def _list(self, args: argparse.Namespace):
+        secrets = self.api.client.secrets.list(self.api.project)
+        print_secrets_table(secrets)
+
+    def _get(self, args: argparse.Namespace):
+        secret = self.api.client.secrets.get(self.api.project, name=args.name)
+        print_secrets_table([secret])
+
+    def _set(self, args: argparse.Namespace):
+        self.api.client.secrets.create_or_update(
+            self.api.project,
+            name=args.name,
+            value=args.value,
+        )
+        console.print("[grey58]OK[/]")
+
+    def _delete(self, args: argparse.Namespace):
+        if not args.yes and not confirm_ask(f"Delete the secret [code]{args.name}[/]?"):
+            console.print("\nExiting...")
+            return
+
+        with console.status("Deleting secret..."):
+            self.api.client.secrets.delete(
+                project_name=self.api.project,
+                names=[args.name],
+            )
+        console.print("[grey58]OK[/]")

src/dstack/_internal/cli/main.py

@@ -17,6 +17,7 @@
 from dstack._internal.cli.commands.offer import OfferCommand
 from dstack._internal.cli.commands.project import ProjectCommand
 from dstack._internal.cli.commands.ps import PsCommand
+from dstack._internal.cli.commands.secrets import SecretCommand
 from dstack._internal.cli.commands.server import ServerCommand
 from dstack._internal.cli.commands.stats import StatsCommand
 from dstack._internal.cli.commands.stop import StopCommand
@@ -72,6 +73,7 @@ def main():
     MetricsCommand.register(subparsers)
     ProjectCommand.register(subparsers)
     PsCommand.register(subparsers)
+    SecretCommand.register(subparsers)
     ServerCommand.register(subparsers)
     StatsCommand.register(subparsers)
     StopCommand.register(subparsers)

src/dstack/_internal/cli/services/completion.py

@@ -75,6 +75,11 @@ def fetch_resource_names(self, api: Client) -> Iterable[str]:
         return [r.name for r in api.client.gateways.list(api.project)]
 
 
+class SecretNameCompleter(BaseAPINameCompleter):
+    def fetch_resource_names(self, api: Client) -> Iterable[str]:
+        return [r.name for r in api.client.secrets.list(api.project)]
+
+
 class ProjectNameCompleter(BaseCompleter):
     """
     Completer for local project names.

src/dstack/_internal/cli/utils/secrets.py

@@ -0,0 +1,25 @@
+from typing import List
+
+from rich.table import Table
+
+from dstack._internal.cli.utils.common import add_row_from_dict, console
+from dstack._internal.core.models.secrets import Secret
+
+
+def print_secrets_table(secrets: List[Secret]) -> None:
+    console.print(get_secrets_table(secrets))
+    console.print()
+
+
+def get_secrets_table(secrets: List[Secret]) -> Table:
+    table = Table(box=None)
+    table.add_column("NAME", no_wrap=True)
+    table.add_column("VALUE")
+
+    for secret in secrets:
+        row = {
+            "NAME": secret.name,
+            "VALUE": secret.value or "*" * 6,
+        }
+        add_row_from_dict(table, row)
+    return table

src/dstack/_internal/core/models/secrets.py

@@ -1,9 +1,16 @@
+from typing import Optional
+from uuid import UUID
+
 from dstack._internal.core.models.common import CoreModel
 
 
 class Secret(CoreModel):
+    id: UUID
     name: str
-    value: str
+    value: Optional[str] = None
 
     def __str__(self) -> str:
-        return f'Secret(name="{self.name}", value={"*" * len(self.value)})'
+        displayed_value = "*"
+        if self.value is not None:
+            displayed_value = "*" * len(self.value)
+        return f'Secret(name="{self.name}", value={displayed_value})'