|
| 1 | +# Bug: Passphrase-Protected SSH Key Silently Fails During Automated Deployment |
| 2 | + |
| 3 | +**Issue**: #411 |
| 4 | +**Parent Epic**: None |
| 5 | +**Related**: #405 - Deploy Hetzner Demo Tracker and Document the Process |
| 6 | + |
| 7 | +## Overview |
| 8 | + |
| 9 | +When a user configures a passphrase-protected SSH private key in `ssh_credentials`, the |
| 10 | +deployer fails silently during the `provision` step with a misleading |
| 11 | +`Permission denied (publickey,password)` error. The root cause — that the key is |
| 12 | +encrypted and cannot be decrypted without a passphrase in an unattended environment — |
| 13 | +is never surfaced to the user. |
| 14 | + |
| 15 | +This bug was triggered during the Hetzner demo deployment (#405) where a fresh |
| 16 | +deployment key was created with a passphrase for security. In an interactive terminal |
| 17 | +session the OS SSH agent transparently decrypts the key, so the error only appears when |
| 18 | +running the deployer in an automated context (Docker container, CI/CD pipeline) where no |
| 19 | +agent is available. |
| 20 | + |
| 21 | +There is no code fix required — passphrase-protected keys are a valid configuration for |
| 22 | +some workflows (e.g. with SSH agent forwarding into the container). The two actions |
| 23 | +needed are: |
| 24 | + |
| 25 | +1. **Add an early warning** in `create environment` when the private key file is detected |
| 26 | + as passphrase-protected, so users can make an informed decision before reaching the |
| 27 | + `provision` step. |
| 28 | +2. **Add documentation** covering SSH key requirements and the three supported workflows. |
| 29 | + |
| 30 | +## Goals |
| 31 | + |
| 32 | +- [ ] Detect passphrase-protected private keys during `create environment` and emit a |
| 33 | + user-visible warning (not an error — the choice belongs to the user) |
| 34 | +- [ ] Add a documentation section to the user guide on SSH key handling, covering key |
| 35 | + requirements and all supported workflows |
| 36 | +- [ ] Update the Hetzner provider guide to call out the passphrase requirement for |
| 37 | + Docker-based deployments |
| 38 | + |
| 39 | +## Specifications |
| 40 | + |
| 41 | +### Root Cause |
| 42 | + |
| 43 | +SSH private keys can be stored in two formats: |
| 44 | + |
| 45 | +- **Unencrypted**: the key material is in plaintext in the PEM file. |
| 46 | +- **Encrypted (passphrase-protected)**: the key material is encrypted; decryption |
| 47 | + requires the passphrase, an SSH agent holding the unlocked key, or a TTY for |
| 48 | + interactive prompting. |
| 49 | + |
| 50 | +The deployer invokes the system `ssh` binary for connectivity probes and remote |
| 51 | +commands. When running inside a Docker container, there is no SSH agent socket and no |
| 52 | +TTY. If the key file is encrypted, `ssh` cannot authenticate and every attempt returns |
| 53 | +`Permission denied (publickey,password)`. This is indistinguishable from a wrong key |
| 54 | +or an unconfigured `authorized_keys` file — the log output reveals nothing about the |
| 55 | +passphrase being the cause. |
| 56 | + |
| 57 | +An encrypted OpenSSH private key file contains `ENCRYPTED` in its PEM header: |
| 58 | + |
| 59 | +```text |
| 60 | +-----BEGIN OPENSSH PRIVATE KEY----- ← unencrypted |
| 61 | +-----BEGIN OPENSSH PRIVATE KEY----- ← also unencrypted (need to read body) |
| 62 | +``` |
| 63 | + |
| 64 | +The reliable detection approach is to read the first line of the PEM file: |
| 65 | + |
| 66 | +- RSA/EC legacy PEM format: encrypted files contain `ENCRYPTED` in the header: |
| 67 | + `-----BEGIN ENCRYPTED PRIVATE KEY-----` or `Proc-Type: 4,ENCRYPTED` |
| 68 | +- OpenSSH format: the body begins with `bcrypt` if passphrase-protected; the header |
| 69 | + alone is not sufficient — the first few bytes of the decoded body must be checked. |
| 70 | + |
| 71 | +The simplest robust approach in Rust is to read the raw bytes of the key file and check: |
| 72 | + |
| 73 | +- For legacy PEM: `ENCRYPTED` appears in the header |
| 74 | +- For OpenSSH format: after the base64-decoded body, the string `bcrypt` appears near |
| 75 | + the start (OpenSSH uses bcrypt KDF for passphrase derivation) |
| 76 | + |
| 77 | +The check only needs to be a best-effort heuristic — it is used to emit a warning, not |
| 78 | +to block the user. A false negative (missing the warning) is acceptable; a false |
| 79 | +positive (warning for an unencrypted key) would be confusing and should be avoided. |
| 80 | + |
| 81 | +### Warning Behavior |
| 82 | + |
| 83 | +The warning should be emitted during the `create environment` command, after the |
| 84 | +configuration is loaded and the private key path is resolved but before the environment |
| 85 | +state is persisted. It must: |
| 86 | + |
| 87 | +- Be non-blocking — the environment is still created normally. |
| 88 | +- Be clearly labelled as a warning, not an error. |
| 89 | +- Explain the consequence (automated runs without an SSH agent will fail). |
| 90 | +- Describe all three resolution options (see below). |
| 91 | + |
| 92 | +Example warning text: |
| 93 | + |
| 94 | +```text |
| 95 | +⚠ Warning: SSH private key appears to be passphrase-protected. |
| 96 | + Key: /home/deployer/.ssh/torrust_tracker_deployer_ed25519 |
| 97 | +
|
| 98 | + Automated deployment (e.g. Docker, CI/CD) requires an SSH key that can be used |
| 99 | + without interactive input. A passphrase-protected key will cause the `provision` |
| 100 | + step to fail with "Permission denied" unless one of the following is arranged: |
| 101 | +
|
| 102 | + Option 1 — Remove the passphrase (recommended for dedicated deployment keys): |
| 103 | + ssh-keygen -p -f /path/to/your/private_key |
| 104 | +
|
| 105 | + Option 2 — Forward your SSH agent socket into the Docker container: |
| 106 | + docker run ... -v "$SSH_AUTH_SOCK:/tmp/ssh-agent.sock" \ |
| 107 | + -e SSH_AUTH_SOCK=/tmp/ssh-agent.sock ... |
| 108 | +
|
| 109 | + Option 3 — Use a separate passphrase-free deployment key and configure it in |
| 110 | + ssh_credentials.private_key_path. |
| 111 | +
|
| 112 | + You can continue now — the environment will be created. If you plan to run |
| 113 | + the deployer without an SSH agent, resolve this before running `provision`. |
| 114 | +``` |
| 115 | + |
| 116 | +### Affected Modules and Types |
| 117 | + |
| 118 | +#### Detection utility |
| 119 | + |
| 120 | +A small free function (or method on `SshCredentials`) to check whether a key file |
| 121 | +appears to be passphrase-protected. Location: `src/adapters/ssh/ssh/credentials.rs` or |
| 122 | +a new `src/adapters/ssh/ssh/key_inspector.rs`. |
| 123 | + |
| 124 | +The function signature could be: |
| 125 | + |
| 126 | +```rust |
| 127 | +/// Returns `true` if the private key at `path` appears to be passphrase-protected. |
| 128 | +/// Returns `false` if the key is unencrypted or if the file cannot be read/parsed. |
| 129 | +pub fn is_passphrase_protected(path: &Path) -> bool |
| 130 | +``` |
| 131 | + |
| 132 | +This is best-effort: it returns `false` on any I/O or parse error (no key found, |
| 133 | +unrecognized format) to avoid blocking normal flow with spurious warnings. |
| 134 | + |
| 135 | +#### `create environment` handler |
| 136 | + |
| 137 | +`src/presentation/cli/controllers/create/subcommands/environment/handler.rs`: |
| 138 | + |
| 139 | +After configuration is loaded (the `LoadConfiguration` step), call the detection |
| 140 | +function on `config.ssh_credentials.ssh_priv_key_path`. If it returns `true`, emit the |
| 141 | +warning through `user_output` before proceeding to the `CreateEnvironment` step. |
| 142 | + |
| 143 | +No changes are needed in the application or domain layers — this is a pure |
| 144 | +presentation-layer concern. |
| 145 | + |
| 146 | +### Documentation |
| 147 | + |
| 148 | +#### New page: SSH Key Handling |
| 149 | + |
| 150 | +Create `docs/user-guide/ssh-keys.md` covering: |
| 151 | + |
| 152 | +- Why the deployer requires SSH keys (remote provisioning, configuration, release, run) |
| 153 | +- Key requirements for unattended automation (no passphrase, or agent forwarding) |
| 154 | +- The three workflows: |
| 155 | + 1. Passphrase-free dedicated deployment key (recommended) |
| 156 | + 2. SSH agent forwarding into Docker |
| 157 | + 3. Direct (non-Docker) execution with an SSH agent running on the host |
| 158 | +- How to generate a deployment key pair: |
| 159 | + |
| 160 | + ```bash |
| 161 | + ssh-keygen -t ed25519 -C "torrust-tracker-deployer" \ |
| 162 | + -f ~/.ssh/torrust_tracker_deployer_ed25519 |
| 163 | + # Leave passphrase empty for automated use |
| 164 | + ``` |
| 165 | + |
| 166 | +- How to remove an existing passphrase: |
| 167 | + |
| 168 | + ```bash |
| 169 | + ssh-keygen -p -f ~/.ssh/torrust_tracker_deployer_ed25519 |
| 170 | + ``` |
| 171 | + |
| 172 | +- Security notes: dedicated deployment keys, key rotation after use, filesystem |
| 173 | + permissions (`0600`) |
| 174 | +- Reference to the `ssh_credentials` fields in the environment config JSON schema |
| 175 | + |
| 176 | +#### Update: Hetzner provider guide |
| 177 | + |
| 178 | +`docs/user-guide/providers/hetzner.md` — add a "SSH Key Requirements" section or |
| 179 | +callout box noting that Docker-based deployments require a passphrase-free key (or agent |
| 180 | +forwarding) and linking to the new SSH keys page. |
| 181 | + |
| 182 | +#### Update: `create environment` command docs |
| 183 | + |
| 184 | +`docs/user-guide/commands/create.md` — mention that a warning is shown if the |
| 185 | +configured private key appears to be passphrase-protected. |
| 186 | + |
| 187 | +## Implementation Plan |
| 188 | + |
| 189 | +### Phase 1: Detection and warning (code change) |
| 190 | + |
| 191 | +- [ ] Implement `is_passphrase_protected(path: &Path) -> bool` in |
| 192 | + `src/adapters/ssh/ssh/credentials.rs` (or a new `key_inspector.rs` module) |
| 193 | + - Check for `ENCRYPTED` in PEM header (legacy format) |
| 194 | + - Check for `bcrypt` near the start of the decoded OpenSSH body |
| 195 | + - Return `false` on any I/O or parse error |
| 196 | +- [ ] In the `create environment` handler |
| 197 | + (`handler.rs`): after `LoadConfiguration`, call the detection function and emit |
| 198 | + a warning via `user_output` if the key appears to be passphrase-protected |
| 199 | +- [ ] Add unit test `it_detects_passphrase_protected_key` (using a test fixture key |
| 200 | + with and without passphrase if available, or by constructing the minimal PEM |
| 201 | + structure in the test) |
| 202 | + |
| 203 | +### Phase 2: Documentation |
| 204 | + |
| 205 | +- [ ] Create `docs/user-guide/ssh-keys.md` covering all workflows and security notes |
| 206 | +- [ ] Update `docs/user-guide/providers/hetzner.md` with an SSH key requirements note |
| 207 | +- [ ] Update `docs/user-guide/commands/create.md` to mention the passphrase warning |
| 208 | +- [ ] Update `docs/user-guide/README.md` to link to the new `ssh-keys.md` page |
| 209 | + |
| 210 | +### Phase 3: Linting and pre-commit |
| 211 | + |
| 212 | +- [ ] Run linters: `cargo run --bin linter all` |
| 213 | +- [ ] Run pre-commit: `./scripts/pre-commit.sh` |
| 214 | + |
| 215 | +## Acceptance Criteria |
| 216 | + |
| 217 | +> **Note for Contributors**: These criteria define what the PR reviewer will check. |
| 218 | +> Use this as your pre-review checklist before submitting the PR to minimize |
| 219 | +> back-and-forth iterations. |
| 220 | + |
| 221 | +**Quality Checks**: |
| 222 | + |
| 223 | +- [ ] Pre-commit checks pass: `./scripts/pre-commit.sh` |
| 224 | + |
| 225 | +**Task-Specific Criteria**: |
| 226 | + |
| 227 | +- [ ] `create environment` emits a visible warning (not an error) when the configured |
| 228 | + private key file is passphrase-protected |
| 229 | +- [ ] `create environment` still succeeds (environment is created) even when the warning |
| 230 | + is emitted — the user is not blocked |
| 231 | +- [ ] `create environment` emits no warning when the key is unencrypted |
| 232 | +- [ ] The warning message names all three resolution options (remove passphrase, agent |
| 233 | + forwarding, separate key) |
| 234 | +- [ ] `docs/user-guide/ssh-keys.md` exists and covers key requirements, workflows, and |
| 235 | + security notes |
| 236 | +- [ ] `docs/user-guide/providers/hetzner.md` references the SSH key requirements |
| 237 | + |
| 238 | +## Related Documentation |
| 239 | + |
| 240 | +- [docs/deployments/hetzner-demo-tracker/commands/provision/problems.md](../deployments/hetzner-demo-tracker/commands/provision/problems.md) — root cause analysis and resolution for the Hetzner deployment failure |
| 241 | +- [src/adapters/ssh/ssh/credentials.rs](../../src/adapters/ssh/ssh/credentials.rs) — `SshCredentials` struct |
| 242 | +- [src/presentation/cli/controllers/create/subcommands/environment/handler.rs](../../src/presentation/cli/controllers/create/subcommands/environment/handler.rs) — where the warning is added |
| 243 | +- [docs/user-guide/providers/hetzner.md](../user-guide/providers/hetzner.md) — Hetzner provider guide |
0 commit comments