Merge #436: fix: [#428] Docker vulnerability remediation pass 1 (all 8 images)

7a44e51 fix: [#428] restore SSH private key permission normalization (confirmed root cause) (Jose Celano)
f081976 test: [#428] temporarily disable key permission normalization for CI root-cause check (Jose Celano)
c6bf1e5 test: [#428] harden SSH test key permissions and print execute errors (Jose Celano)
6453494 test: [#428] add SSH CI timeout diagnostics (Jose Celano)
3b14397 docs: [#428] normalize scan table column widths (Jose Celano)
a284272 fix: [#428] correct prometheus tag in docker_image test and workflow action policy (Jose Celano)
54bddfc docs: [#428] close mysql checklist and finalize plan acceptance (Jose Celano)
f01833a docs: [#428] finalize mysql scan documentation and status (Jose Celano)
ed7f4b3 docs: [#428] record mysql remediation decision and validation (Jose Celano)
e23d7b8 docs: [#428] mark mysql triage subtask complete (Jose Celano)
457f232 docs: [#428] close grafana checklist with follow-up #434 (Jose Celano)
2fb4a22 docs: [#428] record Grafana upgrade verification results (Jose Celano)
72772af feat: [#428] upgrade default Grafana image to 12.4.2 (Jose Celano)
8fef8ac docs: [#428] mark grafana triage subtask complete (Jose Celano)
c44741c docs: [#428] close prometheus checklist with follow-up #433 (Jose Celano)
cbc8396 docs: [#428] record prometheus upgrade verification results (Jose Celano)
90aaffe feat: [#428] upgrade default Prometheus image to v3.5.1 (Jose Celano)
e5d4cb8 docs: [#428] mark prometheus triage subtask complete (Jose Celano)
5df0d5c docs: [#428] close caddy checklist with follow-up #432 (Jose Celano)
1081a79 docs: [#428] record caddy upgrade verification results (Jose Celano)
dc0312b infra: [#428] upgrade caddy reference to 2.10.2 (Jose Celano)
232f826 docs: [#428] mark caddy triage subtask complete (Jose Celano)
ab053a6 docs: [#428] complete provisioned-instance remediation cycle (Jose Celano)
8b5a7a3 build: [#428] harden provisioned-instance package install (Jose Celano)
9c7d17d docs: [#428] mark provisioned-instance triage subtask complete (Jose Celano)
26e830c docs: [#428] complete ssh-server verification and close checklist (Jose Celano)
3e67e12 fix: [#428] correct ssh-server entrypoint script generation (Jose Celano)
fa045d9 build: [#428] remediate ssh-server base package findings (Jose Celano)
fdd3635 docs: [#428] mark ssh-server triage subtask complete (Jose Celano)
ab051e7 docs: [#428] close backup checklist with follow-up #431 (Jose Celano)
d4bb5c8 docs: [#428] record backup remediation verification results (Jose Celano)
3f1e1e3 build: [#428] apply backup base package upgrade remediation (Jose Celano)
364a152 docs: [#428] mark backup triage subtask complete (Jose Celano)
b418db2 docs: [#428] normalize scan table formatting (Jose Celano)
7393a81 docs: [#428] close deployer checklist with follow-up #430 (Jose Celano)
213a72a docs: [#428] record deployer verification and scan delta (Jose Celano)
320334b build: [#428] reduce deployer runtime gnupg exposure (Jose Celano)
0df9623 docs: [#428] mark deployer triage subtask complete (Jose Celano)
bfab622 docs: [#428] add April 2026 scan analysis baseline (Jose Celano)

Pull request description:

  ## Summary

  This PR implements Docker vulnerability remediation pass 1 for all 8 images tracked in issue #428. Each image was scanned with Trivy, remediation was applied where possible, results were verified, and follow-up issues were created for remaining unresolved CVEs.

  Closes #428

  ## Changes by Image

  ### 1. `torrust/tracker-deployer` (trixie) — partial remediation
  - Removed `gnupg` from runtime layer to reduce attack surface
  - After: 44 HIGH, 1 CRITICAL (was 49 HIGH)
  - Follow-up: #429

  ### 2. `torrust/tracker-backup` (trixie) — remediation no change
  - Added `apt-get upgrade -y` to base layer
  - After: 6 HIGH, 0 CRITICAL (no change — upstream packages not yet patched)
  - Follow-up: #431

  ### 3. `torrust/tracker-ssh-server` (Alpine 3.23.3) — fully remediated ✅
  - Added `apk upgrade --no-cache` to base layer
  - Fixed malformed entrypoint script (`echo` → `printf` for multi-line in Alpine)
  - After: 0 HIGH, 0 CRITICAL

  ### 4. `torrust/tracker-provisioned-instance` (Ubuntu 24.04) — fully remediated ✅
  - Added `--no-install-recommends` + `apt-get upgrade -y` to base layer
  - After: 0 HIGH, 0 CRITICAL

  ### 5. `caddy` (3rd-party) — partial remediation
  - Upgraded tag `2.10` → `2.10.2` in `docker-compose.yml.tera` and security scan CI workflow
  - After: 14 HIGH, 4 CRITICAL (was 18 HIGH, 6 CRITICAL)
  - Follow-up: #432

  ### 6. `prom/prometheus` (3rd-party) — partial remediation
  - Upgraded default image tag `v3.5.0` → `v3.5.1` in `src/domain/prometheus/config.rs`
  - After: 6 HIGH, 4 CRITICAL (was 16 HIGH, 4 CRITICAL)
  - Follow-up: #433

  ### 7. `grafana/grafana` (3rd-party) — partial remediation
  - Upgraded default image tag `12.3.1` → `12.4.2` in `src/domain/grafana/config.rs`
  - After: 4 HIGH, 0 CRITICAL (was 18 HIGH, 6 CRITICAL — CRITICAL fully cleared)
  - Follow-up: #434

  ### 8. `mysql` (3rd-party) — monitored, no safe upgrade
  - `mysql:8.4` (floating tag resolves to 8.4.8) = 7 HIGH, 1 CRITICAL
  - All pinned minor tags (`8.4.1`–`9.1`) have 98–100 HIGH — floating tag is already optimal
  - Runtime validated: `mysql:8.4` → Ver 8.4.8
  - Follow-up: #435

  ## Documentation Updates

  - `docs/security/docker/scans/` — added remediation pass 1 sections to all 8 scan files
  - `docs/security/docker/scans/README.md` — updated global summary table
  - `docs/issues/428-docker-vulnerability-analysis-apr8-2026.md` — all checklists and acceptance criteria complete

  ## Follow-up Issues Created

  | Issue | Image | Remaining |
  |-------|-------|-----------|
  | #429 | deployer | 44 HIGH, 1 CRITICAL |
  | #431 | backup | 6 HIGH, 0 CRITICAL |
  | #432 | caddy | 14 HIGH, 4 CRITICAL |
  | #433 | prometheus | 6 HIGH, 4 CRITICAL |
  | #434 | grafana | 4 HIGH, 0 CRITICAL |
  | #435 | mysql | 7 HIGH, 1 CRITICAL |

ACKs for top commit:
  josecelano:
    ACK 7a44e51

Tree-SHA512: e4283f6ea115d984cae2c4805c8932c08da235115489ed990253a1c2665996404810bb3d0e5d483e2e45de2289f83af146f6a5030164d859fd2af21ef2fba2bc

Author: josecelano

.github/workflows/docker-security-scan.yml

@@ -101,15 +101,18 @@ jobs:
     timeout-minutes: 10
     outputs:
       # JSON array of Docker image references for use in scan matrix
-      # Example: ["torrust/tracker:develop","mysql:8.4","prom/prometheus:v3.5.0","grafana/grafana:12.3.1","caddy:2.10"]
+      # Example: ["torrust/tracker:develop","mysql:8.4","prom/prometheus:v3.5.1","grafana/grafana:12.4.2","caddy:2.10.2"]
       images: ${{ steps.extract.outputs.images }}
 
     steps:
       - name: Checkout code
         uses: actions/checkout@v5
 
       - name: Install Rust toolchain
-        uses: actions-rust-lang/setup-rust-toolchain@v1
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Cache Rust dependencies
+        uses: Swatinem/rust-cache@v2
 
       - name: Build deployer CLI
         run: cargo build --release
@@ -179,7 +182,7 @@ jobs:
               .docker_images.mysql,
               .docker_images.prometheus,
               .docker_images.grafana
-            ] | map(select(. != null)) + ["caddy:2.10"]')
+            ] | map(select(. != null)) + ["caddy:2.10.2"]')
 
           echo "Detected images: $images"
           echo "images=$images" >> "$GITHUB_OUTPUT"

docker/backup/Dockerfile

@@ -39,6 +39,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     sqlite3 \
     gzip \
     tar \
+    && apt-get upgrade -y \
     && rm -rf /var/lib/apt/lists/*
 
 # =============================================================================

docker/deployer/Dockerfile

@@ -78,7 +78,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     openssh-client \
     # Required for downloading tools
     curl \
-    gnupg \
     # Python for Ansible
     python3 \
     python3-pip \
@@ -89,6 +88,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     ca-certificates \
     # Additional utilities
     sudo \
+    && apt-get upgrade -y \
     && rm -rf /var/lib/apt/lists/*
 
 # Install Ansible via pipx (isolated environment)
@@ -105,10 +105,13 @@ RUN pipx runpip ansible-core install ansible \
 
 # Install OpenTofu
 # Using the official installation script with deb method for Debian
-RUN curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh \
+RUN apt-get update && apt-get install -y --no-install-recommends gnupg \
+    && curl -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh \
     && chmod +x install-opentofu.sh \
     && ./install-opentofu.sh --install-method deb \
-    && rm install-opentofu.sh
+    && rm install-opentofu.sh \
+    && apt-get purge -y --auto-remove gnupg dirmngr \
+    && rm -rf /var/lib/apt/lists/*
 
 # Build arguments for customization
 ARG USER_ID=1000

docker/provisioned-instance/Dockerfile

@@ -16,7 +16,7 @@ ENV DEBIAN_FRONTEND=noninteractive
 ENV TZ=UTC
 
 # Update package list and install essential packages
-RUN apt-get update && apt-get install -y \
+RUN apt-get update && apt-get install -y --no-install-recommends \
     # SSH server for Ansible connectivity
     openssh-server \
     # Sudo for privilege escalation
@@ -35,6 +35,7 @@ RUN apt-get update && apt-get install -y \
     apt-transport-https \
     # iptables for Docker networking
     iptables \
+    && apt-get upgrade -y \
     # Clean up package cache
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*

docker/ssh-server/Dockerfile

@@ -20,7 +20,8 @@ RUN apk add --no-cache \
     curl \
     wget \
     # Network tools for testing
-    net-tools
+    net-tools \
+    && apk upgrade --no-cache
 
 # Generate SSH host keys
 RUN ssh-keygen -A
@@ -56,14 +57,12 @@ RUN echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCw16sai+XVnawp/P/Q23kcXKekygZ6AL
     chmod 600 /home/testuser/.ssh/authorized_keys && \
     chown testuser:testuser /home/testuser/.ssh/authorized_keys
 
-# Generate SSH host keys
-RUN ssh-keygen -A
-
 # Create a simple entrypoint script
-RUN echo '#!/bin/sh\n\
-# Start SSH daemon in foreground\n\
-exec /usr/sbin/sshd -D\n\
-' > /entrypoint.sh && chmod +x /entrypoint.sh
+RUN printf '%s\n' \
+    '#!/bin/sh' \
+    'exec /usr/sbin/sshd -D' \
+    > /entrypoint.sh \
+    && chmod +x /entrypoint.sh
 
 # Expose SSH port
 EXPOSE 22

docs/issues/428-docker-vulnerability-analysis-apr8-2026.md

@@ -0,0 +1,147 @@
+# Address Docker Image Vulnerabilities - April 8, 2026 Scan
+
+**Issue**: #428
+**Parent Epic**: #250 - Implement Automated Docker Image Vulnerability Scanning
+**Related**:
+
+- [Docker Security Scanning Guide](../security/docker/README.md)
+- [Vulnerability Scan Results](../security/docker/scans/README.md)
+
+## Overview
+
+The April 8, 2026 security scan revealed increased vulnerabilities across all Docker images. While primarily caused by Trivy database updates, several real issues require investigation and remediation.
+
+**Current Status by Image**:
+
+1. Deployer: 49 HIGH (regression from 1)
+2. Backup: 6 HIGH (improvement from 7)
+3. SSH Server: 1 HIGH (stable, test artifact)
+4. Provisioned Instance: 12 HIGH (minor increase)
+5. Caddy: 24 HIGH (Go dependency updates)
+6. Prometheus: 20 HIGH (Go binary updates)
+7. Grafana: 24 HIGH (mixed base + Go issues)
+8. MySQL: 8 HIGH (gosu binary, Python)
+
+## Goals
+
+- [ ] Investigate Trivy database update impact
+- [ ] Filter false positives from real vulnerabilities
+- [ ] Prioritize remediations by deployability impact
+- [ ] Complete high-impact fixes
+- [ ] Document findings and next steps
+
+## Implementation Plan
+
+### Working Rule
+
+- [ ] Process exactly one image at a time
+- [ ] Do not start the next image until the current image checklist is complete
+- [ ] Update this file after each image step to keep progress visible
+
+### Standard Steps (Repeat Per Image)
+
+For each image, execute these steps in order:
+
+1. Analysis and triage
+2. Remediation attempt
+3. Verification (rebuild + re-scan + smoke test)
+4. Documentation update
+5. Follow-up issue (only if unresolved)
+
+### Per-Image Progress Tracking
+
+#### 1. Deployer (`torrust/tracker-deployer`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved; N/A - resolved)
+- [x] Image marked done
+
+#### 2. Backup (`torrust/tracker-backup`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved)
+- [x] Image marked done
+
+#### 3. SSH Server (`torrust/tracker-ssh-server`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved)
+- [x] Image marked done
+
+#### 4. Provisioned Instance (`torrust/tracker-provisioned-instance`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved; N/A - resolved)
+- [x] Image marked done
+
+#### 5. Caddy (`caddy:2.10`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved)
+- [x] Image marked done
+
+#### 6. Prometheus (`prom/prometheus:v3.5.0`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved)
+- [x] Image marked done
+
+#### 7. Grafana (`grafana/grafana:12.3.1`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved)
+- [x] Image marked done
+
+#### 8. MySQL (`mysql:8.4`)
+
+- [x] Analysis and triage completed
+- [x] Easy remediation implemented (if available; no safe tag improvement found)
+- [x] Image rebuilt and validated
+- [x] Trivy re-scan completed and compared
+- [x] Scan docs updated
+- [x] Follow-up issue created (only if unresolved)
+- [x] Image marked done
+
+## Acceptance Criteria
+
+- [x] All 8 image checklists above are complete
+- [x] Each image was processed sequentially (one-at-a-time)
+- [x] Easy fixes were applied where possible and verified
+- [x] Scan documentation reflects post-remediation results
+- [x] Remaining unresolved cases have dedicated follow-up issues
+- [x] Pre-commit checks pass
+- [x] Changes reviewed
+
+## References
+
+- [Docker Security Scans](../security/docker/scans/README.md)
+- [Trivy Documentation](https://aquasecurity.github.io/trivy/)
+- [Debian Security Tracker](https://security-tracker.debian.org/)

docs/security/docker/scans/README.md

@@ -4,18 +4,18 @@ This directory contains historical security scan results for Docker images used
 
 ## Current Status Summary
 
-| Image                                  | Version | HIGH | CRITICAL | Status               | Last Scan    | Details                                         |
-| -------------------------------------- | ------- | ---- | -------- | -------------------- | ------------ | ----------------------------------------------- |
-| `torrust/tracker-deployer`             | trixie  | 1    | 0        | ✅ Improved (Trixie) | Feb 5, 2026  | [View](torrust-tracker-deployer.md)             |
-| `torrust/tracker-backup`               | trixie  | 7    | 0        | ℹ️ Monitored         | Feb 5, 2026  | [View](torrust-tracker-backup.md)               |
-| `torrust/tracker-ssh-server`           | 3.23.3  | 1    | 0        | ✅ Secure (Alpine)   | Feb 5, 2026  | [View](torrust-ssh-server.md)                   |
-| `torrust/tracker-provisioned-instance` | 24.04   | 11   | 0        | ℹ️ Ubuntu LTS        | Feb 5, 2026  | [View](torrust-tracker-provisioned-instance.md) |
-| `caddy`                                | 2.10    | 3    | 1        | ⚠️ Monitored         | Jan 13, 2026 | [View](caddy.md)                                |
-| `prom/prometheus`                      | v3.5.0  | 0    | 0        | ✅ SECURE            | Dec 29, 2025 | [View](prometheus.md)                           |
-| `grafana/grafana`                      | 12.3.1  | 0    | 0        | ✅ SECURE            | Dec 29, 2025 | [View](grafana.md)                              |
-| `mysql`                                | 8.4     | 0    | 0        | ✅ SECURE            | Dec 29, 2025 | [View](mysql.md)                                |
-
-**Overall Status**: ✅ **Major improvement** - Deployer updated to Debian 13 (trixie) reducing HIGH vulnerabilities from 25 to 1. SSH server and provisioned instance scans added. Backup image vulnerabilities documented with mitigation strategies.
+| Image                                  | Version | HIGH | CRITICAL | Status                    | Last Scan   | Details                                         |
+| -------------------------------------- | ------- | ---- | -------- | ------------------------- | ----------- | ----------------------------------------------- |
+| `torrust/tracker-deployer`             | trixie  | 44   | 1        | ⚠️ Partial remediation    | Apr 8, 2026 | [View](torrust-tracker-deployer.md)             |
+| `torrust/tracker-backup`               | trixie  | 6    | 0        | ℹ️ Remediation no change  | Apr 8, 2026 | [View](torrust-tracker-backup.md)               |
+| `torrust/tracker-ssh-server`           | 3.23.3  | 0    | 0        | ✅ Remediated (vuln scan) | Apr 8, 2026 | [View](torrust-ssh-server.md)                   |
+| `torrust/tracker-provisioned-instance` | 24.04   | 0    | 0        | ✅ Remediated (vuln scan) | Apr 8, 2026 | [View](torrust-tracker-provisioned-instance.md) |
+| `caddy`                                | 2.10.2  | 14   | 4        | ⚠️ Partial remediation    | Apr 8, 2026 | [View](caddy.md)                                |
+| `prom/prometheus`                      | v3.5.1  | 6    | 4        | ⚠️ Partial remediation    | Apr 8, 2026 | [View](prometheus.md)                           |
+| `grafana/grafana`                      | 12.4.2  | 4    | 0        | ⚠️ Partial remediation    | Apr 8, 2026 | [View](grafana.md)                              |
+| `mysql`                                | 8.4     | 7    | 1        | ⚠️ Monitored              | Apr 8, 2026 | [View](mysql.md)                                |
+
+**Overall Status**: ⚠️ **CVE database update detected** - Most images still show increased vulnerability counts from previous scans (Feb-Dec 2025). Deployer has a first remediation pass applied (49 HIGH -> 44 HIGH, with 1 CRITICAL still open).
 
 ## Scan Archives
 

docs/security/docker/scans/caddy.md

@@ -1,16 +1,16 @@
 # Caddy Security Scan History
 
-**Image**: `caddy:2.10`
+**Image**: `caddy:2.10.2`
 **Purpose**: TLS termination proxy for HTTPS support
 **Documentation**: [Caddy TLS Proxy Evaluation](../../research/caddy-tls-proxy-evaluation/README.md)
 
 ## Current Status
 
-| Version | HIGH | CRITICAL | Status       | Scan Date    |
-| ------- | ---- | -------- | ------------ | ------------ |
-| 2.10    | 3    | 1        | ⚠️ Monitored | Jan 13, 2026 |
+| Version | HIGH | CRITICAL | Status                               | Scan Date   |
+| ------- | ---- | -------- | ------------------------------------ | ----------- |
+| 2.10.2  | 14   | 4        | ⚠️ Partial improvement after upgrade | Apr 8, 2026 |
 
-**Deployment Status**: ✅ Safe to deploy with monitoring
+**Deployment Status**: ⚠️ Requires follow-up - upgrading from `2.10` to `2.10.2` reduced findings, but HIGH/CRITICAL issues remain in Caddy binary dependencies
 
 ## Vulnerability Summary
 
@@ -23,6 +23,32 @@ All vulnerabilities have fixed versions available upstream and are expected to b
 
 ## Scan History
 
+### April 8, 2026 - Remediation Pass 1 (Issue #428)
+
+**Scanner**: Trivy v0.68.2
+**Scan Mode**: `--scanners vuln --severity HIGH,CRITICAL`
+**Image**: `caddy:2.10.2`
+**Status**: ⚠️ **18 vulnerabilities** (14 HIGH, 4 CRITICAL)
+
+#### Summary
+
+Easy remediation applied by upgrading Caddy image tag from `2.10` to `2.10.2`.
+
+Vulnerability comparison:
+
+- Previous (`2.10`): 18 HIGH, 6 CRITICAL
+- Current (`2.10.2`): 14 HIGH, 4 CRITICAL
+
+Improvement: -4 HIGH, -2 CRITICAL
+
+#### Target Breakdown (`2.10.2`)
+
+| Target        | Type     | HIGH | CRITICAL |
+| ------------- | -------- | ---- | -------- |
+| usr/bin/caddy | gobinary | 14   | 4        |
+
+Remaining issues are in upstream Caddy binary dependencies and require vendor/upstream updates.
+
 ### January 13, 2026 - caddy:2.10
 
 **Scanner**: Trivy v0.68
@@ -59,7 +85,7 @@ All vulnerabilities have fixed versions available upstream and are expected to b
 ## How to Rescan
 
 ```bash
-trivy image --severity HIGH,CRITICAL caddy:2.10
+trivy image --severity HIGH,CRITICAL caddy:2.10.2
 ```
 
 ## Security Advisories