✨ feat(security): add Windows VM support and Secret-based cloud-init storage

Implement comprehensive Windows VM support and enhance security through
unified Kubernetes Secret storage for all VM credentials and initialization data.

**Windows VM Support:**
- Add Cloudbase-Init support for Windows VMs (pwsh shell)
- Shell-based OS detection: bash → Linux, pwsh → Windows
- OS-specific cloud-init generation (Linux: bcrypt, Windows: plaintext)
- Cloudbase-Init plugin configuration and troubleshooting guide

**Security Enhancements:**
- Unified Kubernetes Secret storage for SSH credentials + cloud-init userdata
- Use UserDataSecretRef to reference cloud-init from Secret (no inline userdata)
- Passwords never visible in VM specs (kubectl describe vmi)
- Secret cleanup in garbage collection (gc.go) to prevent accumulation
- Secret cleanup in job cleanup phase (cleanup.go)

**Testing & Documentation:**
- Add comprehensive test suite (security_test.go) with 20+ sub-tests
- Test coverage: password generation, bcrypt hashing, Linux/Windows cloud-init
- Update README with Windows configuration examples and requirements
- Document Secret structure and security benefits

**Backward Compatibility:**
- Zero breaking changes - existing Linux configs work unchanged
- CreateSSHSecret maintained as backward-compatible alias

Author: thpham

README.md

@@ -12,6 +12,7 @@ GitLab Runner custom executor for running CI/CD jobs in VMs on Kubernetes using
 - **KubeVirt Native**: Leverages Kubernetes for VM orchestration
 - **Custom Executor**: Integrates seamlessly with GitLab Runner
 - **Security Hardened**: Regular dependency updates and CVE scanning
+- **Secure Credential Management**: Kubernetes Secrets with RBAC protection
 
 ## Prerequisites
 
@@ -44,8 +45,7 @@ runners:
           "--default-image", "registry.example.com/runner:latest",
           "--default-machine-type", "microvm",
           "--default-architecture", "x86_64",
-          "--ssh-user", "runner",
-          "--ssh-password", "runner"
+          "--ssh-user", "runner"
         ]
         run_exec = "/bin/gitlab-runner-kubevirt"
         run_args = ["run"]
@@ -124,13 +124,13 @@ build-heavy:
 
 ### Garbage Collection
 
-Orphaned VMs are automatically tagged with TTL labels for cleanup:
+Orphaned VMs are automatically tagged with TTL labels for cleanup. The garbage collector automatically deletes both VMs and their associated credential Secrets (containing SSH credentials and cloud-init data):
 
 ```bash
 # List VMs with their expiration info
 kubectl get vmi -n gitlab-runner -l io.kubevirt.gitlab-runner/id --show-labels
 
-# Manual cleanup of expired VMs
+# Manual cleanup of expired VMs (also deletes associated Secrets)
 gitlab-runner-kubevirt gc --namespace gitlab-runner
 
 # Dry-run mode (see what would be deleted)
@@ -140,6 +140,8 @@ gitlab-runner-kubevirt gc --dry-run
 gitlab-runner-kubevirt gc --max-age 1h
 ```
 
+**Note:** Garbage collection cleans up both VirtualMachineInstances and their credential Secrets (containing SSH credentials and cloud-init userdata) to prevent Secret accumulation.
+
 **Automated cleanup** with CronJob:
 
 ```yaml
@@ -180,6 +182,210 @@ The container image includes both `gitlab-runner` (standard GitLab Runner) and `
 
 **Note:** When using the GitLab Runner Helm chart with custom executor configuration, you don't need to set this variable as the executor directly calls `/bin/gitlab-runner-kubevirt`.
 
+### Security
+
+#### SSH Credential Management
+
+GitLab Runner KubeVirt uses a secure credential management system to protect SSH access to VMs:
+
+**Automatic Security Features:**
+
+1. **Unique Random Passwords**: Each VM receives a cryptographically secure 32-character random password
+2. **Kubernetes Secrets**: Credentials and cloud-init data stored in RBAC-protected Secrets (not plaintext in VM specs)
+3. **Cloud-init Injection**: Cloud-init userdata stored as Secret, referenced by VM (passwords never visible in VM objects)
+4. **Automatic Cleanup**: Secrets automatically deleted when VMs are cleaned up
+5. **No Plaintext Storage**: Passwords never stored in VM annotations, logs, or VM specifications
+
+**How it works:**
+
+```
+Prepare Phase:
+  1. Generate random password (crypto/rand)
+  2. Generate cloud-init with bcrypt-hashed password (Linux) or plaintext (Windows)
+  3. Create Kubernetes Secret with:
+     - SSH username and password (for Run phase)
+     - Cloud-init userdata (for VM initialization)
+  4. Create VM with Secret reference (not inline userdata)
+  5. Store only Secret reference in VM annotation
+
+Run Phase:
+  1. Retrieve Secret reference from VM annotation
+  2. Fetch SSH credentials from Kubernetes Secret
+  3. Connect to VM via SSH
+
+Cleanup Phase:
+  1. Retrieve Secret reference from VM annotation
+  2. Delete Kubernetes Secret (removes both SSH creds and cloud-init data)
+  3. Delete VM
+```
+
+**RBAC Requirements:**
+
+The GitLab Runner service account requires these permissions for Secret management:
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: gitlab-runner-kubevirt
+  namespace: gitlab-runner
+rules:
+  # VirtualMachineInstance permissions
+  - apiGroups: ["kubevirt.io"]
+    resources: ["virtualmachineinstances"]
+    verbs: ["get", "list", "create", "delete", "watch"]
+
+  # Secret permissions for SSH credentials
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["create", "get", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: gitlab-runner-kubevirt
+  namespace: gitlab-runner
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: gitlab-runner-kubevirt
+subjects:
+  - kind: ServiceAccount
+    name: gitlab-runner
+    namespace: gitlab-runner
+```
+
+**Secret Structure:**
+
+Each VM gets a single Kubernetes Secret containing:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: vm-creds-<job-id>
+  labels:
+    io.kubevirt.gitlab-runner/id: <job-id>
+    io.kubevirt.gitlab-runner/type: vm-credentials
+stringData:
+  user: runner # SSH username
+  password: <random-32> # SSH password
+  userdata: | # Cloud-init YAML
+    #cloud-config
+    users:
+      - name: runner
+        passwd: <bcrypt-hash>  # Linux: hashed, Windows: plaintext
+        ...
+```
+
+**Security Benefits:**
+
+- **Single Secret Per VM**: One Secret contains all credentials (SSH + cloud-init)
+- **RBAC Protected**: Secret access controlled via Kubernetes RBAC policies
+- **No Plaintext in VM Specs**: Password never visible in `kubectl describe vmi`
+- **Scoped to Jobs**: Each CI/CD job gets unique credentials
+- **Automatic Cleanup**: Secret deleted when VM is cleaned up
+- **Audit Trail**: All Secret access logged by Kubernetes audit logs
+
+#### Windows VM Support
+
+GitLab Runner KubeVirt supports both Linux and Windows VMs with automatic OS detection based on the shell parameter:
+
+**OS Detection:**
+
+- `--shell bash` → Linux VM (cloud-init with bcrypt-hashed passwords)
+- `--shell pwsh` → Windows VM (Cloudbase-Init with plaintext passwords)
+
+**Windows Requirements:**
+
+Your Windows container disk images must include:
+
+1. **Cloudbase-Init**: Windows port of cloud-init for VM initialization
+
+   - Download: https://cloudbase-init.readthedocs.io/en/latest/
+   - Minimum version: 1.1.0+
+   - **Required Configuration**: Must enable `SetUserPasswordPlugin` in `cloudbase-init.conf`:
+
+     ```ini
+     [DEFAULT]
+     username=Administrator
+     inject_user_password=true
+
+     plugins=cloudbaseinit.plugins.common.mtu.MTUPlugin,
+             cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin,
+             cloudbaseinit.plugins.windows.createuser.CreateUserPlugin,
+             cloudbaseinit.plugins.common.setuserpassword.SetUserPasswordPlugin
+     ```
+
+2. **OpenSSH Server**: For remote access via SSH
+
+   - Built-in on Windows Server 2019+ and Windows 10 1809+
+   - Must be installed and configured to start automatically
+   - Port 22 must be accessible
+
+3. **PowerShell**: For script execution
+   - PowerShell 5.1+ or PowerShell Core 7+
+
+**Windows Configuration Example:**
+
+```yaml
+runners:
+  executor: custom
+  config: |
+    [[runners]]
+      name = "kubevirt-windows"
+      executor = "custom"
+
+      [runners.custom]
+        config_exec = "/bin/gitlab-runner-kubevirt"
+        config_args = ["config"]
+        prepare_exec = "/bin/gitlab-runner-kubevirt"
+        prepare_args = [
+          "prepare",
+          "--shell", "pwsh",  # Windows indicator
+          "--default-image", "registry.example.com/windows-server-2022:latest",
+          "--ssh-user", "runner"
+        ]
+        run_exec = "/bin/gitlab-runner-kubevirt"
+        run_args = ["run"]
+        cleanup_exec = "/bin/gitlab-runner-kubevirt"
+        cleanup_args = ["cleanup"]
+```
+
+**Security Notes for Windows:**
+
+- Cloudbase-Init uses plaintext passwords (industry standard)
+- Credentials still protected by Kubernetes Secret RBAC
+- Random 32-character passwords meet Windows complexity requirements
+- Administrator group membership required for CI/CD operations
+
+**Supported Windows Versions:**
+
+- Windows Server 2019, 2022
+- Windows 10 version 1809+
+- Windows 11
+
+**Troubleshooting Windows VM Initialization:**
+
+If password authentication fails on Windows VMs, verify Cloudbase-Init configuration:
+
+```powershell
+# Check Cloudbase-Init logs
+Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log"
+
+# Verify SetUserPasswordPlugin is enabled
+Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init.conf" | Select-String "SetUserPasswordPlugin"
+
+# Check if inject_user_password is enabled
+Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init.conf" | Select-String "inject_user_password"
+```
+
+Common issues:
+
+- **Password not set**: `SetUserPasswordPlugin` not enabled in plugins list
+- **Random password instead of cloud-config password**: `inject_user_password=false` or not set
+- **User not created**: `CreateUserPlugin` not enabled in plugins list
+
 ## Development
 
 ### Quick Start

cleanup.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 	"os"
 	"strings"
@@ -43,6 +44,25 @@ func (cmd *CleanupCmd) Run(ctx context.Context, client kubevirt.KubevirtClient,
 
 	fmt.Fprintf(os.Stderr, "Deleting Virtual Machine instance %v\n", vm.ObjectMeta.Name)
 
+	// Delete SSH credentials Secret before deleting the VM
+	var rc RunConfig
+	if err := json.Unmarshal([]byte(vm.Annotations[RunConfigKey]), &rc); err == nil {
+		if rc.SSH.SecretRef != "" {
+			cfg, err := KubeConfig()
+			if err != nil {
+				fmt.Fprintf(os.Stderr, "Warning: failed to get kubernetes config for secret cleanup: %v\n", err)
+			} else {
+				if err := DeleteSSHSecret(ctx, cfg, jctx.Namespace, rc.SSH.SecretRef); err != nil {
+					fmt.Fprintf(os.Stderr, "Warning: failed to delete SSH secret %s: %v\n", rc.SSH.SecretRef, err)
+				} else {
+					fmt.Fprintf(os.Stderr, "Deleted SSH credentials secret: %s\n", rc.SSH.SecretRef)
+				}
+			}
+		}
+	} else {
+		fmt.Fprintf(os.Stderr, "Warning: failed to unmarshal RunConfig for secret cleanup: %v\n", err)
+	}
+
 	deleteOptions := &metav1.DeleteOptions{}
 
 	if err := client.VirtualMachineInstance(jctx.Namespace).Delete(ctx, vm.ObjectMeta.Name, *deleteOptions); err != nil {

gc.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 	"os"
 	"time"
@@ -84,6 +85,21 @@ func (cmd *GCCmd) Run(ctx context.Context, jctx *JobContext) error {
 				fmt.Fprintf(os.Stderr, "🗑️  [DRY-RUN] Would delete VM %s (age: %s, ttl: %s)\n", vm.Name, age.Round(time.Second), ttl)
 			} else {
 				fmt.Fprintf(os.Stderr, "🗑️  Deleting expired VM %s (age: %s, ttl: %s)\n", vm.Name, age.Round(time.Second), ttl)
+
+				// Delete SSH credentials Secret before deleting the VM
+				var rc RunConfig
+				if err := json.Unmarshal([]byte(vm.Annotations[RunConfigKey]), &rc); err == nil {
+					if rc.SSH.SecretRef != "" {
+						if err := DeleteSSHSecret(ctx, config, namespace, rc.SSH.SecretRef); err != nil {
+							fmt.Fprintf(os.Stderr, "⚠️  Warning: failed to delete SSH secret %s: %v\n", rc.SSH.SecretRef, err)
+						} else {
+							fmt.Fprintf(os.Stderr, "   Deleted SSH credentials secret: %s\n", rc.SSH.SecretRef)
+						}
+					}
+				} else if vm.Annotations[RunConfigKey] != "" {
+					fmt.Fprintf(os.Stderr, "⚠️  Warning: failed to unmarshal RunConfig for secret cleanup: %v\n", err)
+				}
+
 				err := client.VirtualMachineInstance(namespace).Delete(ctx, vm.Name, metav1.DeleteOptions{})
 				if err != nil {
 					fmt.Fprintf(os.Stderr, "❌ Failed to delete VM %s: %v\n", vm.Name, err)