Configure SSH Service Port During Infrastructure Configuration

[josecelano]

Overview

The deployer currently accepts a custom SSH port configuration in the environment JSON file (ssh_credentials.port), and this value is correctly propagated to firewall rules, Ansible inventory, and all connection attempts. However, the SSH service (sshd) on the remote instance is never reconfigured to listen on the custom port - it continues listening only on the default port 22.

This creates a critical configuration mismatch where the provision command fails immediately after VM creation because Ansible cannot connect to the instance on the configured custom port.

Specification

See detailed specification: docs/issues/configure-ssh-service-port.md

(Link will be updated after file rename with issue number)

Reproduction Evidence

Manual testing confirmed:

• ✅ SSH service listening ONLY on port 22 (confirmed via ss -tlnp)
• ✅ Port 22 works perfectly
• ✅ Port 2222 connection refused (SSH not listening there)
• ✅ Ansible configured for port 2222
• ✅ Provision fails at WaitSshConnectivity with "Connection refused"

Full reproduction details in specification document.

🏗️ Architecture Requirements

DDD Layer: Application Layer (Steps) + Infrastructure Layer (Ansible playbook template)
Module Path:

• Application: src/application/steps/system/configure_ssh_port.rs
• Infrastructure: templates/ansible/configure-ssh-port.yml

Pattern: Step (Application) + Ansible Playbook (Infrastructure)

Module Structure Requirements

• Follow DDD layer separation (see docs/codebase-architecture.md)
• System configuration belongs in src/application/steps/system/
• Ansible playbook belongs in templates/ansible/ (static, no .tera extension)
• Step uses AnsibleClient to execute the playbook
• Respect dependency flow: Application → Infrastructure

Architectural Constraints

• Error handling follows project conventions
• Use structured logging with tracing crate
• Follow three-level architecture: Command → Step → Action
• State tracking must record which step failed

Implementation Plan

Phase 1: Create Ansible Playbook (1-2 hours)

• Create templates/ansible/configure-ssh-port.yml
• Playbook must override ansible_port: 22 to connect initially
• Implement idempotent check (skip if port already 22)
• Add validation and SSH service restart
• Test playbook syntax

Phase 2: Register Playbook (15 minutes)

• Modify src/infrastructure/external_tools/ansible/template/renderer/project_generator.rs
• Add to copy_static_templates method

Phase 3: Create Application Step (1 hour)

• Create src/application/steps/system/configure_ssh_port.rs
• Follow pattern from configure_firewall.rs
• Export step in module hierarchy

Phase 4: Update Domain State (30 minutes)

• Add ConfigureSshPort to ConfigureStep enum

Phase 5: Integrate into Configure Command (1 hour)

• Modify execute_configuration_with_tracking in configure handler
• Add as FIRST step (before all other configuration)
• Implement skip logic for containers

Phase 6: Testing (2-3 hours)

• Unit tests for step
• E2E test with port 22 (default)
• E2E test with port 2222 (custom)
• Verify SSH connectivity after reconfiguration

Phase 7: Documentation (1 hour)

• Update user guide
• Update deployment overview
• Create ADR

Estimated Total Time: 8-10 hours

Acceptance Criteria

Note for Contributors: These criteria define what the PR reviewer will check. Use this as your pre-review checklist before submitting the PR.

Quality Checks:

• Pre-commit checks pass: ./scripts/pre-commit.sh

Functional Requirements:

• Playbook configure-ssh-port.yml exists and is registered
• Playbook modifies /etc/ssh/sshd_config with correct port
• Playbook validates and restarts SSH service
• Playbook verifies SSH listening on new port
• Step ConfigureSshPortStep exists and is integrated
• Step executes as FIRST configuration step
• ConfigureStep enum includes ConfigureSshPort variant

Conditional Execution:

• Step skips for port 22 (idempotent)
• Step respects container skip environment variable
• Step execution is logged with tracing

Testing Requirements:

• E2E test with default port 22 succeeds
• E2E test with custom port 2222 succeeds
• Subsequent commands connect on custom port

Error Handling:

• Configuration validation errors are caught
• Service restart failures reported with recovery instructions
• Failed step tracked correctly in failure context

Documentation:

• User guide updated with SSH port configuration
• Deployment overview updated
• ADR documenting design decisions

Related

• Parent Epic: TBD
• Roadmap: Roadmap #1
• Specification: docs/issues/configure-ssh-service-port.md (will be renamed to include issue number)

[josecelano]

Implementation Complete - Different Approach Used

The SSH port configuration feature has been implemented, but we used a significantly different approach than originally proposed in this issue.

Original Plan (Ansible-based)

The issue initially suggested using an Ansible playbook during the configure phase to reconfigure the SSH port after provisioning.

Actual Implementation (Cloud-Init with Reboot)

After extensive testing and analysis, we implemented SSH port configuration via cloud-init during the provision phase using a reboot pattern.

Why the Change?

Critical Timing Issue

The original Ansible-based approach wouldn't work because:

1. The provision phase includes WaitForCloudInitStep that uses Ansible to wait for cloud-init completion
2. This step already requires SSH connectivity on the custom port from the inventory
3. By the time we reach the configure phase, provision would have already failed

Technical Challenges Discovered

During implementation, we discovered several issues with alternative approaches:

1. systemctl restart ssh - Doesn't kill the old SSH process when the port changes, resulting in SSH listening on BOTH port 22 and the custom port
2. bootcmd disable approach - systemd automatically re-enables and starts SSH ~3 seconds after bootcmd disables it
3. Manual service management - Cloud-init runcmd execution often completes without SSH actually restarting

Solution: Reboot Pattern

Following Hetzner's cloud-config best practices, we implemented:

write_files:
  - path: /etc/ssh/sshd_config.d/99-custom-port.conf
    content: |
      # Custom SSH port configuration
      Port {{ ssh_port }}
    permissions: '0644'
    owner: root:root

runcmd:
  - reboot

This ensures:

• SSH service cleanly restarts with new configuration
• No lingering processes on port 22
• All services start in a consistent state

Additional Changes Required

1. Provision handler fix: Changed wait_for_readiness() to use the configured SSH port instead of hardcoded port 22
2. Timeout increase: Raised DEFAULT_MAX_RETRY_ATTEMPTS from 30 to 60 (120 seconds total) to account for cloud-init completion (~70-80s) + reboot time

Documentation

• ADR: Cloud-Init SSH Port Configuration with Reboot
• Updated Issue Spec: docs/issues/222-configure-ssh-service-port.md
• Pull Request: fix: [#222] Configure SSH port via cloud-init with reboot pattern #223

Testing Results

✅ Manual E2E testing confirmed:

• SSH listens only on the configured custom port (e.g., 2222)
• No lingering SSH process on port 22
• Provision phase completes successfully
• Ansible can connect on the custom port

The implementation is architecturally cleaner: infrastructure configuration (SSH port) happens during infrastructure provisioning (provision phase), not as a post-provisioning application configuration step.