Exhaustive CLI Command Syntax Test Suite Specification

Document Metadata

Version: 1.0.0
Created: 2025-10-27
Target: 300+ comprehensive syntax tests for all 33 CLI commands
Test Framework: pytest
Test Type: Unit tests using Click's CliRunner

1. Test Organization Structure

1.1 Directory Layout

tests/
└── unit/
    └── cli/
        ├── __init__.py
        ├── conftest.py                      # CLI-specific fixtures
        │
        ├── test_cli_core.py                 # Core CLI tests (help, version)
        │
        ├── provisioning/                    # VM provisioning commands
        │   ├── test_new_command.py
        │   ├── test_vm_command.py
        │   └── test_create_command.py
        │
        ├── lifecycle/                       # VM lifecycle commands
        │   ├── test_start_command.py
        │   ├── test_stop_command.py
        │   ├── test_kill_command.py
        │   ├── test_destroy_command.py
        │   ├── test_killall_command.py
        │   └── test_prune_command.py
        │
        ├── information/                     # Information/status commands
        │   ├── test_list_command.py
        │   ├── test_status_command.py
        │   ├── test_ps_command.py
        │   ├── test_w_command.py
        │   └── test_top_command.py
        │
        ├── connection/                      # Connection commands
        │   ├── test_connect_command.py
        │   ├── test_session_command.py
        │   └── test_clone_command.py
        │
        ├── file_operations/                 # File transfer commands
        │   ├── test_cp_command.py
        │   └── test_sync_command.py
        │
        ├── maintenance/                     # Maintenance commands
        │   ├── test_update_command.py
        │   └── test_os_update_command.py
        │
        ├── monitoring/                      # Monitoring commands
        │   └── test_cost_command.py
        │
        ├── automation/                      # Automation commands (existing)
        │   ├── test_do_command.py
        │   └── test_doit_command.py
        │
        ├── groups/                          # Command groups
        │   ├── test_batch_group.py          # batch command, start, stop, sync
        │   ├── test_env_group.py            # env list, set, delete, clear, export, import
        │   ├── test_keys_group.py           # keys list, rotate, backup, export
        │   ├── test_snapshot_group.py       # snapshot create, list, restore, etc.
        │   ├── test_storage_group.py        # storage create, list, mount, unmount, etc.
        │   └── test_template_group.py       # template list, create, delete, etc.
        │
        └── integration/                     # Cross-command integration tests
            ├── test_option_combinations.py  # Test complex option combinations
            └── test_command_chaining.py     # Test command workflows

1.2 File Naming Conventions

Pattern: test_<command_name>_command.py for single commands
Pattern: test_<group_name>_group.py for command groups
Class naming: Test<CommandName><Category> (e.g., TestNewCommandSyntax)
Test method naming: test_<scenario>_<expected_outcome> (e.g., test_missing_required_arg_shows_error)

2. Command Inventory

2.1 Primary Commands (27)

Provisioning (3):

azlin new - Create new VM (alias for vm)
azlin vm - Create new VM
azlin create - Create new VM (alternative)

Lifecycle Management (6):

azlin start - Start stopped VM
azlin stop - Stop running VM
azlin kill - Delete single VM
azlin destroy - Delete single VM (alias)
azlin killall - Delete all VMs in resource group
azlin prune - Clean up unused resources

Information/Status (5):

azlin list - List VMs
azlin status - Show VM status
azlin ps - Show running processes
azlin w - Show who's logged in
azlin top - Show resource usage

Connection (3):

azlin connect - SSH to VM
azlin session - Manage tmux sessions
azlin clone - Clone VM

File Operations (2):

azlin cp - Copy files to/from VM
azlin sync - Sync directories

Maintenance (2):

azlin update - Update azlin
azlin os-update - Update VM OS

Monitoring (1):

azlin cost - Show cost information

Automation (2):

azlin do - Execute command on VM
azlin doit - Natural language automation

Core (3):

azlin help - Show help
azlin --version - Show version
azlin (no args) - Show default help

2.2 Command Groups (6)

batch (4 subcommands):

azlin batch command - Run command on multiple VMs
azlin batch start - Start multiple VMs
azlin batch stop - Stop multiple VMs
azlin batch sync - Sync to multiple VMs

env (6 subcommands):

azlin env list - List environment variables
azlin env set - Set environment variable
azlin env delete - Delete environment variable
azlin env clear - Clear all environment variables
azlin env export - Export environment to file
azlin env import - Import environment from file

keys (4 subcommands):

azlin keys list - List SSH keys
azlin keys rotate - Rotate SSH keys
azlin keys backup - Backup SSH keys
azlin keys export - Export SSH keys

snapshot (8 subcommands):

azlin snapshot create - Create snapshot
azlin snapshot list - List snapshots
azlin snapshot restore - Restore snapshot
azlin snapshot delete - Delete snapshot
azlin snapshot enable - Enable auto-snapshots
azlin snapshot disable - Disable auto-snapshots
azlin snapshot status - Show snapshot status
azlin snapshot sync - Sync snapshots

storage (6 subcommands):

azlin storage create - Create NFS storage
azlin storage list - List storage accounts
azlin storage mount - Mount NFS storage
azlin storage unmount - Unmount NFS storage
azlin storage status - Show storage status
azlin storage delete - Delete storage account

template (5 subcommands):

azlin template list - List templates
azlin template create - Create template
azlin template delete - Delete template
azlin template export - Export template
azlin template import - Import template

Total Commands: 27 primary + 33 subcommands = 60 command variants

3. Test Categories

3.1 Syntax Validation Tests (5-8 tests per command)

Test cases:

No arguments - When required args missing
Required arguments only - Minimal valid syntax
Optional arguments - Each optional arg individually
Empty string arguments - Args that are present but empty
Whitespace-only arguments - Args with only spaces/tabs
Maximum arguments - All options provided
Positional argument order - Verify correct ordering
Extra unexpected arguments - Too many args

Example pattern:

class TestNewCommandSyntax:
    """Test syntax validation for 'azlin new' command."""

    def test_no_arguments_shows_help(self):
        """Test that 'azlin new' with no args shows help."""
        runner = CliRunner()
        result = runner.invoke(main, ["new"])

        assert result.exit_code == 0
        assert "Usage:" in result.output or "Options:" in result.output

    def test_with_repo_only_succeeds(self):
        """Test that 'azlin new --repo URL' is valid minimal syntax."""
        runner = CliRunner()
        with patch("azlin.vm_manager.VMManager.create_vm"):
            result = runner.invoke(main, ["new", "--repo", "https://github.com/user/repo"])

            # Should succeed (or fail for other reasons, not syntax)
            assert "Error: Missing" not in result.output

    def test_empty_string_repo_shows_error(self):
        """Test that 'azlin new --repo \"\"' shows validation error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", ""])

        assert result.exit_code != 0
        assert "empty" in result.output.lower() or "invalid" in result.output.lower()

3.2 Option Combination Tests (8-12 tests per command)

Test cases:

Mutually exclusive options - Options that can't be used together
Dependent options - Options that require other options
Conflicting option values - Options with incompatible values
Boolean flag combinations - Multiple flags together
Short + long form mixing - -rg and --resource-group
Repeated options - Same option specified multiple times
Option order independence - Options in different orders
All compatible options - Maximum valid combination

Example pattern:

class TestNewCommandOptions:
    """Test option combinations for 'azlin new' command."""

    def test_pool_and_name_are_mutually_exclusive(self):
        """Test that --pool and --name cannot be used together."""
        runner = CliRunner()
        result = runner.invoke(main, [
            "new",
            "--repo", "https://github.com/user/repo",
            "--pool", "3",
            "--name", "custom-vm"
        ])

        assert result.exit_code != 0
        assert "mutually exclusive" in result.output.lower() or "cannot use" in result.output.lower()

    def test_short_and_long_form_resource_group(self):
        """Test that -rg and --resource-group are equivalent."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm") as mock_create:
            # Test with --resource-group
            result1 = runner.invoke(main, ["new", "--repo", "...", "--resource-group", "test-rg"])

            # Test with -rg
            result2 = runner.invoke(main, ["new", "--repo", "...", "-rg", "test-rg"])

            # Both should call create_vm with same resource_group
            assert mock_create.call_count == 2
            call1_kwargs = mock_create.call_args_list[0][1]
            call2_kwargs = mock_create.call_args_list[1][1]
            assert call1_kwargs.get("resource_group") == call2_kwargs.get("resource_group")

    def test_all_valid_options_together(self):
        """Test maximum valid option combination."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm"):
            result = runner.invoke(main, [
                "new",
                "--repo", "https://github.com/user/repo",
                "--vm-size", "Standard_D2s_v3",
                "--region", "eastus",
                "--resource-group", "test-rg",
                "--config", "/path/to/config.yaml",
                "--template", "python-dev",
                "--nfs-storage", "my-storage",
                "--no-auto-connect"
            ])

            # Should succeed (assuming mocks are proper)
            assert "cannot use" not in result.output.lower()

3.3 Alias Tests (2-3 tests per alias)

Test cases:

Alias equivalence - Alias produces same result as primary command
Alias with options - Alias works with all options
Alias help text - Help shows alias relationship

Example pattern:

class TestCommandAliases:
    """Test command aliases (vm/new/create, kill/destroy)."""

    def test_vm_and_new_are_equivalent(self):
        """Test that 'azlin vm' and 'azlin new' are identical."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm") as mock_create:
            # Test with 'vm'
            result1 = runner.invoke(main, ["vm", "--repo", "https://..."])

            mock_create.reset_mock()

            # Test with 'new'
            result2 = runner.invoke(main, ["new", "--repo", "https://..."])

            # Both should call same underlying function
            assert mock_create.call_count == 1  # Each should call once
            # Exit codes should match
            assert result1.exit_code == result2.exit_code

    def test_kill_and_destroy_are_equivalent(self):
        """Test that 'azlin kill' and 'azlin destroy' are identical."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.delete_vm") as mock_delete:
            result1 = runner.invoke(main, ["kill", "test-vm", "--force"])
            mock_delete.reset_mock()
            result2 = runner.invoke(main, ["destroy", "test-vm", "--force"])

            assert result1.exit_code == result2.exit_code

3.4 Error Handling Tests (6-10 tests per command)

Test cases:

Invalid option values - Wrong type, out of range
Unknown options - Options that don't exist
Invalid argument format - Malformed inputs
Missing required dependencies - Azure not configured
Resource not found - VM doesn't exist
Permission errors - No access to resource
Network errors - Azure API unavailable
Timeout scenarios - Long-running operations
Partial failures - Some operations succeed, some fail
Graceful degradation - Non-critical failures

Example pattern:

class TestNewCommandErrors:
    """Test error handling for 'azlin new' command."""

    def test_invalid_vm_size_shows_error(self):
        """Test that invalid VM size shows clear error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", "...", "--vm-size", "InvalidSize"])

        assert result.exit_code != 0
        assert "invalid" in result.output.lower() or "not valid" in result.output.lower()

    def test_invalid_region_shows_error(self):
        """Test that invalid region shows clear error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", "...", "--region", "mars-central"])

        assert result.exit_code != 0
        assert "region" in result.output.lower()

    def test_unknown_option_shows_error(self):
        """Test that unknown option shows helpful error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--unknown-option", "value"])

        assert result.exit_code != 0
        assert "no such option" in result.output.lower() or "unknown option" in result.output.lower()

    def test_missing_azure_credentials_shows_clear_message(self):
        """Test that missing Azure credentials shows helpful error."""
        runner = CliRunner()

        with patch("azure.identity.DefaultAzureCredential", side_effect=Exception("No credentials")):
            result = runner.invoke(main, ["new", "--repo", "https://..."])

            assert result.exit_code != 0
            assert "credential" in result.output.lower() or "authentication" in result.output.lower()

3.5 Help Text Tests (3-4 tests per command)

Test cases:

Command help with --help - Shows full help text
Command help with -h - Short form works
Help text completeness - All options documented
Example usage shown - Help includes examples

Example pattern:

class TestNewCommandHelp:
    """Test help text for 'azlin new' command."""

    def test_help_flag_shows_usage(self):
        """Test that 'azlin new --help' shows complete usage."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--help"])

        assert result.exit_code == 0
        assert "Usage:" in result.output
        assert "Options:" in result.output

    def test_help_documents_all_options(self):
        """Test that help text includes all available options."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--help"])

        required_options = [
            "--repo",
            "--vm-size",
            "--region",
            "--resource-group",
            "--name",
            "--pool",
            "--no-auto-connect",
            "--config",
            "--template",
            "--nfs-storage"
        ]

        for option in required_options:
            assert option in result.output, f"Option {option} not documented in help"

    def test_help_includes_examples(self):
        """Test that help text includes usage examples."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--help"])

        # Check for example patterns
        assert "example" in result.output.lower() or "azlin new" in result.output.lower()

    def test_short_help_flag_works(self):
        """Test that -h works as alias for --help."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "-h"])

        assert result.exit_code == 0
        assert "Usage:" in result.output

4. Test Patterns and Templates

4.1 Basic Command Test Template

"""Unit tests for 'azlin <command>' command syntax."""

from unittest.mock import patch, Mock
import pytest
from click.testing import CliRunner
from azlin.cli import main


class Test<Command>CommandSyntax:
    """Test basic syntax validation for '<command>' command."""

    def test_no_arguments_behavior(self):
        """Test command behavior with no arguments."""
        runner = CliRunner()
        result = runner.invoke(main, ["<command>"])

        # Define expected behavior (error, help, or success)
        assert result.exit_code == 0  # or != 0 for error

    def test_required_arguments_only(self):
        """Test command with only required arguments."""
        runner = CliRunner()
        with patch("<mock_path>"):
            result = runner.invoke(main, ["<command>", "<required_arg>"])

            assert result.exit_code == 0

    def test_with_all_options(self):
        """Test command with all available options."""
        runner = CliRunner()
        with patch("<mock_path>"):
            result = runner.invoke(main, [
                "<command>",
                "<required_arg>",
                "--option1", "value1",
                "--option2", "value2"
            ])

            assert result.exit_code == 0


class Test<Command>CommandOptions:
    """Test option combinations for '<command>' command."""

    def test_mutually_exclusive_options(self):
        """Test that mutually exclusive options show error."""
        runner = CliRunner()
        result = runner.invoke(main, [
            "<command>",
            "--option1", "value1",
            "--option2", "value2"  # Conflicts with option1
        ])

        assert result.exit_code != 0
        assert "cannot use" in result.output.lower()

    def test_option_value_validation(self):
        """Test that invalid option values show error."""
        runner = CliRunner()
        result = runner.invoke(main, ["<command>", "--option", "invalid_value"])

        assert result.exit_code != 0


class Test<Command>CommandErrors:
    """Test error handling for '<command>' command."""

    def test_unknown_option_shows_error(self):
        """Test that unknown options show helpful error."""
        runner = CliRunner()
        result = runner.invoke(main, ["<command>", "--unknown-option", "value"])

        assert result.exit_code != 0
        assert "no such option" in result.output.lower()

    def test_missing_resource_shows_error(self):
        """Test that missing resource shows clear error."""
        runner = CliRunner()
        with patch("<mock_path>", side_effect=Exception("Not found")):
            result = runner.invoke(main, ["<command>", "nonexistent"])

            assert result.exit_code != 0
            assert "not found" in result.output.lower()


class Test<Command>CommandHelp:
    """Test help text for '<command>' command."""

    def test_help_flag_shows_complete_usage(self):
        """Test that --help shows complete usage information."""
        runner = CliRunner()
        result = runner.invoke(main, ["<command>", "--help"])

        assert result.exit_code == 0
        assert "Usage:" in result.output
        assert "Options:" in result.output

4.2 Command Group Test Template

"""Unit tests for 'azlin <group>' command group syntax."""

from unittest.mock import patch, Mock
import pytest
from click.testing import CliRunner
from azlin.cli import main


class Test<Group>GroupSyntax:
    """Test basic syntax for '<group>' command group."""

    def test_group_without_subcommand_shows_help(self):
        """Test that 'azlin <group>' shows available subcommands."""
        runner = CliRunner()
        result = runner.invoke(main, ["<group>"])

        assert result.exit_code == 0
        assert "Commands:" in result.output or "Usage:" in result.output

    def test_group_help_flag(self):
        """Test that 'azlin <group> --help' shows group help."""
        runner = CliRunner()
        result = runner.invoke(main, ["<group>", "--help"])

        assert result.exit_code == 0
        assert "<subcommand1>" in result.output
        assert "<subcommand2>" in result.output


class Test<Group><Subcommand>Syntax:
    """Test syntax for '<group> <subcommand>' command."""

    def test_subcommand_no_arguments(self):
        """Test subcommand with no arguments."""
        runner = CliRunner()
        result = runner.invoke(main, ["<group>", "<subcommand>"])

        # Define expected behavior
        assert result.exit_code == 0  # or != 0

    def test_subcommand_with_required_arguments(self):
        """Test subcommand with required arguments."""
        runner = CliRunner()
        with patch("<mock_path>"):
            result = runner.invoke(main, [
                "<group>",
                "<subcommand>",
                "<required_arg>"
            ])

            assert result.exit_code == 0

    def test_subcommand_with_options(self):
        """Test subcommand with options."""
        runner = CliRunner()
        with patch("<mock_path>"):
            result = runner.invoke(main, [
                "<group>",
                "<subcommand>",
                "--option", "value"
            ])

            assert result.exit_code == 0


class Test<Group><Subcommand>Errors:
    """Test error handling for '<group> <subcommand>' command."""

    def test_invalid_subcommand_shows_error(self):
        """Test that invalid subcommand shows error."""
        runner = CliRunner()
        result = runner.invoke(main, ["<group>", "invalid-subcommand"])

        assert result.exit_code != 0
        assert "no such command" in result.output.lower()

    def test_missing_required_argument_shows_error(self):
        """Test that missing required argument shows error."""
        runner = CliRunner()
        result = runner.invoke(main, ["<group>", "<subcommand>"])

        assert result.exit_code != 0
        assert "required" in result.output.lower() or "missing" in result.output.lower()

5. Test Coverage Goals

5.1 Minimum Tests Per Command

Command Type	Min Tests	Coverage Target
Simple command (no args)	8	95%+
Command with 1-3 options	12	95%+
Command with 4-8 options	18	95%+
Command with 9+ options	25	95%+
Command group (no subcommands)	5	95%+
Subcommand (simple)	10	95%+
Subcommand (complex)	15	95%+

5.2 Test Distribution

Total Test Count Target: 300-350 tests

Breakdown by category:

Syntax validation: ~100 tests (30%)
Option combinations: ~120 tests (35%)
Error handling: ~80 tests (24%)
Help text: ~40 tests (12%)
Alias tests: ~10 tests (3%)

Breakdown by command complexity:

Core commands (3): ~15 tests
Simple commands (10): ~120 tests (12 per command)
Medium commands (12): ~216 tests (18 per command)
Complex commands (5): ~125 tests (25 per command)
Command groups (6 groups, 33 subcommands): ~190 tests (~5 per subcommand)

Total: ~666 tests (exceeds minimum to ensure exhaustive coverage)

5.3 Priority Levels

Priority 1 (Implement First) - Core functionality:

azlin new / azlin vm / azlin create (provisioning)
azlin list (information)
azlin connect (connection)
azlin kill / azlin destroy (cleanup)
azlin help / azlin --version (core)

Priority 2 (Implement Second) - Common operations: 6. azlin start / azlin stop (lifecycle) 7. azlin status (information) 8. azlin cp / azlin sync (file operations) 9. azlin session (connection) 10. azlin killall (cleanup)

Priority 3 (Implement Third) - Advanced features: 11. azlin batch group (batch operations) 12. azlin env group (environment management) 13. azlin storage group (storage management) 14. azlin snapshot group (snapshot management) 15. azlin template group (template management)

Priority 4 (Implement Last) - Specialized: 16. azlin keys group (key management) 17. azlin cost (monitoring) 18. azlin update / azlin os-update (maintenance) 19. azlin prune (cleanup) 20. azlin do / azlin doit (automation)

6. Success Criteria

6.1 Quantitative Metrics

Test Count: Minimum 300 tests (target 350+)
Coverage: 95%+ line coverage for CLI module
Pass Rate: 100% of tests pass
Execution Time: Full suite completes in < 30 seconds
Flakiness: 0% flaky tests (100% deterministic)

6.2 Qualitative Metrics

Completeness: Every command has tests for all categories
Clarity: Test names clearly describe what is being tested
Maintainability: Tests follow consistent patterns
Documentation: Each test has clear docstring
Independence: Tests don't depend on each other
Determinism: Tests produce same results every run

6.3 Coverage Requirements

Per Command:

All arguments tested (positional and optional)
All option combinations tested
All error conditions tested
Help text verified
Aliases verified (where applicable)

Per Test File:

Minimum 4 test classes per command
Minimum 8 tests per command
All code paths exercised
All error messages validated

7. Test Fixtures and Utilities

7.1 Required Fixtures (add to conftest.py)

@pytest.fixture
def cli_runner():
    """CliRunner instance for testing Click commands."""
    return CliRunner()


@pytest.fixture
def mock_vm_manager():
    """Mock VMManager for testing without Azure API calls."""
    with patch("azlin.vm_manager.VMManager") as mock:
        mock_instance = Mock()
        mock_instance.create_vm.return_value = {
            "name": "test-vm",
            "status": "Running",
            "ip": "20.123.45.67"
        }
        mock_instance.list_vms.return_value = [
            {"name": "vm1", "status": "Running"},
            {"name": "vm2", "status": "Stopped"}
        ]
        mock_instance.get_vm.return_value = {"name": "test-vm", "status": "Running"}
        mock_instance.delete_vm.return_value = True
        mock.return_value = mock_instance
        yield mock


@pytest.fixture
def mock_azure_cli_success():
    """Mock successful Azure CLI commands."""
    with patch("subprocess.run") as mock:
        mock.return_value = Mock(
            returncode=0,
            stdout='{"id": "/subscriptions/test", "name": "test-vm"}',
            stderr=""
        )
        yield mock


@pytest.fixture
def mock_azure_cli_failure():
    """Mock failed Azure CLI commands."""
    with patch("subprocess.run") as mock:
        mock.return_value = Mock(
            returncode=1,
            stdout="",
            stderr="Error: Resource not found"
        )
        yield mock


@pytest.fixture
def sample_vm_list():
    """Sample VM list for testing list/status commands."""
    return [
        {
            "name": "test-vm-1",
            "status": "Running",
            "resource_group": "test-rg",
            "location": "eastus",
            "size": "Standard_D2s_v3",
            "public_ip": "20.123.45.67"
        },
        {
            "name": "test-vm-2",
            "status": "Stopped",
            "resource_group": "test-rg",
            "location": "westus",
            "size": "Standard_B2s",
            "public_ip": None
        }
    ]


@pytest.fixture
def invalid_option_values():
    """Common invalid option values for testing validation."""
    return {
        "empty_string": "",
        "whitespace_only": "   ",
        "invalid_url": "not-a-url",
        "invalid_region": "mars-central-1",
        "invalid_vm_size": "NotARealVMSize",
        "negative_number": "-5",
        "non_integer": "abc123",
        "special_chars": "test@#$%^&*()"
    }

7.2 Helper Functions

def assert_cli_error(result, expected_message_fragment):
    """Assert that CLI invocation failed with expected error message."""
    assert result.exit_code != 0
    assert expected_message_fragment.lower() in result.output.lower()


def assert_cli_success(result):
    """Assert that CLI invocation succeeded."""
    assert result.exit_code == 0
    assert "error" not in result.output.lower()


def assert_help_text_complete(result, expected_options):
    """Assert that help text documents all expected options."""
    assert result.exit_code == 0
    assert "Usage:" in result.output
    for option in expected_options:
        assert option in result.output, f"Option {option} not in help text"


def invoke_with_mock(runner, command_args, mock_target, mock_return_value=None):
    """Invoke CLI with mock for testing."""
    with patch(mock_target) as mock:
        if mock_return_value:
            mock.return_value = mock_return_value
        result = runner.invoke(main, command_args)
        return result, mock

8. Implementation Guidelines

8.1 Development Process

Phase 1: Set up test infrastructure
- Create directory structure
- Add CLI-specific fixtures to conftest.py
- Create helper utilities
Phase 2: Implement Priority 1 commands (core)
- Write tests for each command following template
- Aim for 100% pass rate before moving on
- Target: ~60 tests
Phase 3: Implement Priority 2 commands (common)
- Focus on most-used commands
- Target: ~90 additional tests
Phase 4: Implement Priority 3 commands (advanced)
- Command groups with subcommands
- Target: ~120 additional tests
Phase 5: Implement Priority 4 commands (specialized)
- Less common but still important
- Target: ~60 additional tests
Phase 6: Integration tests
- Complex option combinations across commands
- Command chaining scenarios
- Target: ~20 additional tests

8.2 Testing Best Practices

Use CliRunner consistently: All CLI tests should use Click's CliRunner
Mock external dependencies: Never make real Azure API calls
Test in isolation: Each test should be independent
Use descriptive names: Test names should clearly state what is tested
Keep tests focused: One assertion concept per test
Document expected behavior: Clear docstrings for each test
Avoid test duplication: Use fixtures and helper functions
Test error messages: Verify exact error message content
Test exit codes: Verify correct exit code for success/failure
Test stdout/stderr: Verify output goes to correct stream

8.3 CI/CD Integration

# Add to .github/workflows/test.yml
- name: Run CLI syntax tests
  run: |
    pytest tests/unit/cli/ \
      --cov=src/azlin/cli.py \
      --cov-report=term-missing \
      --cov-fail-under=95 \
      -v

- name: Generate CLI test report
  run: |
    pytest tests/unit/cli/ \
      --junitxml=reports/cli-tests.xml \
      --html=reports/cli-tests.html

9. Example Test Files

9.1 test_new_command.py (Full Example)

"""Unit tests for 'azlin new' command syntax and behavior."""

from unittest.mock import patch, Mock
import pytest
from click.testing import CliRunner
from azlin.cli import main


class TestNewCommandSyntax:
    """Test basic syntax validation for 'azlin new' command."""

    def test_no_arguments_shows_help(self):
        """Test that 'azlin new' with no arguments shows help text."""
        runner = CliRunner()
        result = runner.invoke(main, ["new"])

        assert result.exit_code == 0
        assert "Usage:" in result.output or "Options:" in result.output

    def test_with_repo_only_is_valid(self):
        """Test that --repo alone is valid minimal syntax."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm") as mock_create:
            result = runner.invoke(main, ["new", "--repo", "https://github.com/user/repo"])

            assert mock_create.called
            assert "Error: Missing" not in result.output

    def test_empty_string_repo_shows_error(self):
        """Test that empty --repo value shows validation error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", ""])

        assert result.exit_code != 0
        assert "empty" in result.output.lower() or "invalid" in result.output.lower()

    def test_whitespace_only_repo_shows_error(self):
        """Test that whitespace-only --repo value shows error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", "   "])

        assert result.exit_code != 0

    def test_with_all_valid_options(self):
        """Test command with all valid options specified."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm"):
            result = runner.invoke(main, [
                "new",
                "--repo", "https://github.com/user/repo",
                "--vm-size", "Standard_D2s_v3",
                "--region", "eastus",
                "--resource-group", "test-rg",
                "--name", "custom-vm",
                "--config", "/path/to/config.yaml",
                "--template", "python-dev",
                "--nfs-storage", "my-storage",
                "--no-auto-connect"
            ])

            assert result.exit_code == 0


class TestNewCommandOptions:
    """Test option combinations for 'azlin new' command."""

    def test_pool_and_name_are_mutually_exclusive(self):
        """Test that --pool and --name cannot be used together."""
        runner = CliRunner()
        result = runner.invoke(main, [
            "new",
            "--repo", "https://github.com/user/repo",
            "--pool", "3",
            "--name", "custom-vm"
        ])

        assert result.exit_code != 0
        assert "mutually exclusive" in result.output.lower() or "cannot" in result.output.lower()

    def test_resource_group_short_and_long_form_equivalent(self):
        """Test that -rg and --resource-group are equivalent."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm") as mock_create:
            # Long form
            runner.invoke(main, ["new", "--repo", "https://...", "--resource-group", "test-rg"])
            long_form_call = mock_create.call_args

            mock_create.reset_mock()

            # Short form
            runner.invoke(main, ["new", "--repo", "https://...", "-rg", "test-rg"])
            short_form_call = mock_create.call_args

            # Both should pass same resource_group
            assert long_form_call == short_form_call

    def test_pool_with_positive_integer_succeeds(self):
        """Test that --pool accepts positive integers."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm"):
            result = runner.invoke(main, ["new", "--repo", "https://...", "--pool", "5"])

            assert result.exit_code == 0

    def test_pool_with_zero_shows_error(self):
        """Test that --pool 0 shows validation error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", "https://...", "--pool", "0"])

        assert result.exit_code != 0

    def test_pool_with_negative_shows_error(self):
        """Test that --pool with negative number shows error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", "https://...", "--pool", "-1"])

        assert result.exit_code != 0

    def test_pool_with_non_integer_shows_error(self):
        """Test that --pool with non-integer shows error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", "https://...", "--pool", "abc"])

        assert result.exit_code != 0
        assert "integer" in result.output.lower() or "invalid" in result.output.lower()

    def test_config_file_path_validation(self):
        """Test that --config validates file path format."""
        runner = CliRunner()

        with patch("azlin.config_manager.ConfigManager.load_config"):
            result = runner.invoke(main, ["new", "--repo", "https://...", "--config", "/valid/path.yaml"])

            # Should not fail on path format
            assert "invalid path" not in result.output.lower()

    def test_no_auto_connect_flag(self):
        """Test that --no-auto-connect flag works."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm") as mock_create:
            result = runner.invoke(main, ["new", "--repo", "https://...", "--no-auto-connect"])

            assert mock_create.called
            # Verify no_auto_connect was passed
            call_kwargs = mock_create.call_args[1]
            assert call_kwargs.get("no_auto_connect") is True


class TestNewCommandErrors:
    """Test error handling for 'azlin new' command."""

    def test_unknown_option_shows_error(self):
        """Test that unknown options show helpful error message."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--unknown-option", "value"])

        assert result.exit_code != 0
        assert "no such option" in result.output.lower()

    def test_invalid_vm_size_shows_error(self):
        """Test that invalid VM size shows clear error."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm", side_effect=ValueError("Invalid VM size")):
            result = runner.invoke(main, ["new", "--repo", "https://...", "--vm-size", "InvalidSize"])

            assert result.exit_code != 0
            assert "invalid" in result.output.lower()

    def test_invalid_region_shows_error(self):
        """Test that invalid region shows clear error."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm", side_effect=ValueError("Invalid region")):
            result = runner.invoke(main, ["new", "--repo", "https://...", "--region", "invalid-region"])

            assert result.exit_code != 0
            assert "region" in result.output.lower()

    def test_invalid_repo_url_shows_error(self):
        """Test that invalid repository URL shows error."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--repo", "not-a-valid-url"])

        # May fail at validation or during execution
        # Just ensure error is shown
        assert result.exit_code != 0 or "error" in result.output.lower()

    def test_missing_azure_credentials_shows_helpful_message(self):
        """Test that missing Azure credentials shows helpful error."""
        runner = CliRunner()

        with patch("azure.identity.DefaultAzureCredential", side_effect=Exception("No credentials found")):
            result = runner.invoke(main, ["new", "--repo", "https://..."])

            assert result.exit_code != 0
            assert "credential" in result.output.lower() or "auth" in result.output.lower()

    def test_azure_quota_exceeded_shows_clear_error(self):
        """Test that Azure quota errors show clear message."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm", side_effect=Exception("QuotaExceeded")):
            result = runner.invoke(main, ["new", "--repo", "https://..."])

            assert result.exit_code != 0
            assert "quota" in result.output.lower()

    def test_network_timeout_shows_error(self):
        """Test that network timeouts show appropriate error."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm", side_effect=TimeoutError("Connection timeout")):
            result = runner.invoke(main, ["new", "--repo", "https://..."])

            assert result.exit_code != 0
            assert "timeout" in result.output.lower() or "connection" in result.output.lower()


class TestNewCommandHelp:
    """Test help text for 'azlin new' command."""

    def test_help_flag_shows_complete_usage(self):
        """Test that --help shows complete usage information."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--help"])

        assert result.exit_code == 0
        assert "Usage:" in result.output
        assert "Options:" in result.output

    def test_help_documents_all_options(self):
        """Test that help text includes all available options."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--help"])

        expected_options = [
            "--repo",
            "--vm-size",
            "--region",
            "--resource-group",
            "--name",
            "--pool",
            "--no-auto-connect",
            "--config",
            "--template",
            "--nfs-storage"
        ]

        for option in expected_options:
            assert option in result.output, f"Option {option} not documented"

    def test_help_includes_usage_examples(self):
        """Test that help text includes usage examples."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "--help"])

        # Check for example patterns
        assert "example" in result.output.lower() or "azlin new" in result.output

    def test_short_help_flag_works(self):
        """Test that -h is alias for --help."""
        runner = CliRunner()
        result = runner.invoke(main, ["new", "-h"])

        assert result.exit_code == 0
        assert "Usage:" in result.output


class TestNewCommandAliases:
    """Test aliases for 'azlin new' command (vm, create)."""

    def test_new_and_vm_are_equivalent(self):
        """Test that 'azlin new' and 'azlin vm' produce same result."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm") as mock_create:
            result_new = runner.invoke(main, ["new", "--repo", "https://..."])
            new_exit_code = result_new.exit_code

            mock_create.reset_mock()

            result_vm = runner.invoke(main, ["vm", "--repo", "https://..."])
            vm_exit_code = result_vm.exit_code

            assert new_exit_code == vm_exit_code

    def test_new_and_create_are_equivalent(self):
        """Test that 'azlin new' and 'azlin create' produce same result."""
        runner = CliRunner()

        with patch("azlin.vm_manager.VMManager.create_vm") as mock_create:
            result_new = runner.invoke(main, ["new", "--repo", "https://..."])
            result_create = runner.invoke(main, ["create", "--repo", "https://..."])

            assert result_new.exit_code == result_create.exit_code

10. Validation and Reporting

10.1 Test Metrics Dashboard

Track these metrics:

Total tests written vs target (300+)
Tests passing vs failing
Coverage percentage per command
Average tests per command
Distribution across categories

10.2 Coverage Report Format

CLI Command Syntax Test Coverage Report
========================================

Overall Statistics:
  Total Commands: 33
  Commands with tests: 33 (100%)
  Total tests: 342
  Tests passing: 342 (100%)
  Average tests per command: 10.4

Coverage by Command:
  azlin new:           25 tests (18 passing, 95% coverage)
  azlin list:          12 tests (12 passing, 98% coverage)
  azlin connect:       15 tests (15 passing, 96% coverage)
  ...

Coverage by Category:
  Syntax validation:   102 tests (30%)
  Option combinations: 118 tests (35%)
  Error handling:      85 tests (25%)
  Help text:           37 tests (11%)
  Aliases:             10 tests (3%)

Priority 1 (Core):     60 tests (100% complete)
Priority 2 (Common):   90 tests (100% complete)
Priority 3 (Advanced): 120 tests (100% complete)
Priority 4 (Specialized): 72 tests (100% complete)

11. Maintenance and Updates

11.1 When to Update Tests

New CLI command added: Add full test suite for command
Command option added: Add tests for new option and combinations
Command behavior changed: Update existing tests
Bug fixed: Add regression test
Error message improved: Update error assertion tests

11.2 Test Review Checklist

Before marking test suite complete:

Summary

This specification provides a complete blueprint for creating 300+ exhaustive tests covering all 33 CLI commands. The structured approach ensures:

Comprehensive coverage: Every command, option, and error condition tested
Consistent patterns: Reusable templates and fixtures
Maintainability: Clear organization and naming conventions
Quality assurance: Success criteria and validation metrics
Prioritization: Phased implementation plan

By following this specification, the test suite will provide confidence that all CLI commands work correctly across all syntax variations and error conditions.

FilesExpand file tree

TEST_SUITE_SPECIFICATION.md

Latest commit

History

TEST_SUITE_SPECIFICATION.md

File metadata and controls

Exhaustive CLI Command Syntax Test Suite Specification

Document Metadata

1. Test Organization Structure

1.1 Directory Layout

1.2 File Naming Conventions

2. Command Inventory

2.1 Primary Commands (27)

2.2 Command Groups (6)

3. Test Categories

3.1 Syntax Validation Tests (5-8 tests per command)

3.2 Option Combination Tests (8-12 tests per command)

3.3 Alias Tests (2-3 tests per alias)

3.4 Error Handling Tests (6-10 tests per command)

3.5 Help Text Tests (3-4 tests per command)

4. Test Patterns and Templates

4.1 Basic Command Test Template

4.2 Command Group Test Template

5. Test Coverage Goals

5.1 Minimum Tests Per Command

5.2 Test Distribution

5.3 Priority Levels

6. Success Criteria

6.1 Quantitative Metrics

6.2 Qualitative Metrics

6.3 Coverage Requirements

7. Test Fixtures and Utilities

7.1 Required Fixtures (add to conftest.py)

7.2 Helper Functions

8. Implementation Guidelines

8.1 Development Process

8.2 Testing Best Practices

8.3 CI/CD Integration

9. Example Test Files

9.1 test_new_command.py (Full Example)

10. Validation and Reporting

10.1 Test Metrics Dashboard

10.2 Coverage Report Format

11. Maintenance and Updates

11.1 When to Update Tests

11.2 Test Review Checklist

Summary