feat: add smoke-test-aws CLI command with full lifecycle test

[abrichr]

Summary

• Add oa-vm smoke-test-aws command that runs incremental verification stages against real AWS infrastructure
• Fix two bugs in AWSVMManager where deallocate_vm and delete_vm returned before the instance reached its target state

Stages

#	Stage	Read-only?	What it tests
1	Credentials	yes	boto3 sts get_caller_identity
2	SSH key	yes	~/.ssh/id_rsa.pub exists
3	AMI lookup	yes	_find_latest_ubuntu_ami() returns valid AMI
4	Instance type availability	yes	find_available_size_and_region() finds offering
5	VPC infrastructure	yes*	_ensure_vpc_infrastructure() (idempotent)
6	Create VM	no	create_vm("waa-smoke-test", ...) with cheap type
7	SSH connectivity	no	wait_for_ssh() + ssh_run("hostname")
8	Stop/Start	no	deallocate_vm() → start_vm() → verify IP refresh
9	Cleanup	no	delete_vm() → verify instance terminated

Stages 6-9 only run with --full. Uses m5a.xlarge ($0.17/hr) not .metal ($4.60/hr).

Bug fixes in aws_vm.py

• deallocate_vm(): Was returning immediately after stop_instances() without waiting for stopped state. A subsequent start_vm() would fail with IncorrectInstanceState because the instance was still stopping.
• delete_vm(): Was returning immediately after terminate_instances() without waiting for terminated state. Callers couldn't verify the instance was actually gone.

Both now use boto3 waiters (instance_stopped / instance_terminated), matching how start_vm() already used instance_running.

Verified: 9/9 stages passed

Full lifecycle test output against real AWS (us-east-1), ~1m42s total:

[2026-03-02 15:31:18] [SMOKE-AWS] ============================================================
[2026-03-02 15:31:18] [SMOKE-AWS] AWS Smoke Test (full lifecycle)
[2026-03-02 15:31:18] [SMOKE-AWS] ============================================================
[2026-03-02 15:31:18] [SMOKE-AWS]   Account: ************, ARN: arn:aws:iam::************:user/openadapt
[2026-03-02 15:31:18] [SMOKE-AWS] Stage 1: AWS credentials ... PASS
[2026-03-02 15:31:18] [SMOKE-AWS]   Key: /Users/abrichr/.ssh/id_rsa.pub (380 bytes)
[2026-03-02 15:31:18] [SMOKE-AWS] Stage 2: SSH public key ... PASS
[2026-03-02 15:31:18] [SMOKE-AWS]   AMI: ami-04680790a315cd58d (region: us-east-1)
[2026-03-02 15:31:18] [SMOKE-AWS] Stage 3: AMI lookup ... PASS
[2026-03-02 15:31:19] [SMOKE-AWS]   Type: m5.metal, Region: us-east-1, Cost: $4.608/hr
[2026-03-02 15:31:19] [SMOKE-AWS] Stage 4: Instance type availability ... PASS
[2026-03-02 15:31:20] [SMOKE-AWS]   Subnet: subnet-XXXXXXXXXXXXXXXXX
[2026-03-02 15:31:20] [SMOKE-AWS]   Security group: sg-XXXXXXXXXXXXXXXXX
[2026-03-02 15:31:20] [SMOKE-AWS]   Key pair: waa-pool-key
[2026-03-02 15:31:20] [SMOKE-AWS] Stage 5: VPC infrastructure ... PASS
[2026-03-02 15:31:20] [SMOKE-AWS] 
[2026-03-02 15:31:20] [SMOKE-AWS] --- Full lifecycle (creates real EC2 resources) ---
[2026-03-02 15:31:20] [SMOKE-AWS]   Creating waa-smoke-test (m5a.xlarge in us-east-1)...
[2026-03-02 15:31:37] [SMOKE-AWS]   IP: XXX.XXX.XXX.XXX
[2026-03-02 15:31:37] [SMOKE-AWS] Stage 6: Create VM ... PASS
[2026-03-02 15:31:37] [SMOKE-AWS]   Waiting for SSH on XXX.XXX.XXX.XXX...
[2026-03-02 15:31:41] [SMOKE-AWS]   Hostname: ip-10-0-1-248
[2026-03-02 15:31:41] [SMOKE-AWS] Stage 7: SSH connectivity ... PASS
[2026-03-02 15:31:41] [SMOKE-AWS]   Stopping VM...
[2026-03-02 15:32:12] [SMOKE-AWS]   VM stopped. Starting VM...
[2026-03-02 15:32:29] [SMOKE-AWS]   New IP after restart: YYY.YYY.YYY.YYY
[2026-03-02 15:32:29] [SMOKE-AWS] Stage 8: Stop/Start cycle ... PASS
[2026-03-02 15:32:29] [SMOKE-AWS]   Deleting waa-smoke-test...
[2026-03-02 15:33:00] [SMOKE-AWS]   VM terminated and cleaned up.
[2026-03-02 15:33:00] [SMOKE-AWS] Stage 9: Cleanup ... PASS
[2026-03-02 15:33:00] [SMOKE-AWS] 
[2026-03-02 15:33:00] [SMOKE-AWS] ============================================================
[2026-03-02 15:33:00] [SMOKE-AWS] RESULT: 9/9 stages passed, 0 failed
[2026-03-02 15:33:00] [SMOKE-AWS] ============================================================

Usage

# Read-only (default) — verifies credentials, AMI, instance types, VPC
oa-vm smoke-test-aws

# Full lifecycle — creates/stops/starts/deletes a real EC2 instance
oa-vm smoke-test-aws --full

# Override region or instance type
oa-vm smoke-test-aws --full --region us-west-2 --instance-type m5a.large

Windows 11 running on AWS EC2

Full WAA stack verified on m5.metal in us-east-1: EC2 → Docker → QEMU/KVM → Windows 11.

Windows 11 desktop (fully booted)

Windows 11 installer (mid-install)

AWS cost analysis

Measured costs from this verification session:

Resource	Duration	Rate	Cost
Smoke test runs (3x): m5a.xlarge	~5 min total	$0.172/hr	$0.01
WAA full stack test: m5.metal	~55 min	$4.608/hr	$4.22
Data transfer	negligible	—	~$0.00
		Total	~$4.23

m5.metal time breakdown:

• VM creation + SSH: ~1 min
• Docker install: ~13 min
• Docker image build: ~7 min
• Windows download + install: ~20 min
• Idle/screenshot: ~5 min
• Cleanup: ~1 min

For comparison, the read-only smoke test (oa-vm smoke-test-aws without --full) costs $0.00 — it only makes API calls, no instances created.

Test plan

• Read-only stages pass (5/5)
• Full lifecycle stages pass (9/9)
• deallocate_vm bug fix verified (stop/start cycle now works)
• delete_vm bug fix verified (cleanup completes cleanly)
• No leftover AWS resources after test
• Full WAA stack on AWS: Windows 11 boots to desktop on m5.metal via QEMU/KVM

🤖 Generated with Claude Code