Merge pull request #72 from amazon-mq/docs/gh-71-review-all-docs

Review all documentation for technical correctness

Author: lukebakken

AGENTS.md

@@ -77,7 +77,7 @@ This plugin provides a production-ready solution for migrating mirrored classic
 ### Prerequisites (BEFORE Migration)
 
 1. **Cluster Health**: All cluster nodes must be running
-2. **Plugin Enabled**: `rabbitmq_queue_migration` and `rabbitmq_shovel` plugins loaded
+2. **Plugins Enabled**: `rabbitmq_queue_migration` and `rabbitmq_shovel` must be loaded; the `rabbitmq_queue_migration` plugin must additionally have completed its async initialization on every running node (validation chain step `plugin_ready_on_all_nodes`)
 3. **Queue Eligibility**: Only mirrored classic queues with HA policies
 4. **Khepri Disabled**: Classic Mnesia required (Khepri not compatible)
 5. **AWS Configuration**: For EBS snapshots, AWS credentials and region must be configured
@@ -233,10 +233,10 @@ GET  /api/queue-migration/status/:id      # Specific migration details
 
 ### Progress Tracking
 
-- **Update Frequency**: Configurable via `progress_update_frequency` (default: 10)
-- **Calculation**: Messages processed ÷ 4 for progress count
-- **Real-time Updates**: Database updated during message migration
-- **Percentage Calculation**: `(completed / total) * 100`
+- **Update Frequency**: Configurable via `progress_update_frequency` (default: 10 messages between status writes)
+- **Verification on completion**: After the shovel reports completion, `rqm:verify_and_update_progress/5` reads the source and destination message counts and writes the actual delivered count via `rqm_db:update_queue_status_progress/3`
+- **Real-time Updates**: Database updated as queues complete each phase
+- **Percentage Calculation**: `rqm_mgmt:calculate_progress_percentage/2` computes `trunc((completed / total) * 100)`
 
 **Note:** The setting `quorum_queue.property_equivalence.relaxed_checks_on_redeclaration = true` must be enabled **before** migration (validated during pre-migration checks). This allows applications to redeclare queues with classic arguments after migration without errors.
 
@@ -288,11 +288,13 @@ GET  /api/queue-migration/status/:id      # Specific migration details
 **System Checks:**
 - `check_shovel_plugin/0` - Verify shovel plugin enabled
 - `check_khepri_disabled/0` - Verify Khepri not enabled
+- `check_relaxed_checks_setting/0` - Verify `quorum_queue.property_equivalence.relaxed_checks_on_redeclaration` is enabled
 - `check_leader_balance/1` - Verify queue leaders balanced
-- `check_disk_space/2` - Verify sufficient disk space
+- `check_disk_space/2` - Verify sufficient disk space (signature: `(VHost, UnsuitableQueues)`)
 - `check_active_alarms/0` - Check for memory/disk alarms
 - `check_memory_usage/0` - Verify memory usage acceptable
 - `check_snapshot_not_in_progress/0` - Verify no concurrent snapshots
+- `check_plugin_ready_on_all_nodes/0` - Verify the queue migration plugin reports `ready` on every running node
 - `check_cluster_partitions/0` - Verify no network partitions
 
 **Queue Checks:**
@@ -317,6 +319,10 @@ GET  /api/queue-migration/status/:id      # Specific migration details
 **Key Functions:**
 - All configuration getters return values from application environment or defaults
 - See Configuration Parameters section for complete list
+
+### `rqm_gatherer.erl` (Distributed Result Collection)
+
+**Key Functions:**
 - `fork/1` - Declare intent to produce results (increment producer count)
 - `finish/1` - Signal completion of work (decrement producer count)
 - `in/2` - Add result to gatherer queue (async)
@@ -375,10 +381,11 @@ GET  /api/queue-migration/status/:id      # Specific migration details
 - `check_leader_balance/1` - Ensure balanced queue distribution
 - `check_queue_synchronization/1` - Verify all mirrors are synchronized
 - `check_queue_suitability/1` - Comprehensive queue eligibility
-- `check_disk_space/1` - Disk space estimation and validation based on worker pool concurrency
+- `check_disk_space/2` - Disk space estimation and validation (signature: `(VHost, UnsuitableQueues)`); required free = total queue data x peak multiplier + min disk space buffer
 - `check_active_alarms/0` - Check for active RabbitMQ alarms
 - `check_memory_usage/0` - Validate memory usage is within limits
 - `check_snapshot_not_in_progress/0` - Verify no concurrent EBS snapshots (delegates to `rqm_snapshot`)
+- `check_plugin_ready_on_all_nodes/0` - Verify the queue migration plugin is `ready` on every running cluster node
 - `check_cluster_partitions/0` - Verify no cluster partitions and all nodes up
 - `check_system_migration_readiness/1` - Overall system readiness
 
@@ -393,9 +400,6 @@ GET  /api/queue-migration/status/:id      # Specific migration details
 - `format_migration_readiness_response/1` - Format readiness check response
 - `format_system_checks_for_ui/1` - Format system checks for web UI
 - `format_queue_checks_for_ui/1` - Format queue checks for web UI
-- `check_queue_suitability/1` - Comprehensive queue eligibility
-- `check_disk_space/1` - Disk space estimation and validation
-- `check_system_migration_readiness/1` - Overall system readiness
 
 ## Configuration Parameters
 
@@ -674,7 +678,7 @@ Performance varies significantly based on:
 ### AWS Configuration Requirements
 - **Credentials**: AWS credentials via environment variables, config files, or EC2 instance roles
 - **Region**: AWS region configuration (defaults to us-east-1, should match peer discovery)
-- **Permissions**: EC2 permissions for `CreateSnapshot`, `DescribeVolumes`, `DescribeSnapshots`, and `DescribeInstances`
+- **Permissions**: EC2 permissions for `ec2:DescribeVolumes`, `ec2:CreateSnapshot`, `ec2:DescribeSnapshots`, and `ec2:DeleteSnapshot`. The plugin does not call `ec2:CreateTags` or `ec2:DescribeInstances`.
 - **Volume Setup**: RabbitMQ data directory must be on EBS volume at configured device path
 
 ### Snapshot Timing
@@ -693,7 +697,7 @@ Performance varies significantly based on:
 
 ### Skip Unsuitable Queues Feature
 - **Skip Mode**: `skip_unsuitable_queues` parameter allows migration to proceed by skipping problematic queues
-- **Skip Reasons**: Tracks why queues were skipped (unsynchronized, too_many_queues, unsuitable_overflow, interrupted)
+- **Skip Reasons**: Tracks why queues were skipped. Validation produces five reasons via `#unsuitable_queue{reason = ...}` (`unsynchronized`, `too_many_queues`, `unsuitable_overflow`, `queue_expires`, `message_ttl`); a sixth `interrupted` status is recorded for queues from a previously interrupted migration
 - **Integration Tests**: SkipUnsuitableTest validates feature works correctly
 
 ### Batch Migration Feature

CODE_OF_CONDUCT.md

@@ -1,4 +1,4 @@
-## Code of Conduct
+# Code of Conduct
 This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
 For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
 opensource-codeofconduct@amazon.com with any additional questions or comments.

CONTRIBUTING.md

@@ -17,7 +17,7 @@ To build this plugin in your RabbitMQ development environment:
 git clone https://github.com/rabbitmq/rabbitmq-server.git
 cd rabbitmq-server
 git clone https://github.com/amazon-mq/rabbitmq-queue-migration.git deps/rabbitmq_queue_migration
-make -C deps/aws
+make -C deps/rabbitmq_queue_migration
 ```
 
 To run RabbitMQ with this plugin:

README.md

@@ -4,8 +4,6 @@ A RabbitMQ plugin for migrating mirrored classic queues to quorum queues in Rabb
 
 ## Overview
 
-A RabbitMQ plugin for migrating mirrored classic queues to quorum queues in RabbitMQ 3.13.x clusters.
-
 - Two-phase migration algorithm with message-by-message transfer
 - Automatic binding preservation and rollback support
 - Selective migration by queue name or batch size
@@ -44,6 +42,7 @@ Restarting any broker node, performing a full cluster reboot, or applying a main
 - [Configuration Reference](docs/CONFIGURATION.md) - Configuration parameter reference
 - [EC2 Setup](docs/EC2_SETUP.md) - AWS EC2 and IAM configuration if EBS snapshots are used
 - [Performance Reference](docs/PERFORMANCE.md) - Real-world migration timing data
+- [Troubleshooting Guide](docs/TROUBLESHOOTING.md) - Error messages, diagnostics, and recovery
 
 See [the `docs/` directory](https://github.com/amazon-mq/rabbitmq-queue-migration/tree/main/docs).
 

RELEASE.md

@@ -31,8 +31,8 @@ git push --tags
 * Ensure that Erlang/OTP 26.x and Elixir 1.16.x are in your `PATH`
 
 ```
-git clone --branch v4.2.x https://github.com/rabbitmq/rabbitmq-server.git rabbitmq-server_4.2.x
-cd rabbitmq-server_4.2.x
+git clone --branch v3.13.7 https://github.com/rabbitmq/rabbitmq-server.git rabbitmq-server_v3.13.7
+cd rabbitmq-server_v3.13.7
 git clone --branch "$VER" https://github.com/amazon-mq/rabbitmq-queue-migration.git deps/rabbitmq_queue_migration
 make
 make -C deps/rabbitmq_queue_migration

docs/API_EXAMPLES.md

@@ -1,7 +1,5 @@
 # API Examples - RabbitMQ Queue Migration Plugin
 
-**Last Updated:** January 21, 2026
-
 This document provides practical examples for all HTTP API endpoints with actual response structures verified against a running broker.
 
 ---
@@ -322,7 +320,7 @@ curl -u guest:guest \
       "total_queues": 50,
       "completed_queues": 45,
       "skipped_queues": 5,
-      "progress_percentage": 100,
+      "progress_percentage": 90,
       "skip_unsuitable_queues": true,
       "tolerance": 10.0,
       "error": null
@@ -383,7 +381,7 @@ curl -u guest:guest \
     "total_queues": 50,
     "completed_queues": 45,
     "skipped_queues": 5,
-    "progress_percentage": 100,
+    "progress_percentage": 90,
     "skip_unsuitable_queues": true,
     "error": null,
     "snapshots": [

docs/CONFIGURATION.md

@@ -1,7 +1,5 @@
 # Configuration Reference
 
-**Last Updated:** January 21, 2026
-
 Complete reference for all configuration parameters in the RabbitMQ Queue Migration Plugin.
 
 ---
@@ -114,9 +112,14 @@ queue_migration.disk_usage_peak_multiplier = 3.0
 
 **Calculation:**
 ```
-Required space = (concurrent_workers × avg_queue_size × multiplier) + buffer
+Required space = (total_queue_data × multiplier) + min_disk_space_buffer
 ```
 
+Where:
+- `total_queue_data` = sum of disk usage across all migratable classic queues in the vhost (computed by `rqm_checks:estimate_migration_disk_usage/2`)
+- `multiplier` = the value of this setting
+- `min_disk_space_buffer` = the value of the [`min_disk_space_buffer`](#min_disk_space_buffer) setting (default 500 MB)
+
 ---
 
 ## Memory Configuration

docs/EC2_SETUP.md

@@ -11,7 +11,6 @@ The RabbitMQ queue migration plugin requires EC2 API access to:
 - Query attached EBS volumes (`ec2:DescribeVolumes`)
 - Create snapshots of those volumes (`ec2:CreateSnapshot`)
 - Check snapshot status (`ec2:DescribeSnapshots`)
-- Tag snapshots for identification (`ec2:CreateTags`)
 - Delete snapshots after successful migration (`ec2:DeleteSnapshot`)
 
 When running on EC2 instances, the `rabbitmq_aws` SDK automatically retrieves temporary credentials (including session tokens) from the
@@ -77,7 +76,6 @@ aws iam put-role-policy \
         "ec2:DescribeVolumes",
         "ec2:CreateSnapshot",
         "ec2:DescribeSnapshots",
-        "ec2:CreateTags",
         "ec2:DeleteSnapshot"
       ],
       "Resource": "*"
@@ -97,7 +95,6 @@ restricting permissions to specific volumes or using condition keys to limit sco
       "ec2:DescribeVolumes",
       "ec2:CreateSnapshot",
       "ec2:DescribeSnapshots",
-      "ec2:CreateTags",
       "ec2:DeleteSnapshot"
     ],
     "Resource": "*",
@@ -232,8 +229,7 @@ VOLUME_ID=$(aws ec2 describe-volumes \
 aws ec2 create-snapshot \
   --region us-east-1 \
   --volume-id "$VOLUME_ID" \
-  --description "Test snapshot for RabbitMQ queue migration" \
-  --tag-specifications "ResourceType=snapshot,Tags=[{Key=Purpose,Value=Test}]"
+  --description "Test snapshot for RabbitMQ queue migration"
 ```
 
 If successful, you'll receive a snapshot ID. Clean up the test snapshot:
@@ -307,9 +303,10 @@ The `rabbitmq_queue_migration` plugin performs the following operations during m
 
 1. **Query Volumes**: Calls `ec2:DescribeVolumes` to find EBS volumes attached to the instance
 2. **Create Snapshots**: Calls `ec2:CreateSnapshot` to create crash-consistent snapshots before migration
-3. **Tag Snapshots**: Calls `ec2:CreateTags` to label snapshots with migration metadata
-4. **Monitor Progress**: Calls `ec2:DescribeSnapshots` to check snapshot completion status
-5. **Cleanup Snapshots**: Calls `ec2:DeleteSnapshot` to remove snapshots after successful migration (when `cleanup_snapshots_on_success` is enabled)
+3. **Monitor Progress**: Calls `ec2:DescribeSnapshots` to check snapshot completion status
+4. **Cleanup Snapshots**: Calls `ec2:DeleteSnapshot` to remove snapshots after successful migration (when `cleanup_snapshots_on_success` is enabled)
+
+Snapshots are created with a `Description` of the form `RabbitMQ migration snapshot <timestamp> on <node>`, but the plugin does not attach EC2 tags to them (`ec2:CreateTags` is not required).
 
 The plugin uses the AWS SDK's default credential provider chain, which automatically:
 - Queries the EC2 instance metadata service at `http://169.254.169.254/latest/meta-data/iam/security-credentials/`
@@ -340,8 +337,7 @@ EBS snapshots incur storage costs. Monitor snapshot usage:
 ```bash
 aws ec2 describe-snapshots \
   --owner-ids self \
-  --filters "Name=tag:Purpose,Values=RabbitMQ-Migration" \
-  --query 'Snapshots[].[SnapshotId,VolumeSize,StartTime]' \
+  --query 'Snapshots[?starts_with(Description, `RabbitMQ migration snapshot`)].[SnapshotId,VolumeSize,StartTime,Description]' \
   --output table
 ```
 
@@ -350,7 +346,7 @@ aws ec2 describe-snapshots \
 After completing these steps, your RabbitMQ EC2 instances will have the necessary IAM permissions to:
 - Query attached EBS volumes
 - Create snapshots during queue migration
-- Tag and monitor snapshot progress
+- Monitor snapshot status and delete snapshots after migration completes
 
 The `rabbitmq_aws` library used by this plugin will automatically retrieve temporary credentials from the instance metadata
 service, including the required session token, without any explicit configuration.

docs/HTTP_API.md

@@ -286,17 +286,17 @@ curl -u guest:guest http://localhost:15672/api/queue-migration/status
 - `completed_queues` - Queues completed so far
 - `skipped_queues` - Queues skipped
 - `progress_percentage` - Progress (0-100)
-- `status` - Migration status (see values below)
-- `skip_unsuitable_queues` - Whether skip mode was enabled
-- `tolerance` - Message count tolerance percentage (null if not set)
-- `error` - Error details (null if no error)
+- `status` - Migration status:
   - `pending` - Not started yet
   - `in_progress` - Currently migrating
   - `completed` - Successfully completed
   - `failed` - Migration failed
   - `interrupted` - Migration was interrupted
   - `rollback_pending` - Requires rollback
   - `rollback_completed` - Rollback completed
+- `skip_unsuitable_queues` - Whether skip mode was enabled
+- `tolerance` - Message count tolerance percentage (null if not set)
+- `error` - Error details (null if no error)
 
 ---
 
@@ -685,10 +685,12 @@ curl -u guest:guest http://localhost:15672/api/queue-migration/rollback-pending
 
 ## Migration ID Format
 
-Migration IDs are base64url-encoded Erlang terms containing:
-- Timestamp
-- Node name
-- Unique identifier
+Migration IDs are base64url-encoded Erlang terms. Each ID encodes a 2-tuple of:
+
+- **Timestamp** - milliseconds since the epoch (`erlang:system_time(millisecond)`)
+- **Node name** - the Erlang node atom on which the migration was started (`node()`)
+
+Source: `rqm_util:generate_migration_id/0` and `rqm_util:format_migration_id/1`.
 
 **Example:**
 ```

docs/INTEGRATION_TESTING.md

@@ -35,15 +35,30 @@ This will:
 1. Build the plugin as an ez file
 2. Create custom Docker images with the plugin
 3. Start a 3-node RabbitMQ cluster
-4. Run end-to-end migration test (10 queues, 1000 messages)
-5. Validate migration success
+4. Run six end-to-end test scenarios against that cluster:
+   - `end-to-end` - default migration (10 queues, 1000 messages)
+   - `interrupt-test` - interrupting a migration mid-flight and resuming
+   - `skip-unsuitable-test` - migration with `skip_unsuitable_queues=true`
+   - `batch-test` - batch migration
+   - `empty-queue-test` - migration of zero-message queues (fast path)
+   - `already-quorum-test` - migration when destination type is already quorum
+5. Validate each scenario's success
 6. Stop the cluster
 
-**Duration:** ~2-3 minutes
+**Duration:** ~10-15 minutes for the full six-scenario run; the
+`end-to-end` scenario alone is ~2-3 minutes.
 
 ## What Gets Tested
 
-### Test Phases
+The `integration-test` target runs six scenarios against the same
+3-node Docker cluster: `end-to-end`, `interrupt-test`,
+`skip-unsuitable-test`, `batch-test`, `empty-queue-test`, and
+`already-quorum-test`. The phase breakdown below describes the
+`end-to-end` scenario, which is the most thorough; the other
+scenarios reuse most of the same setup and validation but with
+narrower payloads or specific options.
+
+### Test Phases (`end-to-end`)
 
 **Phase 0: Cleanup**
 - Removes previous test artifacts (queues, exchanges, policies)
@@ -89,7 +104,7 @@ Setup complete in 5165 ms
 Pre-migration: 10 classic queues, 1000 total messages
 
 === Phase 3: Triggering queue migration ===
-Migration started: g2gCbgYA8U2NKJsBdxhyYWJiaXQtMUBTRUEtM0xHNUhWSlVXSks
+Migration started: <base64-encoded-migration-id>
 
 === Phase 4: Monitoring migration progress ===
 [00:00:05] Migration in progress: 20% (2/10 queues)
@@ -185,7 +200,7 @@ The integration test supports various configurations. See [test/integration/READ
 ### Hostname Resolution Fails
 
 ```
-ERROR: node0: Temporary failure in name resolution
+ERROR: rmq0: Temporary failure in name resolution
 ```
 
 **Solution:** Add `/etc/hosts` entries for rmq0, rmq1, rmq2