PXC-5141 - [DOCS] - [feedback] It doesn't work with podman compose 8.4

patrickbirch · patrickbirch · commit ca7cc38dab43 · 2026-03-19T07:24:43.000-05:00
modified:   docs/docker-compose.md
diff --git a/docs/docker-compose.md b/docs/docker-compose.md
@@ -8,26 +8,21 @@ We gather [Telemetry data](telemetry.md) in the Percona packages and Docker imag
 
 --8<--- "get-help-snip.md"
 
-This guide shows you how to deploy a three-node Percona XtraDB Cluster 8.4 using Docker Compose. You generate SSL certificates on the first node and copy them to the other two nodes to enable secure communication.
+The guide shows you how to deploy a three-node Percona XtraDB Cluster 8.4 using Docker Compose. You generate SSL certificates on the first node and copy them to the other two nodes to enable secure communication.
 
-The following procedure describes setting up a simple 3-node cluster
-for evaluation and testing purposes. Do not use these instructions in a
-production environment because the MySQL certificates generated in this
-procedure are self-signed.
-
-In a production environment, you should generate and store the certificates to be used by Docker and configure proper storage, security, backup, and monitoring systems.
+The following procedure is for evaluation and testing only. Do not use these instructions in a production environment because the MySQL certificates generated here are self-signed. In production, generate and store proper certificates and configure storage, security, backup, and monitoring.
 
 ## Prerequisites
 
-* Docker and Docker Compose installed
+* Docker and Docker Compose installed (or Podman 4.1 or later and Podman Compose; see [Appendix: Podman Alternative](#appendix-podman-alternative))
 
 * At least 3 GB of memory per container
-	
+
 * Familiarity with Docker volumes and networks
 
 ## Directory Structure
 
-You must create a separate directory structure to organize your configuration, certificate files, and Docker Compose setup. This step keeps your deployment clean and easy to manage.
+You must create a separate directory structure to organize your configuration, certificate files, and Docker Compose setup. The directory structure keeps your deployment clean and easy to manage.
 
 Run the following commands to create the directory structure:
 
@@ -36,13 +31,17 @@ mkdir -p pxc-cluster/{certs,conf.d,init}
 cd pxc-cluster
 ```
 
+Expected result: No output. The current directory is `pxc-cluster` with subdirectories `certs`, `conf.d`, and `init`.
+
 After running these commands, your working directory (pxc-cluster/) will contain:
 
 ![PXC cluster directories](_static/pxc-cluster-dirs.png)
 
-This structure helps manage configuration files, TLS/SSL certificates, and setup scripts, ensuring a tidy and easy-to-manage deployment.
+The directory structure helps manage configuration files, TLS/SSL certificates, and setup scripts, ensuring a tidy and easy-to-manage deployment.
 {.power-number}
 
+## Configuration Files
+
 1. Create conf.d/custom.cnf with minimal SSL settings:
 
   
@@ -62,6 +61,8 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
 
     Add `.env` to your .gitignore file to prevent committing secrets to version control.
 
+## SSL Certificate Generation
+
 3. Copy the SSL certificate generation script. Save the script as `init/create-ssl-certs.sh`:
 
     ```text
@@ -95,43 +96,50 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
     chmod +x init/create-ssl-certs.sh
     ```
 
+    Expected result: No output.
+
 5. Run the script to create the certs:
 
     ```shell
     ./init/create-ssl-certs.sh
     ```
 
-4. Copy Certificates to All Nodes
+    Expected result: No output. The script creates `ca-key.pem`, `ca.pem`, `server-key.pem`, `server-req.pem`, and `server-cert.pem` in `certs/`.
 
-    All three nodes in the cluster must use the same set of SSL certificates. After generating the certificates in the certs/ directory on node 1, you need to copy them to the directories for node 2 and node 3.
-    
-    If you run all containers from a single project directory (like with Docker Compose on one host), you can reuse the same certs/ directory for all nodes. However, you must explicitly copy the certificates if you’re organizing them into separate directories or deploying on separate hosts.
-  
-    To create the directories for node 2 and node 3:
-
-    
-    ```shell
-    mkdir -p certs-node2
-    mkdir -p certs-node3
-    ```
+    Note: The certificates expire in 10 years. For production environments, implement certificate rotation and expiration monitoring.
 
-    Then copy the certificates:
+4. Copy Certificates to All Nodes (only for multi-host)
 
-    
-    ```shell
-    cp -r certs/* certs-node2/
-    cp -r certs/* certs-node3/
-    ```
+    All three nodes in the cluster must use the same set of SSL certificates. The `docker-compose.yml` in the guide mounts the same `./certs` directory for all three services (pxc1, pxc2, pxc3). For a single-host deployment, you do not need to create separate certificate directories; the `certs/` you created in step 5 is used by all nodes.
 
-    If you’re deploying on separate machines, run the following from node 1:
+    If you deploy on separate machines instead of one host, copy the certificates to each machine so each has its own `certs/` (or equivalent path) for its Compose project. From node 1, run:
 
     
     ```shell
     scp -r ./certs/ user@node2-host:/path/to/pxc-cluster/certs
     scp -r ./certs/ user@node3-host:/path/to/pxc-cluster/certs
     ```
 
-    Ensure each container mounts its own copy of the certs/ directory.
+    Expected result: Progress or confirmation for each file copied to each host. No output indicates success after the transfer completes.
+
+    Ensure each host's Compose file mounts that host's certificate directory. For multi-host, see also [Multi-host deployment](#multi-host-deployment) below for firewall, name resolution, and time synchronization.
+
+## Multi-host deployment
+
+If you run each node on a separate machine, configure the following in addition to copying certificates.
+
+Firewall: Allow cluster and client traffic between the hosts. Open these ports on each host for the other hosts' IPs (or subnet):
+
+* 3306 (MySQL client)
+* 4567 (Galera replication)
+* 4568 (Galera incremental state transfer)
+* 4444 (Percona XtraBackup snapshot transfer)
+
+Name resolution: Containers on one host must reach containers on other hosts by hostname or IP. The Compose file uses service names (pxc1, pxc2, pxc3). When each host runs its own Compose stack, those names resolve only within that host. On each host, ensure the other nodes are reachable by a name or IP that you use in `CLUSTER_JOIN` (for example, the other hosts' FQDNs or IPs). Add DNS records or `/etc/hosts` entries on each host so that the hostname or IP used for each node resolves to the correct machine.
+
+Time synchronization: PXC and Galera require consistent time across nodes. Synchronize the clock on every host with NTP (or chrony, systemd-timesyncd). Skew between hosts can cause replication issues and cluster instability. Ensure NTP is enabled and running before starting the cluster.
+
+## Docker Compose Setup
 
 5. Create docker-compose.yml:
 
@@ -143,11 +151,11 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         environment:
           - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
           - CLUSTER_NAME=pxc-cluster
-          - CLUSTER_JOIN=pxc2,pxc3
           - XTRABACKUP_PASSWORD=${XTRABACKUP_PASSWORD}
         volumes:
           - ./certs:/etc/mysql/certs:ro
           - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
+          - ./data-pxc1:/var/lib/mysql
         networks:
           - pxcnet
         ports:
@@ -164,11 +172,12 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         environment:
           - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
           - CLUSTER_NAME=pxc-cluster
-          - CLUSTER_JOIN=pxc1,pxc3
+          - CLUSTER_JOIN=pxc1
           - XTRABACKUP_PASSWORD=${XTRABACKUP_PASSWORD}
         volumes:
           - ./certs:/etc/mysql/certs:ro
           - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
+          - ./data-pxc2:/var/lib/mysql
         networks:
           - pxcnet
         healthcheck:
@@ -182,11 +191,12 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         environment:
           - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
           - CLUSTER_NAME=pxc-cluster
-          - CLUSTER_JOIN=pxc1,pxc2
+          - CLUSTER_JOIN=pxc1
           - XTRABACKUP_PASSWORD=${XTRABACKUP_PASSWORD}
         volumes:
           - ./certs:/etc/mysql/certs:ro
           - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
+          - ./data-pxc3:/var/lib/mysql
         networks:
           - pxcnet
         healthcheck:
@@ -199,6 +209,10 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         driver: bridge
     ```
 
+    SELinux: On systems with SELinux enabled (Docker or Podman), the container may not be able to read the mounted `certs/`, `conf.d/`, or data directories. Add `:Z` (private) or `:z` (shared) to each volume mount so the runtime relabels the mount for the container. For example, use `./certs:/etc/mysql/certs:ro,Z`, `./conf.d:/etc/percona-xtradb-cluster.conf.d:ro,Z`, and `./data-pxc1:/var/lib/mysql,Z` (and the same for pxc2 and pxc3). Apply the same suffix to every volume in every service.
+
+## Deployment
+
 6. Start the Cluster
 
     Start node 1 to initialize the cluster:
@@ -207,23 +221,156 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
     ```shell
     docker compose up -d pxc1
     ```
+    Expected result: A line such as `[+] Running 1/1 - Container pxc1  Started` (or similar). The container pxc1 is running.
+
+    (With Podman: `podman compose up -d pxc1` or `podman-compose up -d pxc1`.)
+
+    Wait for pxc1 to be fully healthy before starting pxc2 and pxc3. The bootstrap node must be ready and accepting connections, or the other nodes may fail to join. You can use a wait loop:
+
+    ```shell
+    until docker exec pxc1 mysqladmin ping -h localhost &>/dev/null; do
+      echo "Waiting for pxc1..."
+      sleep 2
+    done
+    ```
+
+    Expected result: The loop prints "Waiting for pxc1..." until the node responds, then exits with no further output.
 
     Then, start the remaining nodes:
 
-    
     ```shell
     docker compose up -d pxc2 pxc3
     ```
+    Expected result: Lines such as `[+] Running 2/2 - Container pxc2  Started - Container pxc3  Started` (or similar). Both containers are running.
+
+    (With Podman: `podman compose up -d pxc2 pxc3` or `podman-compose up -d pxc2 pxc3`.)
+
+## Validation
 
 7. Validate the Cluster
 
-    Check the status of each node:
+    Check the status of each node. Exact commands (run from the host; use the same password as in your `.env`). First verify cluster size and node status:
+
+    ```shell
+    docker exec pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
+    docker exec pxc2 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_status';"
+    ```
+
+    Expected result: For `wsrep_cluster_size`, a row with value `3`. For `wsrep_cluster_status`, a row with value `Primary`.
+
+    Then verify additional cluster health indicators on any node (for example, pxc1). Expect `wsrep_ready` and `wsrep_connected` to be `ON`, and `wsrep_local_state_comment` to be `Synced`:
 
-    
     ```shell
-    docker exec -it pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
-    docker exec -it pxc2 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_status';"
+    docker exec pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_ready';"
+    docker exec pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_connected';"
+    docker exec pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"
     ```
 
-You should see all three nodes joined and synchronized.
+    Expected result: `wsrep_ready` = `ON`, `wsrep_connected` = `ON`, `wsrep_local_state_comment` = `Synced`.
+
+    (With Podman: use `podman exec` instead of `docker exec`.)
+
+You should see all three nodes joined and synchronized, with the health indicators above showing the expected values.
+
+## Backup
+
+Even for testing, back up data before major changes or shutdown. Example: dump all databases from one node (pxc1) to a file on the host:
+
+```shell
+docker exec pxc1 mysqldump -uroot -p${MYSQL_ROOT_PASSWORD} --all-databases > backup.sql
+```
+
+Expected result: The file `backup.sql` is created in the current directory with SQL statements for all databases. With Podman, use `podman exec` instead of `docker exec`.
+
+## Troubleshooting
+
+Viewing logs for debugging:
+
+```shell
+docker compose logs -f pxc1
+docker compose logs -f pxc2
+docker compose logs -f pxc3
+```
+
+Run one of the commands above to stream that container's logs (stdout and stderr). The `-f` option follows the log output; omit `-f` for a one-off dump. Expected result: Log lines from MySQL and PXC until you press Ctrl+C. With Podman, use `podman compose logs -f pxc1` (and so on).
+
+* Containers exit or fail to start: Check logs with `docker compose logs pxc1` (and pxc2, pxc3). Ensure the bootstrap node (pxc1) is healthy before starting pxc2 and pxc3.
+
+* Cluster size stays at 1: Start pxc1 first, wait for pxc1 to be ready (healthcheck passing), then start pxc2 and pxc3. If nodes start too soon, the nodes may not join; restart pxc2 and pxc3 after pxc1 is up.
+
+* Permission denied on certs or config: Ensure `certs/` and `conf.d/` are readable by the container. On systems with SELinux enabled (Docker or Podman), add `:Z` or `:z` to all volume mounts (see the SELinux note in [Docker Compose Setup](#docker-compose-setup) and [Appendix: Podman Alternative](#appendix-podman-alternative)).
+
+* Nodes cannot reach each other: On a single host, verify all containers are on the same network (`docker network inspect pxc-cluster_pxcnet` or equivalent). On multiple hosts, see [Multi-host deployment](#multi-host-deployment) for firewall rules, name resolution (DNS or `/etc/hosts`), and NTP.
+
+## Cleanup and Shutdown
+
+To stop the cluster and remove containers (data in `data-pxc1/`, `data-pxc2/`, `data-pxc3/` is preserved):
+
+```shell
+docker compose down
+```
+
+Expected result: Containers pxc1, pxc2, and pxc3 are stopped and removed. Project network is removed. Data in `data-pxc1/`, `data-pxc2/`, and `data-pxc3/` remains on the host.
+
+To stop and remove containers and volumes (the `-v` option deletes all database data):
+
+```shell
+docker compose down -v
+```
+
+Expected result: Containers and any named volumes are removed. Bind-mounted data directories (`data-pxc1/`, etc.) are not removed by this command.
+
+Note: The Compose file in the guide uses bind mounts (`./data-pxc1`, etc.), not named volumes, so `-v` removes only any named volumes if present. To fully reset, remove the data directories manually: `rm -rf data-pxc1 data-pxc2 data-pxc3` (only when the containers are stopped).
+
+## Appendix: Podman Alternative
+
+Podman can be used as an alternative to Docker because Podman supports the same container images. However, Podman is not fully compatible with Docker Compose.
+
+To run the deployment with Podman, you may need to use tools such as podman compose, podman-compose, or a configured Docker Compose compatibility layer, depending on your environment.
+
+Podman uses a different architecture (for example, pods and rootless containers), so networking, volume mounts, and service behavior may differ from Docker Compose.
+
+The deployment is designed and tested with Docker Compose. Podman support is not guaranteed. If you use Podman, verify that the cluster operates correctly before using the cluster beyond testing or experimentation.
+
+You can use the same Compose file and workflow with [Podman](https://podman.io/). Use Podman 4.1 or later; the built-in `podman compose` subcommand requires 4.1 or newer. With older Podman, use the separate podman-compose tool. Then:
+
+* Prerequisites: Podman and Podman Compose (for example, `pip install podman-compose` or your distro's package) if you are not using built-in `podman compose`. For rootless Podman, ensure the user has enough resources (for example, `sysctl user.max_user_namespaces` and subuid/subgid ranges).
+
+* Commands: Replace `docker compose` with `podman compose` or `podman-compose`, and `docker exec` with `podman exec`. Examples:
+
+  * Start node 1: `podman compose up -d pxc1` (or `podman-compose up -d pxc1`)
+
+  * Start other nodes: `podman compose up -d pxc2 pxc3`
+
+  * Validate: `podman exec -it pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_size';"`
+
+* Directory structure: The same layout (`pxc-cluster/` with `certs/`, `conf.d/`, and `init/`) works with Podman. Run `podman compose` from the project directory (for example, `pxc-cluster/`) so the relative volume paths in the Compose file resolve correctly.
+
+* Compose file: The `docker-compose.yml` in the guide works as-is with Podman; the `bridge` network and volume mounts are supported. For rootless Podman, see the volume mount options below.
+
+### Handling rootless permissions (most important for Podman)
+
+In Docker, the daemon runs as root and can override file permissions. In Podman, if you run as a normal user, the MySQL process inside the container (usually UID 1001 or 999) may not have permission to read the files you created on the host. Ensure the files are readable by the container. On systems with SELinux, use the `:Z` (private) or `:z` (shared) mount option so Podman relabels the volume for the container. In your `docker-compose.yml`, use:
+
+```text
+volumes:
+  - ./certs:/etc/mysql/certs:ro,Z
+  - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro,Z
+```
+
+Apply the same volume options to every service (pxc1, pxc2, pxc3). Without `:Z` or `:z`, the container may fail to read certs or config when running rootless on SELinux.
+
+### podman-compose vs docker-compose
+
+* If you use podman-compose (the Python tool): podman-compose reads the directory structure and `.env` file the same way Docker does.
+
+* If you use docker-compose with the Podman socket: Using docker-compose with the Podman socket is often more stable for PXC. Set `DOCKER_HOST` to point at the Podman socket (for example, `unix:///run/user/$(id -u)/podman/podman.sock`) so `docker compose` talks to Podman.
+
+### Network considerations
+
+PXC uses specific ports for cluster communication (4567, 4568, 4444). Podman uses different networking (netavark or CNI). If nodes cannot find each other:
+
+* Keep using a named network in the Compose file (for example, `pxcnet`) so Podman can resolve container names (pxc1, pxc2, pxc3).
+
+* If problems persist, try setting `network_mode: slirp4netns` on the services, or run the stack rootful to use the default bridge.
 
