PXC-5141 - [DOCS] - [feedback] It doesn't work with podman compose 8.4

patrickbirch · patrickbirch · commit 3a9bb7626a44 · 2026-03-19T07:07:09.000-05:00
modified:   docs/docker-compose.md
diff --git a/docs/docker-compose.md b/docs/docker-compose.md
@@ -8,26 +8,21 @@ We gather [Telemetry data](telemetry.md) in the Percona packages and Docker imag
 
 --8<--- "get-help-snip.md"
 
-This guide shows you how to deploy a three-node Percona XtraDB Cluster 8.4 using Docker Compose. You generate SSL certificates on the first node and copy them to the other two nodes to enable secure communication.
+The guide shows you how to deploy a three-node Percona XtraDB Cluster 8.4 using Docker Compose. You generate SSL certificates on the first node and copy them to the other two nodes to enable secure communication.
 
-The following procedure describes setting up a simple 3-node cluster
-for evaluation and testing purposes. Do not use these instructions in a
-production environment because the MySQL certificates generated in this
-procedure are self-signed.
-
-In a production environment, you should generate and store the certificates to be used by Docker and configure proper storage, security, backup, and monitoring systems.
+The following procedure is for evaluation and testing only. Do not use these instructions in a production environment because the MySQL certificates generated here are self-signed. In production, generate and store proper certificates and configure storage, security, backup, and monitoring.
 
 ## Prerequisites
 
-* Docker and Docker Compose installed
+* Docker and Docker Compose installed (or Podman 4.1 or later and Podman Compose; see [Appendix: Podman Alternative](#appendix-podman-alternative))
 
 * At least 3 GB of memory per container
-	
+
 * Familiarity with Docker volumes and networks
 
 ## Directory Structure
 
-You must create a separate directory structure to organize your configuration, certificate files, and Docker Compose setup. This step keeps your deployment clean and easy to manage.
+You must create a separate directory structure to organize your configuration, certificate files, and Docker Compose setup. The directory structure keeps your deployment clean and easy to manage.
 
 Run the following commands to create the directory structure:
 
@@ -40,9 +35,11 @@ After running these commands, your working directory (pxc-cluster/) will contain
 
 ![PXC cluster directories](_static/pxc-cluster-dirs.png)
 
-This structure helps manage configuration files, TLS/SSL certificates, and setup scripts, ensuring a tidy and easy-to-manage deployment.
+The directory structure helps manage configuration files, TLS/SSL certificates, and setup scripts, ensuring a tidy and easy-to-manage deployment.
 {.power-number}
 
+## Configuration Files
+
 1. Create conf.d/custom.cnf with minimal SSL settings:
 
   
@@ -62,6 +59,8 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
 
     Add `.env` to your .gitignore file to prevent committing secrets to version control.
 
+## SSL Certificate Generation
+
 3. Copy the SSL certificate generation script. Save the script as `init/create-ssl-certs.sh`:
 
     ```text
@@ -101,37 +100,21 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
     ./init/create-ssl-certs.sh
     ```
 
-4. Copy Certificates to All Nodes
+4. Copy Certificates to All Nodes (only for multi-host)
 
-    All three nodes in the cluster must use the same set of SSL certificates. After generating the certificates in the certs/ directory on node 1, you need to copy them to the directories for node 2 and node 3.
-    
-    If you run all containers from a single project directory (like with Docker Compose on one host), you can reuse the same certs/ directory for all nodes. However, you must explicitly copy the certificates if you’re organizing them into separate directories or deploying on separate hosts.
-  
-    To create the directories for node 2 and node 3:
-
-    
-    ```shell
-    mkdir -p certs-node2
-    mkdir -p certs-node3
-    ```
-
-    Then copy the certificates:
-
-    
-    ```shell
-    cp -r certs/* certs-node2/
-    cp -r certs/* certs-node3/
-    ```
+    All three nodes in the cluster must use the same set of SSL certificates. The `docker-compose.yml` in the guide mounts the same `./certs` directory for all three services (pxc1, pxc2, pxc3). For a single-host deployment, you do not need to create separate certificate directories; the `certs/` you created in step 5 is used by all nodes.
 
-    If you’re deploying on separate machines, run the following from node 1:
+    If you deploy on separate machines instead of one host, copy the certificates to each machine so each has its own `certs/` (or equivalent path) for its Compose project. From node 1, run:
 
     
     ```shell
     scp -r ./certs/ user@node2-host:/path/to/pxc-cluster/certs
     scp -r ./certs/ user@node3-host:/path/to/pxc-cluster/certs
     ```
 
-    Ensure each container mounts its own copy of the certs/ directory.
+    Ensure each host's Compose file mounts that host's certificate directory.
+
+## Docker Compose Setup
 
 5. Create docker-compose.yml:
 
@@ -143,11 +126,11 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         environment:
           - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
           - CLUSTER_NAME=pxc-cluster
-          - CLUSTER_JOIN=pxc2,pxc3
           - XTRABACKUP_PASSWORD=${XTRABACKUP_PASSWORD}
         volumes:
           - ./certs:/etc/mysql/certs:ro
           - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
+          - ./data-pxc1:/var/lib/mysql
         networks:
           - pxcnet
         ports:
@@ -164,11 +147,12 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         environment:
           - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
           - CLUSTER_NAME=pxc-cluster
-          - CLUSTER_JOIN=pxc1,pxc3
+          - CLUSTER_JOIN=pxc1
           - XTRABACKUP_PASSWORD=${XTRABACKUP_PASSWORD}
         volumes:
           - ./certs:/etc/mysql/certs:ro
           - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
+          - ./data-pxc2:/var/lib/mysql
         networks:
           - pxcnet
         healthcheck:
@@ -182,11 +166,12 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         environment:
           - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
           - CLUSTER_NAME=pxc-cluster
-          - CLUSTER_JOIN=pxc1,pxc2
+          - CLUSTER_JOIN=pxc1
           - XTRABACKUP_PASSWORD=${XTRABACKUP_PASSWORD}
         volumes:
           - ./certs:/etc/mysql/certs:ro
           - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
+          - ./data-pxc3:/var/lib/mysql
         networks:
           - pxcnet
         healthcheck:
@@ -199,6 +184,8 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
         driver: bridge
     ```
 
+## Deployment
+
 6. Start the Cluster
 
     Start node 1 to initialize the cluster:
@@ -207,23 +194,105 @@ This structure helps manage configuration files, TLS/SSL certificates, and setup
     ```shell
     docker compose up -d pxc1
     ```
+    (With Podman: `podman compose up -d pxc1` or `podman-compose up -d pxc1`.)
 
     Then, start the remaining nodes:
 
     
     ```shell
     docker compose up -d pxc2 pxc3
     ```
+    (With Podman: `podman compose up -d pxc2 pxc3` or `podman-compose up -d pxc2 pxc3`.)
+
+## Validation
 
 7. Validate the Cluster
 
-    Check the status of each node:
+    Check the status of each node. Exact commands (run from the host; use the same password as in your `.env`):
 
-    
     ```shell
     docker exec -it pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
     docker exec -it pxc2 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_status';"
     ```
+    (With Podman: use `podman exec` instead of `docker exec`.)
 
 You should see all three nodes joined and synchronized.
 
+## Troubleshooting
+
+* Containers exit or fail to start: Check logs with `docker compose logs pxc1` (and pxc2, pxc3). Ensure the bootstrap node (pxc1) is healthy before starting pxc2 and pxc3.
+
+* Cluster size stays at 1: Start pxc1 first, wait for pxc1 to be ready (healthcheck passing), then start pxc2 and pxc3. If nodes start too soon, the nodes may not join; restart pxc2 and pxc3 after pxc1 is up.
+
+* Permission denied on certs or config: Ensure `certs/` and `conf.d/` are readable by the container. On SELinux with Podman, use the `:Z` or `:z` volume option (see [Appendix: Podman Alternative](#appendix-podman-alternative)).
+
+* Nodes cannot reach each other: Verify all containers are on the same network (`docker network inspect pxc-cluster_pxcnet` or equivalent). Open firewall ports 3306, 4567, 4568, and 4444 if running on separate hosts.
+
+## Cleanup and Shutdown
+
+To stop the cluster and remove containers (data in `data-pxc1/`, `data-pxc2/`, `data-pxc3/` is preserved):
+
+```shell
+docker compose down
+```
+
+To stop and remove containers and volumes (the `-v` option deletes all database data):
+
+```shell
+docker compose down -v
+```
+
+Note: The Compose file in the guide uses bind mounts (`./data-pxc1`, etc.), not named volumes, so `-v` removes only any named volumes if present. To fully reset, remove the data directories manually: `rm -rf data-pxc1 data-pxc2 data-pxc3` (only when the containers are stopped).
+
+## Appendix: Podman Alternative
+
+Podman can be used as an alternative to Docker because Podman supports the same container images. However, Podman is not fully compatible with Docker Compose.
+
+To run the deployment with Podman, you may need to use tools such as podman compose, podman-compose, or a configured Docker Compose compatibility layer, depending on your environment.
+
+Podman uses a different architecture (for example, pods and rootless containers), so networking, volume mounts, and service behavior may differ from Docker Compose.
+
+The deployment is designed and tested with Docker Compose. Podman support is not guaranteed. If you use Podman, verify that the cluster operates correctly before using the cluster beyond testing or experimentation.
+
+You can use the same Compose file and workflow with [Podman](https://podman.io/). Use Podman 4.1 or later; the built-in `podman compose` subcommand requires 4.1 or newer. With older Podman, use the separate podman-compose tool. Then:
+
+* Prerequisites: Podman and Podman Compose (for example, `pip install podman-compose` or your distro's package) if you are not using built-in `podman compose`. For rootless Podman, ensure the user has enough resources (for example, `sysctl user.max_user_namespaces` and subuid/subgid ranges).
+
+* Commands: Replace `docker compose` with `podman compose` or `podman-compose`, and `docker exec` with `podman exec`. Examples:
+
+  * Start node 1: `podman compose up -d pxc1` (or `podman-compose up -d pxc1`)
+
+  * Start other nodes: `podman compose up -d pxc2 pxc3`
+
+  * Validate: `podman exec -it pxc1 mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SHOW STATUS LIKE 'wsrep_cluster_size';"`
+
+* Directory structure: The same layout (`pxc-cluster/` with `certs/`, `conf.d/`, and `init/`) works with Podman. Run `podman compose` from the project directory (for example, `pxc-cluster/`) so the relative volume paths in the Compose file resolve correctly.
+
+* Compose file: The `docker-compose.yml` in the guide works as-is with Podman; the `bridge` network and volume mounts are supported. For rootless Podman, see the volume mount options below.
+
+### Handling rootless permissions (most important for Podman)
+
+In Docker, the daemon runs as root and can override file permissions. In Podman, if you run as a normal user, the MySQL process inside the container (usually UID 1001 or 999) may not have permission to read the files you created on the host. Ensure the files are readable by the container. On systems with SELinux, use the `:Z` (private) or `:z` (shared) mount option so Podman relabels the volume for the container. In your `docker-compose.yml`, use:
+
+```text
+volumes:
+  - ./certs:/etc/mysql/certs:ro,Z
+  - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro,Z
+```
+
+Apply the same volume options to every service (pxc1, pxc2, pxc3). Without `:Z` or `:z`, the container may fail to read certs or config when running rootless on SELinux.
+
+### podman-compose vs docker-compose
+
+* If you use podman-compose (the Python tool): podman-compose reads the directory structure and `.env` file the same way Docker does.
+
+* If you use docker-compose with the Podman socket: Using docker-compose with the Podman socket is often more stable for PXC. Set `DOCKER_HOST` to point at the Podman socket (for example, `unix:///run/user/$(id -u)/podman/podman.sock`) so `docker compose` talks to Podman.
+
+### Network considerations
+
+PXC uses specific ports for cluster communication (4567, 4568, 4444). Podman uses different networking (netavark or CNI). If nodes cannot find each other:
+
+* Keep using a named network in the Compose file (for example, `pxcnet`) so Podman can resolve container names (pxc1, pxc2, pxc3).
+
+* If problems persist, try setting `network_mode: slirp4netns` on the services, or run the stack rootful to use the default bridge.
+
