Merge pull request #982 from plone/issue-975

Deployment training 2025

Author: ericof

docs/plone-deployment/_static/request_flow.png

@@ -18,7 +18,7 @@ This guide outlines the steps to deploy the project using a Docker stack compris
 - **Plone Backend:** The API service.
 - **Postgres 14 Database:** Handles data persistence.
 
-You can find this stack at {file}`devops/stacks/ploneconf2024-<your-github-username>.tangrama.com.br.yml`. It's modular, allowing easy integration of additional services like {term}`Varnish`, `Solr`, or `ElasticSearch`.
+You can find this stack at {file}`devops/stacks/ploneconf2025-<your-github-username>.tangrama.com.br.yml`. It's modular, allowing easy integration of additional services like {term}`Varnish`, `Solr`, or `ElasticSearch`.
 
 ```{seealso}
 [Traefik Proxy with HTTPS](https://dockerswarm.rocks/traefik/)
@@ -27,7 +27,7 @@ You can find this stack at {file}`devops/stacks/ploneconf2024-<your-github-usern
 ## Building Docker Images
 
 Ensure you build the Docker images for the Frontend and Backend servers before deployment.
-GitHub Actions, configured in {file}`.github/workflows/backend.yml` and {file}`.github/workflows/frontend.yml`, facilitate this process.
+GitHub Actions, configured in {file}`.github/workflows/main.yml`, facilitate this process.
 
 ````{important}
 Before deploying, run the following commands from your project's root directory to format the code and run tests.
@@ -56,43 +56,48 @@ Watch Usage
 After these commands succeed, commit all code changes, push to GitHub, and ensure all GitHub Actions successfully complete their runs.
 ````
 
-
-## Manual Deployment with `devops/Makefile`
+## Manual Deployment with Ansible
 
 Utilize the {file}`Makefile` at {file}`devops/Makefile` for manual deployment.
 
 ### Deploying the Stack
 
-Execute the following command from your project's {file}`devops` directory to deploy the stack defined in {file}`devops/stacks/ploneconf2024-<your-github-username>.tangrama.com.br.yml` to the remote server.
+Execute the following command from your project's {file}`devops/ansible` directory to deploy the stack defined in {file}`devops/stacks/ploneconf2025-<your-github-username>.tangrama.com.br.yml` to the remote server.
 
 ```shell
-make stack-deploy
+uv run ansible-playbook playbooks/deploy.yml --tags project
 ```
 
 ### Verifying Stack Status
 
-To check the status of all services in your stack, run:
+To check the status of all services in your stack, access the remote server:
+
+```shell
+ssh root@ploneconf2025-<your-github-username>.tangrama.com.br
+```
+
+And then run the command:
 
 ```shell
-make stack-status
+docker stack ps ploneconf2025-<your-github-username>-tangrama-com-br
 ```
 
 ### Creating Plone Site
 
 On the initial deployment, the frontend containers might be unhealthy due to the unconfigured Plone site on the backend. Create a new site with:
 
 ```shell
-make stack-create-site
+docker exec $(docker ps -qf 'name=_backend'|head -n1) ./docker-entrypoint.sh create-site
 ```
 
 ### Monitoring Logs
 
 Monitor the logs of each service with these commands:
 
--   Traefik: `make stack-logs-webserver`
--   Frontend: `make stack-logs-frontend`
--   Backend: `make stack-logs-backend`
--   Backend: `make stack-logs-db`
+-   Traefik: `docker service logs traefik_traefik --follow`
+-   Frontend: `docker service logs <stack-name>_frontend --follow`
+-   Backend: `ocker service logs <stack-name>_backend --follow`
+-   Database: `ocker service logs <stack-name>_db --follow`
 
 ## Automating Deployment with GitHub Actions
 
@@ -130,15 +135,6 @@ Add secrets in the `Secrets` section of your environment. Refer to the table bel
 | DEPLOY_SSH  | Content of {file}`devops/etc/keys/plone_prod_deploy_rsa` | The private SSH key for connection.                       |
 | ENV_FILE    | Content of {file}`devops/.env_file_gha`                  | File containing environment variables for the stack file. |
 
-#### Adding Repository Variables
-
-Navigate to {menuselection}`Settings --> Secrets and Variables --> Actions`. Under {guilabel}`Variables`, add the repository variable:
-
-| Name     | Value |
-|----------|-------|
-| LIVE_ENV | The name of the earlier created environment |
-
-This variable is referenced in {file}`.github/workflows/manual_deploy.yml`.
 
 ## Initiating Manual Deployment
 

docs/plone-deployment/_static/request_flow.svg

@@ -32,12 +32,12 @@ Many sections may be zipped through in a class, noting to students that the full
 intro
 setup
 plone-stack
+virtual-host
 plone-docker-images
 project-new
 project-start
 project-edit
 server-setup
 deploy
-virtual-host
 troubleshoot
 ```

docs/plone-deployment/deploy.md

@@ -9,9 +9,8 @@ myst:
 
 # Introduction
 
-This training material provides insights into deploying Plone for production.
-We aim to impart skills for automating deployment processes,
-ensuring that you can transform a fresh Linux server into an efficient and robust Plone server.
+This training provides practical guidance for deploying Plone in production.
+It focuses on automating deployments so you can reliably transform a fresh Linux server into a production-ready Plone instance.
 
 ## Target Audience
 
@@ -20,17 +19,29 @@ or considering a virtual machine on your personal computer for testing, this tra
 
 ## Objectives
 
+- Describe the lifecycle of a Plone project.
 - Achieve repeatable deployments.
 - Ensure consistency across multiple deployment instances.
 - Impart knowledge on the tools reflecting the Plone community's preferences.
 
-## Training Content
+## Different types of Plone codebases
 
-We will delve into a basic setup, scalable to accommodate extensive Plone installations.
+In the Plone ecosystem there are three common codebase types:
+
+| Type | Definition | Example |
+| --  | -- | -- |
+| Distribution  | An opinionated packaging of Plone, usually focused on a vertical market or specific use case. | kitconcept Intranet, Quaive, ioCommune |
+| Add-On  | A package (or group of packages) that extends Plone with new features. | EEA Accordion, Volto Form Support, Volto Light Theme, Collective Tech Event |
+| Project  | A specific implementation of Plone (or a Plone-based distribution) used to create a site, intranet, or knowledge base. | Plone Conference Site, Plone.org, DLR.de |
+
+Add-ons are typically tested against a matrix of supported versions: backend add-ons against multiple Python and Plone versions, and frontend add-ons against Node and Volto version matrices. Distributions are released with a narrower, opinionated set of pinned dependencies to reduce integration complexity.
+
+Project codebases generally require strict repeatability and therefore pin exact dependency versions.
+For projects managed with `zc.buildout`, it has been best practice to include a `versions.cfg` file that controls package versions. In this training we follow the same principle: we use a backend lockfile ( `uv.lock`) and `pnpm-lock.yaml` for the frontend.
 
 (deployment-training-choices)=
 
-### Training Choices
+### Training choices
 
 #### Linux
 
@@ -39,21 +50,19 @@ However, any system capable of running Python should suffice for development pur
 
 #### Major Distributions
 
-We support and recommend Ubuntu LTS for server setup. Debian and other distributions with up-to-date packages are also compatible.
+We recommend Ubuntu LTS for server deployments. Debian and other stable distributions with up-to-date packages are also supported.
 
 #### Platform Packages
 
-Prioritize platform packages to leverage automatic updates. Opt for usability over the latest versions to ensure stability and security.
+Prefer using distribution-packaged system packages where practical to benefit from automatic security updates and easier maintenance. Favor stability and maintainability over running the absolute latest upstream versions on production systems.
 
 #### Ansible
 
-Ansible stands out for its serverless nature, Python foundation, and user-friendly YAML configuration language,
-making it a preferred choice for deployment automation.
+Ansible is a natural fit for our automation needs: it's agentless, written in Python, and uses readable YAML playbooks. Its idempotent design and role-based structure make it a good choice for repeatable server provisioning.
 
 #### Docker and Docker Swarm
 
-Embrace the consistency and repeatability offered by containers. Docker, complemented by Docker Swarm,
-simplifies the setup process, making it accessible even for beginners.
+Containers provide consistency and repeatability. Docker (and Docker Compose for local orchestration) is used throughout this training; Docker Swarm is the recommended option for lightweight orchestration in production scenarios covered here.
 
 #### GitHub and GitHub Actions
 
@@ -62,4 +71,4 @@ The principles outlined are adaptable to GitLab, Jenkins, and similar platforms.
 
 #### Kubernetes
 
-While the current training edition doesn't cover Kubernetes, we anticipate its inclusion in future updates.
+This edition of the training does not cover Kubernetes, but we expect to include Kubernetes-based deployment patterns in a future update. The Plone community maintains Helm charts that can be used as a starting point: https://github.com/plone/helm-charts

docs/plone-deployment/index.md

@@ -9,15 +9,15 @@ myst:
 
 # Plone Docker Images
 
-Since the release of Plone 6, the community has a new set of Docker images offering more base options.
+Since the release of Plone 6, the community has a new set of public Docker images offering most base options, and documenting the configuration. They are meant as a way to quickly start a project, and provide inspiration for your own projects advanced requirements.
 
 ## `plone/plone-frontend`
 
-Repository available at https://github.com/plone/plone-frontend/.
+Repository is available at https://github.com/plone/plone-frontend/.
 
 Installs the Plone 6 user-experience using the React-powered frontend, Volto.
 
-Should be used to showcase the Plone 6 experience, as new projects will probably implement their own Docker images (with a similar Dockerfile), like the one below:
+Should be used to showcase the Plone 6 experience, and how to build images in multiple stages to reduce image size. New projects will probably implement their own Docker images (with a similar Dockerfile), like the one below:
 
 ```Dockerfile
 # syntax=docker/dockerfile:1
@@ -58,10 +58,11 @@ EOT
 
 Repository available at https://github.com/plone/plone-backend/.
 
-Installs the Plone 6 backend using a pip-based installation.
-This approach makes it easier and faster to extend this image in your own project.
+Installs the Plone 6 backend using a package-based installation.
 
-One example of such extension would be:
+There are currently two distinct approaches to use the backend base images in your project:
+
+### Usage with `pip` + `mxdev`
 
 ```Dockerfile
 # syntax=docker/dockerfile:1
@@ -93,7 +94,7 @@ EOT
 FROM plone/server-prod-config:${PLONE_VERSION}
 
 LABEL maintainer="Plone Community <collective@plone.org>" \
-      org.label-schema.name="ploneconf2024-training-backend" \
+      org.label-schema.name="ploneconf2025-training-backend" \
       org.label-schema.description="Plone Conference Training Backend." \
       org.label-schema.vendor="Plone Community"
 
@@ -106,8 +107,85 @@ RUN <<EOT
 EOT
 ```
 
+### Usage with `uv` (Experimental)
+
+To use these images, your project should be already using `uv` and have a `pyproject.toml` section with additional dependencies to be installed inside a container:
+
+```toml
+[dependency-groups]
+...
+container = [
+    "plone.app.upgrade",
+    "psycopg2==2.9.10",
+    "relstorage==4.1.1",
+    "zeo==6.0.0",
+]
+```
+
+The `Dockerfile` will look like:
+
+```Dockerfile
+# syntax=docker/dockerfile:1.9
+ARG PYTHON_VERSION=3.12
+FROM plone/server-builder:uv-${PYTHON_VERSION} AS builder
+
+# Install dependencies
+RUN --mount=type=cache,target=/root/.cache \
+    --mount=type=bind,source=uv.lock,target=uv.lock \
+    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
+    uv sync \
+        --locked \
+        --no-dev \
+        --no-group test \
+        --group container \
+        --no-install-project
+
+COPY . /src
+WORKDIR /src
+
+# Install package
+RUN --mount=type=cache,target=/root/.cache \
+    uv sync \
+        --locked \
+        --no-dev \
+        --no-group test \
+        --group container \
+        --no-editable
+
+# Move skeleton files to /app
+RUN <<EOT
+    mv /app_skeleton/* /app
+    rm -Rf /app_skeleton
+    mv container/docker-entrypoint.sh /app/docker-entrypoint.sh
+    chmod +x /app/docker-entrypoint.sh
+    mv scripts/create_site.py /app/scripts/create_site.py
+EOT
+
+# Compile translation files
+RUN <<EOT
+    /app/bin/python /compile_mo.py
+EOT
+
+FROM plone/server-prod-config:uv-${PYTHON_VERSION}
+
+LABEL maintainer="Plone Community <collective@plone.org>" \
+      org.label-schema.name="ploneconf2025-training-backend" \
+      org.label-schema.description="Plone Conference Training Backend." \
+      org.label-schema.vendor="Plone Community"
+
+# Copy the pre-built `/app` directory to the runtime container
+# and change the ownership to user app and group app in one step.
+COPY --from=builder --chown=500:500 /app /app
+
+RUN <<EOT
+    ln -s /data /app/var
+    chown -R 500:500 /data
+EOT
+```
+
+
 ## `plone/plone-zeo`
 
 Repository available at https://github.com/plone/plone-zeo/.
 
-Installs a ZEO database server.
+Provides a ZEO database server for your container based Plone CMS Stack. This allows you to scale to multiple backends without a relational Database like PostgreSQL or MySQL to store the content data.

docs/plone-deployment/intro.md

@@ -10,7 +10,7 @@ myst:
 
 Explore the intricate components of the Plone stack, tailored for deploying a modern Plone 6 site with a ReactJS-powered frontend.
 For deployments focusing on Plone Classic UI with server-side HTML rendering, the frontend component is excluded.
-This guide won't cover the integration of a web accelerator or the setup of an edge caching service.
+This guide won't explain all the details regarding the integration of a web accelerator or the setup of an edge caching service.
 
 ## Components of the Plone Stack
 
@@ -26,15 +26,22 @@ To understand the rewrite rules used in Traefik, please read our reference about
 
 ### Plone Frontend
 
-Hosted on a Node.js HTTP-server running on port 3000, the Plone frontend constitutes the default user interface and requires access to the Plone Backend and the web server.
+Hosted on a Node.js HTTP-server running on port 3000, the Plone frontend constitutes the default user interface. It serves a sever side rendered html version of the webpage and delivers the javascript bundles for the dynamic web application runnin gin the browser. It requires access to the Plone Backend and is exposed through the webserver and Accelerator.
 
 ### Plone Backend
 
-Operating on port 8080, the Plone Backend, a WSGI process, serves as the HTTP server hosting the Plone API. It's optimal to pair it with a specialized database like ZEO server or a relational database via RelStorage, supporting PostgreSQL, MySQL/MariaDB, and Oracle. A separate shared file system is essential for storing binary data if ZEO is employed.
+Operating on port 8080, the Plone Backend, Python  WSGI process, serves as the Application server hosting the Plone REST API. It delivers the content to the frontend Server and Browsers, stores it persistently in a backendd database or on filesystrem, and provides management API and security, like moving/copying content, advanced worfklows, security, etc.
+
+It's optimal to pair it with a specialized database like ZEO server or a relational database
 
 ### Database
 
-The database layer can range from a simple ZODB with file storage to more scalable options like a ZEO server or a relational database. This training will focus on using a PostgreSQL instance.
+The database layer can range from a simple ZODB with file storage to more scalable options like a ZEO server or a relational database via RelStorage, supporting PostgreSQL, MySQL/MariaDB, and Oracle. In the most simple version, the Plone backend can open a database on the local filesystem. But in this setup you are limited to only one backend. If you use ZEO, multiple Plone backends open a connection to the ZEO server, which manages the ZODB files. A separate shared file system like NFS is essential for storing binary data and sharing it to the backends with a ZEO setup.
+
+This training will focus on using a PostgreSQL instance. With a relational database, the backends connect through the RelStorage Adapter to the Database, where the ZODB is mapped in a very basic way to a number of specialised tables and fields to hold all data.
+
+Storing binary data (blobs) in a relational database or keeping them seperate in a blob storage is an advanced discussion and has different trade offs when looking at
+performance and high availability.
 
 ## Deployment Configurations
 
@@ -70,10 +77,10 @@ Web server → Web Accelerator → Plone Frontend → Plone Backend → Database
 
 In a multi-server environment, load distribution and redundancy are achieved through various configurations, enabling horizontal scaling.
 
-```{figure} _static/request_flow.png
-:alt: Flow of a request to `https://example.com`
+```{figure} _static/request_flow.svg
+:alt: Flow of a request to `https://plone.example.com`
 
-Flow of a request to https://example.com
+Flow of a request to https://plone.example.com
 ```
 
 #### Web server and Web Accelerator Layer
@@ -88,3 +95,9 @@ Hosts scalable Plone Frontend and Backend processes, enhancing performance and r
 
 Can host a ZEO server or a relational database. Managed relational database services with backup and replication are
 recommended for those unfamiliar with database management, with PostgreSQL being a preferred choice.
+
+### Scaling and High Availability
+
+The components of the CMS Stack as described above, can all run in a parallelized, distributed, and optionally high available setup. There are however extra requirements for some of the building blocks. The ingress web server (Traefik) can run multiple instances, but needs a way to sync TLS certificates and route information. The Varnish server can be run multiple times, but purge requests for changed/stale contenct, will need to be multiplied to all cache instances.
+
+The Plone frontend and backend servers can and should run with multiple instances in a larger setup to offer enough throughput. Lastly, the database should be dimensioned large enough to support al storage/retrieve requests from the Backends. When high availability is required, the database layer should be set up with Relstorage and a Relational Database that is configured accordingly (for example primary/secondary with active failover, or a a three node DB cluster).