diff --git a/.github/workflows/deployment.yml b/.github/workflows/deployment.yml index e8708b6af..018944d77 100644 --- a/.github/workflows/deployment.yml +++ b/.github/workflows/deployment.yml @@ -6,7 +6,9 @@ on: - 'v*' pull_request: paths-ignore: - - 'CHANGELOG.md' + - '**.md' + - 'docs/**' + - 'mkdocs.yml' schedule: - cron: '0 10 * * *' diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 11608b033..1ba658d78 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -4,11 +4,17 @@ on: push: branches: - main + paths-ignore: + - '**.md' + - 'docs/**' + - 'mkdocs.yml' pull_request: branches: - main paths-ignore: - - 'CHANGELOG.md' + - '**.md' + - 'docs/**' + - 'mkdocs.yml' defaults: run: diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 37648d0e9..c56bc9a45 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -4,11 +4,17 @@ on: push: branches: - main + paths-ignore: + - '**.md' + - 'docs/**' + - 'mkdocs.yml' pull_request: branches: - main paths-ignore: - - 'CHANGELOG.md' + - '**.md' + - 'docs/**' + - 'mkdocs.yml' concurrency: group: ${{ github.workflow }}-${{ github.ref }} diff --git a/docs/admin/how-to/install/register-a-vo.md b/docs/admin/how-to/install/register-a-vo.md index 58f9ab56f..50b4ce153 100644 --- a/docs/admin/how-to/install/register-a-vo.md +++ b/docs/admin/how-to/install/register-a-vo.md @@ -3,7 +3,6 @@ Registering a VO means allowing users of a specific Virtual Organization to use DiracX services. In practice, this can be done once an external IdP is used for managing such VO. By default all VOs managed by a DIRAC/DiracX instance are already "registered" (or "enabled", in other words). There is anyway the possibility to "Disable" VOs by filling the option `DiracX > DisabledVOs` with the list of VOs that can not use DiracX: -```` ??? example "Configuration example" ```yaml @@ -12,7 +11,6 @@ By default all VOs managed by a DIRAC/DiracX instance are already "registered" ( DisabledVOs = vo_x, vo_y } ``` -```` ## Prerequisites diff --git a/docs/admin/how-to/make-a-release.md b/docs/admin/how-to/make-a-release.md new file mode 100644 index 000000000..0348664da --- /dev/null +++ b/docs/admin/how-to/make-a-release.md @@ -0,0 +1,71 @@ +This page describes how to mechanically make releases of DiracX and extensions. See the [reference](../reference/making-releases.md) documentation for more details about how releases are made. + +# How to make a new DiracX release + +Execute the following points in the presented order. + +1. Find the `release-please` [pull request](https://github.com/DIRACGrid/diracx/pulls?q=is%3Apr+is%3Aopen+%22chore%28main%29%3A+release%22) for your release. If you can't find it, it means that the currently existing latest release includes already all code changes. + +2. Review the release number **and** the content of the changelog + + - If you need to change the release number, follow the [ `release-please` doc](https://github.com/googleapis/release-please?tab=readme-ov-file#how-do-i-change-the-version-number) + + ```bash + git commit --allow-empty -m "chore: release 0.0.3" -m "Release-As: v0.0.3" + git push upstream HEAD:main + ``` + + - If you want to update the release notes manually, follow the [`release-please` doc](https://github.com/googleapis/release-please?tab=readme-ov-file#how-can-i-fix-release-notes) + +3. Merge the `release-please` PR + +4. Check the [deployment action](https://github.com/DIRACGrid/diracx/actions/workflows/deployment.yml?query=event%3Apush+actor%3ADIRACGridBot) + +5. When the above will be finished, check that the [CI](https://github.com/DIRACGrid/diracx-charts/actions/workflows/chart-ci.yml?query=actor%3ADIRACGridBot) in `diracx-charts` is successful + +At this stage, there is a new `diracx` and `diracx-chart` release. + +# How to make a new DiracX-Web release + +Execute the following points in the presented order. + +1. Find the `release-please` [pull request](https://github.com/DIRACGrid/diracx-web/pulls?q=is%3Apr+is%3Aopen+%22chore%28main%29%3A+release%22) for your release. If you can't find it, it means that the currently existing latest release includes already all code changes. + +2. Review the release number **and** the content of the changelog + + - If you need to change the release number, follow the [`release-please` doc](https://github.com/googleapis/release-please?tab=readme-ov-file#how-do-i-change-the-version-number) + + ```bash + git commit --allow-empty -m "chore: release 0.0.3" -m "Release-As: v0.0.3" + git push upstream HEAD:main + ``` + + - If you want to update the release notes manually, follow the [`release-please` doc](https://github.com/googleapis/release-please?tab=readme-ov-file#how-can-i-fix-release-notes) + +3. Merge the `release-please` PR + +4. Check the [deployment action](https://github.com/DIRACGrid/diracx-web/actions/workflows/deployment.yml?query=event%3Apush) + +5. When the above is finished, check that the [CI](https://github.com/DIRACGrid/diracx-charts/actions/workflows/chart-ci.yml) in `diracx-charts` is successful + +At this stage, there is a new `diracx-web` release and `diracx-chart` release. + +## Making the extension release + +Making a release of an extension depends on how the extension has been set up. Below are recommended workflows: + +=== "Generic GitLab" + + 1. Ensure any desired merge requests have been merged. + 2. Ensure the `pyproject.toml` contains the desired DiracX dependency + - If it does not, run the pipeline schedule to run renovate, which should trigger a merge request to update the version. + - Merge the renovate-generated MR. + 3. Create tag (and optionally [release](<>)) for your extension + 4. Updates to extension's "charts" repo will be pushed automatically or via merge requests depending on your setup. + +=== "LHCb" + + - The LHCb installation follows the "Generic GitLab" workflow. + - The renovate job runs every Monday at 22:10 UTC. It can be manually triggered [here](https://gitlab.cern.ch/lhcb-dirac/lhcbdiracx/-/pipeline_schedules). + - The LHCb certification set up is updated automatically, see [here](https://gitlab.cern.ch/lhcb-dirac/lhcbdiracx-charts/-/pipelines?page=1&scope=all&ref=lhcbdiracx-cert) for the pipelines. + - The LHCb production setup is deployed automatically when the associated merge request is merged. See [here](https://gitlab.cern.ch/lhcb-dirac/lhcbdiracx-charts/-/merge_requests/?sort=created_date&state=opened&author_username=project_217407_bot_a1f6511ecbeab04354572cf37500b0a3&first_page_size=20) for the pending updates. diff --git a/docs/admin/reference/making-releases.md b/docs/admin/reference/making-releases.md new file mode 100644 index 000000000..d38e4b3f9 --- /dev/null +++ b/docs/admin/reference/making-releases.md @@ -0,0 +1,93 @@ +# Making and deploying releases + +!!! warning "Version compatibility" + + DiracX should be upgraded before upgrading legacy DIRAC installations. + +The versioning of `DIRAC`, `DIRACCommon`, `diracx`, `diracx-charts` and all the extensions are linked to one another. + +## Dependencies between diracx, DIRAC and DIRACCommon + +Both `DIRAC` and `diracx` depend on `DIRACCommon`. However, `DIRACCommon` is in the same repository as `DIRAC`, and versioned together. This means that if we need to move code from `DIRAC` to `DIRACCommon` such that it is usable by `diracx`, we need first a new release of `DIRAC` and `DIRACCommon` just moving the code, before being able to use it in `diracx`. + +The releases of `DIRAC` and `DIRACCommon` are managed as they have always been, and are described in the `DIRAC` documentation + +## Dependencies between diracx and diracx-charts + +`diracx-charts` has a version number, as well as an `appversion`, which corresponds to the `diracx` version. This means that a new release of `diracx` requires a new `diracx-charts` release, which will update: + +- the diracx-charts version +- the AppVersion +- the container versions + +The `update-charts` [CI job](https://github.com/DIRACGrid/diracx/blob/e552651ac682e7461bc338856d295b0651ec2a26/.github/workflows/deployment.yml#L168) in `diracx` pushes a new commit in `diracx-charts`, which results in a new `diracx-charts` release. + +## Dependencies between diracx, diracx-web and diracx-charts + +The `diracx-web` version is only very loosely coupled with the `diracx` version. +A new `diracx-web` release will trigger a new `diracx-charts` updating: + +- the chart version +- the container versions +- NOT the AppVersion + +It does not bump the AppVersion because it refers to the `diracx` version, which is more relevant. + +```mermaid +flowchart LR + subgraph diracx-block["diracx"] + diracx-charts("diracx-charts") + diracx("diracx") + diracx-web("diracx-web") + end + + + diracx -->|commit| diracx-charts + diracx-web -->|commit| diracx-charts + + linkStyle 0 stroke: blue + linkStyle 1 stroke: blue + +``` + +## diracx release + +Releases are managed by `ReleasePlease`. The token used to perform it belongs to `DIRACGridBot` github user. + +!!! info "Why a bot token?" + + `GITHUB_TOKEN` cannot trigger downstream workflows (GitHub's anti-loop protection). Using a bot PAT ensures that the tag created by `release-please` triggers the deployment and chart update jobs. + +A new release creates: + +- a [github release](https://github.com/DIRACGrid/diracx/releases) +- a [pypi release](https://pypi.org/project/diracx/) +- new [docker images](https://github.com/DIRACGrid/diracx/) + +### Force version number + +You can force a specific version number following the [doc](https://github.com/googleapis/release-please?tab=readme-ov-file#how-do-i-change-the-version-number) + +```bash +git commit --allow-empty -m "chore: release 0.0.3" -m "Release-As: v0.0.3" +git push upstream HEAD:main +``` + +## diracx-web release + +Releases are managed by `ReleasePlease`. The token used to perform it belongs to `DIRACGridBot` github user, same as `diracx`. + +!!! info "Why a bot token?" + + `GITHUB_TOKEN` cannot trigger downstream workflows (GitHub's anti-loop protection). Using a bot PAT ensures that the tag created by `release-please` triggers the deployment and chart update jobs. + +A new release creates: + +- a [github release](https://github.com/DIRACGrid/diracx-web/releases) +- an [npm release](https://www.npmjs.com/package/@dirac-grid/diracx-web-components) for `diracx-web-components` +- new [docker images](https://github.com/DIRACGrid/diracx-web/) +- updated [Storybook documentation](https://diracgrid.github.io/diracx-web/) on GitHub Pages + +## diracx-charts releases + +Every commit in master triggers a new release. The releases are published on a [helm repo](https://charts.diracgrid.org/index.yaml) diff --git a/docs/dev/explanations/extensions.md b/docs/dev/explanations/extensions.md index fe2856cec..ec3c21b91 100644 --- a/docs/dev/explanations/extensions.md +++ b/docs/dev/explanations/extensions.md @@ -35,6 +35,7 @@ For most components you can extend them in isolation without needing to have an The exception to this is when extending `diracx-routers` you MUST also extend `diracx-client` and your client extension MUST be regenerated for every DiracX release. More details about this can be found in the [dedicated how to](../how-to/client-extension.md). + Additionally, you will need an umbrella helm chart for your extension. ## Gubbins @@ -225,3 +226,61 @@ Note that even if you have your own `testing` package depending on `diracx-testi ## `Dockerfile` `extensions/gubbins/containers/Dockerfile` contains an example of `Dockerfile` for extensions. + +## Extension chart + +Although you could simply replace the docker images used in the `diracx-charts` by your own, having the umbrella extension is strongly recommended. The reason is that you will be able to properly manage your release cycle and manage proper dependencies, `AppVersion` corresponding to your app, etc. + +Note that your `value.yaml` needs an extra `diracx` layer: + +```yaml +diracx: + diracx: + settings: + DIRACX_CONFIG_BACKEND_URL: "some github url" +``` + +## Repository setup + +How the repositories are setup is very extension specific and flexible. Here we describe one possible set up. + +This setup works by having one GitLab repository for you extension's code and a separate GitLab repository for an extension to `diracx-charts`. One or more installations are then managed via the GitLab CI of `-charts`. Updating the DiracX dependency of `` is handled via [`renovate`](https://github.com/renovatebot/renovate). `Renovate` should be configured to update the `pyproject.toml` and the `dockerfile` (see [lhcb configuration](https://gitlab.cern.ch/lhcb-dirac/lhcbdiracx/-/blob/master/renovate.json)) Updating the `-charts` repository is handled by the GitLab CI job of `` whenever a Git tag is created. + +A new extension release should trigger a release of your extension chart. (See `deploy-chart` job in [lhcb ci config](https://gitlab.cern.ch/lhcb-dirac/lhcbdiracx/-/blob/master/.gitlab-ci.yml)) +In this case, a new extension-chart release means updating: + +- the chart version +- the AppVersion +- the containers versions +- the diracx-charts version to the latest compatible with the diracx version required by lhcbdiracx + +```mermaid +flowchart LR + subgraph diracx-block["diracx"] + diracx-charts("diracx-charts") + diracx("diracx") + diracx-web("diracx-web") + end + + subgraph extension-block["extension"] + extension("extension") + extension-web("extension-web") + extension-charts("extension-charts") + end + + + diracx -->|commit| diracx-charts + diracx-web -->|commit| diracx-charts + + diracx -->|renovate| extension + diracx-web -->|renovate| extension-web + extension -->|commit and bump diracx-charts| extension-charts + extension-web -->|commit| extension-charts + + linkStyle 0 stroke: blue + linkStyle 1 stroke: blue + linkStyle 2 stroke: red + linkStyle 3 stroke: red + linkStyle 4 stroke: blue + linkStyle 5 stroke: blue +``` diff --git a/mkdocs.yml b/mkdocs.yml index 1b7cf5095..d2f5a61d0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -92,6 +92,7 @@ nav: - Debugging: admin/how-to/debugging.md - Upgrading: admin/how-to/upgrading.md - Rotate a secret: admin/how-to/rotate-a-secret.md + - Make a release: admin/how-to/make-a-release.md - Explanations: - admin/explanations/index.md - Auth in DiracX: admin/explanations/auth-with-diracx.md @@ -111,6 +112,7 @@ nav: - Security Model: admin/reference/security_model.md - Settings and Preferences: admin/reference/settings-and-preferences.md - Environment variables: admin/reference/env-variables.md + - Making and deploying releases: admin/reference/making-releases.md - Charts values: admin/reference/values.md - RUN_PROD.md - SSO.md