Skip to content

Commit 3685448

Browse files
eduralpheduralph
authored
ci: forward-port current pipeline (harness + timeout) to gramps60 (#41)
Bring maintenance/gramps60's .github/ up to feature/ci-cd-pipeline-upstream (the 820 PR head, which is gramps60-based), same forward-port as gramps61. gramps60 was further behind — also updates the gramps-ci Dockerfile and docker-build.yml in addition to the run_addon_tests.py harness, addon_system_deps.py, gi_bootstrap, and ci.yml wiring. Branch-neutral ci.yml unchanged in intent. Forward-port only; no addon changes.
1 parent 400180d commit 3685448

8 files changed

Lines changed: 1127 additions & 46 deletions

File tree

‎.github/CI-MAINTAINER.md‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# CI Maintainer Notes
2+
3+
Operational steps that a `gramps-project/addons-source` maintainer
4+
needs to handle once PR 820 (and the branch-neutral follow-up) lands.
5+
The pipeline is otherwise self-driving — this document is only about
6+
the rough edges.
7+
8+
For day-to-day addon-release maintenance see [MAINTAINERS.md](../MAINTAINERS.md);
9+
for the contributor-facing summary of what a green CI check means on
10+
an unreleased branch see [CONTRIBUTING.md](../CONTRIBUTING.md#work-towards-a-merge).
11+
12+
## Contents
13+
14+
1. [One-time setup when the PR first merges](#one-time-setup-when-the-pr-first-merges)
15+
2. [Creating a new maintenance branch](#creating-a-new-maintenance-branch)
16+
3. [When a Gramps minor release lands on PyPI](#when-a-gramps-minor-release-lands-on-pypi)
17+
4. [Diagnostic log markers](#diagnostic-log-markers)
18+
5. [Optional future-proofing knobs](#optional-future-proofing-knobs)
19+
20+
## One-time setup when the PR first merges
21+
22+
### 1. Make the `gramps-ci` GHCR package public
23+
24+
`docker-build.yml` pushes images to
25+
`ghcr.io/gramps-project/addons-source/gramps-ci:<suffix>` using the
26+
workflow's `GITHUB_TOKEN`. GHCR creates the package as **private** the
27+
first time. Same-repo CI keeps working (token covers own packages),
28+
but **fork PRs cannot pull the image** because their `GITHUB_TOKEN`
29+
has no read access to private packages in `gramps-project`. Fork-PR
30+
container jobs would fail at "Initialize containers" with an
31+
authentication error.
32+
33+
Fix once, immediately after the first `Build Docker Images` run
34+
finishes:
35+
36+
1. Go to <https://github.com/orgs/gramps-project/packages/container/addons-source%2Fgramps-ci/settings>
37+
2. Under "Danger Zone" → "Change visibility" → set to **Public**
38+
39+
Every existing and future `gramps-ci:<suffix>` tag inherits public
40+
visibility from this single setting.
41+
42+
### 2. Expect the first-push race on `maintenance/gramps60`
43+
44+
The first push event after merge fires both workflows in parallel:
45+
46+
- `Build Docker Images` builds and pushes `gramps-ci:gramps60`
47+
(~5 min cold).
48+
- `CI` runs `setup` (~2 s), then its container jobs try to pull the
49+
image.
50+
51+
Because both start on the same push, the CI container jobs race the
52+
image push and may fail at "Initialize containers" the first time.
53+
This race only happens once per branch:
54+
55+
1. Wait for `Build Docker Images` to complete.
56+
2. Open the failed CI run → click "Re-run failed jobs".
57+
3. Subsequent pushes find the image already in GHCR — no race.
58+
59+
This is also explained in the header comment of `ci.yml`.
60+
61+
## Creating a new maintenance branch
62+
63+
When a new Gramps minor series goes into development and addons need
64+
a corresponding branch (e.g. `maintenance/gramps62` once 6.2 opens):
65+
66+
```
67+
git branch maintenance/gramps62 maintenance/gramps61
68+
git push origin maintenance/gramps62
69+
```
70+
71+
No workflow edits required. The workflows derive everything from
72+
`github.ref_name`. The first push fires the same race described
73+
above — re-run the failed CI jobs once `Build Docker Images`
74+
finishes.
75+
76+
The setup job's regex (`gramps[0-9][0-9]`) requires a two-digit
77+
suffix. When Gramps 10.0 opens this regex needs updating in two
78+
places (`ci.yml` setup job and `docker-build.yml` params step).
79+
80+
## When a Gramps minor release lands on PyPI
81+
82+
The hybrid Dockerfile auto-detects PyPI availability:
83+
84+
- `pip install "gramps==X.Y.*"` succeeds → image installs the tagged
85+
PyPI release (`::notice::` log line).
86+
- pip reports "No matching distribution found" → image falls back to a
87+
SHA-pinned `git clone` of `gramps-project/gramps@maintenance/grampsNN`
88+
at the SHA captured by `docker-build.yml`'s params step
89+
(`::warning::` log line).
90+
91+
When `gramps==6.1.0` is finally published to PyPI, the next image
92+
rebuild on `maintenance/gramps61` silently switches from "git tip" to
93+
"PyPI release" — no maintainer action needed.
94+
95+
To **immediately** rebuild against the new PyPI release without
96+
waiting for the next push:
97+
98+
1. Open the `Build Docker Images` workflow in the Actions tab.
99+
2. "Run workflow" → select `maintenance/grampsNN` → Run.
100+
101+
The `::notice::` line in the build log confirms the switch.
102+
103+
## Diagnostic log markers
104+
105+
When investigating a CI failure, the install-step output in
106+
`Build Docker Images` carries these annotations:
107+
108+
| Annotation | Meaning |
109+
| --- | --- |
110+
| `::notice::installed gramps==X.Y.* from PyPI` | Released-path install. CI is testing against a tagged release. |
111+
| `::warning::no gramps==X.Y.* on PyPI; installing from gramps-project/gramps@maintenance/grampsNN at <sha>` | Unreleased-branch fallback. CI is testing against the upstream branch tip at `<sha>`. Visible to contributors via the CONTRIBUTING.md note. |
112+
| `::error::no gramps==X.Y.* on PyPI and GRAMPS_FALLBACK_SHA is unset` | `git ls-remote` in `docker-build.yml`'s params step returned no SHA for `maintenance/grampsNN` on `gramps-project/gramps`. The matching upstream branch is missing, or the addons-source branch is misnamed. |
113+
| `::error::pip install gramps failed (non-version reason)` | pip failed for a network/registry reason, not because the version is missing. Captured stderr is dumped after the line. The build does **not** fall back to git in this case — by design, so a transient PyPI hiccup cannot silently flip a released branch into "git tip" mode. |
114+
115+
Other useful entry points:
116+
117+
- `ci.yml` setup job log shows the derived `branch_suffix` and
118+
`ci_image`. A failure here means the branch name doesn't match
119+
`maintenance/gramps[0-9][0-9]`.
120+
- `docker-build.yml` params step log shows the captured
121+
`fallback_sha`. An empty value means upstream `gramps-project/gramps`
122+
has no matching maintenance branch (warning issued; image build will
123+
fail iff the fallback is needed).
124+
125+
## Optional future-proofing knobs
126+
127+
Not needed today; record here in case they ever come up.
128+
129+
### Upstream gramps repo URL
130+
131+
The Dockerfile hardcodes `https://github.com/gramps-project/gramps.git`
132+
as the fallback source. If the project ever reorgs or renames, this
133+
URL must change in one place (`.github/docker/gramps-ci/Dockerfile`).
134+
It could be parameterised as a build arg (`ARG GRAMPS_UPSTREAM_REPO`)
135+
with the current URL as default, but a hardcoded value keeps the
136+
Dockerfile simpler and a rename is a sufficiently large event that
137+
editing one string is not the bottleneck.
138+
139+
### GHCR tag retention
140+
141+
`docker-build.yml` pushes both a moving `gramps-ci:<suffix>` tag (e.g.
142+
`gramps60`) and a per-commit `gramps-ci:<suffix>-<short-sha>` tag.
143+
The moving tag is always overwritten; the SHA tags accumulate over
144+
time. Set a retention policy on the GHCR package settings page if
145+
the count becomes inconvenient — the moving tags are what CI consumes.

‎.github/docker/gramps-ci/Dockerfile‎

Lines changed: 72 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
# .github/docker/gramps-ci/Dockerfile
22
#
3-
# Unified Gramps 6.0 CI image. Includes everything jobs need:
3+
# Gramps CI image. The Gramps minor series (6.0, 6.1, …) is picked at
4+
# build time via the GRAMPS_SERIES build arg, so the same Dockerfile
5+
# produces gramps-ci:gramps60, gramps-ci:gramps61, … without per-branch
6+
# edits. Includes everything jobs need:
47
# - Python + pip-installed Gramps, PyGObject, pycairo
58
# - GTK typelibs (so addon modules that `from gi.repository import Gtk`
69
# at module load time are importable — widgets still need xvfb to render)
@@ -13,11 +16,44 @@
1316
# `--init` (or use a container runtime that injects tini) because xvfb-run
1417
# hangs if it inherits PID 1.
1518
#
19+
# Gramps install path. PyPI is tried first; when no gramps==${SERIES}.*
20+
# release exists (e.g. while 6.1 is still in development) the build
21+
# falls back to a SHA-pinned git clone of
22+
# gramps-project/gramps@maintenance/gramps${SERIES_NODOT}. Other pip
23+
# failures (network, etc.) are NOT silently retried so a transient
24+
# PyPI outage cannot flip a normally-released branch to "test against
25+
# moving tip" mode. docker-build.yml captures the upstream SHA via
26+
# git ls-remote and passes it in as GRAMPS_FALLBACK_SHA; the SHA
27+
# becomes part of the buildx cache key so a moved upstream tip
28+
# actually re-runs the install layer.
29+
#
30+
# Local build (released series):
31+
# docker build --build-arg GRAMPS_SERIES=6.0 .github/docker/gramps-ci
32+
#
33+
# Local build (unreleased series, e.g. 6.1):
34+
# sha=$(git ls-remote https://github.com/gramps-project/gramps.git \
35+
# refs/heads/maintenance/gramps61 | awk '{print $1}')
36+
# docker build --build-arg GRAMPS_SERIES=6.1 \
37+
# --build-arg GRAMPS_FALLBACK_SHA=$sha \
38+
# .github/docker/gramps-ci
39+
#
1640
ARG PYTHON_VERSION=3.12
1741
FROM python:${PYTHON_VERSION}-slim
1842

43+
# Gramps minor series to install (e.g. 6.0, 6.1). No default — must be
44+
# passed explicitly so a wrong default cannot silently produce an image
45+
# for the wrong Gramps series. docker-build.yml derives this from the
46+
# branch ref.
47+
ARG GRAMPS_SERIES
48+
# Commit SHA on gramps-project/gramps@maintenance/grampsNN. Only used
49+
# when no gramps==${GRAMPS_SERIES}.* release exists on PyPI. Ignored
50+
# otherwise. Empty default is intentional — on released branches the
51+
# fallback never fires.
52+
ARG GRAMPS_FALLBACK_SHA=""
53+
RUN [ -n "$GRAMPS_SERIES" ] || { echo "GRAMPS_SERIES is required (e.g. 6.0)"; exit 1; }
54+
1955
LABEL org.opencontainers.image.source="https://github.com/gramps-project/addons-source"
20-
LABEL org.opencontainers.image.description="Unified Gramps 6.0 CI image (Python, Gramps, GTK typelibs, xvfb)"
56+
LABEL org.opencontainers.image.description="Gramps ${GRAMPS_SERIES} CI image (Python, Gramps, GTK typelibs, xvfb)"
2157

2258
RUN apt-get update && apt-get install -y --no-install-recommends \
2359
libgirepository-2.0-dev \
@@ -26,6 +62,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
2662
gir1.2-pango-1.0 \
2763
gir1.2-gdkpixbuf-2.0 \
2864
gir1.2-atk-1.0 \
65+
gir1.2-gexiv2-0.10 \
2966
gcc \
3067
pkg-config \
3168
python3-dev \
@@ -37,13 +74,39 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
3774
xauth \
3875
&& rm -rf /var/lib/apt/lists/*
3976

40-
RUN pip install --no-cache-dir \
41-
PyGObject \
42-
pycairo \
43-
"gramps>=6.0,<6.1" \
44-
orjson \
45-
ruff \
46-
dbf
77+
# Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are
78+
# NOT baked in here — ci.yml's "Install addon runtime deps (derived from
79+
# requires_mod)" step pip-installs them at CI runtime from every
80+
# .gpr.py's requires_mod list, matching what Gramps' Addon Manager does
81+
# for an end user. Keeps .gpr.py the single source of truth.
82+
RUN pip install --no-cache-dir PyGObject pycairo orjson ruff
83+
84+
# Install gramps: PyPI first, SHA-pinned git clone as fallback.
85+
RUN /bin/bash -eo pipefail <<'BASH'
86+
suffix_nodot="${GRAMPS_SERIES//./}"
87+
if pip install --no-cache-dir "gramps==${GRAMPS_SERIES}.*" 2>/tmp/pip.err; then
88+
echo "::notice::installed gramps==${GRAMPS_SERIES}.* from PyPI"
89+
elif grep -q "No matching distribution found for gramps==" /tmp/pip.err; then
90+
if [ -z "${GRAMPS_FALLBACK_SHA}" ]; then
91+
echo "::error::no gramps==${GRAMPS_SERIES}.* on PyPI and GRAMPS_FALLBACK_SHA is unset"
92+
exit 1
93+
fi
94+
echo "::warning::no gramps==${GRAMPS_SERIES}.* on PyPI; installing from gramps-project/gramps@maintenance/gramps${suffix_nodot} at ${GRAMPS_FALLBACK_SHA}"
95+
mkdir -p /tmp/gramps
96+
cd /tmp/gramps
97+
git init -q
98+
git remote add origin https://github.com/gramps-project/gramps.git
99+
git fetch --depth 1 origin "${GRAMPS_FALLBACK_SHA}"
100+
git checkout FETCH_HEAD
101+
pip install --no-cache-dir .
102+
cd /
103+
rm -rf /tmp/gramps
104+
else
105+
echo "::error::pip install gramps failed (non-version reason); aborting rather than silently switching to git tip"
106+
cat /tmp/pip.err
107+
exit 1
108+
fi
109+
BASH
47110

48111
RUN apt-get purge -y gcc python3-dev pkg-config && apt-get autoremove -y
49112

‎.github/environment.yml‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,17 @@ dependencies:
66
- pygobject
77
- gtk3
88
- pip
9+
# Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are
10+
# installed at CI runtime by ci.yml's auto-derive step from .gpr.py
11+
# requires_mod — single source of truth. Keep only the stable base
12+
# here (Gramps + orjson for plugin registration).
13+
#
14+
# conda-forge has no gramps 6.1 yet, so on a maintenance/gramps61 (or later)
15+
# branch this resolves to 6.0.x — i.e. the Windows lane validates addons
16+
# against conda-forge's newest in-range gramps, not the branch's exact series
17+
# (the Linux CI image git-builds the exact series; conda-Windows cannot — see
18+
# ci.yml's "Report gramps-vs-branch series" step). When 6.1 reaches
19+
# conda-forge the pin picks it up automatically.
920
- pip:
1021
- "gramps>=6.0,<6.1"
1122
- orjson
12-
- dbf

0 commit comments

Comments
 (0)