Skip to content

Commit 24c90d1

Browse files
cailmdaleyclaude
cailmdaley
and
claude
committed
docker: split Dockerfile into runtime and dev targets
Rewrites the slim-bookworm Dockerfile as a two-target multi-stage build: - runtime — minimal image for canfar batch jobs and downstream FROM clauses. Ships astromatic binaries + jupyter + fitsio extras only. - dev — runtime plus everyday CLI tools (vim, less, tmux, htop, ripgrep, fd, jq, bat, git-lfs, …) and *all* extras pre-installed via the `dev` meta-extra (test, lint, doc, release, jupyter, fitsio). Default target when `--target` is omitted; published as `:<branch>` (no suffix). Both stages share a `base` stage carrying system deps + uv + the lockfile copy, so the heavy apt + wheel resolution is cached once. Addresses Martin's PR feedback on #719: - `uv run pytest: No such file or directory` — pytest now ships in the dev image (was previously only in the never-built `--extra test`). - `uv sync --extra X: Read-only file system` — the dev image has every extra pre-baked, so live `uv sync` is no longer needed at all. As a belt-and-braces also `chmod -R go+rwX /app` so non-root users on canfar/skaha can still mutate the venv when they want to. - "no vim in the slim image" — `vim`, `less`, `tmux`, `htop`, plus a curated subset of cailmdaley/containers (rg/fd/jq/bat/…) now ship in the dev target. deploy-image.yml builds, smoke-tests, and pushes both targets: - runtime: source-extractor / weightwatcher / psfex / shapepipe_run all invoked inside the built image. - dev: vim, ripgrep, pytest verified runnable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f1d414b commit 24c90d1

2 files changed

Lines changed: 146 additions & 31 deletions

File tree

‎.github/workflows/deploy-image.yml‎

Lines changed: 64 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ on: [push, workflow_dispatch]
33
env:
44
REGISTRY: ghcr.io
55
IMAGE_NAME: ${{ github.repository }}
6-
BRANCH: ${{ github.ref }}
76
jobs:
87
build-and-push-image:
98
runs-on: ubuntu-latest
@@ -26,37 +25,86 @@ jobs:
2625
with:
2726
driver-opts: network=host
2827

29-
- name: Extract metadata (tags, labels) for Docker
30-
id: meta
28+
# Two parallel tag sets. `dev` is the default (no suffix, e.g. `:latest`,
29+
# `:develop`); `runtime` carries a `-runtime` suffix.
30+
- name: Tags — dev (default)
31+
id: meta-dev
3132
uses: docker/metadata-action@v5
3233
with:
3334
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
3435

35-
- name: Build and export to Docker
36+
- name: Tags — runtime
37+
id: meta-runtime
38+
uses: docker/metadata-action@v5
39+
with:
40+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
41+
flavor: |
42+
suffix=-runtime,onlatest=true
43+
44+
# Build runtime first (smaller, used to smoke-test pipeline binaries)
45+
- name: Build runtime (load)
3646
uses: docker/build-push-action@v6
3747
with:
48+
context: .
49+
target: runtime
3850
load: true
39-
tags: ${{ steps.meta.outputs.tags }}
40-
labels: ${{ steps.meta.outputs.labels }}
51+
tags: ${{ steps.meta-runtime.outputs.tags }}
52+
labels: ${{ steps.meta-runtime.outputs.labels }}
53+
cache-from: type=gha
54+
cache-to: type=gha,mode=max
4155

42-
# Smoke-test the binaries baked into the image. Catches the class of
43-
# regression where the image builds but a runtime tool (sextractor,
44-
# weightwatcher) is missing or unrunnable on the deployment target.
45-
- name: Test — binaries
56+
# Smoke-test the binaries baked into the runtime image. Catches the
57+
# class of regression where the image builds but a runtime tool
58+
# (sextractor, weightwatcher) is missing or unrunnable.
59+
- name: Test runtime — binaries
4660
run: |
47-
IMAGE=$(echo "${{ steps.meta.outputs.tags }}" | head -n1)
61+
IMAGE=$(echo "${{ steps.meta-runtime.outputs.tags }}" | head -n1)
4862
docker run --rm "$IMAGE" source-extractor --version
4963
docker run --rm "$IMAGE" weightwatcher --version
5064
docker run --rm "$IMAGE" psfex --version
5165
52-
- name: Test — shapepipe entry point
66+
- name: Test runtime — shapepipe entry point
5367
run: |
54-
IMAGE=$(echo "${{ steps.meta.outputs.tags }}" | head -n1)
68+
IMAGE=$(echo "${{ steps.meta-runtime.outputs.tags }}" | head -n1)
5569
docker run --rm "$IMAGE" shapepipe_run -c /app/example/config.ini
5670
57-
- name: Push
71+
# Build dev (reuses cached `base` layer)
72+
- name: Build dev (load)
73+
uses: docker/build-push-action@v6
74+
with:
75+
context: .
76+
target: dev
77+
load: true
78+
tags: ${{ steps.meta-dev.outputs.tags }}
79+
labels: ${{ steps.meta-dev.outputs.labels }}
80+
cache-from: type=gha
81+
cache-to: type=gha,mode=max
82+
83+
# Verify the dev-only additions are present and runnable.
84+
- name: Test dev — interactive tools and test extras
85+
run: |
86+
IMAGE=$(echo "${{ steps.meta-dev.outputs.tags }}" | head -n1)
87+
docker run --rm "$IMAGE" vim --version | head -n1
88+
docker run --rm "$IMAGE" rg --version | head -n1
89+
docker run --rm "$IMAGE" pytest --version
90+
91+
# Push both targets
92+
- name: Push runtime
93+
uses: docker/build-push-action@v6
94+
with:
95+
context: .
96+
target: runtime
97+
push: true
98+
tags: ${{ steps.meta-runtime.outputs.tags }}
99+
labels: ${{ steps.meta-runtime.outputs.labels }}
100+
cache-from: type=gha
101+
102+
- name: Push dev
58103
uses: docker/build-push-action@v6
59104
with:
105+
context: .
106+
target: dev
60107
push: true
61-
tags: ${{ steps.meta.outputs.tags }}
62-
labels: ${{ steps.meta.outputs.labels }}
108+
tags: ${{ steps.meta-dev.outputs.tags }}
109+
labels: ${{ steps.meta-dev.outputs.labels }}
110+
cache-from: type=gha

‎Dockerfile‎

Lines changed: 82 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,27 @@
1-
FROM python:3.12-slim-bookworm
1+
# syntax=docker/dockerfile:1.7
2+
#
3+
# Two-target image:
4+
# --target runtime → minimal, for canfar batch jobs and downstream stacks
5+
# --target dev → runtime + everyday CLI tools + all extras (test,
6+
# lint, doc, …); default if --target is omitted
7+
#
8+
# Both share the `base` stage (system deps + uv + lockfile copy), so the
9+
# heavy apt + wheel-resolution work is cached once.
10+
11+
# ----------------------------------------------------------------------
12+
# base — system deps shared by every target
13+
# ----------------------------------------------------------------------
14+
FROM python:3.12-slim-bookworm AS base
215

3-
# Metadata
416
LABEL maintainer="martin.kilbinger@cea.fr"
5-
LABEL description="ShapePipe base image — slim Python + uv-frozen deps"
617

718
ENV SHELL=/bin/bash \
819
QT_QPA_PLATFORM=offscreen \
920
PIP_NO_CACHE_DIR=1 \
10-
DEBIAN_FRONTEND=noninteractive
21+
DEBIAN_FRONTEND=noninteractive \
22+
LANG=C.UTF-8
1123

12-
# System dependencies. Three categories:
24+
# System dependencies — three categories:
1325
# - astromatic binaries (psfex, source-extractor, weightwatcher) ship as
1426
# Debian packages on bookworm; preferred over building from source.
1527
# - compilers and dev libs needed to build the heavier wheels (galsim,
@@ -32,24 +44,30 @@ RUN apt-get update -y --quiet && \
3244
psfex source-extractor weightwatcher && \
3345
apt-get clean && rm -rf /var/lib/apt/lists/*
3446

35-
# Install uv — fast, reproducible dependency resolver and installer.
36-
# Deps are declared in pyproject.toml; exact transitive versions are frozen
37-
# in uv.lock. `uv sync --frozen` installs exactly what uv.lock specifies,
47+
# uv — fast reproducible Python deps installer. pyproject.toml + uv.lock
48+
# are the SSOT; `uv sync --frozen` installs exactly what uv.lock specifies,
3849
# so upstream changes only land when we deliberately regenerate the lockfile.
3950
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
4051

4152
WORKDIR /app
4253
COPY pyproject.toml uv.lock /app/
4354

44-
# Install runtime + jupyter + fitsio extras from the lockfile into /app/.venv.
45-
# `--no-install-project` skips installing shapepipe itself (the source isn't
46-
# copied yet); we install it `--no-deps` below once the source is available.
55+
# ----------------------------------------------------------------------
56+
# runtime — minimal target for batch jobs and downstream FROM clauses
57+
# ----------------------------------------------------------------------
58+
FROM base AS runtime
59+
LABEL description="ShapePipe runtime — slim Python + uv-frozen deps"
60+
61+
# Lockfile-frozen Python deps + jupyter + fitsio. Test/lint/doc extras
62+
# are intentionally left out here; they live in the dev target.
4763
RUN uv sync --frozen --no-install-project --extra jupyter --extra fitsio
4864

4965
# Copy the source and install shapepipe into the same venv.
5066
COPY . /app/.
51-
RUN chown -R root:root /app && chmod -R u+rwX /app
52-
RUN uv pip install --no-deps -e . && \
67+
# go+rwX so non-root users on canfar/skaha can read/traverse /app and
68+
# write into the venv when they need to (e.g. uv add for ad-hoc deps).
69+
RUN chmod -R go+rwX /app && \
70+
uv pip install --no-deps -e . && \
5371
for ext in .py .sh .bash; do \
5472
for script in /app/scripts/*/*$ext; do \
5573
link_name=$(basename $script $ext); \
@@ -59,5 +77,54 @@ RUN uv pip install --no-deps -e . && \
5977

6078
# Activate the uv-managed venv on container start so shapepipe_run etc
6179
# resolve against it without explicit activation.
62-
ENV PATH="/app/.venv/bin:${PATH}"
63-
ENV VIRTUAL_ENV=/app/.venv
80+
ENV PATH="/app/.venv/bin:${PATH}" \
81+
VIRTUAL_ENV=/app/.venv
82+
83+
# ----------------------------------------------------------------------
84+
# dev — everyday working environment (default target)
85+
# ----------------------------------------------------------------------
86+
FROM base AS dev
87+
LABEL description="ShapePipe dev — runtime + interactive CLI tools + all extras"
88+
89+
# Interactive tools for working inside the container. Curated subset of
90+
# cailmdaley/containers focused on the search/edit/process loop; heavier
91+
# tooling (neovim, polspice, quarto, zellij) is intentionally not here.
92+
RUN apt-get update -y --quiet && \
93+
apt-get install -y --no-install-recommends \
94+
vim \
95+
less \
96+
tmux \
97+
htop \
98+
procps \
99+
ripgrep \
100+
fd-find \
101+
jq \
102+
bat \
103+
curl \
104+
ca-certificates \
105+
git-lfs \
106+
rsync \
107+
unzip zip \
108+
openssh-client \
109+
locales && \
110+
if command -v batcat >/dev/null; then ln -sf "$(command -v batcat)" /usr/local/bin/bat; fi && \
111+
if command -v fdfind >/dev/null; then ln -sf "$(command -v fdfind)" /usr/local/bin/fd; fi && \
112+
apt-get clean && rm -rf /var/lib/apt/lists/*
113+
114+
# All extras pre-installed (dev = doc + jupyter + lint + release + test +
115+
# fitsio). Pre-installing avoids the read-only-fs failure Martin hit when
116+
# trying to live `uv sync --extra test` inside the runtime image on canfar.
117+
RUN uv sync --frozen --no-install-project --extra dev
118+
119+
COPY . /app/.
120+
RUN chmod -R go+rwX /app && \
121+
uv pip install --no-deps -e . && \
122+
for ext in .py .sh .bash; do \
123+
for script in /app/scripts/*/*$ext; do \
124+
link_name=$(basename $script $ext); \
125+
ln -s $script /usr/local/bin/$link_name; \
126+
done; \
127+
done
128+
129+
ENV PATH="/app/.venv/bin:${PATH}" \
130+
VIRTUAL_ENV=/app/.venv

0 commit comments

Comments
 (0)