Skip to content

Commit 981c8e0

Browse files
cailmdaleyclaude
andcommitted
docs: make README a general front door; move candide detail to container.md
The README had tunnel-visioned onto a candide-specific SLURM walkthrough — wrong altitude for the project's landing page, where the broader community arrives. Restructure it as a front door: a one-sentence-deeper description of what ShapePipe does, a Quickstart that runs the bundled example straight from the published container in one command (apptainer or docker, no install, no cluster specifics), the image tag scheme, and a Documentation signpost to the published pages. Move the candide cluster walkthrough (quota-safe pull -> sbatch candide_smp.sh, the SPDIR bind-mount, the MPI PMIx note) into a new "Running on a cluster (SLURM)" section in container.md, which the README links to. Drop the test-assertion prose ("logs ... and exits 0") that read like a CI check rather than user docs. Both quickstart commands verified on candide against :develop-runtime (including the no-pre-pull `apptainer exec docker://...` form): the bundled example runs to completion, 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 52f0e41 commit 981c8e0

2 files changed

Lines changed: 65 additions & 31 deletions

File tree

README.rst

Lines changed: 27 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -16,44 +16,40 @@ ShapePipe
1616
:target: https://github.com/CosmoStat/shapepipe/releases/latest
1717

1818
ShapePipe is a galaxy shape measurement pipeline developed within the
19-
CosmoStat lab at CEA Paris-Saclay.
19+
CosmoStat lab at CEA Paris-Saclay. It runs the full chain from raw survey
20+
images to calibrated shear catalogues — object detection, PSF modelling, and
21+
shape measurement — and produced the first UNIONS cosmic-shear release.
2022

21-
See the `documentation <https://cosmostat.github.io/shapepipe>`_ for details
22-
on how to install and run ShapePipe.
23+
Quickstart
24+
----------
2325

24-
Quickstart on a cluster (candide)
25-
---------------------------------
26-
27-
ShapePipe ships as a container image — the supported way to run it (see
28-
``docs/source/container.md``). On a SLURM cluster such as candide, pull the slim
29-
``runtime`` image once and submit the bundled example, which runs the pipeline
30-
on a single CFIS tile:
26+
ShapePipe ships as a container image, so you can run the bundled example
27+
pipeline — a single CFIS tile through the full chain — without installing
28+
anything:
3129

3230
.. code-block:: bash
3331
34-
# 0. Get a clone (holds the example configs, data, and job scripts).
35-
git clone https://github.com/CosmoStat/shapepipe.git
36-
cd shapepipe
32+
# Apptainer (HPC, no root needed):
33+
apptainer exec docker://ghcr.io/cosmostat/shapepipe:develop-runtime shapepipe_run_example
34+
35+
# ...or Docker:
36+
docker run --rm ghcr.io/cosmostat/shapepipe:develop-runtime shapepipe_run_example
3737
38-
# 1. Keep the SIF and Apptainer's scratch off the quota-limited $HOME.
39-
# candide's home quota is tight; a pull there fails with "disk quota
40-
# exceeded". Point both at a roomy data partition instead.
41-
export DATA=/n17data/$USER # adjust to your data partition
42-
export APPTAINER_CACHEDIR=$DATA/.apptainer
38+
The image is published on every push to the `GitHub Container Registry
39+
<https://github.com/CosmoStat/shapepipe/pkgs/container/shapepipe>`_:
40+
``:develop`` tracks the integration branch, release tags (e.g. ``:v1.1.0``) a
41+
stable cut, and the ``-runtime`` suffix selects the slim batch image over the
42+
full interactive one.
4343

44-
# 2. Pull the runtime image (≈850 MB).
45-
apptainer pull "$DATA/shapepipe-runtime.sif" \
46-
docker://ghcr.io/cosmostat/shapepipe:develop-runtime
44+
Documentation
45+
-------------
4746

48-
# 3. Submit the example pipeline (SMP, single node).
49-
SP_IMAGE="$DATA/shapepipe-runtime.sif" SPDIR="$PWD" \
50-
sbatch example/pbs/candide_smp.sh
47+
Full documentation lives at https://cosmostat.github.io/shapepipe. Good places
48+
to start:
5149

52-
A clean run logs ``A total of 0 errors were recorded`` and exits ``0``. To span
53-
multiple nodes with hybrid MPI, swap in ``example/pbs/candide_mpi.sh`` (same two
54-
variables) — see the comments in each script for the SLURM directives.
50+
- `Installation <https://cosmostat.github.io/shapepipe/installation.html>`_ — getting ShapePipe onto your machine or cluster.
51+
- `Basic execution <https://cosmostat.github.io/shapepipe/basic_execution.html>`_ and `configuration <https://cosmostat.github.io/shapepipe/configuration.html>`_ — running ``shapepipe_run`` and writing pipeline configs.
52+
- `Container workflow <https://cosmostat.github.io/shapepipe/container.html>`_ — the two image targets, the ``pyproject.toml`` / ``uv.lock`` / ``Dockerfile`` layers, and how to run on a SLURM cluster (with a worked candide example).
5553

56-
The ``:develop-runtime`` tag tracks the integration branch; for a stable cut use
57-
a release tag (e.g. ``:v1.1.0-runtime``). The interactive ``dev`` image (no
58-
``-runtime`` suffix) carries ``vim``, ``pytest``, and the full toolchain for
59-
working *inside* the container; ``docs/source/container.md`` covers both.
54+
If you use ShapePipe in academic work, please cite Guinot et al. (2022) and
55+
Farrens et al. (2022).

docs/source/container.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,44 @@ in, all because something tries to write under `/app` or `$HOME`:
9191
If you bypass `/tmp` (e.g. with a custom apptainer profile) you may need
9292
to override these manually.
9393

94+
## Running on a cluster (SLURM)
95+
96+
On a batch cluster you pull the slim `runtime` image once to a SIF, then
97+
submit a job that runs `shapepipe_run` through it. The repo ships ready
98+
SLURM scripts for the **candide** cluster in `example/pbs/`
99+
`candide_smp.sh` (single node) and `candide_mpi.sh` (multi-node hybrid
100+
MPI) — that you can copy and adapt. The example below runs the bundled
101+
single-tile pipeline end to end:
102+
103+
```bash
104+
# 1. Keep the SIF and Apptainer's scratch off the quota-limited $HOME.
105+
# On candide a pull under $HOME fails with "disk quota exceeded";
106+
# point both at a roomy data partition instead.
107+
export DATA=/n17data/$USER # adjust to your data partition
108+
export APPTAINER_CACHEDIR=$DATA/.apptainer
109+
110+
# 2. Pull the runtime image (~850 MB).
111+
apptainer pull "$DATA/shapepipe-runtime.sif" \
112+
docker://ghcr.io/cosmostat/shapepipe:develop-runtime
113+
114+
# 3. Submit the example pipeline. SPDIR is your local clone; it is
115+
# bind-mounted at the same path inside the container so the config's
116+
# $SPDIR-relative input/output directories resolve identically in and
117+
# out of the container.
118+
SP_IMAGE="$DATA/shapepipe-runtime.sif" SPDIR="/path/to/shapepipe" \
119+
sbatch example/pbs/candide_smp.sh
120+
```
121+
122+
Both job scripts read `SP_IMAGE` (the SIF) and `SPDIR` (the clone) from
123+
the environment, so the same script serves the example and a real run —
124+
point the config inside the script at your own pipeline. The MPI script
125+
additionally needs the host's OpenMPI to match the container's PMIx wire
126+
protocol; it `module load`s a compatible OpenMPI (the image ships the
127+
5.0.x series), and the script's header comments explain the contract.
128+
129+
Adapting to another SLURM cluster is mostly the `#SBATCH` directives and
130+
the `module load` line — the `apptainer exec` invocation carries over.
131+
94132
## Three configuration layers
95133

96134
Three files determine what the image contains. Each has a clear role; the

0 commit comments

Comments
 (0)