Merge pull request #741 from CosmoStat/ngmix_v2.0

Ngmix v2.0 (CI mirror of #740)

Author: cailmdaley

.github/workflows/deploy-image.yml

@@ -123,7 +123,7 @@ jobs:
       - name: Test dev — pytest suite
         run: |
           IMAGE=$(echo "${{ steps.meta-dev.outputs.tags }}" | head -n1)
-          docker run --rm -e HYPOTHESIS_PROFILE=ci "$IMAGE" pytest -rX
+          docker run --rm -e HYPOTHESIS_PROFILE=ci -e SHAPEPIPE_ON_CANDIDE=0 "$IMAGE" pytest -rX
 
       # ----------------------------------------------------------------
       # Publish (push events only — never on pull_request, incl. forks).

CLAUDE.md

@@ -36,6 +36,30 @@ way to get all of that is the container.
   sandbox with a host clone of the repo bind-mounted in and `pip install -e`
   pointed at it, so edits on the host are live inside the container.
 
+**Shadow a library build with `PYTHONPATH` (no image rebuild).** When you need
+the container to run a *different* build of a pure-Python library than the one
+baked into its venv — a feature worktree, an unreleased branch, this repo's own
+`src/` against a stale image — prepend the host checkout to `PYTHONPATH`. Python
+resolves the prepended path first, so the on-disk version shadows
+`/app/.venv/...` without touching the image. This is the local-testing
+counterpart of a git-ref dependency (e.g. `cs_util @ develop` in
+`pyproject.toml`): the dep change makes CI build the right version; the shadow
+lets you test that version *now*, before any rebuild. The recipe:
+
+```bash
+apptainer exec --bind /n17data,/automnt <image.sif> bash -c \
+  "cd <repo-worktree> && \
+   PYTHONPATH=/path/to/libfoo-checkout:<repo-worktree>/src \
+   python -m pytest <targets> -o addopts='' -q"
+```
+
+Notes: the checkout path is the **parent** of the importable package dir (the
+dir containing `foo/`, not `foo/` itself); list several `:`-separated to stack
+shadows; `-o addopts=''` clears `pyproject.toml`'s pytest defaults when a plugin
+they reference (e.g. `pytest-cov`) isn't in the image. Use this for a quick
+verify; land the real fix as the `pyproject.toml` / `uv.lock` dep change so CI
+and the next image agree.
+
 **Testing container changes: build remotely, pull locally.** Don't
 `apptainer build` images on a cluster — quotas are tight and the build is slow.
 The loop for any change to `Dockerfile` / `pyproject.toml` / `uv.lock` is: edit
@@ -56,7 +80,10 @@ Full detail: `docs/source/installation.md` and `docs/source/container.md`.
   I/O; `utilities/`; `canfar/` is CANFAR/cluster job orchestration. Console entry
   points (`shapepipe_run`, `summary_run`, `canfar_*`) are defined under
   `[project.scripts]`.
-- `src/shapepipe/tests/` — unit tests; `tests/unit/` holds newer test scaffolding.
+- `tests/` — the whole test suite, one discovery root: `module/` (per-module
+  unit/property/integration tests), `unit/` (structural), `science/` (fast
+  guardrails), `cluster/` (candide-only), `helpers/` (shared library code).
+  See `tests/README.md`.
 - `example/` — a runnable example pipeline (`example/config.ini`) on a single
   CFIS tile; doubles as the CI smoke test.
 - `scripts/` — shell / Python / notebook helpers (`sh/`, `python/`, `jupyter/`),
@@ -75,8 +102,6 @@ Full detail: `docs/source/installation.md` and `docs/source/container.md`.
   from `master` (`cd.yml`).
 - **Style**: PEP 8; numpydoc docstrings on public modules, classes, and methods.
   Match the surrounding code.
-- **ngmix** is pinned to a fork branch (`aguinot/ngmix@stable_version`) until the
-  fixes land upstream — don't bump that dependency line.
 
 ## Project knowledge — felt
 

Dockerfile.jupyter

@@ -1,7 +1,7 @@
 # Dockerfile for ShapePipe Jupyter Lab
 # Uses the shared base image
 
-FROM ghcr.io/cosmostat/shapepipe:v2.0
+FROM ghcr.io/cosmostat/shapepipe:develop
 
 # Default command starts Jupyter Lab
 ENTRYPOINT ["jupyter","lab","--ip=0.0.0.0","--port=8888","--no-browser","--allow-root"]

bin/canfar_monitor.py

@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+from shapepipe.canfar_run import run_log
+run_log()

bin/canfar_monitor_log.py

@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+from shapepipe.canfar_run import run_monitor_log
+run_monitor_log()

bin/canfar_submit_job.py

@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+from shapepipe.canfar_run import run_job
+run_job()

bin/shapepipe_run.py

@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+from shapepipe.shapepipe_run import main
+main()

bin/summary_run.py

@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+from shapepipe.summary_run import main
+main()

conftest.py

@@ -1,10 +1,106 @@
-"""Shared pytest configuration for the ShapePipe test suite."""
+"""Shared pytest configuration for the ShapePipe test suite.
+
+Markers, environment detection, and the candide skip policy live here so
+every test module — wherever it sits in the tree — sees the same rules.
+
+Markers (also declared in ``pyproject.toml`` so ``--strict-markers`` is on):
+
+* ``slow``    — heavy compute (minutes), not part of the fast inner loop.
+* ``candide`` — needs the candide cluster and/or its real on-disk data;
+  meaningless (and auto-skipped) anywhere else.
+
+A ``candide``-marked test only runs on a candide node. Everywhere else it
+is skipped with a clear reason, so the same suite is green on a laptop, in
+CI, and on the cluster — the cluster-only tests simply do not fire off it.
+"""
 
 import os
+import re
+import socket
+
+import pytest
 
 from hypothesis import settings
 
 
 settings.register_profile("ci", derandomize=True, max_examples=50)
 settings.register_profile("dev", max_examples=200)
 settings.load_profile(os.environ.get("HYPOTHESIS_PROFILE", "ci"))
+
+
+# --------------------------------------------------------------------------- #
+# Candide detection
+# --------------------------------------------------------------------------- #
+
+# Candide compute / login nodes are named c01, c03, n22..n36, etc. The login
+# host this suite is most often driven from is ``c03``. We match the candide
+# node-name families rather than a fixed list so new nodes are covered, and
+# allow an explicit override for CI or odd hostnames.
+_CANDIDE_HOST_RE = re.compile(r"^(c\d|n\d{2})", re.IGNORECASE)
+
+
+def on_candide():
+    """Return True when running on a candide node.
+
+    The check is, in order: an explicit ``SHAPEPIPE_ON_CANDIDE`` override
+    (``1``/``0``), then the hostname against the candide node-name families
+    (``c0x`` login, ``nXX`` compute). Cheap, import-safe, no cluster calls.
+    """
+    override = os.environ.get("SHAPEPIPE_ON_CANDIDE")
+    if override is not None:
+        return override == "1"
+    return bool(_CANDIDE_HOST_RE.match(socket.gethostname()))
+
+
+# --------------------------------------------------------------------------- #
+# Marker registration + skip policy
+# --------------------------------------------------------------------------- #
+
+
+def pytest_configure(config):
+    config.addinivalue_line(
+        "markers",
+        "slow: heavy compute (minutes); excluded from the fast inner loop.",
+    )
+    config.addinivalue_line(
+        "markers",
+        "candide: needs the candide cluster and/or its real data; "
+        "auto-skipped elsewhere.",
+    )
+
+
+def pytest_collection_modifyitems(config, items):
+    """Skip ``candide`` tests off-cluster.
+
+    Collection still happens everywhere — so ``pytest --collect-only`` shows
+    the cluster tests exist — they are just marked skipped at run time when
+    not on candide.
+    """
+    if on_candide():
+        return
+    skip_candide = pytest.mark.skip(
+        reason="needs candide (set SHAPEPIPE_ON_CANDIDE=1 to force)"
+    )
+    for item in items:
+        if "candide" in item.keywords:
+            item.add_marker(skip_candide)
+
+
+# --------------------------------------------------------------------------- #
+# Shared fixtures
+# --------------------------------------------------------------------------- #
+
+
+@pytest.fixture(scope="session")
+def artifacts_dir():
+    """Directory where guardrail tests drop plots + status summaries.
+
+    The artifacts SEAM: a later GitHub Pages / status step publishes from
+    here. Created on demand so a clean checkout has nothing to commit until
+    a test actually emits.
+    """
+    from pathlib import Path
+
+    path = Path(__file__).parent / "tests" / "_artifacts"
+    path.mkdir(parents=True, exist_ok=True)
+    return path

docs/source/configuration.md

@@ -84,7 +84,10 @@ The following options can be added to the `[FILE]` section of the config file
   (*e.g.* `.`, `-`, `:`, *etc.*). *optional*ly a regular expression can also be
   passed if it is preceded by `RE:` (*e.g.* `RE:-\d{9}`).
 - `NUMBER_LIST` : (`str` or `list`, *optional*) A list of number strings
-  matching the numbering scheme or a file name.
+  matching the numbering scheme or a file name. Restricts the run to these
+  numbers; every entry must match an input file found on disk, otherwise the
+  run fails at start-up. This is also how a single image is processed per
+  job (formerly the `-e`/`--exclusive` command-line flag).
 - `CORRECT_FILE_PATTERN` : (`bool`, *optional*) Option to allow substring file
   patterns. Default value is `True`.