|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## What is ShapePipe |
| 6 | + |
| 7 | +ShapePipe is a galaxy shape measurement pipeline for weak gravitational lensing, developed at CosmoStat (CEA Paris-Saclay). It orchestrates astronomical processing modules (SExtractor, PSFEx, MCCD, ngmix, etc.) into configurable pipeline runs. Source: `src/shapepipe/`. |
| 8 | + |
| 9 | +## Build & Install |
| 10 | + |
| 11 | +```bash |
| 12 | +# Development install (editable, with all extras) |
| 13 | +pip install -e ".[dev]" |
| 14 | + |
| 15 | +# Full install with external dependencies (Conda + external tools) |
| 16 | +./install_shapepipe --develop |
| 17 | + |
| 18 | +# Container (recommended for full pipeline) |
| 19 | +# ghcr.io/cosmostat/shapepipe — see Dockerfile |
| 20 | +``` |
| 21 | + |
| 22 | +Requires Python >= 3.11. Core deps: joblib, modopt, numpy. Many modules need external executables (source-extractor, psfex, etc.) installed separately. |
| 23 | + |
| 24 | +## Running |
| 25 | + |
| 26 | +```bash |
| 27 | +# Run pipeline with config file |
| 28 | +shapepipe_run -c <config.ini> |
| 29 | + |
| 30 | +# MPI mode |
| 31 | +mpiexec -n <cores> shapepipe_run -c <config.ini> |
| 32 | +``` |
| 33 | + |
| 34 | +Config files are INI format with sections: `[DEFAULT]`, `[EXECUTION]`, `[FILE]`, `[JOB]`, `[WORKER]`, plus per-module sections like `[SEXTRACTOR_RUNNER]`. See `example/config.ini` for annotated reference, `example/cfis/` for production configs. |
| 35 | + |
| 36 | +## Testing |
| 37 | + |
| 38 | +```bash |
| 39 | +# Run all tests (pytest configured in pyproject.toml) |
| 40 | +pytest |
| 41 | + |
| 42 | +# Run a single test file |
| 43 | +pytest src/shapepipe/tests/test_pipeline.py |
| 44 | + |
| 45 | +# Run a specific test |
| 46 | +pytest src/shapepipe/tests/test_pipeline.py::ExecuteTestCase::test_execute |
| 47 | +``` |
| 48 | + |
| 49 | +pytest is configured with `--verbose --cov=shapepipe`, testpaths = `["shapepipe"]`. Tests use `unittest.TestCase` with `numpy.testing`. |
| 50 | + |
| 51 | +## Linting |
| 52 | + |
| 53 | +```bash |
| 54 | +black src/shapepipe/ |
| 55 | +isort src/shapepipe/ |
| 56 | +``` |
| 57 | + |
| 58 | +## Architecture |
| 59 | + |
| 60 | +### Pipeline engine (`pipeline/`) |
| 61 | + |
| 62 | +`ShapePipe` class in `run.py` is the main orchestrator: |
| 63 | +1. `args.py` parses CLI args → `config.py` (`CustomParser`) reads INI config |
| 64 | +2. `file_handler.py` resolves input/output paths, discovers files by pattern/numbering |
| 65 | +3. `dependency_handler.py` checks Python + executable dependencies |
| 66 | +4. `job_handler.py` dispatches work via joblib (SMP) or mpi4py (MPI) |
| 67 | +5. `worker_handler.py` + `execute.py` run each module, with `timeout.py` support |
| 68 | + |
| 69 | +`file_handler.py` and `file_io.py` are the largest/most complex files — they manage the numbering schemes, inter-module file routing (`last:MODULE`, `all:MODULE` patterns), and output directory structure. |
| 70 | + |
| 71 | +### Module system (`modules/`) |
| 72 | + |
| 73 | +Each processing step is a module with: |
| 74 | +- `<name>_runner.py` — decorated with `@module_runner` from `module_decorator.py` |
| 75 | +- `<name>_package/` — implementation classes/functions |
| 76 | + |
| 77 | +The `@module_runner` decorator declares: `version`, `input_module`, `file_pattern`, `file_ext`, `depends`, `executes`, `numbering_scheme`, `run_method` (parallel/serial). |
| 78 | + |
| 79 | +Runner signature: `def runner(input_file_list, run_dirs, file_number_string, config, module_config_sec, w_log)` → returns `(stdout, stderr)`. |
| 80 | + |
| 81 | +When a module appears multiple times in `[EXECUTION] MODULE`, its config section gets a `_RUN_N` suffix (e.g., `PYTHON_EXAMPLE_RUNNER_RUN_1`, `PYTHON_EXAMPLE_RUNNER_RUN_2`). |
| 82 | + |
| 83 | +### Utilities (`utilities/`) |
| 84 | + |
| 85 | +- `cfis.py` — CFIS survey-specific helpers (largest utility, ~35KB) |
| 86 | +- `summary.py` — pipeline run summary generation |
| 87 | +- `file_system.py`, `galaxy.py` — general helpers |
| 88 | + |
| 89 | +### Scripts (`scripts/`) |
| 90 | + |
| 91 | +Python and shell scripts symlinked into the environment during install. Used for catalog creation, merging, and CANFAR cluster operations. |
| 92 | + |
| 93 | +## Style conventions |
| 94 | + |
| 95 | +- Single quotes preferred over double quotes |
| 96 | +- Explicit floats: `1.0` not `1.` |
| 97 | +- f-strings for formatting |
| 98 | +- Line length: 79 characters max; break with `()` not `\` |
| 99 | +- Numpydoc-style docstrings |
| 100 | +- String concatenation in f-strings: use `+` at start of continuation line |
0 commit comments