Skip to content

Commit e6988f5

Browse files
committed
Merge remote-tracking branch 'origin/develop' into probe-ngmix
# Conflicts: # uv.lock
2 parents f912d08 + 5bfc5a4 commit e6988f5

52 files changed

Lines changed: 1842 additions & 2881 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.felt/dependabot-pr-triage/dependabot-pr-triage.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
---
2+
id: 01KTCHWZSHQXRQHTHH9Q7Q4MT7
23
name: Triage open dependabot uv.lock PRs
34
status: closed
45
tags:
@@ -84,3 +85,7 @@ Update this fiber's `outcome:` to reflect what landed (e.g. "5 merged, 1 deferre
8485
## Skills
8586

8687
Activate `felt` (you'll touch the fiber). The `shuttle` skill is already loaded by virtue of being the dispatched worker.
88+
89+
## Follow-on: PR #742 (2026-06-11)
90+
91+
PR #742 ("chore(deps): bump the lockfile-minor-patch group, 4 updates": mpi4py 4.1.1→4.1.2, numpy 2.4.4→2.4.6, snakemake 9.20.0→9.21.0, black) arrived after the original batch. CI green (`build-test-publish` passes). Not merged — pyproject.toml modified alongside uv.lock: lower bounds tightened (`mpi4py>=4.0→4.1.2`, `numpy>=2.0→2.4.6`, `snakemake>=9.20.0→9.21.0`). Tightening `>=` floors is a policy call, not a mechanical bump. Comment posted at https://github.com/CosmoStat/shapepipe/pull/742#issuecomment-4675762821 asking maintainers to confirm intent before merging.

.felt/docker-multistage/docker-multistage.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
---
2+
id: 01KTCHWZV7263MYPNB8DT1FXPX
23
name: 'Docker multi-stage: runtime + dev targets'
34
tags:
45
- shapepipe

.felt/docker-uv-revert/docker-uv-revert.md

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,18 @@
11
---
2+
id: 01KTCHWZW8DT8VRYQJ4CEPDRGB
23
name: 'Docker: revert skaha→python base, switch to uv lockfile'
3-
status: active
4+
status: closed
45
tags:
56
- shapepipe
67
- docker
78
- infra
89
created-at: 2026-04-27T11:26:45.677512058+02:00
9-
outcome: 'PR #719 (chore: switch Dockerfile to slim Python + uv lockfile) opened and CI-green on first try (3m31s); ready for review. Drops conda double-install, makes pyproject SSOT + uv.lock the pinned manifest, switches WeightWatcher from sed-patched source build to Debian''s pre-patched 1.12+dfsg-3 package, adds binary smoke tests to deploy-image.yml.'
10+
closed-at: 2026-06-10T17:14:45.965931602+02:00
11+
outcome: 'Superseded: conda removal landed via #733 (merged); the #719-era docker-uv work is absorbed into ci-green-on-develop, which tracks remaining follow-ups.'
1012
decisions:
1113
base:
1214
label: Base image
13-
rationale: Conda double-install was the actual problem; cleanest resolution is to drop conda entirely. The canfar deployment concern is satisfied as long as the slim image works on canfar.
15+
rationale: Conda double-install was the actual problem; cleanest resolution is to drop conda entirely. Martin's canfar concern is satisfied as long as the slim image works on canfar.
1416
default: python-slim
1517
options:
1618
python-slim:
@@ -50,15 +52,15 @@ decisions:
5052
label: uv + pyproject + uv.lock; uv sync --frozen in Dockerfile
5153
modernize:
5254
label: Modernize package versions
53-
rationale: 'We determined which versions MUST stay pinned: only ngmix (pinned to a stable_version fork branch — replacement is tracked separately). Everything else can move to current latest because uv resolved cleanly and CI smoke test still passes (3m42s). If a real pipeline run on canfar surfaces a numpy-2 / pandas-3 break, the fix is a targeted constraint + uv lock, not a wholesale revert.'
55+
rationale: 'We determined which versions MUST stay pinned: only ngmix (Axel''s stable_version branch — replacement is tracked separately). Everything else can move to current latest because uv resolved cleanly and CI smoke test still passes (3m42s). If a real pipeline run on canfar surfaces a numpy-2 / pandas-3 break, the fix is a targeted constraint + uv lock, not a wholesale revert.'
5456
default: stay-current
5557
options:
5658
stay-conservative:
5759
label: Keep pre-v2 minimums (numpy 1.26, astropy 6.1, pandas 2.2); only bump when forced
5860
excluded: true
5961
excluded_reason: Drift between pyproject signal and lockfile reality; loses the chance to surface numpy-2/pandas-3 incompatibilities at PR time when CI is fast
6062
stay-current:
61-
label: Bump pyproject minimums to current major versions (numpy 2, astropy 7, pandas 3, galsim 2.8, mpi4py 4.1, etc.); pin ngmix to its stable_version fork branch
63+
label: Bump pyproject minimums to current major versions (numpy 2, astropy 7, pandas 3, galsim 2.8, mpi4py 4.1, etc.); pin ngmix to Axel's stable_version branch
6264
insights:
6365
ci-fast:
6466
claim: 'First CI run on PR #719 went green in 3m31s. uv installed 238 packages in 322ms — everything resolved to prebuilt wheels, no source compilation of galsim/mpi4py/python-pysap/etc. Massive speedup vs. previous build.'
@@ -97,10 +99,11 @@ The `--frozen` flag is the discipline mechanism: a stale lockfile cannot ship.
9799
## Followups
98100

99101
- Watch CI on #719. The slim-base apt list is conjectural — galsim/mpi4py/python-pysap pull a lot of system deps and we may need to add more (`libatlas-base-dev`, `libblas-dev`, etc).
100-
- If CI needs anything beyond what's in the apt block, that's worth noting for next time.
101-
- After this lands, PRs #708 and #714 may need a small rebase.
102-
- Optional: separate `Dockerfile.canfar` building on skaha if there's a concrete deployment reason. Currently conjectural — floated as a possibility, but slim should work on canfar.
102+
- If CI needs anything beyond what's in the apt block, that's the surface that benefits from a [[shapepipe/prs-in-flight]] note for next time.
103+
- After this lands, [[shapepipe/prs-in-flight]] PRs #708 and #714 may need a small rebase.
104+
- Optional: separate `Dockerfile.canfar` building on skaha if there's a concrete deployment reason. Currently conjectural — Martin floated it but we agreed slim should work on canfar.
103105

104106
## Connections
105107

106108
- [[shapepipe]] — root
109+
- [[shapepipe/prs-in-flight]] — touches the testing-scaffold xfail set and the develop-bugs PR
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
id: 01KTCHWZWEXTJ337APFSH4NF6X
3+
name: Fabian's coord-propagation bug + image-sim code on github
4+
tags:
5+
- shapepipe
6+
- bug
7+
- collaboration
8+
- future
9+
created-at: 2026-04-27T11:26:52.878118978+02:00
10+
outcome: 'Fabian: 1-line fix in shapepipe needs porting; first need him to put image-sim code/configs on github so it''s testable. Beg if necessary.'
11+
---

.felt/ngmix-update/ngmix-update.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
---
2-
name: ngmix library upgrade + wrapper sync
2+
id: 01KTCHWZWY48FVCEDX6DGE6CTY
3+
name: ngmix library upgrade + Lucy wrapper sync
34
tags:
45
- shapepipe
56
- ngmix
67
- future
78
created-at: 2026-04-27T11:26:51.026191639+02:00
8-
outcome: 'Replace the pinned ngmix fork (a stable_version branch carrying not-yet-upstreamed fixes) with upstream ngmix once those land; reconcile the wrapper afterward.'
9+
outcome: 'Future: replace Axel''s stable_version fork with upstream ngmix; reconcile with Lucy''s cleaned-up wrapper from her visit'
910
---
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
id: 01KTCHWZWYYTDSYHAP4FPE7V3V
3+
name: PRs in flight after v2 merge
4+
tags:
5+
- shapepipe
6+
- pr
7+
created-at: 2026-04-27T11:26:49.300097608+02:00
8+
outcome: 'Post-v2 + post-propagation: infra stream now landed (#718 setuptools, #719 uv-lockfile, #728 dependabot+SHA-pin), supply-chain hygiene done (20 → 0 alerts). Issue #712 empirically verified resolved against current `:develop` (all 11 packages in Martin''s May 18 list import in both read-only and writable sandbox modes); comment posted, awaiting Martin reply before closing. Science PRs still open: #714 develop-bugs (closes #709 + #711 only — #712 closes separately), #708 testing-scaffold (mine); #725 centroid shift (Axel), several older Martin PRs (#704 #703 #699 #660 #650 #636), #670 lbaumo file_io. Next thread: merge #714.'
9+
insights:
10+
714-already-redundant:
11+
claim: 'Surprise from rebasing #714: its Dockerfile commit (cf304f8f, adding astroquery/numba/fitsio + setuptools<81 pin) was *already* redundant on current develop — the v2 merge silently put astroquery/numba/fitsio into pyproject and the v2 Dockerfile installs them via ''pip install -e ".[fitsio]"'' at the end. setuptools<81 went away via #718. So ''rebase to drop the obsolete commit'' wasn''t waiting on #719 — it was already obsolete the moment v2 merged. Worth checking sooner next time before assuming a fix is still load-bearing.'
12+
xfail-mostly-fixable:
13+
claim: 'Most #708 xfails are about to be resolved: canfar_monitor IndentationError (4 xfails) and summary_run -h (1 xfail) are fixed in #714; astroquery/numba/fitsio import xfails (5 modules) resolve in #719 because uv sync installs them from pyproject. Only stile/treecorr corr2 (4 modules) is a separate issue requiring stile removal or upstream patch.'
14+
dependabot-policy:
15+
claim: 'shapepipe now ships `.github/dependabot.yml` (#728) with 14-day cooldown, monthly grouped lockfile PRs, github-actions ecosystem opted in, and SHA-pinned actions across all four workflows. Reasoning lives in the file itself + the #728 PR body. Companion fiber [[shapepipe/sqlitedict-pickle-smell]] tracks the single dismissed alert.'
16+
712-empirically-resolved:
17+
claim: 'Issue #712 is empirically resolved against current `ghcr.io/cosmostat/shapepipe:develop` (dev target, post-#728). Both the original packages (astroquery, numba, fitsio) and Martin''s May 18 follow-up list (scipy, joblib, importlib_metadata, tqdm, LSSTDESC.Coord, pyyaml, astropy_iers_data, pyerfa) import cleanly in both read-only and writable sandbox modes, as do the three originally-flagged runner modules. Pyproject confirms astroquery/numba/joblib/tqdm are core deps; the rest are transitives of astropy/mccd/modopt/galsim; fitsio is gated in both runtime (`--extra jupyter --extra fitsio`) and dev (`--extra dev`) targets. Comment posted; awaiting Martin reply before closing. Likely root cause of the May 18 report: cached/older image.'
18+
decisions:
19+
setuptools-pin:
20+
label: drop setuptools<81 pin
21+
default: merged
22+
options:
23+
merged:
24+
label: 'Already merged as #718 (c9e71df8) — small one-liner, agreed in transcript'
25+
---
26+
27+
Snapshot of CosmoStat/shapepipe PR state, maintained as a living index.
28+
29+
## Open — infra
30+
31+
(All infra PRs landed. The dependabot stream is resolved; supply-chain
32+
posture set; SHA-pins in place. See [[shapepipe/sqlitedict-pickle-smell]]
33+
for the one open security-fiber.)
34+
35+
## Open — issues (mine)
36+
37+
| # | What | Status |
38+
|---|---|---|
39+
| #712 | Dockerfile missing runtime deps | Empirically resolved against current `:develop` ([comment](https://github.com/CosmoStat/shapepipe/issues/712#issuecomment-4562085977)). Both original list (astroquery/numba/fitsio) and Martin's May 18 follow-up (scipy/joblib/importlib_metadata/tqdm/LSSTDESC.Coord/pyyaml/astropy_iers_data/pyerfa) import cleanly in read-only + writable sandbox modes. Awaiting Martin reply before closing. |
40+
| #711 | summary_run -h crashes | Fixed by #714 (auto-closes on merge) |
41+
| #709 | canfar_monitor IndentationError | Fixed by #714 (auto-closes on merge) |
42+
43+
## Open — mine (science / fixes)
44+
45+
| # | Branch | What | Status |
46+
|---|---|---|---|
47+
| #731 | `chore/smoke-test-read-only` | smoke-test in read-only mode | Open. Adds `shapepipe_run_example` wrapper; CI now runs the entry-point smoke under `docker --read-only --tmpfs /tmp:rw`. See [[shapepipe/smoke-test-read-only]]. |
48+
| #714 | `fix/develop-bugs` | small develop bugs (#709, #711) | Open. Originally a multi-bug fix; the Dockerfile portion got absorbed into #719. Worth checking what's still load-bearing here vs already-fixed-upstream. |
49+
| #708 | `chore/testing-scaffold` | Tier 0–2 test scaffolding | Open. Some xfails should have flipped to xpass after the v2 + uv-lockfile work; needs a rebase + xfail-list audit. |
50+
51+
## Open — others' PRs awaiting attention
52+
53+
| # | Author | What |
54+
|---|---|---|
55+
| #741 | martinkilbinger / lbaumo | **Ngmix v2.0** — upstream ngmix 2.4.0 + Lucy's new classes. Canonical PR (CI mirror; fork PR #740 closed by Martin). OPEN, mergeable, CI green. Two-part review + next-steps triage delivered; 11 findings open (5 cut-and-dry, 5 decisions, 1 resume), 2 are merge-gates (weight-norm, `*_psfo`). See [[review-ngmix-v2-pr740]]. |
56+
| #725 | aguinot | Fix centroid shift (overlaps #741 centroid work — cross-ref/supersede?) |
57+
| #704 | martinkilbinger | Contributors |
58+
| #703 | martinkilbinger | V1.3.x |
59+
| #699 | martinkilbinger | Coverage mask |
60+
| #670 | lbaumo | file_io handles sextractor header |
61+
| #660 | martinkilbinger | Existing output directory |
62+
| #650 | martinkilbinger | Third-party catalogue for tile objects |
63+
| #636 | martinkilbinger | Rho statistics: flexible training/test split |
64+
65+
## Recently closed
66+
67+
- **#728** `chore/dependabot-config` — dependabot.yml + SHA-pin all actions. Merged 2026-05-28.
68+
- **#727, #726, #724, #722, #721, #720** — dependabot security bumps for idna/urllib3/gitpython/mistune/jupyter-server/jupyterlab. All squash-merged 2026-05-28 (see [[shapepipe/dependabot-pr-triage]]).
69+
- **#719** `chore/uv-lockfile` — merged 2026-05-05 (Martin).
70+
- **#718** `chore/drop-setuptools-pin` — merged.
71+
- **v2.0 PR** — merged. Source of the skaha/conda situation that #719 unwound.
72+
73+
## Connections
74+
75+
- [[shapepipe]] — root
76+
- [[shapepipe/docker-uv-revert]] — drove #719
77+
- [[shapepipe/dependabot-pr-triage]] — drove the 6 security-bump merges (closed)
78+
- [[shapepipe/sqlitedict-pickle-smell]] — future-work fiber for the one dismissed alert

0 commit comments

Comments
 (0)