Skip to content

Commit 693a7be

Browse files
cailmdaleyclaude
andcommitted
felt: record docs work — versioning, cluster tree, rescued v2.0 plan
Three fibers from this session's docs work: - docs-versioning: the versioned-site + switcher design (#738) and the recurring unexercised-path bit-rot pattern. - docs-cluster-tree: the machine-specific clusters.md decision (#739) and why a single page beat a thin standalone general page. - v2-run-plan: the v2.0 run wishlist rescued from the deleted work_flow_v2.0.md docs page before removal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent c893dfd commit 693a7be

3 files changed

Lines changed: 92 additions & 0 deletions

File tree

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
name: Machine-specific cluster docs tree + freshness pass
3+
tags:
4+
- shapepipe
5+
- docs
6+
created-at: 2026-05-31T22:15:20.539028984+02:00
7+
outcome: 'PR #739 (off develop): introduced a single clusters.md (general pattern + candide/CANFAR/ccin2p3 sections) under a ''Running on a cluster'' toctree caption — chosen over a standalone general page (too thin: shared content is ~3 paragraphs, submission models diverge) and over per-machine pages (canfar walkthrough would unbalance). Deleted obsolete canfar.md/pipeline_v2.0.md/work_flow_v2.0.md (planning rescued to [[shapepipe/v2-run-plan]]); kept pipeline_canfar.md as the linked deep CANFAR walkthrough. Rewrote dependencies.md against pyproject (SSOT framing, ngmix fork, dropped CDSclient); stripped removed rho-stats + dead prepare_tiles_for_final from post_processing/random_cat; cosmetics. Audit found install/container/testing/API pages already fresh. Local sphinx build green. Sequencing: candide section assumes #737''s SLURM scripts; publishability needs [[shapepipe/docs-versioning]] (#738).'
8+
---
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
name: Versioned docs site with a version switcher
3+
tags:
4+
- shapepipe
5+
- docs
6+
- ci
7+
created-at: 2026-05-31T20:39:06.459699229+02:00
8+
outcome: 'PR #738 (off develop): docs site now versioned — stable@root (non-destructive to existing URLs), develop@/latest/, tags@/vX.Y.Z/, switcher.json drives the dropdown. Fixes that the site published only from master (~78 commits behind) so container.md/canfar.md were never published. CI green incl. real dev-image build + PR-preview artifact.'
9+
---
10+
11+
## Why
12+
13+
`cd.yml` published the Sphinx site only on push to `master`, which sits ~78
14+
commits behind `develop`. So the public docs described a stale release while
15+
everyone runs `develop`, and six pages written on `develop``container.md`,
16+
`canfar.md`, four others — had **never been published** (`container.html`
17+
404). Surfaced when checking where the README's container link would land.
18+
19+
## Design
20+
21+
Versioned, not just "publish from develop." gh-pages layout is **additive**:
22+
`master`→ site **root** (stable; keeps every existing `/installation.html`-style
23+
URL working), `develop``/latest/`, tags `v*``/<tag>/`. A root `switcher.json`
24+
(tracked in `docs/`) drives the pydata/`sphinx-book-theme` dropdown;
25+
`DOCS_VERSION` (set by the workflow per ref) tells `conf.py` which entry to
26+
highlight. Each deploy uses `keep_files: true`, so it only writes its own slice
27+
and never clobbers siblings — every step is non-destructive, including the first
28+
`develop` run (adds `/latest/` + `switcher.json`, leaves root untouched).
29+
30+
**Per-ref builds, not `sphinx-multiversion`.** autodoc imports `shapepipe`, so an
31+
old tag must build against *its own* deps. Multiversion uses one shared env
32+
across all refs → breaks autodoc on old tags. Building each ref in its own dev
33+
image at its own time sidesteps that entirely. PRs touching docs build + upload
34+
the HTML as an artifact but don't deploy → broken builds caught pre-merge,
35+
reviewers get a preview. `conf.py` version now comes from package metadata
36+
(killed the hand-maintained `1.0.1`, actual is `1.1.0`).
37+
38+
## The recurring bit-rot pattern
39+
40+
Same shape as [[shapepipe/mpi-hybrid]]: a path the repo's CI never exercises
41+
silently rots, and *enabling* it surfaces the breakage. Here, the docs build
42+
only ever ran on `master` (old flat layout), so `sphinx-apidoc … shapepipe`
43+
went stale when the package moved to src-layout (`src/shapepipe`); the first
44+
`develop` build failed `/app/shapepipe is not a directory`. Fixed to
45+
`src/shapepipe`. The lesson keeps repeating — unexercised paths (MPI, docs
46+
deploy) drift undetected; the fix is to *run them in CI*, which this PR now does
47+
for docs on every push and every docs-touching PR.
48+
49+
## Follow-ups (not in PR)
50+
51+
Seed `switcher.json` with the first real release entry when the next version is
52+
tagged; one-time `master` release to refresh root with current docs. Policy
53+
nod: the site now publishes `develop` as `latest`, so in-development docs are
54+
publicly visible — right for an active pipeline, but a conscious shift.
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
name: v2.0 ShapePipe run — workflow wishlist
3+
tags:
4+
- shapepipe
5+
- planning
6+
created-at: 2026-05-31T22:09:28.586302759+02:00
7+
outcome: Forward-looking design wishlist for a future 'v2.0' production run, rescued from the obsolete docs/source/work_flow_v2.0.md before deletion (it was an unrealized plan sitting in published docs). Not yet implemented.
8+
---
9+
10+
## The wishlist (from work_flow_v2.0.md, 2020-era)
11+
12+
A future "v2.0" run was sketched with these goals — none yet implemented; kept
13+
here so the intent isn't lost when the stale docs page is removed:
14+
15+
- **Run per tile** — pipeline runs a single tile end-to-end, or to any
16+
user-specified stopping point.
17+
- **Central exposure store** — used exposures stored once in a central place and
18+
linked from each tile, rather than the current mix of per-patch central storage
19+
and per-tile copies.
20+
- **Drop the patch organisation** — runs no longer organised into patches.
21+
- **Use `/scratch`** for processing speed.
22+
- **Smarter file layout**~20000 tiles and a few ×10⁴ single-HDU exposure files
23+
need subdir bucketing, e.g. `tiles/123/123.456`, `exp/12/…`.
24+
- **Metadata over symlinks** — replace the symlink-based "used exposures per tile"
25+
with a YAML (or DB) manifest.
26+
- **Update ngmix** (can come later).
27+
- **External detection catalogue** — replace SExtractor detection on tiles with an
28+
external ASCII catalogue; still run SExtractor on exposures to identify stars.
29+
- **Remove spread-model (SM)** — note: `spread_model` still exists in the code as
30+
of 2026-05, so this remains unrealized.

0 commit comments

Comments
 (0)