DES-Y3 3x2pt

[joezuntz]

This is an attempt at pulling everything together for a full 3x2pt re-run of DES Y3.

It's been run on a sub-deg2 test patch and now I'm looking at pushing it through the full catalog on NERSC.

The main remaining TODO apart from the full run is to try to improve the photo-z treatment. RAIL team members have pointed me towards some runs on Cardinal. It would be nice to try to incorporate this once the current attempt to process the full catalog is done.

This would also be a chance to try and use the RAIL tomography steps.

[Copilot]

Pull request overview

This PR pulls together a runnable end-to-end configuration for a DES Y3 “3x2pt” re-run in TXPipe, including new DES Y3 ingestion stages, updated source-selection/tomography plumbing, and updates to LSS weights + RAIL summarization to support the intended DES Y3 workflow (including nested HEALPix handling and “bin_all” masked summarization).

Changes:

• Add DES Y3 ingestion stages (metacal shear + source n(z) from legacy 2pt file) and a DES Y3-specific metacal source selector that consumes the public index catalog binning.
• Refactor tomography assignment in source selection into a single overridable hook (make_tomographic_bin_chooser_function) and add an HDF hardlink (tomography/class_id) for RAIL masked summarizers.
• Improve LSS weights linear-model fitting (design-matrix precompute + unit test), parallelize the null-test stage, and tighten nested/ring handling across several HEALPix call sites.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
txpipe/utils/pixel_schemes.py	Respect nest when computing HEALPix pixel boundaries.
txpipe/test/test_lss_corr.py	New test verifying design-matrix linear model matches map-based computation.
txpipe/source_selection/metacal.py	Add compatibility for catalogs with only a single flags column; add DES Y3 selector subclass.
txpipe/source_selection/base.py	Centralize tomography assignment via make_tomographic_bin_chooser_function; add class_id hardlink.
txpipe/random_cats.py	Allow selecting a band-specific depth map and respect nested ordering in random generation.
txpipe/rail/summarize.py	Ensure bin_all is included; use RAIL’s TOMOGRAPHY_ALL for masked summarizers.
txpipe/metadata.py	Stop copying R_total* response datasets into metadata output.
txpipe/masks.py	Minor operator map ordering tweak.
txpipe/lssweights.py	Add “no sysmaps” error, parallelize null tests, switch to design-matrix linear model, add consistency check.
txpipe/lsstools/lsstools.py	Add normalization control for correlations; add design-matrix precompute + fast linear model; update map-based linear model function.
txpipe/lsstools/init.py	Export linear_model_from_maps.
txpipe/lens_selector.py	Read mask metadata and use nest in ang2pix.
txpipe/ingest/stage3/desy3.py	Add DES Y3 metacal shear ingestion and source n(z) ingestion stage.
txpipe/ingest/base.py	Write ingested map metadata with nest: True.
txpipe/base_stage.py	Improve robustness of provenance gathering; support parallel opening for directory outputs under MPI.
examples/lssweights/des-y1/pipeline_binned.yml	Update example paths/site/log locations.
examples/lssweights/des-y1/config_binned.yml	Update example sysmap paths; minor comment change.
examples/desy3/pipeline.yml	New DES Y3 pipeline recipe integrating ingestion, RAIL photo-z, selection, 2pt, and mapping stages.
examples/desy3/config.yml	New DES Y3 configuration for the above pipeline (ingestion inputs, mask cuts, DNF, LSS weights, etc.).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[empEvil]

Looks good

[joezuntz]

@empEvil think I've fixed all this - can you approve?