AnaCal ingestion by empEvil · Pull Request #446 · LSSTDESC/TXPipe

empEvil · 2026-02-27T17:07:08Z

This is a pull request for ingesting the DP2 version of AnaCal. currently it is ingesting a DP1 version of the catalog

mr-superonion

Thanks for the preparing the PR, here is my comment

mr-superonion · 2026-07-03T02:58:20Z

+            The indicies of the objects selected from this chunk of data
+        """
+        data = _DataWrapper(data, "")
+        select = self.selector(data, *args, **kwargs)


i think the select also need a shear response calculation like what is done in self.get_submask()

Not quite sure what you mean

@mr-superonion The select function is passed to the code that does the shear calibration. The DataWrapper objects pass in different subsets of data - if the select function asks for data["s2n"], for example, then DataWrapper will give it different columns depending on the second argument used when creating it.

thanks for the explanation

i am implementing it here (e112dea#r3592442390) to create s2n_1p, s2n_1m, s2n_2p, s2n_2m

then the data wrapper is used here (e112dea#r3592459804) to create different shifted version

joezuntz · 2026-07-14T12:20:09Z

+config: examples/Anacal_test/config.yml
+
+inputs:
+    anacal_catalog: /pscratch/sd/x/xiangchl/data/DP1/catalogs/anacal_catalog_a360.fits


Should the example do the butler version?

joezuntz · 2026-07-14T12:22:30Z

+        except Exception as e:
+            raise RuntimeError(error_msg) from e
+
+        if self.config["select_field"]:


Will we have anacal for DP2? If so we might code that applies there?

Not sure what you mean? The plan is to generate an Anacal catalog for DP2

I just meant that all the fields are DP1 fields so it might not be that useful here.

ahh okay, I also added the option of just providing tracts

2. correct for the typo in calibration_calibrators.py, det_respponse_e1 3. in conversion.py f was not defined, changed to flux

mr-superonion · 2026-07-15T05:55:43Z

It seems that i do not have permission to modify the pr here; therefore i opened a pr to correct for some bugs/typo in this pr:
#482

Two equivalent bookkeeping conventions give an unbiased shear: (A) <e> = Σ(wsel·e) / N R = Σ(de/dg·wsel + dw/dg·e) / N → R_sel normalized by count. (B) <e> = Σ(wsel·e) / N / <wsel> R = Σ(de/dg·wsel + dw/dg·e) / N / <wsel> → R_sel normalized by sum(wsel). The rest of the code — R_shape, R_weight, and every mean-e computation — uses convention (B), so R_sel must match with a sum_weights denominator, not count. Also: nanojansky_err_to_mag_ab returns NaN for flux <= 0 via np.divide(where=flux>0) instead of producing negative sigma_mag and RuntimeWarnings.

anacal ingest bug correction

joezuntz · 2026-07-15T11:17:26Z

@mr-superonion I've added you to the team that gives you permissions for future things.

as a second stage after ingest, with a matching config.yml block: delta_gamma: 0.01, mask_threshold: 40, s2n_cut: 5.0, T_cut: 0.1, 5-bin source_zbin_edges, input_pz: true. 2. Ingested photo-z point estimates. TXIngestAnacal now reads zmode_{0,1p,1m,2p,2m} from the merged catalog and exposes them on the shear catalog as mean_z, mean_z_{1p,1m,2p,2m} — baseline for tomographic binning, the four variants for the R_sel bin-migration term. 3. Built ±γ variants of every quantity the selector cuts on. In TXSourceSelectorAnaCal.apply_simple_redshift_cut, injected s2n_{1p,1m,2p,2m} = s2n ± δγ · ds2n_dg{1,2}, and analogous variants of m00, m20 (from dm00_dg, dm20_dg) and zbin (from mean_z_*). All four cut quantities now respond to shear via a single uniform injection step. 4. Unified the calculator's cut logic under _DataWrapper. Deleted the ad-hoc get_submask size-cut machinery in AnaCalCalculator.add_data. The selector now applies mask + s2n + size + zbin in one place; the calculator just runs the same selector on five wrappers (baseline + _1p/_1m/_2p/_2m), and the wrapper's suffix-lookup routes the shifted columns automatically. AnaCalCalculator.__init__ dropped its mask_threshold argument as a consequence. 5. Consistent response formulae. Rewrote R_shape, R_weight, R_sel_{1,2} in the calculator so all three share the same code shape: Σ(w · X) / sum_weights for R_shape and R_weight, and [Σ(w·e)|sel_+ − Σ(w·e)|sel_−] / (2 δγ · sum_weights) for R_sel. R_shape switched from a ParallelMean to a plain scalar sum for parity; det_response_* renamed to weight_response_* to match the "R_weight" label; a small _reduce helper replaced 30 lines of MPI reduce boilerplate. 6. Promoted the size threshold to config. The AnaCal size cut, previously hard-coded as (m00 + m20) / m00 > 0.1, is now > T_cut — driven by the base selector's T_cut config option that was previously unused for AnaCal. TXSourceSelectorAnaCal.config_options gives it a default of 0.1 alongside a new s2n_cut default of 5.0, so a minimal yaml block still runs; both can be overridden in the yaml for tighter/looser cuts.

mr-superonion

I made a few update to include the selection bias correction for s2n and add selection stage into the example. I will check over the dp1 data to see whether i can get consistent result to the hsc code.

mr-superonion · 2026-07-16T03:30:40Z

+        pz_data["s2n_1m"] = s2n - dg * shear_data["ds2n_dg1"]
+        pz_data["s2n_2p"] = s2n + dg * shear_data["ds2n_dg2"]
+        pz_data["s2n_2m"] = s2n - dg * shear_data["ds2n_dg2"]
+


here is the code to create shifted columns for s2n.

mr-superonion · 2026-07-16T03:32:40Z

-            shear_cols += ["mean_z"]
+            # Baseline photo-z + the four shifted variants used to derive
+            # zbin_{1p,1m,2p,2m} for the selection-response calculation.
+            shear_cols += ["mean_z", "mean_z_1p", "mean_z_1m", "mean_z_2p", "mean_z_2m"]


Here we need columns for the shifted redshifts to get the selection bias due to the binning.

mr-superonion · 2026-07-16T03:34:59Z

+        data_1p = _DataWrapper(data, "_1p")
+        data_1m = _DataWrapper(data, "_1m")
+        data_2p = _DataWrapper(data, "_2p")
+        data_2m = _DataWrapper(data, "_2m")


data wrapper is used here to get different shifted version of the catalog

mr-superonion · 2026-07-16T03:35:34Z

+            The indicies of the objects selected from this chunk of data
+        """
+        data = _DataWrapper(data, "")
+        select = self.selector(data, *args, **kwargs)


thanks for the explanation

i am implementing it here (e112dea#r3592442390) to create s2n_1p, s2n_1m, s2n_2p, s2n_2m

then the data wrapper is used here (e112dea#r3592459804) to create different shifted version

mr-superonion · 2026-07-16T03:36:13Z

+    # (zmode_{1p,1m,2p,2m}) in xlens' photoZPipe (currently ±0.01 in
+    # DISTORTIONS).
+    delta_gamma: 0.01
+


i add the selection stage in the example.

But we don't want to run them at the same time. Sadly from how things are set up on NERSC, we want to run the ingestion stage in one environment and then the rest of the pipeline should be run in a dedicated environment for TXPipe. But we want to be able to run the Selection stage later, where we might be able to use different photo-z's for example

empEvil · 2026-07-16T13:10:57Z

@mr-superonion Thank you for the additions, I however do not think we should move away from parallel mean if we can avoid it! I am a little confused about what has been moved from the calibration stage to the source-selection, and why?

empEvil added 30 commits November 4, 2025 13:40

inital commit of anacal ingester

cac4c6c

fixed, first attempt

52093b0

Merge branch 'master' into AnaCal_ingestion

0542c09

Merge branch 'master' into AnaCal_ingestion

cb0bead

Merge branch 'master' into AnaCal_ingestion

5f985ad

fixing the ingestion part

abba469

updating test configs

fd42ff6

more input edits

1c99217

typo fix

8126a27

change naming

96914db

WIP on anacal calibrator

09bf4e3

WIP simplifying the calibration

21103dd

added calibration calculator

5881da1

WIP

7966f16

initial version of source selector

6fb51b9

adding missing stage to init

5c444a1

more import

3cc7017

Merge branch 'master' into AnaCal_ingestion

54b1846

added butler version to Anacal WIP

a7b56a6

fixing butler calls

fcc9c07

fixin input columns

75f3e4a

fixing ingestion

71f004f

removing missing detail

dcb80cc

fixing prefix

901b0c2

typo

b39c0d2

fixing inputs

cb0d002

fixing flux reading

c6f2a3b

typo

ba9fe19

another typo

cbce432

fixing sourceselection

276d805

adding tests for anacal's calibrator

8a900fe

mr-superonion reviewed Jul 2, 2026

View reviewed changes

Comment thread txpipe/ingest/anacal.py Outdated

mr-superonion reviewed Jul 6, 2026

View reviewed changes

mr-superonion reviewed Jul 9, 2026

View reviewed changes

Comment thread txpipe/ingest/anacal.py

empEvil added 3 commits July 10, 2026 13:53

small upates

1ec3129

adding s2n derivatives

20f554e

fixing response flux to magnitude

baaadff

joezuntz requested changes Jul 14, 2026

View reviewed changes

empEvil and others added 13 commits July 14, 2026 16:43

fixing mask_threshold

cd1a7fe

fixing folder

d80dd63

fixing catalog name

5a66b75

minor fixes

69f08ef

fixing doctstring

a0275a9

fixing error message

89b6543

adding select_tracts option

3a72a90

trying to fix the calibration of the detection response

7346ae7

removing unused things

5131798

fixing threshold flagging

0e5890a

fixing missing input in test

32d4979

correcting test

8d3a0be

1. unify shear_catalog_type to "anacal". There was 'Anacal' and 'anacal'

e98a09a

2. correct for the typo in calibration_calibrators.py, det_respponse_e1 3. in conversion.py f was not defined, changed to flux

mr-superonion and others added 3 commits July 14, 2026 23:41

update the examples/anacal so that they are consistent

04dc546

Merge pull request #482 from mr-superonion/AnaCal_ingestion

3e7ad1a

anacal ingest bug correction

mr-superonion reviewed Jul 16, 2026

View reviewed changes

Moving the PZ_SUFFIXES for clarity

6cf485e

Uh oh!

Conversation

empEvil commented Feb 27, 2026

Uh oh!

Uh oh!

mr-superonion left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mr-superonion commented Jul 15, 2026

Uh oh!

joezuntz commented Jul 15, 2026

Uh oh!

mr-superonion left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

empEvil commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants