pipeline_canfar.md

Running ShapePipe processing and post-processing pipelines on CANFAR

Documentation to create ShapePipe output products for catalogues v1.x.

Initial Setup

CANFAR Login

Login to the canfar system with

canfar auth login

This can be done from at notebook or terminal within the canfar science portal, or any remote terminal that has the canfar library installed.

Check authentication status with

canfar auth list

If not on "default", run

canfar auth switch default

Set variables (optional)

Set the current patch in the shell as

patch=P[1-9]

For convenience, the current PSF model can be set as environment variable, e.g.:

psf="psfex"

Allowed are psfex and mccd.

Setting the terminal title to display the patch can be useful for long jobs, to keep track of which terminal runs which patch:

echo -ne "\033]0;$patch\007"

Prepare run directory

First, go to the dedicated directory with

cd /path/to/version/$patch

Next, set links to the tile number list and configuration directory:

ln -s ~/shapepipe/auxdir/CFIS/tiles_202106/tiles_$patch.txt tile_numbers.txt
ln -s ~/shapepipe/example/cfis

Create output and debug log directories

mkdir -p output
mkdir -p debug

Finally, create and link to central image storage directories for tiles and exposures:

mkdir -p ~/cosmostat/v2/data_tiles/$patch
ln -s ~/cosmostat/v2/data_tiles/$patch data_tiles
mkdir -p ~/cosmostat/v2/data_exp/$patch
ln -s ~/cosmostat/v2/data_tiles/$patch data_exp

ShapePipe processing

Now, everything should be ready to start running ShapePipe for the weak lensing processing. The following details all necessary steps.

Get Images

We first download images, and in a second run create symbolic links with the proper pipeine naming scheme.

Download and move tiles

When running the main ShapePipe script shapepipe_run, the following env variable needs to point to the current working directory

export SP_RUN=`pwd`

Now we run the first module (get_images_runner) to download the tile images together with the weight files. This run can get interrupted by VOSpace I/O or connection errors. In that case, we move new files to the image storage directory, remove the previous (now void of images) run directory, and update the run log file. We also check the number of previous and new tiles.

shapepipe_run -c cfis/config_Git_vos.ini
ls -l data_tiles/ | wc
mv -i output/run_sp_Git_*/get_images_runner/output/CFIS.???.???.*fits* data_tiles
ls -l data_tiles/ | wc
rm -rf output/run_sp_Git_*
update_runs_log_file.py

Repeat the above block as needed.

Find Exposures

With all tile images (= stacks) downloaded, we can inquire their headers to identify the exposures that were used to create the stacks. This call to the pipeline also creates the symbolic links to the downloaded tile images.

shapepipe_run -c cfis/config_GitFe_symlink.ini

(One could also run Fe alone.)

Download and Move Exposures

The last module create exposure lists on output. These are now used to download all exposures. As for the tile downloads, we have to account for VOSpace errors.

shapepipe_run -c cfis/config_Gie_vos.ini
mv -i output/run_sp_Gie_*/get_images_runner/output/*.fits*fz data_exp
rm -rf output/run_sp_Gie_*
update_runs_log_file.py

Repeat the above by hand, or peform it in an automatic loop:

while true; do
  shapepipe_run -c cfis/config_Gie_vos.ini
  ls -l data_exp/ | wc
  mv -i output/run_sp_Gie_*/get_images_runner/output/*.fits*fz data_exp
  ls -l data_exp/ | wc
  rm -rf output/run_sp_Gie_*
  update_runs_log_file.py
done

Note: Make sure that after all images are downloaded there is no Gie run in the output directory. This would mess up later modules since last:get_image_runner could point to this run.

Create tile links again (necessary?)

If necessary, e.g. because a previous Git run is no longer valid, re-create the symbolic links to the downloaded tiles with

job_sp_canfar.bash -p $psf `cat tile_numbers.txt` -j 1 -r symlink

Uncompress tile weights

The downloaded tile weights are compressed. The following call uncompresses all.

shapepipe_run -c cfis/config_tile_Uz.ini

Mask tiles

This step is done globally for all tiles. There might be job failures or interruptions. The following command to the ShapePipe job script can be run repeatedly; already created masks will be skipped.

job_sp_canfar.bash -p $psf -n $OMP_NUM_THREADS -j 4

If masks were created in more than one run, i.e. situated in more than one output directory, these have to be combined for subsequent pipeline module runs. This is done by creating a new output directory with symbolic links, using the script

combine_runs.bash -c flag_tile

Tile detection

We can finally run our first module using the canfar submission system. First, determine the number of optimal jobs such that at a given time the allowed maximum of 512 running jobs is not exceeded. Set as N_PAR (number of parallel jobs) a number between 1 and 8.

canfar_submit_job -j 16 -f tile_numbers.txt -P N_PAR -v -s

Now, run the previous command with that number JMAX

canfar_submit_job -j 16 -f tile_numbers.txt -P N_PAR -v -J JMAX

Exposure Processing

Option 0: Global split and exp masks (deprecated; used for earlier v1.x patch runs)

For this option, set sp_local=0.

**TODO: ** Split Uz and SpMh

For sp_local=- both mh_local (0, 1) are ok:

export mh_local=0

Option 0: Mask exposures (deprecated)

Run repeatedly if necessary:

job_sp_canfar.bash -p $psf -n $OMP_NUM_THREADS -j 8

Combine all runs:

combine_runs.bash -c flag_exp

Option 1: Local split and mask exposures (recommended)

Optional: Enable flags for local split processing and merge header runs as

export sp_local=1
export mh_local=1

These flags are automatically set to 1 in the new job scripts.

Get single-HDU single-exposure IDs file (from missing 32 job):

summary_run P$patch 32
cp summary/missing_job_32_all.txt exp_shdu.txt

Split exposures

First, determine the number of maximum jobs with the option -s (see above). Then, submit with

canfar_submit_job -j 2 -v -f exp_shdu.txt -v -P N_PAR -J JMAX

Mask exposures

canfar_submit_job -j 8 -f exp_shdu.txt -v -P N_PAR -J JMAX

Exposure detection

canfar_submit_job -j 32 -f exp_shdu.txt -v -P N_PAR -J JMAX

Tile preparation

canfar_submit_job -j 64 -f tile_numbers.txt

Tile shape measurement

canfar_submit_job -j 128 -f tile_numbers.txt

Merge sub-catalogues

canfar_submit_job -j 256 -f tile_numbers.txt

Create final catalogues

canfar_submit_job -j 512 -f tile_numbers.txt

This was the last ShapePipe module to run for main processing.

Merge all final catalogues

The last step of ShapePipe processing is, per patch, to merget all final catalogues. This is done via a python script, as follows. First, change to parent directory /path/to/version and run the following command for all patches

patchnum=`tr $patch P ''`
create_final_cat.py -m final_cat_$patch.hdf5 -i . -p $patch/cfis/final_cat.param \
    -P $patchnum -o $patch/n_tiles_final.txt -v

Additional ShapePipe processing

Create star Catalogue

We can additionaly create a combined star catalogue, with star shapes projecte from detector to world coordinates. This is useful for validation and galaxy-PSF/star correlation diagnostics.

Combine all PSF runs

In each patch directory /path/to/version/$patch, run

combine_runs.bash -p $psf -c psf

to create a single output directory of PSF files (symbolic links).

Optionally, to create and plot results for this patch only:

shapepipe_run -c $SP_CONFIG/config_Ms_$psf.ini
shapepipe_run -c $SP_CONFIG/config_Pl_$psf.ini

Convert star catalogue to wCS

Convert all input validation PSF files and create directories per patch P?. Create files validation_psf_conv-<patchnum>-<idx>.fits (for the v1.4 setup only one file):

cd /path/to/version
mkdir stat_car
cd star_cat

For each patch run

convert_psf_pix2world.py -i .. -P $patchnum -v

Combine previously created files as links within one ShapePipe run directory (for the v1.4 setup only one link). First (and optiohnal), create a subdir for a run and link to the input patches:

cd /path/to/version/star_cat
mkdir v1.6
ln -s ../P1
ln -s ../P2
...

Next, create links to all validation_conv runs:

combine_runs.bash -p psfex -c psf_conv

Merge all converted star catalogues and create final-starcat.fits:

export SP_RUN=`pwd`
shapepipe_run -c ~/shapepipe/example/cfis/config_Ms_psfex_conv.ini

Rename to general PSF and star catalogue used for all ("a") sub-versions:

cp output/run_sp_Ms/merge_starcat_runner/output/full_starcat-0000000.fits \
    unions_shapepipe_psf_2024_v1.6.a.fits 

The FITS file CATTYPE (newer version) should be validation_psf_conf.

Post-processing

The following post-processing steps are performed with the library sp_validation.

Extract Information

First, we extract all information from the final catalogue, per patch. We copy the parameter file and set links to the catalogues and ShapePipe config directory.

cd /path/to/version/$patch
cp ~/astro/repositories/github/sp_validation/notebooks/params.py .
ln -s /path/to/final_cat_$patchnum.hdf5  # not relative path ../final_cat_P$patchnum.hdf5 !
ln -s output/run_sp_MsPl/mccd_merge_starcat_runner/output/full_starcat-0000000.fits
ln -s ~/astro/repositories/github/shapepipe/example/cfis

Then edit params.py: Set patch name; set wrap_ra for P2.

Now we can run the script, recommended via job submission on candide. For large patches, this requies a job with a large memory, e.g. with mem=380000

[squeue] python ~/astro/repositories/github/sp_validation/notebooks/extract_info.py

This creates a patch-wise comprehensive catalogue.

Create global comprehensive catalogues

cd /patch/to/version
[squeue] python ~/astro/repositories/github/sp_validation/scripts/create_joint_comprehensive_cat.py \
    -v v1.6.c -v -p P1+P2+P3+P4+P5+P6+P7+P8+P9

This creates the file unions_shapepipe_comprehensive_2024_v1.6.c.hdf5.

Apply structural masks

First, edit the Python script ~/astro/repositories/github/sp_validation/notebooks/demo_apply_hsp_masks.py to match catalogue name. Check the coverage mask input file (see below). Run the script to apply the healsparse structural masks:

[squeue] python ~/astro/repositories/github/sp_validation/notebooks/demo_apply_hsp_masks.py

This creates the file unions_shapepipe_comprehensive_struct_2024_v1.6.c.hdf5.

Define sample, calibrate catalogue

We are close to finally perform the last post-processing step, which is the calibration. First, the final galaxy sample in question needs to be defined, with masks and cuts to apply from a yaml config file. A number of pre-defined files can be found in ~/astro/repositories/github/sp_validation/calibration.

For example, to create v1.6.6, the steps are:

cd /path/to/version
mkdir -p v1.6.6
cd v1.6.6
ln -s ~/astro/repositories/github/sp_validation/calibration/mask_v1.X.6.yaml config_mask.yaml
ln -s ..//unions_shapepipe_comprehensive_struct_2024_v1.6.c.hdf5 unions_shapepipe_comprehensive_struct_2024_v1.X.c.hdf5
[squeue] python ~/astro/repositories/github/sp_validation/calibrate_comprehensive_cat.py

calibrate_comprehensive

Create matched star catalogue

For diagnostics, a catalogue with multi-epoch shapes measured by ngmix matched with the validation star catalogue is used. This is created as follows:

cd /path/to/version
merge_psf_cat.py [-V v1.6|-P P1+P2+...] -v

This creates the joint catalogue unions_shapepipe_star_2024_v1.6.a.fits .

Create coverage mask

First, on canfar, move to the directory that has the patch subdirectories.

cd /path/to/version

Get exposure numbers

If the file $patch/exp_numbers.txt does not exist for a given patch, create it with the summary program

summary_run $patch 1

Now, create the list of CCDs that have PSF information with

get_ccds_with_psf -v -V v1.6

Next, download exposures headers; indicate (with -d) a directory of already downloaded headers; those will be linked and duplicated download skipped.

download_headers -i ccds_with_psfs_v1.6.txt -o headers_v1.6 -d headers_v1.3 -v

From the headers, the CCD corner coordinates are extracted with

extract_field_corners -i headers_v1.6 -v

Then, build the healsparse coverage mask file as

build_coverage_map -i exp_ra_dec_v1.6 -o coverage_v1.6.x.hsp -c 128 -n 131072 -v

The healsparse resolutions (128, 131072) match the bit masks.

Use plot_coverage_map to create plots of the coverage mask.

Building and plotting for a range of versions is done with build_and_plot_coverage_maps.sh.

Extra Utilities

Run in Terminal in Parallel

cat IDs.txt | xargs -I {} -P 16 bash -c 'init_run_exclusive_canfar.sh -j 512 -e {}'