|
| 1 | +{ |
| 2 | + "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "markdown", |
| 5 | + "id": "4227b7d4-a861-4839-b4b7-52a37534a283", |
| 6 | + "metadata": {}, |
| 7 | + "source": [ |
| 8 | + "# Demo for reducing the Euclid flagship simulations:\n", |
| 9 | + "\n", |
| 10 | + "The full simulations are available at https://cosmohub.pic.es/catalogs/353. If you use the catalog for any publication, please use the acknowledgements listed on this website.\n", |
| 11 | + "\n", |
| 12 | + "Notice that the catalog inherently has a cut for the Euclid h band at ${\\tt euclid\\_nisp\\_h} < 26.6$. The full catalog covers 1 octant of the sky but the example file only contain objects on 1 pixel (out of a total of 256), and subsampled to be 1% of the original size. Notice also there is a small region (150 < RA < 155 deg., 5 < DEC < 10 deg.) where there is no magnitude or flux cut, which has significantly higher number density. This small region *is* part of the example file." |
| 13 | + ] |
| 14 | + }, |
| 15 | + { |
| 16 | + "cell_type": "code", |
| 17 | + "execution_count": null, |
| 18 | + "id": "c2d0ca66-8c50-410d-9a35-f844f6b518aa", |
| 19 | + "metadata": {}, |
| 20 | + "outputs": [], |
| 21 | + "source": [ |
| 22 | + "# import relevant packages\n", |
| 23 | + "import os\n", |
| 24 | + "from rail.projects import library\n", |
| 25 | + "from rail.projects import RailProject\n", |
| 26 | + "import pandas as pd\n", |
| 27 | + "\n", |
| 28 | + "check_dir = os.path.basename(os.path.abspath(os.curdir))\n", |
| 29 | + "if check_dir == 'examples':\n", |
| 30 | + " os.chdir('..')" |
| 31 | + ] |
| 32 | + }, |
| 33 | + { |
| 34 | + "cell_type": "markdown", |
| 35 | + "id": "193228cc-9394-4807-b86d-34d54e2dab97", |
| 36 | + "metadata": {}, |
| 37 | + "source": [ |
| 38 | + "We load the project configuration for flagship below. This config file is very similar to the RomanRubin example, but you notice a few difference. \n", |
| 39 | + "Firstly, I changed the `CommonPaths` to the path to the flagship example catalog, which currently sits on my scratch. The subsequent file structure is similar to the RomanRubin case, so we can use the same library file, `ci_project_libraray.yaml`. I changed the `catalog_tag` under `Baseline` to `flagship` as well. Finally, I changed `IterationVars` to match the iteration pattern, in this case I only have one pixel, `1`. " |
| 40 | + ] |
| 41 | + }, |
| 42 | + { |
| 43 | + "cell_type": "code", |
| 44 | + "execution_count": null, |
| 45 | + "id": "a0302bd0-b62e-484d-8bd0-6fcdfa39e494", |
| 46 | + "metadata": {}, |
| 47 | + "outputs": [], |
| 48 | + "source": [ |
| 49 | + "# load the project configuration:\n", |
| 50 | + "project_flagship = RailProject.load_config(\"examples/ci_project-flagship.yaml\")" |
| 51 | + ] |
| 52 | + }, |
| 53 | + { |
| 54 | + "cell_type": "markdown", |
| 55 | + "id": "bba045f2-c38b-4f2d-b8dc-ac11ee460b9d", |
| 56 | + "metadata": {}, |
| 57 | + "source": [ |
| 58 | + "We can check if we've found the catalog we specified:" |
| 59 | + ] |
| 60 | + }, |
| 61 | + { |
| 62 | + "cell_type": "code", |
| 63 | + "execution_count": null, |
| 64 | + "id": "5b63ac86-0177-4226-8abf-77bd5b1eac63", |
| 65 | + "metadata": {}, |
| 66 | + "outputs": [], |
| 67 | + "source": [ |
| 68 | + "catalog_files_truth = project_flagship.get_catalog_files(\"truth\")\n", |
| 69 | + "print(catalog_files_truth)" |
| 70 | + ] |
| 71 | + }, |
| 72 | + { |
| 73 | + "cell_type": "markdown", |
| 74 | + "id": "8f3768c3-b700-4617-b2e4-9944c806bbef", |
| 75 | + "metadata": {}, |
| 76 | + "source": [ |
| 77 | + "Yes, this is the correct directory. Now we can reduce the raw catalog to the correct format we want.\n", |
| 78 | + "To do this, we simply changed the `reducer_class_name` to `flagship`. This is defined in the library file, and calls the `FlagshipReducer`. This reducer does the following things:\n", |
| 79 | + "- Convert all fluxes (here, we use columns `[fluxtype]_el_model3_ext` which includes fluxes from both the continuum and the emission lines from the galaxy, and `[fluxtype]` denotes LSST $ugrizy$ and Euclid nisp y,j,h and Euclid vis) into magnitudes, and rename the columns to standard values.\n", |
| 80 | + "- Compute the semi-major and minor axes of the galaxy in arcsec and save in the `major` and `minor` columns. Some intermediate quantites such as orientation angles are also saved.\n", |
| 81 | + "- Apply a `gold` magnitude cut, also defined in the library, which corresponds to $i<25.5$. " |
| 82 | + ] |
| 83 | + }, |
| 84 | + { |
| 85 | + "cell_type": "code", |
| 86 | + "execution_count": null, |
| 87 | + "id": "1d64106c-eedb-4607-a4af-b34e4a0d18c3", |
| 88 | + "metadata": {}, |
| 89 | + "outputs": [], |
| 90 | + "source": [ |
| 91 | + "project_flagship.reduce_data(\n", |
| 92 | + " catalog_template=\"truth\",\n", |
| 93 | + " output_catalog_template=\"reduced\",\n", |
| 94 | + " reducer_class_name=\"flagship\",\n", |
| 95 | + " input_selection=\"\",\n", |
| 96 | + " selection=\"gold\",\n", |
| 97 | + ")" |
| 98 | + ] |
| 99 | + }, |
| 100 | + { |
| 101 | + "cell_type": "markdown", |
| 102 | + "id": "c90dcf8d-8049-4f2a-b9a1-c6d7ac671236", |
| 103 | + "metadata": {}, |
| 104 | + "source": [ |
| 105 | + "You can check the input and output catalogs, to see if the reducer has worked:" |
| 106 | + ] |
| 107 | + }, |
| 108 | + { |
| 109 | + "cell_type": "code", |
| 110 | + "execution_count": null, |
| 111 | + "id": "8250a69d-dab5-4b8c-8e31-3b2d84783ead", |
| 112 | + "metadata": {}, |
| 113 | + "outputs": [], |
| 114 | + "source": [ |
| 115 | + "f_truth=pd.read_parquet(\"/pscratch/sd/q/qhang//Flagship/ci_test_flagship_v1/1/part-0.pq\")\n", |
| 116 | + "f_reduce=pd.read_parquet(\"/pscratch/sd/q/qhang//Flagship/ci_test_flagship_v1_gold/1/part-0.pq\")" |
| 117 | + ] |
| 118 | + }, |
| 119 | + { |
| 120 | + "cell_type": "code", |
| 121 | + "execution_count": null, |
| 122 | + "id": "57a5615a-f388-4cd9-a607-abad6c9743dc", |
| 123 | + "metadata": {}, |
| 124 | + "outputs": [], |
| 125 | + "source": [ |
| 126 | + "f_truth" |
| 127 | + ] |
| 128 | + }, |
| 129 | + { |
| 130 | + "cell_type": "code", |
| 131 | + "execution_count": null, |
| 132 | + "id": "aa68bc7b-7f0f-4349-91c4-387730cb3e09", |
| 133 | + "metadata": {}, |
| 134 | + "outputs": [], |
| 135 | + "source": [ |
| 136 | + "f_reduce" |
| 137 | + ] |
| 138 | + }, |
| 139 | + { |
| 140 | + "cell_type": "markdown", |
| 141 | + "id": "181c15af-1ce0-4ac0-99c5-1f2b9011fe2b", |
| 142 | + "metadata": {}, |
| 143 | + "source": [ |
| 144 | + "From here on you can follow examples in `rail_project_example.ipynb` to run the rest of the pipeline." |
| 145 | + ] |
| 146 | + }, |
| 147 | + { |
| 148 | + "cell_type": "code", |
| 149 | + "execution_count": null, |
| 150 | + "id": "3bacb742-35f5-4589-929f-e69450f290ed", |
| 151 | + "metadata": {}, |
| 152 | + "outputs": [], |
| 153 | + "source": [] |
| 154 | + } |
| 155 | + ], |
| 156 | + "metadata": { |
| 157 | + "kernelspec": { |
| 158 | + "display_name": "rail_dev", |
| 159 | + "language": "python", |
| 160 | + "name": "rail_dev" |
| 161 | + }, |
| 162 | + "language_info": { |
| 163 | + "codemirror_mode": { |
| 164 | + "name": "ipython", |
| 165 | + "version": 3 |
| 166 | + }, |
| 167 | + "file_extension": ".py", |
| 168 | + "mimetype": "text/x-python", |
| 169 | + "name": "python", |
| 170 | + "nbconvert_exporter": "python", |
| 171 | + "pygments_lexer": "ipython3", |
| 172 | + "version": "3.12.9" |
| 173 | + } |
| 174 | + }, |
| 175 | + "nbformat": 4, |
| 176 | + "nbformat_minor": 5 |
| 177 | +} |
0 commit comments