adding reducer for flagship sim

examples/ci_project-flagship.yaml

@@ -0,0 +1,52 @@
+Project:
+
+  Name: ci_test_flagship
+
+  # Include other configuration files
+  Includes:
+    - tests/ci_project_library.yaml
+
+  PathTemplates: {}
+
+  CommonPaths:
+    root: /pscratch/sd/q/qhang/
+    scratch_root: "{root}"
+    catalogs_dir: "{root}/Flagship"
+    project: ci_test_flagship
+    sim_version: v1
+
+  # Baseline configuraiton, included in others by default
+  Baseline:
+    catalog_tag: flagship
+    pipelines: ['all']
+    file_aliases:  # Set the training and test files
+      test: test_file_100k
+      train: train_file_100k
+      train_zCOSMOS: train_file_zCOSMOS_100k
+      wide: wide_file_full
+      deep: deep_file_full
+      spec: spec_file_full
+
+  # These define the variant configurations for the various parts of the analysis
+  Flavors:
+    - Flavor:
+        name: train_cosmos
+        pipelines: ['pz', 'tomography']
+        file_aliases:  # Set the training and test files
+          test: test_file_100k
+          train: train_file_zCOSMOS_100k
+    - Flavor:
+        name: gpz_gl
+        pipelines: ['pz']  # only run the pz pipeline
+        pipeline_overrides:  # Override specifics for particular pipelines
+          default:
+            kwargs:
+              algorithms: ['gpz']  # Only run gpz
+          inform:
+            inform_gpz:
+              gpz_method: GL
+
+  # These are variables that we iterate over when running over entire catalogs
+  IterationVars:
+    healpix:
+      - 1

examples/rail_project_flagship_reducer_example.ipynb

@@ -0,0 +1,177 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4227b7d4-a861-4839-b4b7-52a37534a283",
+   "metadata": {},
+   "source": [
+    "# Demo for reducing the Euclid flagship simulations:\n",
+    "\n",
+    "The full simulations are available at https://cosmohub.pic.es/catalogs/353. If you use the catalog for any publication, please use the acknowledgements listed on this website.\n",
+    "\n",
+    "Notice that the catalog inherently has a cut for the Euclid h band at ${\\tt euclid\\_nisp\\_h} < 26.6$. The full catalog covers 1 octant of the sky but the example file only contain objects on 1 pixel (out of a total of 256), and subsampled to be 1% of the original size. Notice also there is a small region (150 < RA < 155 deg., 5 < DEC < 10 deg.) where there is no magnitude or flux cut, which has significantly higher number density. This small region *is* part of the example file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c2d0ca66-8c50-410d-9a35-f844f6b518aa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# import relevant packages\n",
+    "import os\n",
+    "from rail.projects import library\n",
+    "from rail.projects import RailProject\n",
+    "import pandas as pd\n",
+    "\n",
+    "check_dir = os.path.basename(os.path.abspath(os.curdir))\n",
+    "if check_dir == 'examples':\n",
+    "    os.chdir('..')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "193228cc-9394-4807-b86d-34d54e2dab97",
+   "metadata": {},
+   "source": [
+    "We load the project configuration for flagship below. This config file is very similar to the RomanRubin example, but you notice a few difference. \n",
+    "Firstly, I changed the `CommonPaths` to the path to the flagship example catalog, which currently sits on my scratch. The subsequent file structure is similar to the RomanRubin case, so we can use the same library file, `ci_project_libraray.yaml`. I changed the `catalog_tag` under `Baseline` to `flagship` as well. Finally, I changed `IterationVars` to match the iteration pattern, in this case I only have one pixel, `1`. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a0302bd0-b62e-484d-8bd0-6fcdfa39e494",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load the project configuration:\n",
+    "project_flagship = RailProject.load_config(\"examples/ci_project-flagship.yaml\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bba045f2-c38b-4f2d-b8dc-ac11ee460b9d",
+   "metadata": {},
+   "source": [
+    "We can check if we've found the catalog we specified:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5b63ac86-0177-4226-8abf-77bd5b1eac63",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "catalog_files_truth = project_flagship.get_catalog_files(\"truth\")\n",
+    "print(catalog_files_truth)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f3768c3-b700-4617-b2e4-9944c806bbef",
+   "metadata": {},
+   "source": [
+    "Yes, this is the correct directory. Now we can reduce the raw catalog to the correct format we want.\n",
+    "To do this, we simply changed the `reducer_class_name` to `flagship`. This is defined in the library file, and calls the `FlagshipReducer`. This reducer does the following things:\n",
+    "- Convert all fluxes (here, we use columns `[fluxtype]_el_model3_ext` which includes fluxes from both the continuum and the emission lines from the galaxy, and `[fluxtype]` denotes LSST $ugrizy$ and Euclid nisp y,j,h and Euclid vis) into magnitudes, and rename the columns to standard values.\n",
+    "- Compute the semi-major and minor axes of the galaxy in arcsec and save in the `major` and `minor` columns. Some intermediate quantites such as orientation angles are also saved.\n",
+    "- Apply a `gold` magnitude cut, also defined in the library, which corresponds to $i<25.5$. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1d64106c-eedb-4607-a4af-b34e4a0d18c3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "project_flagship.reduce_data(\n",
+    "    catalog_template=\"truth\",\n",
+    "    output_catalog_template=\"reduced\",\n",
+    "    reducer_class_name=\"flagship\",\n",
+    "    input_selection=\"\",\n",
+    "    selection=\"gold\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c90dcf8d-8049-4f2a-b9a1-c6d7ac671236",
+   "metadata": {},
+   "source": [
+    "You can check the input and output catalogs, to see if the reducer has worked:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8250a69d-dab5-4b8c-8e31-3b2d84783ead",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "f_truth=pd.read_parquet(\"/pscratch/sd/q/qhang//Flagship/ci_test_flagship_v1/1/part-0.pq\")\n",
+    "f_reduce=pd.read_parquet(\"/pscratch/sd/q/qhang//Flagship/ci_test_flagship_v1_gold/1/part-0.pq\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "57a5615a-f388-4cd9-a607-abad6c9743dc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "f_truth"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "aa68bc7b-7f0f-4349-91c4-387730cb3e09",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "f_reduce"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "181c15af-1ce0-4ac0-99c5-1f2b9011fe2b",
+   "metadata": {},
+   "source": [
+    "From here on you can follow examples in `rail_project_example.ipynb` to run the rest of the pipeline."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3bacb742-35f5-4589-929f-e69450f290ed",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "rail_dev",
+   "language": "python",
+   "name": "rail_dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}