Updates to handle large Monte Carlo runs (#18)

* handle different base years of K and pop in sliiders

* allow for restarting of surge lookup creation

* fix issues when seg_var != "seg_adm"

* fix io funcs for on-the-fly quantiles, fix storage_options default, add other selectors to prep_sliiders, allow other name options for dims of input stores, improve performance of create_template_dataarray, drop now unnecessary save_to_zarr_region wrapper

* handle upcoming xarray default merge changes, allow for skipping check for surge lookup completeness to save time, allow chunking along other dims than seg, use as_completed instead of client.gather, add some print stmts for model status, fix use of scen_mc_filter in execute_pyciam, remove erroneously left testing return in execute_pyciam, allow for unique spec of quantile to use for refA, drop custom save_to_zarr_region func

* optimize slr loading by using pint later in the process

* add vsl to the variables that are condensed to segment in collapse_econ_inputs_to_seg

* improve performance of surge lookup creation by loading only min and max of slr data; improve dtypes of template zarrs;

* remove all pins in environment.yml except python=3.14

* add numpy >= 2 dependency to pyproject.toml

* explicitly specify "quantile"

* add comment for when _load_scenario_mc only returns scenario filter, not MC

* fix i/o for when passing scen_mc_filter

* fix bug in using pandas index for xr.concat with pandas string dtype in pandas 3

* update execute_pyciam to work with large ensemble runs

* extract some functionality into private funcs and confirm it works for end-to-end large ensemble run

* remove unnecessary block

Author: bolliger32

environment/environment.yml

@@ -2,34 +2,31 @@ name: pyciam
 channels:
   - conda-forge
 dependencies:
-  - bottleneck=1.3
-  - bokeh=2.4.3  # for use of dask dashboard
-  - cartopy=0.21.1
-  - cloudpathlib=0.13
-  - distributed=2023.3.1
-  - flox=0.6.8
-  - fsspec=2023.3.0
-  - geopandas=0.12.2
-  - gitpython=3.1.31  # unmarked dependency of rhg_compute_tools
-  - jupyterlab=3.6.1
-  - matplotlib=3.7.1
-  - numpy=1.24.2
-  - numpy_groupies=0.9.20
-  - oct2py=5.6.0
-  - octave=7.3.0
-  - openpyxl=3.1.1
-  - pandas=1.5.3
-  - papermill=2.3.4
-  - pint-xarray=0.3
-  - pip=23.0.1
-  - python=3.11.0
-  - requests=2.28.2
-  - scikit-learn=1.2.2
-  - scipy=1.10.1
-  - tqdm=4.65.0
-  - xarray=2023.2.0
-  - zarr=2.14.2
+  - bottleneck
+  - bokeh  # for use of dask dashboard
+  - cartopy
+  - cloudpathlib
+  - distributed
+  - flox
+  - fsspec
+  - geopandas
+  - gitpython  # unmarked dependency of rhg_compute_tools
+  - jupyterlab
+  - matplotlib
+  - numpy
+  - openpyxl
+  - pandas
+  - papermill
+  - pint-xarray
+  - pip
+  - python=3.14
+  - requests
+  - scikit-learn
+  - scipy
+  - tqdm
+  - xarray
+  - zarr
   - pip:
-    - python-ciam==1.1
-    - rhg_compute_tools==1.3
-    - parameterize_jobs==0.1
+    - python-ciam
+    - rhg_compute_tools
+    - parameterize_jobs