Move README into documentation (#63)

Leaving the old documentation intact.

Goal: Have a "single source of truth" for eCLM static file generator
documentation.

* collapse deprecated docs using clickable "<details>" tag

* Provide all options for reading the documentation

* Corrected phrasing about the licence

There are different MIT licences.  Usually the Expat licence is used,
but this name is not as commonly used.  Using the undefined article
at least makes the phrasing correct.

* Source environment and set CSMDATA

With moving to the docs/ structure this got lost.  It applies to most
sections, so it must be stated before section 1.

---------

Co-authored-by: Marco van Hulten <Marco.van.Hulten@uni-bonn.de>

Author: jjokella

README.md

@@ -1,256 +1,24 @@
 # eCLM static file generator
 
+[![docs](https://github.com/HPSCTerrSys/eCLM_static-file-generator/actions/workflows/doc.yml/badge.svg)](https://github.com/HPSCTerrSys/eCLM_static-file-generator/actions/workflows/doc.yml)
+
+## Introduction
+
 This repository shows how to generate curvilinear surface and domain fields for eCLM simulations.
 The generator follows the official CLM-workflow but makes a few adaptions.
 
-It is not necessary to clone [CTSM](https://github.com/ESCOMP/CTSM.git) and [cime](https://github.com/ESMCI/cime.git), as this generator is independent.
-However, the basis is the CLM5.0 release and there might be newer developments in the official repositories.
+## Usage / Documentation
 
-By sourcing the provided environment file
+If on [JSC HPC](https://www.fz-juelich.de/en/jsc/), start with sourcing the provided environment file and set the raw data path:
 
 ```
 source jsc.2024_Intel.sh
+export CSMDATA="/p/scratch/cslts/shared_data/rlmod_eCLM/inputdata/" # this works for JSC users only
 ```
 
-the necessary compilations in this repository can be performed consistently. It also contains the export of necessary paths for netCDF.
-If you are on a [JSC](https://www.fz-juelich.de/en/ias/jsc) machine, it will be very useful to set this data path:
-
-```
-export CSMDATA="/p/largedata2/detectdata/CentralDB/projects/z04"
-```
-
-If not, you'll have to get the relevant raw data at the respective steps and either set `$CSMDATA` to a different path or replace it with your respective local directory.
-
-To create all static files needed to run eCLM, you first need to create a gridfile, then mapping and domain files and only then you can create surface and forcing files.
-For this the grid has to be properly defined in one of the following formats:
-
-- SCRIP
-- ESMF-Mesh
-- CF-convention-file
-
-SCRIP is a very old format not maintained anymore but is still the most effective solution.
-ESMF is able to basically handle any netCDF file that follows the CF-conventions version 1.6 and includes lat/lon values and corners.
-This means that ESMF mesh files are also able to describe unstructured grids.
-
-**Hint:** Once a static file generation is started, it is best
-practice to **not anymore copy or move the repository**. This ensures
-tracability of absolute paths, e.g. absolute paths of map files saved
-in the domain file as netCDF attributes.
-
-## Creation of gridfile
-
-First, we need to create a gridfile that describes our simulation domain.
-These come in two resolutions: ~12 km resolution (EUR-R13B05) and ~3 km resolution (EUR-R13B07).
-Choose one of these two grids if you want to use this with TSMP2.
-In this guide the default is EUR-R13B05, meaning that you will end up with all low-resolution static files on the *icosahedral* grid if you do not modify any of the following commands or scripts.
-For other purposes guidelines for the *rectilinear* and *curvilinear* grids are here as well.
-A curvilinear grid is normally used for CLM5, and when you run eCLM stand-alone you might choose to use a curvilinear grid as well.
-
-The relevant scripts to create the grid files are in `./mkmapgrids/`.
-The gridfile will contain arrays of longitudes and latitudes of the gridboxes' centres and corners.
-The simulation domain is the EURO-CORDEX pan-European domain, which at high latitudes, for the Earth's canonical curvilinear grid, has significant convergence of the zonal dimension with increasing latitude.
-Therefore we *rotate* the grid (of a same size) centred at the equator to the pan-European domain.
-
-Grid files are available on the JSC machines in the DETECT CentralDB below `/p/largedata2/detectdata/CentralDB/projects/z04/detect_grid_specs/grids/` (`$CSMDATA/detect_grid_specs/grids`).
-This is part of a Git repository that is also available [here](https://gitlab.jsc.fz-juelich.de/detect/detect_z03_z04/detect_grid_specification).
-
-### Rectilinear grid
-
-You can create a SCRIP file from a *rectilinear grid* with [`mkscrip_rect.py`](mkmapgrids/mkscrip_rect.py).
-The Python packages numpy, xarray and dask-expr need to be available.
-They are loaded by the [environment file](jsc.2024_Intel.sh) (sourced above).
-On machines without the modules from the environment file, you can install the Python packages with `pip3`, probably best in a [virtual environment](https://docs.python.org/3/library/venv.html), if they are not all available as a package in your operating system.
-The script was modified from `mesh_maker.py` from the CTSM repository to accept 2D lon / lat.
-The main caveat is that the resulting surface files are in 1D which makes them harder to handle.
-The python script `mkscrip_rect.py` can create SCRIP files including the calculation of corners.
-It takes command line arguments like this:
-
-```
-./mkscrip_rect.py --ifile EUR-regLonLat01deg_1204x548_grid_inclbrz_v2.nc --ofile EUR-regLonLat01deg_659792_grid_SCRIP.nc --oformat SCRIP
-```
-
-`--help` provides additional information.
-
-#### Rectilinear grid II: Python script for direct SCRIP-grid-file generation
-
-One can create a SCRIP file for a rectilinear script using the Python
-script `mkmapgrids/mkscripgrid.py`.
-
-For its usage, in particular setting inputs, please refer to the
-script itself.
-
-#### Rectilinear grid III: External Perl script from CTSM/CLM5 repo
-
-The following Perl script from CTSM/CLM5 provides an interface to
-similar functionality as the Python script
-`mkmapgrids/mkscripgrid.py`:
-
-https://github.com/ESCOMP/CTSM/blob/994e02983cf557410fe455b6bd64ee61ca50d488/tools/site_and_regional/mknoocnmap.pl
-
-### Curvilinear grid
-
-eCLM can be used with a *curvilinear grid*.
-
-You can, e.g., use the 450x438 gridfile including boundary relaxation zone, `EUR-11_450x438_grid_inclbrz13gp_v2.nc`, as the input file.
-If you want a high-resolution curvilinear grid, use `EUR-0275_1600x1552_grid_inclbrz_v2.nc`.
-However, in eCLM we use a slightly smaller domain, so you must truncate the files:
-
-```
-cd mkmapgrids/
-ncks -d rlat,3,434 -d rlon,3,446 $CSMDATA/detect_grid_specs/grids/EUR-11_450x438_grid_inclbrz13gp_v2.nc EUR-11_444x432_grid_inclbrz13gp_v2.nc
-ncks -d rlat,3,1548 -d rlon,3,1596 $CSMDATA/detect_grid_specs/grids/EUR-0275_1600x1552_grid_inclbrz_v2.nc EUR-0275_1594x1546_grid_inclbrz_v2.nc
-```
-
-At the moment SCRIP generation from curvilinear grids can be done and is tested to work with the NCAR Command Language (NCL), even though it is [deprecated](https://www.ncl.ucar.edu/open_letter_to_ncl_users.shtml).
-NCL can be installed through Conda.
-If you have no Conda yet on your system, you can install it, including the conda-forge channel, following [this guide](https://github.com/conda-forge/miniforge?tab=readme-ov-file#unix-like-platforms-macos-linux--wsl).
-Then follow [this guide](https://yonsci.github.io/yon_academic/portfolio/portfolio-9/#installing-ncl) to install NCL.
-The repository contains the NCL-script [`mkscrip_curv.ncl`](mkmapgrids/mkscrip_curv.ncl) that can create a SCRIP file from a netCDF that contains the lat- and lon-center coordinates.
-It is not necessary to provide the corners because the internal routine of NCL seems to calculate them correctly for the later steps.
-Adapt the variable `f` in `mkscrip_curv.ncl` to your gridfile and execute:
-
-```
-conda activate NCL_environment
-ncl mkscrip_curv.ncl
-conda deactivate
-conda deactivate
-```
-
-### Icosahedral grid
-
-The atmospheric model ICON runs on an *icosahedral grid*, sometimes called *triangular grid*.
-The land model eCLM, when coupled to ICON (in TSMP2), also uses this grid.
-
-Check out https://zonda.ethz.ch/ for generating icosahedral input grids for `mkscrip_icos.py` (specified under option `--ifile`).
-
-Then convert your ICON gridfile to the SCRIP format with the python script [`mkscrip_icos.py`](mkmapgrids/mkscrip_icos.py):
-
-```
-./mkscrip_icos.py --ifile EUR-R13B05_199920_grid_inclbrz_v2.nc --ofile EUR-R13B05_199920_grid_SCRIP.nc
-```
-
-For [TSMP2](https://github.com/HPSCTerrSys/TSMP2), on a 0.11 degree (~12 km) resolution, you probably want to use the EUR-R13B05 grid including boundary relaxation zone, `EUR-R13B05_199920_grid_inclbrz_v2.nc`, as the input file.
-If you want a high-resolution icosahedral grid, use `EUR-R13B07_2473796_grid_inclbrz_v1.nc`.
-
-Further information about the DETECT grid specification can be found [here](https://gitlab.jsc.fz-juelich.de/detect/detect_z03_z04/detect_grid_specification).
-
-## Creation of mapping files
-
-For the creation of the mapping files of CLM inputdata to our grid use `mkmapdata/runscript_mkmapdata.sh`.
-Adjust the Slurm directives to your compute time project and partition.
-Below the Slurm directives, modify `GRIDNAME` and `GRIDFILE` to your grid and previously created SCRIP file.
-The script can be used on JURECA and JUWELS but it is advisable to use the large memory partitions for larger domains.
-
-The files are at JSC available below `$CSMDATA/lnd/clm2/mappingdata/grids/`.
-If you don't have access to the CLM mappingdata you have to download it.
-
-There are two possible ways to download the grids
-
-1. Use:
-```
-wget --no-check-certificate -i clm_mappingfiles.txt
-```
-
-2. Run `download_grids.sh` after adapting inputs (in particular the path `myraw`).
-   You need Subversion (`svn`) for this.
-```
-./download_grids.sh
-```
-
-When all grids are downloaded, start the creation of the weights with
-```
-sbatch runscript_mkmapdata.sh
-```
-
-This will create regridding and netCDF mapping files in the current
-directory.
-
-### Icosahedral grid
-
-Experience has shown that conservative remapping does not always work for icosahedral grids (EUR-R13B05 and EUR-R13B07).
-In that case `runscript_mkmapdata.sh` decides to create a mapping file that bilinearly interpolates fields.
-
-## Creation of domain files
-
-The CIME submodule under `./gen_domain_files/` generates the domain files for CLM.
-This repository contains a simplified version of `gen_domain` which is easier to compile on JSC machines and you do not need the CIME repository.
-Required modules are imkl, netCDF and netCDF-Fortran (all provided by `jsc.2024_Intel.sh`).
-Then compile `src/gen_domain.F90` with
-
-```
-gfortran -o gen_domain src/gen_domain.F90 -mkl -I${INC_NETCDF} -lnetcdff -lnetcdf
-```
-
-If the Intel Math Kernel Library (MKL) is not available on your system, you can remove the `-mkl` flag without consequences for the output.
-After the compilation you can execute `gen_domain` with $MAPFILE being one of the mapping files created in the step before (in `mkmapdata/`) and $GRIDNAME being a string with the name of your grid, e.g. `EUR-R13B05` for the ~12-km icosahedral grid.
-The choice of $MAPFILE does not influence the lat- and longitude values in the domain file but can influence the land/sea mask.
-
-**Hint:** For better reproducibility, specify the absolute path of
-`$MAPFILE`. The absolute path to the file can be printed using
-`realpath $MAPFILE`.
-
-```
-./gen_domain -m $MAPFILE -o $GRIDNAME -l $GRIDNAME -u $USER
-```
-
-The created domain file will later be modified.
-
-## Creation of surface file
-
-The surface creation tool is found under `./mksurfdata/`.
-The required modules Intel and netCDF-Fortran are loaded by `jsc.2024_Intel.sh`.
-
-First, compile `mksurfdata_map`:
-
-```
-cd mksurfdata/src
-gmake
-```
-
-Then check that `mksurfdata_map` is available inside `./mksurfdata/`.
-
-After compilation execute
-
-```
-export GRIDNAME="EUR-R13B05"    # in case of the low-resolution icosahedral grid
-export CDATE="`date +%y%m%d`"   # should match mapping files creation date
-
-# generate surfdata
-./mksurfdata.pl -r usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -l $CSMDATA -allownofile -y 2005 -hirespft
-```
-
-to create a real domain with hires pft.
-Again, you need to have set $GRIDNAME, the date $CDATE in yymmdd format (matching the mapping files) and the path $CSMDATA where the raw data of CLM is stored.
-You have to download the data from https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/rawdata/ if you have no access to JSC machines.
-Also make sure that mksurfdata and mkmapdata have the same parent directory.
-
-At least in the original UCAR rawdata input files, not all variables are available (under the correct name).
-Instead you can create the surface file with *mostly* constant values (apparently `SOIL_COLOR` is not homogeneous) with the following command.
-In a later step we are anyway going to replace variables in this file that are specific for eCLM (TSMP2).
-
-```
-./mksurfdata.pl -r usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -l $CSMDATA -allownofile -y 2005 -hirespft -usr_mapdir="../mkmapdata/" -no-crop -pft_idx 13 -pft_frc 100 -soil_cly 60 -soil_col 10 -soil_fmx 0.5 -soil_snd 40
-```
-
-
-## Modification of the surface and domain file
-
-The created surface and domain file have negative longitudes that CLM5 does not accept and inherently has no landmask. To modify the longitudes and to add a landmask use `mod_domain.sh` after inserting the paths to your files.
-
-At least for TSMP2, further modification of the surface file is needed and not yet included in this (tested) workflow.
-The necessary replacement routines can be found in the `dev_replace_tsmp2` branch in the [`mksurfdata/`](https://github.com/HPSCTerrSys/eCLM_static-file-generator/tree/dev_replace_tsmp2/mksurfdata) directory.
-
-
-## Surface File: Use landcover GLC2000 and soil texture from SOILGRIDS
-
-Only for BGC mode!
-
-The following script updates landcover using GLC2000 and soils using SOILGRIDS.
-
-```
-./replace_surfdata.py
-```
+Otherwise you need to provide for the data and software environment yourself (as documented below).
 
-Additionally, this script checks CLM gridcells to make sure the
-percentages per landunit in each gridcell sum up to one.
+Then please check the [generated documentation](https://hpscterrsys.github.io/eCLM_static-file-generator/INDEX.html) online, [generate a static version](docs/README.md) yourself or browse it locally: `$VISUAL docs/users_guide/?_*.md` with `$VISUAL` your editor or viewer of choice.
 
+## License
+eCLM static file generator is open source software and is licensed under an [MIT License](https://github.com/HPSCTerrSys/eCLM_static-file-generator/blob/master/LICENSE).

docs/INDEX.md

@@ -4,5 +4,6 @@
 **Welcome!** You are viewing the first version of the documentation for eCLM static file generator. This is a living document, which means it will be continuously updated and improved. Please check back regularly for the latest information and updates.
 ```
 
-For now: look at the main README for a workflow. Goal: Port this
-README (except general info) to this doc.
+It is not necessary to clone [CTSM](https://github.com/ESCOMP/CTSM.git) and [cime](https://github.com/ESMCI/cime.git), as this generator is independent.
+However, the basis is the CLM5.0 release and there might be newer developments in the official repositories.
+