Brief summary of bug

CTSM initialization consumes excessive memory (and time) at very high resolution (e.g. ne1024), to the point of risking out-of-memory during startup. The cost is dominated by global-sized arrays replicated identically on every MPI rank, NetCDF/PIO file handles held open long past their last use, and duplicate ESMF mesh construction for CDEPS streams whose data already lives on the model grid.

General bug information

CTSM version you are using: ctsm5.4.044-3-gd3078f4e0

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: High-resolution land configurations, in particular ne1024

Details of bug

1. Per-rank replication of the global land mask (src/cpl/share_esmf/lnd_set_decomp_and_domain.F90). In both lnd_set_lndmask_from_maskmesh and lnd_set_lndmask_from_lndmesh, lndmask_glob(gsize) is allocated on every MPI rank and built with an all-rank ESMF_VMAllReduce into a second global-sized integer temporary (itemp_glob). At ne1024 this is two global-sized integer arrays held per rank with identical contents across every rank on a node — pure duplication that scales with rank count.

2. NetCDF/PIO file handles held open past their last use. Several init routines open a file, finish reading, and only close the handle much later — or, on at least one path, never:

• src/main/clm_instMod.F90 — params_ncid closed at the end of init_accflds rather than right after its last use (bgc_vegetation_inst%Init).
• src/main/initVerticalMod.F90 — ncid closed at the end of initVertical rather than after the last read (STD_ELEV).
• src/biogeophys/UrbanParamsType.F90 — ncid leaked on the nlevurb == 0 early-return path in UrbanInput.
• src/main/organicFileMod.F90 — no close after reading ORGANIC in organicrd.
• src/main/surfrdMod.F90 — handles left open after the dimension reads in surfrd_get_num_patches and surfrd_get_nlevurb.

3. Duplicate ESMF mesh construction for redist streams (src/cpl/share_esmf/PrigentRoughnessStreamType.F90 and src/cpl/share_esmf/UrbanTimeVarType.F90). For streams already on the model grid (mapalgo == 'redist') — as for the ne1024 Prigent-roughness and urban-time-varying inputs — shr_strdata_init_from_inline still reads the stream mesh file and builds a second full ESMF mesh identical to the already-built CLM model mesh. This duplicate is a large init memory/time cost at ne1024. Avoiding it requires CDEPS to accept a caller-supplied mesh (stream_mesh_in), added in Reduce the memory usage that is important for ne1024 simulation CDEPS#414.

Important details of your setup / configuration so we can reproduce the bug

FHISTC_LTso compset at ne1024 resolution, Derecho, Intel compiler