case.setup "--keep" option NOT working when it gets to the case.build step

[ekluzek]

The --keep option is NOT working in case.setup in cime6.1.169 with other submodules from ctsm5.4.034.

How to reproduce. I setup a case, I'm using SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop. I then kept a copy of the original keep files (env_mach_specific.xml, Macros.make, and Macros.cmake) (copied to one with .orig). Then I modified the case version by adding comments to the files. And then kept a copy of that (copied to .new). I made sure those files were readonly and then went through the case steps to see where it fails.

The case.setup works correctly, but on the case.build step it tries to overwrite the files and fails. I'll give details below

[ekluzek]

First run case.setup and show that it works correctly:

chmod -w Macros.* env_mach_specific.xml*
ls -l Macros.* env_mach_specific.xml*
-r--r--r-- 1 erik cgdtss 5183 Apr 26 16:57 env_mach_specific.xml
-r--r--r-- 1 erik cgdtss 5183 Apr 26 16:57 env_mach_specific.xml.new
-r--r--r-- 1 erik cgdtss 5162 Apr 26 16:55 env_mach_specific.xml.orig
-r--r--r-- 1 erik cgdtss 1321 Apr 26 16:56 Macros.cmake
-r--r--r-- 1 erik cgdtss 1321 Apr 26 17:07 Macros.cmake.new
-r--r--r-- 1 erik cgdtss 1306 Apr 26 16:56 Macros.cmake.orig
-r--r--r-- 1 erik cgdtss 1321 Apr 26 16:57 Macros.cmak.new
-r--r--r-- 1 erik cgdtss 2267 Apr 26 16:56 Macros.make
-r--r--r-- 1 erik cgdtss 2267 Apr 26 17:05 Macros.make.new
-r--r--r-- 1 erik cgdtss 2252 Apr 26 16:55 Macros.make.orig
./case.setup --reset --keep 'env_mach_specific.xml' --keep Macros.make --keep Macros.cmake
Setting up test for SMS in directory /glade/derecho/scratch/erik/tests_ctsm5434erikhacknia/SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop.GC.ctsm5434erikhacknia_nvh
WARNING: Found difference in test REST_OPTION: case: none original value $STOP_OPTION
Successfully cleaned .case.test
Successfully cleaned cmake_macros
ls -l Macros.make Macros.cmake env_mach_specific.xml
-r--r--r-- 1 erik cgdtss 5183 Apr 26 16:57 env_mach_specific.xml
-r--r--r-- 1 erik cgdtss 1321 Apr 26 17:07 Macros.cmake
-r--r--r-- 1 erik cgdtss 2267 Apr 26 17:05 Macros.make

This step works correctly as it doesn't list the files to clean in the list of "Successfully cleaned " log output. And we verify that the files didn't get changed afterwards.

But, on the case.build step it tries to overwrite the Macros.make file and fails, because my permission change doesn't allow it.

./case.build 
Building test for SMS in directory /glade/derecho/scratch/erik/tests_ctsm5434erikhacknia/SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop.GC.ctsm5434erikhacknia_nvh
WARNING: Found difference in test REST_OPTION: case: none original value $STOP_OPTION
Building case in directory /glade/derecho/scratch/erik/tests_ctsm5434erikhacknia/SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop.GC.ctsm5434erikhacknia_nvh
sharedlib_only is True
model_only is False
Generating component namelists as part of build
  2026-04-26 17:20:32 atm 
Create namelist for component datm
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/components/cdeps/datm/cime_config/buildnml
  2026-04-26 17:20:32 lnd 
Create namelist for component clm
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/cime_config/buildnml
NOTE: LND_TUNING_MODE is clm6_0_GSWP3v1 which is NOT tuned, but is based on clm5_0_GSWP3v1
  2026-04-26 17:20:32 ice 
Create namelist for component sice
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/non_py/src/components/stub_comps_nuopc/sice/cime_config/buildnml
  2026-04-26 17:20:32 ocn 
Create namelist for component socn
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/non_py/src/components/stub_comps_nuopc/socn/cime_config/buildnml
  2026-04-26 17:20:32 rof 
Create namelist for component mosart
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/components/mosart//cime_config/buildnml
  2026-04-26 17:20:32 glc 
Create namelist for component sglc
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
  2026-04-26 17:20:32 wav 
Create namelist for component swav
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
  2026-04-26 17:20:32 esp 
Create namelist for component sesp
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
  2026-04-26 17:20:32 cpl 
Create namelist for component drv
   Calling /glade/work/erik/ctsm_worktrees/nvidia_hack/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ROF']
libs from case_support_libraries ['gptl', 'pio', 'csm_share', 'FTorch', 'CDEPS']
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/tests_ctsm5434erikhacknia/SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop.GC.ctsm5434erikhacknia_nvh/./case.build", line 269, in <module>
    _main_func(__doc__)
  File "/glade/derecho/scratch/erik/tests_ctsm5434erikhacknia/SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop.GC.ctsm5434erikhacknia_nvh/./case.build", line 243, in _main_func
    success = test.build(
              ^^^^^^^^^^^
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/SystemTests/system_tests_common.py", line 446, in build
    self.build_phase(
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/SystemTests/system_tests_common.py", line 494, in build_phase
    self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/SystemTests/system_tests_common.py", line 501, in build_indv
    build.case_build(
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/build.py", line 1339, in case_build
    return run_and_log_case_status(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/status.py", line 96, in run_and_log_case_status
    rv = func()
         ^^^^^^
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/build.py", line 1323, in <lambda>
    functor = lambda: _case_build_impl(
                      ^^^^^^^^^^^^^^^^^
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/build.py", line 1213, in _case_build_impl
    logs = _build_libraries(
           ^^^^^^^^^^^^^^^^^
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/build.py", line 813, in _build_libraries
    generate_makefile_macro(case, caseroot)
  File "/glade/work/erik/ctsm_worktrees/nvidia_hack/cime/CIME/build.py", line 206, in generate_makefile_macro
    with open(os.path.join(caseroot, "Macros.make"), "w") as fd:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/glade/derecho/scratch/erik/tests_ctsm5434erikhacknia/SMS_D.f10_f10_mg37.I2000Clm60BgcCrop.derecho_nvhpc.clm-crop.GC.ctsm5434erikhacknia_nvh/Macros.make'

[ekluzek]

Note, that when I run similar to above, but allow the Macros.make file to be changed, both the env_mach_specific.xml AND Macros.cmake files get modified and overwritten. So this needs to be fixed for those cases to even use this in a simple manner.

[ekluzek]

I noted this in #4971 as well, but the other thing I think is needed is to add a unit test that makes sure that using the --keep option works correctly after both case.setup and case.build.