cime6.3.6 brings in changes so that the batch system can be used to build tests. Currently the builds are done per compiler. So all the intel as one job, all gnu as another, and nvhpc as the third. Because the list of tests aren't balanced between compilers this means the intel builds take the longest, gnu in the middle and nvhpc as a distant third. As noted as a box in #275 balancing the test lists would make this more efficient. However, even better would be to send all the builds to the batch system.
In order to do this, we might need to make a couple changes to run_sys_tests. Currently it sends each compiler run of create_test to a batch job using qsub. Without the builds this step will take a fraction of the time. But, it still probably isn't a problem to send it this way, the create_newcase step for each case does take a while to do.
I also thought that we'd need to add settings about this for the build. But, it looks like BATCHED_BUILD is normally default FALSE. But, it assumes TRUE for testing. But, we should probably add the new argument "--no-batch-build" argument to create_test to run_sys_tests.
This might be something that we have turned on by default, by adding a setting for it in ccs_config. See ESMCI/ccs_config_cesm#290
Definition of done:
cime6.3.6 brings in changes so that the batch system can be used to build tests. Currently the builds are done per compiler. So all the intel as one job, all gnu as another, and nvhpc as the third. Because the list of tests aren't balanced between compilers this means the intel builds take the longest, gnu in the middle and nvhpc as a distant third. As noted as a box in #275 balancing the test lists would make this more efficient. However, even better would be to send all the builds to the batch system.
In order to do this, we might need to make a couple changes to run_sys_tests. Currently it sends each compiler run of create_test to a batch job using qsub. Without the builds this step will take a fraction of the time. But, it still probably isn't a problem to send it this way, the create_newcase step for each case does take a while to do.
I also thought that we'd need to add settings about this for the build. But, it looks like BATCHED_BUILD is normally default FALSE. But, it assumes TRUE for testing. But, we should probably add the new argument "--no-batch-build" argument to create_test to run_sys_tests.
This might be something that we have turned on by default, by adding a setting for it in ccs_config. See ESMCI/ccs_config_cesm#290
Definition of done: