ENH: Auto-cleanup BigIO test outputs and enforce serial execution

[hjmjohnson]

Summary

• Adds ITK_REMOVE_BIGIO_FILES_ON_SUCCESS (advanced CMake option, default ON) that automatically removes multi-gigabyte BigIO test output files after successful completion
• Enforces RUN_SERIAL on all BigIO tests to prevent concurrent disk exhaustion
• New CMake function itk_add_bigio_test_cleanup() using CTest fixtures

Motivation

The 11 BigIO write-read tests produce up to 47 GB of temporary files:

Test	Output size
LargeImage04 (.mhd/.raw, .tif)	9.1 GB each
LargeImageWriteReadTest_3D (.mha)	6.7 GB
LargeImage03 (.mhd/.raw, .tif)	4.7 GB each
LargeImage02 (.mhd/.raw, .tif)	3.0 GB each
LargeImage01 (.mhd/.raw, .tif)	1.7 GB each

On CI runners (GitHub Actions ubuntu-22.04: 146 GB total, ~89 GB free after build), running these tests without cleanup risks disk exhaustion. PR #6011 addressed this with post-build .o/.a cleanup, but the root cause is that BigIO outputs accumulate across test runs.

How it works

CTest fixture semantics:

• The main test is FIXTURES_SETUP — it runs and produces output files
• A cleanup test (cmake -E rm -f) is FIXTURES_CLEANUP — it runs only after SETUP passes
• On test failure, output files are retained for debugging
• For .mhd files, companion .raw/.zraw data files are also removed
• Set ITK_REMOVE_BIGIO_FILES_ON_SUCCESS=OFF to always retain files

Design alternatives considered

An in-test C++ self-cleanup approach was evaluated and rejected. See Documentation/docs/design/BigIO_test_cleanup_design.md for the full rationale (multiple exit paths, format-dependent companion files, reader lifetime, separation of concerns, no runtime configurability).

Affected modules

• IO/Meta: LargeMetaImageWriteReadTest 1–4
• IO/TIFF: LargeTIFFImageWriteReadTest 1–4
• IO/ImageBase: LargeImageWriteReadTest 2D/3D, WriteConvertRead

Test plan

• Verify ITK_REMOVE_BIGIO_FILES_ON_SUCCESS=ON (default): BigIO output files are removed after passing tests
• Verify ITK_REMOVE_BIGIO_FILES_ON_SUCCESS=OFF: BigIO output files are retained
• Verify test failure retains output files for debugging
• Verify RUN_SERIAL prevents concurrent BigIO tests in ctest -jN

🤖 Generated with Claude Code

[hjmjohnson]

Design Alternatives: BigIO Test Output Cleanup

Chosen approach: CTest fixture cleanup (CMake-side)

A new CMake function itk_add_bigio_test_cleanup() uses CTest fixture semantics to wire up automatic file removal after successful test completion:

• The main test is registered as FIXTURES_SETUP
• A cleanup test (cmake -E rm -f) is registered as FIXTURES_CLEANUP
• CTest only runs CLEANUP when the SETUP test passes
• On test failure, output files are retained for debugging
• Controlled by advanced option ITK_REMOVE_BIGIO_FILES_ON_SUCCESS (default ON; set OFF to retain files)

For .mhd files, companion .raw/.zraw data files are also removed.

Alternative considered: in-test C++ self-cleanup

All 4 test source files follow the same pattern: write a large image, read it back, compare pixels, return EXIT_SUCCESS or EXIT_FAILURE. Adding std::remove(filename.c_str()) before return EXIT_SUCCESS would be straightforward.

Why this was rejected:

1. Multiple exit paths. Each test has 2–4 return EXIT_FAILURE paths (catch blocks, pixel comparison failures). Cleanup must be placed only before success returns; an accidental placement before a failure return would destroy evidence needed for debugging.

2. Format-dependent companion files. .mhd writes produce two files (.mhd header + .raw data). Each test would need format-aware cleanup logic duplicated in C++. The CMake function handles this centrally.

3. Reader lifetime concerns. After reader->Update(), the reader and its IO object are still alive. On Windows, IO objects may hold file handles. Deleting files before these objects are destroyed could be platform-problematic.

4. Separation of concerns. The tests exist to verify IO correctness, not manage disk space. Mixing infrastructure cleanup into test logic makes test code harder to review and reason about.

5. No runtime configurability. Developers inspecting output files for debugging would need to modify and recompile the test source. The CMake option ITK_REMOVE_BIGIO_FILES_ON_SUCCESS=OFF provides that control without touching code.

6. No C++17 filesystem dependency. The CMake approach uses cmake -E rm -f, avoiding any need for <filesystem> headers or platform-specific removal APIs in test code.

Serial execution requirement

All BigIO tests now set RUN_SERIAL True to prevent CTest from running multiple multi-gigabyte tests concurrently. Without this, ctest -j3 could launch three 9 GB tests simultaneously, requiring ~27 GB of temporary disk space on top of regular test output.

The RESOURCE_LOCK MEMORY_SIZE property was already present but only serializes tests within the same resource group. RUN_SERIAL ensures no other test runs concurrently regardless of resource groups.

[greptile-apps]

Greptile Summary

This PR introduces automatic cleanup of multi-gigabyte BigIO test output files using CTest fixtures and enforces serial execution of all BigIO tests to prevent concurrent disk exhaustion on CI runners. The new itk_add_bigio_test_cleanup() CMake function in ITKModuleTest.cmake registers each BigIO write-read test as a FIXTURES_SETUP and adds a companion _cleanup test as FIXTURES_CLEANUP that deletes the output files. The three affected test modules (IO/ImageBase, IO/Meta, IO/TIFF) consolidate their existing per-test RESOURCE_LOCK properties into single set_tests_properties blocks and add RUN_SERIAL True.

• CTest cleanup semantics are incorrect in the design: FIXTURES_CLEANUP tests always run after the fixture completes, regardless of whether the setup passed or failed (confirmed by CMake 4.3 docs). The in-code comment "only if the fixture's SETUP passed" and the PR description's claim that "output files are retained for debugging on failure" are both inaccurate — files will be deleted even after a failed test.
• Cleanup tests won't auto-run in filtered ctest invocations: Because no test has FIXTURES_REQUIRED for the new fixtures, CTest will not automatically pull in _cleanup tests when running a subset (e.g. ctest -R LargeMetaImage). In a full unfiltered run the cleanup tests are already in the set and fixture ordering is respected, so the primary CI use-case works correctly.
• Variable naming inconsistency: The local variable NAME_WE in itk_add_bigio_test_cleanup is populated with the NAME_WLE component — a minor but confusing mismatch.
• The RUN_SERIAL consolidation is clean and correct; removing the redundant per-test set_property APPEND RUN_SERIAL blocks is a good simplification.

Confidence Score: 3/5

Safe to merge with caveats — the disk-cleanup mechanism works for full CI runs but the stated 'retain on failure' guarantee is incorrect per CTest semantics

The core CI goal (prevent disk exhaustion on full test runs) will work correctly. However the stated design invariant about retaining debug files on failure is factually wrong per CTest docs, and filtered reruns silently skip cleanup. These are real correctness issues but do not break the primary CI workflow.

CMake/ITKModuleTest.cmake — the itk_add_bigio_test_cleanup function's fixture semantics and its in-code comment both need correction

Important Files Changed

Filename	Overview
CMake/ITKModuleTest.cmake	Adds ITK_REMOVE_BIGIO_FILES_ON_SUCCESS option and itk_add_bigio_test_cleanup() function; the function's CTest fixture semantics are documented incorrectly — cleanup runs unconditionally, not only on success, and filtered reruns won't auto-trigger cleanup
Modules/IO/ImageBase/test/CMakeLists.txt	Consolidates per-test RESOURCE_LOCK properties into a single set_tests_properties block, adds RUN_SERIAL and BigIO/RUNS_LONG labels, and registers cleanup hooks for three LargeImage tests
Modules/IO/Meta/test/CMakeLists.txt	Adds RUN_SERIAL to existing set_tests_properties block replacing old appended set_property calls, and registers cleanup hooks for all four LargeMetaImage tests
Modules/IO/TIFF/test/CMakeLists.txt	Same RUN_SERIAL consolidation pattern as IO/Meta; removes four separate set_property APPEND RUN_SERIAL blocks and registers TIFF cleanup hooks

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["ctest selects BigIO test\ne.g. itkLargeMetaImageWriteReadTest1"] --> B{"Full run or\nfiltered run?"}
    B -- "Full run\n(no -R filter)" --> C["Both main test AND _cleanup\ntest are in the set"]
    B -- "Filtered run\n(-R LargeMetaImage)" --> D["Only main test matches\nNo FIXTURES_REQUIRED present"]
    C --> E["CTest fixture ordering:\nSETUP runs before CLEANUP"]
    D --> F["_cleanup test NOT auto-added\nCleanup silently skipped"]
    E --> G{"Main test result?"}
    G -- "PASS" --> H["_cleanup runs\ncmake -E rm -f files"]
    G -- "FAIL" --> I["_cleanup STILL runs\n(CMake docs: cleanup is unconditional)\nFiles deleted despite failure"]
    H --> J["Disk space reclaimed ✓"]
    I --> K["Disk space reclaimed\nbut debug files lost\n(contradicts PR design intent)"]

Loading

_{Reviews (1): Last reviewed commit: "ENH: Auto-cleanup BigIO test outputs and..." | Re-trigger Greptile}

[blowekamp]

Please separate the change in properties from the addition of the clean up step. The current concept and behavior of the RESOURCE_LOCK may have been missed. If there is a test in here that is missing this lock, then it could be added in this PR.

I think these ctest properties are a blocker for converting tests to the google test framework. I don't know of any work around for that.

This looks like a nice usage of this ctest fixture feature. However I think "itk_add_bigio_test_cleanup" may be too specific, and it could be generalized to "itk_add_file_test_cleanup" to just delete listed files

[hjmjohnson]

Thanks @blowekamp — all three concerns addressed, branch force-pushed:

1. Separated property changes from cleanup fixture — now two commits:

   • STYLE: Consolidate BigIO test scheduling-property blocks — only the RESOURCE_LOCK/RUN_SERIAL/label consolidations, no cleanup calls
   • ENH: Add itk_add_file_test_cleanup for automatic test output cleanup — only the fixture mechanism and call sites

2. Removed newly-added RUN_SERIAL — dropped RUN_SERIAL True from IO/ImageBase (all three tests) and IO/Meta tests 1–3. These already have RESOURCE_LOCK MEMORY_SIZE which serializes them against each other while letting other tests run freely — exactly the behavior you described for nightly valgrind/coverage builds. RUN_SERIAL is retained only where it already existed before this PR (TIFF tests 1–4, Meta test4), consolidated into the main set_tests_properties block.

3. Renamed to itk_add_file_test_cleanup — function, CMake option (ITK_REMOVE_TEST_FILES_ON_SUCCESS), fixture prefix (Cleanup_), and label (CLEANUP) are all format-agnostic now.

[dzenanz]

A very welcome feature. I have a bunch of local builds, some of which are big due to temporary test outputs. Some inline requests.

[hjmjohnson]

FYI: Failing test for mac is fixed in #6015