PERF: Optimize Group C morphological filters for out-of-core

Optimizes all 5 Group C (Multi-Iteration Morphological / Neighbor
Replacement) filters for out-of-core performance using Z-slice rolling
buffers, bulk I/O via copyIntoBuffer/copyFromBuffer, and Z-slice
buffered tuple transfer.

Algorithm changes:
* Z-slice rolling buffer for neighbor lookups in all 5 filters — all
  6 face-neighbor reads come from RAM buffers instead of OOC stores
* SliceBufferedTransfer utility for type-dispatched Z-slice buffered
  tuple copy — eliminates per-element copyTuple OOC overhead
* Bulk I/O for all slice reads/writes using copyIntoBuffer/copyFromBuffer
* Cache CrystalStructures ensemble array locally in
  NeighborOrientationCorrelation to avoid per-element virtual dispatch
* Sequential per-array transfer ordering to prevent OOC cache thrashing
* Deferred transfer in ErodeDilateCoordinationNumber for bulk I/O

Bug fixes:
* Honor XDirOn/YDirOn/ZDirOn direction flags in ErodeDilateBadData
  and ErodeDilateMask — parameters were declared but never applied

Test changes:
* Parameterized BuildTestData builders for small (20^3) and large (200^3)
* All test helpers use Z-slice batched bulk reads for OOC efficiency
* Tier 2 tests use programmatic input with ForceOocAlgorithmGuard
* Tier 1 tests use single most-representative parameter combo
* PreferencesSentinel updated from "Zarr" to "HDF5-OOC"
* Retired legacy 6_6_ test archives in favor of programmatic data

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>

Author: joeykleingers

src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.cpp

@@ -10,34 +10,76 @@
 namespace nx::core
 {
 
+/**
+ * @struct NeighborOrientationCorrelationInputValues
+ * @brief Holds all user-supplied parameters for the NeighborOrientationCorrelation algorithm.
+ */
 struct ORIENTATIONANALYSIS_EXPORT NeighborOrientationCorrelationInputValues
 {
-  DataPath ImageGeomPath;
-  float32 MinConfidence;
-  float32 MisorientationTolerance;
-  int32 Level;
-  DataPath ConfidenceIndexArrayPath;
-  DataPath CellPhasesArrayPath;
-  DataPath QuatsArrayPath;
-  DataPath CrystalStructuresArrayPath;
-  MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths;
+  DataPath ImageGeomPath;                                        ///< Path to the ImageGeom that defines the voxel grid dimensions
+  float32 MinConfidence;                                         ///< Cells with confidence index below this value are candidates for replacement
+  float32 MisorientationTolerance;                               ///< Angular tolerance (degrees) for comparing neighbor orientations
+  int32 Level;                                                   ///< Minimum neighbor agreement count required to replace a cell (cleanup level)
+  DataPath ConfidenceIndexArrayPath;                             ///< Path to the float32 confidence index array
+  DataPath CellPhasesArrayPath;                                  ///< Path to the int32 cell phases array
+  DataPath QuatsArrayPath;                                       ///< Path to the float32 quaternion array (4 components per tuple)
+  DataPath CrystalStructuresArrayPath;                           ///< Path to the uint32 crystal structures ensemble array
+  MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths; ///< Data arrays excluded from the neighbor-copy transfer step
 };
 
 /**
- * @class
+ * @class NeighborOrientationCorrelation
+ * @brief Corrects low-confidence EBSD voxels by replacing their cell data with
+ * data from the most orientation-correlated face neighbor.
+ *
+ * The algorithm iterates through multiple "cleanup levels" (from 6 down to the
+ * user-specified Level). At each level, every voxel whose confidence index is
+ * below MinConfidence is examined. For that voxel, the 6 face neighbors are
+ * compared pairwise: two neighbors "agree" if they share the same nonzero phase
+ * and their misorientation is within MisorientationTolerance. Each neighbor
+ * accumulates a similarity count (how many other neighbors agree with it). The
+ * neighbor with the highest agreement is chosen as the replacement source.
+ *
+ * ## Z-Slice Buffering (Out-of-Core Optimization)
+ *
+ * To avoid random-access thrashing of out-of-core (OOC) compressed chunk stores,
+ * the algorithm maintains a rolling window of 3 adjacent Z-slices for the
+ * quaternion and phase arrays, plus 1 Z-slice for the confidence index. At each
+ * Z-step, the window advances by swapping buffer slots and reading only the new
+ * z+1 slice. All neighbor lookups then read from these local buffers instead of
+ * the backing DataArray, eliminating repeated chunk decompressions.
+ *
+ * After identifying the best neighbor for every low-confidence voxel in a level,
+ * all cell-level DataArrays (except ignored ones) are updated in parallel using
+ * ParallelTaskAlgorithm, copying tuple data from each best neighbor.
  */
 class ORIENTATIONANALYSIS_EXPORT NeighborOrientationCorrelation
 {
 public:
+  /**
+   * @brief Constructs the algorithm with all required references and parameters.
+   * @param dataStructure The DataStructure containing all input/output arrays
+   * @param mesgHandler Handler for sending progress messages to the UI
+   * @param shouldCancel Atomic flag checked between iterations to support cancellation
+   * @param inputValues User-supplied parameters controlling the algorithm behavior
+   */
   NeighborOrientationCorrelation(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel,
                                  NeighborOrientationCorrelationInputValues* inputValues);
+
+  /**
+   * @brief Default destructor.
+   */
   ~NeighborOrientationCorrelation() noexcept;
 
   NeighborOrientationCorrelation(const NeighborOrientationCorrelation&) = delete;
   NeighborOrientationCorrelation(NeighborOrientationCorrelation&&) noexcept = delete;
   NeighborOrientationCorrelation& operator=(const NeighborOrientationCorrelation&) = delete;
   NeighborOrientationCorrelation& operator=(NeighborOrientationCorrelation&&) noexcept = delete;
 
+  /**
+   * @brief Executes the neighbor orientation correlation algorithm.
+   * @return Result<> indicating success or any errors encountered during execution
+   */
   Result<> operator()();
 
 private:

src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.hpp

@@ -9,7 +9,21 @@ namespace nx::core
 {
 /**
  * @class NeighborOrientationCorrelationFilter
- * @brief This filter will ....
+ * @brief Cleans up EBSD data by replacing low-confidence voxels with data from
+ * the most orientation-correlated face neighbor.
+ *
+ * This filter identifies voxels whose confidence index falls below a
+ * user-specified threshold, then examines the 6 face neighbors of each such
+ * voxel. Neighbor pairs are compared using crystallographic misorientation;
+ * the neighbor that agrees most with the other neighbors (within a given
+ * angular tolerance) is selected as the replacement source. All cell-level
+ * DataArrays are updated to reflect the replacement. The process repeats
+ * across multiple cleanup levels, progressively relaxing the required neighbor
+ * agreement count.
+ *
+ * The underlying algorithm uses Z-slice buffering to maintain efficient
+ * sequential access patterns, which is critical for out-of-core (OOC) data
+ * where arrays are stored as compressed Zarr chunks on disk.
  */
 class ORIENTATIONANALYSIS_EXPORT NeighborOrientationCorrelationFilter : public IFilter
 {

src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/NeighborOrientationCorrelationFilter.hpp

@@ -3,14 +3,20 @@
 #include "OrientationAnalysisTestUtils.hpp"
 
 #include "simplnx/Core/Application.hpp"
+#include "simplnx/DataStructure/AttributeMatrix.hpp"
+#include "simplnx/DataStructure/Geometry/ImageGeom.hpp"
 #include "simplnx/Parameters/ArraySelectionParameter.hpp"
 #include "simplnx/Parameters/BoolParameter.hpp"
 #include "simplnx/Parameters/Dream3dImportParameter.hpp"
 #include "simplnx/Parameters/GeometrySelectionParameter.hpp"
+#include "simplnx/Parameters/MultiArraySelectionParameter.hpp"
 #include "simplnx/UnitTest/UnitTestCommon.hpp"
+#include "simplnx/Utilities/AlgorithmDispatch.hpp"
+#include "simplnx/Utilities/DataStoreUtilities.hpp"
 
 #include <catch2/catch.hpp>
 
+#include <cmath>
 #include <filesystem>
 
 namespace fs = std::filesystem;
@@ -38,6 +44,8 @@ using namespace nx::core::UnitTest;
 TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Small IN100 Pipeline", "[OrientationAnalysis][NeighborOrientationCorrelationFilter]")
 {
   UnitTest::LoadPlugins();
+  // 1 Z-slice of quats (largest array): 189*201*4*4 = 607824 bytes
+  const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true);
 
   const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "neighbor_orientation_correlation.tar.gz", "neighbor_orientation_correlation.dream3d");
 
@@ -194,3 +202,165 @@ TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Small IN10
 
   UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths);
 }
+
+namespace
+{
+const std::string k_GeomName("Image Geometry");
+const std::string k_CellDataName("Cell Data");
+
+const DataPath k_GeomPath({k_GeomName});
+const DataPath k_CellDataPath = k_GeomPath.createChildPath(k_CellDataName);
+const DataPath k_CIPath = k_CellDataPath.createChildPath("Confidence Index");
+const DataPath k_QuatsPath = k_CellDataPath.createChildPath("Quats");
+const DataPath k_PhasesPath = k_CellDataPath.createChildPath("Phases");
+const DataPath k_CrystalStructuresPath = k_GeomPath.createChildPath("Ensemble Data").createChildPath("CrystalStructures");
+
+void BuildTestData(DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ, usize blockSize)
+{
+  const ShapeType cellTupleShape = {dimZ, dimY, dimX};
+  const usize sliceSize = dimX * dimY;
+
+  auto* imageGeom = ImageGeom::Create(dataStructure, k_GeomName);
+  imageGeom->setDimensions({dimX, dimY, dimZ});
+  imageGeom->setSpacing({1.0f, 1.0f, 1.0f});
+  imageGeom->setOrigin({0.0f, 0.0f, 0.0f});
+
+  auto* cellAM = AttributeMatrix::Create(dataStructure, k_CellDataName, cellTupleShape, imageGeom->getId());
+  imageGeom->setCellData(*cellAM);
+
+  auto quatsDataStore = DataStoreUtilities::CreateDataStore<float32>(cellTupleShape, {4}, IDataAction::Mode::Execute);
+  auto* quatsArray = DataArray<float32>::Create(dataStructure, "Quats", quatsDataStore, cellAM->getId());
+  auto& quatsStore = quatsArray->getDataStoreRef();
+
+  auto phasesDataStore = DataStoreUtilities::CreateDataStore<int32>(cellTupleShape, {1}, IDataAction::Mode::Execute);
+  auto* phasesArray = DataArray<int32>::Create(dataStructure, "Phases", phasesDataStore, cellAM->getId());
+  auto& phasesStore = phasesArray->getDataStoreRef();
+
+  auto ciDataStore = DataStoreUtilities::CreateDataStore<float32>(cellTupleShape, {1}, IDataAction::Mode::Execute);
+  auto* ciArray = DataArray<float32>::Create(dataStructure, "Confidence Index", ciDataStore, cellAM->getId());
+  auto& ciStore = ciArray->getDataStoreRef();
+
+  const usize blocksPerDimX = dimX / blockSize;
+  const usize blocksPerDimY = dimY / blockSize;
+
+  std::vector<float32> quatsBuf(sliceSize * 4);
+  std::vector<int32> phasesBuf(sliceSize);
+  std::vector<float32> ciBuf(sliceSize);
+
+  for(usize z = 0; z < dimZ; z++)
+  {
+    for(usize y = 0; y < dimY; y++)
+    {
+      for(usize x = 0; x < dimX; x++)
+      {
+        const usize inSlice = y * dimX + x;
+        phasesBuf[inSlice] = 1;
+
+        usize bx = x / blockSize;
+        usize by = y / blockSize;
+        usize bz = z / blockSize;
+        float32 angle = static_cast<float32>(bz * blocksPerDimY * blocksPerDimX + by * blocksPerDimX + bx) * 0.1f;
+        float32 sinHalf = std::sin(angle * 0.5f);
+        float32 cosHalf = std::cos(angle * 0.5f);
+
+        const usize qIdx = inSlice * 4;
+        quatsBuf[qIdx] = cosHalf;
+        quatsBuf[qIdx + 1] = sinHalf * 0.577350269f; // 1/sqrt(3)
+        quatsBuf[qIdx + 2] = sinHalf * 0.577350269f;
+        quatsBuf[qIdx + 3] = sinHalf * 0.577350269f;
+
+        bool isBoundary = (x % blockSize == 0) || (y % blockSize == 0) || (z % blockSize == 0);
+        bool isNoisy = ((x * 7 + y * 13 + z * 29) % 10 == 0);
+        ciBuf[inSlice] = (isBoundary || isNoisy) ? 0.05f : 0.9f;
+      }
+    }
+    const usize zOffset = z * sliceSize;
+    quatsStore.copyFromBuffer(zOffset * 4, nonstd::span<const float32>(quatsBuf.data(), sliceSize * 4));
+    phasesStore.copyFromBuffer(zOffset, nonstd::span<const int32>(phasesBuf.data(), sliceSize));
+    ciStore.copyFromBuffer(zOffset, nonstd::span<const float32>(ciBuf.data(), sliceSize));
+  }
+
+  // Ensemble data — small enough for per-element writes
+  auto* ensembleAM = AttributeMatrix::Create(dataStructure, "Ensemble Data", {2}, imageGeom->getId());
+  auto crystalStructuresDataStore = DataStoreUtilities::CreateDataStore<uint32>({2}, {1}, IDataAction::Mode::Execute);
+  auto* crystalStructuresArray = DataArray<uint32>::Create(dataStructure, "CrystalStructures", crystalStructuresDataStore, ensembleAM->getId());
+  std::array<uint32, 2> csData = {999, 1}; // Unknown, Cubic-High (m-3m)
+  crystalStructuresArray->getDataStoreRef().copyFromBuffer(0, nonstd::span<const uint32>(csData.data(), 2));
+}
+} // namespace
+
+TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Generate Test Data", "[OrientationAnalysis][NeighborOrientationCorrelationFilter][.GenerateTestData]")
+{
+  const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data" / "neighbor_orientation_correlation";
+  fs::create_directories(outputDir);
+
+  // Large input data (200x200x200, blockSize=25)
+  {
+    DataStructure buildDS;
+    BuildTestData(buildDS, 200, 200, 200, 25);
+    UnitTest::WriteTestDataStructure(buildDS, outputDir / "large_input.dream3d");
+    fmt::print("Generated large input: {}\n", (outputDir / "large_input.dream3d").string());
+  }
+}
+
+TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: 200x200x200 Large OOC", "[OrientationAnalysis][NeighborOrientationCorrelationFilter]")
+{
+  UnitTest::LoadPlugins();
+  bool forceOocAlgo = GENERATE(false, true);
+  const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo);
+  // 200x200x200, Quats (float32, 4-comp) => 200*200*4*4 = 640,000 bytes/slice
+  const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 640000, true);
+
+  DYNAMIC_SECTION("forceOoc: " << forceOocAlgo)
+  {
+    constexpr usize k_Dim = 200;
+    constexpr usize k_Block = 25;
+
+    DataStructure dataStructure;
+    BuildTestData(dataStructure, k_Dim, k_Dim, k_Dim, k_Block);
+
+    const NeighborOrientationCorrelationFilter filter;
+    Arguments args;
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_ImageGeometryPath_Key, std::make_any<DataPath>(k_GeomPath));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_MinConfidence_Key, std::make_any<float32>(0.2f));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_MisorientationTolerance_Key, std::make_any<float32>(5.0f));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_Level_Key, std::make_any<int32>(2));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_CorrelationArrayPath_Key, std::make_any<DataPath>(k_CIPath));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_CellPhasesArrayPath_Key, std::make_any<DataPath>(k_PhasesPath));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_QuatsArrayPath_Key, std::make_any<DataPath>(k_QuatsPath));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_CrystalStructuresArrayPath_Key, std::make_any<DataPath>(k_CrystalStructuresPath));
+    args.insertOrAssign(NeighborOrientationCorrelationFilter::k_IgnoredDataArrayPaths_Key, std::make_any<MultiArraySelectionParameter::ValueType>(MultiArraySelectionParameter::ValueType{}));
+
+    auto preflightResult = filter.preflight(dataStructure, args);
+    SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions);
+
+    auto executeResult = filter.execute(dataStructure, args);
+    SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result);
+
+    // Some low-CI voxels should have been modified — use Z-slice batched reads for OOC efficiency
+    const auto& ciAfter = dataStructure.getDataRefAs<Float32Array>(k_CIPath).getDataStoreRef();
+    const usize sliceSize = k_Dim * k_Dim;
+    std::vector<float32> ciBuf(sliceSize);
+    usize modifiedCount = 0;
+    for(usize z = 0; z < k_Dim; z++)
+    {
+      ciAfter.copyIntoBuffer(z * sliceSize, nonstd::span<float32>(ciBuf.data(), sliceSize));
+      for(usize y = 0; y < k_Dim; y++)
+      {
+        for(usize x = 0; x < k_Dim; x++)
+        {
+          const usize inSlice = y * k_Dim + x;
+          bool wasBoundary = (x % k_Block == 0) || (y % k_Block == 0) || (z % k_Block == 0);
+          bool wasNoisy = ((x * 7 + y * 13 + z * 29) % 10 == 0);
+          if((wasBoundary || wasNoisy) && ciBuf[inSlice] != 0.05f)
+          {
+            modifiedCount++;
+          }
+        }
+      }
+    }
+    REQUIRE(modifiedCount > 0);
+
+    UnitTest::CheckArraysInheritTupleDims(dataStructure);
+  }
+}