Backport non-CI PRs (#1759): #1744 #1732 #1722 #1733 #1738 #1740 #1749

* Fix: Late unique_ptr puts without CLOSE_FILE or ADVANCE operations (#1744)

* Add failing test

* Add failing test

* Revert "Add failing test"

This reverts commit 5e04ece.

* Reactivate writing from unique_ptr in finalize()

* BP5+groupbased: allow only up to 100 steps (#1732)

* BP5+groupbased: allow only up to 1000 steps

* Configure this via env variable

OPENPMD_BP5_GROUPENCODING_MAX_STEPS=1000

* Add documentation

* Lower limit to 100

* Add compile-time check for #1720 (#1722)

* WarpX: Repo Moved (#1733)

Update a link to WarpX.

* Fix zero-sized storeChunk for Span API in Python (#1738)

* working around an unusual encounter when the joined_dim has actual value "max::size_t - 1" (#1740)

* working around an unusal encounter when the joined_dim has actual
value "max::size_t - 1"

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add test for redundant resetDataset()

* Merge check into above logic

* Better error messages in verifyDataset

* Add further safety guards to createDataset and extendDataset tasks

* Move joinedDim logic into middle-end for extendDataset

* Update include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

* ADIOS2 bugfix: Always use CurrentStep() in mode::Read (#1749)

* Always use CurrentStep() in mode::Read

* Remove manual step counting

m_currentStep only necessary for SetStepSelection, it seems

* Clean up logic that is no longer needed

* Add test

---------

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Co-authored-by: Junmin Gu <guj@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: 4 people

docs/source/analysis/paraview.rst

@@ -44,7 +44,7 @@ The file contains the same string as one would put in an openPMD ``Series("...."
 .. warning::
 
    As of ParaView 5.11 and older, the axisLabel is not yet read for fields.
-   See, e.g., `WarpX issue 21162 <https://github.com/ECP-WarpX/WarpX/issues/1803>`__.
+   See, e.g., `WarpX issue 21162 <https://github.com/BLAST-WarpX/warpx/issues/1803>`__.
    Please apply rotation of, e.g., ``0 -90 0`` to mesh data where needed.
 
 .. warning::

docs/source/backends/adios2.rst

@@ -78,23 +78,24 @@ The ADIOS2 SST engine for streaming can be picked by specifying the ending ``.ss
 The following environment variables control ADIOS2 I/O behavior at runtime.
 Fine-tuning these is especially useful when running at large scale.
 
-===================================== ========== ================================================================================
-environment variable                  default    description
-===================================== ========== ================================================================================
-``OPENPMD_ADIOS2_HAVE_PROFILING``     ``1``      Turns on/off profiling information right after a run.
-``OPENPMD_ADIOS2_HAVE_METADATA_FILE`` ``1``      Online creation of the adios journal file (``1``: yes, ``0``: no).
-``OPENPMD_ADIOS2_NUM_SUBSTREAMS``     ``0``      Number of files to be created, 0 indicates maximum number possible.
-``OPENPMD_ADIOS2_ENGINE``             ``File``   `ADIOS2 engine <https://adios2.readthedocs.io/en/latest/engines/engines.html>`_
-``OPENPMD_ADIOS2_PRETEND_ENGINE``     *empty*    Pretend that an (unknown) ADIOS2 engine is in fact another one (also see the ``adios2.pretend_engine`` :ref:`parameter <backendconfig-adios2>`).
-``OPENPMD2_ADIOS2_USE_GROUP_TABLE``   ``0``      Use group table (see below)
-``OPENPMD_ADIOS2_STATS_LEVEL``        ``0``      whether to generate statistics for variables in ADIOS2. (``1``: yes, ``0``: no).
-``OPENPMD_ADIOS2_ASYNC_WRITE``        ``0``      ADIOS2 BP5 engine: 1 means setting "AsyncWrite" in ADIOS2 to "on". Flushes will go to the buffer by default (see ``preferred_flush_target``).
-``OPENPMD_ADIOS2_BP5_BufferChunkMB``  ``0``      ADIOS2 BP5 engine: applies when using either EveryoneWrites or EveryoneWritesSerial aggregation
-``OPENPMD_ADIOS2_BP5_MaxShmMB``       ``0``      ADIOS2 BP5 engine: applies when using TwoLevelShm aggregation
-``OPENPMD_ADIOS2_BP5_NumSubFiles``    ``0``      ADIOS2 BP5 engine: num of subfiles
-``OPENPMD_ADIOS2_BP5_NumAgg``         ``0``      ADIOS2 BP5 engine: num of aggregators
-``OPENPMD_ADIOS2_BP5_TypeAgg``        *empty*    ADIOS2 BP5 engine: aggregation type. (EveryoneWrites, EveryoneWritesSerial, TwoLevelShm)
-===================================== ========== ================================================================================
+======================================= ========== ================================================================================
+environment variable                    default    description
+======================================= ========== ================================================================================
+``OPENPMD_ADIOS2_HAVE_PROFILING``       ``1``      Turns on/off profiling information right after a run.
+``OPENPMD_ADIOS2_HAVE_METADATA_FILE``   ``1``      Online creation of the adios journal file (``1``: yes, ``0``: no).
+``OPENPMD_ADIOS2_NUM_SUBSTREAMS``       ``0``      Number of files to be created, 0 indicates maximum number possible.
+``OPENPMD_ADIOS2_ENGINE``               ``File``   `ADIOS2 engine <https://adios2.readthedocs.io/en/latest/engines/engines.html>`_
+``OPENPMD_ADIOS2_PRETEND_ENGINE``       *empty*    Pretend that an (unknown) ADIOS2 engine is in fact another one (also see the ``adios2.pretend_engine`` :ref:`parameter <backendconfig-adios2>`).
+``OPENPMD2_ADIOS2_USE_GROUP_TABLE``     ``0``      Use group table (see below)
+``OPENPMD_ADIOS2_STATS_LEVEL``          ``0``      whether to generate statistics for variables in ADIOS2. (``1``: yes, ``0``: no).
+``OPENPMD_ADIOS2_ASYNC_WRITE``          ``0``      ADIOS2 BP5 engine: 1 means setting "AsyncWrite" in ADIOS2 to "on". Flushes will go to the buffer by default (see ``preferred_flush_target``).
+``OPENPMD_ADIOS2_BP5_BufferChunkMB``    ``0``      ADIOS2 BP5 engine: applies when using either EveryoneWrites or EveryoneWritesSerial aggregation
+``OPENPMD_ADIOS2_BP5_MaxShmMB``         ``0``      ADIOS2 BP5 engine: applies when using TwoLevelShm aggregation
+``OPENPMD_ADIOS2_BP5_NumSubFiles``      ``0``      ADIOS2 BP5 engine: num of subfiles
+``OPENPMD_ADIOS2_BP5_NumAgg``           ``0``      ADIOS2 BP5 engine: num of aggregators
+``OPENPMD_ADIOS2_BP5_TypeAgg``          *empty*    ADIOS2 BP5 engine: aggregation type. (EveryoneWrites, EveryoneWritesSerial, TwoLevelShm)
+``OPENPMD_BP5_GROUPENCODING_MAX_STEPS`` ``100``    ADIOS2 BP5 engine: max number of allowed output steps in group encoding.
+======================================= ========== ================================================================================
 
 Please refer to the `ADIOS2 documentation <https://adios2.readthedocs.io/en/latest/engines/engines.html>`_ for details on I/O tuning.
 
@@ -315,6 +316,21 @@ Rather than by reallocation as in BP4, this is done by appending a new chunk, le
 The default is to flush to disk (except when specifying ``OPENPMD_ADIOS2_ASYNC_WRITE=1``), but the default ``preferred_flush_target`` can also be specified via JSON/TOML at the ``Series`` level.
 
 
+The BP5 engine is known to perform extremely bad for group-based encoding with many Iterations, since its design assumes that the metadata structure will be constant across output steps, while group-based encoding will add new variables and attributes for each Iteration.
+The openPMD-api will hence cancel operation after 100 written Iterations in group-based encoding for BP5.
+Experiments with PIConGPU show that the metadata (!) size grows from 10MB to 1GB when going from 100 to 1000 output steps in this setup.
+The environment variable ``OPENPMD_BP5_GROUPENCODING_MAX_STEPS`` may be used to change this limit (specifying the limit as ``0`` will disable the check).
+
+For workarounds you may follow these guidelines:
+
+* Use file encoding by including an expansion pattern ``%T`` in the filename.
+* If output to a single file is required, then:
+
+  * Use another ADIOS2 engine, recommended is the BP4 engine by selecting file ending ``.bp4``.
+  * Use another openPMD backend, recommended is HDF5 by selecting file ending ``.h5``.
+  * (experimental) use variable encoding with BP5, either by using the API call ``Series::setIterationEncoding(IterationEncoding::variableBased)`` / ``Series.iteration_encoding = Iteration_Encoding.variable_based`` or by using the JSON/TOML configuration ``{"iteration_encoding": "variable_based"}`` / ``iteration_encoding = "variable_based"``.
+    Note that there is at this point no complete read support for variable-encoded outputs.
+
 
 
 Known Issues

examples/10_streaming_write.cpp

@@ -20,12 +20,16 @@ int main()
     }
 
     // open file for writing
+    // use QueueFullPolicy = Discard in order to create a situation where from
+    // the reader's perspective steps are skipped. This tests the bug reported
+    // in https://github.com/openPMD/openPMD-api/issues/1747.
     Series series = Series("electrons.sst", Access::CREATE, R"(
 {
   "adios2": {
     "engine": {
       "parameters": {
-        "DataTransport": "WAN"
+        "DataTransport": "WAN",
+        "QueueFullPolicy": "Discard"
       }
     }
   }

include/openPMD/Dataset.hpp

@@ -64,5 +64,6 @@ class Dataset
     bool empty() const;
 
     std::optional<size_t> joinedDimension() const;
+    static std::optional<size_t> joinedDimension(Extent const &);
 };
 } // namespace openPMD

include/openPMD/IO/ADIOS/ADIOS2File.hpp

@@ -25,6 +25,7 @@
 #include "openPMD/IO/IOTask.hpp"
 #include "openPMD/IO/InvalidatableFile.hpp"
 #include "openPMD/config.hpp"
+#include <optional>
 
 #if openPMD_HAVE_ADIOS2
 #include <adios2.h>
@@ -414,11 +415,7 @@ class ADIOS2File
     ADIOS2IOHandlerImpl *m_impl;
     std::optional<adios2::Engine> m_engine; //! ADIOS engine
 
-    /*
-     * Not all engines support the CurrentStep() call, so we have to
-     * implement this manually.
-     */
-    size_t m_currentStep = 0;
+    std::optional<size_t> m_max_steps_bp5 = std::make_optional<size_t>(100);
 
     /*
      * ADIOS2 does not give direct access to its internal attribute and

include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp

@@ -32,6 +32,7 @@
 #include "openPMD/IterationEncoding.hpp"
 #include "openPMD/ThrowError.hpp"
 #include "openPMD/auxiliary/JSON_internal.hpp"
+#include "openPMD/auxiliary/StringManip.hpp"
 #include "openPMD/backend/Writable.hpp"
 #include "openPMD/config.hpp"
 #include <stdexcept>
@@ -473,20 +474,44 @@ class ADIOS2IOHandlerImpl
             }
         }
         auto joinedDim = joinedDimension(shape);
-        if (joinedDim.has_value())
+        auto make_runtime_error = [&](char const *message) {
+            std::stringstream s;
+            s << "[ADIOS2IOHandlerImpl::verifyDataset()] " << message;
+            s << "\nNote: Variable '" << varName << "' has shape ";
+            auxiliary::write_vec_to_stream(s, shape)
+                << ", is accessed from offset ";
+            auxiliary::write_vec_to_stream(s, offset) << " with extent ";
+            auxiliary::write_vec_to_stream(s, extent);
+            if (joinedDim.has_value())
+            {
+                s << " (joined dimension on index " << *joinedDim << ").";
+            }
+            else
+            {
+                s << " (no joined dimension).";
+            }
+            return std::runtime_error(s.str());
+        };
+        if (joinedDim.has_value() ||
+            var.ShapeID() == adios2::ShapeID::JoinedArray)
         {
             if (!offset.empty())
             {
-                throw std::runtime_error(
-                    "[ADIOS2] Offset must be an empty vector in case of joined "
-                    "array.");
+                throw make_runtime_error(
+                    "Offset must be an empty vector in case of joined array.");
+            }
+            if (!joinedDim.has_value())
+            {
+                throw make_runtime_error(
+                    "Trying to access a dataset as a non-joined array, but it "
+                    "has previously been array.");
             }
             for (unsigned int i = 0; i < actualDim; i++)
             {
                 if (*joinedDim != i && extent[i] != shape[i])
                 {
-                    throw std::runtime_error(
-                        "[ADIOS2] store_chunk extent of non-joined dimensions "
+                    throw make_runtime_error(
+                        "store_chunk extent of non-joined dimensions "
                         "must be equivalent to the total extent.");
                 }
             }
@@ -498,8 +523,7 @@ class ADIOS2IOHandlerImpl
                 if (!(joinedDim.has_value() && *joinedDim == i) &&
                     offset[i] + extent[i] > shape[i])
                 {
-                    throw std::runtime_error(
-                        "[ADIOS2] Dataset access out of bounds.");
+                    throw make_runtime_error("Dataset access out of bounds.");
                 }
             }
         }

include/openPMD/IO/IOTask.hpp

@@ -98,6 +98,13 @@ struct OPENPMDAPI_EXPORT AbstractParameter
 
     virtual std::unique_ptr<AbstractParameter> to_heap() && = 0;
 
+    // Used as a tag in some constructors to make callsites explicitly
+    // acknowledge that joined dimensions will not be automatically resolved
+    struct I_dont_want_to_use_joined_dimensions_t
+    {};
+    constexpr static I_dont_want_to_use_joined_dimensions_t
+        I_dont_want_to_use_joined_dimensions{};
+
 protected:
     // avoid object slicing
     // by allow only child classes to use these things for defining their own
@@ -327,7 +334,17 @@ template <>
 struct OPENPMDAPI_EXPORT Parameter<Operation::CREATE_DATASET>
     : public AbstractParameter
 {
-    Parameter() = default;
+    Parameter(Dataset const &ds)
+        : extent(ds.extent)
+        , dtype(ds.dtype)
+        , options(ds.options)
+        , joinedDimension(ds.joinedDimension())
+    {}
+
+    // default constructor, but callsites need to explicitly acknowledge that
+    // joined dimensions will not be automatically configured when using it
+    Parameter(I_dont_want_to_use_joined_dimensions_t)
+    {}
     Parameter(Parameter &&) = default;
     Parameter(Parameter const &) = default;
     Parameter &operator=(Parameter &&) = default;
@@ -362,7 +379,15 @@ template <>
 struct OPENPMDAPI_EXPORT Parameter<Operation::EXTEND_DATASET>
     : public AbstractParameter
 {
-    Parameter() = default;
+    Parameter(Extent e) : joinedDimension(Dataset::joinedDimension(e))
+    {
+        this->extent = std::move(e);
+    }
+
+    // default constructor, but callsites need to explicitly acknowledge that
+    // joined dimensions will not be automatically configured when using it
+    Parameter(I_dont_want_to_use_joined_dimensions_t)
+    {}
     Parameter(Parameter &&) = default;
     Parameter(Parameter const &) = default;
     Parameter &operator=(Parameter &&) = default;
@@ -375,6 +400,7 @@ struct OPENPMDAPI_EXPORT Parameter<Operation::EXTEND_DATASET>
     }
 
     Extent extent = {};
+    std::optional<size_t> joinedDimension;
 };
 
 template <>

include/openPMD/RecordComponent.tpp

@@ -309,18 +309,14 @@ RecordComponent::storeChunk(Offset o, Extent e, F &&createBuffer)
     if (!written())
     {
         auto &rc = get();
-        Parameter<Operation::CREATE_DATASET> dCreate;
-        dCreate.name = rc.m_name;
-        dCreate.extent = getExtent();
-        dCreate.dtype = getDatatype();
-        dCreate.joinedDimension = joinedDimension();
         if (!rc.m_dataset.has_value())
         {
             throw error::WrongAPIUsage(
                 "[RecordComponent] Must specify dataset type and extent before "
                 "using storeChunk() (see RecordComponent::resetDataset()).");
         }
-        dCreate.options = rc.m_dataset.value().options;
+        Parameter<Operation::CREATE_DATASET> dCreate(rc.m_dataset.value());
+        dCreate.name = rc.m_name;
         IOHandler()->enqueue(IOTask(this, dCreate));
     }
     Parameter<Operation::GET_BUFFER_VIEW> getBufferView;

include/openPMD/auxiliary/StringManip.hpp

@@ -242,5 +242,40 @@ namespace auxiliary
         });
         return std::forward<S>(s);
     }
+
+    template <typename Stream, typename Vec>
+    auto write_vec_to_stream(Stream &&s, Vec const &vec) -> Stream &&
+    {
+        if (vec.empty())
+        {
+            s << "[]";
+        }
+        else
+        {
+            s << '[';
+            auto it = vec.begin();
+            s << *it++;
+            auto end = vec.end();
+            for (; it != end; ++it)
+            {
+                s << ", " << *it;
+            }
+            s << ']';
+        }
+        return std::forward<Stream>(s);
+    }
+
+    template <typename Vec>
+    auto vec_as_string(Vec const &vec) -> std::string
+    {
+        if (vec.empty())
+        {
+            return "[]";
+        }
+        else
+        {
+            return write_vec_to_stream(std::stringstream(), vec).str();
+        }
+    }
 } // namespace auxiliary
 } // namespace openPMD

include/openPMD/openPMD.hpp

@@ -25,6 +25,21 @@
 namespace openPMD
 {}
 
+#if defined(CUDA_VERSION) && !defined(OPENPMD_SKIP_CHECK_ISSUE_1720)
+static_assert(__cplusplus < 202002L || CUDA_VERSION >= 12040, R"(
+Cannot use the openPMD-api in C++20 projects under a Cuda version lower
+than 12.4.0 due to a bug in the implementation of std::variant.
+Further information at:
+https://github.com/openPMD/openPMD-api/issues/1720
+https://forums.developer.nvidia.com/t/nvcc-c-20-std-variant-complie-failed/270162/5
+This cannot be fixed on our side, please either upgrade to CUDA >= 12.4.0
+or use C++17.
+If you think that this assertion is shown wrongly, please apply
+'#define OPENPMD_SKIP_CHECK_ISSUE_1720' before including
+'<openPMD/openPMD.hpp>'.
+)");
+#endif
+
 // IWYU pragma: begin_exports
 #include "openPMD/Dataset.hpp"
 #include "openPMD/Datatype.hpp"