HIP build fails: `UniquePtr.cpp:68:25: error: reference to __host__ function '~unique_ptr' in __host__ __device__ function`

[pgrete]

Inspired by the warpx environment instructions provided in #1792 (comment)

Performed steps
The following steps

Module env

# required dependencies
module load cmake/3.30.5
module load craype-accel-amd-gfx90a
module load rocm/6.2.4
module load cray-mpich/8.1.31
module load cce/18.0.1  # must be loaded after rocm
# https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#compatible-compiler-rocm-toolchain-versions

# Fix for OpenMP Runtime (OLCFHELP-21543)
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${ROCM_PATH}/llvm/lib

module load ninja

# optional: for openPMD support
module load adios2/2.10.2-mpi
module load hdf5/1.14.3-mpi

# optional: for Python bindings or libEnsemble
module load cray-python/3.11.5

# make output group-readable by default
umask 0027

# GPU-aware MPI
export MPICH_GPU_SUPPORT_ENABLED=1

# compiler environment hints
export CC=$(which hipcc)
export CXX=$(which hipcc)
export FC=$(which ftn)
export CFLAGS="-I${ROCM_PATH}/include"
export CXXFLAGS="-I${ROCM_PATH}/include -Wno-pass-failed"
export LDFLAGS="-L${ROCM_PATH}/lib -lamdhip64 ${PE_MPICH_GTL_DIR_amd_gfx90a} -lmpi_gtl_hsa"

Python env inst.:

#!/bin/bash
#
# Copyright 2023 The WarpX Community
#
# This file is part of WarpX.
#
# Author: Axel Huebl
# License: BSD-3-Clause-LBNL

# Exit on first error encountered #############################################
#
set -eu -o pipefail



# remove common user mistakes in python, located in .local instead of a venv
python3 -m pip uninstall -qq -y pywarpx
python3 -m pip uninstall -qq -y warpx
python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true


# Python ######################################################################
#
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade virtualenv
python3 -m pip cache purge
rm -rf /ccs/proj/ast146/proj-shared/software/py-oct25-hip
python3 -m venv /ccs/proj/ast146/proj-shared/software/py-oct25-hip
source /ccs/proj/ast146/proj-shared/software/py-oct25-hip/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade build
python3 -m pip install --upgrade packaging
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade setuptools[core]
python3 -m pip install --upgrade "cython>=3.0"
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade h5py
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
MPICC="cc -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py
#python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade yt
python3 -m pip install --upgrade unyt

OpenPMD config

$ cmake -Bbuild-oct25-hip-21adf5a -DCMAKE_INSTALL_PREFIX=/ccs/proj/ast146/proj-shared/openpmd-api/oct25-hip-21adf5a -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE=$(which python3)

-- The C compiler identification is Clang 18.0.0
-- The CXX compiler identification is Clang 18.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rocm-6.2.4/bin/hipcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rocm-6.2.4/bin/hipcc - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- openPMD-api superbuild: ON
-- Found MPI_CXX: /opt/cray/pe/mpich/8.1.31/ofi/cray/17.0/lib/libmpi_cray.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: CXX
-- Downloading nlohmann_json ...
-- nlohmann_json source: https://github.com/nlohmann/json/archive/refs/tags/v3.11.3.tar.gz
-- Using the multi-header code from /ccs/proj/ast146/pgrete/src/openPMD-api/build-oct25-hip-21adf5a/_deps/fetchednlohmann_json-src/include/
-- Downloading toml11 ...
-- toml11 source: https://github.com/ToruNiina/toml11/archive/refs/tags/v4.2.0.tar.gz
CMake Deprecation Warning at build-oct25-hip-21adf5a/_deps/fetchedtoml11-src/CMakeLists.txt:23 (cmake_policy):
  The OLD behavior for policy CMP0127 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


-- Performing Test TOML11_COMPILER_SUPPORTS_WALL
-- Performing Test TOML11_COMPILER_SUPPORTS_WALL - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WEXTRA
-- Performing Test TOML11_COMPILER_SUPPORTS_WEXTRA - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WPEDANTIC
-- Performing Test TOML11_COMPILER_SUPPORTS_WPEDANTIC - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WERROR
-- Performing Test TOML11_COMPILER_SUPPORTS_WERROR - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WSIGN_CONVERSION
-- Performing Test TOML11_COMPILER_SUPPORTS_WSIGN_CONVERSION - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WCONVERSION
-- Performing Test TOML11_COMPILER_SUPPORTS_WCONVERSION - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WDUPLICATED_COND
-- Performing Test TOML11_COMPILER_SUPPORTS_WDUPLICATED_COND - Failed
-- Performing Test TOML11_COMPILER_SUPPORTS_WDUPLICATED_BRANCHES
-- Performing Test TOML11_COMPILER_SUPPORTS_WDUPLICATED_BRANCHES - Failed
-- Performing Test TOML11_COMPILER_SUPPORTS_WLOGICAL_OP
-- Performing Test TOML11_COMPILER_SUPPORTS_WLOGICAL_OP - Failed
-- Performing Test TOML11_COMPILER_SUPPORTS_WDOUBLE_PROMOTION
-- Performing Test TOML11_COMPILER_SUPPORTS_WDOUBLE_PROMOTION - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WRANGE_LOOP_ANALYSIS
-- Performing Test TOML11_COMPILER_SUPPORTS_WRANGE_LOOP_ANALYSIS - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WUNDEF
-- Performing Test TOML11_COMPILER_SUPPORTS_WUNDEF - Success
-- Performing Test TOML11_COMPILER_SUPPORTS_WSHADOW
-- Performing Test TOML11_COMPILER_SUPPORTS_WSHADOW - Success
-- Found HDF5: hdf5-shared (found version "1.14.3") found components: C
-- Found MPI_C: /opt/cray/pe/mpich/8.1.31/ofi/cray/17.0/lib/libmpi_cray.so (found version "3.1")
CMake Warning (dev) at /autofs/nccs-svm1_sw/frontier/spack-envs/core-25.03/opt/gcc-13.2/cmake-3.30.5-3megaha56irdlal5or2bjkasn2zj5fvp/share/cmake-3.30/Modules/FindMPI.cmake:1481 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'MPI_CXX_SKIP_MPICXX'.
Call Stack (most recent call first):
  /sw/frontier/spack-envs/cpe24.11-cpu/opt/cce-18.0.1/adios2-2.10.2-ssnfnmbbyipk7jo25xhfg457x5kl5hke/lib64/cmake/adios2/FindMPI.cmake:10 (include)
  /autofs/nccs-svm1_sw/frontier/spack-envs/core-25.03/opt/gcc-13.2/cmake-3.30.5-3megaha56irdlal5or2bjkasn2zj5fvp/share/cmake-3.30/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  /sw/frontier/spack-envs/cpe24.11-cpu/opt/cce-18.0.1/adios2-2.10.2-ssnfnmbbyipk7jo25xhfg457x5kl5hke/lib64/cmake/adios2/adios2-config-common.cmake:77 (find_dependency)
  /sw/frontier/spack-envs/cpe24.11-cpu/opt/cce-18.0.1/adios2-2.10.2-ssnfnmbbyipk7jo25xhfg457x5kl5hke/lib64/cmake/adios2/adios2-config.cmake:29 (include)
  CMakeLists.txt:371 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found MPI: TRUE (found version "3.1")
-- Found ADIOS2: /sw/frontier/spack-envs/cpe24.11-cpu/opt/cce-18.0.1/adios2-2.10.2-ssnfnmbbyipk7jo25xhfg457x5kl5hke/lib64/cmake/adios2/adios2-config.cmake (found suitable version "2.10.2", minimum required is "2.9.0") found components: CXX MPI
-- Found Python: /ccs/proj/ast146/proj-shared/software/py-oct25-hip/bin/python3 (found suitable version "3.11.5", minimum required is "3.7.0") found components: Interpreter Development.Module
-- Downloading pybind11 ...
-- pybind11 source: https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.tar.gz
-- pybind11 v2.13.6 
-- Performing Test HAS_FLTO_THIN
-- Performing Test HAS_FLTO_THIN - Success
-- Downloading Catch2 ...
-- Catch2 source: https://github.com/catchorg/Catch2/archive/refs/tags/v2.13.10.tar.gz
-- Python LTO/IPO: ON
-- Note: Skipping example and tool runs (missing openPMD-example-datasets)
-- Note: run
    /ccs/proj/ast146/pgrete/src/openPMD-api/share/openPMD/download_samples.sh
to add example files to samples/git-sample/ directory!
-- Found mpi4py: TRUE

openPMD build configuration:
  library Version: 0.17.0
  openPMD Standard: 1.1.0
  C++ Compiler: Clang 18.0.0 
    /opt/rocm-6.2.4/bin/hipcc

  Install with RPATHs: ON
  Installation prefix: /ccs/proj/ast146/proj-shared/openpmd-api/oct25-hip-21adf5a
        bin: bin
        lib: lib64
    include: include
      cmake: lib64/cmake/openPMD
     python: lib64/python3.11/site-packages

  Build Type: Release
  Library: shared
  CLI Tools: ON
  Examples: ON
  Testing: ON
  Invasive Tests: OFF
  Internal VERIFY: ON
  Build Options:
    MPI: ON
    HDF5: ON
    ADIOS2: ON
    PYTHON: ON
    CUDA_EXAMPLES: OFF

-- Configuring done (30.8s)
-- Generating done (1.0s)
-- Build files have been written to: /ccs/proj/ast146/pgrete/src/openPMD-api/build-oct25-hip-21adf5a

led to the following output/problem during install

$ cmake --build /ccs/proj/ast146/pgrete/src/openPMD-api/build-oct25-hip-21adf5a -j1
[  0%] Building CXX object CMakeFiles/openPMD.dir/src/auxiliary/UniquePtr.cpp.o
/ccs/proj/ast146/pgrete/src/openPMD-api/src/auxiliary/UniquePtr.cpp:68:25: error: reference to __host__ function '~unique_ptr' in __host__ __device__ function
   68 | UniquePtrWithLambda<T>::UniquePtrWithLambda() = default;
      |                         ^
/ccs/proj/ast146/pgrete/src/openPMD-api/src/auxiliary/UniquePtr.cpp:68:25: note: in defaulted default constructor for 'openPMD::UniquePtrWithLambda<char>' first required here
/ccs/proj/ast146/pgrete/src/openPMD-api/src/auxiliary/UniquePtr.cpp:101:34: note: in instantiation of member function 'openPMD::UniquePtrWithLambda<char>::UniquePtrWithLambda' requested here
  101 | OPENPMD_FOREACH_DATASET_DATATYPE(OPENPMD_INSTANTIATE_WITH_AND_WITHOUT_EXTENT)
      |                                  ^
/usr/lib64/gcc/x86_64-suse-linux/13/../../../../include/c++/13/bits/unique_ptr.h:398:7: note: '~unique_ptr' declared here
  398 |       ~unique_ptr() noexcept
      |       ^
/ccs/proj/ast146/pgrete/src/openPMD-api/src/auxiliary/UniquePtr.cpp:68:25: error: reference to __host__ function '~unique_ptr' in __host__ __device__ function
   68 | UniquePtrWithLambda<T>::UniquePtrWithLambda() = default;
      |                         ^

...
/usr/lib64/gcc/x86_64-suse-linux/13/../../../../include/c++/13/bits/unique_ptr.h:398:7: note: '~unique_ptr' declared here
  398 |       ~unique_ptr() noexcept
      |       ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated when compiling for gfx90a.
failed to execute:/opt/rocm-6.2.4/lib/llvm/bin/clang++  --offload-arch=gfx90a  -DADIOS2_USE_MPI -DH5_BUILT_AS_DYNAMIC_LIB -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_MPICC_H -D_POSIX_C_SOURCE=200809L -DopenPMD_EXPORTS -DopenPMD_USE_FILESYSTEM_HEADER=0 -DopenPMD_USE_VERIFY=1 -I/ccs/proj/ast146/pgrete/src/openPMD-api/include -I/ccs/proj/ast146/pgrete/src/openPMD-api/build-oct25-hip-21adf5a/include -isystem /ccs/proj/ast146/pgrete/src/openPMD-api/build-oct25-hip-21adf5a/_deps/fetchednlohmann_json-src/include -isystem /ccs/proj/ast146/pgrete/src/openPMD-api/build-oct25-hip-21adf5a/_deps/fetchedtoml11-src/include -isystem /sw/frontier/spack-envs/cpe24.11-cpu/opt/cce-18.0.1/hdf5-1.14.3-ftanfovhcorzlcw2j7q5f2qugloroqkl/include -isystem /opt/cray/pe/mpich/8.1.31/ofi/cray/17.0/include -isystem /sw/frontier/spack-envs/cpe24.11-cpu/opt/cce-18.0.1/adios2-2.10.2-ssnfnmbbyipk7jo25xhfg457x5kl5hke/include -Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code -Wsign-compare -I/opt/rocm-6.2.4/include -Wno-pass-failed -O3 -DNDEBUG -std=c++17 -fPIC -MD -MT CMakeFiles/openPMD.dir/src/auxiliary/UniquePtr.cpp.o -MF CMakeFiles/openPMD.dir/src/auxiliary/UniquePtr.cpp.o.d -o "CMakeFiles/openPMD.dir/src/auxiliary/UniquePtr.cpp.o" -c -x hip /ccs/proj/ast146/pgrete/src/openPMD-api/src/auxiliary/UniquePtr.cpp
gmake[2]: *** [CMakeFiles/openPMD.dir/build.make:398: CMakeFiles/openPMD.dir/src/auxiliary/UniquePtr.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:983: CMakeFiles/openPMD.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

System environment

• machine: Frontier

Additional context
Current dev version of the repo.

[pgrete]

FWIW no issue to build/install version 0.16.1 in the environment above.

[franzpoeschel]

Thank you for the report. We had similar troubles in other files not so long ago, I will check what is going on.

[ax3l]

Looks like a std::unique_ptr is copied to or create inside a GPU kernel.
The file is new in dev https://github.com/openPMD/openPMD-api/blob/dev/src/auxiliary/UniquePtr.cpp and not yet part of a release.

[franzpoeschel]

#1801 should hopefully fix this, sorry for the delay.
I'll wait for @ax3l to have a look at it before merging.

[ax3l]

While we will work-around this with #1801, let us send a reproducer to OLCF help so they report this to AMD. That way, this work-around will at some point not be needed anymore.

Can we simplify this to a compile reproducer that just clones a specific commit of openPMD-api and builds it on Frontier?

[franzpoeschel]

The error can be reproduced by this minimal example:

#include <memory>

template <typename T>
class MyUniquePtr : public std::unique_ptr<T>
{
private:
    using BasePtr = std::unique_ptr<T>;

public:
    MyUniquePtr();

    MyUniquePtr(MyUniquePtr &&) noexcept;
    MyUniquePtr &operator=(MyUniquePtr &&) noexcept;

    MyUniquePtr(MyUniquePtr const &) = delete;
    MyUniquePtr &operator=(MyUniquePtr const &) = delete;
};

template <typename T>
MyUniquePtr<T>::MyUniquePtr() = default;
template <typename T>
MyUniquePtr<T>::MyUniquePtr(MyUniquePtr &&) noexcept = default;
template <typename T>
MyUniquePtr<T> &MyUniquePtr<T>::operator=(MyUniquePtr &&) noexcept = default;

template class MyUniquePtr<char>;

I have opened a ticket with OLCF support.