Linking errors and warnings lead to failure in generating shared library

[wangxdgg]

I followed the timeloop tutorial to install it, and then followed the timeloop-python Installing, but pip3 install -e. The following error message is returned.

(base) root@2b5da774569c:/home/workspace/gamma/accelergy-timeloop-infrastructure/src/timeloop-python# pip3 install .
Processing /home/workspace/gamma/accelergy-timeloop-infrastructure/src/timeloop-python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pyyaml in /root/miniforge-pypy3/lib/python3.9/site-packages (from pytimeloop==0.0.1) (6.0.1)
Building wheels for collected packages: pytimeloop
  Building wheel for pytimeloop (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pytimeloop (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [150 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-pypy39
      creating build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/__init__.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/accelergy_interface.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/config.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/engine.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/mapper.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/mapping.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/mapspace.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/model.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/problem.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      copying pytimeloop/search.py -> build/lib.linux-x86_64-pypy39/pytimeloop
      creating build/lib.linux-x86_64-pypy39/tests
      copying tests/__init__.py -> build/lib.linux-x86_64-pypy39/tests
      copying tests/test_accelerator.py -> build/lib.linux-x86_64-pypy39/tests
      copying tests/test_buffer.py -> build/lib.linux-x86_64-pypy39/tests
      copying tests/test_config.py -> build/lib.linux-x86_64-pypy39/tests
      copying tests/test_mapper.py -> build/lib.linux-x86_64-pypy39/tests
      copying tests/test_model.py -> build/lib.linux-x86_64-pypy39/tests
      copying tests/test_topology.py -> build/lib.linux-x86_64-pypy39/tests
      copying tests/util.py -> build/lib.linux-x86_64-pypy39/tests
      creating build/lib.linux-x86_64-pypy39/pytimeloop/app
      copying pytimeloop/app/__init__.py -> build/lib.linux-x86_64-pypy39/pytimeloop/app
      copying pytimeloop/app/call_utils.py -> build/lib.linux-x86_64-pypy39/pytimeloop/app
      copying pytimeloop/app/mapper.py -> build/lib.linux-x86_64-pypy39/pytimeloop/app
      copying pytimeloop/app/model.py -> build/lib.linux-x86_64-pypy39/pytimeloop/app
      creating build/lib.linux-x86_64-pypy39/tests/demo
      copying tests/demo/__init__.py -> build/lib.linux-x86_64-pypy39/tests/demo
      copying tests/demo/demo.py -> build/lib.linux-x86_64-pypy39/tests/demo
      running build_ext
      CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.
      
        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.
      
      
      -- The C compiler identification is GNU 11.4.0
      -- The CXX compiler identification is GNU 11.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Using timeloop includes from: /home/workspace/gamma/accelergy-timeloop-infrastructure/src/timeloop/include
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
      -- Found Threads: TRUE
      -- pybind11 v2.13.0 dev1
      -- Found PythonInterp: /root/miniforge-pypy3/bin/python3.9 (found suitable version "3.9.18", minimum required is "3.6")
      -- Found PythonLibs: python3.9
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      -- Configuring done (0.9s)
      -- Generating done (0.0s)
      -- Build files have been written to: /home/workspace/gamma/accelergy-timeloop-infrastructure/src/timeloop-python/build/temp.linux-x86_64-pypy39
      [  3%] Building CXX object CMakeFiles/define_gterminate.dir/bindings/gterminate.cpp.o
      [  7%] Linking CXX static library libdefine_gterminate.a
      [  7%] Built target define_gterminate
      [ 10%] Building CXX object CMakeFiles/pytimeloop.dir/src/mapper/coupled-mapper.cpp.o
      [ 14%] Building CXX object CMakeFiles/pytimeloop.dir/src/mapspace/status.cpp.o
      [ 17%] Building CXX object CMakeFiles/pytimeloop.dir/src/model/accelerator.cpp.o
      [ 21%] Building CXX object CMakeFiles/pytimeloop.dir/src/model/bounded-queue-accelerator-pool.cpp.o
      [ 25%] Building CXX object CMakeFiles/pytimeloop.dir/src/model/unbounded-queue-accelerator-pool.cpp.o
      [ 28%] Building CXX object CMakeFiles/pytimeloop.dir/src/model/util.cpp.o
      [ 32%] Linking CXX shared library /home/workspace/gamma/accelergy-timeloop-infrastructure/src/timeloop-python/build/lib.linux-x86_64-pypy39/libpytimeloop.so
      lto-wrapper: warning: using serial compilation of 39 LTRANS jobs
      ../../../timeloop/src/loop-analysis/mapping-to-isl/mapping-to-isl.cpp: In function 'LogicalBufSkewsFromMapping':
      ../../../timeloop/src/loop-analysis/mapping-to-isl/mapping-to-isl.cpp:284:34: warning: 'MEM[(const int &)&last_y_idx_opt]' may be used uninitialized in this function [-Wmaybe-uninitialized]
        284 |       for (int i = last_y_idx; i < loop_idx; ++i)
            |                                  ^
      ../../../timeloop/src/loop-analysis/mapping-to-isl/mapping-to-isl.cpp:174:22: note: 'MEM[(const int &)&last_y_idx_opt]' was declared here
        174 |   std::optional<int> last_y_idx_opt = std::nullopt;
            |                      ^
      ../../../timeloop/src/loop-analysis/mapping-to-isl/mapping-to-isl.cpp:246:7: warning: 'MEM[(const int &)&last_x_idx_opt]' may be used uninitialized in this function [-Wmaybe-uninitialized]
        246 |       for (auto i = last_x_idx; i < loop_idx; ++i)
            |       ^
      ../../../timeloop/src/loop-analysis/mapping-to-isl/mapping-to-isl.cpp:173:22: note: 'MEM[(const int &)&last_x_idx_opt]' was declared here
        173 |   std::optional<int> last_x_idx_opt = std::nullopt;
            |                      ^
      /usr/bin/ld: /tmp/cchzahKo.ltrans30.ltrans.o: relocation R_X86_64_TPOFF32 against symbol `gCtx' can not be used when making a shared object; recompile with -fPIC
      /usr/bin/ld: failed to set dynamic section sizes: bad value
      collect2: error: ld returned 1 exit status
      gmake[2]: *** [CMakeFiles/pytimeloop.dir/build.make:179: /home/workspace/gamma/accelergy-timeloop-infrastructure/src/timeloop-python/build/lib.linux-x86_64-pypy39/libpytimeloop.so] Error 1
      gmake[1]: *** [CMakeFiles/Makefile2:130: CMakeFiles/pytimeloop.dir/all] Error 2
      gmake: *** [Makefile:91: all] Error 2
      Traceback (most recent call last):
        File "/root/miniforge-pypy3/lib/pypy3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/miniforge-pypy3/lib/pypy3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/root/miniforge-pypy3/lib/pypy3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/build_meta.py", line 410, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 97, in <module>
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
          self.build_extensions()
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 479, in build_extensions
          self._build_extensions_serial()
        File "/tmp/pip-build-env-8c1iw72k/overlay/lib/pypy3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 505, in _build_extensions_serial
          self.build_extension(ext)
        File "<string>", line 91, in build_extension
        File "/root/miniforge-pypy3/lib/pypy3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '--build', '.']' returned non-zero exit status 2.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pytimeloop
Failed to build pytimeloop
ERROR: Could not build wheels for pytimeloop, which is required to install pyproject.toml-based projects

Thank you for your assistance!

[gilbertmike]

I haven't seen this error before, but it seems like the timeloop libraries are built without PIC and that is causing the issue. Would you mind providing some information about how you built timeloop? E.g., what commands did you call to build timeloop.

[EMJzero]

Hi,
I apologize for the disturbance, but I am also encountering the same issue after pulling the most recent version of the Accelergy-Timeloop Infrastructure.
After pulling the infrastructure, I also updated some of the repositories to their most recent commit:

• pytimeloop: 54cdc1e50a1d0b34dedd3c39f1c649e14f983253
• Timeloop: 450af7939ce84b6f577ce3566fa915f50dd4bd8b

I then built the infrastructure as explained in: https://timeloop.csail.mit.edu/v4/installation, thereby building Timeloop through SConstruct with scons -j4 --with-isl --static --accelergy.
Subsequently I proceeded to install pytimeloop as per its README: https://github.com/Accelergy-Project/timeloop-python/blob/main/README.md
As a result, the same error as above occurs.

I managed to work around the issue somewhat by modifying Timeloop's SConstruct as follows:

# ...
env = Environment(ENV = os.environ)

if not GetOption('clang'):
    env.Append(CCFLAGS=['-fPIC'])
#...

And after re-building Timeloop, I got a different error while running pip3 install -e . for pytimeloop:

# ...
    [ 46%] Building CXX object CMakeFiles/bindings.dir/src/model/bounded-queue-accelerator-pool.cpp.o
    [ 50%] Building CXX object CMakeFiles/bindings.dir/src/model/unbounded-queue-accelerator-pool.cpp.o
    [ 53%] Building CXX object CMakeFiles/bindings.dir/src/model/util.cpp.o
    [ 56%] Building CXX object CMakeFiles/bindings.dir/bindings/accelergy_interface.cpp.o
    [ 60%] Building CXX object CMakeFiles/bindings.dir/bindings/applications.cpp.o
    [ 63%] Building CXX object CMakeFiles/bindings.dir/bindings/bindings.cpp.o
    [ 66%] Building CXX object CMakeFiles/bindings.dir/bindings/buffer.cpp.o
    [ 70%] Building CXX object CMakeFiles/bindings.dir/bindings/config.cpp.o
    [ 73%] Building CXX object CMakeFiles/bindings.dir/bindings/looptree.cpp.o
    /home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/bindings/looptree.cpp: In function ‘void pytimeloop::looptree_bindings::BindLooptree(pybind11::module&)’:
    /home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/bindings/looptree.cpp:14:16: error: ‘application’ was not declared in this scope; did you mean ‘Application’?
       14 |     py::class_<application::LooptreeModel>(m, "LooptreeModelApp")
          |                ^~~~~~~~~~~
          |                Application
    /home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/bindings/looptree.cpp:14:42: error: template argument 1 is invalid
       14 |     py::class_<application::LooptreeModel>(m, "LooptreeModelApp")
          |                                          ^
    /home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/bindings/looptree.cpp:18:32: error: ‘application’ is not a class, namespace, or enumeration
       18 |         .def_readwrite("ops", &application::LooptreeModel::Result::ops)
          |                                ^~~~~~~~~~~
    /home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/bindings/looptree.cpp:19:33: error: ‘application’ is not a class, namespace, or enumeration
       19 |         .def_readwrite("fill", &application::LooptreeModel::Result::fill)
          |                                 ^~~~~~~~~~~
    /home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/bindings/looptree.cpp:20:38: error: ‘application’ is not a class, namespace, or enumeration
       20 |         .def_readwrite("occupancy", &application::LooptreeModel::Result::occupancy);
          |                                      ^~~~~~~~~~~
    gmake[2]: *** [CMakeFiles/bindings.dir/build.make:244: CMakeFiles/bindings.dir/bindings/looptree.cpp.o] Error 1
    gmake[1]: *** [CMakeFiles/Makefile2:156: CMakeFiles/bindings.dir/all] Error 2
    gmake: *** [Makefile:91: all] Error 2
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/setup.py", line 97, in <module>
        setup(
      File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
        return run_commands(dist)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
        dist.run_commands()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
        cmd_obj.run()
      File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
        cmd_obj.run()
      File "/usr/lib/python3/dist-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/setup.py", line 91, in build_extension
        subprocess.check_call(
      File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '--build', '.']' returned non-zero exit status 2.
    [end of output]

To solve this error, I temporarily rolled back the last commit of pytimeloop, going from 54cdc1e50a1d0b34dedd3c39f1c649e14f983253 to 95e4a731d7e518ea16a59ed217fbac5330a6b777.
After this, I have been able to successfully run pip3 install -e . for pytimeloop.
However, once I try to import it, I get the following error:

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytimeloop
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/pytimeloop/init.py", line 1, in <module>
    from bindings import *
ImportError: /home/zero/Desktop/Timeloop/accelergy-timeloop-infrastructure/src/timeloop-python/bindings.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZTIN5boost13serialization13typeid_system27extended_type_info_typeid_0E

I don't know if the steps I took were meaningful in fixing the problem, but anyway I did not manage to go any further from here.
If there is any further information that you think could be of help, I will promptly to provide it.

Thank you in advance.

---

On a side note, I also tried to use pytimeloop within the Docker container mentioned in the documentation, however I cannot successfully import pytimeloop there too, with the following error (on a freshly created container):

workspace@accelergy-timeloop:~$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytimeloop
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/pytimeloop/init.py", line 1, in <module>
    from bindings import *
ImportError: libbarvinok.so.23: cannot open shared object file: No such file or directory