Generation of binaries compatible with most Linux distributions

[joseasoler]

Since todds supports static builds, it is possible to generate a binary that contains all of the dependencies. The remaining issue has always been glibc. So far, we have been building binaries using increasingly older Linux distributions, in the hopes of finally finding a combination that works for everyone.

This is time consuming, as it requires setting up a new VM and figuring out how to compile todds in that case. Also, lately we have been reaching a point where using even older distros would mean switching from GCC 13 to older versions, which could potentially cause performance impacts. We need an alternative which addresses these problems.

One potential way to address all of this would be to use crosstool-ng. This is explained in the "Setup 2" part of this excellent Stackoverflow response: https://stackoverflow.com/a/52550158

This setup should give us freedom to choose which glibc version to use without requiring a VM.

[joseasoler]

I have spent a lot of time trying to get crosstool-NG working without success. Which is frustrating, as it has a CentOS 7 toolchain which should do exactly what we need. In the end I have decided to go back to VMs, using CentOS 7 this time.

[lisanhu]

Hi! I'm having an issue running todds release binary

 todds_Linux_x86_64_0.4.1 $ ./todds 
./todds: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by ./todds)
./todds: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by ./todds)

I'm running it on Pop!_OS 22.04 LTS x86_64. I can compile and run the built target myself, but I want to get it work inside RimSort, which downloads the latest release from github.

I find this issue and I know it is not easy to resolve, I'm wondering are there any solutions / ideas now about static linking with libstdc++?

I personally tried to add the static linking flags to every target_link_libraries possible in the CMakeLists.txt files all over the project and still got the dependencies using ldd. I think it could be the reason that Boost and OpenCV are depending on libstdc++, but I'm not sure these libraries will cause this behaviour.

I could not verify this because I don't have a machine that I can test it on. ( On my machine it will always build with the correct ABI, but not sure if it's built somewhere else and run on my system. I want to know if those dependencies are purely from Boost and OpenCV, is it possible to run on my machine with the static linked built binary from another machine? )

I think the release binary is compiled using a VM in CentOS 7? Am I correct?

[joseasoler]

@lisanhu Which todds release binary are you running? Is it https://github.com/todds-encoder/todds/releases/download/0.4.1/todds_Linux_x86_64_0.4.1.zip?

If that is the case, what happens if you try with https://github.com/todds-encoder/todds/releases/download/0.0.0/todds_Linux_x86_64_0.0.0.zip?

Regarding static linking, it is a very complex issue. My comments above are an incomplete summary of the different attempts I made at resolving this problem. Other attempts included using a musl based toolchain (see https://build-your-own.org/blog/20221229_alpine/ for details), building with increasingly newer Ubuntu versions starting at 14.04 LTS, and so on.

As you mentioned, part of the difficulty comes from dependencies such as Boost and OpenCV. These are huge libraries with complex build systems that were developed long before these static linking solutions were developed.

After all of the time I spent trying to fix this issue without success, I decided it was more cost-effective to just build using a relatively old Linux distro using vcpkg. vcpkg takes care of linking almost every dependency statically, while the old version of glibc provided by the Linux distribution that is dynamically linked against todds keeps compatibility as high as possible.

The todds_Linux_x86_64_0.0.0.zip binary I mentioned early is an alternative todds 0.4.1 binary built on Debian 11 using the new cross-platform release workflow kindly provided by @twstagg. Earlier releases (including todds_Linux_x86_64_0.4.1.zip) were built using Ubuntu 20.04.

[lisanhu]

@lisanhu Which todds release binary are you running? Is it https://github.com/todds-encoder/todds/releases/download/0.4.1/todds_Linux_x86_64_0.4.1.zip?

Yes, this is the version I was using.

If that is the case, what happens if you try with https://github.com/todds-encoder/todds/releases/download/0.0.0/todds_Linux_x86_64_0.0.0.zip?

It's still not working for me.

 todds_Linux_x86_64_0.0.0 $ ./todds 
./todds: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by ./todds)
./todds: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by ./todds)

And I realized both versions are reporting the same error messages, it's very strange to see behaviours like this. So I dig a little bit more into this and I realized that the release binaries are using new ABIs and my system is only supporting older versions.

The system compiler on my system is gnu c++ 11, and glibcxx versions supported by my system is:

 todds_Linux_x86_64_0.4.1 $ strings /lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_3.4.22
GLIBCXX_3.4.23
GLIBCXX_3.4.24
GLIBCXX_3.4.25
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28
GLIBCXX_3.4.29
GLIBCXX_3.4.30
GLIBCXX_DEBUG_MESSAGE_LENGTH

The strings output for both 0.4.1 and 0.0.0:

[ lisanhu@system76 : ~/Downloads/todds_Linux_x86_64_0.4.1 ]
 todds_Linux_x86_64_0.4.1 $ strings todds  | grep GLIBCXX_3.4.32
GLIBCXX_3.4.32
_ZSt21ios_base_library_initv@@GLIBCXX_3.4.32
[ lisanhu@system76 : ~/Downloads/todds_Linux_x86_64_0.4.1 ]
 todds_Linux_x86_64_0.4.1 $ strings todds  | grep GLIBCXX_3.4.31
GLIBCXX_3.4.31
_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE15_M_replace_coldEPcmPKcmm@@GLIBCXX_3.4.31
[ lisanhu@system76 : ~/Downloads/todds_Linux_x86_64_0.4.1 ]
 todds_Linux_x86_64_0.4.1 $ cd ../todds_Linux_x86_64_0.0.0/
[ lisanhu@system76 : ~/Downloads/todds_Linux_x86_64_0.0.0 ]
 todds_Linux_x86_64_0.0.0 $ strings todds  | grep GLIBCXX_3.4.31
GLIBCXX_3.4.31
_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE15_M_replace_coldEPcmPKcmm@GLIBCXX_3.4.31
[ lisanhu@system76 : ~/Downloads/todds_Linux_x86_64_0.0.0 ]
 todds_Linux_x86_64_0.0.0 $ strings todds  | grep GLIBCXX_3.4.32
GLIBCXX_3.4.32
_ZSt21ios_base_library_initv@GLIBCXX_3.4.32

[joseasoler]

Thank you for looking into it! That information is very useful.

I am seeing similar results after analyzing todds with objdump:

Version References:
  required from ld-linux-x86-64.so.2:
    0x0d696913 0x00 41 GLIBC_2.3
  required from libdl.so.2:
    0x09691a75 0x00 23 GLIBC_2.2.5
  required from libgcc_s.so.1:
    0x0b792654 0x00 33 GCC_3.4
    0x0b792650 0x00 19 GCC_3.0
  required from libm.so.6:
    0x06969187 0x00 42 GLIBC_2.27
    0x06969189 0x00 28 GLIBC_2.29
    0x09691a75 0x00 12 GLIBC_2.2.5
  required from libstdc++.so.6:
    0x0297f842 0x00 40 GLIBCXX_3.4.32
    0x0297f870 0x00 39 GLIBCXX_3.4.20
    0x0bafd178 0x00 38 CXXABI_1.3.8
    0x0297f841 0x00 37 GLIBCXX_3.4.31
    0x0297f867 0x00 36 GLIBCXX_3.4.17
    0x0297f869 0x00 34 GLIBCXX_3.4.19
    0x0bafd179 0x00 32 CXXABI_1.3.9
    0x0bafd172 0x00 29 CXXABI_1.3.2
    0x0297f879 0x00 27 GLIBCXX_3.4.29
    0x0297f868 0x00 26 GLIBCXX_3.4.18
    0x0297f876 0x00 25 GLIBCXX_3.4.26
    0x02297f89 0x00 22 GLIBCXX_3.4.9
    0x056bafd3 0x00 21 CXXABI_1.3
    0x0bafd173 0x00 20 CXXABI_1.3.3
    0x0297f861 0x00 17 GLIBCXX_3.4.11
    0x0afd17f1 0x00 16 CXXABI_1.3.11
    0x0297f864 0x00 15 GLIBCXX_3.4.14
    0x0bafd175 0x00 13 CXXABI_1.3.5
    0x0297f872 0x00 10 GLIBCXX_3.4.22
    0x0afd17f3 0x00 09 CXXABI_1.3.13
    0x0297f865 0x00 08 GLIBCXX_3.4.15
    0x0297f871 0x00 06 GLIBCXX_3.4.21
    0x08922974 0x00 05 GLIBCXX_3.4
  required from libpthread.so.0:
    0x09691a75 0x00 07 GLIBC_2.2.5
    0x09691972 0x00 04 GLIBC_2.3.2
  required from libc.so.6:
    0x06969188 0x00 35 GLIBC_2.28
    0x0d696917 0x00 31 GLIBC_2.7
    0x06969194 0x00 30 GLIBC_2.14
    0x0d696914 0x00 24 GLIBC_2.4
    0x06969185 0x00 18 GLIBC_2.25
    0x0d696913 0x00 14 GLIBC_2.3
    0x0d696916 0x00 11 GLIBC_2.6
    0x09691a75 0x00 03 GLIBC_2.2.5
    0x09691974 0x00 02 GLIBC_2.3.4

As you mentioned, libstdc++.so.6 is bringing a newer glibc version; other libraries are using the expected, older version. The issue seems related to the problem mentioned in this StackOverflow post, where the prebuilt package of g++-13 brings in a version of libstdc++.so that requires a newer glibc version. I do not see any way around this issue except for replacing the g++ package with a self-compiled version built against an older glibc version.

I also noticed that I forgot to answer your question about static linking and glibc. todds is linked statically against every dependency with just two exceptions, the C standard library (glibc) and the C++ standard library (in this case, libstdc++). In the case of libstdc++, this happens because it also uses glibc.

In the case of glibc, dynamic linking is used because this library is not designed to be linked statically. Attempting to do so will generate a binary that fails to run in the best case, or has subtle bugs in the worst case. A more in-depth explanation can be found here: https://stackoverflow.com/questions/57476533/why-is-statically-linking-glibc-discouraged

I am going to spend some additional time trying to find out another solution to this issue, there might be something I have not tried yet. If this fails, I will fall back to getting an older Ubuntu LTS and compiling all dependencies manually.

[lisanhu]

Hi! Thanks for your reply! I just found this: https://stackoverflow.com/questions/13636513/linking-libstdc-statically-any-gotchas

The choosen answer suggested a different approach (in the last paragraph): include the libstdc++.so with the binary, and use RPATH to make sure the linker is linking to relative path. I think this might be a better and easier solution?

[joseasoler]

That included libstdc++.so file would still depend on the newer glibc version, as mentioned in one of the follow-up answers on that post (https://stackoverflow.com/a/13636770/7001854).

[joseasoler]

@lisanhu I just realized that I was looking for glibc errors, while the errors you are experiencing are glibc++ errors; I am sorry about the confusion.

In this case the solution you suggest should work, as long as I can coerce vcpkg to statically link against libstdc++. I will investigate more into this and try to generate a binary that does this.

[joseasoler]

While searching for ways to make vcpkg handle and respect the -static-libstdc++ setting, I ran into another issue. libstdc++ is under the GPL license, so it is not allowed to statically link a MPL2 licensed program such as todds against it without relicensing todds to use the GPL.

A similar issue happens with distributing libstdc++.so. Although distributing a dinamically linked GPL file along with a MPL2 binary seems to be allowed, if I am understanding the legalese language correctly then the GPL would require me to also distribute the source of libstdc++.so as well.

In principle I am not opposed to relicensing todds under the terms of the GPL but it is something I need to consider. Besides thinking about the pros and cons, this will involve checking every dependency for GPL compatibility and so on.

All of these are quite complex factors that will require some time to think about on my part. For now I would recommend trying to self-compile todds and using the resulting binary to replace the one provided by RimSort.