[BUG]: finer grained ABI compatibility

[beckermr]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

Based on the extensive discussion in conda-forge plus offline discussion with @isuruf, we think there may be a better approach to ABI compatibility with respect to compilers and c++ runtimes.

Currently, pybind11 uses a combination of various things. The one of interest here is the __GXX_ABI_VERSION. This appears to be too strict, especially for forward-compatible libraries commonly in use.

The suggestion is to replace __GXX_ABI_VERSION with

• __GLIBCXX_USE_CXX11_ABI
• __GLIBCXX_USE_DEPRECATED
• __SEH__ (only in clang I think)
• __SJLJ_EXCEPTIONS__

These capture the fine-grained, core ABI changes without being overly prescriptive. The last two are related to exception handling. There may be more macros to use to capture dwarf2 exceptions explicitly.

Thoughts @henryiii @isuruf?

Reproducible example code

No response

[allanleal]

I'm wondering if what you guys want to solve here is related to the issue I'm facing for quite some time already.

I have two C++ libraries (autodiff and Reaktoro) for which pybind11 is used to generate their Python bindings. When one is compiled with a different version than the other, I start to have issues like the following:

The ChemicalState::temperature method is expecting autodiff::real. However, this type is constructible from arithmetic types, so state.temperature(100, "celsius") works fine ONLY WHEN autodiff and Reaktoro are compiled with exact same compilers. Otherwise I get these errors as shown in the snapshot.

I have not been able to resolve this from my side; have tried already multiple solutions, additional conversion overloads, with no avail. I would appreciate to know if this is an unresolved issue in pybind11 or if there is a solution to this already.

[beckermr]

Is this different compilers or different C++ stdlib ABIs?

[allanleal]

The issue happens with both cases:

• different compilers (gcc for one package, clang for the other)
• same compilers, different versions (e.g., gcc 9.4 for one, gcc 10.3 for the other).

For example, this happened today when a new conda-forge package for Reaktoro was built with gcc 10.3, but the autodiff package available in conda-forge was built days ago with gcc 9.4.

[beckermr]

So I have a PR open to handle the conda-forge issues via the pybind11-abi package: conda-forge/pybind11-feedstock#79

The first item is out of my wheel house.

[tdegeus]

I just want to state my support for make the ABI compatibly finer. I use classes from different modules and ship them using conda-forge. The current behaviour is just a nightmare on Linux: all involved modules have to be compiled with the same GCC major version. This means that when the compiler of conda-forge is updated:

1. All modules have to be reshipped with the latest compiler.
2. Even then, things might not work, as not all my systems would have easy access to that compiler.

[wjakob]

The key question is: when is the ABI compatibility criterion in pybind11 too strict, and what can be done to make it more relaxed without running a risk of breaking things? (The same also applies to nanobind, which uses essentially the same kind of ABI tag)

Let me add one correction: simply using different versions of GCC alone is not enough to cause issues -- pybind11 queries the ABI compatibility tag of libstdc++ (__GXX_ABI_VERSION). And that's where this whole thing gets kind of tricky: when pybind11 isolates different extensions, it's does so intentionally because a change in this ABI version.

At that point, it's basically your word against that of the libstdc++ authors: they are saying that there was an ABI break, and you wish to disable it because it is inconvenient.

These ABI tags haven't been introduced just for fun -- it's because things were crashing badly and in weird ways, and it was really complicated to track down what was going on. So any change here would need a very careful justification on why the coarser-grained flags are 100% guaranteed to avoid such issues, now and in the future. The alternative proposal @beckermr made at the top will cause less things from being isolated from each other, but it wasn't clear to me why it would provide such a guarantee. Note that this isn't just about pybind11 usage of libstdc++ (which uses only relatively simple and presumably stable components like std::string, std::unordered_map, etc.). It also includes the extensions, which could use pybind11 to exchange arbitrary C++ instances and segfault during this process because of an ABI break.

Taking a step back, this seems to me to not be a pybind11 issue but a complaint about the libstdc++ authors changing __GXX_ABI_VERSION relatively often. And that is just not an issue we can fix here..

One thing I have never tried, but which may be worth a shot, is to provide -fabi-version=.., which is a GCC parameter that supposedly forces GCC to a specific ABI version. Maybe that's a feasible route to pin the ABI version of multiple extensions so that they aren't isolated from each other. Barring that, building with the same GCC major version doesn't strike me as the end of the world (we're talking about major, not minor versions, right?)

[isuruf]

GXX_ABI_VERSION is too broad. Yes, there might have been an ABI break because of name mangling issues, but there is a compatibility scheme that adds a aliased symbol. See -fabi-compat-version in https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/C_002b_002b-Dialect-Options.html

Barring that, building with the same GCC major version doesn't strike me as the end of the world (we're talking about major, not minor versions, right?)

It is when you have hundreds of downstream packages.

Another issue with this system is that the current system breaks things silently whereas we haven't had an issue with the libstdc++ version for years. That's because of the -fabi-compat-version workaround.

[wjakob]

It is when you have hundreds of downstream packages.

Extensions which need to be part of the same "pybind11 domain" for C++ interop with custom types are sort of "buddies" with each other. Typical pybind11 extensions bind their own custom types but don't need such an intimate relationship with another extension. So that statement about hundreds of packages being affected seems high to me.

I have a few follow-up questions:

1. Can you explain how -fabi-compat-version=.. is used to create the workaround you mentioned in CondaForge?
2. Does this cause GCC to set a #define internally so that it could be detected?
3. Do you know the difference between -fabi-version=.. and -fabi-compat-version=.. in the linked GCC docs. Those two seem very related (I'm confused).

[isuruf]

1. Can you explain how -fabi-compat-version=.. is used to create the workaround you mentioned in CondaForge?

It's automatically done by GCC. Nothing to do with conda-forge.

2. No

3. The only changes to the G++ ABI between -fabi-version=13 and -fabi-version=18 are name mangling differences. If a function foo had name mangling foo_mangle1 in -fabi-version=13 and foo_mangle2 in -fabi-version=18, then GCC would generate a symbol called foo_mangle2 and make an alias for foo_mangle1.
   When two packages are compiled with -fabi-version=13, they generate the symbols foo_mangle1, but when only the upstream is updated to compile with -fabi-version=18, GCC generates both foo_mangle1 and foo_mangle2.

So, you have to consider that -fabi-version={13,14,15,16,17,18} are all interchangeable on Linux and other systems with strong aliasing. (GCC 8.2 - GCC 13.2)