Nav2 crashes after latest Rolling sync

[mini-1235]

Describe the bug
The latest sync on rolling updates BTCPP from 4.7.2 to 4.8.0, this causes Nav2 to crash immediately

Backtrace:

[planner_server-18] #14   Source "../csu/libc-start.c", line 360, in __libc_start_main_impl [0x7bfa7f26728a]
[planner_server-18] #13   Source "../sysdeps/nptl/libc_start_call_main.h", line 58, in __libc_start_call_main [0x7bfa7f2671c9]
[planner_server-18] #12   Object "/root/nav2_ws/build/nav2_planner/planner_server", at 0x5e15b1f8d501, in main
[planner_server-18] #11   Object "/root/nav2_ws/install/nav2_planner/lib/libplanner_server_core.so", at 0x7bfa7fa0d2a1, in nav2_planner::PlannerServer::PlannerServer(rclcpp::NodeOptions const&)
[planner_server-18] #10   Object "/root/nav2_ws/install/nav2_planner/lib/libplanner_server_core.so", at 0x7bfa7fa50fc1, in pluginlib::ClassLoader<[nav2_core::GlobalPlanner](https://github.com/ros-navigation/navigation2/issues/nav2_core::GlobalPlanner)>::ClassLoader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >)
[planner_server-18] #9    Object "/root/nav2_ws/install/nav2_planner/lib/libplanner_server_core.so", at 0x7bfa7fa50961, in pluginlib::ClassLoader<[nav2_core::GlobalPlanner](https://github.com/ros-navigation/navigation2/issues/nav2_core::GlobalPlanner)>::determineAvailableClasses(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)
[planner_server-18] #8    Object "/root/nav2_ws/install/nav2_planner/lib/libplanner_server_core.so", at 0x7bfa7fa4dda4, in pluginlib::ClassLoader<[nav2_core::GlobalPlanner](https://github.com/ros-navigation/navigation2/issues/nav2_core::GlobalPlanner)>::processSingleXMLPluginFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, pluginlib::ClassDesc, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, pluginlib::ClassDesc> > >&)
[planner_server-18] #7    Object "/opt/ros/rolling/lib/libbehaviortree_cpp.so", at 0x7bfa7f128769, in tinyxml2::XMLDocument::~XMLDocument()
[planner_server-18] #6    Source "./malloc/malloc.c", line 3398, in __libc_free [0x7bfa7f2eadad]
[planner_server-18] #5    Source "./malloc/malloc.c", line 4507, in _int_free [0x7bfa7f2e838b]
[planner_server-18] #4    Source "./malloc/malloc.c", line 5772, in malloc_printerr [0x7bfa7f2e5ff4]
[planner_server-18] #3    Source "../sysdeps/posix/libc_fatal.c", line 134, in __libc_message_impl [0x7bfa7f2667b5]
[planner_server-18] #2    Source "./stdlib/abort.c", line 79, in abort [0x7bfa7f2658fe]
[planner_server-18] #1    Source "../sysdeps/posix/raise.c", line 26, in raise [0x7bfa7f28227d]
[planner_server-18] #0  | Source "./nptl/pthread_kill.c", line 89, in __pthread_kill_internal
[planner_server-18]     | Source "./nptl/pthread_kill.c", line 78, in __pthread_kill_implementation
[planner_server-18]       Source "./nptl/pthread_kill.c", line 44, in __pthread_kill [0x7bfa7f2dbb2c]
[planner_server-18] Aborted (Signal sent by tkill() 606035 0)

How to Reproduce

ros2 launch nav2_bringup tb4_simulation_launch.py headless:=0

Possibly due to tinyxml2 version mismatch. I tried downgrading to 4.7.2 and it works, reverting c0bc00b also works

[facontidavide]

good to know. I shall try to address this ASAP

[SteveMacenski]

I see this as well and correlates with @mini-1235 's report

[facontidavide]

So, i am perplexed by the fact that in Ubuntu 22.04 the official version of the library is 11.0, the same of BT.CPP 4.8

https://launchpad.net/ubuntu/+source/tinyxml2

therefore my best hypothesis is that you have some plugins compiled with 4.7 and the main application with 4.8, or something like this.

You may need to recompile everything, to fix this mismatch

[facontidavide]

If that doesn't fix the problem, how can I reproduce the issue?

[mini-1235]

https://github.com/ros/rosdistro/blob/8466a7104e19ab69ca8750d494a3541045e2fae4/rosdep/base.yaml#L9797 this points to libtinyxml2-dev

https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=tinyxml&searchon=names

On Ubuntu 22.04, it is using 9.0
On Ubuntu 24.04, it is using 10.0

jammy (22.04LTS) (libdevel): TinyXML2 library - header and static library [universe]
9.0.0+dfsg-3: amd64 arm64 armhf ppc64el riscv64 s390x
noble (24.04LTS) (libdevel): TinyXML2 library - header and static library [universe]
10.0.0+dfsg-2: amd64 arm64 armhf ppc64el riscv64 s390x

I think the tinyxml2 version in this repo has to match with the version tinyxml2 points to, does that make sense to you

[mini-1235]

I think this is also related to #78

[SteveMacenski]

I recompiled from scratch on Ubuntu 24.04 / updated Rolling binaries and I see this as still broken. This was working properly before I apt upgraded to new rolling packages. On 24.04:

mkdir -p nav2_ws/src
cd nav2_ws/src
git clone git@github.com:ros-navigation/navigation2.git
cd ../../
apt update
rosdep install -r -y --from-paths nav2_ws/src --ignore-src
apt upgrade -y 
colcon build 
ros2 launch nav2_bringup tb4_simulation_launch.py

[robin-mueller]

I'm not sure whether this is related, but I'm also experiencing an issue after the latest rolling package sync. For me there is a Segmentation fault (core dumped) error as soon as I run a program that instantiates a tinyxml2::XMLDocument object while being linked against behaviortree_cpp::behaviortree_cpp. The segfault seems to be related to the destruction of the XMLDocument instance (debugging the code step by step doesn't throw).

A minimal program that segfaults after building it with colcon and linking it with BT.CPP under rolling is

#include <tinyxml2.h>
#include <string>
#include <stdexcept>
int main()
{
  try {
    if (std::string("not empty").empty()) {
      throw std::runtime_error("error");
    }
    tinyxml2::XMLDocument doc;
  } catch (const std::exception & e) {
    return EXIT_FAILURE;
  }
  return EXIT_SUCCESS;
}

The following variations run fine:

#include <tinyxml2.h>
#include <string>
#include <stdexcept>
int main()
{
  try {
    if (std::string("not empty").empty()) {
      throw std::runtime_error("error");
    }
    // tinyxml2::XMLDocument doc;  commented out
  } catch (const std::exception & e) {
    return EXIT_FAILURE;
  }
  return EXIT_SUCCESS;
}

#include <tinyxml2.h>
#include <string>
#include <stdexcept>
int main()
{
  try {
    if (false) {  // true or false doesn't matter 
      throw std::runtime_error("error");
    }
    tinyxml2::XMLDocument doc;
  } catch (const std::exception & e) {
    return EXIT_FAILURE;
  }
  return EXIT_SUCCESS;
}

#include <tinyxml2.h>
#include <string>
#include <stdexcept>
int main()
{
  try {
    // removing the whole if clause fixes it as well
    tinyxml2::XMLDocument doc;
  } catch (const std::exception & e) {
    return EXIT_FAILURE;
  }
  return EXIT_SUCCESS;
}

#include <tinyxml2.h>
#include <string>
#include <stdexcept>
int main()
{
  // Without the try except, the originally failing program runs fine as well
  if (std::string("not empty").empty()) {
    throw std::runtime_error("error");
  }
  tinyxml2::XMLDocument doc;
}

I'll have to admit, I'm quite confused (probably some compiler magic optimizing away the XMLDocument instantiation and destruction) ..

[robin-mueller]

Valgrind output is (with execuable path /path/to/minimal/program)

valgrind --leak-check=full --show-leak-kinds=all -s /path/to/minimal/program

==68264== Memcheck, a memory error detector
==68264== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==68264== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==68264== Command: /path/to/minimal/program
==68264== 
==68264== Jump to the invalid address stated on the next line
==68264==    at 0xA: ???
==68264==    by 0x4CA028A: __libc_start_main@@GLIBC_2.34 (libc-start.c:360)
==68264==    by 0x10A344: (below main) (in /path/to/minimal/program)
==68264==  Address 0xa is not stack'd, malloc'd or (recently) free'd
==68264== 
==68264== 
==68264== Process terminating with default action of signal 11 (SIGSEGV)
==68264==  Bad permissions for mapped region at address 0xA
==68264==    at 0xA: ???
==68264==    by 0x4CA028A: __libc_start_main@@GLIBC_2.34 (libc-start.c:360)
==68264==    by 0x10A344: (below main) (in /path/to/minimal/program)
==68264== 
==68264== HEAP SUMMARY:
==68264==     in use at exit: 75,808 bytes in 3 blocks
==68264==   total heap usage: 3 allocs, 0 frees, 75,808 bytes allocated
==68264== 
==68264== 1,040 bytes in 1 blocks are still reachable in loss record 1 of 3
==68264==    at 0x484D953: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==68264==    by 0x4CBD0BC: __new_exitfn (cxa_atexit.c:112)
==68264==    by 0x4CBD230: __internal_atexit (cxa_atexit.c:44)
==68264==    by 0x4CBD230: __cxa_atexit (cxa_atexit.c:68)
==68264==    by 0x48B361D: ??? (in /opt/ros/rolling/lib/libbehaviortree_cpp.so)
==68264==    by 0x400571E: call_init.part.0 (dl-init.c:74)
==68264==    by 0x4005823: call_init (dl-init.c:120)
==68264==    by 0x4005823: _dl_init (dl-init.c:121)
==68264==    by 0x401F59F: ??? (in /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
==68264== 
==68264== 1,040 bytes in 1 blocks are still reachable in loss record 2 of 3
==68264==    at 0x484D953: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==68264==    by 0x4CBD0BC: __new_exitfn (cxa_atexit.c:112)
==68264==    by 0x4CBD230: __internal_atexit (cxa_atexit.c:44)
==68264==    by 0x4CBD230: __cxa_atexit (cxa_atexit.c:68)
==68264==    by 0x48B385D: ??? (in /opt/ros/rolling/lib/libbehaviortree_cpp.so)
==68264==    by 0x400571E: call_init.part.0 (dl-init.c:74)
==68264==    by 0x4005823: call_init (dl-init.c:120)
==68264==    by 0x4005823: _dl_init (dl-init.c:121)
==68264==    by 0x401F59F: ??? (in /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
==68264== 
==68264== 73,728 bytes in 1 blocks are still reachable in loss record 3 of 3
==68264==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==68264==    by 0x4A8138E: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33)
==68264==    by 0x400571E: call_init.part.0 (dl-init.c:74)
==68264==    by 0x4005823: call_init (dl-init.c:120)
==68264==    by 0x4005823: _dl_init (dl-init.c:121)
==68264==    by 0x401F59F: ??? (in /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
==68264== 
==68264== LEAK SUMMARY:
==68264==    definitely lost: 0 bytes in 0 blocks
==68264==    indirectly lost: 0 bytes in 0 blocks
==68264==      possibly lost: 0 bytes in 0 blocks
==68264==    still reachable: 75,808 bytes in 3 blocks
==68264==         suppressed: 0 bytes in 0 blocks
==68264== 
==68264== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
==68264== 
==68264== 1 errors in context 1 of 1:
==68264== Jump to the invalid address stated on the next line
==68264==    at 0xA: ???
==68264==    by 0x4CA028A: __libc_start_main@@GLIBC_2.34 (libc-start.c:360)
==68264==    by 0x10A344: (below main) (in /path/to/minimal/program)
==68264==  Address 0xa is not stack'd, malloc'd or (recently) free'd
==68264== 
==68264== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

So very unexpectedly /opt/ros/rolling/lib/libbehaviortree_cpp.so appears, even though, the code does not include any BT.CPP headers. The only connection is, that the executable is linked against behaviortree_cpp::behaviortree_cpp. However, in my project, I link it to both ${TinyXML2_TARGETS} AND behaviortree_cpp::behaviortree_cpp.

After I try to link only against ${TinyXML2_TARGETS} instead of behaviortree_cpp::behaviortree_cpp while still find_package() both tyinyxml2_vendor AND behaviortree_cpp (I need tinyxml2 in my original application), I get undefined link errors:

undefined reference to `tinyxml2::XMLDocument::XMLDocument(bool, tinyxml2::Whitespace)
undefined reference to `tinyxml2::XMLDocument::~XMLDocument()

Maybe the local tinyxml2 copy in BT.CPP is interfering with the official ros2 vendor package when find_package()-ing both in one project?!

EDIT:
Yes indeed, after I link against the plain library path of the Ubuntu 24 version of tinyxml2 (version 10.0.0 as described by @mini-1235) the program runs without segfault! So the issue is definitely related to BT.CPP managing its own tinyxml2 dependency and it not being synchronized with the system version.

[LucienMorey]

Hopefully, I am not adding more noise...

Is it reasonable to add a depend tag for tinyxml2 to the package.xml and set USE_VENDORED_TINYXML2 to false if in an ament build system?

Things seem painful because the core of ROS2 depends on tinyxml2 through fastdds and other packages. That version will almost always conflict with the version here because of the range of OSs on which ROS is deployable. I figure if tinyxml2 is already in the core ROS installation, it's safe to add it as a dependency and let the ROS ecosystem pull in the version it needs for that particular distribution.

[facontidavide]

@LucienMorey

Yes, that is the solution. I am busy with ROSCon and life in general, so feel free to test and submit a PR

[LucienMorey]

Done the bits with #1033