2.16.0 icinga2 constantly crashes on secondary master with enabled graphite feature

[legioner0]

After upgrading Icinga2 to 2.16.0, process icinga2 starts to constantly crashes on secondary master. Master master node and all connected agents working correctly.

From /var/log/icinga2/error.log:

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::PerfdataWriterConnection]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2026-05-08 17:20:06 +0300

From /var/log/icinga2/crash/report.1778250006.600936:

Caught SIGABRT.
Current time: 2026-05-08 17:20:06 +0300

  Application version: r2.16.0-1

System information:
  Platform: Ubuntu
  Platform version: 24.04.4 LTS (Noble Numbat)
  Kernel: Linux
  Kernel version: 6.8.0-111-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 13.3.0
  Build host: runner-aa2jha3hm-project-575-concurrent-7
  OpenSSL version: OpenSSL 3.0.13 30 Jan 2024

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Stacktrace:
 0# icinga::Application::SigAbrtHandler(int) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x0000756677245330 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
 3# gsignal in /lib/x86_64-linux-gnu/libc.so.6
 4# abort in /lib/x86_64-linux-gnu/libc.so.6
 5# 0x000075667722881B in /lib/x86_64-linux-gnu/libc.so.6
 6# 0x000075667723B517 in /lib/x86_64-linux-gnu/libc.so.6
 7# 0x00005BB53A819917 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 8# icinga::GraphiteWriter::StatsFunc(boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::Array> const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 9# std::_Function_handler<icinga::Value (std::vector<icinga::Value, std::allocator<icinga::Value> > const&), icinga::WrapFunction<void (*)(boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::Array> const&)>(void (*)(boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::Array> const&))::{lambda(std::vector<icinga::Value, std::allocator<icinga::Value> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<icinga::Value, std::allocator<icinga::Value> > const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
10# icinga::Function::Invoke(std::vector<icinga::Value, std::allocator<icinga::Value> > const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
11# icinga::CIB::GetFeatureStats() in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
12# icinga::IcingaCheckTask::ScriptFunc(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
13# std::_Function_handler<icinga::Value (std::vector<icinga::Value, std::allocator<icinga::Value> > const&), icinga::WrapFunction<void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool)>(void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool))::{lambda(std::vector<icinga::Value, std::allocator<icinga::Value> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<icinga::Value, std::allocator<icinga::Value> > const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
14# icinga::Function::Invoke(std::vector<icinga::Value, std::allocator<icinga::Value> > const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
15# icinga::CheckCommand::Execute(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
16# icinga::Checkable::ExecuteRemoteCheck(boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
17# icinga::ClusterEvents::ExecuteCheckFromQueue(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
18# icinga::ClusterEvents::RemoteCheckThreadProc() in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
19# 0x00007566776ECDB4 in /lib/x86_64-linux-gnu/libstdc++.so.6
20# 0x000075667729CAA4 in /lib/x86_64-linux-gnu/libc.so.6
21# 0x0000756677329C6C in /lib/x86_64-linux-gnu/libc.so.6

...
Thread 2 (Thread 0x7566766746c0 (LWP 3969373) "Remote Check Sc"):
#0  0x0000756677310813 in wait4 () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00005bb53a14cef0 in icinga::Application::AttachDebugger(icinga::String const&, bool) ()
No symbol table info available.
#2  0x00005bb53a15695c in icinga::Application::SigAbrtHandler(int) ()
No symbol table info available.
#3  <signal handler called>
No symbol table info available.
#4  0x000075667729eb2c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x000075667724527e in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#6  0x00007566772288ff in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#7  0x000075667722881b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#8  0x000075667723b517 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#9  0x00005bb53a819917 in ?? ()
No symbol table info available.
#10 0x00005bb53a76fe9c in icinga::GraphiteWriter::StatsFunc(boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::Array> const&) ()
No symbol table info available.
#11 0x00005bb53a147b13 in std::_Function_handler<icinga::Value (std::vector<icinga::Value, std::allocator<icinga::Value> > const&), icinga::WrapFunction<void (*)(boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::Array> const&)>(void (*)(boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::Array> const&))::{lambda(std::vector<icinga::Value, std::allocator<icinga::Value> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<icinga::Value, std::allocator<icinga::Value> > const&) ()
No symbol table info available.
#12 0x00005bb53a1862d6 in icinga::Function::Invoke(std::vector<icinga::Value, std::allocator<icinga::Value> > const&) ()
No symbol table info available. 
#13 0x00005bb53a4bef79 in icinga::CIB::GetFeatureStats() ()
No symbol table info available. 
#14 0x00005bb53a5f9286 in icinga::IcingaCheckTask::ScriptFunc(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) ()
No symbol table info available. 
#15 0x00005bb53a60319c in std::_Function_handler<icinga::Value (std::vector<icinga::Value, std::allocator<icinga::Value> > const&), icinga::WrapFunction<void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool)>(void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool))::{lambda(std::vector<icinga::Value, std::allocator<icinga::Value> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<icinga::Value, std::allocator<icinga::Value> > const&) ()
No symbol table info available. 
#16 0x00005bb53a1862d6 in icinga::Function::Invoke(std::vector<icinga::Value, std::allocator<icinga::Value> > const&) ()
No symbol table info available.
#17 0x00005bb53a4b93e2 in icinga::CheckCommand::Execute(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) ()
No symbol table info available.
#18 0x00005bb53a4b962f in icinga::Checkable::ExecuteRemoteCheck(boost::intrusive_ptr<icinga::WaitGroup> const&, boost::intrusive_ptr<icinga::Dictionary> const&) ()
No symbol table info available. 
#19 0x00005bb53a4dbb79 in icinga::ClusterEvents::ExecuteCheckFromQueue(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) ()
No symbol table info available. 
#20 0x00005bb53a4cab4f in icinga::ClusterEvents::RemoteCheckThreadProc() ()
No symbol table info available. 
#21 0x00007566776ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#22 0x000075667729caa4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available. 
#23 0x0000756677329c6c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Package version

# dpkg -l icinga2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version              Architecture Description
+++-==============-====================-============-==================================
ii  icinga2        2.16.0-1+ubuntu24.04 amd64        host and network monitoring system

Workaround

icinga2 feature disable graphite

[jschmidt-icinga]

I'm aware of this issue. I'll make a PR with a fix soon.

[julianbrost]

For 2.16.1, this will be implicitly fixed by #10851 which reverts more changes to the writers due to other more serious issues that still need to be investigated in detail. This issue will remain open until it's also fixed in the master branch (which is the lower priority at the moment).