2.16.0 fails to build due to perfdata test failures

[sebastic]

Describe the bug

The Debian package fails to build due to test failures.

In the Salsa CI (GitLab) environment perfdata_connection/stuck_reading_response fails often:

251/254 Test #246: perfdata_connection/stuck_reading_response .................................***Failed   10.07 sec
Running 1 test case...
./test/perfdata-perfdatatargetfixture.hpp(83): fatal error: in "perfdata_connection/stuck_reading_response": Exception in shutdown(): shutdown: stream truncated [asio.ssl.stream:1]
terminate called after throwing an instance of 'boost::execution_aborted'
unknown location(0): fatal error: in "perfdata_connection/stuck_reading_response": signal: SIGABRT (application abort requested)
./test/perfdata-perfdatatargetfixture.hpp(83): last checkpoint
*** 2 failures are detected in the test module "icinga2"

https://salsa.debian.org/nagios-team/icinga2/-/jobs/9456641
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9456640
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9456642
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9456643

This test seems to be flaky as it doesn't always fail:
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9456643 (first run: failed)
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9456639 (second run: succeeded)

On the build daemons more tests fail on certain architectures:

armhf

The following tests FAILED:
	231 - perfdata_elasticsearchwriter/connect (Failed)     network perfdata
	248 - perfdata_connection/http_send_retry (Failed)      network perfdata
Errors while running CTest

https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=armhf&ver=2.16.0-1&stamp=1776886720&raw=0
The failures are not the same for the prior build:
https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=armhf&ver=2.16.0-1&stamp=1776874475&raw=0

ppc64el

The following tests FAILED:
	231 - perfdata_elasticsearchwriter/connect (Failed)     network perfdata
	232 - perfdata_elasticsearchwriter/pause_with_pending_work (SEGFAULT) network perfdata
	233 - perfdata_gelfwriter/connect (Failed)              network perfdata
	234 - perfdata_gelfwriter/pause_with_pending_work (Failed) network perfdata
	235 - perfdata_graphitewriter/connect (Failed)          network perfdata
	236 - perfdata_graphitewriter/pause_with_pending_work (SEGFAULT) network perfdata
	237 - perfdata_influxdbwriter/connect (Failed)          network perfdata
	238 - perfdata_influxdbwriter/pause_with_pending_work (Timeout) network perfdata
	239 - perfdata_opentsdbwriter/connect (Failed)          network perfdata
	240 - perfdata_opentsdbwriter/pause_with_pending_work (Subprocess aborted) network perfdata
	241 - perfdata_connection/connection_refused (Failed)   network perfdata
	242 - perfdata_connection/ensure_connected (Failed)     network perfdata
	243 - perfdata_connection/finish_during_timeout (Failed) network perfdata
	244 - perfdata_connection/stuck_in_handshake (Failed)   network perfdata
	246 - perfdata_connection/stuck_reading_response (Failed) network perfdata
	247 - perfdata_connection/reconnect_failed (Failed)     network perfdata
	248 - perfdata_connection/http_send_retry (Failed)      network perfdata
Errors while running CTest

https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=ppc64el&ver=2.16.0-1&stamp=1776888375&raw=0
The failures are not the same for the prior build:
https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=ppc64el&ver=2.16.0-1&stamp=1776875844&raw=0

s390x

The following tests FAILED:
	231 - perfdata_elasticsearchwriter/connect (SEGFAULT)   network perfdata
Errors while running CTest

https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=s390x&ver=2.16.0-1&stamp=1776886785&raw=0
The failures are not the same for the prior build:
https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=s390x&ver=2.16.0-1&stamp=1776875153&raw=0

I've added a patch to disable these tests: perfdata-tests.patch

To Reproduce

1. Build the Debian package

Expected behavior

No test failures

Screenshots

N/A

Your Environment

• Version used (icinga2 --version): 2.16.0
• Operating System and version: Debian unstable

[jschmidt-icinga]

I apologize for the inconvenience. We've had some problems with theses tests, especially on non x86 architectures.
Can you test and see if it works with the additional fixes in #10780? 6777d71 and fa1c1d2 are the most relevant.

Edit: Added another commit 4ca38d5 that should fix the stream truncated errors. I already had looked at that fix before, but thought it was unnecessary because I never saw that error again after fixing the other issues.

[yhabteab]

Just noticed in the output, yeah, the tests seem to run in parallel.

dh_auto_test
cd obj-s390x-linux-gnu && make -j2 test ARGS+=--verbose ARGS+=-j2
make[2]: Entering directory '/build/reproducible-path/icinga2-2.16.0/obj-s390x-linux-gnu'
Running tests...
/usr/bin/ctest --verbose -j2

If the commits shared by @jschmidt-icinga don't help, maybe running the tests sequentially could fix the issue for the time being.

[sebastic]

Can you test and see if it works with the additional fixes in #10780?

With the first 4 commits, the tests still fail on armhf & s390x at least:

https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=armhf&ver=2.16.0-3%7Eexp1&stamp=1776935164&raw=0
https://buildd.debian.org/status/fetch.php?pkg=icinga2&arch=s390x&ver=2.16.0-3%7Eexp1&stamp=1776935094&raw=0
https://buildd.debian.org/status/package.php?p=icinga2&suite=experimental

The CI jobs succeeded:

https://salsa.debian.org/nagios-team/icinga2/-/pipelines/1071961

With the 5th commit CI jobs are failing again:

https://salsa.debian.org/nagios-team/icinga2/-/jobs/9461561
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9461568
https://salsa.debian.org/nagios-team/icinga2/-/pipelines/1072015

[jschmidt-icinga]

It seems you followed @yhabteab's suggestion and disabled parallel testing. Might be best for now. I'll try to fix these test cases until 2.16.1.

I'm not sure this will fix the weird stack smashing issues on armhf and s390x, though. I wasn't even aware of those and they seem unrelated to the immediate test cases they occur in. I'll try and see if I can reproduce and fix them in our own infrastructure.

Edit: I can reproduce the armhf issue and will further debug it when I have the time.

[sebastic]

maybe running the tests sequentially could fix the issue for the time being.

It does not, still various perfdata failures on armhf, ppc64el, and s390x at least:

https://buildd.debian.org/status/package.php?p=icinga2&suite=experimental

[jschmidt-icinga]

I found the issue for the tests failing on armhf (and hopefully the two others as well). It was related to boost coroutines not meshing well with the thread local storage used in std::promise, something similar to what we discovered in #10789. (Edit: For a more detailed description see the commit message of 1b568ac)

@sebastic Can you please give the current state of #10780 a shot when you have time?

[sebastic]

Still flaky tests in CI:

https://salsa.debian.org/nagios-team/icinga2/-/jobs/9500792
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9500794
https://salsa.debian.org/nagios-team/icinga2/-/jobs/9500799

The ~exp4 revision was uploaded 6 hours ago but has still not hit the archive, keep an eye on the buildd status where failures are still expected:

https://buildd.debian.org/status/package.php?p=icinga2&suite=experimental

[jschmidt-icinga]

@sebastic Thanks for your help. At least the SIGSEGV are now entirely gone it seams. (Edit: No, the PPC build still shows similar errors, so I'll have to do another debugging run on emulated ppc64le.)

The remaining issues are the truncated stream errors in perfdata_connection/stuck_reading_response, which I swear should've been fixed in 0839cf8 (they were everywhere I could reproduce them) and those weird hangs in every test case on ppc64le I haven't seen before anywhere. I'll look into both and report back.