HAProxy Ingress 3.2.9: libblock_secrets.so segfaults HAProxy

[HaveFun83]

Deployment: helm
Chartversion: 1.52.0

We see the following warning messages within the haproxy log and and maybe there’s a relation to a problem with faulty WebSocket connections.

WARNING! thread 9 has stopped processing traffic for 123 milliseconds
    with 0 streams currently blocked, prevented from making any progress.
    While this may occasionally happen with inefficient configurations
    involving excess of regular expressions, map_reg, or heavy Lua processing,
    this must remain exceptional because the system's stability is now at risk.
    Timers in logs may be reported incorrectly, spurious timeouts may happen,
    some incoming connections may silently be dropped, health checks may
    randomly fail, and accesses to the CLI may block the whole process. The
    blocking delay before emitting this warning may be adjusted via the global
    'warn-blocked-traffic-after' directive. Please check the trace below for
    any clues about configuration elements that need to be corrected:

* Thread 9 : id=0x7d41394ffb30 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
      1/9    loops=5728 ctxsw=7564 stuck=0 prof=0 harmless=0 isolated=0
             cpu_ns: poll=178166898 now=302094658 diff=123927760
             curr_task=0x7d4136ea50c0 (task) calls=113 last=0
               fct=0x5b6b17959de0(process_chk) ctx=0x7d4139127498
             lock_hist: S:SERVER U:SERVER R:TASK_WQ U:TASK_WQ S:SERVER U:SERVER R:TASK_WQ U:TASK_WQ
             call trace(7):
             | 0x5b6b17953979 <48 89 df e8 c7 f4 ff ff]: ha_dump_backtrace+0x3449 > ha_dump_backtrace+0x2910
             | 0x5b6b17a466a3 <00 00 00 e8 fd d1 f0 ff]: main+0x311863 > ha_dump_backtrace+0x3370
             | 0x7d413bb41ec2 <31 d2 e9 8c ff ff ff 90]: main+0x21d62440d082
             | 0x5b6b179f504c <4e 30 01 e8 34 f3 ff ff]: process_runnable_tasks+0x3fc > run_tasks_from_lists
             | 0x5b6b1796559a <02 00 00 e8 b6 f6 08 00]: run_poll_loop+0x8a > process_runnable_tasks
             | 0x5b6b17965d91 <00 00 00 e8 7f f7 ff ff]: run_poll_loop+0x881 > run_poll_loop
             | 0x7d413bb5029a <0f 05 48 8b 7b 08 ff 13]: main+0x21d62441b45a
 => Trying to gracefully recover now (pid 89083).

WARNING! thread 24 has stopped processing traffic for 110 milliseconds
    with 0 streams currently blocked, prevented from making any progress.
    While this may occasionally happen with inefficient configurations
    involving excess of regular expressions, map_reg, or heavy Lua processing,
    this must remain exceptional because the system's stability is now at risk.
    Timers in logs may be reported incorrectly, spurious timeouts may happen,
    some incoming connections may silently be dropped, health checks may
    randomly fail, and accesses to the CLI may block the whole process. The
    blocking delay before emitting this warning may be adjusted via the global
    'warn-blocked-traffic-after' directive. Please check the trace below for
    any clues about configuration elements that need to be corrected:

* Thread 24: id=0x7d41378d2b30 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
      1/24   loops=6179 ctxsw=7565 stuck=0 prof=0 harmless=0 isolated=0
             cpu_ns: poll=191400471 now=302138495 diff=110738024
             curr_task=0x7d4136e9b2a0 (task) calls=225 last=0
               fct=0x5b6b17959de0(process_chk) ctx=0x7d4139012498
             lock_hist: U:SERVER R:TASK_WQ U:TASK_WQ S:SERVER U:SERVER R:TASK_WQ U:TASK_WQ S:SERVER locked: SERVER(S)
             call trace(12):
             | 0x5b6b17953979 <48 89 df e8 c7 f4 ff ff]: ha_stuck_warning+0xd9 > ha_dump_backtrace+0x2910
             | 0x5b6b17a466a3 <00 00 00 e8 fd d1 f0 ff]: main+0x311863 > ha_dump_backtrace+0x3370
             | 0x7d413bb41ec2 <31 d2 e9 8c ff ff ff 90]: main+0x21d62440d082
             | 0x5b6b17940cc2 <48 89 e5 e8 0e 16 08 00]: cli_io_handler+0x2e362 > main+0x28d490
             | 0x5b6b17940d87 <00 31 c0 e8 29 ff ff ff]: cli_io_handler+0x2e427 > cli_io_handler+0x2e350
             | 0x5b6b179426d3 <83 ec 08 e8 3d e6 ff ff]: cli_io_handler+0x2fd73 > cli_io_handler+0x2e3b0
             | 0x5b6b179599af <4c 89 f7 e8 11 8d fe ff]: ha_dump_backtrace+0x947f > cli_io_handler+0x2fd60
             | 0x5b6b179f472c <84 ff 01 00 00 41 ff d1]: run_tasks_from_lists+0x3ac
             | 0x5b6b179f504c <4e 30 01 e8 34 f3 ff ff]: process_runnable_tasks+0x3fc > run_tasks_from_lists
             | 0x5b6b1796559a <02 00 00 e8 b6 f6 08 00]: run_poll_loop+0x8a > process_runnable_tasks
             | 0x5b6b17965d91 <00 00 00 e8 7f f7 ff ff]: run_poll_loop+0x881 > run_poll_loop
             | 0x7d413bb5029a <0f 05 48 8b 7b 08 ff 13]: main+0x21d62441b45a
 => Trying to gracefully recover now (pid 89083).

WARNING! thread 32 has stopped processing traffic for 122 milliseconds
    with 0 streams currently blocked, prevented from making any progress.
    While this may occasionally happen with inefficient configurations
    involving excess of regular expressions, map_reg, or heavy Lua processing,
    this must remain exceptional because the system's stability is now at risk.
    Timers in logs may be reported incorrectly, spurious timeouts may happen,
    some incoming connections may silently be dropped, health checks may
    randomly fail, and accesses to the CLI may block the whole process. The
    blocking delay before emitting this warning may be adjusted via the global
    'warn-blocked-traffic-after' directive. Please check the trace below for
    any clues about configuration elements that need to be corrected:

* Thread 32: id=0x7d413318db30 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
      1/32   loops=7540 ctxsw=10663 stuck=0 prof=0 harmless=0 isolated=0
             cpu_ns: poll=179103693 now=301663166 diff=122559473
             curr_task=0x7d4136e99860 (task) calls=225 last=0
               fct=0x5b6b17959de0(process_chk) ctx=0x7d4138fe8498
             lock_hist: U:SERVER R:TASK_WQ U:TASK_WQ S:SERVER U:SERVER R:TASK_WQ U:TASK_WQ S:SERVER locked: SERVER(S)
             call trace(12):
             | 0x5b6b17953979 <48 89 df e8 c7 f4 ff ff]: ha_stuck_warning+0xd9 > ha_thread_dump_one
             | 0x5b6b17a466a3 <00 00 00 e8 fd d1 f0 ff]: wdt_handler+0x1e3 > ha_stuck_warning
             | 0x7d413bb41ec2 <31 d2 e9 8c ff ff ff 90]: ld-musl-x86_64:+0x4dec2
             | 0x5b6b17940cc2 <48 89 e5 e8 0e 16 08 00]: sedesc_new+0x12 > __pool_alloc
             | 0x5b6b17940d87 <00 31 c0 e8 29 ff ff ff]: cli_io_handler+0x2e427 > sedesc_new
             | 0x5b6b179426d3 <83 ec 08 e8 3d e6 ff ff]: sc_new_from_check+0x13 > cli_io_handler+0x2e3b0
             | 0x5b6b179599af <4c 89 f7 e8 11 8d fe ff]: process_chk_conn+0x104f > sc_new_from_check
 => Trying to gracefully recover now (pid 89083).

Maybe someone can help or had an advise howto mitigate or fix this?
Thanks in advance

[BuJo]

haproxy version is 3.2.19-06868cd6e / haproxytech/kubernetes-ingress:3.2.9

We found and fixed those "has stopped processing". The problem in a Kubernetes Cluster is, that all CPU cores are visible to HAProxy. This means, it will start one thread per core. In a big cluster, this means that even if using a cgroup limit of 1 CPU, all the notes of the performance tuning for big systems apply.
Compounding that is that if the Kubernetes Cluster is run on a VM instead of bare metal, the Level 3 cache layout cannot be known, it is flattened.

The solution was to simply set the nbthread to 1, which is completely sufficient for our case.

We traced the crashing WebSocket connections to segfaulting instances of HAProxy (we didn't see those over all the other logs at first):

[NOTICE]   (278) : haproxy version is 3.2.19-06868cd6e
[ALERT]    (278) : Current worker (290) exited with code 139 (Segmentation fault)
[WARNING]  (278) : A worker process unexpectedly died and this can only be explained by a bug in haproxy or its dependencies.
Please check that you are running an up to date and maintained version of haproxy and open a bug report.
[ALERT]    (278) : exit-on-failure: killing every processes with SIGTERM

Tracing into that is hard, as I'm not as proficient in K8s as if I had a VM.

Program received signal SIGSEGV, Segmentation fault.
0x00007abf2d3244a7 in ?? ()
(gdb) bt full
#0  0x00007abf2d3244a7 in ?? ()
No symbol table info available.
#1  0x00007abf2d2c3996 in ?? ()
No symbol table info available.
#2  0x0000000000000000 in ?? ()

The memory map for that process:

7abf2d2c2000-7abf2d2c3000 r--p 00000000 00:175 205737768                 /usr/local/lib/libblock_secrets.so
7abf2d2c3000-7abf2d2c6000 r-xp 00001000 00:175 205737768                 /usr/local/lib/libblock_secrets.so
7abf2d2c6000-7abf2d2c7000 r--p 00004000 00:175 205737768                 /usr/local/lib/libblock_secrets.so
7abf2d2c7000-7abf2d2c8000 r--p 00005000 00:175 205737768                 /usr/local/lib/libblock_secrets.so
7abf2d2c8000-7abf2d2c9000 rw-p 00006000 00:175 205737768                 /usr/local/lib/libblock_secrets.so
7abf2d2c9000-7abf2d2cb000 rw-p 00000000 00:00 0 
7abf2d2cb000-7abf2d2df000 r--p 00000000 00:175 205737560                 /lib/ld-musl-x86_64.so.1
7abf2d2df000-7abf2d333000 r-xp 00014000 00:175 205737560                 /lib/ld-musl-x86_64.so.1
7abf2d333000-7abf2d369000 r--p 00068000 00:175 205737560                 /lib/ld-musl-x86_64.so.1
7abf2d369000-7abf2d36a000 r--p 0009d000 00:175 205737560                 /lib/ld-musl-x86_64.so.1
7abf2d36a000-7abf2d36b000 rw-p 0009e000 00:175 205737560                 /lib/ld-musl-x86_64.so.1

Checking Frame #0 Address...
Address 0x00007abf2d3244a7 found in region:
  Range:  0x7abf2d2df000-0x7abf2d333000
  Mapping: 00014000 00:175 205737560 /lib/ld-musl-x86_64.so.1

Checking Frame #1 Address...
Address 0x00007abf2d2c3996 found in region:
  Range:  0x7abf2d2c3000-0x7abf2d2c6000
  Mapping: 00001000 00:175 205737768 /usr/local/lib/libblock_secrets.so

Using strace on a dying process:

strace output

``` close(22) = 0 --- SIGALRM {si_signo=SIGALRM, si_code=SI_TIMER, si_timerid=0, si_overrun=0, si_int=0, si_ptr=NULL} --- timer_settime(0, 0, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=100000000}}, NULL) = 0 rt_sigreturn({mask=[]}) = 516 --- SIGALRM {si_signo=SIGALRM, si_code=SI_TIMER, si_timerid=0, si_overrun=0, si_int=0, si_ptr=NULL} --- rt_sigprocmask(SIG_BLOCK, [], [ALRM URG], 8) = 0 rt_sigprocmask(SIG_SETMASK, [ALRM URG], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [ALRM URG], [ALRM URG], 8) = 0 readlink("\220\316f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\220\316f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\213\5!t(", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\213\5!t(", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\360\316f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\360\316f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\213\25?\374\21", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\213\25?\374\21", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\200\334f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\200\334f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("H\307\300\17", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("H\307\300\17", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\20\335f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\20\335f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("A)\307\205\300tIE\205\377~DH\213\5!O\25", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("A)\307\205\300tIE\205\377~DH\213\5!O\25", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\200\335f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\200\335f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\203=\303X'", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\203=\303X'", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\360\335f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\360\335f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("L\213%\250\5 ", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("L\213%\250\5 ", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("P\340f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("P\340f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("1\300\350\371\273-", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("1\300\350\371\273-", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\230\340f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\230\340f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\211\307\350vy\377\377ATHc\306I\211\374UH\215|\302\10H\211\305SH\2132H\211\323\350\20\376\377\377H\211\332\211\356L\211\347H\215\5\242\377\377\377[]A\\\377\340AVdL\2134%", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\211\307\350vy\377\377ATHc\306I\211\374UH\215|\302\10H\211\305SH\2132H\211\323\350\20\376\377\377H\211\332\211\356L\211\347H\215\5\242\377\377\377[]A\\\377\340AVdL\2134%", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("\n\375f:\375\177", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("\n\375f:\375\177", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) readlink("-sf", 0x7ffd3a6677c0, 4096) = -1 ENOENT (No such file or directory) stat("-sf", 0x7ffd3a66a7e0) = -1 ENOENT (No such file or directory) stat("/usr/local/sbin/haproxy", {st_mode=S_IFREG|0755, st_size=26463104, ...}) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} --- +++ killed by SIGSEGV (core dumped) +++ ```

It looks like memory corruption after the ALRM. Some of the raw bytes can be decoded to addresses, ending with argv.

Replicating all that is hard, the crashes happen only intermittently and it's more likely to end the worker processes with SIGUSR1 for reloading. And all the binaries are stripped. I haven't found a non-alpine debug container or I would try that to capture a better stacktrace.

I've never seen such a behavior on a VM with HAProxy ever, so something must be different. jemalloc, musl, or the libblock_secrets.so which I'm suspecting right now. That, or the hardware is failing (which I don't think so at the moment, as the other Kubernetes workload is unaffected).

Current working theory:

• SIGALRM happens
• The watchdog gets triggered wdt_handler()
• At one point it goes to ha_dump_backtrace() calling my_backtrace() which calls may_access()
• may_access() calls stat() internally with random addresses, which explains the stat calls from the strace having odd input.
• The block_secrets.c has real_stat = dlsym(RTLD_NEXT, "stat"); which might somehow affect the workings of the may_access(), because it calls path_blocked() on that random ram contents.

[BuJo]

Looking further into printing the backtrace, another image with glibc I would not have found that issue probably, as HA_HAVE_WORKING_BACKTRACE would have used another way to construct the backtrace and would not have tripped the stat() calls.

[BuJo]

We've been running the patched libblock_secrets.so since yesterday, the segfaults crashing the haproxy worker disappeared.

In case someone has the same problems, here's a Dockerfile to compile and build a patched image.

Dockerfile

```dockerfile FROM haproxytech/haproxy-alpine:3.2 AS builder-c RUN apk add --no-cache build-base gcc musl-dev WORKDIR /src

COPY block_secrets.c .
RUN gcc -O3 -std=c11 -pipe
-Wall -Wextra -Wformat=2 -Wformat-security
-Wnull-dereference -Wshadow -Wstrict-prototypes
-D_FORTIFY_SOURCE=2
-fstack-protector-strong -fstack-clash-protection
-fno-omit-frame-pointer
-flto -fPIC -shared -s
-Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack -Wl,--as-needed -Wl,-z,defs
-o libblock_secrets.so block_secrets.c -ldl

FROM haproxytech/kubernetes-ingress:3.2.9

Overwrite only the breaking preloaded library

COPY --from=builder-c /src/libblock_secrets.so /usr/local/lib/libblock_secrets.so

</details>

[vholer]

We are also experiencing segfaulting HAProxy recently, in our case it's easily reproducible by using custom configuration code with userlist and involve http_auth_group(...) for basic authentication in the backend. Every request where the authentication is involved segfaults (and restarts) HAproxy.

[pid 56496] stat("ZZZZZZZ", 0x7ffd7d41b460) = -1 ENOENT (No such file or directory)
[pid 56496] stat("", 0x7ffd7d41b460)    = -1 ENOENT (No such file or directory)
[pid 56496] readlink("YYYYYYY", 0x7ffd7d418440, 4096) = -1 ENOENT (No such file or directory)
[pid 56496] stat("YYYYYYY", 0x7ffd7d41b460) = -1 ENOENT (No such file or directory)
[pid 56496] readlink("XXXXXX", 0x7ffd7d418440, 4096) = -1 ENOENT (No such file or directory)
[pid 56496] stat("XXXXX", 0x7ffd7d41b460) = -1 ENOENT (No such file or directory)
[pid 56496] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
[pid 56497] +++ killed by SIGSEGV (core dumped) +++
+++ killed by SIGSEGV (core dumped) +++

Last working Helm chart release 1.49.0 with controller 3.2.6 (HAProxy 3.2.12).

Newer releases segfault:

• 1.52.0 (controller 3.2.9)
• 1.51.1 (controller 3.2.8)

[dkorunic]

This looks interesting and so far all points to libblock_secrets. We'll investigate and fix as soon as possible.

[dkorunic]

Root cause: Commit e6c0123 expanded the block_secrets LD_PRELOAD interposer from the open() family (6 syscalls) to ~50 libc calls, including metadata-only syscalls (stat, lstat, fstatat, statx, access, faccessat, readlink, opendir) that HAProxy calls constantly on its hot path — including inside process_chk while holding the per-SERVER lock.
Each intercepted call runs is_blocked(), whose slow path calls realpath(), which is blocking and non-async-signal-safe. Executing realpath() in HAProxy's event loop stalled threads on the SERVER lock, tripping the watchdog.

Secondary issue: On musl libc (Alpine-based images), the LFS64 aliases (open64, stat64, fstatat64, truncate64, renameat2) are not exported as distinct symbols. dlsym(RTLD_NEXT, ...) returned NULL, causing the exported *64 hooks to return ENOSYS and re-dlsym() on every call (holding the global loader lock).

Fix (commit 6a5241f4):

1. Narrowed interception back to calls that actually read or mutate file contents (open/openat/fopen/freopen/creat and write hooks). Metadata-only calls now bind straight to libc and never detour through realpath().
2. In the constructor, alias NULL *64 pointers to their base counterparts (ABI-identical on 64-bit), and special-case renameat2(flags==0) → renameat.
3. Added behavioural tests (block_secrets_test.{c,sh}) covering the security invariant and both regressions, runnable under the musl/Alpine build image.

Security impact of the fix: None. Reading the service-account token requires open(); metadata calls only leak existence, which was already observable. The security invariant is preserved and verified by the new test suite.

MR with fix is in the queue.