Skip to content

7.1 regression: server fails to start on noexec data dir (oshi/JNA UnsatisfiedLinkError) #26471

Description

@patrickmann

Summary

On 7.1, the server fails to start on hosts where the Graylog data directory (<data_dir>, e.g. /var/lib/graylog-server) lives on a noexec-mounted filesystem. Startup aborts during Guice injector creation with:

NoClassDefFoundError: Could not initialize class oshi.software.os.linux.LinuxOperatingSystemJNA
...
Caused by: UnsatisfiedLinkError: /var/lib/graylog-server/libnative/jna/jnaXXXX.tmp: failed to map
segment from shared object

This is a regression introduced in 7.1.0 — the same host configuration starts fine on 7.0.

Root cause

NodeMetricPeriodical (new in 7.1.0, #24493) eagerly creates a CpuLoadGauge in a field initializer:

// NodeMetricPeriodical.java
private final CpuLoadGauge cpuLoadGauge = new CpuLoadGauge();

CpuLoadGauge in turn touches OSHI native code in its own field initializer:

// CpuLoadGauge.java
private long[] lastTicks = processor().getSystemCpuLoadTicks();   // -> new
SystemInfo().getHardware().getProcessor()

Because NodeMetricPeriodical is a @Singleton Periodical (eagerly instantiated when Guice builds the injector), this forces OSHI to load its native library at startup. OSHI/JNA unpacks the native .so into <data_dir>/libnative/jna (we deliberately moved jna.tmpdir there in #21223) and dlopen()s it. On a noexec mount the kernel refuses to map the executable segments -> UnsatisfiedLinkError -> injector creation fails -> the JVM never starts.

7.0 had no boot-time consumer that forced the OSHI native load, so the same noexec data dir was tolerated. 7.1 turns a latent, harmless misconfiguration into a hard boot failure.

noexec on /var, /var/lib, /tmp, etc. is common on CIS-benchmarked / STIG-hardened systems, so this is likely to hit security-conscious customers specifically.

Impact

  • Server will not start at all (not a degraded-metrics situation — total boot failure).
  • Affects hardened/noexec deployments upgrading 7.0 -> 7.1.

Workaround

Point the native lib dir at an exec-capable location via JVM options:

-Djna.tmpdir=/opt/graylog/libnative/jna -Dio.netty.native.workdir=/opt/graylog/libnative

(directory must be writable by the graylog user and on a non-noexec mount).

Proposed fix

Node metrics are a nice-to-have; their unavailability must not prevent startup.

  1. Make OSHI/CpuLoadGauge initialization lazy and guarded — wrap the native access in try/catch, log a single warning, and disable the CPU-load metric (return null/NaN) instead of propagating the error.
  2. Remove the eager field initializers (new CpuLoadGauge() in NodeMetricPeriodical, and lastTicks = processor()... in CpuLoadGauge) so nothing forces the native load during injector creation; defer to first periodical run.
  3. Consider applying the same defensive guard to OshiService so any OSHI-backed startup path degrades gracefully on noexec/restricted hosts.

Affected versions

7.1.0 and later. Not present in 7.0.x.

References

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions