chore(test): split container-backed unit tests into their own runner

Running `tox` over the full unit test suite was hanging on the first
container-backed plugin: py39 timed out after 20min on keycloak-version,
py310 got OOM-killed on cpu-usage, and py311 failed even earlier on
netifaces source-builds. The container tests are valuable but they are
integration tests, not multi-Python unit tests - they each build a
podman image per target OS, inject the plugin plus lib/, and exercise
the check against a live service.

- tools/run-unit-tests: add `--no-container` and `--only-container`
  mutually-exclusive flags. A plugin counts as container-based when
  its `unit-test/` dir has a `containerfiles/` subdirectory. Without
  a flag, behavior is unchanged (runs everything).
- tools/run-container-tests: new thin wrapper that calls
  `run-unit-tests --only-container` for the full podman-driven sweep.
- tox.ini: call `tools/run-unit-tests --no-container` so the
  multi-Python matrix only runs the 103 fast tests and skips the 7
  container-backed ones.

Split is 103 fast / 7 container (+ 1 skipped `example` template).

Author: markuslf

CHANGELOG.md

@@ -33,6 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * rocketchat-stats: add missing `lib.txt` import so the user-count pluralization no longer crashes with `AttributeError` at runtime
 * sap-open-concur-com: `choices=services.append('All')` was effectively `choices=None` because `list.append()` returns `None`. The `--service` argument therefore accepted any value without validation. Use `choices=services + ['All']` to build a proper choices list ([#1070](https://github.com/Linuxfabrik/monitoring-plugins/issues/1070))
 * starface-java-memory-usage: the heap and non-heap state checks used `state += lib.base.get_worst(used_state, state)` which adds state integers together and corrupts the accumulated state (e.g. WARN+WARN=2 reads as CRIT, WARN+CRIT=3 is outside the valid range). Replaced with `state = lib.base.get_worst(state, used_state)` so the accumulated state remains a valid STATE_OK / STATE_WARN / STATE_CRIT value ([#1070](https://github.com/Linuxfabrik/monitoring-plugins/issues/1070))
+* tools/run-unit-tests: add `--no-container` and `--only-container` flags so container-backed unit tests (podman-driven integration suites that take minutes per plugin) can be filtered out of the fast default run. `tools/run-container-tests` is a thin wrapper around `--only-container` for the full integration sweep. `tox` now invokes `tools/run-unit-tests --no-container` so the multi-Python matrix actually completes instead of hanging / OOM-killing on the first container stage
 * tools/run-unit-tests: skip the `example` plugin in default runs. Its unit test is a template meant to be copy-pasted into new plugins and does not correspond to a real check. Explicit `python tools/run-unit-tests example` still runs it
 * tox.ini: disable the sdist build (`no_package = true`) so `tox` no longer trips over the flat top-level layout with "Multiple top-level packages discovered". The repo is a collection of plugin scripts, not a Python package
 * tuned-profile, crypto-policy: make the "expected vs actual" branches structurally exclusive via an explicit `else:`. The old code relied on `lib.base.oao()` exiting to skip the follow-up WARN call, which works but is fragile to read ([#1070](https://github.com/Linuxfabrik/monitoring-plugins/issues/1070))

tools/run-container-tests

@@ -0,0 +1,42 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8; py-indent-offset: 4 -*-
+#
+# Author:  Linuxfabrik GmbH, Zurich, Switzerland
+# Contact: info (at) linuxfabrik (dot) ch
+#          https://www.linuxfabrik.ch/
+# License: The Unlicense, see LICENSE file.
+
+"""Discover and run all container-based plugin unit tests.
+
+A plugin's unit test counts as container-based when its `unit-test/`
+directory has a `containerfiles/` subdirectory. These tests build a
+podman image per target OS, inject the plugin plus `lib/`, and exercise
+the check against a live service. They take minutes per plugin and
+need podman on the host.
+
+This runner is a thin wrapper around `tools/run-unit-tests
+--only-container` so the two entry points (fast tests and container
+tests) are clearly separated. `tox` runs `tools/run-unit-tests
+--no-container` for multi-Python-version coverage; use this runner for
+a full integration sweep before a release.
+
+Usage:
+    python tools/run-container-tests                 # run all container tests
+    python tools/run-container-tests keycloak-version  # run one
+"""
+
+import os
+import subprocess
+import sys
+
+
+def main():
+    here = os.path.dirname(os.path.abspath(__file__))
+    runner = os.path.join(here, 'run-unit-tests')
+    argv = [sys.executable, runner, '--only-container', *sys.argv[1:]]
+    result = subprocess.run(argv, check=False)
+    sys.exit(result.returncode)
+
+
+if __name__ == '__main__':
+    main()

tools/run-unit-tests

@@ -11,12 +11,32 @@
 Each unit test is executed from its own directory (check-plugins/<name>/unit-test/)
 because the test files use relative paths for test data and plugin references.
 
+Unit tests come in two shapes:
+
+- **fast tests**: pure-Python tests that use `--test` fixtures and run in
+  well under a second. These are safe to run in the multi-Python-version
+  tox matrix.
+- **container tests**: tests that build a podman container per target OS,
+  inject the plugin plus `lib/`, and exercise the check against a live
+  service. They take minutes per plugin and need podman. They are not
+  safe for the tox matrix, and get their own runner
+  (`tools/run-container-tests`). A unit test counts as a container test
+  if the plugin's `unit-test/` directory has a `containerfiles/`
+  subdirectory.
+
 Usage:
-    python tools/run-unit-tests                 # run all
+    python tools/run-unit-tests                 # run everything
+    python tools/run-unit-tests --no-container  # skip container tests
+    python tools/run-unit-tests --only-container  # run only container tests
     python tools/run-unit-tests disk-smart      # run one plugin
     python tools/run-unit-tests disk-smart xml  # run specific plugins
+    python tools/run-unit-tests --no-container procs  # flags + plugins mix
+
+When specific plugins are requested on the command line, filters like
+`--no-container` still apply.
 """
 
+import argparse
 import os
 import subprocess
 import sys
@@ -27,27 +47,57 @@ import sys
 SKIP_PLUGINS = {'example'}
 
 
+def is_container_test(check_plugins, plugin):
+    """Return True if the plugin's unit test builds/runs containers."""
+    return os.path.isdir(
+        os.path.join(check_plugins, plugin, 'unit-test', 'containerfiles')
+    )
+
+
 def main():
     """Find and run unit tests, report results."""
+    parser = argparse.ArgumentParser(description=__doc__.strip().split('\n\n')[0])
+    group = parser.add_mutually_exclusive_group()
+    group.add_argument(
+        '--no-container',
+        action='store_true',
+        help='Skip container-based unit tests (anything with a containerfiles/ dir).',
+    )
+    group.add_argument(
+        '--only-container',
+        action='store_true',
+        help='Run only container-based unit tests. Useful for tools/run-container-tests.',
+    )
+    parser.add_argument(
+        'plugins',
+        nargs='*',
+        help='Optional list of plugin names to restrict the run to.',
+    )
+    args = parser.parse_args()
+
     root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
     check_plugins = os.path.join(root, 'check-plugins')
 
-    # filter to specific plugins if given as arguments
-    requested = sys.argv[1:] if len(sys.argv) > 1 else None
+    requested = set(args.plugins) if args.plugins else None
 
     # discover tests
     plugins = []
     for plugin in sorted(os.listdir(check_plugins)):
-        if requested and plugin not in requested:
+        if requested is not None and plugin not in requested:
             continue
         # when a plugin is explicitly requested on the command line, run it
         # even if it is normally skipped; otherwise honor the skip set
-        if plugin in SKIP_PLUGINS and not requested:
+        if plugin in SKIP_PLUGINS and requested is None:
             continue
-        unit_test_dir = os.path.join(check_plugins, plugin, 'unit-test')
-        run_file = os.path.join(unit_test_dir, 'run')
-        if os.path.isfile(run_file):
-            plugins.append(plugin)
+        run_file = os.path.join(check_plugins, plugin, 'unit-test', 'run')
+        if not os.path.isfile(run_file):
+            continue
+        container = is_container_test(check_plugins, plugin)
+        if args.no_container and container:
+            continue
+        if args.only_container and not container:
+            continue
+        plugins.append(plugin)
 
     if not plugins:
         print('No unit tests found.')

tox.ini

@@ -20,4 +20,7 @@ deps =
     vici
     xmltodict
 commands =
-    python tools/run-unit-tests {posargs}
+    # `--no-container` keeps tox focused on fast pure-Python unit tests so
+    # the multi-Python matrix actually completes. Run `tools/run-container-tests`
+    # separately for the container-backed integration suite (podman required).
+    python tools/run-unit-tests --no-container {posargs}