test: iterate platform lists via lib.lftest.attach_each for accurate test counts

Convert the last 7 run files that still used a plain for/subTest loop
(apache-httpd-status, cpu-usage, keycloak-memory-usage, keycloak-stats,
keycloak-version, logfile, mysql-connections) to lib.lftest.attach_each
so each platform, image or scenario shows up as its own unittest test
method. Plain subTest loops collapse into a single method and make
./run report "Ran 1 test" regardless of actual coverage.

Document both the TESTS-list pattern (attach_tests) and the
platform-list pattern (attach_each) in CONTRIBUTING.md, and update the
container-based test example to match.

Author: markuslf

CONTRIBUTING.md

@@ -693,7 +693,7 @@ pylint check-plugins/my-check/my-check
 
 ### Unit Tests
 
-Unit tests are implemented using the `unittest` framework (<https://docs.python.org/3/library/unittest.html>) with a declarative, data-driven approach. Test definitions are a list of dicts, executed via `lib.lftest.run()` and `unittest.subTest()`. See the [example](check-plugins/example/unit-test/run) plugin for the reference implementation.
+Unit tests are implemented using the `unittest` framework (<https://docs.python.org/3/library/unittest.html>) with a declarative, data-driven approach. Test definitions are a list of dicts (or a list of platform/image items for container tests), materialised into one real `unittest` test method per item via `lib.lftest.attach_tests()` or `lib.lftest.attach_each()`. See the [example](check-plugins/example/unit-test/run) plugin for the reference implementation.
 
 
 #### Test directory structure
@@ -802,6 +802,20 @@ Available assertion keys in each testcase dict:
 * `assert-stderr` (`str`, optional): Expected stderr content. Default: `''`.
 
 
+#### Iterating over TESTS vs. platforms
+
+Two iteration shapes show up in the test files, and each has its own helper. Both materialise one real `unittest` test method per item so `./run` reports an accurate count and `./run -v` names every case.
+
+* **TESTS list** (the default): a list of testcase dicts executed by `lib.lftest.run()`. Use `lib.lftest.attach_tests(TestCheck, TESTS)`. This is the right shape for everything that injects fixture data via `--test=stdout/...`.
+* **Platform list** (container-based tests): a list of images, Containerfiles, or scenario dicts where each item needs its own setup (spin up a container, build an image, reset a cache DB). Use `lib.lftest.attach_each(TestCheck, ITEMS, action, id_func=...)` with an `action(test, item)` callable that does the per-item work. The `id_func` turns one item into the test method name. Examples in-tree:
+
+    * `check-plugins/mysql-connections/unit-test/run` iterates over an `IMAGES` list of `(image, label)` tuples, with `id_func=lambda it: it[1]`.
+    * `check-plugins/cpu-usage/unit-test/run` iterates over a `CONTAINERFILES` list of strings, with the default `id_func=str`.
+    * `check-plugins/apache-httpd-status/unit-test/run` iterates over a `SCENARIOS` list of dicts where the action resets a cache DB and then replays a multi-step sequence.
+
+Do not fall back to a plain `for ... subTest()` loop. It still executes every case and failures still surface, but unittest collapses the whole loop into a single method and reports `Ran 1 test`, which hides the real coverage count from `./run` and from the `tox` summary.
+
+
 #### Running tests
 
 Unit tests come in two flavors:
@@ -877,42 +891,47 @@ IMAGES = [
 
 
 class TestCheck(unittest.TestCase):
-    def test(self):
-        for image, version_tag in IMAGES:
-            with self.subTest(image=image):
-                with lib.lftest.run_container(
-                    image,
-                    env={
-                        'KEYCLOAK_ADMIN': 'admin',
-                        'KEYCLOAK_ADMIN_PASSWORD': 'admin',
-                    },
-                    ports=[8080],
-                    command='start-dev',
-                    wait_log='Listening on:',
-                ) as container:
-                    url = f'http://{container.get_container_host_ip()}:{container.get_exposed_port(8080)}'
-                    result = subprocess.run(
-                        ['python3', '../keycloak-version',
-                         f'--url={url}', '--username=admin', '--password=admin',
-                         '--path=/nonexistent'],
-                        capture_output=True, text=True,
-                    )
-                    self.assertRegex(
-                        result.stdout + result.stderr,
-                        rf'Keycloak\s+{version_tag}',
-                    )
-                    self.assertIn(
-                        result.returncode, (STATE_OK, STATE_WARN, STATE_CRIT),
-                    )
+    pass
+
+
+def _check_image(test, image_pair):
+    image, version_tag = image_pair
+    with lib.lftest.run_container(
+        image,
+        env={
+            'KEYCLOAK_ADMIN': 'admin',
+            'KEYCLOAK_ADMIN_PASSWORD': 'admin',
+        },
+        ports=[8080],
+        command='start-dev',
+        wait_log='Listening on:',
+    ) as container:
+        url = f'http://{container.get_container_host_ip()}:{container.get_exposed_port(8080)}'
+        result = subprocess.run(
+            ['python3', '../keycloak-version',
+             f'--url={url}', '--username=admin', '--password=admin',
+             '--path=/nonexistent'],
+            capture_output=True, text=True,
+        )
+        test.assertRegex(
+            result.stdout + result.stderr,
+            rf'Keycloak\s+{version_tag}',
+        )
+        test.assertIn(
+            result.returncode, (STATE_OK, STATE_WARN, STATE_CRIT),
+        )
+
+
+lib.lftest.attach_each(TestCheck, IMAGES, _check_image, id_func=lambda it: it[1])
 ```
 
 Rules and tips:
 
 * **Pull upstream images whenever possible.** You do not need a custom `Containerfile` that injects Python into the service image, because the plugin runs from the host and connects to the container via the exposed port. That is the common case for API-driven checks.
 * **Wait on a log marker, not a sleep.** The `wait_log` argument takes a substring that the service writes to stdout/stderr when it is ready (e.g. `Listening on:` for Keycloak, `ready for connections.` for MariaDB). Use `wait_log_timeout` for services that take longer than 2 minutes to start.
 * **Do not hardcode state-shifting assertions.** If the plugin reports something that depends on today's date (EOL windows, "last seen N days ago", "expires in X days"), assert only that the plugin returned a valid state (any of `STATE_OK`, `STATE_WARN`, `STATE_CRIT`) and that the output contains the expected version / service identifier. Locking in a specific state will break the test every time the calendar moves past a boundary.
-* **Multi-version matrix** goes in an `IMAGES` list at the top of the test file, iterated via `self.subTest(image=...)`. Add a new major release at the bottom of the list when it becomes available upstream.
-* **Rootless podman**: testcontainers-python works, but the Ryuk cleanup container needs to be disabled. Set `TESTCONTAINERS_RYUK_DISABLED=true` and `DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock` before running the tests. A lightweight wrapper can live in `tools/run-container-tests` to set these for you.
+* **Multi-version matrix** goes in an `IMAGES` list (or `CONTAINERFILES`, `SCENARIOS`, ...) at the top of the test file, materialised into one real test method per item via `lib.lftest.attach_each()`. Add a new major release at the bottom of the list when it becomes available upstream. See the "Iterating over TESTS vs. platforms" subsection below for the rationale.
+* **Rootless podman**: testcontainers-python works, but the Ryuk cleanup container needs to be disabled. Set `TESTCONTAINERS_RYUK_DISABLED=true` and `CONTAINER_HOST=unix:///run/user/$UID/podman/podman.sock` before running the tests. `tools/run-unit-tests` sets both automatically when it detects a container-based test.
 * **Do not run container tests via `tox`.** They are integration tests and belong in `tools/run-container-tests`, not in the multi-Python matrix. `tools/run-unit-tests` detects them automatically by inspecting the `run` file for `podman` or `testcontainers` references.
 * **Keep hand-rolled podman orchestration out of new tests.** If you find a plugin that still builds containers via `subprocess.run(['podman', 'build', ...])`, migrate it to `lib.lftest.run_container()`; the old pattern is being retired.
 

check-plugins/apache-httpd-status/unit-test/run

@@ -415,19 +415,20 @@ SCENARIOS = [
 
 
 class TestCheck(unittest.TestCase):
-
     check = '../apache-httpd-status'
 
-    def test(self):
-        for scenario in SCENARIOS:
-            with self.subTest(scenario=scenario['id']):
-                # Reset the sqlite cache so previous-value lookups
-                # start from a clean slate for this scenario.
-                if os.path.exists(CACHE_DB):
-                    os.remove(CACHE_DB)
-                for i, step in enumerate(scenario['steps'], start=1):
-                    with self.subTest(scenario=scenario['id'], step=i):
-                        lib.lftest.run(self, self.check, step)
+
+def _run_scenario(test, scenario):
+    # Reset the sqlite cache so previous-value lookups start from a
+    # clean slate for this scenario.
+    if os.path.exists(CACHE_DB):
+        os.remove(CACHE_DB)
+    for i, step in enumerate(scenario['steps'], start=1):
+        with test.subTest(step=i):
+            lib.lftest.run(test, test.check, step)
+
+
+lib.lftest.attach_each(TestCheck, SCENARIOS, _run_scenario, id_func=lambda s: s['id'])
 
 
 if __name__ == '__main__':

check-plugins/cpu-usage/unit-test/run

@@ -41,6 +41,7 @@ import unittest
 
 sys.path.insert(0, '..')
 
+import lib.lftest
 from lib.globals import STATE_OK
 
 try:
@@ -78,53 +79,56 @@ CONTAINERFILES = [
 
 
 class TestCheck(unittest.TestCase):
-    def test(self):
-        for containerfile in CONTAINERFILES:
-            with self.subTest(containerfile=containerfile):
-                print(f'\n=== Testing {containerfile} ===', flush=True)
-                image_tag = f'lfmp-cpu-usage-{containerfile}'.lower()
-                with DockerImage(
-                    path=HERE,
-                    dockerfile_path=f'containerfiles/{containerfile}',
-                    tag=image_tag,
-                ) as image:
-                    # `,Z` relabels the mount for SELinux on podman/rhel hosts
-                    # so the container can actually read the bind-mounted files.
-                    container = (
-                        DockerContainer(str(image.tag))
-                        .with_volume_mapping(LIB_DIR, '/tmp/lib', mode='ro,Z')
-                        .with_volume_mapping(PLUGIN, '/tmp/cpu-usage', mode='ro,Z')
-                    )
-                    with container:
-                        # Some containerfiles install dependencies into a venv
-                        # at /tmp/venv (debian, ubuntu, archlinux), others
-                        # install them system-wide (fedora, rhel). Detect and
-                        # use the right python invocation.
-                        probe = container.exec(
-                            ['sh', '-c', 'test -d /tmp/venv && echo venv || echo system']
-                        )
-                        use_venv = b'venv' in probe.output
-                        if use_venv:
-                            cmd = [
-                                'sh',
-                                '-c',
-                                '. /tmp/venv/bin/activate && cd /tmp && python3 ./cpu-usage',
-                            ]
-                        else:
-                            cmd = ['sh', '-c', 'cd /tmp && python3 ./cpu-usage']
-                        print(f'Run plugin: {" ".join(cmd)}', flush=True)
-                        result = container.exec(cmd)
-                        output = (result.output or b'').decode('utf-8', errors='replace')
-                        print(f'Script output:\n{output.strip()}', flush=True)
-
-                        # Plugin must exit cleanly and produce its normal
-                        # "<percent>% - ..., user: X%, ..." line. Use a
-                        # multi-line-tolerant regex because the "user:"
-                        # segment can land on the first or the second line
-                        # depending on psutil's by-value sort when several
-                        # counters read 0%.
-                        self.assertEqual(result.exit_code, STATE_OK)
-                        self.assertRegex(output, r'(?s)\d+\.\d+%.*user:')
+    pass
+
+
+def _check_containerfile(test, containerfile):
+    print(f'\n=== Testing {containerfile} ===', flush=True)
+    image_tag = f'lfmp-cpu-usage-{containerfile}'.lower()
+    with DockerImage(
+        path=HERE,
+        dockerfile_path=f'containerfiles/{containerfile}',
+        tag=image_tag,
+    ) as image:
+        # `,Z` relabels the mount for SELinux on podman/rhel hosts so
+        # the container can actually read the bind-mounted files.
+        container = (
+            DockerContainer(str(image.tag))
+            .with_volume_mapping(LIB_DIR, '/tmp/lib', mode='ro,Z')
+            .with_volume_mapping(PLUGIN, '/tmp/cpu-usage', mode='ro,Z')
+        )
+        with container:
+            # Some containerfiles install dependencies into a venv at
+            # /tmp/venv (debian, ubuntu, archlinux), others install
+            # them system-wide (fedora, rhel). Detect and use the
+            # right python invocation.
+            probe = container.exec(
+                ['sh', '-c', 'test -d /tmp/venv && echo venv || echo system']
+            )
+            use_venv = b'venv' in probe.output
+            if use_venv:
+                cmd = [
+                    'sh',
+                    '-c',
+                    '. /tmp/venv/bin/activate && cd /tmp && python3 ./cpu-usage',
+                ]
+            else:
+                cmd = ['sh', '-c', 'cd /tmp && python3 ./cpu-usage']
+            print(f'Run plugin: {" ".join(cmd)}', flush=True)
+            result = container.exec(cmd)
+            output = (result.output or b'').decode('utf-8', errors='replace')
+            print(f'Script output:\n{output.strip()}', flush=True)
+
+            # Plugin must exit cleanly and produce its normal
+            # "<percent>% - ..., user: X%, ..." line. Use a
+            # multi-line-tolerant regex because the "user:" segment
+            # can land on the first or the second line depending on
+            # psutil's by-value sort when several counters read 0%.
+            test.assertEqual(result.exit_code, STATE_OK)
+            test.assertRegex(output, r'(?s)\d+\.\d+%.*user:')
+
+
+lib.lftest.attach_each(TestCheck, CONTAINERFILES, _check_containerfile)
 
 
 if __name__ == '__main__':

check-plugins/keycloak-memory-usage/unit-test/run

@@ -39,59 +39,62 @@ IMAGES = [
 
 
 class TestCheck(unittest.TestCase):
-    def test(self):
-        """Spin up each Keycloak image and run the plugin against the API."""
-        for image, version_tag in IMAGES:
-            with self.subTest(image=image):
-                print(f'\n=== Testing {image} ===', flush=True)
-                with lib.lftest.run_container(
-                    image,
-                    env={
-                        'KEYCLOAK_ADMIN': 'admin',
-                        'KEYCLOAK_ADMIN_PASSWORD': 'admin',
-                    },
-                    ports=[8080],
-                    command='start-dev',
-                    wait_log='Listening on:',
-                    wait_log_timeout=180,
-                ) as container:
-                    url = 'http://{}:{}'.format(
-                        container.get_container_host_ip(),
-                        container.get_exposed_port(8080),
-                    )
-                    cmd = [
-                        'python3',
-                        '../keycloak-memory-usage',
-                        f'--url={url}',
-                        '--username=admin',
-                        '--password=admin',
-                    ]
-                    print(f'Run plugin: {" ".join(cmd)}', flush=True)
-                    result = subprocess.run(
-                        cmd,
-                        cwd=os.path.dirname(os.path.abspath(__file__)),
-                        capture_output=True,
-                        text=True,
-                    )
-                    combined = result.stdout + result.stderr
-                    print(f'Script output:\n{combined.strip()}', flush=True)
-
-                    # The plugin reports "<percent>% - total: <size>, used: <size>,
-                    # free: <size>" plus perfdata. Check the message shape instead
-                    # of specific numbers (which depend on the container host's
-                    # current memory pressure).
-                    self.assertRegex(
-                        combined,
-                        r'\d+(?:\.\d+)?% - total:\s*\d+\.\d+(?:M|G)iB,'
-                        r'\s*used:\s*\d+\.\d+(?:M|G)iB,'
-                        r'\s*free:\s*\d+\.\d+(?:M|G)iB',
-                    )
-                    # Any valid monitoring state. Threshold-based alerts on heap
-                    # depend on how much memory the test host allocates to
-                    # Keycloak; not something we want to pin in the test.
-                    self.assertIn(
-                        result.returncode, (STATE_OK, STATE_WARN, STATE_CRIT)
-                    )
+    pass
+
+
+def _check_image(test, image_pair):
+    """Spin up one Keycloak image and run the plugin against the API."""
+    image, version_tag = image_pair
+    print(f'\n=== Testing {image} ===', flush=True)
+    with lib.lftest.run_container(
+        image,
+        env={
+            'KEYCLOAK_ADMIN': 'admin',
+            'KEYCLOAK_ADMIN_PASSWORD': 'admin',
+        },
+        ports=[8080],
+        command='start-dev',
+        wait_log='Listening on:',
+        wait_log_timeout=180,
+    ) as container:
+        url = 'http://{}:{}'.format(
+            container.get_container_host_ip(),
+            container.get_exposed_port(8080),
+        )
+        cmd = [
+            'python3',
+            '../keycloak-memory-usage',
+            f'--url={url}',
+            '--username=admin',
+            '--password=admin',
+        ]
+        print(f'Run plugin: {" ".join(cmd)}', flush=True)
+        result = subprocess.run(
+            cmd,
+            cwd=os.path.dirname(os.path.abspath(__file__)),
+            capture_output=True,
+            text=True,
+        )
+        combined = result.stdout + result.stderr
+        print(f'Script output:\n{combined.strip()}', flush=True)
+
+        # The plugin reports "<percent>% - total: <size>, used: <size>,
+        # free: <size>" plus perfdata. Check the message shape instead
+        # of specific numbers (which depend on the container host's
+        # current memory pressure).
+        test.assertRegex(
+            combined,
+            r'\d+(?:\.\d+)?% - total:\s*\d+\.\d+(?:M|G)iB,'
+            r'\s*used:\s*\d+\.\d+(?:M|G)iB,'
+            r'\s*free:\s*\d+\.\d+(?:M|G)iB',
+        )
+        # Any valid monitoring state. Threshold-based alerts on heap
+        # depend on how much memory the test host allocates to
+        # Keycloak; not something we want to pin in the test.
+        test.assertIn(result.returncode, (STATE_OK, STATE_WARN, STATE_CRIT))
+
+
+lib.lftest.attach_each(TestCheck, IMAGES, _check_image, id_func=lambda it: it[1])
 
 
 if __name__ == '__main__':