diff --git a/.gitignore b/.gitignore index 236cffbd..05ec277a 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,7 @@ tests/output/* playbooks/test.yml roles/test context/ +extensions/molecule/**/*.retry # mkdocs documentation /docs/CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 655384e2..d4935ff6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +* **testing**: Add a Molecule-based test framework that runs the playbooks (and through them the roles) against throwaway libvirt/KVM VMs or Podman containers. Scenarios live under `extensions/molecule`; see the Testing section in `CONTRIBUTING.md`. * **role:icinga2_master, role:icingadb, role:icingaweb2, role:icingaweb2_module_reporting, role:icingaweb2_module_x509**: Add explicit Ubuntu variable files, making Ubuntu support visible alongside Debian. The Icinga repository, GPG key and package names were verified on Debian 13 and Ubuntu 24.04. * **role:nextcloud**: Add `meta/argument_specs.yml` declaring the user-facing variables, so role-entry validation catches type mismatches and missing mandatory variables. * **role:clamav**: Add `meta/argument_specs.yml` declaring the user-facing variables, so role-entry validation catches type mismatches and unknown variables. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e35d2113..0df6df32 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -922,6 +922,116 @@ Unit tests are **mandatory** for every in-house plugin. Any pull request that ad * The `Linuxfabrik: Unit Tests` workflow runs the controller matrix on every push and pull request. +### Testing + +Molecule is used as the framework to test the LFOps playbooks (and therefore indirectly the roles). The test scenarios and configurations live in `extensions/molecule` and are structured as follows: + +``` +extensions +└── molecule + ├── apps -- test scenario, named after the playbook name + │ ├── install -- if needed, sub-scenario + │ │ ├── converge.yml -- the actual test phase. this is where the playbook under test runs against the hosts + │ │ ├── inventory -- scenario-specific inventory with variables that are needed for the playbook under test and optionally additional hosts (e.g. for a cluster test setup). overwrites the shared inventory (extensions/molecule/inventory) + │ │ │ ├── group_vars + │ │ │ │ └── systems_under_test.yml -- by convention, the "systems_under_test" group contains all our hosts against which the tests are run + │ │ │ └── hosts.yml -- here we select against which hosts we want to run (most of the time the hosts come from the shared inventory) and put them into the correct group for the playbook, here "lfops_apps" + │ │ ├── molecule.yml -- scenario marker; the file is required even if empty. can also be used to overwrite settings from the extensions/molecule/config.yml, for example which playbooks are used by Molecule (e.g. to switch between VM and container provisioning playbooks) + │ │ └── verify.yml -- runs after the test phase and uses ansible to check if the result is as expected + │ └── remove -- additional sub-scenario + │ └── ... + ├── config.yml -- valid for all scenarios, can be overwritten in each scenario's molecule.yml (content and structure are the same) + ├── default -- we are not using the "default" scenario, but molecule needs this to run at all. could be used to share config (e.g. prepare.yml) across *all* scenarios + │ └── molecule.yml + ├── example -- fully commented reference scenario (install + remove sub-scenarios); copy it when adding a new test, like the example role + │ ├── install + │ │ └── ... + │ └── remove + │ └── ... + ├── inventory -- shared inventory across all scenarios and therefore available in all scenarios. contains a basic set of VMs/containers that are commonly used + │ ├── hosts.yml -- required, even if empty, that Ansible can detect this inventory + │ └── host_vars + │ ├── debian11-container.yml + │ ├── debian11-vm.yml + │ └── ... + ├── monitoring_plugins -- a scenario with no sub-scenarios + │ ├── converge.yml + │ ├── inventory + │ │ └── ... + │ ├── molecule.yml + │ └── verify.yml + ├── playbooks -- shared playbooks used by Molecule for running the scenarios + │ ├── container-create.yml + │ ├── container-destroy.yml + │ └── ... + └── requirements.yml +``` + +The `extensions/molecule/example` scenario mirrors the `example` role: it is a fully commented, non-functional reference that walks through every file of a scenario (a non-functional reference because the `example` playbook installs the fictional "Example" application). Copy it as the starting point when adding a test for a playbook. + +Tests can be run against a subset of targets by providing them as a comma-separated list via the project-specific `LFOPS_TEST_TARGETS` environment variable. The variable is optional: unset, every target in the scenario runs. `localhost` (the hypervisor) is included automatically, so you only ever pass the targets themselves: + +```shell +# all targets in the scenario +molecule test --scenario-name apps/install + +# a subset +LFOPS_TEST_TARGETS='rocky*' molecule test --scenario-name apps/install +``` + + +Known Limitations: + +* VM-based testing requires passwordless sudo on the Ansible controller. The cloud image and per-VM disks are written and built directly in the root-owned libvirt pool directory (`get_url`, `qemu-img`, `virt-customize`), which is plain filesystem I/O and needs root. The read-only libvirt API calls already run unprivileged via the `libvirt` group; it is only the pool writes that require sudo. Trying to make the whole run rootless is not worth it: the only way to provision VMs without root-equivalent rights at all is the user session (`qemu:///session`), which the test cannot use because its address discovery reads the host's ARP/neighbour table for the libvirt-managed `default` network that only the system connection (`qemu:///system`) provides. Every other route still grants effective root: a member of the `libvirt` group (which the read-only calls already require) can define a domain backed by any host device and drive QEMU as root. Swapping the `sudo` for a user-owned `qemu:///system` pool therefore only trades an explicit, on-demand escalation for a standing root-equivalent privilege plus looser filesystem permissions, which is a worse posture, not a better one. +* Does not work inside an Ansible Execution Environment (Ansible Navigator). Provisioning runs as `localhost`, which inside an EE is the container, yet it has to act on the host's libvirt and podman. The disk-build tools (`qemu-img`, `virt-customize`, `virt-sysprep`) are filesystem-bound to the pool and have no libvirt-socket equivalent, so an EE would have to bind-mount the host libvirt/podman sockets and the pool directory and use host networking, which removes most of the isolation an EE exists to provide. + + +#### How a scenario runs + +`molecule test --scenario-name ` runs the steps listed in the `test_sequence` of `config.yml`, in order: + +* `dependency`: installs the collections from `requirements.yml`. +* `create`: provisions the instances (libvirt/KVM VMs or Podman containers). +* `prepare`: waits until the instances are reachable and gathers facts. +* `converge`: runs the playbook under test (`converge.yml`). +* `verify`: runs `verify.yml` against the converged instances. +* `idempotence`: runs the playbook a second time and fails if it reports any change. +* `verify`: runs `verify.yml` again, now against the idempotent state. +* `destroy`: tears the instances down. + + +#### What to verify + +Verify the observable end result, not the steps the role took to get there. Ansible and the role already guarantee their own mechanics, so re-checking those only tests Ansible. The guiding question is "what can only be confirmed by looking at the running system?". + +Two guarantees come for free, so do not rebuild them in `verify.yml`: + +* If the playbook errors out, the `converge` step fails and `verify.yml` never runs. `verify.yml` is therefore only ever about the *result* of a successful run, not about whether the run crashed. +* Idempotence is enforced by the dedicated `idempotence` step. Never add tasks that check "running it a second time changes nothing". + +Do **not** assert: + +* That a templated file exists or contains a given line. If the `template` task ran, the file is there with the rendered content; asserting it only exercises Jinja and the `template` module. +* That a package was installed or a file was written, as the goal of the test. The module already reports `changed`/`ok` for that. A one-line "the package is installed" smoke check is fine as a floor, but it is not where the value of the test lies. + +Do assert what only the running system can confirm, that is, that the pieces actually work together: + +* The application is running and enabled (`ansible.builtin.service_facts`), and reachable on its port (`ansible.builtin.wait_for`, or a request that would fail if it were not). A service that starts proves the deployed config is at least valid, which the role's own tasks cannot tell you. +* The application actually *uses* the configured values. Ask the running application (an API or status endpoint via `ansible.builtin.uri`, or a CLI that prints the effective configuration) and assert it reports the value the scenario set in `group_vars`. This is the important one: it proves the whole chain, `group_vars` to template to the service reading the file to its behaviour, which is exactly what grepping the config file does not. +* End state managed outside the package and file layer (users, databases, API objects) is present, or absent in a removal scenario. + +A useful rule of thumb: if an assertion would still pass while the service is dead or running with the wrong configuration, it is testing the wrong thing. + + +#### Troubleshooting + +**`molecule test` aborts with `ansible_compat.errors.InvalidPrerequisiteError: Command ansible-galaxy collection install -vvv --force /path/to/lfops` during prerun while installing the local collection** + +* Before running a scenario, Molecule's prerun step tries to install the current repository as a collection with `ansible-galaxy collection install --force `. That build fails because `galaxy.yml` carries a non-semver `version` (`main`), which `ansible-galaxy` rejects. +* Option 1: disable the prerun so Molecule stops trying to build and install the local collection, by setting `prerun: false` as a top-level key in the `config.yml`. If you do this, you have to make sure that LFOps is installed yourself. +* Option 2: If you installed LFOps by symlinking it, make sure the link points to the **same** folder that you are running `molecule` in (`ln -sf "$(pwd)" ~/.ansible/collections/ansible_collections/linuxfabrik/lfops`). + + ### Credits * diff --git a/extensions/molecule/apps/install/converge.yml b/extensions/molecule/apps/install/converge.yml new file mode 100644 index 00000000..24ae2f23 --- /dev/null +++ b/extensions/molecule/apps/install/converge.yml @@ -0,0 +1,2 @@ +- name: 'Converge apps playbook' + ansible.builtin.import_playbook: 'linuxfabrik.lfops.apps' diff --git a/extensions/molecule/apps/install/inventory/group_vars/systems_under_test.yml b/extensions/molecule/apps/install/inventory/group_vars/systems_under_test.yml new file mode 100644 index 00000000..c559815d --- /dev/null +++ b/extensions/molecule/apps/install/inventory/group_vars/systems_under_test.yml @@ -0,0 +1,3 @@ +apps__apps__group_var: + - name: 'zsh' + state: 'present' diff --git a/extensions/molecule/apps/install/inventory/hosts.yml b/extensions/molecule/apps/install/inventory/hosts.yml new file mode 100644 index 00000000..129b4f61 --- /dev/null +++ b/extensions/molecule/apps/install/inventory/hosts.yml @@ -0,0 +1,17 @@ +# yamllint disable rule:empty-values +lfops_apps: + children: + systems_under_test: + +systems_under_test: + hosts: + debian11-vm: + debian12-vm: + debian13-vm: + rocky8-vm: + rocky9-vm: + rocky10-vm: + ubuntu2004-vm: + ubuntu2204-vm: + ubuntu2404-vm: + ubuntu2604-vm: diff --git a/extensions/molecule/apps/install/molecule.yml b/extensions/molecule/apps/install/molecule.yml new file mode 100644 index 00000000..1e47cbff --- /dev/null +++ b/extensions/molecule/apps/install/molecule.yml @@ -0,0 +1 @@ +# Molecule scenario marker diff --git a/extensions/molecule/apps/install/verify.yml b/extensions/molecule/apps/install/verify.yml new file mode 100644 index 00000000..b1eb7bd1 --- /dev/null +++ b/extensions/molecule/apps/install/verify.yml @@ -0,0 +1,10 @@ +- name: 'Verify apps are installed' + hosts: 'systems_under_test' + gather_facts: false + tasks: + - name: 'Gather the package facts' + ansible.builtin.package_facts: # yamllint disable-line rule:empty-values + + - name: 'Assert that zsh is installed' + ansible.builtin.assert: + that: '"zsh" in ansible_facts["packages"]' diff --git a/extensions/molecule/apps/remove/converge.yml b/extensions/molecule/apps/remove/converge.yml new file mode 100644 index 00000000..24ae2f23 --- /dev/null +++ b/extensions/molecule/apps/remove/converge.yml @@ -0,0 +1,2 @@ +- name: 'Converge apps playbook' + ansible.builtin.import_playbook: 'linuxfabrik.lfops.apps' diff --git a/extensions/molecule/apps/remove/inventory/group_vars/systems_under_test.yml b/extensions/molecule/apps/remove/inventory/group_vars/systems_under_test.yml new file mode 100644 index 00000000..e7bd8516 --- /dev/null +++ b/extensions/molecule/apps/remove/inventory/group_vars/systems_under_test.yml @@ -0,0 +1,3 @@ +apps__apps__group_var: + - name: 'less' + state: 'absent' diff --git a/extensions/molecule/apps/remove/inventory/hosts.yml b/extensions/molecule/apps/remove/inventory/hosts.yml new file mode 100644 index 00000000..129b4f61 --- /dev/null +++ b/extensions/molecule/apps/remove/inventory/hosts.yml @@ -0,0 +1,17 @@ +# yamllint disable rule:empty-values +lfops_apps: + children: + systems_under_test: + +systems_under_test: + hosts: + debian11-vm: + debian12-vm: + debian13-vm: + rocky8-vm: + rocky9-vm: + rocky10-vm: + ubuntu2004-vm: + ubuntu2204-vm: + ubuntu2404-vm: + ubuntu2604-vm: diff --git a/extensions/molecule/apps/remove/molecule.yml b/extensions/molecule/apps/remove/molecule.yml new file mode 100644 index 00000000..1e47cbff --- /dev/null +++ b/extensions/molecule/apps/remove/molecule.yml @@ -0,0 +1 @@ +# Molecule scenario marker diff --git a/extensions/molecule/apps/remove/verify.yml b/extensions/molecule/apps/remove/verify.yml new file mode 100644 index 00000000..07c6a6e9 --- /dev/null +++ b/extensions/molecule/apps/remove/verify.yml @@ -0,0 +1,10 @@ +- name: 'Verify apps are not installed' + hosts: 'systems_under_test' + gather_facts: false + tasks: + - name: 'Gather the package facts' + ansible.builtin.package_facts: # yamllint disable-line rule:empty-values + + - name: 'Assert that less is not installed' + ansible.builtin.assert: + that: '"less" not in ansible_facts["packages"]' diff --git a/extensions/molecule/config.yml b/extensions/molecule/config.yml new file mode 100644 index 00000000..94637976 --- /dev/null +++ b/extensions/molecule/config.yml @@ -0,0 +1,92 @@ +# Shared Molecule base config. It is merged into every scenario and is read whenever Molecule +# resolves a scenario (e.g. `molecule test`, `molecule converge`, `molecule create`). Individual +# scenarios can override any of these settings in their own molecule.yml. + +ansible: + cfg: + defaults: + ansible_managed: 'This file is managed by Ansible - do not edit' + callbacks_enabled: 'profile_tasks' + fact_caching: 'jsonfile' + fact_caching_connection: '${MOLECULE_EPHEMERAL_DIRECTORY}/.ansible_cache' + fact_caching_timeout: 86400 + forks: 30 + gathering: 'smart' + host_key_checking: false + inventory: 'hosts' + inventory_ignore_extensions: + - '~' + - '.orig' + - '.bak' + - '.ini' + - '.cfg' + - '.retry' + - '.pyc' + - '.pyo' + - '.csv' + - '.md' + inventory_ignore_patterns: '(host|group)_files' + log_path: '${MOLECULE_EPHEMERAL_DIRECTORY}/ansible.log' + nocows: 1 + retry_files_enabled: true + stdout_callback: 'yaml' + timeout: 60 + ssh_connections: + pipelining: true + ssh_args: '-o ControlMaster=auto -o ControlPersist=60s' + executor: + backend: 'ansible-playbook' # or 'ansible-navigator' + # The limit always includes localhost (vm-create.yml's controller-setup play targets it) and + # defaults LFOPS_TEST_TARGETS to 'all', so the variable is optional: unset runs every target, + # and when set you only pass the targets (e.g. 'rocky*'), never localhost. + args: + ansible_navigator: + - '--inventory=${MOLECULE_PROJECT_DIRECTORY}/extensions/molecule/inventory/' + - '--inventory=${MOLECULE_SCENARIO_DIRECTORY}/inventory/' + - '--limit=localhost,${LFOPS_TEST_TARGETS:-all}' + ansible_playbook: + - '--inventory=${MOLECULE_PROJECT_DIRECTORY}/extensions/molecule/inventory/' + - '--inventory=${MOLECULE_SCENARIO_DIRECTORY}/inventory/' + - '--limit=localhost,${LFOPS_TEST_TARGETS:-all}' + +# The 'galaxy' dependency below installs the collections listed in requirements.yml via +# `ansible-galaxy collection install`. Molecule only runs the 'dependency' stage when it is part +# of the sequence being executed. Our custom test_sequence therefore lists +# 'dependency' explicitly; without it `molecule test` would skip the install and rely on the +# collections already being present on the controller. +# The galaxy dependency also runs a roles sub-step that ignores requirements-file and reads +# role-file instead. We have no Galaxy roles, so role-file points at an empty +# requirements-roles.yml purely to stop it warning about a missing roles requirements file. +dependency: + name: 'galaxy' + options: + requirements-file: '${MOLECULE_PROJECT_DIRECTORY}/extensions/molecule/requirements.yml' + role-file: '${MOLECULE_PROJECT_DIRECTORY}/extensions/molecule/requirements-roles.yml' + +# Inventory model (why there is no 'platforms:' block and no managed-driver instance_config): +# We test playbooks, which are inventory-driven (hosts: lfops_*, group_vars, host_vars). The +# scenarios therefore use real inventory files (extensions/molecule/inventory plus each +# scenario's own inventory) so the test inventory mirrors a production one and converge runs the +# playbook the way an admin would. Molecule's managed driver only injects connection details +# (ansible_host, ansible_user, ...) for hosts listed under 'platforms:', which the inventory +# model does not use - so vm-create.yml discovers each VM's IP and writes the connection details +# into Molecule's ephemeral inventory directory itself (Molecule includes that directory +# automatically). Host key checking is turned off above (host_key_checking) because the VMs are +# throwaway and reuse IPs from the libvirt default network. + +provisioner: + playbooks: + create: '${MOLECULE_PROJECT_DIRECTORY}/extensions/molecule/playbooks/vm-create.yml' + destroy: '${MOLECULE_PROJECT_DIRECTORY}/extensions/molecule/playbooks/vm-destroy.yml' + prepare: '${MOLECULE_PROJECT_DIRECTORY}/extensions/molecule/playbooks/vm-prepare.yml' + +scenario: + test_sequence: + - 'dependency' # must stay in the list so requirements.yml is installed (see header note) + - 'create' + - 'prepare' + - 'converge' + - 'verify' + - 'idempotence' + - 'verify' + - 'destroy' diff --git a/extensions/molecule/default/molecule.yml b/extensions/molecule/default/molecule.yml new file mode 100644 index 00000000..1e47cbff --- /dev/null +++ b/extensions/molecule/default/molecule.yml @@ -0,0 +1 @@ +# Molecule scenario marker diff --git a/extensions/molecule/example/install/converge.yml b/extensions/molecule/example/install/converge.yml new file mode 100644 index 00000000..b3183fed --- /dev/null +++ b/extensions/molecule/example/install/converge.yml @@ -0,0 +1,9 @@ +# converge.yml is the test phase. Molecule runs it after 'create' and +# 'prepare' have provisioned and readied the systems under test. +# +# Import the playbook under test by its collection FQCN, so the test exercises +# the real playbook (and through it, the roles) instead of a copy that could +# drift. Keep this file to the single import; the test inputs live in the +# scenario inventory, and the checks live in verify.yml. +- name: 'Converge example playbook' + ansible.builtin.import_playbook: 'linuxfabrik.lfops.example' diff --git a/extensions/molecule/example/install/inventory/group_vars/systems_under_test.yml b/extensions/molecule/example/install/inventory/group_vars/systems_under_test.yml new file mode 100644 index 00000000..569a6617 --- /dev/null +++ b/extensions/molecule/example/install/inventory/group_vars/systems_under_test.yml @@ -0,0 +1,27 @@ +# Variables the playbook under test needs, applied to every system under test. +# This is the test's input: set the mandatory variables, plus any optional ones +# whose behaviour the scenario should exercise. + +# Mandatory: the example role requires a version. +example__version: '3.2.1' + +# Skip the fictional repo_example role the playbook pulls in. A test for a real +# playbook would normally let the repo role run so the package source is set up. +example__skip_repo_example: true + +# Optional simple values, set through the __group_var injection slot so the +# role's combined_var logic merges them over its defaults. +example__conf_log_level__group_var: 'debug' +example__conf_max_connections__group_var: 200 + +# Combined variables are lists of dicts with a state. The install scenario adds +# one plugin and one user so verify.yml has concrete state to assert on. The +# matching remove scenario flips these to state: 'absent'. +example__plugins__group_var: + - name: 'example-plugin-auth-ldap' + state: 'present' + +example__users__group_var: + - name: 'example-admin' + password: 'linuxfabrik' + state: 'present' diff --git a/extensions/molecule/example/install/inventory/hosts.yml b/extensions/molecule/example/install/inventory/hosts.yml new file mode 100644 index 00000000..ca03a844 --- /dev/null +++ b/extensions/molecule/example/install/inventory/hosts.yml @@ -0,0 +1,28 @@ +# yamllint disable rule:empty-values + +# Scenario inventory. It is layered on top of the shared inventory +# (extensions/molecule/inventory) through the two --inventory flags in +# config.yml, so the host_vars defined there (connection, image, kvm_vm__*) +# stay available; this file only adds the host-to-group mapping. + +# Put the systems under test into the group the playbook targets. The example +# playbook runs against 'lfops_example' (see playbooks/example.yml: hosts). +lfops_example: + children: + systems_under_test: + +# 'systems_under_test' is the conventional group holding the actual hosts. +# Select the platforms the role supports; here the standard VM set. Trim this +# (or use LFOPS_TEST_TARGETS at runtime) to test against fewer hosts. +systems_under_test: + hosts: + debian11-vm: + debian12-vm: + debian13-vm: + rocky8-vm: + rocky9-vm: + rocky10-vm: + ubuntu2004-vm: + ubuntu2204-vm: + ubuntu2404-vm: + ubuntu2604-vm: diff --git a/extensions/molecule/example/install/molecule.yml b/extensions/molecule/example/install/molecule.yml new file mode 100644 index 00000000..17526526 --- /dev/null +++ b/extensions/molecule/example/install/molecule.yml @@ -0,0 +1,12 @@ +# Molecule scenario marker. Required even when empty, so that Molecule +# discovers the scenario. Settings from extensions/molecule/config.yml can be +# overridden here when a scenario needs it, for example to point the +# create/prepare/destroy hooks at the container playbooks instead of the VM +# ones. +# +# Like the 'example' role, this 'example' scenario is a non-functional +# reference: the 'example' playbook installs the fictional "Example" +# application, so the scenario does not converge against a real system. It +# exists to show how a Molecule test for an LFOps playbook is structured and +# commented. Copy it as the starting point when adding a test for a real +# playbook. diff --git a/extensions/molecule/example/install/verify.yml b/extensions/molecule/example/install/verify.yml new file mode 100644 index 00000000..f8612061 --- /dev/null +++ b/extensions/molecule/example/install/verify.yml @@ -0,0 +1,55 @@ +# verify.yml runs after converge, and again after the idempotence step (see the +# test_sequence in config.yml). It checks the observable end result with Ansible +# facts and read-only modules - never by re-running the role. +# +# The guiding question is "what can only be confirmed by looking at the running +# system?". We do not assert that /etc/example/example.conf exists or contains a +# certain line: if the template task ran, that is already guaranteed, so it would +# only test Jinja and the template module. We confirm instead that the service +# runs and that it actually uses the values this scenario set in group_vars. +- name: 'Verify example is installed and configured' + hosts: 'systems_under_test' + gather_facts: false + tasks: + + # Floor check: did the install happen at all? On its own this would still + # pass even if the service never starts, so it is the start, not the goal. + - name: 'Gather the package facts' + ansible.builtin.package_facts: # yamllint disable-line rule:empty-values + + - name: 'Assert that the example-server package is installed' + ansible.builtin.assert: + that: '"example-server" in ansible_facts["packages"]' + + # Real check: is the service up? This catches a config that deployed without + # error but makes the service crash on start - something the role's own + # tasks cannot detect. + - name: 'Gather the service facts' + ansible.builtin.service_facts: # yamllint disable-line rule:empty-values + + - name: 'Assert that example.service is running' + ansible.builtin.assert: + that: 'ansible_facts["services"]["example.service"]["state"] == "running"' + + # The important check: does the running application actually use the value we + # configured? Ask the application (here an illustrative status endpoint), not + # the config file. This scenario sets example__conf_max_connections to 200, + # so the live configuration must report 200. This proves the whole chain + # works: group_vars -> template -> service reads the file -> behaviour. + - name: 'curl http://127.0.0.1:8080/api/status' + ansible.builtin.uri: + url: 'http://127.0.0.1:8080/api/status' + return_content: true + register: '__molecule__example_status_result' + + - name: 'Assert the running application uses the configured max_connections' + ansible.builtin.assert: + that: '__molecule__example_status_result["json"]["max_connections"] == 200' + + # End state managed over the REST API is invisible to package/service facts, + # so query it directly. The install scenario creates this user, so a GET for + # it returns 200. + - name: 'curl http://127.0.0.1:8080/api/user/example-admin' + ansible.builtin.uri: + url: 'http://127.0.0.1:8080/api/user/example-admin' + status_code: 200 diff --git a/extensions/molecule/example/remove/converge.yml b/extensions/molecule/example/remove/converge.yml new file mode 100644 index 00000000..0ba8d439 --- /dev/null +++ b/extensions/molecule/example/remove/converge.yml @@ -0,0 +1,5 @@ +# Same playbook as the install sub-scenario; only the scenario inventory +# differs. The remove sub-scenario flips the plugin and user to state: 'absent' +# (see its group_vars), which exercises the role's removal path. +- name: 'Converge example playbook' + ansible.builtin.import_playbook: 'linuxfabrik.lfops.example' diff --git a/extensions/molecule/example/remove/inventory/group_vars/systems_under_test.yml b/extensions/molecule/example/remove/inventory/group_vars/systems_under_test.yml new file mode 100644 index 00000000..a26d01b4 --- /dev/null +++ b/extensions/molecule/example/remove/inventory/group_vars/systems_under_test.yml @@ -0,0 +1,18 @@ +# The remove scenario runs the same playbook but flips the plugin and user to +# state: 'absent'. This exercises the list-with-state removal pattern the LFOps +# roles use, and gives verify.yml something concrete to assert is gone. +# +# In a real suite, run the install sub-scenario first to create the state, then +# this one to remove it. As a standalone run it still demonstrates that a second +# converge with state: 'absent' leaves the host without the plugin/user. + +example__version: '3.2.1' +example__skip_repo_example: true + +example__plugins__group_var: + - name: 'example-plugin-auth-ldap' + state: 'absent' + +example__users__group_var: + - name: 'example-admin' + state: 'absent' diff --git a/extensions/molecule/example/remove/inventory/hosts.yml b/extensions/molecule/example/remove/inventory/hosts.yml new file mode 100644 index 00000000..47abd4b3 --- /dev/null +++ b/extensions/molecule/example/remove/inventory/hosts.yml @@ -0,0 +1,20 @@ +# yamllint disable rule:empty-values + +# Same host-to-group mapping as the install sub-scenario. Sub-scenarios do not +# share inventory with each other, so each repeats the hosts it runs against. +lfops_example: + children: + systems_under_test: + +systems_under_test: + hosts: + debian11-vm: + debian12-vm: + debian13-vm: + rocky8-vm: + rocky9-vm: + rocky10-vm: + ubuntu2004-vm: + ubuntu2204-vm: + ubuntu2404-vm: + ubuntu2604-vm: diff --git a/extensions/molecule/example/remove/molecule.yml b/extensions/molecule/example/remove/molecule.yml new file mode 100644 index 00000000..a683437d --- /dev/null +++ b/extensions/molecule/example/remove/molecule.yml @@ -0,0 +1,8 @@ +# Molecule scenario marker for the 'remove' sub-scenario. A scenario can have +# several sub-scenarios (here 'install' and 'remove'); each is a directory with +# its own converge/inventory/verify, run with +# `molecule test --scenario-name example/remove`. +# +# Like the 'install' sub-scenario, this is a non-functional reference: the +# 'example' playbook targets the fictional "Example" application. It shows how +# to test the removal half of the role's list-with-state pattern. diff --git a/extensions/molecule/example/remove/verify.yml b/extensions/molecule/example/remove/verify.yml new file mode 100644 index 00000000..d0df5900 --- /dev/null +++ b/extensions/molecule/example/remove/verify.yml @@ -0,0 +1,31 @@ +# Verify the removal. Same mindset as install/verify.yml: check the observable +# result, and confirm the removal did not break the still-installed application. +- name: 'Verify the example plugin and user are removed' + hosts: 'systems_under_test' + gather_facts: false + tasks: + + # Removing a plugin and a user must not take the application down with them, + # so the most important check here is that the service is still running. + - name: 'Gather the service facts' + ansible.builtin.service_facts: # yamllint disable-line rule:empty-values + + - name: 'Assert that example.service is still running' + ansible.builtin.assert: + that: 'ansible_facts["services"]["example.service"]["state"] == "running"' + + # The plugin is an OS package, so its removal is observable in the package + # facts - the counterpart to install's "is present" check. + - name: 'Gather the package facts' + ansible.builtin.package_facts: # yamllint disable-line rule:empty-values + + - name: 'Assert that the example plugin package is absent' + ansible.builtin.assert: + that: '"example-plugin-auth-ldap" not in ansible_facts["packages"]' + + # The user is managed over the REST API, so assert its absence the same way + # install asserts its presence: a GET for it now returns 404. + - name: 'curl http://127.0.0.1:8080/api/user/example-admin' + ansible.builtin.uri: + url: 'http://127.0.0.1:8080/api/user/example-admin' + status_code: 404 diff --git a/extensions/molecule/inventory/group_vars/all.yml b/extensions/molecule/inventory/group_vars/all.yml new file mode 100644 index 00000000..312e60d9 --- /dev/null +++ b/extensions/molecule/inventory/group_vars/all.yml @@ -0,0 +1,5 @@ +# Prefix host-visible resource names (libvirt domains, podman containers, and the guest hostname +# the kvm_vm role derives from kvm_vm__name) so Molecule's throwaway instances are easy to spot +# in `virsh list` / `podman ps` and never clash with anything else on the host. inventory_hostname +# stays the short name that Ansible itself uses to identify the host. +__molecule__instance_name: 'lfops-molecule-{{ inventory_hostname }}' diff --git a/extensions/molecule/inventory/host_vars/debian11-container.yml b/extensions/molecule/inventory/host_vars/debian11-container.yml new file mode 100644 index 00000000..7ffbd690 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/debian11-container.yml @@ -0,0 +1,4 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_image: 'docker.io/library/debian:11' diff --git a/extensions/molecule/inventory/host_vars/debian11-vm.yml b/extensions/molecule/inventory/host_vars/debian11-vm.yml new file mode 100644 index 00000000..903931f0 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/debian11-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'debian-11-genericcloud-amd64.qcow2' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'debian11' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2' diff --git a/extensions/molecule/inventory/host_vars/debian12-container.yml b/extensions/molecule/inventory/host_vars/debian12-container.yml new file mode 100644 index 00000000..db02979a --- /dev/null +++ b/extensions/molecule/inventory/host_vars/debian12-container.yml @@ -0,0 +1,4 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_image: 'docker.io/library/debian:12' diff --git a/extensions/molecule/inventory/host_vars/debian12-vm.yml b/extensions/molecule/inventory/host_vars/debian12-vm.yml new file mode 100644 index 00000000..5b891589 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/debian12-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'debian-12-genericcloud-amd64.qcow2' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'debian12' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2' diff --git a/extensions/molecule/inventory/host_vars/debian13-container.yml b/extensions/molecule/inventory/host_vars/debian13-container.yml new file mode 100644 index 00000000..c209eccd --- /dev/null +++ b/extensions/molecule/inventory/host_vars/debian13-container.yml @@ -0,0 +1,4 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_image: 'docker.io/library/debian:13' diff --git a/extensions/molecule/inventory/host_vars/debian13-vm.yml b/extensions/molecule/inventory/host_vars/debian13-vm.yml new file mode 100644 index 00000000..6f5779e3 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/debian13-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'debian-13-genericcloud-amd64.qcow2' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'debian13' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2' diff --git a/extensions/molecule/inventory/host_vars/localhost.yml b/extensions/molecule/inventory/host_vars/localhost.yml new file mode 100644 index 00000000..4f862fb4 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/localhost.yml @@ -0,0 +1 @@ +ansible_python_interpreter: '/usr/bin/python3' # needs to point to the python3 version that has the libvirt module (`python3 -c 'import libvirt'` has to work) diff --git a/extensions/molecule/inventory/host_vars/rocky10-container.yml b/extensions/molecule/inventory/host_vars/rocky10-container.yml new file mode 100644 index 00000000..555bbd3f --- /dev/null +++ b/extensions/molecule/inventory/host_vars/rocky10-container.yml @@ -0,0 +1,6 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_command: '/usr/sbin/init' +molecule__container_image: 'docker.io/rockylinux/rockylinux:10-ubi-init' +molecule__container_systemd: true diff --git a/extensions/molecule/inventory/host_vars/rocky10-vm.yml b/extensions/molecule/inventory/host_vars/rocky10-vm.yml new file mode 100644 index 00000000..97afc9d8 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/rocky10-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'Rocky-10-GenericCloud-Base.latest.x86_64.qcow2' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'rocky10' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2' diff --git a/extensions/molecule/inventory/host_vars/rocky8-container.yml b/extensions/molecule/inventory/host_vars/rocky8-container.yml new file mode 100644 index 00000000..fcebbdd6 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/rocky8-container.yml @@ -0,0 +1,6 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_command: '/usr/sbin/init' +molecule__container_image: 'docker.io/rockylinux/rockylinux:8-ubi-init' +molecule__container_systemd: true diff --git a/extensions/molecule/inventory/host_vars/rocky8-vm.yml b/extensions/molecule/inventory/host_vars/rocky8-vm.yml new file mode 100644 index 00000000..ed34d5da --- /dev/null +++ b/extensions/molecule/inventory/host_vars/rocky8-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'Rocky-8-GenericCloud-Base.latest.x86_64.qcow2' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'rocky8' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://dl.rockylinux.org/pub/rocky/8/images/x86_64/Rocky-8-GenericCloud-Base.latest.x86_64.qcow2' diff --git a/extensions/molecule/inventory/host_vars/rocky9-container.yml b/extensions/molecule/inventory/host_vars/rocky9-container.yml new file mode 100644 index 00000000..b89ffec6 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/rocky9-container.yml @@ -0,0 +1,6 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_command: '/usr/sbin/init' +molecule__container_image: 'docker.io/rockylinux/rockylinux:9-ubi-init' +molecule__container_systemd: true diff --git a/extensions/molecule/inventory/host_vars/rocky9-vm.yml b/extensions/molecule/inventory/host_vars/rocky9-vm.yml new file mode 100644 index 00000000..63ad3768 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/rocky9-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'Rocky-9-GenericCloud-Base.latest.x86_64.qcow2' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'rocky9' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2004-container.yml b/extensions/molecule/inventory/host_vars/ubuntu2004-container.yml new file mode 100644 index 00000000..cfc57553 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2004-container.yml @@ -0,0 +1,4 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_image: 'docker.io/library/ubuntu:20.04' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2004-vm.yml b/extensions/molecule/inventory/host_vars/ubuntu2004-vm.yml new file mode 100644 index 00000000..59fba93f --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2004-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'focal-server-cloudimg-amd64.img' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'ubuntu20.04' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2204-container.yml b/extensions/molecule/inventory/host_vars/ubuntu2204-container.yml new file mode 100644 index 00000000..ea7cf7f4 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2204-container.yml @@ -0,0 +1,4 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_image: 'docker.io/library/ubuntu:22.04' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2204-vm.yml b/extensions/molecule/inventory/host_vars/ubuntu2204-vm.yml new file mode 100644 index 00000000..97e0235c --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2204-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'jammy-server-cloudimg-amd64.img' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'ubuntu22.04' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2404-container.yml b/extensions/molecule/inventory/host_vars/ubuntu2404-container.yml new file mode 100644 index 00000000..58352666 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2404-container.yml @@ -0,0 +1,4 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_image: 'docker.io/library/ubuntu:24.04' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2404-vm.yml b/extensions/molecule/inventory/host_vars/ubuntu2404-vm.yml new file mode 100644 index 00000000..773f4ffe --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2404-vm.yml @@ -0,0 +1,16 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'noble-server-cloudimg-amd64.img' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +kvm_vm__osinfo: 'ubuntu24.04' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2604-container.yml b/extensions/molecule/inventory/host_vars/ubuntu2604-container.yml new file mode 100644 index 00000000..3ddaadd8 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2604-container.yml @@ -0,0 +1,4 @@ +ansible_connection: 'containers.podman.podman' +ansible_host: '{{ __molecule__instance_name }}' + +molecule__container_image: 'docker.io/library/ubuntu:26.04' diff --git a/extensions/molecule/inventory/host_vars/ubuntu2604-vm.yml b/extensions/molecule/inventory/host_vars/ubuntu2604-vm.yml new file mode 100644 index 00000000..20b48458 --- /dev/null +++ b/extensions/molecule/inventory/host_vars/ubuntu2604-vm.yml @@ -0,0 +1,19 @@ +ansible_user: 'root' + +kvm_vm__autostart: false +kvm_vm__base_image: 'resolute-server-cloudimg-amd64.img' +kvm_vm__boot_disk_size: '20G' +kvm_vm__host: 'localhost' +kvm_vm__memory: 4096 +kvm_vm__network_connections: + - name: 'enp1s0' + network_name: 'default' + dhcp4: true +# libosinfo does not ship a dedicated 'ubuntu26.04' entry yet, so use the LTS-latest alias. It +# only seeds hypervisor hardware defaults (identical for recent Ubuntu) and starts resolving to +# 26.04 once libosinfo learns it. +kvm_vm__osinfo: 'ubuntu-lts-latest' +kvm_vm__packages: [] +kvm_vm__vcpus: 2 + +molecule__vm_image_url: 'https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img' diff --git a/extensions/molecule/inventory/hosts.yml b/extensions/molecule/inventory/hosts.yml new file mode 100644 index 00000000..ef30ee4a --- /dev/null +++ b/extensions/molecule/inventory/hosts.yml @@ -0,0 +1 @@ +# Required for this directory to be an Ansible inventory source diff --git a/extensions/molecule/kernel_settings/sysctl/converge.yml b/extensions/molecule/kernel_settings/sysctl/converge.yml new file mode 100644 index 00000000..b0cbd950 --- /dev/null +++ b/extensions/molecule/kernel_settings/sysctl/converge.yml @@ -0,0 +1,2 @@ +- name: 'Converge kernel_settings playbook' + ansible.builtin.import_playbook: 'linuxfabrik.lfops.kernel_settings' diff --git a/extensions/molecule/kernel_settings/sysctl/inventory/group_vars/systems_under_test.yml b/extensions/molecule/kernel_settings/sysctl/inventory/group_vars/systems_under_test.yml new file mode 100644 index 00000000..b7927d41 --- /dev/null +++ b/extensions/molecule/kernel_settings/sysctl/inventory/group_vars/systems_under_test.yml @@ -0,0 +1,3 @@ +kernel_settings__sysctl__group_var: + - name: 'vm.overcommit_memory' + value: 1 diff --git a/extensions/molecule/kernel_settings/sysctl/inventory/hosts.yml b/extensions/molecule/kernel_settings/sysctl/inventory/hosts.yml new file mode 100644 index 00000000..e5ead358 --- /dev/null +++ b/extensions/molecule/kernel_settings/sysctl/inventory/hosts.yml @@ -0,0 +1,10 @@ +# yamllint disable rule:empty-values +lfops_kernel_settings: + children: + systems_under_test: + +systems_under_test: + hosts: + rocky8-vm: + rocky9-vm: + rocky10-vm: diff --git a/extensions/molecule/kernel_settings/sysctl/molecule.yml b/extensions/molecule/kernel_settings/sysctl/molecule.yml new file mode 100644 index 00000000..1e47cbff --- /dev/null +++ b/extensions/molecule/kernel_settings/sysctl/molecule.yml @@ -0,0 +1 @@ +# Molecule scenario marker diff --git a/extensions/molecule/kernel_settings/sysctl/verify.yml b/extensions/molecule/kernel_settings/sysctl/verify.yml new file mode 100644 index 00000000..82f1b5d4 --- /dev/null +++ b/extensions/molecule/kernel_settings/sysctl/verify.yml @@ -0,0 +1,12 @@ +- name: 'Verify vm.overcommit_memory is set' + hosts: 'systems_under_test' + gather_facts: false + tasks: + - name: 'Read sysctl vm.overcommit_memory from procfs' + ansible.builtin.slurp: + src: '/proc/sys/vm/overcommit_memory' + register: '__molecule__sysctl_vm_overcommit_memory_result' + + - name: 'Assert that vm.overcommit_memory is set to 1' + ansible.builtin.assert: + that: '__molecule__sysctl_vm_overcommit_memory_result["content"] | ansible.builtin.b64decode | int == 1' diff --git a/extensions/molecule/monitoring_plugins/converge.yml b/extensions/molecule/monitoring_plugins/converge.yml new file mode 100644 index 00000000..fc3b8a9d --- /dev/null +++ b/extensions/molecule/monitoring_plugins/converge.yml @@ -0,0 +1,2 @@ +- name: 'Converge monitoring_plugins playbook' + ansible.builtin.import_playbook: 'linuxfabrik.lfops.monitoring_plugins' diff --git a/extensions/molecule/monitoring_plugins/inventory/group_vars/systems_under_test.yml b/extensions/molecule/monitoring_plugins/inventory/group_vars/systems_under_test.yml new file mode 100644 index 00000000..d59c92ea --- /dev/null +++ b/extensions/molecule/monitoring_plugins/inventory/group_vars/systems_under_test.yml @@ -0,0 +1 @@ +monitoring_plugins__version: '5.0.0' diff --git a/extensions/molecule/monitoring_plugins/inventory/hosts.yml b/extensions/molecule/monitoring_plugins/inventory/hosts.yml new file mode 100644 index 00000000..8b8a4e59 --- /dev/null +++ b/extensions/molecule/monitoring_plugins/inventory/hosts.yml @@ -0,0 +1,21 @@ +# yamllint disable rule:empty-values +lfops_monitoring_plugins: + children: + systems_under_test: + +lfops_repo_monitoring_plugins: + children: + systems_under_test: + +systems_under_test: + hosts: + debian11-vm: + debian12-vm: + debian13-vm: + rocky8-vm: + rocky9-vm: + rocky10-vm: + ubuntu2004-vm: + ubuntu2204-vm: + ubuntu2404-vm: + ubuntu2604-vm: diff --git a/extensions/molecule/monitoring_plugins/molecule.yml b/extensions/molecule/monitoring_plugins/molecule.yml new file mode 100644 index 00000000..1e47cbff --- /dev/null +++ b/extensions/molecule/monitoring_plugins/molecule.yml @@ -0,0 +1 @@ +# Molecule scenario marker diff --git a/extensions/molecule/monitoring_plugins/verify.yml b/extensions/molecule/monitoring_plugins/verify.yml new file mode 100644 index 00000000..ea52c172 --- /dev/null +++ b/extensions/molecule/monitoring_plugins/verify.yml @@ -0,0 +1,12 @@ +- name: 'Verify monitoring plugins are installed' + hosts: 'systems_under_test' + gather_facts: false + tasks: + - name: 'stat /usr/lib64/nagios/plugins/about-me' + ansible.builtin.stat: + path: '/usr/lib64/nagios/plugins/about-me' + register: '__molecule__about_me_plugin_stat_result' + + - name: 'Assert that about-me monitoring plugin is installed' + ansible.builtin.assert: + that: '__molecule__about_me_plugin_stat_result["stat"]["exists"]' diff --git a/extensions/molecule/playbooks/README.md b/extensions/molecule/playbooks/README.md new file mode 100644 index 00000000..d3f35f1b --- /dev/null +++ b/extensions/molecule/playbooks/README.md @@ -0,0 +1,137 @@ +# Molecule Provisioning Playbooks + +These shared playbooks back the Molecule test scenarios under `extensions/molecule`. They provision the systems under test as either containers or VMs and are wired into Molecule through the `create`, `prepare`, and `destroy` hooks in `extensions/molecule/config.yml`. A scenario can switch backends by overriding those hooks in its own `molecule.yml`. + +This directory also serves as a reference for how the LFOps test setup spins instances up and down. The variables consumed here are set per host in the shared inventory (`extensions/molecule/inventory/host_vars`) or in a scenario-specific inventory. + +Two backends are provided: + +* libvirt/KVM virtual machines, provisioned through the [linuxfabrik.lfops.kvm_vm](https://github.com/Linuxfabrik/lfops/tree/main/roles/kvm_vm) role. +* Podman containers. + + +## Mandatory Requirements + +* The collections listed in `extensions/molecule/requirements.yml` (`community.crypto`, `community.libvirt`, `containers.podman`). Molecule installs them during the `dependency` stage. + + +## Optional Requirements + +Depending on the backend a scenario uses: + +* VM backend: + + * A libvirt/KVM hypervisor reachable as the create host (`kvm_vm__host`, `localhost` by default), with a `default` storage pool and a `default` network. + * `virsh` on the Ansible controller. + * Passwordless sudo on the Ansible controller. + +* Container backend: + + * Podman on the Ansible controller. + + +## Playbooks + +`vm-create.yml` + +* Generates an ephemeral SSH keypair, downloads the cloud image into the libvirt storage pool (when `molecule__vm_image_url` is set), creates the VMs through the `kvm_vm` role, waits for each to obtain an IP address, and writes a dynamic inventory so Molecule can reach them over SSH. + +`vm-prepare.yml` + +* Waits for SSH to become available on each VM and gathers facts. + +`vm-destroy.yml` + +* Removes the VMs through the `kvm_vm` role and deletes the ephemeral keypair and dynamic inventory. + +`container-create.yml` + +* Starts one Podman container per host. Fails early and prints the container log when a container does not come up. + +`container-prepare.yml` + +* Installs Python inside the container when the image does not ship it, then gathers facts. + +`container-destroy.yml` + +* Removes the containers. + + +## Mandatory Variables + +`molecule__container_image` + +* Container backend only. The image to start the container from. +* Type: String. + +Example: +```yaml +# mandatory (container backend) +molecule__container_image: 'docker.io/rockylinux/rockylinux:9-ubi-init' +``` + + +## Optional Variables + +`molecule__container_capabilities` + +* Container backend only. Linux capabilities to add to the container. +* Type: List. +* Default: unset (Podman default). + +`molecule__container_command` + +* Container backend only. The command the container runs to stay alive. +* Type: String. +* Default: `'sleep 1d'` + +`molecule__container_log_driver` + +* Container backend only. The Podman log driver. +* Type: String. +* Default: `'json-file'` + +`molecule__container_privileged` + +* Container backend only. Run the container in privileged mode. +* Type: Bool. +* Default: `false` + +`molecule__container_systemd` + +* Container backend only. Run the container with systemd enabled. +* Type: Bool. +* Default: `false` + +`molecule__container_volumes` + +* Container backend only. Volumes to mount into the container. +* Type: List. +* Default: unset (no volumes). + +`molecule__vm_image_url` + +* VM backend only. URL of the cloud image to download into the libvirt storage pool. When unset, no image is downloaded and the base image is assumed to exist already. +* Type: String. + +The VM backend additionally consumes the `kvm_vm__*` variables, which are passed straight to the [linuxfabrik.lfops.kvm_vm](https://github.com/Linuxfabrik/lfops/tree/main/roles/kvm_vm) role; see that role's README for their meaning. The playbooks set `kvm_vm__ssh_authorized_keys` and `kvm_vm__state` themselves. + +Example: +```yaml +# optional (container backend) +molecule__container_command: '/usr/sbin/init' +molecule__container_systemd: true + +# optional (VM backend) +molecule__vm_image_url: 'https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2' +``` + + +## License + +[The Unlicense](https://unlicense.org/) + + +## Author Information + +[Linuxfabrik GmbH, Zurich](https://www.linuxfabrik.ch) diff --git a/extensions/molecule/playbooks/container-create.yml b/extensions/molecule/playbooks/container-create.yml new file mode 100644 index 00000000..b90c89ad --- /dev/null +++ b/extensions/molecule/playbooks/container-create.yml @@ -0,0 +1,47 @@ +- name: 'Create container instances' + hosts: 'systems_under_test' + gather_facts: false + tasks: + + - name: 'Create the container and confirm it is running' + block: + + - name: 'Create the container from inventory' + containers.podman.podman_container: + capabilities: '{{ molecule__container_capabilities | default(omit) }}' + command: '{{ molecule__container_command | default("sleep 1d") }}' + hostname: '{{ __molecule__instance_name }}' + image: '{{ molecule__container_image }}' + log_driver: '{{ molecule__container_log_driver | default("json-file") }}' + name: '{{ __molecule__instance_name }}' + privileged: '{{ molecule__container_privileged | default(false) | bool }}' + state: 'started' + systemd: '{{ molecule__container_systemd | default(false) | bool }}' + volumes: '{{ molecule__container_volumes | default(omit) }}' + register: '__molecule__container_result' + delegate_to: 'localhost' + + # podman reports the container as 'started' even when the entrypoint exits immediately, + # so assert on the actual state. A failure here drops into the rescue, which surfaces the + # container log to explain why it did not come up. + - name: 'Assert the container is running' + ansible.builtin.assert: + that: + - '__molecule__container_result["container"]["State"]["ExitCode"] == 0' + - '__molecule__container_result["container"]["State"]["Running"]' + quiet: true + + rescue: + + - name: 'Retrieve the container log' + ansible.builtin.command: + cmd: 'podman logs {{ __molecule__instance_name }}' + changed_when: false + register: '__molecule__container_log_result' + delegate_to: 'localhost' + + - name: 'Fail with the container log' + ansible.builtin.fail: + msg: | + Container {{ __molecule__instance_name }} failed to start properly. + Log output: {{ __molecule__container_log_result["stdout"] | d("No logs available") }} diff --git a/extensions/molecule/playbooks/container-destroy.yml b/extensions/molecule/playbooks/container-destroy.yml new file mode 100644 index 00000000..fef6a3ae --- /dev/null +++ b/extensions/molecule/playbooks/container-destroy.yml @@ -0,0 +1,11 @@ +- name: 'Destroy container instances' + hosts: 'systems_under_test' + gather_facts: false + tasks: + + - name: 'Kill container if running' + containers.podman.podman_container: + name: '{{ __molecule__instance_name }}' + state: 'absent' + timeout: 2 + delegate_to: 'localhost' diff --git a/extensions/molecule/playbooks/container-prepare.yml b/extensions/molecule/playbooks/container-prepare.yml new file mode 100644 index 00000000..8349b9d1 --- /dev/null +++ b/extensions/molecule/playbooks/container-prepare.yml @@ -0,0 +1,14 @@ +- name: 'Prepare containers for Ansible' + hosts: 'systems_under_test' + gather_facts: false + tasks: + - name: 'Install Python using the raw module (in case it is missing from the container image)' + ansible.builtin.raw: | + if command -v dnf > /dev/null 2>&1; then + dnf install -y python3 + elif command -v apt-get > /dev/null 2>&1; then + apt-get update && apt-get install -y python3 + fi + + - name: 'Gather facts after Python is available' + ansible.builtin.setup: # yamllint disable-line rule:empty-values diff --git a/extensions/molecule/playbooks/vm-create.yml b/extensions/molecule/playbooks/vm-create.yml new file mode 100644 index 00000000..999a232c --- /dev/null +++ b/extensions/molecule/playbooks/vm-create.yml @@ -0,0 +1,129 @@ +# Shared, controller-side setup. Runs once on localhost (the hypervisor), so there is no +# delegate_to / run_once juggling. The keypair and pool info become normal facts on localhost +# that the later plays read via hostvars["localhost"]. +- name: 'Prepare the controller for VM creation' + hosts: 'localhost' + gather_facts: false + vars: + __molecule__ephemeral_dir: '{{ lookup("env", "MOLECULE_EPHEMERAL_DIRECTORY") }}' + tasks: + + - name: 'Generate the ephemeral SSH keypair' + community.crypto.openssh_keypair: + path: '{{ __molecule__ephemeral_dir }}/molecule_key' + type: 'ed25519' + comment: 'molecule' + register: '__molecule__keypair_result' + + - name: 'Get the libvirt storage pool info' + community.libvirt.virt_pool: + command: 'info' + uri: '{{ kvm_vm__connect_url | d("qemu:///system") }}' + register: '__molecule__pool_result' + + - name: 'Ensure the ephemeral inventory directory exists' + ansible.builtin.file: + path: '{{ __molecule__ephemeral_dir }}/inventory' + state: 'directory' + mode: 0o755 + + +# One VM per host in systems_under_test. The kvm_vm role delegates to kvm_vm__host, so this play +# never connects to the (not yet existing) target; the play-level become is for the role's +# privileged libvirt operations and the image download into the root-owned pool. +- name: 'Create VM instances using the kvm_vm role' + hosts: 'systems_under_test' + become: true + gather_facts: false + vars: + kvm_vm__name: '{{ __molecule__instance_name }}' + # Throwaway test VMs: set a known root password for console login during debugging. SSH still + # uses the ephemeral keypair below. + kvm_vm__root_password: 'linuxfabrik' + kvm_vm__ssh_authorized_keys: + - '{{ hostvars["localhost"]["__molecule__keypair_result"]["public_key"] }}' + kvm_vm__state: 'running' + pre_tasks: + + # Raise timeout well above the 10s default: the cloud images are several hundred MB and one + # play downloads all of them at once (forks), so individual connections stall past 10s under + # the shared bandwidth and hit "read operation timed out". get_url defaults to force: false, so + # an image already present in the pool is left untouched and only a missing one is fetched. + - name: 'Download the cloud image into the storage pool' + ansible.builtin.get_url: + url: '{{ molecule__vm_image_url }}' + dest: '{{ hostvars["localhost"]["__molecule__pool_result"]["pools"][kvm_vm__pool | d("default")]["path"] }}/{{ kvm_vm__base_image }}' + mode: 0o644 + timeout: 120 + delegate_to: 'localhost' + when: 'molecule__vm_image_url is defined' + + roles: + - role: 'linuxfabrik.lfops.kvm_vm' + + +# Discover each VM's address via its libvirt DHCP lease and write it into Molecule's ephemeral +# inventory directory. Molecule includes that directory automatically, so converge and verify +# reach the VMs over SSH. See the inventory-model note in config.yml for why this is hand-written +# instead of using Molecule's instance_config. +- name: 'Discover VM addresses and write the dynamic inventory' + hosts: 'systems_under_test' + gather_facts: false + vars: + __molecule__ephemeral_dir: '{{ lookup("env", "MOLECULE_EPHEMERAL_DIRECTORY") }}' + tasks: + + # Use --source arp: libvirt reads the host's neighbour cache (ip neigh on the bridge), so it + # only ever reports addresses currently live on the network. We rejected the two alternatives: + # --source lease reads libvirt's DHCP lease table, which keeps every still-unexpired lease. + # The VMs reuse a stable MAC across destroy/create cycles but regenerate their machine-id + # (hence DHCP client-id) on each rebuilt boot disk, so dnsmasq hands out a fresh IP every + # time and the old leases linger. domifaddr then returns several IPs for one MAC and there + # is no way to tell the live one from the stale ones without sorting by lease expiry. + # --source agent needs qemu-guest-agent running inside the guest, which only the Rocky cloud + # image ships preinstalled - the VMs install no extra packages (kvm_vm__packages is []), so + # on Debian/Ubuntu the agent never connects. + # --source arp sidesteps both: the neighbour cache holds only what is reachable now, so stale + # leases never appear, and it needs nothing inside the guest. The entry is populated once the + # VM emits traffic (its DHCP exchange does this within seconds), so the retry loop below covers + # the short boot window where it is not yet there. + - name: 'Wait for the VM to report a non-loopback IPv4 address via the host ARP table' + ansible.builtin.command: + cmd: 'virsh --connect {{ kvm_vm__connect_url | d("qemu:///system") }} domifaddr {{ __molecule__instance_name }} --source arp' + register: '__molecule__vm_ipaddr_result' + until: >- + __molecule__vm_ipaddr_result['stdout_lines'] + | select('ansible.builtin.search', 'ipv4') + | reject('ansible.builtin.search', '^\s*lo\s') + | list | length > 0 + retries: 15 + delay: 2 + changed_when: false + delegate_to: 'localhost' + + - name: 'Parse the VM IPv4 address' + ansible.builtin.set_fact: + __molecule__vm_ipaddr: >- + {{ + __molecule__vm_ipaddr_result['stdout_lines'] + | select('ansible.builtin.search', 'ipv4') + | reject('ansible.builtin.search', '^\s*lo\s') + | first + | ansible.builtin.regex_search('([0-9]{1,3}\.){3}[0-9]{1,3}') + }} + + - name: 'Write the dynamic VM inventory' + ansible.builtin.copy: + content: | + molecule: + hosts: + {% for host in ansible_play_hosts %} + {{ host }}: + ansible_host: '{{ hostvars[host]["__molecule__vm_ipaddr"] }}' + ansible_ssh_private_key_file: '{{ __molecule__ephemeral_dir }}/molecule_key' + ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' + {% endfor %} + dest: '{{ __molecule__ephemeral_dir }}/inventory/molecule_vm.yml' + mode: 0o644 + delegate_to: 'localhost' + run_once: true # single write that loops over ansible_play_hosts to emit every host at once diff --git a/extensions/molecule/playbooks/vm-destroy.yml b/extensions/molecule/playbooks/vm-destroy.yml new file mode 100644 index 00000000..fbe76233 --- /dev/null +++ b/extensions/molecule/playbooks/vm-destroy.yml @@ -0,0 +1,26 @@ +- name: 'Destroy VM instances using the kvm_vm role' + hosts: 'systems_under_test' + become: true + gather_facts: false + roles: + - role: 'linuxfabrik.lfops.kvm_vm' + vars: + kvm_vm__name: '{{ __molecule__instance_name }}' + kvm_vm__state: 'absent' + + +- name: 'Clean up Molecule artifacts on the controller' + hosts: 'localhost' + gather_facts: false + vars: + __molecule__ephemeral_dir: '{{ lookup("env", "MOLECULE_EPHEMERAL_DIRECTORY") }}' + tasks: + + - name: 'Remove the ephemeral SSH keypair and dynamic inventory' + ansible.builtin.file: + path: '{{ __molecule__ephemeral_dir }}/{{ item }}' + state: 'absent' + loop: + - 'molecule_key' + - 'molecule_key.pub' + - 'inventory/molecule_vm.yml' diff --git a/extensions/molecule/playbooks/vm-prepare.yml b/extensions/molecule/playbooks/vm-prepare.yml new file mode 100644 index 00000000..d591dbad --- /dev/null +++ b/extensions/molecule/playbooks/vm-prepare.yml @@ -0,0 +1,20 @@ +- name: 'Prepare VMs for Ansible' + hosts: 'systems_under_test' + gather_facts: false + tasks: + - name: 'Wait for SSH to be available' + ansible.builtin.wait_for_connection: + timeout: 120 + + - name: 'Gather facts' + ansible.builtin.setup: # yamllint disable-line rule:empty-values + + # Debian/Ubuntu genericcloud images ship with an empty apt index (/var/lib/apt/lists is cleaned + # at image-build time), so the first package install cannot resolve any package until the cache + # is refreshed. Scenarios that pull in a repo_* role get this for free; ones that install a bare + # package (e.g. apps) would otherwise fail with "No package matching '' is available". + - name: 'apt-get update' + ansible.builtin.apt: + update_cache: true + when: + - 'ansible_facts["os_family"] == "Debian"' diff --git a/extensions/molecule/requirements-roles.yml b/extensions/molecule/requirements-roles.yml new file mode 100644 index 00000000..0b7d8cf8 --- /dev/null +++ b/extensions/molecule/requirements-roles.yml @@ -0,0 +1,5 @@ +# The galaxy dependency always runs both a roles and a collections sub-step. We have no Galaxy +# roles to install (everything is a local collection, see requirements.yml), but pointing the +# roles sub-step at this empty file gives it something to read so it does not warn "Missing roles +# requirements file: requirements.yml" on every run. +roles: [] diff --git a/extensions/molecule/requirements.yml b/extensions/molecule/requirements.yml new file mode 100644 index 00000000..fe829fef --- /dev/null +++ b/extensions/molecule/requirements.yml @@ -0,0 +1,5 @@ +collections: + - name: 'community.crypto' + - name: 'community.libvirt' + - name: 'containers.podman' + version: '>=1.10.0' diff --git a/extensions/molecule/setup_basic/converge.yml b/extensions/molecule/setup_basic/converge.yml new file mode 100644 index 00000000..046a2140 --- /dev/null +++ b/extensions/molecule/setup_basic/converge.yml @@ -0,0 +1,2 @@ +- name: 'Converge setup_basic playbook' + ansible.builtin.import_playbook: 'linuxfabrik.lfops.setup_basic' diff --git a/extensions/molecule/setup_basic/inventory/group_vars/systems_under_test.yml b/extensions/molecule/setup_basic/inventory/group_vars/systems_under_test.yml new file mode 100644 index 00000000..c6f5ca51 --- /dev/null +++ b/extensions/molecule/setup_basic/inventory/group_vars/systems_under_test.yml @@ -0,0 +1,11 @@ +mailto_root__from: 'root@localhost' +mailto_root__to: + - 'root@localhost' + +monitoring_plugins__version: '5.0.0' + +postfix__relayhost: 'mail.example.com' + +setup_basic__skip_duplicity: true +setup_basic__skip_repo_icinga: true +setup_basic__skip_icinga2_agent: true diff --git a/extensions/molecule/setup_basic/inventory/host_vars/rocky10-vm.yml b/extensions/molecule/setup_basic/inventory/host_vars/rocky10-vm.yml new file mode 100644 index 00000000..8491746e --- /dev/null +++ b/extensions/molecule/setup_basic/inventory/host_vars/rocky10-vm.yml @@ -0,0 +1 @@ +setup_basic__skip_glances: true diff --git a/extensions/molecule/setup_basic/inventory/hosts.yml b/extensions/molecule/setup_basic/inventory/hosts.yml new file mode 100644 index 00000000..eb0eda38 --- /dev/null +++ b/extensions/molecule/setup_basic/inventory/hosts.yml @@ -0,0 +1,17 @@ +# yamllint disable rule:empty-values +lfops_setup_basic: + children: + systems_under_test: + +systems_under_test: + hosts: + debian11-vm: + debian12-vm: + debian13-vm: + rocky8-vm: + rocky9-vm: + rocky10-vm: + ubuntu2004-vm: + ubuntu2204-vm: + ubuntu2404-vm: + ubuntu2604-vm: diff --git a/extensions/molecule/setup_basic/molecule.yml b/extensions/molecule/setup_basic/molecule.yml new file mode 100644 index 00000000..1e47cbff --- /dev/null +++ b/extensions/molecule/setup_basic/molecule.yml @@ -0,0 +1 @@ +# Molecule scenario marker diff --git a/extensions/molecule/setup_basic/verify.yml b/extensions/molecule/setup_basic/verify.yml new file mode 100644 index 00000000..84e0ed62 --- /dev/null +++ b/extensions/molecule/setup_basic/verify.yml @@ -0,0 +1,12 @@ +- name: 'Verify setup_basic installed the monitoring plugins' + hosts: 'systems_under_test' + gather_facts: false + tasks: + - name: 'stat /usr/lib64/nagios/plugins/about-me' + ansible.builtin.stat: + path: '/usr/lib64/nagios/plugins/about-me' + register: '__molecule__about_me_plugin_stat_result' + + - name: 'Assert that the about-me monitoring plugin is installed' + ansible.builtin.assert: + that: '__molecule__about_me_plugin_stat_result["stat"]["exists"]'