feat(rhel): add kickstart version stamp, drop dead rm calls, document logs

* Add LF_KICKSTART_VERSION=2026041501 at the top of %pre. The stamp is
  echoed into the pre-log, re-echoed at the start of the generated
  %post (so it shows up in ks-script-*.log), and embedded as a comment
  at the top of the rendered dynamic.ks. Since dynamic.ks is archived
  to /root on the installed system, any host can be traced back to the
  exact lf-rhel.cfg build that installed it. CONTRIBUTING gets a new
  "Versioning the Kickstart File" section that documents the
  YYYYMMDDNN format and when to bump.
* Drop the dead `rm -f /root/anaconda-ks.cfg` and
  `rm -f /root/original-ks.cfg` calls from the cloud post-install
  script. Anaconda writes both files via _writeKS_via_boss and
  CopyLogsTask AFTER all %post scripts, so the rm -f calls ran before
  the files existed and had no effect. A comment at the same spot
  explains the ordering for future maintainers.
* README: add a "Log Files" section to each of the RHEL, Debian and
  Ubuntu chapters, grouping the install-time and post-install log
  files by symptom so an operator can go straight to the right one.
* README: document where to read the kickstart version stamp on an
  installed system and fix the `ll /root` test to match the actual
  cloud image state.

Author: markuslf

CHANGELOG.md

@@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+* `lf-rhel.cfg`: Add a running `LF_KICKSTART_VERSION` build stamp (format `YYYYMMDDNN`) declared at the top of `%pre`. The stamp is echoed during install (ends up in `/tmp/kickstart.install.pre.log`, `/var/log/anaconda/anaconda.log`, and the `%post` `ks-script-*.log`) and embedded as a comment at the top of the rendered `dynamic.ks`, which is archived as `/root/dynamic.ks` on the installed system. This lets admins trace any installed host back to the exact `lf-rhel.cfg` variant that built it. CONTRIBUTING documents when and how to bump the stamp; the README documents where to read it on an installed system
+
 ### Fixed
 
 * `lf-rhel.cfg`: Fix `grub2-mkconfig` being silently skipped on every target version. The `%pre` script was built via an unquoted heredoc (`cat << EOT`), which let bash eat every `"$var"` and `$(cmd)` reference embedded in the generated shell code before the Python helper was even written. As a result the grub finder always ran with empty variables and printed "Could not find a grub.cfg in /boot" — hiding the fact that `/etc/default/grub` edits had not been regenerated since the first release. The heredoc is now quoted (`cat << 'EOT'`), the finder targets `/boot/grub2/grub.cfg` directly (which is the right target on RHEL/Rocky 8/9/10 for both BIOS and UEFI — on 9+ UEFI the `/boot/efi/EFI/<vendor>/grub.cfg` is only a thin wrapper that `configfile`s `/boot/grub2/grub.cfg`), and the console-ordering change in `/etc/default/grub` is actually reflected in the generated `grub.cfg`
@@ -17,12 +21,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
-* `lf-rhel.cfg`: On `lftype=cloud` and `lftype=cloud-cis`, the post-install script now also removes `/root/original-ks.cfg` in addition to `/root/anaconda-ks.cfg`. Both files are written by modern Anaconda and both contain the raw bootstrap kickstart — removing both keeps the base image clean for cloud deployments
-* README: Correct and sharpen the "Modifying this Kickstart" and "Tests" sections. The `lfkeys`, `users_<lftype>` and `post_<lftype>` descriptions now match the actual Python data structures and interpreter behavior, and the `ll /root` test is now explicit about which files to expect per `lftype`
+* README: Add a "Log Files" section to each of the RHEL, Debian and Ubuntu chapters that lists the log files most relevant for diagnosing install-time and post-install problems, grouped by "during install" and "after install", so an operator can go straight to the right log for a given symptom
+* README: Correct and sharpen the "Modifying this Kickstart" and "Tests" sections. The `lfkeys`, `users_<lftype>` and `post_<lftype>` descriptions now match the actual Python data structures and interpreter behavior, and the `ll /root` test now reflects that Anaconda's own kickstart copies (`anaconda-ks.cfg`, `original-ks.cfg`) are present on every install because they are written after all `%post` scripts and cannot be removed from within the kickstart
 
 ### Removed
 
 * `lf-rhel.cfg`: Drop the dead `yum erase authconfig` call from the cloud post-install script. `authconfig` has not existed since RHEL 8 (replaced by `authselect`), so the call was a no-op on every supported target
+* `lf-rhel.cfg`: Drop the dead `rm -f /root/anaconda-ks.cfg` and `rm -f /root/original-ks.cfg` calls from the cloud post-install script. Both files are written by Anaconda's boss module (`_writeKS_via_boss` and `CopyLogsTask`) **after** all `%post` scripts have run, so the removals ran against non-existent files and had no effect. A comment at the same spot now documents the ordering so future maintainers do not reintroduce this
 
 
 ## [1.2.0] - 2026-04-15

CONTRIBUTING.md

@@ -71,6 +71,20 @@ Python packages installed via `pip` inside workflows follow a two-tier policy:
 - One-shot installs such as `ansible-builder`, `build`, `mkdocs`, `pdoc`, and `ruff` in release, docs, or test workflows are version-pinned only (`package==X.Y.Z`) and kept fresh by Dependabot. Scorecard's `pipCommand not pinned by hash` findings for these are considered acceptable risk and may be dismissed.
 
 
+### Versioning the Kickstart File
+
+`lf-rhel.cfg` carries a running build stamp in the shell variable `LF_KICKSTART_VERSION` at the top of the `%pre` section. The value is logged at install time (to `/tmp/kickstart.install.pre.log`, the Anaconda logs, and the `%post` `ks-script-*.log`) and embedded as a comment in the `dynamic.ks` archived under `/root/dynamic.ks` on the installed system, so the origin of any installed host can be traced back to a specific build.
+
+The format is `YYYYMMDDNN`, where `YYYYMMDD` is the build date and `NN` is a two-digit daily sequence number starting at `01`. Example: `2026041501` for the first build on 2026-04-15, `2026041502` for the second on the same day.
+
+Rules:
+
+- Bump `LF_KICKSTART_VERSION` in every commit that changes the effective content of `lf-rhel.cfg` (kickstart logic, embedded shell/Python, comments that end up in the generated output).
+- Pure documentation commits (`README.md`, `CONTRIBUTING.md`, `CHANGELOG.md`) and commits that only touch `lf-debian.cfg` or `lf-ubuntu.cfg` do not bump the RHEL kickstart version.
+- If multiple commits on the same day change `lf-rhel.cfg`, increment `NN` accordingly.
+- Releases (as described in the `Changelog` section) also imply a version bump, because they are also content changes.
+
+
 ### Coding Conventions
 
 - Sort variables, parameters, lists, and similar items alphabetically where possible.

README.md

@@ -147,7 +147,41 @@ What to test within the VM:
 * `sudo dnf -y install nano`: Should work.
 * `systemctl status cloud-init`: Not found on non-cloud, should work on cloud.
 * `systemctl status firewalld`: Should be inactive on non-cloud. On cloud, firewalld is removed.
-* `ll /root`: On `lftype=cis` and `lftype=minimal`, should list `anaconda-ks.cfg` and `original-ks.cfg` (both written by Anaconda), `dynamic.ks` (the rendered kickstart fragment Linuxfabrik applied) and `70-install-ssh-keys.ks` (the SSH key deployment fragment). On `lftype=cloud` and `lftype=cloud-cis`, should list only `dynamic.ks` — both of Anaconda's kickstart copies (`anaconda-ks.cfg` and `original-ks.cfg`) are removed by the cloud post-install script, and no SSH key fragment is generated because `cloud-init` handles keys.
+* `ll /root`: Should list `dynamic.ks` (the rendered kickstart fragment Linuxfabrik applied, always present). Anaconda additionally writes `original-ks.cfg` on every install and, on most targets, `anaconda-ks.cfg` — both are written after all `%post` scripts, so they are not cleaned up by this kickstart. On `lftype=cis` and `lftype=minimal`, `70-install-ssh-keys.ks` is also archived (the SSH key deployment fragment); on `lftype=cloud` and `lftype=cloud-cis`, no SSH key fragment is generated because `cloud-init` handles keys.
+* `grep 'Linuxfabrik Kickstart version' /root/dynamic.ks`: Should show the `YYYYMMDDNN` build stamp of the `lf-rhel.cfg` variant that was applied during installation. The same stamp is also written to `/var/log/anaconda/anaconda.log` and to the `%post` `ks-script-*.log` under `/var/log/anaconda/`, so the origin of any installed host is traceable to a specific `lf-rhel.cfg` build.
+
+
+### Log Files
+
+During and after an Anaconda install, the following log files are the fastest path to diagnosing problems.
+
+**During the install (installer environment, switch to a shell with `Ctrl+Alt+F2` or `Ctrl+Alt+F3`):**
+
+* `/tmp/dynamic.ks`: The dynamically generated kickstart fragment that `%pre` produced. If Anaconda reports a kickstart syntax error in an included file, `cat` this file to see what was actually rendered.
+* `/tmp/kickstart.install.pre.log`: Output of the `%pre` script, including the `LF_KICKSTART_VERSION` stamp, the `lftype`/`lfdisk` detection, and the Python helper's progress messages. First place to look if the install aborts before the package selection.
+* `/tmp/ks-script-*.log`: Per-script output of each `%post` block and each post-script under `/usr/share/anaconda/post-scripts/`. Same files that will later be copied to `/var/log/anaconda/`.
+* `/tmp/pre-script.py`: The Python helper itself. Useful if `%pre` crashes on a specific Python line.
+
+**After a successful install (on the installed system, under `/var/log/anaconda/`):**
+
+* `anaconda.log`: Main Anaconda narrative for the whole install. Start here when a post-install symptom shows up after first boot.
+* `dbus.log`: Anaconda DBus module communication. Only needed for deeper Anaconda-internal issues (for example when the `boss` or `storage` module misbehaves, as with the `original-ks.cfg` write ordering).
+* `journal.log`: Snapshot of the installer's journal at the end of install. Covers kernel, dracut and systemd messages; useful when the installer itself had boot problems.
+* `ks-script-*.log`: Output of each `%post` block. `grep -l 'Linuxfabrik Kickstart version' /var/log/anaconda/ks-script-*.log` identifies the script that ran our `post_cloud`/`post_cis` block; that file contains the yum/dnf transaction output, the systemd-in-chroot warnings, the `grub2-mkconfig` result, and any messages from our cleanup/tweaks.
+* `packaging.log`: DNF/yum transaction detail during the install. Goes deeper than `ks-script-*.log` when a package install or removal looks wrong.
+* `program.log`: External programs Anaconda invoked (e.g. `grub2-mkconfig`, `mkfs.xfs`). Complements the other logs when the failing step was an external binary.
+* `storage.log`: Partitioning, LVM creation, filesystem formatting. First stop for "disk not found", `ignoredisk` or LVM-related failures.
+
+**On the installed system, outside `/var/log/anaconda/`:**
+
+* `/root/anaconda-ks.cfg`: Anaconda's generated copy of the effective kickstart. Not present on targets where `can_save_output_kickstart=False` (including Rocky 10 cloud).
+* `/root/dynamic.ks`: The kickstart fragment Linuxfabrik archived on the installed system. The first comment line reads `# Linuxfabrik Kickstart version: YYYYMMDDNN`, which attributes the host to a specific `lf-rhel.cfg` build.
+* `/root/original-ks.cfg`: Anaconda's verbatim copy of the `lf-rhel.cfg` served from the `inst.ks=` URL.
+
+**For `cloud-init` problems on `lftype=cloud` and `lftype=cloud-cis` (after first boot):**
+
+* `/var/log/cloud-init-output.log`: Stdout/stderr of any user-data scripts that cloud-init executed.
+* `/var/log/cloud-init.log`: Full cloud-init log with DEBUG-level entries. First stop for "my SSH keys aren't there" or "hostname not set".
 
 
 ### Troubleshooting
@@ -217,6 +251,26 @@ What to test within the VM:
 * `sudo apt install -y nano`: Should work.
 
 
+### Log Files
+
+During and after a Debian Installer (d-i) preseed run, the following log files are the fastest path to diagnosing problems.
+
+**During the install (switch to a shell with `Ctrl+Alt+F2`; live installer syslog tails on `Ctrl+Alt+F4`):**
+
+* `/var/log/partman`: Partition manager log. First stop for disk, LVM or filesystem errors during install.
+* `/var/log/syslog`: Main installer syslog. First stop for any preseed or d-i failure.
+
+**After a successful install (on the installed system, under `/var/log/installer/`):**
+
+* `cdebconf/questions.dat`: Preseed questions and the answers that were given (by our preseed file or interactively). Useful to verify that a preseed directive actually reached debconf.
+* `cdebconf/templates.dat`: The debconf template catalogue, in case a preseed key was silently ignored because the template changed.
+* `hardware-summary`: Detected hardware and driver decisions.
+* `lsb-release`, `media-info`: Base distribution and installation media metadata.
+* `partman`: Partition manager log, persisted from the install.
+* `status`: `apt`/`dpkg` status of every package installed during the install phase.
+* `syslog`: Main installer narrative for the whole install.
+
+
 ### References
 
 * [Debian Preseed Documentation](https://www.debian.org/releases/stable/amd64/apb.en.html)
@@ -274,6 +328,24 @@ What to test within the VM:
 * `sudo apt install -y nano`: Should work.
 
 
+### Log Files
+
+During and after a subiquity autoinstall run, the following log files are the fastest path to diagnosing problems.
+
+**During the install (switch to a shell with `Ctrl+Alt+F2`):**
+
+* `/var/log/installer/autoinstall-user-data`: The autoinstall YAML subiquity actually consumed. Useful to verify that the `user-data` file served over HTTP is what subiquity actually parsed.
+* `/var/log/installer/subiquity-server-debug.log`: Subiquity server-side debug log, streams live during install. First stop for any autoinstall/subiquity failure.
+
+**After a successful install (on the installed system, under `/var/log/installer/`):**
+
+* `autoinstall-user-data`: The autoinstall YAML subiquity consumed, persisted.
+* `cloud-init-output.log`, `cloud-init.log`: Cloud-init logs from inside the installer — subiquity uses cloud-init internally to fetch `user-data` and apply some steps.
+* `curtin-install-cfg.yaml`, `curtin-install.log`: Curtin (the actual installer under subiquity) effective config and run log. First stop for partitioning, LVM or grub issues.
+* `installer-journal.txt`: Journal snapshot from the installer environment.
+* `subiquity-client-debug.log`, `subiquity-server-debug.log`: Subiquity's full debug logs.
+
+
 ### References
 
 * [Ubuntu Autoinstall Reference](https://canonical-subiquity.readthedocs-hosted.com/en/latest/reference/autoinstall-reference.html)

lf-rhel.cfg

@@ -74,6 +74,15 @@ services --disabled="kdump" --enabled="NetworkManager,sshd"
 
 
 %pre --logfile /tmp/kickstart.install.pre.log --erroronfail
+# Running kickstart version stamp (YYYYMMDDNN). Bump on every content change
+# per CONTRIBUTING "Versioning". The value is echoed at install time (lands in
+# /tmp/kickstart.install.pre.log and /var/log/anaconda/*) and re-echoed by the
+# %post scripts; the Python helper picks it up from the environment and
+# embeds it as a comment in the generated dynamic.ks so it is visible on the
+# installed system via /root/dynamic.ks.
+LF_KICKSTART_VERSION=2026041501
+export LF_KICKSTART_VERSION
+echo "Linuxfabrik Kickstart version: $LF_KICKSTART_VERSION"
 # rhel8 only has platform-python, rhel9+ has python3, so we'll have to detect what we have
 PYTHON=$(ls /usr/libexec/platform-python /usr/bin/python3 2> /dev/null | head -n1)
 # Use a quoted heredoc delimiter ('EOT') so bash performs no variable or command
@@ -109,6 +118,12 @@ for guess in ['vda', 'sda', 'nvme0n1']:
         lfdisk = guess
         break
 
+# Pick up the running kickstart version from the env var the outer %pre
+# exported. Use 'unknown' as a defensive fallback so we never crash the
+# install if the variable is missing for some reason.
+lf_kickstart_version = os.environ.get('LF_KICKSTART_VERSION', 'unknown')
+print('Linuxfabrik Kickstart version: {}'.format(lf_kickstart_version))
+
 print('Linuxfabrik Kickstart: Set kernel cmdline arguments')
 with open('/proc/cmdline', 'r') as file:
     cmdline = file.read()
@@ -381,9 +396,12 @@ echo 'Remove /etc/machine-id'
 truncate --size=0 /etc/machine-id
 
 echo 'Remove various other log / cache files'
+# Note: /root/anaconda-ks.cfg and /root/original-ks.cfg are intentionally
+# NOT removed here. Modern Anaconda writes both files via the boss module
+# (_writeKS_via_boss and CopyLogsTask) AFTER all %post scripts have run,
+# so any rm -f here would run before the files exist and be a no-op.
+# See pyanaconda/modules/boss/installation.py in rhinstaller/anaconda.
 rm -f /root/.wget-hsts
-rm -f /root/anaconda-ks.cfg
-rm -f /root/original-ks.cfg
 rm -f /root/install.log
 rm -f /root/install.log.syslog
 rm -rf /var/lib/yum/*
@@ -452,6 +470,7 @@ print('Linuxfabrik Kickstart: Write dynamic.ks')
 dynamic = []
 keypost = []
 
+dynamic.append('# Linuxfabrik Kickstart version: {}'.format(lf_kickstart_version))
 dynamic.append('# System users')
 for user in users:
     dynamic.append(user['cmd'])
@@ -494,6 +513,7 @@ dynamic.extend(packages)
 dynamic.append('%end')
 
 dynamic.append('%post --erroronfail')
+dynamic.append('echo "Linuxfabrik Kickstart version: {}"'.format(lf_kickstart_version))
 dynamic.append(post)
 dynamic.append('%end\n')