Forward‑porting Azure-related changes from Jammy to Noble (#476)

* Fix cloud-init logging format

The typo leads to problems when cloud-init attempts to load the logging
configuration. This is evident in the logs, as shown by the following
stack trace:

> cloud-init[515]: 2025-09-24 12:07:29,260 - util.py[WARNING]: Failed
>   loading yaml blob. Invalid format at line 10 column 1: "expected
?   '<document start>', but found '<block mapping start>'
> cloud-init[515]:   in "<unicode string>", line 10, column 1:
> cloud-init[515]:     _log:
> cloud-init[515]:     ^"

* Disable floppy module

* Prevent kernel from auto-loading floppy

In a previous change (#449), loading the floppy module was disabled by
redirecting the floppy command to true. However, Buffer I/O errors
observed in the `dmesg` output indicate that the kernel is still
attempting to load the floppy module when the hardware supports it.

```txt
blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
floppy0: disk absent or changed during operation
```

This suggests that during boot:
1. Kernel detects floppy controller hardware
2. udev/kernel auto-loads floppy module (install directive not active yet)
3. Floppy driver starts, finds no disk → I/O errors
4. install directive becomes active (too late!)

The blacklist in /etc/modprobe.d only affects modprobe after the root
filesystem is active. If the initramfs contains a `floppy.ko` file and
is not rebuilt after the blacklist has been applied, the initramfs
auto-loads the floppy driver.
`update-initramfs` rebuilds the initramfs and includes the /etc/modprobe.d
rules to the root fs.

* Mount /sys in system_kernel_modules for initramfs generation

The system_kernel_modules stage fails when running update-initramfs
with FIPS kernels because the FIPS initramfs hooks require /sys to be
mounted for hardware introspection when MODULES=dep is configured.

The run_in_chroot helper creates an isolated mount namespace via
unshare -m, which only mounts /dev and /proc but not /sys. This causes
mkinitramfs to fail with "MODULES dep requires mounted sysfs on /sys"
when the FIPS hooks attempt to scan hardware capabilities.

Mount /sys explicitly in system_kernel_modules/apply.sh before calling
update-initramfs to ensure the kernel device model and driver
information is available for initramfs generation.

* Add hvperv-kv daemon for Azure stemcells

Azure runs on the Hyper-V hypervisor, and Linux VMs rely on Linux Integration
Services (LIS) to communicate with the host. These services include kernel
modules (like hv_utils) and daemons (hv_kvp_daemon, hv_vss_daemon, etc.) that
handle:

- Host-to-guest communication
- IP address reporting
- VM metadata exchange
- Backup coordination (via VSS)

The hv_kvp_daemon specifically manages key-value pair exchange between the
Azure host and the Linux guest. If the daemon is missing, Azure may not
be able to retrieve guest metadata (e.g., hostname, IP address) This can
affect Azure Resource Graph queries, scripts that rely on guest-reported
data, backup and monitoring tools that use KVP for coordination

cloud-init writes warnings about events that could not be sent to hv-kvp:

```txt
$ grep kvp /var/log/cloud-init.log
failed to truncate kvp pool file, [Errno 2] No such file or directory: '/var/lib/hyperv/.kvp_pool_1'
failed posting events to kvp, [Errno 2] No such file or directory: '/var/lib/hyperv/.kvp_pool_1'
```

* Add missing tests for waagent.service

* Add linux-cloud-tools to expected azure packages

Fixes #453

* Enable cloud-init apt-configure module for Azure stemcells

Azure stemcells now properly use Azure-optimized APT mirrors
(azure.archive.ubuntu.com) by enabling the apt-configure module
in cloud-init configuration.

The LISA (Linux Integration Services Automation) test suite's
verify_repository_installed test was failing because Azure VMs
were using archive.ubuntu.com instead of azure.archive.ubuntu.com
for APT package repositories. While the Azure mirror configuration
existed in /etc/cloud/cloud.cfg.d/90-azure-apt-sources.cfg, it
was never applied because the apt-configure module was not enabled
in cloud-init's module list.

Added apt-configure to the cloud_init_modules list in cloud.cfg,
which instructs cloud-init to read and apply the Azure APT mirror
configuration at VM boot time. This ensures /etc/apt/sources.list
is automatically updated to use Azure-optimized mirrors.

* Update waagent to `v2.15.0.1` and keep systemd config in sync

 * upgrades waagent to the latest version
 * syncs the systemd config, which adds memory-accounting and a dependency
   to cloud-init.service, which ensures that the waagent.service does not start
   before cloud-init has finished.

<https://github.com/Azure/WALinuxAgent/blob/v2.15.0.1/init/ubuntu/walinuxagent.service>

* Initialize log directory required by waagent

waagent assumes that the `/var/log/azure` directory exists and raises an
exception in the telemetry module if it does not.

```sh
ERROR TelemetryEventsCollector ExtHandler Event: name=WALinuxAgent,
op=ExtensionTelemetryEventProcessing, message=Unknown error occurred
when trying to collect extension events:[Errno 2] No such file or
directory: '/var/log/azure'
```

* Improve chrony clock settings

Co-Authored-By: Sebastian Heid <8442432+s4heid@users.noreply.github.com>

* Improve chrony reliability on Azure

add systemd override (FIFO scheduling, priority 50, OOMScoreAdjust -500) change
Hyper-V PTP refclock poll 3 -> -1 (sampling interval max from 8s to 0.5s; with
dpoll -2 dynamic range shifts from 2-8s to 0.125-0.5s) for tighter
offset/jitter. (#442)

Co-Authored-By: Sebastian Heid <8442432+s4heid@users.noreply.github.com>

* Fix path to chrony.service

This change sets the correct path when creating the chrony.service
directory. The wrong path was introduced in #442.

* Set linux-cloud-tools version to 6.8 in expectation

---------

Co-authored-by: Sebastian Max Dusch <sedusch@microsoft.com>
Co-authored-by: Adrian Joian <6505576+naioja@users.noreply.github.com>

Author: 3 people

bosh-stemcell/spec/assets/dpkg-list-ubuntu-azure-additions.txt

@@ -1,5 +1,9 @@
 azure-vm-utils
 cloud-init
+linux-cloud-tools-6.8
+linux-cloud-tools-6.8-generic
+linux-cloud-tools-common
+linux-cloud-tools-generic
 netplan.io
 python-is-python3
 python3-attr

bosh-stemcell/spec/stemcells/azure_spec.rb

@@ -40,4 +40,52 @@
       its(:content) { should include('"PartitionerType": "parted"') }
     end
   end
+
+  context 'cloud-init Azure APT mirror configuration' do
+    describe file('/etc/cloud/cloud.cfg.d/90-azure-apt-sources.cfg') do
+      it { should be_file }
+      its(:content) { should include('http://azure.archive.ubuntu.com/ubuntu/') }
+    end
+
+    describe file('/etc/cloud/cloud.cfg') do
+      it { should be_file }
+      its(:content) { should include('apt-configure') }
+    end
+  end
+
+  context 'installed by system_azure_init', {
+    exclude_on_alicloud: true,
+    exclude_on_aws: true,
+    exclude_on_google: true,
+    exclude_on_vcloud: true,
+    exclude_on_vsphere: true,
+    exclude_on_warden: true,
+    exclude_on_openstack: true,
+    exclude_on_softlayer: true,
+  } do
+    describe 'Hyper-V KVP daemon' do
+      describe command('which hv_kvp_daemon') do
+        its(:exit_status) { should eq 0 }
+      end
+
+      describe service('hv-kvp-daemon') do
+        it { should be_enabled }
+      end
+    end
+
+    describe 'WALinuxAgent configuration' do
+      describe file('/etc/waagent.conf') do
+        it { should be_owned_by('root') }
+      end
+
+      describe file('/lib/systemd/system/walinuxagent.service') do
+        it { should be_mode(0644) }
+        it { should be_owned_by('root') }
+      end
+
+      describe service('walinuxagent') do
+        it { should be_enabled }
+      end
+    end
+  end
 end

bosh-stemcell/spec/support/os_image_linux_kernel_modules_shared_examples.rb

@@ -78,4 +78,10 @@
       its(:content) { should match 'install rds /bin/true' }
     end
   end
+
+  context 'prevent floppy module from being loaded' do
+    describe file('/etc/modprobe.d/blacklist.conf') do
+      its(:content) { should match 'install floppy /bin/true' }
+    end
+  end
 end

stemcell_builder/stages/bosh_azure_chrony/apply.sh

@@ -6,8 +6,24 @@ base_dir=$(readlink -nf $(dirname $0)/../..)
 source $base_dir/lib/prelude_apply.bash
 source $base_dir/lib/prelude_bosh.bash
 
+mkdir -p $chroot/etc/systemd/system/chrony.service.d
+
+cat > $chroot/etc/systemd/system/chrony.service.d/chrony-systemd-override.conf <<EOF
+# created by $0
+[Service]
+# Set the CPU scheduling policy to FIFO (First-In, First-Out), a real-time policy.
+CPUSchedulingPolicy=fifo
+
+# Set the real-time priority to the highest possible value (99).
+# This ensures chronyd runs before any other non-kernel, non-real-time tasks.
+CPUSchedulingPriority=50
+
+# Make the process less likely to be killed by the OOM killer
+OOMScoreAdjust=-500
+EOF
+
 cat > $chroot/etc/chrony/conf.d/azure_ptp.conf <<EOF
 # created by $0
 # https://docs.microsoft.com/en-us/azure/virtual-machines/linux/time-sync#chrony
-refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0
+refclock PHC /dev/ptp_hyperv poll -1 dpoll -2 offset 0 stratum 2
 EOF

stemcell_builder/stages/system_azure_init/apply.sh

@@ -5,11 +5,12 @@ set -e
 base_dir=$(readlink -nf $(dirname $0)/../..)
 source $base_dir/lib/prelude_apply.bash
 
-packages="python3 python3-pyasn1 python3-setuptools python3-distro python-is-python3 cloud-init azure-vm-utils"
+packages="python3 python3-pyasn1 python3-setuptools python3-distro python-is-python3 \
+cloud-init azure-vm-utils linux-cloud-tools-common linux-cloud-tools-generic"
 pkg_mgr install $packages
 
-wala_release=2.9.1.1
-wala_expected_sha1=b61bd57f3b2f7b048d6bab2739690bbf1d9c213b
+wala_release=2.15.0.1
+wala_expected_sha1=155fd6f326a2bf2ff97b4ea2e2c83dc16a9c1768
 
 curl -L https://github.com/Azure/WALinuxAgent/archive/v${wala_release}.tar.gz > /tmp/wala.tar.gz
 sha1=$(cat /tmp/wala.tar.gz | openssl dgst -sha1  | awk 'BEGIN {FS="="}; {gsub(/ /,"",$2); print $2}')
@@ -32,6 +33,7 @@ run_in_chroot $chroot "
   sudo rm -fr WALinuxAgent-${wala_release}
   rm wala.tar.gz
 "
+mkdir -p $chroot/var/log/azure
 cp -f $dir/assets/etc/waagent/waagent.conf $chroot/etc/waagent.conf
 cp -f $dir/assets/etc/waagent/walinuxagent.service $chroot/lib/systemd/system/walinuxagent.service
 chmod 0644 $chroot/lib/systemd/system/walinuxagent.service
@@ -70,3 +72,6 @@ cat $chroot/etc/rsyslog.d/21-cloudinit.conf >> $chroot/etc/rsyslog.d/50-default.
 
 rm $chroot/etc/rsyslog.d/21-cloudinit.conf
 
+
+# Enable Hyper-V KVP daemon (installed via linux-cloud-tools)
+run_in_chroot "$chroot" "systemctl enable hv-kvp-daemon.service"

stemcell_builder/stages/system_azure_init/assets/etc/cloud-init/05-logging.cfg

@@ -1,7 +1,7 @@
-=## This yaml formated config file handles setting
+## This yaml formatted config file handles setting
 ## logger information.  The values that are necessary to be set
 ## are seen at the bottom.  The top '_log' are only used to remove
-## redundency in a syslog and fallback-to-file case.
+## redundancy in a syslog and fallback-to-file case.
 ##
 ## The 'log_cfgs' entry defines a list of logger configs
 ## Each entry in the list is tried, and the first one that
@@ -68,5 +68,4 @@ log_cfgs:
 # This tells cloud-init to redirect its stdout and stderr to
 # 'tee -a /var/log/cloud-init-output.log' so the user can see output
 # there without needing to look on the console.
-output: {all: '| tee -a /var/log/cloud-init-output.log'}
-
+output: {all: '| tee -a /var/log/cloud-init-output.log'}

stemcell_builder/stages/system_azure_init/assets/etc/cloud-init/cloud.cfg

@@ -10,6 +10,7 @@ cloud_init_modules:
  - update_etc_hosts
  - users-groups
  - ssh
+ - apt-configure
 cloud_config_modules:
  - ssh-import-id
  - set-passwords

stemcell_builder/stages/system_azure_init/assets/etc/waagent/walinuxagent.service

@@ -20,6 +20,7 @@ ExecStart=/usr/bin/python3 -u /usr/sbin/waagent -daemon
 Restart=always
 Slice=azure.slice
 CPUAccounting=yes
+MemoryAccounting=yes
 
 [Install]
 WantedBy=multi-user.target

stemcell_builder/stages/system_kernel_modules/apply.sh

@@ -18,7 +18,8 @@ install hfs /bin/true
 install hfsplus /bin/true
 install squashfs /bin/true
 install udf /bin/true
-install rds /bin/true' >> $chroot/etc/modprobe.d/blacklist.conf
+install rds /bin/true
+install floppy /bin/true' >> $chroot/etc/modprobe.d/blacklist.conf
 
 echo '# prevent nouveau from loading
 blacklist nouveau
@@ -28,3 +29,7 @@ alias nouveau off
 alias lbm-nouveau off' >> $chroot/etc/modprobe.d/blacklist-nouveau.conf
 
 rm -rf $chroot/lib/modules/*/kernel/zfs $chroot/usr/src/linux-headers-*/zfs
+
+mount --bind /sys "$chroot/sys"
+add_on_exit "umount $chroot/sys"
+run_in_chroot $chroot "update-initramfs -u -k all"