feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task by rchincha · Pull Request #8096 · Azure/AgentBaker

rchincha · 2026-03-16T05:14:51Z

What this PR does / why we need it

This PR implements RCV1P (Robust Certificate Validation for 1P) — the next-generation mechanism for distributing Azure root CA certificates to AKS nodes. Instead of hardcoding certificate bundles, RCV1P queries the Azure wireserver at provisioning time to download and install the latest root certificates into the OS trust store.

Reference: https://eng.ms/docs/products/onecert-certificates-key-vault-and-dsms/onecert-customer-guide/autorotationandecr/rcv1ptsg

Summary of Changes

1. Linux: Unified cert bootstrap flow (`init-aks-custom-cloud.sh`)

Consolidated 3 scripts into 1: Removed init-aks-custom-cloud-mariner.sh, init-aks-custom-cloud-operation-requests-mariner.sh, and init-aks-custom-cloud-operation-requests.sh. All cert logic now flows through a single init-aks-custom-cloud.sh that detects the distro (Ubuntu, Mariner, AzureLinux, Flatcar, ACL) at runtime.
Extracted repo depot logic: Moved Ubuntu repo depot initialization and chrony configuration into a new init-aks-custom-cloud-repos.sh to keep base customData size small for non-custom-cloud scenarios (critical for Flatcar/ACL which have tight size limits).
Two cert endpoint modes: legacy (ussec/usnat regions) and rcv1p (all other regions), selected by cloud location at runtime.
Fatal failure handling: Wireserver cert retrieval failures are fatal on both Linux and Windows — after exhausting retries, provisioning fails rather than continuing without certificates. The cert mode (legacy vs rcv1p) is always determined and must succeed.

2. Windows: CA cert refresh task and rcv1p support (`kubernetesfunc.ps1`)

New Get-CACertificates with -Location parameter: Determines cert endpoint mode from location, uses legacy endpoint for ussec/usnat, rcv1p for all others.
New Register-CACertificatesRefreshTask: Registers a daily scheduled task to refresh CA certificates, with backward compatibility for older VHDs that don't accept -Location.
New Should-InstallCACertificatesRefreshTask: Gates refresh task registration on wireserver opt-in status.
Pester tests: 260 lines of tests covering URI construction, endpoint mode selection, and refresh task gating.

3. E2E tests (`e2e/scenario_rcv1p_test.go`, `e2e/scenario_rcv1p_win_test.go`)

Linux tests: Ubuntu 2204, Ubuntu 2404, AzureLinux V3, Flatcar, ACL — each validates cert download, trust store installation, and refresh schedule.
Windows tests: Windows 2022, 23H2, 2025 — validates cert download to C:\ca, Windows certificate store import, and scheduled task registration.
Negative test: Test_RCV1P_NotOptedIn verifies that omitting the VM opt-in tag correctly prevents cert installation.
Dedicated pipeline: .pipelines/e2e-rcv1p.yaml runs daily at 3am PST with tag filter rcv1pcertmode=true (not yet enabled).

4. E2E infrastructure: multi-subscription and VM instance tagging

Multi-subscription support: RCV1P tests run in a dedicated subscription (RCV1P_SUBSCRIPTION_ID) with the Microsoft.Compute/PlatformSettingsOverride feature flag. Added SubscriptionID field to scenarios and GetAzure()/GetSubscriptionID() helpers.
VMSS opt-in tagging: Sets the wireserver opt-in tag (platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) on the VMSS at creation time via a VMConfigMutator. VMSS-level tags inherit to VM instances automatically.

⚠️ Critical Design Decisions

1. Cert endpoint mode is determined by cloud location, not a flag

Decision: ussec*/usnat* → legacy mode, everything else → rcv1p mode. This is determined at runtime from the node's Azure location.
Why: Avoids requiring a new API contract field. The location-based approach lets us roll out rcv1p incrementally — ussec/usnat stay on the legacy endpoint that works today, while all other regions use the new rcv1p endpoint with opt-in gating.

2. Two-layer access control for rcv1p

Decision: Both conditions must be met for cert installation:

Subscription feature flag (Microsoft.Compute/PlatformSettingsOverride) enables the wireserver endpoint
VM instance tag (platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) grants per-VM access

Why: Defense in depth — the subscription flag is a coarse gate, the VM tag provides per-node opt-in control. Without the tag, wireserver returns IsOptedInForRootCerts=false.

3. VM opt-in tag is set at VMSS creation time

Decision: The opt-in tag (platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) is set on the VMSS at creation time and inherits to all VM instances automatically.
Why: VMSS-level tags propagate to VM instances, and wireserver reads the tag from the VM instance to determine opt-in status. In E2E tests, the positive tests set the tag via a VMConfigMutator at VMSS creation, while the negative test (Test_RCV1P_NotOptedIn) simply omits the tag to verify wireserver returns IsOptedInForRootCerts=false.

4. `Get-CACertificates` moved outside `IsAKSCustomCloud` guard (Windows)

Decision: Get-CACertificates -Location $Location -FailOnError now runs for all clouds, not just custom clouds.
Why: RCV1P applies to all clouds. The function itself handles the location-based mode selection internally and gracefully skips cert installation when wireserver returns IsOptedInForRootCerts=false (which is the case on public cloud without the feature flag).

5. Wireserver failures are fatal after retries

Decision: If wireserver cert endpoints fail after exhausting retries, provisioning fails (exit 1 on Linux, throw on Windows with -FailOnError).
Why: Cert installation is required for the selected mode. Silently continuing without certificates would leave the node in an inconsistent state. Retries with backoff handle transient wireserver issues (rate limiting, temporary unavailability).

6. Backward compatibility for Windows VHD/CSE version skew

Decision: kuberneteswindowssetup.ps1 guards Register-CACertificatesRefreshTask with Get-Command checks before calling it.
Why: Windows VHD and CSE release independently. Newer CSE must not crash on older VHDs that don't have these functions. The guard falls back gracefully.

Testing Evidence

MSFT tenant (default E2E subscription)

Linux (Build 158446017):

All distros (Ubuntu, ACL, AzureLinux) correctly detect rcv1p mode
IsOptedInForRootCerts check works (skips on public cloud as expected)
Chrony configured per-OS, CSE completes successfully across 113 tests

Windows (Build 158446024):

Get-CACertificates -Location correctly selects rcv1p mode
Should-InstallCACertificatesRefreshTask returns $false on public cloud (correct)
Backward-compat guard works for older VHDs

TME tenant (RCV1P_SUBSCRIPTION_ID set in pipeline, with PlatformSettingsOverride feature flag)

Linux — Validated end-to-end: wireserver returns IsOptedInForRootCerts=true, certificates downloaded and installed into OS trust store, refresh schedule registered. Passed across Ubuntu 2204, Ubuntu 2404, AzureLinux V3, Flatcar, ACL.

Windows (Build 161633049):

5 of 6 jobs passed — all RCV1P tests passed across Windows 2022, 23H2, 2025 (both gen1 and gen2 variants)
Wireserver returns IsOptedInForRootCerts=true, certificates downloaded to C:\ca, scheduled task aks-ca-certs-refresh-task registered
The only failure (windows-2022-containerd job) was a pre-existing Test_Windows2022_VHDCaching issue unrelated to RCV1P
Bug fix validated: The original Windows implementation used the wrong JSON field name (OperationRequests instead of OperationsInfo) when parsing wireserver responses — this was the root cause of empty cert downloads. Fixed in commit b6cd4e4f68.

Note on Windows E2E infrastructure: The published CSE scripts package (v0.0.52 from packages.aks.azure.com) predates the RCV1P code. To test the branch code end-to-end, the tests build a CSE zip from staging/cse/windows/ at test time, upload it to blob storage, and override CseScriptsPackageURL in the bootstrap config. This override is temporary — once a new CSE package is published with RCV1P support, the override can be removed.

Files Changed (31 files, +1979 / -1218)

Area	Files	Description
Linux provisioning	`init-aks-custom-cloud.sh`, `init-aks-custom-cloud-repos.sh` (new), 3 removed	Unified cert flow, repo depot extraction
Windows provisioning	`kubernetesfunc.ps1`, `kuberneteswindowssetup.ps1`	rcv1p support, refresh task, backward compat
Windows tests	`kubernetesfunc.tests.ps1` (new)	260 lines of Pester tests
E2E tests	`scenario_rcv1p_test.go` (new), `scenario_rcv1p_win_test.go` (new)	Linux + Windows + negative tests
E2E infra	`vmss.go`, `types.go`, `validators.go`, `cluster.go`, `config/`	Multi-sub, VM instance tags, validators
Pipeline	`e2e-rcv1p.yaml` (new)	Daily RCV1P test pipeline
AgentBaker service	`baker.go`, `const.go`, `variables.go`	Wire up new scripts

PR File Breakdown: Functionality vs Tests

Functionality (1,859 lines — 51%)

Lines	File	Purpose
455	`parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh`	Linux RCV1P cert provisioning
358	`parts/linux/cloud-init/artifacts/init-aks-custom-cloud-repos.sh`	Split out repo init logic
346	`parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh`	Deleted (consolidated)
236	`parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh`	Deleted (consolidated)
211	`staging/cse/windows/kubernetesfunc.ps1`	Windows RCV1P cert functions
186	`parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh`	Deleted (consolidated)
18	`pkg/agent/variables.go`	Template variable plumbing
13	`parts/windows/kuberneteswindowssetup.ps1`	Windows CSE template
12	`pkg/agent/const.go`	Embedded script references
7	`parts/linux/cloud-init/nodecustomdata.yml`	Cloud-init custom data
7	`aks-node-controller/parser/helper.go`	ANC parser helper
4	`parts/linux/cloud-init/artifacts/cse_cmd.sh`	CSE command script
3	`aks-node-controller/parser/templates/cse_cmd.sh.gtpl`	CSE command template
3	`pkg/agent/baker.go`	Baker template plumbing

Tests / E2E Infra (1,795 lines — 49%)

Lines	File	Purpose
507	`e2e/scenario_rcv1p_test.go`	Linux RCV1P E2E tests + CSE zip helper
260	`staging/cse/windows/kubernetesfunc.tests.ps1`	Windows PowerShell unit tests
195	`e2e/scenario_rcv1p_win_test.go`	Windows RCV1P E2E tests
146	`e2e/validators.go`	RCV1P validation functions
135	`e2e/cluster.go`	Multi-subscription cluster support
111	`e2e/vmss.go`	VMSS tagging and log collection
96	`e2e/config/azure.go`	Azure client config for multi-sub
80	`e2e/test_helpers.go`	Test helper improvements
68	`e2e/types.go`	Scenario type extensions
61	`spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh`	ShellSpec unit tests
47	`e2e/cache.go`	Cluster cache additions
29	`e2e/aks_model.go`	AKS model extensions
24	`e2e/config/config.go`	E2E config additions
19	`.pipelines/e2e-rcv1p.yaml`	Dedicated RCV1P pipeline
14	`e2e/kube.go`	Kube helper improvements
2	`.pipelines/scripts/e2e_run.sh`	E2E run script changes
1	`.pipelines/templates/e2e-template.yaml`	E2E template changes

Summary

Category	Lines Changed	Percentage
Functionality	1,859	51%
Tests / E2E	1,795	49%
Total	3,654	100%

Copilot

Pull request overview

This PR aims to unify the custom-cloud CA certificate bootstrap path (removing the separate “operation-requests” init scripts) and adds a Windows scheduled task to periodically refresh custom-cloud CA certificates.

Changes:

Windows: add a scheduled task to refresh custom-cloud CA certificates; update Get-CACertificates to support legacy vs “rcv1p” modes keyed off location.
Linux: consolidate custom-cloud init to a single init script and update CSE command generation to set a cert-endpoint mode variable.
Regenerate multiple custom data / generated command snapshots to reflect the new templates.

Reviewed changes

Copilot reviewed 74 out of 176 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
staging/cse/windows/kubernetesfunc.ps1	Adds CA refresh scheduled task + updates CA retrieval logic and error behavior
parts/windows/kuberneteswindowssetup.ps1	Wires `Get-CACertificates -Location` and registers refresh task for custom clouds
pkg/agent/variables.go	Always injects `initAKSCustomCloud` payload into cloud-init data
pkg/agent/const.go	Removes separate custom-cloud init script constants; keeps single init script
pkg/agent/baker.go	Simplifies `GetTargetEnvironment`; notes `IsAKSCustomCloud` as deprecated
parts/linux/cloud-init/artifacts/cse_cmd.sh	Updates CSE command to set cert endpoint mode + run custom-cloud init script
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh	Deleted (custom-cloud init consolidation)
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh	Deleted (custom-cloud init consolidation)
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh	Deleted (custom-cloud init consolidation)
aks-node-controller/parser/templates/cse_cmd.sh.gtpl	Mirrors CSE command template updates for aks-node-controller parser
aks-node-controller/parser/testdata/Compatibility+EmptyConfig/generatedCSECommand	Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AzureLinuxv2+Kata+DisableUnattendedUpgrades=false/generatedCSECommand	Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+SSHStatusOn/generatedCSECommand	Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+EnablePubkeyAuth/generatedCSECommand	New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+DisablePubkeyAuth/generatedCSECommand	New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+DefaultPubkeyAuth/generatedCSECommand	New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+CustomOSConfig/generatedCSECommand	Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+CustomCloud/generatedCSECommand	Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+Containerd+MIG/generatedCSECommand	Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+CloudProviderOverrides/generatedCSECommand	New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+China/generatedCSECommand	Regenerated snapshot for new CSE cmd template
pkg/agent/testdata/MarinerV2+Kata/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+China/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOff/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/CustomizedImage/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/CustomizedImageKata/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData	Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/Flatcar/CustomData.inner	Regenerated snapshot (embedded gzip payload changed)
pkg/agent/testdata/ACL/CustomData.inner	Regenerated snapshot (embedded gzip payload changed)

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

This PR aims to unify AKS custom-cloud CA certificate bootstrap behavior (legacy vs “rcv1p/operation-requests” style flows) and adds a Windows scheduled task to periodically refresh custom-cloud CA certificates.

Changes:

Adds Windows CA refresh scheduled task registration and introduces location-based endpoint-mode selection (legacy vs rcv1p).
Refactors Windows CA certificate retrieval to support both endpoint modes and opt-in gating for rcv1p.
Simplifies Linux custom-cloud init script selection by consolidating onto init-aks-custom-cloud.sh and removing older variants; updates generated testdata accordingly.

Reviewed changes

Copilot reviewed 93 out of 99 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
staging/cse/windows/kubernetesfunc.ps1	Adds CA refresh scheduled task and endpoint-mode-aware `Get-CACertificates` implementation.
pkg/agent/variables.go	Simplifies how `initAKSCustomCloud` is added to Linux cloud-init variables.
pkg/agent/testdata/MarinerV2+Kata/CustomData	Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/Flatcar/CustomData.inner	Updates expected Flatcar CustomData snapshot (generated content changed).
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData	Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/CustomizedImageKata/CustomData	Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/CustomizedImage/CustomData	Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData	Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData	Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S119/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S118/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S117/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S116/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData	Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData	Updates expected Windows CustomData snapshot (new `Get-CACertificates` call form + refresh task).
pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData	Updates expected Windows CustomData snapshot (new `Get-CACertificates` call form + refresh task).
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData	Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData	Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData	Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData	Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData	Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData	Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+China/CustomData	Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/ACL/CustomData.inner	Updates expected ACL CustomData snapshot (generated content changed).
pkg/agent/const.go	Consolidates custom-cloud init script constants to a single script.
parts/windows/kuberneteswindowssetup.ps1	Updates Windows setup flow to call `Get-CACertificates` with location and registers CA refresh scheduled task.
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh	Removes operation-requests-specific Linux init script (consolidation).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh	Removes Mariner/AzureLinux operation-requests init script (consolidation).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh	Removes Mariner/AzureLinux legacy init script variant (consolidation).
aks-node-controller/parser/templates/cse_cmd.sh.gtpl	Adds a `LOCATION` shell variable in the generated CSE command template.
aks-node-controller/parser/helper.go	Factors out a shared `getCloudLocation` helper and reuses it in `getCloudTargetEnv`.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

This PR unifies the AKS custom cloud CA certificate bootstrap logic to a single flow and adds a Windows scheduled task to periodically refresh custom cloud CA certificates. It also updates Linux/customdata generation and test snapshots to reflect the new wiring.

Changes:

Add Windows scheduled task registration for daily CA certificate refresh and introduce a location-based cert endpoint mode selector.
Simplify Linux custom cloud init script selection by standardizing on init-aks-custom-cloud.sh, plus add wiring/tests for refresh-mode arguments.
Update aks-node-controller template to export LOCATION, and regenerate CustomData snapshot test artifacts.

Reviewed changes

Copilot reviewed 95 out of 101 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
staging/cse/windows/kubernetesfunc.tests.ps1	Adds Pester coverage for cert endpoint mode selection, scheduled task registration, and CA retrieval behavior.
staging/cse/windows/kubernetesfunc.ps1	Implements unified Windows CA retrieval logic with legacy/rcv1p modes and registers a daily refresh scheduled task.
spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh	Adds ShellSpec assertions to validate refresh-mode argument parsing/wiring in the Linux init script.
pkg/agent/variables.go	Changes how `initAKSCustomCloud` is injected into Linux cloud-init data.
pkg/agent/const.go	Removes per-cloud custom init script constants and standardizes on `init-aks-custom-cloud.sh`.
parts/windows/kuberneteswindowssetup.ps1	Wires CA retrieval call and registers the Windows CA refresh scheduled task during BasePrep.
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh	Removed (operation-requests variant no longer used).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh	Removed (operation-requests Mariner/AzureLinux variant no longer used).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh	Removed (Mariner/AzureLinux legacy variant no longer used).
aks-node-controller/parser/templates/cse_cmd.sh.gtpl	Exports `LOCATION` into the CSE environment for downstream scripts.
aks-node-controller/parser/helper.go	Adds a helper to normalize location and reuses it in cloud target env detection.
pkg/agent/testdata/MarinerV2+Kata/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/Flatcar/CustomData.inner	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/CustomizedImageKata/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/CustomizedImage/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S119/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S118/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S117/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S116/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData	Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+China/CustomData	Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/ACL/CustomData.inner	Regenerated CustomData snapshot due to init/custom cloud wiring changes.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

This PR updates AKS custom cloud certificate bootstrapping to use a single unified flow and adds a Windows scheduled task for periodic custom cloud CA refresh.

Changes:

Added Windows CA refresh task registration plus new logic to select cert retrieval mode and opt-in gating.
Simplified Linux custom cloud init script wiring by removing legacy “operation-requests” variants and normalizing location for refresh mode.
Added/updated tests and refreshed golden testdata outputs to reflect new custom data content.

Reviewed changes

Copilot reviewed 95 out of 101 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
staging/cse/windows/kubernetesfunc.tests.ps1	Adds Pester coverage for endpoint-mode selection, task registration behavior, and CA retrieval failure handling.
staging/cse/windows/kubernetesfunc.ps1	Implements endpoint-mode derivation, opt-in gating, CA retrieval paths, and a Windows scheduled task for refresh.
spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh	Adds ShellSpec checks to ensure init script wiring for ca-refresh mode and LOCATION usage.
pkg/agent/variables.go	Simplifies init script selection and updates how custom cloud init script is injected into cloud-init data.
pkg/agent/const.go	Removes now-unused custom-cloud init script constants; keeps unified init script constant.
parts/windows/kuberneteswindowssetup.ps1	Updates Windows setup to call Get-CACertificates with Location and conditionally register refresh task.
aks-node-controller/parser/templates/cse_cmd.sh.gtpl	Adds LOCATION variable for downstream scripts during custom cloud provisioning.
aks-node-controller/parser/helper.go	Adds getCloudLocation helper and reuses it for cloud target env detection.
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh	Removes legacy operation-requests init script (superseded by unified script).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh	Removes legacy Mariner operation-requests init script (superseded by unified script).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh	Removes legacy Mariner init script variant (superseded by unified script).
pkg/agent/testdata/MarinerV2+Kata/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/Flatcar/CustomData.inner	Updates golden ignition/customData payload for unified custom cloud init content.
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/CustomizedImageKata/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/CustomizedImage/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S119/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S118/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S117/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S116/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData	Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOff/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+China/CustomData	Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/ACL/CustomData.inner	Updates golden ignition/customData payload for unified custom cloud init content.

Comments suppressed due to low confidence (7)

staging/cse/windows/kubernetesfunc.ps1:1

Get-CACertificates used to fail fast via Set-ExitCode on retrieval/parse errors, but now returns $false (and logs warnings) for a wide range of failure cases. Because call sites in the generated setup scripts invoke Get-CACertificates -Location $Location without checking the return value, this can silently proceed without required CA material and lead to harder-to-diagnose TLS failures later in provisioning. Consider restoring fatal behavior for “expected-to-install” scenarios (e.g., legacy mode, or rcv1p when opted-in), or have callers check the return value and invoke Set-ExitCode when it’s $false in those modes.
staging/cse/windows/kubernetesfunc.ps1:1
Get-CACertificates used to fail fast via Set-ExitCode on retrieval/parse errors, but now returns $false (and logs warnings) for a wide range of failure cases. Because call sites in the generated setup scripts invoke Get-CACertificates -Location $Location without checking the return value, this can silently proceed without required CA material and lead to harder-to-diagnose TLS failures later in provisioning. Consider restoring fatal behavior for “expected-to-install” scenarios (e.g., legacy mode, or rcv1p when opted-in), or have callers check the return value and invoke Set-ExitCode when it’s $false in those modes.
pkg/agent/variables.go:1
This change removes the previous cs.IsAKSCustomCloud() guard and injects the custom cloud init script into cloudInitData unconditionally. That can increase customData size for all clusters (risking platform limits) and may introduce unintended side effects if any downstream template writes/executes this script outside custom cloud. Recommend reinstating the custom cloud guard (and only setting initAKSCustomCloud when IsAKSCustomCloud() is true), while still using the unified initAKSCustomCloudScript for all custom clouds.
staging/cse/windows/kubernetesfunc.ps1:1
$resourceFileName is used directly to build a path under C:\ca. If the upstream response ever contains path separators (e.g., ..\foo or nested paths), this can write outside the intended directory. Prefer sanitizing to a basename (e.g., using Split-Path -Leaf or [IO.Path]::GetFileName($resourceFileName)) before Join-Path, and consider rejecting names containing directory traversal characters.
staging/cse/windows/kubernetesfunc.ps1:1
$resourceFileName is used directly to build a path under C:\ca. If the upstream response ever contains path separators (e.g., ..\foo or nested paths), this can write outside the intended directory. Prefer sanitizing to a basename (e.g., using Split-Path -Leaf or [IO.Path]::GetFileName($resourceFileName)) before Join-Path, and consider rejecting names containing directory traversal characters.
staging/cse/windows/kubernetesfunc.ps1:1
The new rcv1p operation-requests flow is non-trivial (multiple requests, JSON shape assumptions, per-item content downloads, and $downloadedAny aggregation), but the added Pester tests only cover legacy mode and the “throws returns false” path. Add tests that (1) exercise the rcv1p path end-to-end with mocked Retry-Command returning operation requests and cert bodies, and (2) verify behavior when operation requests are empty/invalid (ensuring the function returns $false and logs expected warnings).
pkg/agent/variables.go:1
The PR description still contains placeholder text (Fixes # with no linked issue and no explanation of “what/why”). Please update the PR description to summarize the behavior change (unified bootstrap + Windows refresh task) and link the relevant issue or remove the placeholder.

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.

Copilot · 2026-04-25T22:29:25Z

+    if echo "$opt_in_response" | jq -e '.IsOptedInForRootCerts == true' > /dev/null 2>&1; then
+        echo "IsOptedInForRootCerts=true"
+        return 0
    fi


init-aks-custom-cloud.sh uses jq to parse the IsOptedInForRootCerts response, but this script is invoked very early from cse_cmd.sh (before the usual dependency installation). On images where jq isn’t present yet, jq will exit 127 and the script will treat this as “not opted in”, silently skipping cert install and refresh scheduling. Consider either (1) avoiding jq here (parse the boolean with shell tools), or (2) ensuring jq is available before this script runs (e.g., source cse_helpers.sh and call installJq, or vendor a minimal JSON parser).

Suggested change

if echo "$opt_in_response" | jq -e '.IsOptedInForRootCerts == true' > /dev/null 2>&1; then

echo "IsOptedInForRootCerts=true"

return 0

fi

case "$opt_in_response" in

*\"IsOptedInForRootCerts\"*[[:space:]]*:[[:space:]]*true*)

echo "IsOptedInForRootCerts=true"

return 0

;;

esac

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.

Copilot · 2026-04-27T00:21:50Z

+        if ! crontab -l 2>/dev/null | grep -q "\"$scriptPath\" ca-refresh"; then
+            # Quote the script path in the cron entry to avoid issues with spaces or special characters.
+            if ! (crontab -l 2>/dev/null ; printf '%s\n' "0 19 * * * \"$scriptPath\" ca-refresh \"$LOCATION\"") | crontab -; then
+                echo "Failed to install ca-refresh cron job via crontab" >&2
+            fi


The cron installer only adds an entry if it can’t find an existing "$scriptPath" ca-refresh line. That grep will also match older cron entries created by previous versions (which invoked ca-refresh without the location argument), so upgraded nodes may keep the old entry and never get updated to pass the location. In that case, ca-refresh runs with an empty location (cron has no LOCATION env var) and will default endpoint mode to rcv1p, which is wrong for ussec/usnat and can change refresh behavior. Consider rewriting/removing any existing ca-refresh entry for this script and re-adding it with the explicit location argument.

Suggested change

if ! crontab -l 2>/dev/null | grep -q "\"$scriptPath\" ca-refresh"; then

# Quote the script path in the cron entry to avoid issues with spaces or special characters.

if ! (crontab -l 2>/dev/null ; printf '%s\n' "0 19 * * * \"$scriptPath\" ca-refresh \"$LOCATION\"") | crontab -; then

echo "Failed to install ca-refresh cron job via crontab" >&2

fi

current_crontab="$(crontab -l 2>/dev/null || true)"

# Remove any existing ca-refresh entry for this script so upgraded nodes

# are updated to the canonical form that passes the location explicitly.

filtered_crontab="$(printf '%s\n' "$current_crontab" | awk -v script="$scriptPath" 'index($0, "\"" script "\" ca-refresh") == 0 { print }')"

# Quote the script path in the cron entry to avoid issues with spaces or special characters.

if ! { printf '%s\n' "$filtered_crontab"; printf '%s\n' "0 19 * * * \"$scriptPath\" ca-refresh \"$LOCATION\""; } | crontab -; then

echo "Failed to install ca-refresh cron job via crontab" >&2

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated no new comments.

Co-authored-by: Jane Jung <janejung@microsoft.com> Co-authored-by: janenotjung-hue <107402425+janenotjung-hue@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: aks-node-assistant[bot] <190555641+aks-node-assistant[bot]@users.noreply.github.com>

https://eng.ms/docs/products/onecert-certificates-key-vault-and-dsms/onecert-customer-guide/autorotationandecr/overviewrcv https://eng.ms/docs/products/onecert-certificates-key-vault-and-dsms/onecert-customer-guide/autorotationandecr/rcv1ptsg cse_cmd.sh.gtpl: derive cert endpoint mode from target cloud and always run custom-cloud init script. cse_cmd.sh: same mode logic as template; remove LOCATION export. init-aks-custom-cloud.sh: merged legacy + operation-requests logic into one script with distro-aware cert install paths. parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh: removed (merged into unified script). parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh: removed (merged into unified script). parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh: removed (merged into unified script). const.go: keep only unified custom-cloud init script constant. variables.go: simplify script selection to always use unified init script. kubernetesfunc.ps1: add location-aware CA retrieval (legacy/rcv1p) and scheduled refresh task registration helper. kuberneteswindowssetup.ps1: pass location to CA retrieval and register refresh task for custom cloud.

…n location

…cloud spec

…gistration

… for legacy and opted-in rcv1p modes

…E packages

… schedule installation

…istency

…ndpoint mode handling for legacy and rcv1p regions

…et-CACertificates to verify URI handling

…ing of custom cloud file paths

…h optional Location parameter

…nit-aks-custom-cloud.sh

The unified init-aks-custom-cloud.sh script (~22KB) pushed Flatcar and ACL VM customData over Azure's 87,380 character limit, causing 16 E2E failures. Split the script into two files: - init-aks-custom-cloud.sh: cert refresh + scheduling (included for all clouds) - init-aks-custom-cloud-repos.sh: repo depot + chrony (custom cloud only) The main script sources the repos script at runtime if present. For non-custom-cloud VMs, only the smaller main script is embedded, reducing base64(gzip) size from 8,736 to 4,424 chars (-4,312 chars). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add e2e test infrastructure and scenarios to validate RCV1P (Root Certificate V1P) certificate endpoint mode across all supported Linux distros and Windows versions. Infrastructure changes: - Introduce ClusterInfra struct to decouple cluster lifecycle functions from the default Azure subscription, enabling per-subscription clients - Refactor ~20 functions in cluster.go, kube.go, and aks_model.go to accept ClusterInfra instead of hardcoding config.Azure - Add NewAzureClientForSubscription() to construct ARM clients for any subscription, replacing the single-subscription NewAzureClient() - Add CreateVMManagedIdentityInRG() for identity-only creation without blob storage (RCV1P subscription doesn't need shared storage) - Add ClusterRCV1PKubenet cached cluster function and RCV1P-specific resource group/identity cache entries Config and pipeline: - Add RCV1P_SUBSCRIPTION_ID env var to config, with lazy-init of RCV1PAzure client and helper functions - Pass RCV1P_SUBSCRIPTION_ID through e2e-template.yaml and e2e_run.sh - Add dedicated e2e-rcv1p.yaml pipeline with daily schedule Test scenarios: - Linux: Ubuntu 22.04, Ubuntu 24.04, AzureLinux V3, Flatcar, ACL - Windows: Server 2022, 23H2, 2025 - All tests skip gracefully when RCV1P_SUBSCRIPTION_ID is unset Validators: - ValidateRCV1PCertMode (Linux): checks provisioning log for rcv1p mode, verifies certs in /root/AzureCACertificates, validates distro-specific trust store updates, confirms cron/systemd refresh schedule - ValidateRCV1PCertModeWindows: checks C:\AzureCACertificates directory and scheduled refresh task Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…rror handling - e2e/cluster.go: Pass ClusterInfra to ensureMaintenanceConfiguration and createNewMaintenanceConfiguration so RCV1P clusters use the correct subscription and resource group instead of the global default. - e2e/validators.go: Fix ValidateEmptyDirectory shell predicate to succeed when the directory is missing (not just when empty), matching the error message. Also quote dirName in the shell command. - staging/cse/windows/kubernetesfunc.ps1: Add -FailOnError switch to Get-CACertificates so initial provisioning fails fast on cert retrieval errors while the scheduled refresh task remains non-fatal. - parts/windows/kuberneteswindowssetup.ps1: Call Get-CACertificates with -FailOnError during initial provisioning. - staging/cse/windows/kubernetesfunc.tests.ps1: Add tests for -FailOnError behavior (exception and empty data paths). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add Test_RCV1P_Windows_NotOptedIn which creates a Windows VM in the RCV1P subscription without the opt-in tag and validates that: - C:\ca is empty or does not exist (no certificates downloaded) - aks-ca-certs-refresh-task scheduled task is not registered This mirrors the existing Linux Test_RCV1P_NotOptedIn test to ensure the two-layer access control (subscription feature + VM tag) works on Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Wireserver checks tags on the individual VMSS VM instance, not the VMSS resource-level tags. Add VMInstanceTags field to Config and update the VM instance after it appears in the API but before CSE completes. This ensures wireserver sees the opt-in tag when init-aks-custom-cloud.sh queries IsOptedInForRootCerts during provisioning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The previous approach of updating VM instance tags after creation had a race condition: the BeginUpdate took ~108s, but CSE ran init-aks-custom-cloud.sh and queried wireserver before the tag update completed. Now we marshal the VMSS model to JSON, inject tags into virtualMachineProfile, and send a raw ARM PUT request via the SDK pipeline. This ensures the tags are present at VMSS creation time and propagate to VM instances before CSE boots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tion The ARM API does not support virtualMachineProfile.tags for Uniform mode VMSS (400 BadRequest). Instead, use a lightweight PATCH request to update tags on the VM instance after it appears. PATCH only modifies the tags property and should complete in seconds, unlike BeginUpdate which triggers a full model update (~108s). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ON injection" This reverts commit 03efe78.

For Uniform mode VMSS, VM instance tags cannot be set at creation time: - The Go SDK (armcompute v7.x) does not expose a Tags field on VirtualMachineScaleSetVMProfile. - The ARM API rejects virtualMachineProfile.tags for Uniform mode VMSS with: 'Could not find member tags on object of type VirtualMachineProfile'. - PATCH on the Compute VM instance endpoint returns 405 Method Not Allowed. - BeginUpdate (PUT) works but takes ~108s for a full VM model reconciliation, causing a race condition: CSE runs init-aks-custom-cloud.sh and queries wireserver before the tag update completes. Use the Microsoft.Resources/tags API instead, which provides a lightweight PATCH endpoint (/{resourceId}/providers/Microsoft.Resources/tags/default) that updates only tags without triggering a full VM update. The Merge operation adds tags without replacing existing ones. Also moves s.T.Cleanup() registration to immediately after waitForVMSSVM() so the VMSS is always cleaned up even if tag patching or subsequent steps fail, preventing orphaned VMSS resources. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace the Microsoft.Resources/tags API approach (which returns 405 on Uniform VMSS VM instances) with BeginUpdate (full PUT) + deferred CSE. For scenarios requiring VM instance tags (e.g., RCV1P): 1. Create VMSS without CSE extension profile 2. Wait for VMSS creation to complete 3. Apply tags via VMSSVM.BeginUpdate (~108s full PUT) 4. Re-add CSE extension via a second BeginCreateOrUpdate This ensures wireserver sees the per-VM-instance tags before CSE queries it. The delay is acceptable for E2E validation; production would use a different approach (e.g., AKS RP sets tags pre-boot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Verify Microsoft.Compute/PlatformSettingsOverride is registered on the RCV1P subscription before running tests. This fails fast with a clear error if the feature flag is missing, rather than letting tests run and fail with opaque wireserver responses. The check runs once per test run via sync.Once. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Experimental commit to validate whether wireserver detects VM instance tags applied via BeginUpdate after VM creation. Polls for up to ~5 minutes (30x10s). Wireserver reads IsOptedInForRootCerts from the Fabric Controller goal state (CCF/ContainerConfig), NOT directly from ARM tags. The flow is: BeginUpdate -> ARM model update -> FC generates new CCF with platformsettings.host_environment.service.platform_optedin_for_rootcerts -> FC pushes CCF to host agent -> wireserver reflects new state. FC goal state propagation can take several minutes, so the polling window is set to ~5 minutes to give adequate time for detection. Logs the full wireserver response on each attempt for diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 16, 2026 05:14

rchincha temporarily deployed to test March 16, 2026 05:14 — with GitHub Actions Inactive

Copilot started reviewing on behalf of rchincha March 16, 2026 05:15 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

Comment thread staging/cse/windows/kubernetesfunc.ps1

Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated

Comment thread parts/linux/cloud-init/artifacts/cse_cmd.sh Outdated

Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl Outdated

rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from 44ff9ee to a0a1307 Compare March 18, 2026 21:11

rchincha temporarily deployed to test March 18, 2026 21:11 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 18, 2026 22:12

rchincha temporarily deployed to test March 18, 2026 22:12 — with GitHub Actions Inactive

Copilot AI reviewed Mar 18, 2026

View reviewed changes

Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated

Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated

Comment thread staging/cse/windows/kubernetesfunc.ps1

Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl Outdated

rchincha mentioned this pull request Mar 19, 2026

feat: gate custom cloud cert pull on RCV 1P opt-in #7958

Closed

rchincha changed the title ~~feat(custom-cloud): unify cert bootstrap flow and add Windows CA refresh task~~ feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task Mar 19, 2026

rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from 2b3c1d6 to e19a19b Compare March 19, 2026 00:51

rchincha temporarily deployed to test March 19, 2026 00:51 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 19, 2026 01:00

rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from e19a19b to d41856f Compare March 19, 2026 01:00

rchincha temporarily deployed to test March 19, 2026 01:00 — with GitHub Actions Inactive

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated

Comment thread pkg/agent/variables.go

rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from d41856f to 18ba549 Compare March 19, 2026 20:13

rchincha temporarily deployed to test March 19, 2026 20:13 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 19, 2026 22:07

rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from 18ba549 to e94c465 Compare March 19, 2026 22:07

rchincha temporarily deployed to test March 19, 2026 22:07 — with GitHub Actions Inactive

Copilot started reviewing on behalf of rchincha March 19, 2026 22:14 View session

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl

rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from e94c465 to f20d5b8 Compare March 19, 2026 23:28

rchincha temporarily deployed to test March 19, 2026 23:28 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 20, 2026 06:42

rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from f20d5b8 to b53f240 Compare March 20, 2026 06:42

rchincha temporarily deployed to test March 20, 2026 06:42 — with GitHub Actions Inactive

Copilot started reviewing on behalf of rchincha March 20, 2026 06:42 View session

Copilot AI reviewed Apr 25, 2026

View reviewed changes

Copilot AI reviewed Apr 27, 2026

View reviewed changes

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Comment thread e2e/types.go

Comment thread parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Outdated

Copilot AI reviewed Apr 29, 2026

View reviewed changes

aks-node-assistant Bot and others added 26 commits May 1, 2026 19:54

feat: enhance CA certificates refresh task with endpoint mode based o…

a183741

…n location

feat: add tests for certificate endpoint mode handling in AKS custom …

d8bfb24

…cloud spec

feat: simplify certificate endpoint mode handling and refresh task re…

f32dc9f

…gistration

feat: implement conditional CA certificates refresh task registration…

b0c8ecf

… for legacy and opted-in rcv1p modes

feat: enhance CA certificates refresh task registration for legacy CS…

176604b

…E packages

feat: update tests for certificate endpoint mode handling and refresh…

b83899d

… schedule installation

feat: refactor test setup functions for improved readability and cons…

650fedc

…istency

feat: update Get-CustomCloudCertEndpointModeFromLocation to clarify e…

1e3d32e

…ndpoint mode handling for legacy and rcv1p regions

feat: enhance tests for Should-InstallCACertificatesRefreshTask and G…

acb9156

…et-CACertificates to verify URI handling

feat: update cse_cmd.sh and cse_cmd.sh.gtpl to ensure consistent logg…

c496a58

…ing of custom cloud file paths

feat: update CA certificates functions for backward compatibility wit…

1071513

…h optional Location parameter

feat: remove deprecated Ubuntu repository initialization logic from i…

af55d5c

…nit-aks-custom-cloud.sh

Revert "e2e: use lightweight PATCH for VM instance tags instead of JS…

4a5f81d

…ON injection" This reverts commit 03efe78.

Conversation

rchincha commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it

Summary of Changes

1. Linux: Unified cert bootstrap flow (init-aks-custom-cloud.sh)

2. Windows: CA cert refresh task and rcv1p support (kubernetesfunc.ps1)

3. E2E tests (e2e/scenario_rcv1p_test.go, e2e/scenario_rcv1p_win_test.go)

4. E2E infrastructure: multi-subscription and VM instance tagging

⚠️ Critical Design Decisions

1. Cert endpoint mode is determined by cloud location, not a flag

2. Two-layer access control for rcv1p

3. VM opt-in tag is set at VMSS creation time

4. Get-CACertificates moved outside IsAKSCustomCloud guard (Windows)

5. Wireserver failures are fatal after retries

6. Backward compatibility for Windows VHD/CSE version skew

Testing Evidence

MSFT tenant (default E2E subscription)

TME tenant (RCV1P_SUBSCRIPTION_ID set in pipeline, with PlatformSettingsOverride feature flag)

Files Changed (31 files, +1979 / -1218)

PR File Breakdown: Functionality vs Tests

Functionality (1,859 lines — 51%)

Tests / E2E Infra (1,795 lines — 49%)

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

rchincha commented Mar 16, 2026 •

edited

Loading

1. Linux: Unified cert bootstrap flow (`init-aks-custom-cloud.sh`)

2. Windows: CA cert refresh task and rcv1p support (`kubernetesfunc.ps1`)

3. E2E tests (`e2e/scenario_rcv1p_test.go`, `e2e/scenario_rcv1p_win_test.go`)

4. `Get-CACertificates` moved outside `IsAKSCustomCloud` guard (Windows)