What you will find here: A one-row-per-pipeline reference index for every bundled
Step.*.ymlworkflow (GitHub Actions + Azure DevOps twins), with a one-line summary of the job, its triggers, the cmdlets it invokes, what it depends on, and the artefacts it produces. Use this as the at-a-glance reference card after you have read the per-step runbook in the main pipeline README.md.
Each pipeline section heading below uses Step N - <description> to mirror its bundled YAML filename Step.N_<description>.yml exactly. The Pipeline column header in the at-a-glance table below is the column label; the Step N - form is the operator-facing identifier used everywhere else - workflow name: fields, ADO pipeline names, the parent README's section 1.1 table, and these section headings. The execution order in the end-to-end runbook is operational and not strictly numerical - read it for the recommended sequence.
The table below is the ground truth for what each shipped YAML does out of the box. Four of the ten pipelines (fleet-connectivity-status, fleet-update-status, fleet-health-status, apply-updates-schedule-audit) and one optional (inventory-clusters weekly) are pre-wired with schedule: (GH) / schedules: (ADO) blocks. The remaining five (authentication-test, manage-updatering-tags, assess-update-readiness, apply-updates, monitor-updates) are manual-only by design (monitor-updates ships a commented-out */30 * * * * cron sample for in-wave use).
| Pipeline | GitHub Actions trigger | Azure DevOps trigger | Notes |
|---|---|---|---|
Step 0 - authentication-test |
workflow_dispatch only |
trigger: none (manual only) |
Run on initial wiring + after every RBAC / federated-credential / subscription change. Recommended monthly cadence; add your own schedule: block if you want it automated. |
Step 1 - inventory-clusters |
workflow_dispatch + schedule: cron '0 6 * * 1' (Mondays 06:00 UTC) |
trigger: none + schedules: cron '0 6 * * 1' |
Weekly drift detection. Edit the cron to change cadence. |
Step 2 - manage-updatering-tags |
workflow_dispatch only |
trigger: none (manual only) |
Runs on-demand whenever you edit the CSV. |
Step 3 - apply-updates-schedule-audit |
workflow_dispatch + schedule: cron '0 5 * * 1' (Mondays 05:00 UTC) |
trigger: none + schedules: cron '0 5 * * 1' |
Weekly read-only drift advisor: compares apply-updates cron(s) to UpdateStartWindow tags. Runs before the daily fleet pipelines so its annotations are first on Monday mornings. |
Step 4 - fleet-connectivity-status |
workflow_dispatch + schedule: cron '30 5 * * *' (daily 05:30 UTC) |
trigger: none + schedules: cron '30 5 * * *' |
Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot with bidirectional node-coverage reconciliation. Runs 30 min before fleet-update-status so connectivity issues are visible upstream of update reporting. |
Step 5 - assess-update-readiness |
workflow_dispatch only |
trigger: none (manual only) |
Recommended customisation - schedule per ring 12-72h before the matching Step 7 UpdateStartWindow (see Step 5 - Assess Update Readiness below for the per-ring pattern + lead-time table). Always-green at pipeline level; per-cluster readiness gaps surface as JUnit <failure> entries. |
Step 7 - apply-updates |
workflow_dispatch only |
trigger: none (manual only) |
No schedule shipped - see the warning in Step 7 - Apply Updates below. The cluster UpdateStartWindow / UpdateExclusionsWindow tags only gate updates while the pipeline is running; they do not start the pipeline. |
Step 8 - monitor-updates |
workflow_dispatch only (commented-out schedule: cron '*/30 * * * *' sample shipped) |
trigger: none only (commented-out schedules: cron '*/30 * * * *' sample shipped) |
In-flight update progress / long-running-run flag. Disabled by default to avoid 48 idle runs/day between waves. Uncomment the cron when a wave is active. |
Step 9 - fleet-update-status |
workflow_dispatch + schedule: cron '0 6 * * *' (daily 06:00 UTC) |
trigger: none + schedules: cron '0 6 * * *' |
Daily fleet update snapshot. |
Step 10 - fleet-health-status |
workflow_dispatch + schedule: cron '0 7 * * *' (daily 07:00 UTC) |
trigger: none + schedules: cron '0 7 * * *' |
Daily 24-hour health-check snapshot. Offset by one hour from fleet-update-status to avoid contention. |
All times are UTC. GitHub Actions and Azure DevOps schedules both run on UTC; convert from your local timezone when picking cron values. Both platforms can delay scheduled runs by several minutes during high-load periods - do not rely on second-precision alignment.
GitHub Actions only: scheduled workflows are automatically disabled after 60 days of repository inactivity (no commits, PRs, or issue activity). Re-enable them via the Actions UI or run any push. Azure DevOps schedules do not have this auto-disable behaviour.
Common conventions for every row below:
Cmdlets invokedlists only AzLocal.UpdateManagement cmdlets the YAML calls directly (helper cmdlets they themselves call transitively are documented in each cmdlet's section of the module-levelREADME.md).Depends onlists upstream pipelines whose output is consumed or whose work must be in place before this pipeline runs -Nonemeans the pipeline is self-contained.Exit conditionsdescribes what a non-success YAML run state means in practice; per-cluster issues are always surfaced as JUnit<failure>entries rather than failing the whole pipeline unless noted otherwise.ITSMindicates whether the bundled YAMLs ship with the opt-in ServiceNow auto-raise connector (Get-AzLocalItsmConfig+ per-failure ticket creation gated on theraise_itsm_ticket/raiseItsmTicketinput); only Step 4, Step 7, Step 9, and Step 10 ship it - the matrix config lives at./.itsm/azurelocal-itsm.yml(seeAutomation-Pipeline-Examples/ITSM/for the full recipe).
| Aspect | Value |
|---|---|
| Purpose | End-to-end probe of the federated identity, the Azure RBAC role assignment, and the subscriptions the pipeline identity can read. Emits a JUnit-rendered Authentication / Subscription Scope / Resource Graph Reachability report and the full subscription set (subscriptions.csv + subscriptions.json) so you can detect silent scope drift (a new tenant SP suddenly seeing more or fewer subscriptions than yesterday) before downstream fleet reports under- or over-count clusters. |
| Inputs | environment (optional - GitHub Actions only; leave blank to test the branch-scoped federated credential), module_version (optional). |
| Trigger | Manual only (workflow_dispatch / Run pipeline button). No schedule shipped - run it on initial wiring, after every RBAC / federated-credential / subscription change, and on a recommended monthly cadence. Add your own schedule: / schedules: block if you want it automated. |
| Cmdlets invoked | None (uses the az CLI directly for the auth / Resource Graph probes). |
| Depends on | None. This is the first pipeline that must run on a freshly-wired identity. |
| Artefacts | auth-report.xml (JUnit, one <testcase> per probe), subscriptions.json, subscriptions.csv, markdown step / run summary with the subscription detail table. |
| When to run | First, before importing any of the other nine pipelines. Re-run after every RBAC change, federated-credential change, service-connection change, or subscription move. Re-run monthly as a baseline scope-drift check. |
| RBAC | Whatever the pipeline identity has - the probe itself is read-only and intentionally surfaces both over- and under-grants. |
| Exit conditions | Pipeline run is green only when every probe passes. A red run means the identity, its role assignment, or its subscription scope is wrong - downstream pipelines will give nonsense results until this passes. |
| ITSM | Not supported - this is an identity-only probe with no per-cluster failure surface to dispatch on. |
| Introduced | v0.7.70. |
| Aspect | Value |
|---|---|
| Purpose | Enumerate every Azure Local cluster the identity can see and export to CSV. Establishes the baseline cluster list and current tag values for every downstream pipeline. |
| Inputs | None. |
| Trigger | Manual (workflow_dispatch / Run pipeline button) plus weekly scheduled run on Mondays at 06:00 UTC (cron '0 6 * * 1'). Edit the cron in the YAML (inside the # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers block) to change the day / time. |
| Cmdlets invoked | Get-AzLocalClusterInventory. |
| Depends on | Step 0 should be green at least once before relying on this output (otherwise the cluster list may be silently scoped to the wrong subscription set). |
| Artefacts | cluster-inventory.csv (one row per cluster, includes current UpdateRing / UpdateStartWindow / UpdateExclusionsWindow / UpdateExcluded and sideloaded-workflow tags). |
| When to run | First run of a new estate; periodically (default weekly) to detect new clusters or tag drift. The CSV is the input artefact for Step 2 (UpdateRing tag bulk-apply). |
| RBAC | Read-only. Covered by the Azure Stack HCI Update Operator (custom) custom role (Microsoft.AzureStackHCI/clusters/read, Microsoft.ResourceGraph/resources/read, Microsoft.Resources/tags/read). |
| Exit conditions | Pipeline run is green when the inventory query succeeds and at least one cluster is returned. An empty inventory on a non-empty estate is a red flag for scope drift - re-run Step 0. |
| ITSM | Not supported - inventory snapshot has no per-cluster failure surface to ticket on. |
| Aspect | Value |
|---|---|
| Purpose | Bulk-apply UpdateRing, UpdateStartWindow, UpdateExclusionsWindow, and UpdateExcluded tags from a CSV. Default-stamps UpdateExcluded=False on any cluster that does not already carry the tag (v0.7.90) so the operator hard-override is always discoverable in the Azure portal. |
| Inputs | csv_path (required). |
| Trigger | Manual only (workflow_dispatch / Run pipeline button). No schedule shipped - this is a deliberate change-controlled operation that should follow a CSV edit + review. Add a schedule: / schedules: block if your CSV is auto-generated and you want periodic re-application. |
| Cmdlets invoked | Set-AzLocalClusterUpdateRingTag. |
| Depends on | Step 1 produces the input CSV (the operator typically downloads cluster-inventory.csv, edits ring / window / exclusion values in a PR, then re-uploads it as the input to this pipeline). |
| Artefacts | UpdateRingTag_YYYYMMDD_HHmmss.csv (per-cluster CSV log: ClusterName, ResourceGroup, SubscriptionId, ResourceId, Action, PreviousTagValue, NewTagValue, Status, Message) and UpdateRingTag_Results.json (the same rows in JSON form, written from -PassThru) inside the published log artifact. Since v0.8.0 the pipeline job summary tab also renders a result breakdown table (Created / Updated / Already in sync / Skipped / WhatIf / Failed) plus a collapsible per-cluster details block, at parity with Step 3 / Step 5 / Step 7 / Step 9. |
| When to run | After editing the inventory CSV; whenever ring membership or maintenance windows change. |
| RBAC | Write to tags only. The built-in Tag Contributor role on the cluster scope is sufficient (Microsoft.Resources/tags/*). Since v0.7.65, Set-AzLocalClusterUpdateRingTag writes via Microsoft.Resources/tags/default PATCH, so the broader Microsoft.AzureStackHCI/clusters/write is not required. |
| Exit conditions | Pipeline run is green when every CSV row is processed (added / updated / unchanged). Per-cluster tag-write failures surface in the run log; the pipeline does not currently fail on per-row errors. Re-run after triaging is safe (the cmdlet is idempotent). |
| ITSM | Not supported - tag-write operation is operator-driven and idempotent; per-row failures are surfaced in the run log for direct triage rather than ticketed. |
| Aspect | Value |
|---|---|
| Purpose | Read-only advisor that compares the cron schedule(s) in your apply-updates.yml to the UpdateStartWindow tag values present on your clusters, and flags any (UpdateRing, UpdateStartWindow) pair that no cron in apply-updates.yml will ever reach. Never edits tags or YAML. |
| Inputs | pipeline_path (file or folder; default .github/workflows on GitHub Actions, .azure-pipelines on Azure DevOps - the standard consumer locations for the bundled apply-updates.yml sample), lead_time_minutes (0-60, default 5), include_untagged (default false), module_version (optional). |
| Trigger | Manual (workflow_dispatch / Run pipeline button) plus scheduled weekly on Mondays at 05:00 UTC (cron '0 5 * * 1'). Deliberately runs before the daily fleet-connectivity-status (05:30 UTC), fleet-update-status (06:00 UTC), and fleet-health-status (07:00 UTC) pipelines so its drift annotations land at the top of the operator's Monday-morning inbox. Edit the cron in the YAML to change cadence. |
| Cmdlets invoked | Test-AzLocalApplyUpdatesScheduleCoverage, Get-AzLocalApplyUpdatesScheduleConfig (when a schedule.yml is present in the repo). |
| Depends on | Step 1 (cluster inventory + UpdateStartWindow tags), Step 2 (tags applied) for non-trivial output. Runs fine on an empty fleet but the audit is meaningless. |
| Artefacts | schedule-coverage-audit.xml (JUnit, one <testcase> per (UpdateRing, UpdateStartWindow) pair, uncovered = <failure>), schedule-coverage-audit.csv (full Audit view with Status / Recommendation columns), schedule-coverage-matrix.csv (every distinct (Ring, Window) pair with its required cron), schedule-coverage-recommend.md (ready-to-paste GH Actions + Azure DevOps cron blocks), markdown step summary. |
| When to run | Hands-off scheduled. Trigger manually whenever you have just tagged a new ring or changed a maintenance window - see the end-to-end runbook in section 8.3. |
| RBAC | Read-only - Reader on the cluster scope plus Microsoft.ResourceGraph/resources/read. No write actions are ever taken. Covered by the Azure Stack HCI Update Operator (custom) custom role. |
| Exit conditions | Pipeline run is always green - uncovered (Ring, Window) pairs surface as JUnit <failure> entries in the Tests tab so they are visible without failing the whole run. |
| ITSM | Not supported - schedule coverage drift is a one-time YAML-edit fix, not a per-incident triage queue. |
| Introduced | v0.7.65. |
| Aspect | Value |
|---|---|
| Purpose | Daily fleet-wide read-only snapshot of Arc agent connectivity, physical NIC inventory + status histogram, Azure Resource Bridge reachability, and a bidirectional node-coverage reconciliation between each cluster's reportedProperties.nodes and the Arc-tagged physical machines visible in Resource Graph. The v0.7.85 release added the "How to interpret + act on a non-zero reconciliation" subsection in the pipeline summary with per-direction remediation lists (positive delta = Arc has more machines than the cluster reports; negative delta = cluster reports more nodes than Arc can see) and an inline Resource Graph query template for triage. |
| Inputs | subscription_ids (optional - comma-separated; defaults to every subscription the federated identity can read), module_version (optional). v0.7.4-style ITSM toggles are also exposed: raise_itsm_ticket, itsm_config_path, itsm_dry_run, itsm_force_create (all optional, default off). |
| Trigger | Manual (workflow_dispatch / Run pipeline button) plus scheduled daily at 05:30 UTC (cron '30 5 * * *'). Deliberately runs 30 minutes before fleet-update-status (06:00 UTC) so connectivity issues are visible upstream of update reporting. Edit the cron in the YAML to change cadence. |
| Cmdlets invoked | Get-AzLocalFleetConnectivityStatus. When ITSM is enabled: Get-AzLocalItsmConfig. |
| Depends on | None directly. Step 0 should be green at least once so the subscription scope is trusted. This is the upstream "can we see the fleet at all?" probe that other fleet pipelines silently rely on. |
| Artefacts | fleet-connectivity-status.xml (JUnit, one <testcase> per cluster), fleet-cluster-connectivity.csv (per-cluster reconciliation), fleet-arc-status-summary.csv (per-cluster Arc agent counts), fleet-arc-non-connected-machines.csv (per-machine triage list), fleet-physical-nics.csv (NIC issues only), fleet-physical-nic-all.csv (full NIC inventory), fleet-physical-nic-stats.csv (histogram by NicType + NicStatus), fleet-arb-status.csv (Azure Resource Bridge state), markdown job summary including the "How to interpret + act on a non-zero reconciliation" guidance subsection. |
| When to run | Hands-off scheduled. Trigger manually whenever the apply-updates chain returns "zero ready clusters" or the fleet count drops unexpectedly - Step 4 is the upstream "can we see the fleet at all?" probe. |
| RBAC | Read-only - Reader plus Microsoft.ResourceGraph/resources/read, Microsoft.AzureStackHCI/edgeDevices/read, Microsoft.HybridCompute/machines/read, and Microsoft.ResourceConnector/appliances/read. All four already live in the Azure Stack HCI Update Operator (custom) custom role definition shipped in section 3.1. |
| Exit conditions | Pipeline run is green when the snapshot completes. Per-cluster connectivity issues surface as JUnit <failure> entries; non-zero reconciliation deltas surface in the markdown summary with remediation guidance. |
| ITSM | Supported (opt-in via raise_itsm_ticket=true / raiseItsmTicket=true). When enabled, one ServiceNow incident is raised per cluster with an Arc-disconnected machine or non-zero reconciliation delta; dedupe key is per-cluster + reason via the matrix at ./.itsm/azurelocal-itsm.yml. itsm_dry_run builds payloads without creating tickets; itsm_force_create bypasses dedupe. |
| Introduced | v0.7.79 (reconciliation enhanced in v0.7.85). |
Behaviour change in v0.8.81: status icons in the Summary counts, Not-Ready and All-clusters tables now render via the new shared
Get-AzLocalStatusIconMapprivate helper (host-aware: GitHub Markdown shortcodes on theGitHubActionshost, Unicode glyphs everywhere else - fixes literal:white_check_mark:text on Azure DevOps step summaries). The Cluster column wraps in a portal deep-link viaGet-AzLocalClusterPortalLinkand the standing Ctrl-click tip banner is single-sourced viaGet-AzLocalCtrlClickTip. The Up-to-Date bucket gains the readiness-cascade check icon.
Behaviour change in v0.8.74: clusters that have already applied every required update are now classified as Up to Date (counted separately) instead of being lumped into the Not-Ready bucket. The summary's "Up to date" count is now accurate, the "Not-Ready clusters (review first)" table excludes Up-to-Date and Ready clusters, and the "All clusters detail" table shows a readable
Statuscolumn in place of the oldReadyboolean. Classification is shared with Step 7 and Step 9 via theGet-AzLocalClusterReadinessStatushelper.
| Aspect | Value |
|---|---|
| Purpose | Pre-flight, report-only readiness + blocking-health snapshot for a single UpdateRing. Always succeeds - per-cluster failures show up as JUnit test failures rather than failing the whole run. |
| Inputs | update_ring (required), throttle_limit (optional). |
| Trigger | Manual only by default (workflow_dispatch / Run pipeline button). No schedule is shipped because the update_ring input is required and there is no single ring value that would be correct for every consumer. See the recommended customisation note below for the per-ring scheduling pattern + lead-time guidance. |
| Cmdlets invoked | Get-AzLocalClusterInventory, Get-AzLocalClusterUpdateReadiness, Test-AzLocalClusterHealth. |
| Depends on | Step 1 (an UpdateRing tag must exist on at least one cluster to scope the readiness query) and Step 2 (to apply that tag at scale). |
| Artefacts | readiness.xml, readiness.csv, health-blocking.xml, health-blocking.csv. |
| When to run | 12-72 hours before each Step 7 wave for the same ring (12-24h for small rings, 48-72h for large rings of 50+ clusters) - long enough for an operator to triage any <failure> entries before the maintenance window opens, short enough that the readiness signal is still fresh. See the recommended customisation note below for the cron pattern. |
| RBAC | Read-only. Covered by the Azure Stack HCI Update Operator (custom) custom role. |
| Exit conditions | Pipeline run is always green - per-cluster readiness gaps and blocking health checks surface as JUnit <failure> entries in the Tests tab. Caveat: because the pipeline never goes red, a silently-empty readiness.xml (e.g. ring tag typo, zero clusters in scope) will not generate a red email - check the Tests tab / readiness.xml artefact after each run, or wire the JUnit reporter into your existing CI status surface. |
| ITSM | Not supported - this is a pre-flight gate intended for operator review before the Step 7 wave; readiness gaps surface as JUnit <failure> entries and feed the Step 7 ITSM dispatch downstream rather than raising tickets here. |
RECOMMENDED CUSTOMISATION: schedule Step 5 per ring, 12-72h ahead of the matching Step 7 cron. Unlike Step 7 (which silently does nothing if you forget to schedule it), Step 5 is recommended rather than mandatory - Step 7 does its own internal
Get-AzLocalClusterUpdateReadinessper-cluster pre-flight, so skipping Step 5 will not break the apply step. What you lose by skipping it is the human review window - the chance to see "12 clusters in Ring2 have blocking health checks" and intervene before the maintenance window starts.Because
update_ringis a required input there is no single bundled cron - add oneschedule:(GitHub Actions) /schedules:(Azure DevOps) entry per ring you intend to patch, each settingupdate_ring:via the workflow inputs (GH Actionswith:/ ADOparameters:). Anchor each Step 5 cron to the matching Step 7 cron (same days, earlier by the lead time):
Ring size Recommended Step 5 lead time Step 5 cron (UTC, anchoring on a Step 7 Sat 02:00 window) Rationale Small (<= 10 clusters) 12-24 hours ahead '0 2 * * 5'(Fri 02:00 ahead of a Sat 02:00 Step 7)Enough to triage a handful of <failure>entries; fresh enough that the health signal is still meaningful at apply time.Medium (10-50 clusters) 24-48 hours ahead '0 2 * * 4'(Thu 02:00 ahead of a Sat 02:00 Step 7)Lets you raise tickets / engage cluster owners on per-cluster issues before the weekend. Large (50+ clusters) 48-72 hours ahead '0 2 * * 3'(Wed 02:00 ahead of a Sat 02:00 Step 7)Allows time to escalate, swap clusters into a deferral ring ( UpdateExcluded=True), or stage parallel mitigation.Known gap: the Step 3 schedule-coverage audit currently validates Step 7 cron-to-
UpdateStartWindowcoverage only - it does not audit whether each Step 5 cron is correctly anchored to a Step 7 cron. Pair Step 5 and Step 7 cron edits in the same PR so the lead-time relationship is reviewable. The end-to-end runbook in the parent README's section 8.1.1 has worked examples for the most common ring layouts.
Behaviour change in v0.8.81 (Step 6 - Apply-Updates Readiness Gate): NOTE this callout describes the Apply-Updates Readiness Gate report (
Export-AzLocalClusterReadinessGateReport), which is internal to the Step 7 apply-updates pipeline (not the Sideload Updates pipeline documented in the table below). The gate report drops its inline icon switch in favour of the new sharedGet-AzLocalStatusIconMapprivate helper (host-aware: fixes literal:white_check_mark:text on Azure DevOps step summaries); the Cluster column wraps in a portal deep-link viaGet-AzLocalClusterPortalLink; and the standing Ctrl-click tip banner is single-sourced viaGet-AzLocalCtrlClickTip.
OPT-IN, OFF BY DEFAULT, self-hosted runner required. Introduced in v0.8.7. This pipeline is inert unless the repository variable
SIDELOAD_UPDATESis the literal string'true'. Full architecture, the scheduled-task survival model, the shared-state / multi-runner contract, the auth-map, and the catalog format are in sideload.md; robocopy throttling guidance is in sideload-robocopy.md.
| Aspect | Value |
|---|---|
| Purpose | Pre-stage Azure Local solution-update media onto clusters that cannot pull updates from Azure directly (dark / air-gapped / restricted-egress fabrics). It Robocopies the CombinedSolutionBundle (or OEM SBE package) to each cluster's infrastructure import SMB share, verifies the SHA256 over WinRM, runs Add-SolutionUpdate, and flips UpdateSideloaded=True so the downstream Step 7 apply can proceed. |
| Inputs | update_ring (optional - a single ring, a ;-delimited list, or *** for every tagged cluster; default ***), dry_run (optional, default true - previews the plan + transitions with no staging / task / tag changes), module_version (optional pin). |
| Trigger | Manual only by default (workflow_dispatch / Run pipeline button). No schedule ships because the pipeline requires an on-prem self-hosted runner labelled azlocal-sideload (GH) / a self-hosted agent in a pool with the azlocal-sideload demand (ADO) that most repos do not have. Once the runner/agent is online, uncomment the bundled */30 * * * * cron inside the BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers block to drive the state machine. |
| Cmdlets invoked | Resolve-AzLocalSideloadPlan (selects clusters whose next apply window is within SIDELOAD_LEAD_DAYS), Invoke-AzLocalSideloadUpdate (the re-entrant state machine), Export-AzLocalSideloadStatusReport + Add-AzLocalSideloadStepSummary (JUnit + Markdown summary). |
| Re-entrant state machine | Each run advances every in-scope cluster by one transition and exits: Planned -> Copying -> Copied -> Verified -> Imported -> SideloadFlagged. The multi-hour copy itself runs in a detached Windows Scheduled Task (driven by Tools/Invoke-AzLocalSideloadCopyTask.ps1) so no pipeline run is ever long-lived. Re-running is always safe; a stale Copying heartbeat (> SIDELOAD_HEARTBEAT_STALE_MINUTES) is re-driven. |
| Depends on | Step 1 (UpdateRing + sideload tags present, including UpdateAuthAccountId), Step 2 (to apply those tags at scale), and a populated sideload-auth-map.csv + sideload-catalog.yml. The downstream Step 7 apply consumes the UpdateSideloaded=True flag this pipeline sets. |
| Artefacts | sideload-status.xml (JUnit, one cluster per test), sideload-status.csv, plus per-run copy/verify logs under the shared SIDELOAD_STATE_ROOT\logs. |
| Shared state | SIDELOAD_STATE_ROOT must be a UNC path that every runner/agent can read+write (state\, logs\, cache\). Verified media is cached under SIDELOAD_CACHE_ROOT (defaults to <state-root>\cache) so a bundle is downloaded + hashed once and reused across clusters. |
| RBAC | ARG fleet read + Key Vault secret read + UpdateSideloaded tag write for the pipeline identity; a separate Active Directory [pscredential] (built from two Key Vault secrets named in the auth-map row) is used for the cluster WinRM remoting and Add-SolutionUpdate. |
| Exit conditions | Pipeline run is green when every in-scope cluster either advances cleanly or is correctly classified as a skip. Per-cluster copy / verify / import failures surface as JUnit <failure> entries. |
| ITSM | Not supported in v0.8.7 (staging is a pre-apply preparation step; failures surface as JUnit <failure> entries for operator review). |
Behaviour change in v0.8.81: the per-host apply-updates step summary (
Add-AzLocalApplyUpdatesStepSummary) drops its inline$iconSuccess/$iconFail/$iconBlock/$iconSkip/$iconWarndefinitions in favour of the new sharedGet-AzLocalStatusIconMapprivate helper (fixes literal:white_check_mark:text on Azure DevOps step summaries). The Cluster column in both the Cluster Actions and Clusters Skipped at Readiness Gate tables now resolves each cluster's resource id from the readiness CSV (built upstream byGet-AzLocalClusterUpdateReadiness) and wraps the cell in a portal deep-link viaGet-AzLocalClusterPortalLink. Adds the standing Ctrl-click tip viaGet-AzLocalCtrlClickTip. The upstream apply-results JSON shape (ClusterName/Status/UpdateName/Duration/Message) is intentionally unchanged - the cluster id lookup is done in the renderer.
Behaviour change in v0.8.74: the pre-apply readiness gate report (
Export-AzLocalClusterReadinessGateReport) replaces its binaryReady?column with a readableStatuscolumn so fully-patched clusters now show Up to Date instead of a no-entry icon that implied failure. The header line shows a distinctUp to Datecount and the step emits a new additiveUP_TO_DATE_COUNT/UpToDateCountoutput (theREADY_COUNTgate that drives the apply is unchanged). Classification is shared with Step 5 and Step 9 via theGet-AzLocalClusterReadinessStatushelper. When the gate finds no clusters ready, the "No Clusters Ready" summary (Add-AzLocalNoReadyClustersStepSummary) now renders an Up-to-Date vs Not-Ready breakdown - so an idle run makes clear that already-patched clusters are a healthy steady state (a notice, not a warning, when every cluster is up to date) rather than a fault.
Behaviour change in v0.8.78: per-cluster
ScheduleBlocked/SideloadedBlocked/ExcludedByTagoutcomes now render as JUnit<skipped>instead of<failure>sodorny/test-reporterno longer flips the Step.07 check RED on by-design gate-respect outcomes -HealthCheckBlockeddeliberately stays a<failure>because a critical health failure IS actionable. The Step 7 Readiness KPI table also gains optional Already Up to Date and Not Ready (needs attention before updating) rows, wired in both shippedapply-updates.ymltemplates from the upstreamreadiness.UpToDateCount/readiness.NotReadyCountoutputs (emitted since v0.8.74), so operators see the full ring breakdown - eligible / started / skipped / blocked / up-to-date / needs-attention - in one place. GitHub Actions only:actions/download-artifact@v6 -> @v7to silence the Node 20 deprecation warning (v7 chosen over v8 because v8 has a breakingdigest-mismatch=errordefault).
New in v0.8.79 - operator break-glass override (
Force Immediate Update): optional pipeline parameter onapply-updates.yml(force_immediate_updateGitHub Actions choice input /forceImmediateUpdateAzure DevOps boolean parameter) defaulting tofalse. When set totrueon a manual run / queue only, bypasses the per-clusterUpdateStartWindow/UpdateExclusionsWindowStep 3c maintenance-window gate for emergency / out-of-window patching. Defence in depth: GitHub Actions YAML collapses the flag to'false'for any non-workflow_dispatchevent (${{ github.event_name == 'workflow_dispatch' && github.event.inputs.force_immediate_update || 'false' }}); Azure DevOps rechecks$(Build.Reason) -eq 'Manual'at runtime before honouring the parameter. AWARNING:GUI label is prepended on both pipeline hosts; a high-visibility::warning::(GHA) /##vso[task.logissue type=warning](ADO) banner is emitted into the run log when the override fires. The override is unreachable from theapply-updates-schedule.ymlconfiguration file (noforceImmediateUpdatefield exists onNew-AzLocalApplyUpdatesScheduleConfig/Resolve-AzLocalPipelineUpdateRing/Get-AzLocalApplyUpdatesScheduleAudit). Other readiness gates (connectivity, health, sideload status,UpdateExcludedoperator hard-override) continue to apply. Maps toInvoke-AzLocalReadinessGatedClusterUpdate -ForceImmediateUpdate->Start-AzLocalClusterUpdate -IgnoreScheduleTagsfor direct cmdlet use.
| Aspect | Value |
|---|---|
| Purpose | Apply updates to clusters filtered by UpdateRing tag value. The actual mutation step; every other pipeline in this folder is either preparation, monitoring, or reporting. |
| Inputs | update_ring (required), update_name (optional - leave blank for latest), dry_run (optional), throttle_limit (optional). v0.7.4 adds raise_itsm_ticket, itsm_config_path, itsm_dry_run, itsm_force_create (all optional, defaults preserve existing behaviour). |
| Trigger | Manual only by default (workflow_dispatch / Run pipeline button). No schedule is shipped - you must add one. See the mandatory customisation note below and the schedule-alignment guidance in section 8. |
| Cmdlets invoked | Get-AzLocalApplyUpdatesScheduleConfig, Get-AzLocalClusterUpdateReadiness, Start-AzLocalClusterUpdate. When ITSM is enabled: Get-AzLocalItsmConfig. |
| Depends on | Step 1 (UpdateRing tags present), Step 5 (readiness reviewed for the wave). Step 3 (schedule coverage audit) should be green so the cron(s) that start this pipeline actually fire at every tagged UpdateStartWindow. |
| Artefacts | update-results.xml (JUnit, one cluster per test), update-logs/* (CSV + detail). When ITSM is enabled: itsm-results.csv, itsm-results.xml. |
| When to run | During the maintenance window for each ring, after the readiness assessment is reviewed. |
| RBAC | Write to clusters required. Covered by the Azure Stack HCI Update Operator (custom) custom role (Microsoft.AzureStackHCI/clusters/updates/apply/action + cluster-update reads). |
| Exit conditions | Pipeline run is green when every in-scope cluster either succeeds or is correctly classified as ScheduleBlocked / SideloadedBlocked / ExcludedByTag (these are skips, not failures - rendered as JUnit <skipped> from v0.8.78). Per-cluster update failures and HealthCheckBlocked outcomes surface as JUnit <failure> entries; long-running runs are tracked by Step 8. |
| ITSM | Supported (opt-in via raise_itsm_ticket=true / raiseItsmTicket=true). When enabled, one ServiceNow incident is raised per cluster whose update run finished with a failure state; classification skips (ScheduleBlocked / SideloadedBlocked / ExcludedByTag) do not generate tickets. itsm_dry_run builds payloads without creating tickets; itsm_force_create bypasses dedupe. |
MANDATORY CUSTOMISATION: the Apply Updates pipeline does not ship with a schedule. The cluster
UpdateStartWindow/UpdateExclusionsWindowtags only gate updates while the pipeline is already running; they do not start the pipeline. If you (a) useUpdateStartWindowtags to define when updates may be installed and (b) leave the shippedapply-updates.ymlwithworkflow_dispatchonly (GH) /trigger: none(ADO), no updates will ever be applied automatically - the pipeline will simply never start during the window.Add a
schedule:(GitHub Actions) /schedules:(Azure DevOps) block toapply-updates.ymlthat fires at (or a few minutes before) the start of everyUpdateStartWindowyou have tagged. One cron entry per distinct window value. Worked examples and the per-cluster scheduling model are in section 8.Pipeline summaries (Step 7 GH Actions + Azure DevOps) classify per-cluster skips as
ScheduleBlocked(outsideUpdateStartWindowor insideUpdateExclusionsWindow),SideloadedBlocked(UpdateSideloaded=Falseon a sideloaded-workflow cluster) and, new in v0.7.90,ExcludedByTag(UpdateExcluded=Trueoperator hard override). The Actions Required callout points operators at theUpdateExcludedtag when one or more clusters are intentionally held out of automation.
Behaviour change in v0.8.81: the
$stateIcon(Statecolumn) and$statusIcon(Current Step Statuscolumn) switches inExport-AzLocalUpdateRunMonitorReportno longer hard-code GitHub-Markdown shortcodes (:large_blue_circle:/:hourglass_flowing_sand:/:red_circle:/ etc.) which previously rendered as literal text on Azure DevOps step summaries. Both switches now read from the sharedGet-AzLocalStatusIconMapprivate helper using the newState*(StateInProgress/StateSucceeded/StateFailed/StateNotStarted/StateUnknown) andStatus*(StatusInProgress/StatusCancelled) keys. JUnit XML output and CSV columns are unchanged.
| Aspect | Value |
|---|---|
| Purpose | In-flight observability for an active update wave. Reports clusters whose latest update run is currently InProgress with the CURRENT STEP each cluster is on, the PROGRESS (completed/total steps), and the ELAPSED DURATION. Long-running runs are flagged via the long_running_threshold_hours input (default 6h) so an operator can spot a stuck cluster without waiting for the next daily fleet-update-status snapshot. No new ARM calls: data source is Get-AzLocalUpdateRuns -Latest (one row per cluster) - the same call already made by fleet-update-status. v0.7.96 enhancements: (1) every in-flight row now carries a new Status column from properties.progress.status (Success / Error / InProgress / NotStarted / Skipped / Cancelled / Unknown); (2) a new ErrorMessage column surfaces the deepest non-empty errorMessage walked from the nested properties.progress.steps[] tree (uses a coalesce(e9Msg..e1Msg) recursion via the new private Get-DeepestErrorMessage helper); (3) runs whose Status='Error' but elapsed time is BELOW long_running_threshold_hours (the stuck-but-still-fresh Arizona-shaped case) now generate a new failureType='StepError' JUnit entry so they appear in the Checks tab WITHOUT needing the long-running threshold to fire; (4) the always-shown Failed-runs block carries an MS Learn TSG link plus the Progress Status column; (5) markdown summary's Cluster Name and Update Name cells render as <a href="https://portal.azure.com/..."> deep-links to the cluster's Updates blade and the single-instance update-run history view. |
| Inputs | scope (all-clusters (default) or by-update-ring), update_ring (only used when scope=by-update-ring; single ring, Prod;Ring2 semicolon-list, or *** wildcard), long_running_threshold_hours (0-48, default 6), module_version (optional). |
| Trigger | Manual only by default (workflow_dispatch / Run pipeline button). A commented-out schedule: cron '*/30 * * * *' (GH) / schedules: cron '*/30 * * * *' (ADO) sample is shipped inside the # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers block - uncomment it for the duration of an active wave, then re-comment it once the wave drains, so you do not generate 48 empty runs per day between waves. |
| Cmdlets invoked | Get-AzLocalUpdateRuns, Get-AzLocalClusterInventory (used for ring-scope mode). |
| Depends on | Step 7 must be active (at least one InProgress run) for the output to be non-empty. Runs cleanly on an empty fleet (reports "no in-flight runs"). |
| Artefacts | update-monitor.xml (JUnit, one <testcase> per in-flight cluster - the failure message contains current step + progress + elapsed duration when the run is over long_running_threshold_hours, or the deepest-step error detail when failureType='StepError' fires under v0.7.96), update-monitor.csv (full per-cluster rows including the new Status + ErrorMessage columns), markdown job summary with the in-flight table (cluster + update names as portal links from v0.7.96) and long-running flag column. |
| Exit conditions | Pipeline run is green when the snapshot completes. Long-running runs (over long_running_threshold_hours) AND stuck-step errors (failureType='StepError', v0.7.96+) surface as JUnit <failure> entries so they appear in the Tests / Checks tab without failing the whole pipeline. v0.7.96 also adds two new GITHUB_OUTPUT values: STEP_ERRORED (count of runs that hit a step error inside the threshold) and UNRESOLVED_FAILURES (count of Failed runs surfaced from Get-AzLocalUpdateRunFailures). |
| When to run | While a wave is active (manually, or via the commented cron). The two daily snapshot pipelines (Step 9 and Step 10) remain the steady-state daily reports; this pipeline is purpose-built for the "is the apply-updates run I started two hours ago still progressing?" question that those daily snapshots cannot answer between runs. |
| RBAC | Read-only - Reader on the cluster scope, plus the cluster-update API read paths already enumerated in the Azure Stack HCI Update Operator (custom) custom role. |
| ITSM | Not supported - in-flight monitor exists to be a fast feedback loop during an active wave; long-running runs surface as JUnit <failure> entries for the operator on call, and post-wave failures are picked up + ticketed by the daily Step 9 snapshot. |
| Introduced | v0.7.90. Enhanced in v0.7.96 with the Status + ErrorMessage columns, the new StepError JUnit failure type, the always-shown Failed-runs block, portal-linked Cluster Name / Update Name cells, and the new STEP_ERRORED / UNRESOLVED_FAILURES GITHUB_OUTPUT values. Behaviour change in v0.7.98: UX overhaul of the in-flight table - rows now sort by composite SeverityScore (StepError severity x 1000 + RunSeverity x 100 + elapsed-hours bucket) so stuck step errors and runs over 14 days bubble to the top regardless of cluster alphabetical order; each row carries per-cell StateIcon + StatusIcon icons and a horizontal chip stack of flags (STEP-STUCK, RUN-STUCK, UNRESOLVED, RECENT-FAIL); the job summary opens with a single CRITICAL / WARN / OK fleet status badge that collapses the worst row across the fleet into one line (e.g. CRITICAL - 1 stuck step error(s), 1 run(s) > 14d, 1 step(s) > 4h); each failed step's errorMessage is now wrapped in a collapsible <details><summary>Verbose error</summary>...</details> block; and JUnit <testsuite time=".."> + <testcase time=".."> are populated with real per-run elapsed seconds (was time="0" before, which made GitHub Test Reporter render "5 tests were completed in 0ms"). |
Behaviour change in v0.8.81: the GitHub-Markdown shortcodes used by the Critical Health Status table, the Primary Status table, and the Version Distribution Support column are now routed through the shared
Get-AzLocalStatusIconMapprivate helper using the newInfo/GreenCircle/YellowCircle/CycleArrows/GreyQuestion/SupportSupported/SupportUnsupported/SupportUnknownkeys (fixes literal:green_circle:/:yellow_circle:/:information_source:/ etc. text on Azure DevOps step summaries). JUnit XML output and CSV columns are unchanged.
| Aspect | Value |
|---|---|
| Purpose | Daily fleet-wide snapshot of cluster update state. Read-only. v0.7.90 pivots the Version Distribution markdown table by YYMM (leading column Version = 2511, 2604, ...; new Update Versions column lists each distinct full version installed within that YYMM as <version> x <count> separated by <br>; rows sorted ascending by YYMM so the oldest YYMM is at the top). The underlying <testsuite name="Fleet Version Distribution"> JUnit XML is unchanged - still one <testcase> per distinct full CurrentVersion - so machine-readable consumers and CI test-reporters are unaffected. v0.7.96 reworks the Primary Status bucket cascade to align with the Azure portal Update Manager state filter: NeedsAttention is now promoted into the Update Failed bucket; PreparationFailed lands in a new Action Required bucket (with its own actionable guidance in the Actions Required callout because the run never started, so re-arming is required); PreparationInProgress is folded into Update In Progress. The new bucket surfaces via (1) a new <property name="primaryActionRequired" value="$stActionRequired"/> JUnit attribute on the primary testsuite, (2) per-testcase failureType='PreparationFailed' instead of the generic UpdateFailure, (3) a new ACTION_REQUIRED GITHUB_OUTPUT (actionRequired pipeline variable in ADO), (4) Summary.UpdateFailures + Summary.ActionRequired in readiness-status.json, and (5) a new "Action Required (PreparationFailed)" row in the Primary Status markdown table with separate actionable prose in the Actions Required callout. The 📜 Update Run History and Error Details markdown table renders Cluster Name and Update Name cells as <a href="https://portal.azure.com/..."> deep-links (cluster's Updates blade + the single-instance update-run history view) - same linking as the Step.8 in-flight table. |
| Inputs | Scope (-AllClusters or -ScopeByUpdateRingTag), throttle_limit (optional). v0.7.4-style ITSM toggles are also exposed: raise_itsm_ticket, itsm_config_path, itsm_dry_run, itsm_force_create (all optional, default off). |
| Trigger | Manual (workflow_dispatch / Run pipeline button) plus scheduled daily at 06:00 UTC (cron '0 6 * * *'). Edit the cron in the YAML to change cadence. |
| Cmdlets invoked | Get-AzLocalClusterInventory, Get-AzLocalClusterUpdateReadiness, Get-AzLocalUpdateSummary, Get-AzLocalAvailableUpdates, Get-AzLocalUpdateRuns, Get-AzLocalUpdateRunFailures, Get-AzLocalLatestSolutionVersion. When ITSM is enabled: Get-AzLocalItsmConfig. |
| Depends on | None directly. Best-quality output when Step 4 (connectivity) has run earlier the same morning, since clusters Arc can't see won't appear in the readiness or update-summary results. |
| Artefacts | readiness-status.xml / .csv / .json (the .json carries the rolled-up bucket counts under Summary.* - UpdateFailures, ActionRequired, HealthFailures, SbeBlocked, InProgress, ReadyForUpdate, UpToDate, Other - and the new primaryActionRequired field mirrors the JUnit testsuite attribute from v0.7.96 for downstream automation), cluster-inventory.csv, update-summaries.csv, available-updates.csv, update-runs.csv. |
| When to run | Hands-off scheduled. Trigger manually for ad-hoc reporting. |
| RBAC | Read-only. Covered by the Azure Stack HCI Update Operator (custom) custom role. |
| Exit conditions | Pipeline run is green when the snapshot completes. Per-cluster issues (outdated version, failed run history) surface as JUnit <failure> entries. v0.7.96 distinguishes failureType='UpdateFailure' (run executed and failed - landed in NeedsAttention) from failureType='PreparationFailed' (run never started - landed in the new Action Required bucket) so downstream automation can split paging behaviour per failure mode. |
| ITSM | Supported (opt-in via raise_itsm_ticket=true / raiseItsmTicket=true). When enabled, one ServiceNow incident is raised per unresolved failed update run picked up by the daily snapshot; dedupe key combines cluster + update version + failure reason via ./.itsm/azurelocal-itsm.yml. itsm_dry_run builds payloads without creating tickets; itsm_force_create bypasses dedupe. |
| Introduced | Originally Step 7 in v0.7.4; renumbered to Step 8 in v0.7.90 when the in-flight monitor moved into Step 7; renumbered to Step 9 in v0.8.7 when the on-prem sideload pipeline took Step 6. v0.7.90 added the YYMM-pivoted Version Distribution table. v0.7.96 promoted NeedsAttention into the Update Failed bucket, added the new Action Required bucket for PreparationFailed, folded PreparationInProgress into Update In Progress, added the primaryActionRequired JUnit attribute, the ACTION_REQUIRED GITHUB_OUTPUT, Summary.UpdateFailures + Summary.ActionRequired in readiness-status.json, and portal-linked Cluster Name / Update Name cells in the Update Run History and Error Details markdown table. Behaviour change in v0.7.98: each <testcase> inside the 📜 Update Run History and Error Details testsuite now emits time = DurationMinutes * 60 (DurationMinutes from KQL datetime_diff('minute', EndTime, StartTime) via Get-AzLocalUpdateRunFailures), and the testsuite-level time= is the sum of those per-row seconds. The other two testsuites (FleetVersionDistribution, AzureLocalFleetUpdateStatus) intentionally stay at time="0" - they are instantaneous snapshot projections and a synthetic duration would be misleading. |
Behaviour change in v0.8.81: fixes the "Healthy + Unhealthy != Total" KPI counting bug. The previous KPI table summed only two buckets (Healthy + Unhealthy) and silently dropped clusters whose
HealthStatuswasIn progress/Unknown/Health check failedAND had no failure rows. The KPI block is now split into two tables: Cluster Counts (Total / Healthy / Unhealthy / Other, where Other is computed asTotal - Healthy - Unhealthyclamped non-negative so the three buckets always sum to Total) and Failing Checks Breakdown (Total / Critical / Warning / Distinct Reasons). Newother_clusterspipeline step output;-PassThruPSCustomObject gainsOtherClusters. Detailed Results columns reordered to put the most-specific identifier first:Severity | Title | Failure Reason | Description | Health Check Name | Failure Remediation | Target Resource Name | Target Resource Type | Last Occurrence | Resource Group. Description is a collapsible<details>block that surfaces drive/volume-level detail (e.g. the file path emitted by aMicrosoft.Health.FaultType.Volume.FileSystem.Corruption.Correctablewarning); Health Check Name is the raw fault-type name (preserved verbatim so operators can grep / triage by FaultType across runs). Icons in the table are wired through the new sharedGet-AzLocalStatusIconMapprivate helper.
| Aspect | Value |
|---|---|
| Purpose | Daily fleet-wide snapshot of 24-hour system health-check failures surfaced by every Azure Local cluster the identity can see. Independent of update activity - clusters that are "up to date" can still surface Critical / Warning issues that need operator triage. Read-only. |
| Inputs | severity (optional - Critical, Warning, or All; default All), update_ring_tag (optional - narrow to one wave), throttle_limit (optional). v0.7.4-style ITSM toggles are also exposed: raise_itsm_ticket, itsm_config_path, itsm_dry_run, itsm_force_create (all optional, default off). |
| Trigger | Manual (workflow_dispatch / Run pipeline button) plus scheduled daily at 07:00 UTC (cron '0 7 * * *'). Deliberately offset by one hour from fleet-update-status (06:00 UTC) to avoid agent and ARM contention. Edit the cron in the YAML to change cadence. |
| Cmdlets invoked | Get-AzLocalFleetHealthFailures, Get-AzLocalFleetHealthOverview. When ITSM is enabled: Get-AzLocalItsmConfig. |
| Depends on | None directly. Best-quality output when Step 4 (connectivity) has run earlier the same morning. |
| Artefacts | fleet-health-status.xml (JUnit, one <testcase> per failing check, grouped under Critical Health Failures / Warning Health Failures testsuites), fleet-health-detail.csv (one row per failing check), fleet-health-summary.csv (aggregated by FailureReason + Severity), markdown job summary. |
| When to run | Hands-off scheduled. Trigger manually for ad-hoc fleet-wide health triage outside the daily schedule, especially after operational events (capacity changes, network maintenance, certificate rotations) where health-check failures are expected to spike. |
| RBAC | Read-only. Covered by the Azure Stack HCI Update Operator (custom) custom role. |
| Exit conditions | Pipeline run is green when the snapshot completes. Critical and Warning health-check failures surface as JUnit <failure> entries grouped by severity. |
| ITSM | Supported (opt-in via raise_itsm_ticket=true / raiseItsmTicket=true). When enabled, one ServiceNow incident is raised per Critical or Warning health-check failure picked up by the daily snapshot; dedupe key combines cluster + FailureReason + Severity via ./.itsm/azurelocal-itsm.yml. itsm_dry_run builds payloads without creating tickets; itsm_force_create bypasses dedupe. |
| Introduced | v0.7.65. |