You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Step.10 Fleet Health Status
- Fixes the long-standing KPI bug where the legacy top table mixed two axes (cluster bucket counts vs failing-check counts) and produced confusing output like "11 + 8 != 20" when one or more clusters had HealthStatus other than 'Healthy' AND no failing detail rows (e.g. 'In progress', 'Unknown', 'Health check failed').
- Splits the table into two: **Cluster Counts** (Total / Healthy / Unhealthy / Other) and **Failing Checks Breakdown** (Total / Critical / Warning / Distinct Reasons). Adds an OtherClusters bucket so Cluster Counts always sum to Total Clusters in Subscription.
- Detailed Results: Title column now appears FIRST (collapsed when identical to the Failure Reason), adds a collapsible Description column with file paths / volume detail (the "which drive is corrupt" surface that was previously hidden), and adds a raw health-check FailureName column for grep/triage.
- Emits a new step output: other_clusters.
Shared step-summary helpers (Step.05-10)
- New private helpers in AzLocal.UpdateManagement/Private/:
- Get-AzLocalStatusIconMap.ps1 - host-aware status->icon map (Unicode glyphs for GitHub Actions / Local; Markdown shortcodes for Azure DevOps where the renderer prefers them).
- Get-AzLocalClusterPortalLink.ps1 - wraps cluster names in an Azure portal anchor (with ClusterPortalUrl override and -Escape switch).
- Get-AzLocalCtrlClickTip.ps1 - single source of truth for the "Hold Ctrl / Cmd / middle-click" tip line.
- Step.05 (Export-AzLocalClusterUpdateReadinessReport), Step.06 (Export-AzLocalClusterReadinessGateReport), Step.07 (Add-AzLocalApplyUpdatesStepSummary), Step.08 (Export-AzLocalUpdateRunMonitorReport), Step.09 (Export-AzLocalFleetUpdateStatusReport) and Step.10 (Export-AzLocalFleetHealthStatusReport) now all read icons from the helper rather than inlining glyph constants. This eliminates the literal ✅ / 🔵 rendering on Azure DevOps and the duplicated glyph maps across Public cmdlets.
Tests
- 1209 Pester tests pass (40-test regression from a PS 5.1 [char] supplementary-plane trap caught and fixed in Get-AzLocalStatusIconMap before merge).
- New regression Describe blocks cover: helper registration, host-aware icon output, no key-drift between the two return blocks, Step.05/06/07/08/09/10 all import the helper, Step.10 KPI counting (Cluster Counts vs Failing Checks Breakdown, OtherClusters bucket sums correctly, Title-first ordering, Description column visibility).
Docs
- CHANGELOG.md, README.md, docs/release-history.md, Automation-Pipeline-Examples/README.md and Automation-Pipeline-Examples/docs/appendix-pipelines.md all updated with the v0.8.81 entry. 22 YAML pipeline templates re-pinned to GENERATED_AGAINST_MODULE_VERSION=0.8.81.
Copy file name to clipboardExpand all lines: AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -64,10 +64,10 @@ By the end of this guide you will have:
64
64
-**Fleet Connectivity Status** (Step.4, v0.7.79+, enhanced in v0.7.85) - read-only daily snapshot of Arc agent connectivity, physical NIC health, Azure Resource Bridge status, and the node-count reconciliation between cluster `reportedProperties.nodes` and Arc-tagged physical machines. *Scheduled daily 05:30 UTC + manual.*
65
65
-**Assess Update Readiness** (Step.5) - pre-flight, report-only readiness + blocking-health snapshot, published as JUnit XML. *Manual only.*
66
66
-**Sideload Updates** (Step.6, v0.8.7) - **opt-in, off by default, on-prem self-hosted runner/agent required.** Pre-stages solution-update media onto clusters that cannot pull updates from Azure directly (dark / air-gapped fabrics): Robocopy to the cluster import share, remote SHA256 verify, `Add-SolutionUpdate` import, then flip `UpdateSideloaded=True` for the downstream Step.7 apply. Re-entrant state machine driven by a frequent CRON; the multi-hour copy runs in a detached Scheduled Task. *Inert unless `SIDELOAD_UPDATES=true`. See [sideload.md](docs/sideload.md) and [sideload-robocopy.md](docs/sideload-robocopy.md).*
67
-
- **Apply Updates** (Step.7; **v0.8.79 break-glass override**: optional `force_immediate_update` / `forceImmediateUpdate` pipeline parameter (defaults `false`, **manual run / queue only**) bypasses the per-cluster `UpdateStartWindow` / `UpdateExclusionsWindow` maintenance-window gate for emergency / out-of-window patching - the override is unreachable from the scheduled `apply-updates-schedule.yml` configuration file and a high-visibility WARNING banner is emitted into the run log when it fires; **v0.8.78 step-summary UX polish**: ``ScheduleBlocked`` / ``SideloadedBlocked`` / ``ExcludedByTag`` outcomes render as JUnit ``<skipped>`` instead of ``<failure>`` so ``dorny/test-reporter`` no longer flips Step.07 RED on by-design gate-respect outcomes - ``HealthCheckBlocked`` deliberately stays a ``<failure>``; the Readiness KPI table now also surfaces **Already Up to Date** and **Not Ready (needs attention before updating)** rows so operators see the full ring breakdown - eligible, started, skipped, blocked, up-to-date, needs-attention - in one place; GHA-only: ``actions/download-artifact@v6 -> @v7`` silences the Node 20 deprecation warning) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
68
-
- **Monitor In-Flight Updates** (Step.8, v0.7.90; v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `<details>` Verbose Error block per row, and JUnit `<testsuite time="..">` + `<testcase time="..">` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `monitor-updates.yml` (v0.7.92+).*
69
-
-**Fleet Update Status** (Step.9, formerly Step.8; v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `<testcase>` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.*
70
-
-**Fleet Health Status** (Step.10, formerly Step.9) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.*
67
+
- **Apply Updates** (Step.7; **v0.8.81 step-summary polish**: per-host apply-updates step summary now uses the new shared `Get-AzLocalStatusIconMap` host-aware icon helper (fixes literal `:white_check_mark:` text on Azure DevOps step summaries), the Cluster column wraps in a portal deep-link via `Get-AzLocalClusterPortalLink`, and the standing Ctrl-click tip is single-sourced via `Get-AzLocalCtrlClickTip`; **v0.8.79 break-glass override**: optional `force_immediate_update` / `forceImmediateUpdate` pipeline parameter (defaults `false`, **manual run / queue only**) bypasses the per-cluster `UpdateStartWindow` / `UpdateExclusionsWindow` maintenance-window gate for emergency / out-of-window patching - the override is unreachable from the scheduled `apply-updates-schedule.yml` configuration file and a high-visibility WARNING banner is emitted into the run log when it fires; **v0.8.78 step-summary UX polish**: ``ScheduleBlocked`` / ``SideloadedBlocked`` / ``ExcludedByTag`` outcomes render as JUnit ``<skipped>`` instead of ``<failure>`` so ``dorny/test-reporter`` no longer flips Step.07 RED on by-design gate-respect outcomes - ``HealthCheckBlocked`` deliberately stays a ``<failure>``; the Readiness KPI table now also surfaces **Already Up to Date** and **Not Ready (needs attention before updating)** rows so operators see the full ring breakdown - eligible, started, skipped, blocked, up-to-date, needs-attention - in one place; GHA-only: ``actions/download-artifact@v6 -> @v7`` silences the Node 20 deprecation warning) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
68
+
- **Monitor In-Flight Updates** (Step.8, v0.7.90; **v0.8.81 host-aware icons**: the State and Current Step Status columns now use the shared `Get-AzLocalStatusIconMap` helper (fixes literal `:large_blue_circle:` / `:hourglass_flowing_sand:` text on Azure DevOps step summaries); v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `<details>` Verbose Error block per row, and JUnit `<testsuite time="..">` + `<testcase time="..">` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `monitor-updates.yml` (v0.7.92+).*
69
+
-**Fleet Update Status** (Step.9, formerly Step.8; **v0.8.81 host-aware icons**: the Critical Health Status table, the Primary Status table, and the Version Distribution Support column now use the shared `Get-AzLocalStatusIconMap` helper (fixes literal `:green_circle:` / `:yellow_circle:` / `:information_source:` text on Azure DevOps step summaries); v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `<testcase>` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.*
70
+
-**Fleet Health Status** (Step.10, formerly Step.9; **v0.8.81**: fixes the KPI counting bug where Healthy + Unhealthy did not sum to Total - the top KPI table is now split into **Cluster Counts** (Total / Healthy / Unhealthy / **Other**) and **Failing Checks Breakdown**, with a new `other_clusters` step output. Detailed Results columns reordered to put `Title` first and add a collapsible `Description` block that surfaces drive/volume-level detail (e.g. file paths from `Microsoft.Health.FaultType.Volume.FileSystem.Corruption.Correctable` warnings); raw fault-type `Name` preserved for grep/triage) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.*
71
71
- An end-to-end "ring-based" rollout pattern: Pilot -> Wave2 -> Production, with each ring gated on the previous wave's success.
72
72
-**Optional**: a ServiceNow integration that opens deduped incidents for clusters whose run status indicates the module's own retries cannot recover (failures, blocking health checks, sideloaded payload missing) - see [section 7](#7-optional-open-itsm-tickets-for-clusters-needing-operator-action).
0 commit comments