Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ By the end of this guide you will have:
- **Fleet Connectivity Status** (Step.4, v0.7.79+, enhanced in v0.7.85) - read-only daily snapshot of Arc agent connectivity, physical NIC health, Azure Resource Bridge status, and the node-count reconciliation between cluster `reportedProperties.nodes` and Arc-tagged physical machines. *Scheduled daily 05:30 UTC + manual.*
- **Assess Update Readiness** (Step.5) - pre-flight, report-only readiness + blocking-health snapshot, published as JUnit XML. *Manual only.*
- **Sideload Updates** (Step.6, v0.8.7) - **opt-in, off by default, on-prem self-hosted runner/agent required.** Pre-stages solution-update media onto clusters that cannot pull updates from Azure directly (dark / air-gapped fabrics): Robocopy to the cluster import share, remote SHA256 verify, `Add-SolutionUpdate` import, then flip `UpdateSideloaded=True` for the downstream Step.7 apply. Re-entrant state machine driven by a frequent CRON; the multi-hour copy runs in a detached Scheduled Task. *Inert unless `SIDELOAD_UPDATES=true`. See [sideload.md](docs/sideload.md) and [sideload-robocopy.md](docs/sideload-robocopy.md).*
- **Apply Updates** (Step.7) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
- **Apply Updates** (Step.7; **v0.8.78 step-summary UX polish**: ``ScheduleBlocked`` / ``SideloadedBlocked`` / ``ExcludedByTag`` outcomes render as JUnit ``<skipped>`` instead of ``<failure>`` so ``dorny/test-reporter`` no longer flips Step.07 RED on by-design gate-respect outcomes - ``HealthCheckBlocked`` deliberately stays a ``<failure>``; the Readiness KPI table now also surfaces **Already Up to Date** and **Not Ready (needs attention before updating)** rows so operators see the full ring breakdown - eligible, started, skipped, blocked, up-to-date, needs-attention - in one place; GHA-only: ``actions/download-artifact@v6 -> @v7`` silences the Node 20 deprecation warning) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
- **Monitor In-Flight Updates** (Step.8, v0.7.90; v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `<details>` Verbose Error block per row, and JUnit `<testsuite time="..">` + `<testcase time="..">` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `monitor-updates.yml` (v0.7.92+).*
- **Fleet Update Status** (Step.9, formerly Step.8; v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `<testcase>` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.*
- **Fleet Health Status** (Step.10, formerly Step.9) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ parameters:
default: false

variables:
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}'
reportsPath: '$(Build.ArtifactStagingDirectory)/reports'
# v0.8.7 sideload advisor defaults. Override at the pipeline / variable-group
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.77'
value: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down Expand Up @@ -276,6 +276,14 @@ stages:
variables:
readyCount: $[ stageDependencies.CheckReadiness.ReadinessCheck.outputs['readiness.ReadyCount'] ]
totalCount: $[ stageDependencies.CheckReadiness.ReadinessCheck.outputs['readiness.TotalCount'] ]
# v0.8.78: surface the readiness gate's Up-to-Date / Not-Ready breakdown
# in the apply-updates summary so the operator sees the FULL ring picture
# (not just the clusters that entered Apply Updates). Up-to-Date is a
# healthy steady state; Not-Ready includes clusters held back by a
# previously-failed run (state=NeedsAttention / UpdateFailed), an
# in-progress run, pending SBE prerequisites or a critical health failure.
upToDateCount: $[ stageDependencies.CheckReadiness.ReadinessCheck.outputs['readiness.UpToDateCount'] ]
notReadyCount: $[ stageDependencies.CheckReadiness.ReadinessCheck.outputs['readiness.NotReadyCount'] ]
# pull the upstream artifact-stamp through so the download step
# can reconstruct the timestamped artifact name produced by CheckReadiness.
readinessArtifactStamp: $[ stageDependencies.CheckReadiness.ReadinessCheck.outputs['stamp.artifactStamp'] ]
Expand Down Expand Up @@ -486,6 +494,11 @@ stages:
UpdateRing = $env:RESOLVED_UPDATE_RING
TotalCount = "$(totalCount)"
ReadyCount = "$(readyCount)"
# v0.8.78: pass the readiness gate's Up-to-Date / Not-Ready breakdown
# so the Readiness KPI surfaces the FULL ring picture, not just the
# clusters that entered Apply Updates.
UpToDateCount = "$(upToDateCount)"
NotReadyCount = "$(notReadyCount)"
Succeeded = "$(applyUpdates.Succeeded)"
Skipped = "$(applyUpdates.Skipped)"
Failed = "$(applyUpdates.Failed)"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.77'
value: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.77'
value: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.77'
value: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ parameters:
default: ''

variables:
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}'
reportsPath: '$(Build.ArtifactStagingDirectory)/reports'

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,8 @@ The table below is the ground truth for what each shipped YAML does **out of the

> **Behaviour change in v0.8.74**: the pre-apply readiness gate report (`Export-AzLocalClusterReadinessGateReport`) replaces its binary `Ready?` column with a readable `Status` column so fully-patched clusters now show **Up to Date** instead of a no-entry icon that implied failure. The header line shows a distinct `Up to Date` count and the step emits a new additive `UP_TO_DATE_COUNT` / `UpToDateCount` output (the `READY_COUNT` gate that drives the apply is unchanged). Classification is shared with Step 5 and Step 9 via the `Get-AzLocalClusterReadinessStatus` helper. When the gate finds no clusters ready, the "No Clusters Ready" summary (`Add-AzLocalNoReadyClustersStepSummary`) now renders an Up-to-Date vs Not-Ready breakdown - so an idle run makes clear that already-patched clusters are a healthy steady state (a notice, not a warning, when every cluster is up to date) rather than a fault.

> **Behaviour change in v0.8.78**: per-cluster `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` outcomes now render as JUnit `<skipped>` instead of `<failure>` so `dorny/test-reporter` no longer flips the Step.07 check RED on by-design gate-respect outcomes - `HealthCheckBlocked` deliberately stays a `<failure>` because a critical health failure IS actionable. The Step 7 Readiness KPI table also gains optional **Already Up to Date** and **Not Ready (needs attention before updating)** rows, wired in both shipped `apply-updates.yml` templates from the upstream `readiness.UpToDateCount` / `readiness.NotReadyCount` outputs (emitted since v0.8.74), so operators see the full ring breakdown - eligible / started / skipped / blocked / up-to-date / needs-attention - in one place. GitHub Actions only: `actions/download-artifact@v6 -> @v7` to silence the Node 20 deprecation warning (v7 chosen over v8 because v8 has a breaking `digest-mismatch=error` default).

| Aspect | Value |
|---|---|
| **Purpose** | Apply updates to clusters filtered by `UpdateRing` tag value. The actual mutation step; every other pipeline in this folder is either preparation, monitoring, or reporting. |
Expand All @@ -170,7 +172,7 @@ The table below is the ground truth for what each shipped YAML does **out of the
| **Artefacts** | `update-results.xml` (JUnit, one cluster per test), `update-logs/*` (CSV + detail). When ITSM is enabled: `itsm-results.csv`, `itsm-results.xml`. |
| **When to run** | During the maintenance window for each ring, after the readiness assessment is reviewed. |
| **RBAC** | Write to clusters required. Covered by the `Azure Stack HCI Update Operator (custom)` custom role (`Microsoft.AzureStackHCI/clusters/updates/apply/action` + cluster-update reads). |
| **Exit conditions** | Pipeline run is green when every in-scope cluster either succeeds or is correctly classified as `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` (these are skips, not failures). Per-cluster update failures surface as JUnit `<failure>` entries; long-running runs are tracked by Step 8. |
| **Exit conditions** | Pipeline run is green when every in-scope cluster either succeeds or is correctly classified as `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` (these are skips, not failures - rendered as JUnit `<skipped>` from v0.8.78). Per-cluster update failures and `HealthCheckBlocked` outcomes surface as JUnit `<failure>` entries; long-running runs are tracked by Step 8. |
| **ITSM** | Supported (opt-in via `raise_itsm_ticket=true` / `raiseItsmTicket=true`). When enabled, one ServiceNow incident is raised per cluster whose update run finished with a failure state; classification skips (`ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag`) do not generate tickets. `itsm_dry_run` builds payloads without creating tickets; `itsm_force_create` bypasses dedupe. |

> **MANDATORY CUSTOMISATION: the Apply Updates pipeline does not ship with a schedule.** The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags **only gate updates *while the pipeline is already running***; they do **not** start the pipeline. If you (a) use `UpdateStartWindow` tags to define when updates may be installed and (b) leave the shipped `apply-updates.yml` with `workflow_dispatch` only (GH) / `trigger: none` (ADO), **no updates will ever be applied automatically** - the pipeline will simply never start during the window.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ env:
# this to the version actually installed and to the latest on PSGallery, and emits a
# ::notice annotation if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
REQUIRED_MODULE_VERSION: ${{ github.event.inputs.module_version || vars.REQUIRED_MODULE_VERSION || '' }}
# v0.8.4 - opt this workflow into Node.js 24 for all JavaScript actions
# (actions/checkout, actions/download-artifact, actions/upload-artifact,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ env:
# this to the version actually installed and to the latest on PSGallery, and emits a
# ::notice annotation if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.77'
GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): manual workflow_dispatch input >
# repository variable 'REQUIRED_MODULE_VERSION' > empty (latest).
Expand Down Expand Up @@ -354,7 +354,10 @@ jobs:
# health, Step 3c schedule, Step 3b1 sideloaded) as defence in depth in
# case state drifts between the two jobs.
- name: Download Readiness Report
uses: actions/download-artifact@v6
# v0.8.78: bumped from @v6 to @v7. v6 still ran on Node.js 20 by default
# and triggered the runner's "Node.js 20 is deprecated" warning on every
# apply-updates run. @v7 runs natively on Node 24 and silences the warning.
uses: actions/download-artifact@v7
with:
# artifact name now carries a UTC timestamp - resolve via the
# upstream job's outputs.readiness_artifact so re-runs on the same day
Expand Down Expand Up @@ -502,6 +505,13 @@ jobs:
UpdateRing = $env:INPUT_UPDATE_RING
TotalCount = "${{ needs.check-readiness.outputs.total_count }}"
ReadyCount = "${{ needs.check-readiness.outputs.ready_count }}"
# v0.8.78: surface the full ring picture (Up to Date + Not Ready) in
# the apply-updates Readiness KPI, not just the clusters that entered
# Apply Updates. Up-to-Date clusters are a healthy steady state;
# Not-Ready clusters were held back by the readiness gate (e.g.
# state=NeedsAttention / UpdateInProgress / UpdateFailed).
UpToDateCount = "${{ needs.check-readiness.outputs.up_to_date_count }}"
NotReadyCount = "${{ needs.check-readiness.outputs.not_ready_count }}"
Succeeded = "${{ steps.apply-updates.outputs.SUCCEEDED }}"
Skipped = "${{ steps.apply-updates.outputs.SKIPPED }}"
Failed = "${{ steps.apply-updates.outputs.FAILED }}"
Expand Down
Loading
Loading