AzLocal.UpdateManagement v0.8.79: Step.07 Force Immediate Update break-glass override

AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md

@@ -64,7 +64,7 @@ By the end of this guide you will have:
   - **Fleet Connectivity Status** (Step.4, v0.7.79+, enhanced in v0.7.85) - read-only daily snapshot of Arc agent connectivity, physical NIC health, Azure Resource Bridge status, and the node-count reconciliation between cluster `reportedProperties.nodes` and Arc-tagged physical machines. *Scheduled daily 05:30 UTC + manual.*
   - **Assess Update Readiness** (Step.5) - pre-flight, report-only readiness + blocking-health snapshot, published as JUnit XML. *Manual only.*
   - **Sideload Updates** (Step.6, v0.8.7) - **opt-in, off by default, on-prem self-hosted runner/agent required.** Pre-stages solution-update media onto clusters that cannot pull updates from Azure directly (dark / air-gapped fabrics): Robocopy to the cluster import share, remote SHA256 verify, `Add-SolutionUpdate` import, then flip `UpdateSideloaded=True` for the downstream Step.7 apply. Re-entrant state machine driven by a frequent CRON; the multi-hour copy runs in a detached Scheduled Task. *Inert unless `SIDELOAD_UPDATES=true`. See [sideload.md](docs/sideload.md) and [sideload-robocopy.md](docs/sideload-robocopy.md).*
-  - **Apply Updates** (Step.7; **v0.8.78 step-summary UX polish**: ``ScheduleBlocked`` / ``SideloadedBlocked`` / ``ExcludedByTag`` outcomes render as JUnit ``<skipped>`` instead of ``<failure>`` so ``dorny/test-reporter`` no longer flips Step.07 RED on by-design gate-respect outcomes - ``HealthCheckBlocked`` deliberately stays a ``<failure>``; the Readiness KPI table now also surfaces **Already Up to Date** and **Not Ready (needs attention before updating)** rows so operators see the full ring breakdown - eligible, started, skipped, blocked, up-to-date, needs-attention - in one place; GHA-only: ``actions/download-artifact@v6 -> @v7`` silences the Node 20 deprecation warning) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
+  - **Apply Updates** (Step.7; **v0.8.79 break-glass override**: optional `force_immediate_update` / `forceImmediateUpdate` pipeline parameter (defaults `false`, **manual run / queue only**) bypasses the per-cluster `UpdateStartWindow` / `UpdateExclusionsWindow` maintenance-window gate for emergency / out-of-window patching - the override is unreachable from the scheduled `apply-updates-schedule.yml` configuration file and a high-visibility WARNING banner is emitted into the run log when it fires; **v0.8.78 step-summary UX polish**: ``ScheduleBlocked`` / ``SideloadedBlocked`` / ``ExcludedByTag`` outcomes render as JUnit ``<skipped>`` instead of ``<failure>`` so ``dorny/test-reporter`` no longer flips Step.07 RED on by-design gate-respect outcomes - ``HealthCheckBlocked`` deliberately stays a ``<failure>``; the Readiness KPI table now also surfaces **Already Up to Date** and **Not Ready (needs attention before updating)** rows so operators see the full ring breakdown - eligible, started, skipped, blocked, up-to-date, needs-attention - in one place; GHA-only: ``actions/download-artifact@v6 -> @v7`` silences the Node 20 deprecation warning) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
   - **Monitor In-Flight Updates** (Step.8, v0.7.90; v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `<details>` Verbose Error block per row, and JUnit `<testsuite time="..">` + `<testcase time="..">` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `monitor-updates.yml` (v0.7.92+).*
   - **Fleet Update Status** (Step.9, formerly Step.8; v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `<testcase>` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.*
   - **Fleet Health Status** (Step.10, formerly Step.9) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.*

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml

@@ -87,7 +87,7 @@ parameters:
     default: false
 
 variables:
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}'
   reportsPath: '$(Build.ArtifactStagingDirectory)/reports'
   # v0.8.7 sideload advisor defaults. Override at the pipeline / variable-group

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml

@@ -85,6 +85,15 @@ parameters:
     type: boolean
     default: true
 
+  # v0.8.79 break-glass override. Only honoured when $(Build.Reason) -eq 'Manual'
+  # (operator-initiated queue from the Azure DevOps UI / CLI). Schedule-triggered
+  # runs deliberately ignore this flag even if a YAML edit ever sets the default
+  # to true - the apply step rechecks $(Build.Reason) at runtime.
+  - name: forceImmediateUpdate
+    displayName: 'WARNING: FORCE IMMEDIATE UPDATE - bypass UpdateStartWindow / UpdateExclusionsWindow tags and apply updates immediately. Break-glass override for emergency / out-of-window patching. MANUAL QUEUE ONLY - schedule-triggered runs ignore this flag.'
+    type: boolean
+    default: false
+
   # --- ITSM Connector (ServiceNow) ---
   # Opt-in ticket creation. Defaults preserve the previous pipeline behaviour
   # (no ITSM calls). Secrets are wired via the variable group referenced below
@@ -120,7 +129,7 @@ variables:
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
   - name: GENERATED_AGAINST_MODULE_VERSION
-    value: '0.8.78'
+    value: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
@@ -366,6 +375,11 @@ stages:
               # clusters with no matching Ready update are skipped with
               # status 'NotInAllowList'.
               RESOLVED_ALLOWED_UPDATE_VERSIONS: $(resolvedAllowedUpdateVersions)
+              # v0.8.79 break-glass override - parameter value + Build.Reason guard.
+              # The runtime check below collapses force=true on Schedule/CI runs to
+              # force=false so the override only ever takes effect on a Manual queue.
+              FORCE_IMMEDIATE_UPDATE_PARAM: ${{ parameters.forceImmediateUpdate }}
+              BUILD_REASON: $(Build.Reason)
             inputs:
               azureSubscription: 'AzureLocal-ServiceConnection'  # Update with your service connection name
               scriptType: 'pscore'
@@ -380,6 +394,14 @@ stages:
                     OutputDirectory       = "$(Build.ArtifactStagingDirectory)"
                 }
                 if ([System.Convert]::ToBoolean("${{ parameters.dryRun }}")) { $params['DryRun'] = $true }
+                if ($env:FORCE_IMMEDIATE_UPDATE_PARAM -eq 'True') {
+                    if ($env:BUILD_REASON -eq 'Manual') {
+                        $params['ForceImmediateUpdate'] = $true
+                    }
+                    else {
+                        Write-Host "##vso[task.logissue type=warning]forceImmediateUpdate=True ignored: Build.Reason='$env:BUILD_REASON' (override is honoured only for Manual queue runs)."
+                    }
+                }
                 Invoke-AzLocalReadinessGatedClusterUpdate @params
 
           # compute a UTC timestamp variable so both update-logs and itsm-results share a unique per-run suffix.

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml

@@ -70,7 +70,7 @@ variables:
   # the version actually installed and to the latest on PSGallery, and emits a warning
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/authentication-test.yml

@@ -60,7 +60,7 @@ variables:
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
   - name: GENERATED_AGAINST_MODULE_VERSION
-    value: '0.8.78'
+    value: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml

@@ -110,7 +110,7 @@ variables:
   # the version actually installed and to the latest on PSGallery, and emits a warning
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml

@@ -105,7 +105,7 @@ variables:
   # the version actually installed and to the latest on PSGallery, and emits a warning
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml

@@ -93,7 +93,7 @@ variables:
   # the version actually installed and to the latest on PSGallery, and emits a warning
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/inventory-clusters.yml

@@ -43,7 +43,7 @@ variables:
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
   - name: GENERATED_AGAINST_MODULE_VERSION
-    value: '0.8.78'
+    value: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml

@@ -46,7 +46,7 @@ variables:
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
   - name: GENERATED_AGAINST_MODULE_VERSION
-    value: '0.8.78'
+    value: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml

@@ -84,7 +84,7 @@ parameters:
     default: ''
 
 variables:
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}'
   reportsPath: '$(Build.ArtifactStagingDirectory)/reports'
 

AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml

@@ -79,7 +79,7 @@ variables:
   # the version actually installed and to the latest on PSGallery, and emits a warning
   # log if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   # Resolution order for the module version pin (leave all unset to install the latest,
   # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
   # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).

AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md

@@ -162,6 +162,8 @@ The table below is the ground truth for what each shipped YAML does **out of the
 
 > **Behaviour change in v0.8.78**: per-cluster `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` outcomes now render as JUnit `<skipped>` instead of `<failure>` so `dorny/test-reporter` no longer flips the Step.07 check RED on by-design gate-respect outcomes - `HealthCheckBlocked` deliberately stays a `<failure>` because a critical health failure IS actionable. The Step 7 Readiness KPI table also gains optional **Already Up to Date** and **Not Ready (needs attention before updating)** rows, wired in both shipped `apply-updates.yml` templates from the upstream `readiness.UpToDateCount` / `readiness.NotReadyCount` outputs (emitted since v0.8.74), so operators see the full ring breakdown - eligible / started / skipped / blocked / up-to-date / needs-attention - in one place. GitHub Actions only: `actions/download-artifact@v6 -> @v7` to silence the Node 20 deprecation warning (v7 chosen over v8 because v8 has a breaking `digest-mismatch=error` default).
 
+> **New in v0.8.79 - operator break-glass override (`Force Immediate Update`)**: optional pipeline parameter on `apply-updates.yml` (`force_immediate_update` GitHub Actions choice input / `forceImmediateUpdate` Azure DevOps boolean parameter) defaulting to `false`. When set to `true` on a **manual run / queue only**, bypasses the per-cluster `UpdateStartWindow` / `UpdateExclusionsWindow` Step 3c maintenance-window gate for emergency / out-of-window patching. Defence in depth: GitHub Actions YAML collapses the flag to `'false'` for any non-`workflow_dispatch` event (`${{ github.event_name == 'workflow_dispatch' && github.event.inputs.force_immediate_update || 'false' }}`); Azure DevOps rechecks `$(Build.Reason) -eq 'Manual'` at runtime before honouring the parameter. A `WARNING:` GUI label is prepended on both pipeline hosts; a high-visibility `::warning::` (GHA) / `##vso[task.logissue type=warning]` (ADO) banner is emitted into the run log when the override fires. The override is unreachable from the `apply-updates-schedule.yml` configuration file (no `forceImmediateUpdate` field exists on `New-AzLocalApplyUpdatesScheduleConfig` / `Resolve-AzLocalPipelineUpdateRing` / `Get-AzLocalApplyUpdatesScheduleAudit`). Other readiness gates (connectivity, health, sideload status, `UpdateExcluded` operator hard-override) continue to apply. Maps to `Invoke-AzLocalReadinessGatedClusterUpdate -ForceImmediateUpdate` -> `Start-AzLocalClusterUpdate -IgnoreScheduleTags` for direct cmdlet use.
+
 | Aspect | Value |
 |---|---|
 | **Purpose** | Apply updates to clusters filtered by `UpdateRing` tag value. The actual mutation step; every other pipeline in this folder is either preparation, monitoring, or reporting. |

AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml

@@ -96,7 +96,7 @@ env:
   # this to the version actually installed and to the latest on PSGallery, and emits a
   # ::notice annotation if the YAML appears stale - prompting you to refresh via
   # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
-  GENERATED_AGAINST_MODULE_VERSION: '0.8.78'
+  GENERATED_AGAINST_MODULE_VERSION: '0.8.79'
   REQUIRED_MODULE_VERSION: ${{ github.event.inputs.module_version || vars.REQUIRED_MODULE_VERSION || '' }}
   # v0.8.4 - opt this workflow into Node.js 24 for all JavaScript actions
   # (actions/checkout, actions/download-artifact, actions/upload-artifact,