diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md index 87e2cba2..ba31940d 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/README.md @@ -56,17 +56,18 @@ It is written in the same step-by-step style as [`ITSM/README.md`](../ITSM/READM By the end of this guide you will have: - A federated identity (no client secrets) wired into your CI/CD platform with the **minimum** Azure RBAC needed for cluster update management. -- Nine working pipelines committed to your repo and visible in the Actions / Pipelines UI: +- Eleven working pipelines committed to your repo and visible in the Actions / Pipelines UI: - **Authentication Validation and Subscription Scope Report** (Step.0) - probe the federated identity end-to-end, emit a JUnit-rendered authentication / scope / Resource Graph report, and capture the subscription set the pipeline identity can read. *Manual only - re-run after every RBAC change.* - **Inventory** (Step.1) - enumerate every Azure Local cluster the identity can see and export a CSV. *Scheduled weekly + manual.* - **Manage UpdateRing tags** (Step.2) - bulk-apply `UpdateRing`, `UpdateStartWindow`, `UpdateExclusionsWindow`, `UpdateExcluded` tags from that CSV. *Manual only.* - **Apply-Updates Schedule Coverage Audit** (Step.3, v0.7.65) - read-only weekly audit that compares the cron(s) in your `apply-updates` pipeline to the `UpdateStartWindow` tags actually present on your clusters and flags any (UpdateRing, UpdateStartWindow) pair that no cron will reach. *Scheduled weekly Mon 05:00 UTC + manual.* - **Fleet Connectivity Status** (Step.4, v0.7.79+, enhanced in v0.7.85) - read-only daily snapshot of Arc agent connectivity, physical NIC health, Azure Resource Bridge status, and the node-count reconciliation between cluster `reportedProperties.nodes` and Arc-tagged physical machines. *Scheduled daily 05:30 UTC + manual.* - **Assess Update Readiness** (Step.5) - pre-flight, report-only readiness + blocking-health snapshot, published as JUnit XML. *Manual only.* - - **Apply Updates** (Step.6) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 6 - Apply Updates](docs/appendix-pipelines.md#step-6---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).* - - **Monitor In-Flight Updates** (Step.7, v0.7.90; v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `
` Verbose Error block per row, and JUnit `` + `` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `Step.7_monitor-updates.yml` (v0.7.92+).* - - **Fleet Update Status** (Step.8, formerly Step.7; v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.* - - **Fleet Health Status** (Step.9, formerly Step.8) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.* + - **Sideload Updates** (Step.6, v0.8.7) - **opt-in, off by default, on-prem self-hosted runner/agent required.** Pre-stages solution-update media onto clusters that cannot pull updates from Azure directly (dark / air-gapped fabrics): Robocopy to the cluster import share, remote SHA256 verify, `Add-SolutionUpdate` import, then flip `UpdateSideloaded=True` for the downstream Step.7 apply. Re-entrant state machine driven by a frequent CRON; the multi-hour copy runs in a detached Scheduled Task. *Inert unless `SIDELOAD_UPDATES=true`. See [sideload.md](docs/sideload.md) and [sideload-robocopy.md](docs/sideload-robocopy.md).* + - **Apply Updates** (Step.7) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).* + - **Monitor In-Flight Updates** (Step.8, v0.7.90; v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `
` Verbose Error block per row, and JUnit `` + `` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `monitor-updates.yml` (v0.7.92+).* + - **Fleet Update Status** (Step.9, formerly Step.8; v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.* + - **Fleet Health Status** (Step.10, formerly Step.9) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.* - An end-to-end "ring-based" rollout pattern: Pilot -> Wave2 -> Production, with each ring gated on the previous wave's success. - **Optional**: a ServiceNow integration that opens deduped incidents for clusters whose run status indicates the module's own retries cannot recover (failures, blocking health checks, sideloaded payload missing) - see [section 7](#7-optional-open-itsm-tickets-for-clusters-needing-operator-action). @@ -74,24 +75,25 @@ The pipelines are **fully opt-in additive layers** over the module. The PowerShe ### 1.1 Why the pipelines are named `Step.N - ` -The ten YAMLs ship with a `Step.N_` filename prefix **and** a matching `Step.N - ` value in each workflow's `name:` field (GitHub Actions) / header title (Azure DevOps): +The eleven YAMLs ship with a de-numbered filename (e.g. `apply-updates.yml`) **and** a matching `Step.N - ` value in each workflow's `name:` field (GitHub Actions) / header title (Azure DevOps): Each row's name links to the matching `Step N -` section in [`docs/appendix-pipelines.md`](docs/appendix-pipelines.md), which documents what every pipeline does, the cmdlets it calls, its inputs / outputs / dependencies, the RBAC and exit conditions, and the emitted artefacts. The **GH Actions** and **Azure DevOps** columns link directly to the source YAML for each platform. | Step | File / Workflow name | GH Actions | Azure DevOps | |---:|---|---|---| -| 0 | [Step.0 - Authentication Validation and Subscription Scope Report](docs/appendix-pipelines.md#step-0---authentication-validation-and-subscription-scope-report) | [`Step.0_authentication-test.yml`](github-actions/Step.0_authentication-test.yml) | [`Step.0_authentication-test.yml`](azure-devops/Step.0_authentication-test.yml) | -| 1 | [Step.1 - Inventory Azure Local Clusters](docs/appendix-pipelines.md#step-1---inventory-clusters) | [`Step.1_inventory-clusters.yml`](github-actions/Step.1_inventory-clusters.yml) | [`Step.1_inventory-clusters.yml`](azure-devops/Step.1_inventory-clusters.yml) | -| 2 | [Step.2 - Manage UpdateRing Tags](docs/appendix-pipelines.md#step-2---manage-updatering-tags) | [`Step.2_manage-updatering-tags.yml`](github-actions/Step.2_manage-updatering-tags.yml) | [`Step.2_manage-updatering-tags.yml`](azure-devops/Step.2_manage-updatering-tags.yml) | -| 3 | [Step.3 - Apply-Updates Schedule Coverage Audit](docs/appendix-pipelines.md#step-3---apply-updates-schedule-coverage-audit) | [`Step.3_apply-updates-schedule-audit.yml`](github-actions/Step.3_apply-updates-schedule-audit.yml) | [`Step.3_apply-updates-schedule-audit.yml`](azure-devops/Step.3_apply-updates-schedule-audit.yml) | -| 4 | [Step.4 - Fleet Connectivity Status](docs/appendix-pipelines.md#step-4---fleet-connectivity-status) | [`Step.4_fleet-connectivity-status.yml`](github-actions/Step.4_fleet-connectivity-status.yml) | [`Step.4_fleet-connectivity-status.yml`](azure-devops/Step.4_fleet-connectivity-status.yml) | -| 5 | [Step.5 - Assess Update Readiness](docs/appendix-pipelines.md#step-5---assess-update-readiness) | [`Step.5_assess-update-readiness.yml`](github-actions/Step.5_assess-update-readiness.yml) | [`Step.5_assess-update-readiness.yml`](azure-devops/Step.5_assess-update-readiness.yml) | -| 6 | [Step.6 - Apply Updates](docs/appendix-pipelines.md#step-6---apply-updates) | [`Step.6_apply-updates.yml`](github-actions/Step.6_apply-updates.yml) | [`Step.6_apply-updates.yml`](azure-devops/Step.6_apply-updates.yml) | -| 7 | [Step.7 - Monitor In-Flight Updates](docs/appendix-pipelines.md#step-7---monitor-in-flight-updates) | [`Step.7_monitor-updates.yml`](github-actions/Step.7_monitor-updates.yml) | [`Step.7_monitor-updates.yml`](azure-devops/Step.7_monitor-updates.yml) | -| 8 | [Step.8 - Fleet Update Status](docs/appendix-pipelines.md#step-8---fleet-update-status) | [`Step.8_fleet-update-status.yml`](github-actions/Step.8_fleet-update-status.yml) | [`Step.8_fleet-update-status.yml`](azure-devops/Step.8_fleet-update-status.yml) | -| 9 | [Step.9 - Fleet Health Status](docs/appendix-pipelines.md#step-9---fleet-health-status) | [`Step.9_fleet-health-status.yml`](github-actions/Step.9_fleet-health-status.yml) | [`Step.9_fleet-health-status.yml`](azure-devops/Step.9_fleet-health-status.yml) | - -- **GitHub Actions**: the Actions sidebar sorts workflows alphabetically by the `name:` field inside the YAML. Because every `name:` starts with `Step.N - `, the sidebar lists the ten workflows in execution order (Step.0 first, Step.9 last) instead of the cosmetically confusing alphabetical scatter (`Apply Updates`, `Apply-Updates Schedule Coverage Audit`, `Assess Update Readiness`, ...). +| 0 | [Step.0 - Authentication Validation and Subscription Scope Report](docs/appendix-pipelines.md#step-0---authentication-validation-and-subscription-scope-report) | [`authentication-test.yml`](github-actions/authentication-test.yml) | [`authentication-test.yml`](azure-devops/authentication-test.yml) | +| 1 | [Step.1 - Inventory Azure Local Clusters](docs/appendix-pipelines.md#step-1---inventory-clusters) | [`inventory-clusters.yml`](github-actions/inventory-clusters.yml) | [`inventory-clusters.yml`](azure-devops/inventory-clusters.yml) | +| 2 | [Step.2 - Manage UpdateRing Tags](docs/appendix-pipelines.md#step-2---manage-updatering-tags) | [`manage-updatering-tags.yml`](github-actions/manage-updatering-tags.yml) | [`manage-updatering-tags.yml`](azure-devops/manage-updatering-tags.yml) | +| 3 | [Step.3 - Apply-Updates Schedule Coverage Audit](docs/appendix-pipelines.md#step-3---apply-updates-schedule-coverage-audit) | [`apply-updates-schedule-audit.yml`](github-actions/apply-updates-schedule-audit.yml) | [`apply-updates-schedule-audit.yml`](azure-devops/apply-updates-schedule-audit.yml) | +| 4 | [Step.4 - Fleet Connectivity Status](docs/appendix-pipelines.md#step-4---fleet-connectivity-status) | [`fleet-connectivity-status.yml`](github-actions/fleet-connectivity-status.yml) | [`fleet-connectivity-status.yml`](azure-devops/fleet-connectivity-status.yml) | +| 5 | [Step.5 - Assess Update Readiness](docs/appendix-pipelines.md#step-5---assess-update-readiness) | [`assess-update-readiness.yml`](github-actions/assess-update-readiness.yml) | [`assess-update-readiness.yml`](azure-devops/assess-update-readiness.yml) | +| 6 | [Step.6 - Sideload Updates (opt-in, on-prem)](docs/appendix-pipelines.md#step-6---sideload-updates) | [`sideload-updates.yml`](github-actions/sideload-updates.yml) | [`sideload-updates.yml`](azure-devops/sideload-updates.yml) | +| 7 | [Step.7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) | [`apply-updates.yml`](github-actions/apply-updates.yml) | [`apply-updates.yml`](azure-devops/apply-updates.yml) | +| 8 | [Step.8 - Monitor In-Flight Updates](docs/appendix-pipelines.md#step-8---monitor-in-flight-updates) | [`monitor-updates.yml`](github-actions/monitor-updates.yml) | [`monitor-updates.yml`](azure-devops/monitor-updates.yml) | +| 9 | [Step.9 - Fleet Update Status](docs/appendix-pipelines.md#step-9---fleet-update-status) | [`fleet-update-status.yml`](github-actions/fleet-update-status.yml) | [`fleet-update-status.yml`](azure-devops/fleet-update-status.yml) | +| 10 | [Step.10 - Fleet Health Status](docs/appendix-pipelines.md#step-10---fleet-health-status) | [`fleet-health-status.yml`](github-actions/fleet-health-status.yml) | [`fleet-health-status.yml`](azure-devops/fleet-health-status.yml) | + +- **GitHub Actions**: the Actions sidebar sorts workflows alphabetically by the `name:` field inside the YAML. Because every `name:` starts with `Step.N - `, the sidebar lists the eleven workflows in execution order (Step.0 first, Step.10 last) instead of the cosmetically confusing alphabetical scatter (`Apply Updates`, `Apply-Updates Schedule Coverage Audit`, `Assess Update Readiness`, ...). - **Azure DevOps**: the Pipelines list sorts by the pipeline **definition name** chosen at *import time* (not by the YAML filename and not by any top-level `name:` field - the `name:` field in an ADO YAML controls the per-run *build number*, not the pipeline display name). When you import each YAML, the import wizard prefills the suggested pipeline name from the YAML's leading title comment; the YAMLs in this repo open with `# Step.N - `, so the suggested name is already correct. **Accept the suggested name** (or paste `Step.N - ` yourself), and the Pipelines list will sort in execution order. You can rename a pipeline later via *Pipeline -> Edit -> Settings -> Name*. If you prefer a different naming scheme (e.g. `00 - Auth`, `01 - Inventory`, ...), just change the `name:` field in each GH Actions YAML and / or pick a different prefix at ADO import time. Nothing else in the module depends on these display names. @@ -942,7 +944,7 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp 1. **Run the Authentication Validation and Subscription Scope Report first (strongly recommended).** Before exercising all seven workflows, validate that the App Registration, federated credentials, GitHub secrets, environments, and RBAC role assignment all line up - and capture the count + per-subscription detail of subscriptions visible to the pipeline identity - by running **`Step.0 - Authentication Validation and Subscription Scope Report`**. This narrows any failure to *one* small YAML file instead of debugging seven interacting workflows simultaneously. **Re-run periodically** (recommended monthly, or after any RBAC change in the tenant) to confirm the pipeline identity's subscription scope has not silently widened or narrowed - drift here is the earliest signal that downstream fleet reports are about to under- or over-count clusters. - The validation workflow ships with the module at [`github-actions/Step.0_authentication-test.yml`](./github-actions/Step.0_authentication-test.yml). v0.7.70 emits a **JUnit XML** report (Authentication / Subscription Scope / Resource Graph Reachability) rendered in the run's Checks UI via `dorny/test-reporter`, a **markdown summary** with the subscription count + subscription detail table written to `$GITHUB_STEP_SUMMARY`, and a `auth-report` artifact (XML + `subscriptions.json` + `subscriptions.csv`) for ITSM / dashboard ingest. + The validation workflow ships with the module at [`github-actions/authentication-test.yml`](./github-actions/authentication-test.yml). v0.7.70 emits a **JUnit XML** report (Authentication / Subscription Scope / Resource Graph Reachability) rendered in the run's Checks UI via `dorny/test-reporter`, a **markdown summary** with the subscription count + subscription detail table written to `$GITHUB_STEP_SUMMARY`, and a `auth-report` artifact (XML + `subscriptions.json` + `subscriptions.csv`) for ITSM / dashboard ingest. - **If you used the `Copy-AzLocalPipelineExample` shortcut above**: the file is already in `.github/workflows/` alongside the other seven workflow YAMLs - those will ride along on the commit but stay dormant until you trigger them manually (`gh workflow run`) or their schedules fire. Commit and push to the default branch, then run the trigger commands below. - **If you didn't use the shortcut**: copy just that one file into your repo's `.github/workflows/`, commit, and push to the default branch. @@ -951,10 +953,10 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp ```powershell # 1. Branch-scoped run - exercises the 'GitHubActions-main' federated credential. - gh workflow run Step.0_authentication-test.yml --repo $repo + gh workflow run authentication-test.yml --repo $repo # 2. Environment-scoped run - exercises the 'GitHubActions-DevTest' federated credential. - gh workflow run Step.0_authentication-test.yml --repo $repo -f environment=DevTest + gh workflow run authentication-test.yml --repo $repo -f environment=DevTest # Watch the most recent run live (Ctrl+C to stop watching, run continues). gh run watch --repo $repo @@ -1013,33 +1015,33 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp | `Error: Could not fetch access token for Azure` (no AADSTS code) | The workflow lacks `permissions: id-token: write` or the secrets are missing/misspelt. | Confirm the `permissions:` block is present and run `gh secret list --repo $repo` shows all three `AZURE_*` secrets. | | Environment-scoped run hangs in **Waiting for review** | The environment has required-reviewers protection (good!) and is waiting for you to approve. | Approve in the **Actions** tab, or remove required reviewers from the validation run via the environment settings. | - Once the run is green, leave `Step.0_authentication-test.yml` in place and schedule yourself to re-run it monthly (or whenever you change RBAC / federated credentials / subscription assignments). If you used the `Copy-AzLocalPipelineExample` shortcut, the other nine workflows are already on the default branch - skip to step 4 to run them. Otherwise, proceed to step 2 to copy the remaining workflow files. + Once the run is green, leave `authentication-test.yml` in place and schedule yourself to re-run it monthly (or whenever you change RBAC / federated credentials / subscription assignments). If you used the `Copy-AzLocalPipelineExample` shortcut, the other nine workflows are already on the default branch - skip to step 4 to run them. Otherwise, proceed to step 2 to copy the remaining workflow files. 2. Copy every file from [`github-actions/`](./github-actions/) into `.github/workflows/` in your repo: ```text .github/ workflows/ - Step.1_inventory-clusters.yml - Step.2_manage-updatering-tags.yml - Step.3_apply-updates-schedule-audit.yml - Step.4_fleet-connectivity-status.yml - Step.5_assess-update-readiness.yml - Step.6_apply-updates.yml - Step.7_monitor-updates.yml - Step.8_fleet-update-status.yml - Step.9_fleet-health-status.yml + inventory-clusters.yml + manage-updatering-tags.yml + apply-updates-schedule-audit.yml + fleet-connectivity-status.yml + assess-update-readiness.yml + apply-updates.yml + monitor-updates.yml + fleet-update-status.yml + fleet-health-status.yml ``` 3. Commit and push. The workflows appear in the **Actions** tab. -4. Each workflow exposes its inputs via the **Run workflow** button (workflow_dispatch). The scheduled triggers (e.g. `Step.4_fleet-connectivity-status.yml` runs daily at 05:30 UTC, `Step.7_monitor-updates.yml` runs 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window, v0.7.92+ default - edit the cron in the file if your maintenance window differs), `Step.8_fleet-update-status.yml` runs daily at 06:00 UTC, `Step.9_fleet-health-status.yml` runs daily at 07:00 UTC, `Step.3_apply-updates-schedule-audit.yml` runs weekly on Mondays at 05:00 UTC) activate automatically once the file is on the default branch. -5. **Replace the starter `apply-updates-schedule.yml` with one generated from your live fleet** (required for **scheduled** Step.6 / Step.3 runs; manual `workflow_dispatch` runs of Step.6 work without it because they use the `-UpdateRingValue` input verbatim). v0.7.92+ `Copy-AzLocalPipelineExample -Platform GitHub` drops a **starter** `apply-updates-schedule.yml` alongside `.github\workflows\` (i.e. at `.github\apply-updates-schedule.yml`) by default. The starter is a verbatim copy of [`apply-updates-schedule.example.yml`](./apply-updates-schedule.example.yml) and ships with **DEMO** ring names (`Canary`, `DevTest`, `Ring1`, `Ring2`, `Prod`) that almost never match a real estate's `UpdateRing` tag values - regenerate from your live fleet, overwriting the demo file: +4. Each workflow exposes its inputs via the **Run workflow** button (workflow_dispatch). The scheduled triggers (e.g. `fleet-connectivity-status.yml` runs daily at 05:30 UTC, `monitor-updates.yml` runs 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window, v0.7.92+ default - edit the cron in the file if your maintenance window differs), `fleet-update-status.yml` runs daily at 06:00 UTC, `fleet-health-status.yml` runs daily at 07:00 UTC, `apply-updates-schedule-audit.yml` runs weekly on Mondays at 05:00 UTC) activate automatically once the file is on the default branch. +5. **Replace the starter `apply-updates-schedule.yml` with one generated from your live fleet** (required for **scheduled** Step.7 / Step.3 runs; manual `workflow_dispatch` runs of Step.7 work without it because they use the `-UpdateRingValue` input verbatim). v0.8.7+ `Copy-AzLocalPipelineExample -Platform GitHub` drops a **starter** `apply-updates-schedule.yml` into a **`config\` folder at the repo root** (i.e. at `config\apply-updates-schedule.yml`, alongside `config\ClusterUpdateRings.csv`) by default. The starter is a verbatim copy of [`apply-updates-schedule.example.yml`](./apply-updates-schedule.example.yml) and ships with **DEMO** ring names (`Canary`, `DevTest`, `Ring1`, `Ring2`, `Prod`) that almost never match a real estate's `UpdateRing` tag values - regenerate from your live fleet, overwriting the demo file: ```powershell # Regenerate from your fleet's live UpdateRing tag values, overwriting the starter. # -Force is required because the starter file already exists at that path. - New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\.github\apply-updates-schedule.yml -Force + New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\config\apply-updates-schedule.yml -Force ``` - Review the regenerated file, uncomment / edit the rows you want active, then `git add .\.github\apply-updates-schedule.yml ; git commit ; git push`. The starter is safe to leave in place until then - the bundled Step.6 ships with every `cron:` line commented out inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers, so it cannot fire on a `schedule:` trigger until you explicitly add a cron entry. Manual `workflow_dispatch` runs of Step.6 ignore the schedule file entirely (they take `-UpdateRingValue` verbatim from the run-form input). + Review the regenerated file, uncomment / edit the rows you want active, then `git add .\config\apply-updates-schedule.yml ; git commit ; git push`. The starter is safe to leave in place until then - the bundled Step.7 ships with every `cron:` line commented out inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers, so it cannot fire on a `schedule:` trigger until you explicitly add a cron entry. Manual `workflow_dispatch` runs of Step.7 ignore the schedule file entirely (they take `-UpdateRingValue` verbatim from the run-form input). **Safety rails**: `Copy-AzLocalPipelineExample` **never overwrites** an existing `apply-updates-schedule.yml` - any operator-tailored schedule you commit is preserved across re-runs. Pass `-SkipStarterSchedule` to suppress the starter copy entirely (e.g. if you pre-stage the schedule via separate tooling). For the full schema, multi-stage rollouts, weekly-cycle / ring-eligibility model, and the `allowedUpdateVersions` allow-list (schema v2), see [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods). @@ -1047,11 +1049,11 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp 1. **Run the Authentication Validation and Subscription Scope Report first (strongly recommended).** Before importing all eight remaining pipelines, validate that the service connection (Workload Identity Federation), App Registration, and RBAC role assignment all line up - and capture the count + per-subscription detail of subscriptions visible to the pipeline identity - by running **`Step.0 - Authentication Validation and Subscription Scope Report`**. This narrows any failure to *one* small YAML file instead of debugging eight interacting pipelines simultaneously. **Re-run periodically** (recommended monthly, or after any RBAC change in the tenant) to confirm the pipeline identity's subscription scope has not silently widened or narrowed - drift here is the earliest signal that downstream fleet reports are about to under- or over-count clusters. - The validation pipeline ships with the module at [`azure-devops/Step.0_authentication-test.yml`](./azure-devops/Step.0_authentication-test.yml). v0.7.70 emits a **JUnit XML** report (Authentication / Subscription Scope / Resource Graph Reachability) published via `PublishTestResults@2` and rendered in the run's **Tests** tab, a **markdown summary** with the subscription count + subscription detail table uploaded to the run's **Summary** tab via `##vso[task.uploadsummary]`, and a `auth-report` pipeline artifact (XML + `subscriptions.json` + `subscriptions.csv`) for ITSM / dashboard ingest. + The validation pipeline ships with the module at [`azure-devops/authentication-test.yml`](./azure-devops/authentication-test.yml). v0.7.70 emits a **JUnit XML** report (Authentication / Subscription Scope / Resource Graph Reachability) published via `PublishTestResults@2` and rendered in the run's **Tests** tab, a **markdown summary** with the subscription count + subscription detail table uploaded to the run's **Summary** tab via `##vso[task.uploadsummary]`, and a `auth-report` pipeline artifact (XML + `subscriptions.json` + `subscriptions.csv`) for ITSM / dashboard ingest. If you used the `Copy-AzLocalPipelineExample` shortcut above, the file is already in your chosen pipelines folder alongside the other eight pipeline YAMLs - those YAMLs sit dormant until you import each one as a pipeline, so they're harmless at rest. Otherwise, copy just that one file into your repo. Either way, import it as a new pipeline: - - **Pipelines -> New pipeline -> Azure Repos Git -> your repo -> Existing Azure Pipelines YAML file -> `/azure-devops/Step.0_authentication-test.yml`**. + - **Pipelines -> New pipeline -> Azure Repos Git -> your repo -> Existing Azure Pipelines YAML file -> `/azure-devops/authentication-test.yml`**. - **Save and run**. If your service connection has a name other than `AzureLocal-ServiceConnection`, edit `azureSubscription:` in the YAML first (the only configurable line). Unlike GitHub Actions, ADO does **not** have environment-scoped federated credentials at the auth layer - the service connection itself is the federation, so a single pipeline run validates everything. ADO **Environments** (Pipelines -> Environments) are approval gates only, layered on top of the service connection, and are not exercised by this validation pipeline. @@ -1088,15 +1090,15 @@ Both platforms expect the YAML files inside this folder to land in a platform-sp 3. **Pipelines -> New pipeline -> Azure Repos Git -> your repo -> Existing Azure Pipelines YAML file**, then point at the path of each file. Repeat for all eight. 4. After the pipeline is created, click **Save** (not **Run**) until you are ready to execute. 5. Each pipeline references a service connection named `AzureLocal-ServiceConnection`. Either name your service connection to match, or change `azureSubscription:` in each YAML. -6. **Replace the starter `apply-updates-schedule.yml` with one generated from your live fleet** (required for **scheduled** Step.6 / Step.3 runs; manual queue runs of Step.6 work without it because they use the `updateRing` parameter verbatim). v0.7.92+ `Copy-AzLocalPipelineExample -Platform AzureDevOps -Destination .\pipelines\` drops a **starter** `apply-updates-schedule.yml` **one level up** from the pipelines folder (i.e. at `.\apply-updates-schedule.yml` at repo root) by default - this separates the config file from the folder that holds the pipeline YAMLs. The starter is a verbatim copy of [`apply-updates-schedule.example.yml`](./apply-updates-schedule.example.yml) and ships with **DEMO** ring names (`Canary`, `DevTest`, `Ring1`, `Ring2`, `Prod`) that almost never match a real estate's `UpdateRing` tag values - regenerate from your live fleet, overwriting the demo file: +6. **Replace the starter `apply-updates-schedule.yml` with one generated from your live fleet** (required for **scheduled** Step.7 / Step.3 runs; manual queue runs of Step.7 work without it because they use the `updateRing` parameter verbatim). v0.8.7+ `Copy-AzLocalPipelineExample -Platform AzureDevOps -Destination .\pipelines\` drops a **starter** `apply-updates-schedule.yml` into a **`config\` folder at the repo root** (i.e. at `config\apply-updates-schedule.yml`, alongside `config\ClusterUpdateRings.csv`) by default - the same `config\` folder used on GitHub, so the path is identical on both platforms. The starter is a verbatim copy of [`apply-updates-schedule.example.yml`](./apply-updates-schedule.example.yml) and ships with **DEMO** ring names (`Canary`, `DevTest`, `Ring1`, `Ring2`, `Prod`) that almost never match a real estate's `UpdateRing` tag values - regenerate from your live fleet, overwriting the demo file: ```powershell # Regenerate from your fleet's live UpdateRing tag values, overwriting the starter. # -Force is required because the starter file already exists at that path. - New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\apply-updates-schedule.yml -Force + New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\config\apply-updates-schedule.yml -Force ``` - Review the regenerated file, uncomment / edit the rows you want active, then commit and push. Step.6 (ADO) reads the file at `APPLY_UPDATES_SCHEDULE_PATH` (default `./apply-updates-schedule.yml` at repo root, which matches the default starter location) - override the pipeline variable if you keep the schedule elsewhere. The starter is safe to leave in place until then - the bundled Step.6 ships with every `cron:` line commented out inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers, so it cannot fire on a scheduled trigger until you explicitly add a cron entry. Manual queue runs of Step.6 ignore the schedule file entirely (they take `updateRing` verbatim from the run-form input). + Review the regenerated file, uncomment / edit the rows you want active, then commit and push. Step.7 (ADO) reads the file at `APPLY_UPDATES_SCHEDULE_PATH` (default `./config/apply-updates-schedule.yml`, which matches the default starter location) - override the pipeline variable if you keep the schedule elsewhere. The starter is safe to leave in place until then - the bundled Step.7 ships with every `cron:` line commented out inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers, so it cannot fire on a scheduled trigger until you explicitly add a cron entry. Manual queue runs of Step.7 ignore the schedule file entirely (they take `updateRing` verbatim from the run-form input). **Safety rails**: `Copy-AzLocalPipelineExample` **never overwrites** an existing `apply-updates-schedule.yml` - any operator-tailored schedule you commit is preserved across re-runs. Pass `-SkipStarterSchedule` to suppress the starter copy entirely (e.g. if you pre-stage the schedule via separate tooling). For the full schema, multi-stage rollouts, weekly-cycle / ring-eligibility model, and the `allowedUpdateVersions` allow-list (schema v2), see [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods). @@ -1121,7 +1123,7 @@ If your change-control process requires you to pin the module version (so a rele gh variable set REQUIRED_MODULE_VERSION --body '0.7.60' --repo / # Override for a single manual run, leaving the estate-wide pin untouched: -gh workflow run Step.8_fleet-update-status.yml -f module_version=0.7.60 +gh workflow run fleet-update-status.yml -f module_version=0.7.60 # Clear the estate-wide pin to return to latest: gh variable delete REQUIRED_MODULE_VERSION --repo / @@ -1185,44 +1187,44 @@ This is the canonical "nothing wired -> staged rollout working" sequence. Follow ```text +-----------------------------------------------------------------------+ | PHASE 1: INVENTORY | -| 6.1 Step.1_inventory-clusters.yml -> cluster-inventory.csv | +| 6.1 inventory-clusters.yml -> cluster-inventory.csv | +-----------------------------------------------------------------------+ v +-----------------------------------------------------------------------+ | PHASE 2: TAG | | 6.2 Edit the CSV (UpdateRing, UpdateStartWindow, | | UpdateExclusionsWindow, UpdateExcluded) | -| 6.3 Step.2_manage-updatering-tags.yml | +| 6.3 manage-updatering-tags.yml | +-----------------------------------------------------------------------+ v +-----------------------------------------------------------------------+ | PHASE 3: ROLLOUT | -| 6.4 Step.5_assess-update-readiness.yml (report-only pre-flight) | -| 6.5 Step.6_apply-updates.yml Wave1 -> validate -> Wave2 -> Production | +| 6.4 assess-update-readiness.yml (report-only pre-flight) | +| 6.5 apply-updates.yml Wave1 -> validate -> Wave2 -> Production | +-----------------------------------------------------------------------+ v +-----------------------------------------------------------------------+ | PHASE 4: STEADY STATE | -| 6.6 Step.4_fleet-connectivity-status.yml (scheduled, daily 05:30 UTC) | +| 6.6 fleet-connectivity-status.yml (scheduled, daily 05:30 UTC) | | - v0.7.79+, reconciliation enhanced in v0.7.85 | | - "Are Arc agents Connected, NICs healthy, Resource Bridges | | reachable, and does the cluster's node count reconcile | | with Arc-tagged physical machines?" | -| 6.6 Step.7_monitor-updates.yml (5x/day 20-04 UTC every 2h, manual too) | +| 6.6 monitor-updates.yml (5x/day 20-04 UTC every 2h, manual too) | | - v0.7.90 | | - "What is happening right now? Which clusters are mid-update, | | which step are they on, and is anything stuck?" In-flight | | monitor; flags runs over a configurable threshold (default | | 6h) as JUnit failures. | -| 6.6 Step.8_fleet-update-status.yml (scheduled, daily 06:00 UTC) | +| 6.6 fleet-update-status.yml (scheduled, daily 06:00 UTC) | | - "Is each cluster up-to-date?" | -| 6.6 Step.9_fleet-health-status.yml (scheduled, daily 07:00 UTC) - v0.7.65 | +| 6.6 fleet-health-status.yml (scheduled, daily 07:00 UTC) - v0.7.65 | | - "Do clusters have actionable health issues even when | | up-to-date?" Surfaces 24-hour system health-check failures. | -| 6.7 Step.3_apply-updates-schedule-audit.yml (scheduled, weekly Mon 05:00 | +| 6.7 apply-updates-schedule-audit.yml (scheduled, weekly Mon 05:00 | | UTC) - v0.7.65 | | - "Will any tagged UpdateStartWindow never be reached by the cron | -| schedule in Step.6_apply-updates.yml?" Read-only drift advisor. | +| schedule in apply-updates.yml?" Read-only drift advisor. | +-----------------------------------------------------------------------+ ``` @@ -1232,7 +1234,7 @@ Every pipeline emits one or more artifacts (CSV / Markdown / JUnit XML / HTML). ```text +-------------------------------+ - | Step.1_inventory-clusters.yml | + | inventory-clusters.yml | | (read-only ARG) | +-------------------------------+ | @@ -1251,7 +1253,7 @@ Every pipeline emits one or more artifacts (CSV / Markdown / JUnit XML / HTML). | as run artifact for audit) | +-------------------------------+ - | Step.2_manage-updatering-tags.yml | + | manage-updatering-tags.yml | | (writes Microsoft.Resources/ | | tags - DryRun first) | +-------------------------------+ @@ -1259,7 +1261,7 @@ Every pipeline emits one or more artifacts (CSV / Markdown / JUnit XML / HTML). v (cluster tags now committed) | +-------------------------------+ - | Step.5_assess-update-readiness.yml | + | assess-update-readiness.yml | | (read-only; per-ring | | gating evaluation) | +-------------------------------+ @@ -1269,7 +1271,7 @@ Every pipeline emits one or more artifacts (CSV / Markdown / JUnit XML / HTML). | BlockingReasons, HealthState, ...) | +-------------------------------+ - | Step.6_apply-updates.yml | + | apply-updates.yml | | in: cluster-readiness.csv | | (consumes ClusterResourceId | | filtered to ReadyForUpdate) | @@ -1283,10 +1285,10 @@ Every pipeline emits one or more artifacts (CSV / Markdown / JUnit XML / HTML). | | | | v v v v +-------------------------+ +---------------------------+ +-------------------------------+ +---------------------------+ - | Step.8_fleet-update-status.yml | | Step.9_fleet-health-status.yml | | apply-updates-schedule-audit | | (Optional) ITSM forwarder | + | fleet-update-status.yml | | fleet-health-status.yml | | apply-updates-schedule-audit | | (Optional) ITSM forwarder | | daily 06:00 UTC | | daily 07:00 UTC | | .yml | | (section 7) | | out: fleet-update- | | out: fleet-health- | | weekly Mon 05:00 UTC | | consumes: | - | status.csv | | failures.csv | | in: Step.6_apply-updates.yml | | - apply-updates- | + | status.csv | | failures.csv | | in: apply-updates.yml | | - apply-updates- | | fleet-update- | | fleet-health- | | cron entries | | results.csv | | status.html | | summary.html | | out: schedule-coverage- | | - fleet-health- | +-------------------------+ +---------------------------+ | audit.csv | | failures.csv | @@ -1302,7 +1304,7 @@ Key handoffs to remember: - **`cluster-inventory.csv`** is the only artifact the operator edits by hand. Everything downstream is machine-generated. - **`cluster-readiness.csv`** carries `ClusterResourceId` from Assess into Apply. Apply does not re-query ARG to pick targets - it consumes the ID column directly, so a stale or malformed readiness CSV silently produces zero ready clusters. Always treat the most recent readiness run as the source of truth for the next Apply. - **`apply-updates-results.xml`** (JUnit) is what surfaces in the Tests tab on GH Actions and Azure DevOps. Failed-first ordering means actionable rows appear at the top of the reporter UI. -- **`schedule-coverage-recommend.md`** is the only artifact intended to be pasted by hand - directly back into `Step.6_apply-updates.yml`'s `on.schedule` / ADO trigger block when the audit reports `Uncovered` or `PartiallyCovered` rows. +- **`schedule-coverage-recommend.md`** is the only artifact intended to be pasted by hand - directly back into `apply-updates.yml`'s `on.schedule` / ADO trigger block when the audit reports `Uncovered` or `PartiallyCovered` rows. - **Step.4 Fleet Connectivity Status runs parallel to (not downstream of) the apply-updates artifact chain.** It reads ARG directly and emits its own per-scope CSVs (`fleet-cluster-connectivity.csv`, `fleet-arc-status-summary.csv`, `fleet-arc-non-connected-machines.csv`, `fleet-physical-nics.csv`, `fleet-physical-nic-stats.csv`, `fleet-arb-status.csv`) plus a JUnit XML (`fleet-connectivity-status.xml`). No dependency on `cluster-readiness.csv` or `cluster-inventory.csv` - it is the upstream "can we see the fleet at all?" probe. Use it to triage why the apply-updates chain is empty or under-counting. ### 6.1 Inventory the estate @@ -1316,7 +1318,7 @@ Download `cluster-inventory.csv` from the run artifacts. It contains `Subscripti **What a successful inventory run looks like.** The `Run Cluster Inventory` step prints the discovery summary, the absolute path of the exported CSV under the run artifacts, the `UpdateRing` tag distribution across all clusters, and a "Next Steps" block that points at `Set-AzLocalClusterUpdateRingTag` for the next workflow: -![Step.1_inventory-clusters.yml run: Run Cluster Inventory step expanded, showing Inventory Summary with Total Clusters 20 / Clusters with UpdateRing tag 19 / 1 cluster without UpdateRing tag (warning), UpdateRing Distribution Canary=3 / Prod=9 / Ring1=3 / Ring2=4, CSV export path under the artifacts folder, and the Next Steps block guiding the operator to populate the UpdateRing column and then run Set-AzLocalClusterUpdateRingTag](../docs/images/inventory-clusters-run-output.png) +![inventory-clusters.yml run: Run Cluster Inventory step expanded, showing Inventory Summary with Total Clusters 20 / Clusters with UpdateRing tag 19 / 1 cluster without UpdateRing tag (warning), UpdateRing Distribution Canary=3 / Prod=9 / Ring1=3 / Ring2=4, CSV export path under the artifacts folder, and the Next Steps block guiding the operator to populate the UpdateRing column and then run Set-AzLocalClusterUpdateRingTag](../docs/images/inventory-clusters-run-output.png) > If you would rather skip the inventory pipeline entirely, the same operation runs from a local PowerShell session: `Import-Module ./AzLocal.UpdateManagement.psd1; Get-AzLocalClusterInventory -ExportPath ./cluster-inventory.csv`. This is the same code path the pipeline uses. @@ -1387,27 +1389,24 @@ If you do want a hard go / no-go gate (typical for first production wave), have ### 6.5 Generate `apply-updates-schedule.yml` from your live fleet -`Copy-AzLocalPipelineExample` (run earlier when you wired the pipelines into your repo - see [section 5](#5-wire-the-pipeline-files-into-your-repo)) dropped a **starter** `apply-updates-schedule.yml` next to your pipelines on first copy - GitHub Actions: `.github\apply-updates-schedule.yml`, Azure DevOps: `.\apply-updates-schedule.yml` at repo root. That starter ships with **DEMO** ring names (`Canary`, `DevTest`, `Ring1`, `Ring2`, `Prod`) that almost never match a real estate's `UpdateRing` tag values. Now that section 6.3 has tagged your clusters, regenerate the file so the rings, windows, and `allowedUpdateVersions` reflect the live fleet: +`Copy-AzLocalPipelineExample` (run earlier when you wired the pipelines into your repo - see [section 5](#5-wire-the-pipeline-files-into-your-repo)) dropped a **starter** `apply-updates-schedule.yml` into a **`config\` folder at the repo root** on first copy - the SAME `config\apply-updates-schedule.yml` path on both GitHub Actions and Azure DevOps, alongside `config\ClusterUpdateRings.csv`. That starter ships with **DEMO** ring names (`Canary`, `DevTest`, `Ring1`, `Ring2`, `Prod`) that almost never match a real estate's `UpdateRing` tag values. Now that section 6.3 has tagged your clusters, regenerate the file so the rings, windows, and `allowedUpdateVersions` reflect the live fleet: ```powershell -# GitHub Actions -New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\.github\apply-updates-schedule.yml -Force - -# Azure DevOps (default starter location, repo root) -New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\apply-updates-schedule.yml -Force +# GitHub Actions and Azure DevOps (same config\ folder on both) +New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\config\apply-updates-schedule.yml -Force ``` -The cmdlet inspects every `UpdateRing` and `UpdateStartWindow` tag on the clusters the pipeline identity can read, then emits a schema v2 schedule with one block per distinct ring plus a Recommend / cron snippet inside `Step.6_apply-updates.yml`'s `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` marker block. Review the generated file, uncomment the rows you want active, then commit and push. +The cmdlet inspects every `UpdateRing` and `UpdateStartWindow` tag on the clusters the pipeline identity can read, then emits a schema v2 schedule with one block per distinct ring plus a Recommend / cron snippet inside `apply-updates.yml`'s `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` marker block. Review the generated file, uncomment the rows you want active, then commit and push. -**Why this step matters**: the schedule file is **required for scheduled Step.6 / Step.3 runs**. Manual `workflow_dispatch` (GitHub) / queue (Azure DevOps) runs of Step.6 work without it - they take `update_ring` / `updateRing` verbatim from the run-form input - but anything cron-triggered (the steady-state Step.6 wave kicks, the Step.3 weekly drift audit) reads ring eligibility from this file. +**Why this step matters**: the schedule file is **required for scheduled Step.7 / Step.3 runs**. Manual `workflow_dispatch` (GitHub) / queue (Azure DevOps) runs of Step.7 work without it - they take `update_ring` / `updateRing` verbatim from the run-form input - but anything cron-triggered (the steady-state Step.7 wave kicks, the Step.3 weekly drift audit) reads ring eligibility from this file. **Safety rails**: - `Copy-AzLocalPipelineExample` **never overwrites** an existing `apply-updates-schedule.yml` - any operator-tailored schedule already committed is preserved across re-runs. -- The bundled Step.6 ships with every `cron:` line commented out inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers, so Step.6 cannot fire on a `schedule:` trigger until at least one cron is explicitly uncommented - the DEMO starter is safe to leave in place until you regenerate. +- The bundled Step.7 ships with every `cron:` line commented out inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers, so Step.7 cannot fire on a `schedule:` trigger until at least one cron is explicitly uncommented - the DEMO starter is safe to leave in place until you regenerate. - Pass `-SkipStarterSchedule` to `Copy-AzLocalPipelineExample` to suppress the starter drop entirely (e.g. for estates that pre-stage the schedule via separate tooling). -For the full schema, multi-stage rollouts, weekly-cycle / ring-eligibility model, and the `allowedUpdateVersions` allow-list (schema v2), see [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods). The weekly drift detector (`Step.3_apply-updates-schedule-audit.yml`) that catches `(UpdateRing, UpdateStartWindow)` tag combinations no cron in Step.6 will ever reach is summarised in section 6.7 (full runbook in [section 8.3](#83-end-to-end-runbook-apply-updates-schedule-coverage-audit)). +For the full schema, multi-stage rollouts, weekly-cycle / ring-eligibility model, and the `allowedUpdateVersions` allow-list (schema v2), see [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods). The weekly drift detector (`apply-updates-schedule-audit.yml`) that catches `(UpdateRing, UpdateStartWindow)` tag combinations no cron in Step.7 will ever reach is summarised in section 6.7 (full runbook in [section 8.3](#83-end-to-end-runbook-apply-updates-schedule-coverage-audit)). ### 6.6 Apply updates - one wave at a time @@ -1436,7 +1435,7 @@ Use the duration data in `update-runs.csv` from the wave you just finished to si For tighter control around production rollouts, add a manual approval gate between waves: -- **Azure DevOps**: a separate stage with a `ManualValidation@0` step (the `Step.6_apply-updates.yml` shipped here includes a commented-out `WaitForApproval` block ready to enable). +- **Azure DevOps**: a separate stage with a `ManualValidation@0` step (the `apply-updates.yml` shipped here includes a commented-out `WaitForApproval` block ready to enable). - **GitHub Actions**: an `environment:` on the production job with required reviewers, configured in *Settings -> Environments*. ### 6.7 Continuous fleet monitoring @@ -1445,12 +1444,12 @@ The "steady-state" phase ships **three complementary pipelines**, all read-only, | Pipeline | Daily | Answers | Output | |----------|-------|---------|--------| -| `Step.4_fleet-connectivity-status.yml` *(v0.7.79+, enhanced in v0.7.85)* | 05:30 UTC | *"Are all clusters' Arc agents Connected? Are physical NICs healthy? Are Azure Resource Bridges reachable? Does the cluster's reported node count match the Arc-tagged physical machines we see?"* | JUnit + per-scope CSV/JSON + Markdown summary; one test case per cluster, with reconciliation rows that include "How to interpret + act" remediation guidance for any non-zero node-coverage delta | -| `Step.7_monitor-updates.yml` *(v0.7.90; v0.7.92 default cadence)* | 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the overnight maintenance window) + manual | *"What is happening right now? Which clusters are mid-update, which step are they on, and is anything stuck?"* | JUnit + CSV + Markdown summary; one test case per in-flight cluster (failure when elapsed > threshold, default 6h) | -| `Step.8_fleet-update-status.yml` *(formerly Step.7)* | 06:00 UTC | *"Is each cluster up-to-date? Which ones need an apply, which ones are SBE-blocked, which ones failed?"* | JUnit + CSV/JSON + Markdown summary; one test case per cluster | -| `Step.9_fleet-health-status.yml` *(v0.7.65, formerly Step.8)* | 07:00 UTC | *"Do clusters have actionable health issues even when up-to-date? What failure reasons hit the most clusters?"* | JUnit + CSV/JSON + Markdown summary; one test case per (cluster, failing 24-hour health check) grouped under Critical / Warning testsuites | +| `fleet-connectivity-status.yml` *(v0.7.79+, enhanced in v0.7.85)* | 05:30 UTC | *"Are all clusters' Arc agents Connected? Are physical NICs healthy? Are Azure Resource Bridges reachable? Does the cluster's reported node count match the Arc-tagged physical machines we see?"* | JUnit + per-scope CSV/JSON + Markdown summary; one test case per cluster, with reconciliation rows that include "How to interpret + act" remediation guidance for any non-zero node-coverage delta | +| `monitor-updates.yml` *(v0.7.90; v0.7.92 default cadence)* | 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the overnight maintenance window) + manual | *"What is happening right now? Which clusters are mid-update, which step are they on, and is anything stuck?"* | JUnit + CSV + Markdown summary; one test case per in-flight cluster (failure when elapsed > threshold, default 6h) | +| `fleet-update-status.yml` *(formerly Step.8)* | 06:00 UTC | *"Is each cluster up-to-date? Which ones need an apply, which ones are SBE-blocked, which ones failed?"* | JUnit + CSV/JSON + Markdown summary; one test case per cluster | +| `fleet-health-status.yml` *(v0.7.65, formerly Step.9)* | 07:00 UTC | *"Do clusters have actionable health issues even when up-to-date? What failure reasons hit the most clusters?"* | JUnit + CSV/JSON + Markdown summary; one test case per (cluster, failing 24-hour health check) grouped under Critical / Warning testsuites | -The four run in distinct (offset) cron slots so they don't contend for the same agent. `Step.7_monitor-updates.yml` ships **without** an active schedule - turn the `'*/30 * * * *'` cron on only during an active wave. +The four run in distinct (offset) cron slots so they don't contend for the same agent. `monitor-updates.yml` ships **without** an active schedule - turn the `'*/30 * * * *'` cron on only during an active wave. **Fleet Connectivity Status** *(introduced in v0.7.79, enhanced in v0.7.85)* runs daily at 05:30 UTC and answers the upstream question every other steady-state pipeline depends on: *"can the pipeline identity actually see every cluster, every physical node, and every Resource Bridge it is supposed to manage?"* The Step.4 reconciliation table compares each cluster's `reportedProperties.nodes` count against the Arc-tagged physical machines visible in Resource Graph and flags both directions of drift (positive = Arc has more machines than the cluster reports; negative = cluster reports more nodes than Arc can see). The v0.7.85 *"How to interpret + act on a non-zero reconciliation"* subsection in the pipeline summary gives operators per-direction remediation lists and an inline Resource Graph query template for triage. RBAC: `Reader` plus `Microsoft.ResourceGraph/resources/read`, `Microsoft.AzureStackHCI/edgeDevices/read`, `Microsoft.HybridCompute/machines/read`, and `Microsoft.ResourceConnector/appliances/read` - all already in the **`Azure Stack HCI Update Operator (custom)`** custom role definition shipped in [section 3.1](#31-custom-role-azure-stack-hci-update-operator-custom). @@ -1481,7 +1480,7 @@ It calls the new [`Get-AzLocalFleetHealthFailures`](../README.md#get-azlocalflee Configure your CI/CD platform's alerting on the JUnit failures - GitHub Actions surfaces them in the run summary and Azure DevOps shows them in the Tests tab with trend analytics. -**Plus weekly: `Step.3_apply-updates-schedule-audit.yml`** (read-only, runs Mondays at 05:00 UTC by default) catches drift between the cron schedule(s) committed to `Step.6_apply-updates.yml` and the `UpdateRing` / `UpdateStartWindow` tags that operators apply to new clusters. It emits a JUnit + CSV + Markdown "Recommend" snippet that pastes straight back into Step.6 to close any coverage gap. **For the full audit runbook (tag a cluster -> see drift -> paste recommended cron -> re-run and watch it turn green), see [section 8.3](#83-end-to-end-runbook-apply-updates-schedule-coverage-audit).** +**Plus weekly: `apply-updates-schedule-audit.yml`** (read-only, runs Mondays at 05:00 UTC by default) catches drift between the cron schedule(s) committed to `apply-updates.yml` and the `UpdateRing` / `UpdateStartWindow` tags that operators apply to new clusters. It emits a JUnit + CSV + Markdown "Recommend" snippet that pastes straight back into Step.7 to close any coverage gap. **For the full audit runbook (tag a cluster -> see drift -> paste recommended cron -> re-run and watch it turn green), see [section 8.3](#83-end-to-end-runbook-apply-updates-schedule-coverage-audit).** --- @@ -1497,7 +1496,7 @@ This README does not duplicate the setup - it is a single-source-of-truth in [`. > ```powershell > Copy-AzLocalItsmSample > ``` -> This copies `azurelocal-itsm.yml` + `templates/incident-body.md` from the installed module into `.\.itsm\` - the exact relative path that `Step.6_apply-updates.yml` looks for at job runtime (`itsm_config_path` / `itsmConfigPath` default `./.itsm/azurelocal-itsm.yml`). The sample is CI-platform-agnostic; both GitHub Actions and Azure DevOps consume the same YAML, only the secret source differs. To refresh the sample after a module upgrade, re-run with `-Update` (per-file `ShouldContinue` prompt) or `-Update -Confirm:$false` (unattended). See [`Copy-AzLocalItsmSample` reference](../Public/Copy-AzLocalItsmSample.ps1). +> This copies `azurelocal-itsm.yml` + `templates/incident-body.md` from the installed module into `.\.itsm\` - the exact relative path that `apply-updates.yml` looks for at job runtime (`itsm_config_path` / `itsmConfigPath` default `./.itsm/azurelocal-itsm.yml`). The sample is CI-platform-agnostic; both GitHub Actions and Azure DevOps consume the same YAML, only the secret source differs. To refresh the sample after a module upgrade, re-run with `-Update` (per-file `ShouldContinue` prompt) or `-Update -Confirm:$false` (unattended). See [`Copy-AzLocalItsmSample` reference](../Public/Copy-AzLocalItsmSample.ps1). | Step | Where it's documented | |---|---| @@ -1507,14 +1506,14 @@ This README does not duplicate the setup - it is a single-source-of-truth in [`. | Author the trigger matrix at `./.itsm/azurelocal-itsm.yml` (a ready-to-copy version ships in [`./.itsm/`](./.itsm/) - use `Copy-AzLocalItsmSample` to drop it into your repo) | [ITSM/README.md section 5](../ITSM/README.md#5-author-the-trigger-matrix) | | Validate end-to-end with `Test-AzLocalItsmConnection` before flipping the pipeline switch | [ITSM/README.md section 6](../ITSM/README.md#6-validate-before-you-wire-it-into-a-pipeline) | -Once the ServiceNow side is set up, the pipeline-side change is **already in `Step.6_apply-updates.yml`** in this folder. You enable it by: +Once the ServiceNow side is set up, the pipeline-side change is **already in `apply-updates.yml`** in this folder. You enable it by: 1. Setting `raise_itsm_ticket=true` when you trigger Apply Updates (workflow input in GH Actions, parameter in Azure DevOps). 2. Wiring the three secrets the step expects: - `ITSM_SN_INSTANCE_URL` - `ITSM_SN_CLIENT_ID` - `ITSM_SN_CLIENT_SECRET` -3. (Azure DevOps only) Uncomment the `- group: AzureLocal-ITSM-Secrets` line at the top of `Step.6_apply-updates.yml` once the variable group exists. +3. (Azure DevOps only) Uncomment the `- group: AzureLocal-ITSM-Secrets` line at the top of `apply-updates.yml` once the variable group exists. The first production run should keep `itsm_dry_run=true` (the connector still resolves secrets and performs the read-only dedupe lookup so you can validate the matrix + templates against a real workload, without creating tickets). The dry-run output includes a CSV + JUnit projection of "what would have been ticketed" - inspect those before flipping the switch. @@ -1534,9 +1533,9 @@ The `UpdateStartWindow` and `UpdateExclusionsWindow` tags on each cluster contro | `UpdateExclusionsWindow` (v0.7.90; was `UpdateExclusions`) | `YYYY-MM-DD/YYYY-MM-DD`, comma-separated, supports `*` wildcards | `20**-12-20/20**-01-03,2027-06-01/2027-06-10` | No updates start during these dates. **Exclusions override windows.** | | `UpdateExcluded` (v0.7.90) | `True` / `False` / `1` / `0` (case-insensitive) | `True` | Operator hard override. `True` skips the cluster with `Status = ExcludedByTag` regardless of ring scope, sideloaded state, or schedule. `Set-AzLocalClusterUpdateRingTag` default-stamps `False` on any cluster that doesn't already have the tag. | -> **CRITICAL: the `UpdateStartWindow` tag is a *gate*, not a *trigger*.** The tag only controls **whether** the Apply Updates pipeline is allowed to start an update on a given cluster when the pipeline is already running. The tag does **not** schedule the pipeline itself. The shipped `Step.6_apply-updates.yml` samples have **`workflow_dispatch` only** (GitHub Actions) / **`trigger: none`** (Azure DevOps) with no `schedule:` / `schedules:` block - which means **if you never trigger the pipeline manually during a window, no updates are ever applied automatically**, no matter what `UpdateStartWindow` you have tagged on your clusters. +> **CRITICAL: the `UpdateStartWindow` tag is a *gate*, not a *trigger*.** The tag only controls **whether** the Apply Updates pipeline is allowed to start an update on a given cluster when the pipeline is already running. The tag does **not** schedule the pipeline itself. The shipped `apply-updates.yml` samples have **`workflow_dispatch` only** (GitHub Actions) / **`trigger: none`** (Azure DevOps) with no `schedule:` / `schedules:` block - which means **if you never trigger the pipeline manually during a window, no updates are ever applied automatically**, no matter what `UpdateStartWindow` you have tagged on your clusters. > -> **You must add a `schedule:` (GH) / `schedules:` (ADO) block to `Step.6_apply-updates.yml` that fires *inside* (or a few minutes *before*) every `UpdateStartWindow` you have tagged.** If your fleet uses several distinct `UpdateStartWindow` values across rings, add one cron entry per window. Examples (UTC): +> **You must add a `schedule:` (GH) / `schedules:` (ADO) block to `apply-updates.yml` that fires *inside* (or a few minutes *before*) every `UpdateStartWindow` you have tagged.** If your fleet uses several distinct `UpdateStartWindow` values across rings, add one cron entry per window. Examples (UTC): > > | Cluster `UpdateStartWindow` tag | GitHub Actions `cron` | Azure DevOps `cron` | Notes | > |---|---|---|---| @@ -1546,7 +1545,7 @@ The `UpdateStartWindow` and `UpdateExclusionsWindow` tags on each cluster contro > > Inside the pipeline, `Test-AzLocalUpdateScheduleAllowed` is the per-cluster gate - clusters whose `UpdateStartWindow` does not cover "now" (or whose `UpdateExclusionsWindow` does cover "now") are skipped with `Status = ScheduleBlocked`. Running the pipeline outside any window is therefore safe but wasted - **running it during a window is the only way an update ever starts.** Separately, clusters with `UpdateExcluded = True` are skipped with `Status = ExcludedByTag` regardless of schedule. > -> **Tip - one pipeline per ring**: if `Pilot` / `Wave1` / `Production` have different windows, the cleanest pattern is to either (a) copy `Step.6_apply-updates.yml` per ring with the ring's own schedule + `update_ring` hard-coded, or (b) keep one YAML but pass `update_ring` via a matrix indexed by cron entry. Sticking with the default "single manual workflow" is fine for ad-hoc / change-controlled estates - in that case the operator manually clicks **Run workflow** at the start of the maintenance window. +> **Tip - one pipeline per ring**: if `Pilot` / `Wave1` / `Production` have different windows, the cleanest pattern is to either (a) copy `apply-updates.yml` per ring with the ring's own schedule + `update_ring` hard-coded, or (b) keep one YAML but pass `update_ring` via a matrix indexed by cron entry. Sticking with the default "single manual workflow" is fine for ad-hoc / change-controlled estates - in that case the operator manually clicks **Run workflow** at the start of the maintenance window. Grammar details: @@ -1569,39 +1568,39 @@ Test-AzLocalUpdateScheduleAllowed -UpdateStartWindow "Sat_02:00-06:00" -TestTime ### 8.1.1 Recommended Step.5 pre-flight schedule (per ring) -`Step.5_assess-update-readiness.yml` is the **report-only pre-flight** for an Apply Updates wave. The pipeline does not ship with a `schedule:` / `schedules:` block because the `update_ring` input is required and no single ring value is correct for every consumer. **Skipping Step.5 will not break Step.6** - the apply pipeline does its own internal `Get-AzLocalClusterUpdateReadiness` per-cluster pre-flight - but you lose the **human review window** between "we now know Ring2 has 12 clusters with blocking health checks" and "Ring2's maintenance window opens". For non-trivial estates that review window is worth preserving. +`assess-update-readiness.yml` is the **report-only pre-flight** for an Apply Updates wave. The pipeline does not ship with a `schedule:` / `schedules:` block because the `update_ring` input is required and no single ring value is correct for every consumer. **Skipping Step.5 will not break Step.7** - the apply pipeline does its own internal `Get-AzLocalClusterUpdateReadiness` per-cluster pre-flight - but you lose the **human review window** between "we now know Ring2 has 12 clusters with blocking health checks" and "Ring2's maintenance window opens". For non-trivial estates that review window is worth preserving. -The recommendation is to **schedule one Step.5 cron entry per ring you intend to patch, anchored 12-72 hours before that ring's Step.6 cron, with the lead time scaled to ring size**: +The recommendation is to **schedule one Step.5 cron entry per ring you intend to patch, anchored 12-72 hours before that ring's Step.7 cron, with the lead time scaled to ring size**: -| Ring size | Lead time | Step.5 cron (UTC) anchored on a `Sat 02:00` Step.6 window | Rationale | +| Ring size | Lead time | Step.5 cron (UTC) anchored on a `Sat 02:00` Step.7 window | Rationale | |---|---|---|---| | Small (<= 10 clusters) | 12-24 hours | `'0 2 * * 5'` (Fri 02:00) | Enough to triage a handful of `` entries; health signal still fresh at apply time. | | Medium (10-50 clusters) | 24-48 hours | `'0 2 * * 4'` (Thu 02:00) | Lets you raise tickets / engage cluster owners before the weekend. | | Large (50+ clusters) | 48-72 hours | `'0 2 * * 3'` (Wed 02:00) | Allows time to escalate, swap clusters into a deferral ring (`UpdateExcluded=True`), or stage parallel mitigation. | -Worked snippet (GitHub Actions, **one `Step.5_*.yml` per ring** - the recommended pattern, see "Why one YAML per ring" below): +Worked snippet (GitHub Actions, **one `assess-update-readiness*.yml` per ring** - the recommended pattern, see "Why one YAML per ring" below): ```yaml -# .github/workflows/Step.5_assess-update-readiness-Wave1.yml +# .github/workflows/assess-update-readiness-Wave1.yml name: Step.5 - Assess Update Readiness (Wave1) on: workflow_dispatch: inputs: update_ring: { description: 'UpdateRing tag value', required: true, default: 'Wave1' } schedule: - - cron: '0 2 * * 4' # Wave1 - Thu 02:00 UTC, 48h ahead of Sat 02:00 Wave1 Step.6 cron + - cron: '0 2 * * 4' # Wave1 - Thu 02:00 UTC, 48h ahead of Sat 02:00 Wave1 Step.7 cron env: UPDATE_RING: ${{ github.event.inputs.update_ring || 'Wave1' }} # ... job steps below pass $UPDATE_RING into the Assess-AzLocalClusterUpdateReadiness call ... ``` -Copy this file once per ring (e.g. `Step.5_..._Pilot.yml`, `Step.5_..._Wave1.yml`, `Step.5_..._Production.yml`), changing the ring name in three places: workflow `name:`, `default:` value, and the `cron:` day-of-week so each fires the right lead time ahead of *that* ring's Step.6 cron. +Copy this file once per ring (e.g. `assess-update-readiness-Pilot.yml`, `assess-update-readiness-Wave1.yml`, `assess-update-readiness-Production.yml`), changing the ring name in three places: workflow `name:`, `default:` value, and the `cron:` day-of-week so each fires the right lead time ahead of *that* ring's apply-updates cron. -**Why one YAML per ring**: GitHub Actions `schedule:`-triggered runs cannot supply `inputs:` values - they always use the workflow's default `inputs:`. So a single `Step.5_*.yml` with three crons in one `schedule:` block (and `update_ring` required, no default) would never actually run on cron - every cron tick would fail input validation. Splitting one YAML per ring (each with its own default + single cron) is the cleanest fix. Azure DevOps has the same constraint - `schedules:`-triggered runs use the YAML's default `parameters:` values, so the same per-ring split pattern applies. +**Why one YAML per ring**: GitHub Actions `schedule:`-triggered runs cannot supply `inputs:` values - they always use the workflow's default `inputs:`. So a single `assess-update-readiness*.yml` with three crons in one `schedule:` block (and `update_ring` required, no default) would never actually run on cron - every cron tick would fail input validation. Splitting one YAML per ring (each with its own default + single cron) is the cleanest fix. Azure DevOps has the same constraint - `schedules:`-triggered runs use the YAML's default `parameters:` values, so the same per-ring split pattern applies. -**Known gap**: the Step.3 schedule-coverage audit (`Step.3_apply-updates-schedule-audit.yml`) currently validates Step.6 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Step.5 cron is correctly anchored ahead of a Step.6 cron. **Always pair Step.5 + Step.6 cron edits in the same PR** so the lead-time relationship is reviewable by a human at merge time. The per-pipeline appendix entry for [Step 5 - Assess Update Readiness](docs/appendix-pipelines.md#step-5---assess-update-readiness) repeats this guidance with the same lead-time table. +**Known gap**: the Step.3 schedule-coverage audit (`apply-updates-schedule-audit.yml`) currently validates Step.7 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Step.5 cron is correctly anchored ahead of a Step.7 cron. **Always pair Step.5 + Step.7 cron edits in the same PR** so the lead-time relationship is reviewable by a human at merge time. The per-pipeline appendix entry for [Step 5 - Assess Update Readiness](docs/appendix-pipelines.md#step-5---assess-update-readiness) repeats this guidance with the same lead-time table. -**Always-green caveat**: `Step.5_assess-update-readiness.yml` never goes red at the pipeline level - per-cluster readiness gaps surface as JUnit `` entries in the Tests / Checks tab via `readiness.xml`. A silently-empty `readiness.xml` (e.g. an `update_ring` typo with zero clusters in scope) **will not generate a red-build email**. Either check the Tests tab after each scheduled run, or wire the JUnit reporter into the CI status surface you already monitor. +**Always-green caveat**: `assess-update-readiness.yml` never goes red at the pipeline level - per-cluster readiness gaps surface as JUnit `` entries in the Tests / Checks tab via `readiness.xml`. A silently-empty `readiness.xml` (e.g. an `update_ring` typo with zero clusters in scope) **will not generate a red-build email**. Either check the Tests tab after each scheduled run, or wire the JUnit reporter into the CI status surface you already monitor. ### 8.2 Multi-stage rollouts with approval gates @@ -1650,7 +1649,7 @@ jobs: ### 8.3 End-to-end runbook: Apply-Updates Schedule Coverage Audit -*(New in v0.7.65. Pre-wired pipeline samples: [`github-actions/Step.3_apply-updates-schedule-audit.yml`](./github-actions/Step.3_apply-updates-schedule-audit.yml), [`azure-devops/Step.3_apply-updates-schedule-audit.yml`](./azure-devops/Step.3_apply-updates-schedule-audit.yml).)* +*(New in v0.7.65. Pre-wired pipeline samples: [`github-actions/apply-updates-schedule-audit.yml`](./github-actions/apply-updates-schedule-audit.yml), [`azure-devops/apply-updates-schedule-audit.yml`](./azure-devops/apply-updates-schedule-audit.yml).)* This runbook walks through the full loop of **discover -> fix -> verify** for `UpdateStartWindow` / cron drift. Use it the first time you tag a new ring, and rely on the weekly scheduled audit to catch drift afterwards. @@ -1664,7 +1663,7 @@ Copy-AzLocalPipelineExample -Destination .\.github\workflows -Platform GitHub Copy-AzLocalPipelineExample -Destination .\pipelines -Platform AzureDevOps ``` -The audit YAML is one of the files copied. Commit and push. On GitHub it appears in the Actions tab; on Azure DevOps, import `Step.3_apply-updates-schedule-audit.yml` as a new pipeline (same procedure as for the other YAMLs in section 5.2). +The audit YAML is one of the files copied. Commit and push. On GitHub it appears in the Actions tab; on Azure DevOps, import `apply-updates-schedule-audit.yml` as a new pipeline (same procedure as for the other YAMLs in section 5.2). #### Step 2 - Tag a new ring with an UpdateStartWindow @@ -1682,7 +1681,7 @@ Repeat for the rest of the ring. - **GitHub Actions**: *Actions -> Apply-Updates Schedule Coverage Audit -> Run workflow*. - **Azure DevOps**: *Pipelines -> Apply-Updates Schedule Coverage Audit -> Run pipeline*. -Both default `pipelinePath` to the standard consumer location for the platform: `.github/workflows` on GitHub Actions, `.azure-pipelines` on Azure DevOps. If you copied `Step.6_apply-updates.yml` into one of those folders (the recommended layout), the audit finds it on the first run. Override `pipelinePath` only if your copied `Step.6_apply-updates.yml` lives somewhere else (e.g. `.\pipelines`). +Both default `pipelinePath` to the standard consumer location for the platform: `.github/workflows` on GitHub Actions, `.azure-pipelines` on Azure DevOps. If you copied `apply-updates.yml` into one of those folders (the recommended layout), the audit finds it on the first run. Override `pipelinePath` only if your copied `apply-updates.yml` lives somewhere else (e.g. `.\pipelines`). #### Step 4 - Read the run summary @@ -1715,7 +1714,7 @@ Followed by the per-row detail (Uncovered first): And finally the **ready-to-paste cron block** (Recommend view). With the default `firesPerWindow: 2`, every window emits **two** cron entries - one tagged `(open)` that fires `leadTimeMinutes` BEFORE the window opens, and one tagged `(retry)` that fires inside the window at the lesser of the window midpoint or +60 min after it opens: ````yaml -# --- GitHub Actions: paste under Step.6_apply-updates.yml `on:` --- +# --- GitHub Actions: paste under apply-updates.yml `on:` --- # schedule: # - cron: '55 1 * * 6,0' # Sat-Sun_02:00-06:00 (open) (rings: Pilot, 3 cluster(s)) # - cron: '3 3 * * 6,0' # Sat-Sun_02:00-06:00 (retry) (rings: Pilot, 3 cluster(s)) @@ -1727,7 +1726,7 @@ And finally the **ready-to-paste cron block** (Recommend view). With the default #### Step 5 - Apply the recommendation -Open `Step.6_apply-updates.yml`, uncomment / paste the recommended `schedule:` (GH) or `schedules:` (ADO) block, and commit. The audit pipeline emits both blocks even when `-Platform Both` is the default - copy the section that matches your CI/CD platform. +Open `apply-updates.yml`, uncomment / paste the recommended `schedule:` (GH) or `schedules:` (ADO) block, and commit. The audit pipeline emits both blocks even when `-Platform Both` is the default - copy the section that matches your CI/CD platform. #### Why two cron entries per window? (belt-and-braces) @@ -1745,13 +1744,13 @@ Pass `fires_per_window: '1'` (GitHub) or `firesPerWindow: 1` (ADO) on Step.3 to If two clusters share an `UpdateRing` tag but carry **different** `UpdateStartWindow` values (e.g. follow-the-sun, or a typo), Step.3 emits a separate `Covered` / `Uncovered` row for each (Ring, Window) pair AND a single informational `RingMixedWindows` warning row in the Schedule sub-table. The runtime gate reads each cluster's own tag so updates still fire correctly, but the ring no longer represents a single maintenance window for operator mental-model purposes. The warning row's recommendation lists three resolution choices: (a) standardise the ring to one window, (b) split into separate rings, or (c) accept the divergence and document it in the schedule file's `notes` column. `RingMixedWindows` is **not** counted as an `Uncovered` failure - the pipeline does not gate on it. -#### Step 6 - Re-run the audit to verify +#### Step 7 - Re-run the audit to verify Trigger the audit pipeline again. The summary should now show **Uncovered = 0**, the Audit Detail table all `Covered`, and JUnit Test Results all green. -#### Step 7 - Catch drift automatically +#### Step 8 - Catch drift automatically -Leave the weekly Monday 05:00 UTC schedule enabled. Any time someone tags a new cluster with a `UpdateStartWindow` value that the existing crons in `Step.6_apply-updates.yml` do not cover, the next Monday's audit run flips to **Uncovered** for that pair and surfaces it on the Tests tab. Configure your CI/CD alerting (GitHub Actions: branch-protection required check; Azure DevOps: notification on Test results) so the team is notified. +Leave the weekly Monday 05:00 UTC schedule enabled. Any time someone tags a new cluster with a `UpdateStartWindow` value that the existing crons in `apply-updates.yml` do not cover, the next Monday's audit run flips to **Uncovered** for that pair and surfaces it on the Tests tab. Configure your CI/CD alerting (GitHub Actions: branch-protection required check; Azure DevOps: notification on Test results) so the team is notified. #### Ad-hoc / desktop equivalent (no pipeline) @@ -1866,7 +1865,7 @@ Use the schema migrator: ```powershell Update-AzLocalApplyUpdatesScheduleConfig ` - -Path .\apply-updates-schedule.yml ` + -Path .\config\apply-updates-schedule.yml ` -SchemaMigrate ``` @@ -1881,7 +1880,7 @@ After migration the file is a strict superset of v1 - every cluster still resolv #### 8.4.5 Audit pipeline support -`Step.3_apply-updates-schedule-audit.yml` (GitHub Actions and Azure DevOps) automatically emits an **Allow-list coverage (schema v2)** section in its run summary when a `-SchedulePath` is supplied. The section surfaces the **top-level fleet default**, then a per-row table showing the effective `allowedUpdateVersions` for each schedule row (or `inherits top-level` when no row-level override is set), and recommends edit sites for rows that you might want to override. Schema v1 files get a one-line nudge to run the migrator. +`apply-updates-schedule-audit.yml` (GitHub Actions and Azure DevOps) automatically emits an **Allow-list coverage (schema v2)** section in its run summary when a `-SchedulePath` is supplied. The section surfaces the **top-level fleet default**, then a per-row table showing the effective `allowedUpdateVersions` for each schedule row (or `inherits top-level` when no row-level override is set), and recommends edit sites for rows that you might want to override. Schema v1 files get a one-line nudge to run the migrator. --- @@ -1895,7 +1894,7 @@ Only the **apply-side fan-out** still uses parallelism control, because applying | Function | `-ThrottleLimit` exposed | Used by pipeline | |---|---|---| -| `Start-AzLocalClusterUpdate` (apply-side fleet ops) | Internal via `Invoke-FleetJobsInParallel`; no user-facing `-ThrottleLimit` parameter. | Step.6 Apply Updates. | +| `Start-AzLocalClusterUpdate` (apply-side fleet ops) | Internal via `Invoke-FleetJobsInParallel`; no user-facing `-ThrottleLimit` parameter. | Step.7 Apply Updates. | The apply-side parallelism is bounded internally by the module's own job-pool helper; there is no operator-facing throttle knob to tune. @@ -1982,34 +1981,34 @@ Automation-Pipeline-Examples/ templates/ incident-body.md # - Mustache-style ticket body template. github-actions/ - Step.0_authentication-test.yml # 0. Authentication validation + subscription scope report (manual; v0.7.70). - Step.1_inventory-clusters.yml # 1. Inventory (weekly Mon 06:00 UTC + manual). - Step.2_manage-updatering-tags.yml # 2. Apply UpdateRing / UpdateStartWindow / UpdateExclusionsWindow / UpdateExcluded tags (manual). - Step.3_apply-updates-schedule-audit.yml # 3. Weekly read-only audit: UpdateStartWindow tags vs apply-updates cron (Mon 05:00 UTC, v0.7.65). - Step.4_fleet-connectivity-status.yml # 4. Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot + node-coverage reconciliation (daily 05:30 UTC, v0.7.79+; reconciliation enhanced in v0.7.85). - Step.5_assess-update-readiness.yml # 5. Pre-flight readiness report (manual; v0.7.0). - Step.6_apply-updates.yml # 6. Apply updates to one UpdateRing (with optional ITSM step, v0.7.4). - Step.7_monitor-updates.yml # 7. In-flight update monitor: per-cluster current step + elapsed duration; flags long-running runs (manual, optional */30 min cron; v0.7.90). - Step.8_fleet-update-status.yml # 8. Scheduled fleet update-status snapshot (daily 06:00 UTC; formerly Step.7). - Step.9_fleet-health-status.yml # 9. Scheduled fleet 24-hour health-check failure report (daily 07:00 UTC, v0.7.65; formerly Step.8). + authentication-test.yml # 0. Authentication validation + subscription scope report (manual; v0.7.70). + inventory-clusters.yml # 1. Inventory (weekly Mon 06:00 UTC + manual). + manage-updatering-tags.yml # 2. Apply UpdateRing / UpdateStartWindow / UpdateExclusionsWindow / UpdateExcluded tags (manual). + apply-updates-schedule-audit.yml # 3. Weekly read-only audit: UpdateStartWindow tags vs apply-updates cron (Mon 05:00 UTC, v0.7.65). + fleet-connectivity-status.yml # 4. Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot + node-coverage reconciliation (daily 05:30 UTC, v0.7.79+; reconciliation enhanced in v0.7.85). + assess-update-readiness.yml # 5. Pre-flight readiness report (manual; v0.7.0). + apply-updates.yml # 6. Apply updates to one UpdateRing (with optional ITSM step, v0.7.4). + monitor-updates.yml # 7. In-flight update monitor: per-cluster current step + elapsed duration; flags long-running runs (manual, optional */30 min cron; v0.7.90). + fleet-update-status.yml # 8. Scheduled fleet update-status snapshot (daily 06:00 UTC; formerly Step.8). + fleet-health-status.yml # 9. Scheduled fleet 24-hour health-check failure report (daily 07:00 UTC, v0.7.65; formerly Step.9). azure-devops/ - Step.0_authentication-test.yml - Step.1_inventory-clusters.yml - Step.2_manage-updatering-tags.yml - Step.3_apply-updates-schedule-audit.yml - Step.4_fleet-connectivity-status.yml - Step.5_assess-update-readiness.yml - Step.6_apply-updates.yml - Step.7_monitor-updates.yml - Step.8_fleet-update-status.yml - Step.9_fleet-health-status.yml + authentication-test.yml + inventory-clusters.yml + manage-updatering-tags.yml + apply-updates-schedule-audit.yml + fleet-connectivity-status.yml + assess-update-readiness.yml + apply-updates.yml + monitor-updates.yml + fleet-update-status.yml + fleet-health-status.yml ``` --- ## 14. Pipeline reference -Moved to [docs/appendix-pipelines.md](docs/appendix-pipelines.md) - one section per pipeline (`Step 0 - ...` ... `Step 9 - ...`) mapping 1:1 to the bundled `Step.N_*.yml` workflows, with purpose, inputs, trigger, cmdlets invoked, dependencies, artefacts, RBAC, and exit conditions for each. Kept out-of-line to keep this README focused on the runbook. +Moved to [docs/appendix-pipelines.md](docs/appendix-pipelines.md) - one section per pipeline (`Step 0 - ...` ... `Step 10 - ...`) mapping 1:1 to the bundled `*.yml` workflows, with purpose, inputs, trigger, cmdlets invoked, dependencies, artefacts, RBAC, and exit conditions for each. Kept out-of-line to keep this README focused on the runbook. ## Appendix B: Release history diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml index ec7efbdd..633fc4e8 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/apply-updates-schedule.example.yml @@ -5,7 +5,7 @@ # This is the single source of truth for "which UpdateRing(s) is/are # eligible to apply updates on a given UTC date". It is consumed by: # -# * Step.6_apply-updates.yml - reads it at every cron firing and +# * Step.7_apply-updates.yml - reads it at every cron firing and # resolves the UpdateRingValue (and, # schema v2+, the AllowedUpdateVersions # allow-list) to use for that run. @@ -59,13 +59,13 @@ cycleAnchorYear: 2026 # ---- AllowedUpdateVersions (schema v2, MANDATORY top-level) ------- # >>> ALLOWED-UPDATE-VERSIONS-V2 <<< -# Fleet-wide allow-list of Azure Local update identifiers that Step.6 +# Fleet-wide allow-list of Azure Local update identifiers that Step.7 # (apply-updates) is permitted to install. # # Default 'Latest' (case-insensitive) is a reserved sentinel meaning # 'no constraint - install the latest Ready update on each cluster' # (the historic v0.7.88 default). Leave it as 'Latest' to keep that -# behaviour. When set to an explicit list, Step.6 only installs +# behaviour. When set to an explicit list, Step.7 only installs # updates whose `name` OR `properties.version` is an EXACT # (case-insensitive) match for one of the entries. If a cluster has # no Ready update that matches, that cluster is SKIPPED with status @@ -98,7 +98,7 @@ cycleAnchorYear: 2026 # customer ask in v0.7.89): install ONLY the YY04 + YY10 feature # updates plus the cumulative updates immediately preceding each # feature update (YY03 + YY09). With v2 schema you list those four -# update names here once and Step.6 ignores every other 'Ready' +# update names here once and Step.7 ignores every other 'Ready' # update. Example using real Azure Local update names: # allowedUpdateVersions: 'Solution12.2603.1002.15;Solution12.2604.1003.1005;Solution12.2609.1002.XX;Solution12.2610.1003.XX' # diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.3_apply-updates-schedule-audit.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml similarity index 90% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.3_apply-updates-schedule-audit.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml index b6ef65c6..916f09d6 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.3_apply-updates-schedule-audit.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates-schedule-audit.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: apply-updates-schedule-audit # Step.3 - Apply-Updates Schedule Coverage Audit Pipeline # This pipeline runs the read-only Test-AzLocalApplyUpdatesScheduleCoverage advisor # weekly to detect drift between the cron schedule(s) in your apply-updates pipeline @@ -5,13 +6,13 @@ # # WHY THIS PIPELINE EXISTS # ------------------------ -# Step.6_apply-updates.yml ships with NO default schedule (trigger:none) on purpose, so +# Step.7_apply-updates.yml ships with NO default schedule (trigger:none) on purpose, so # customers must consciously choose when updates fire. After you have configured -# UpdateStartWindow tags on your clusters and added cron entries to Step.6_apply-updates.yml, this +# UpdateStartWindow tags on your clusters and added cron entries to Step.7_apply-updates.yml, this # pipeline is the safety net that catches: -# - a new ring tagged with UpdateStartWindow that no cron in Step.6_apply-updates.yml will reach +# - a new ring tagged with UpdateStartWindow that no cron in Step.7_apply-updates.yml will reach # - an UpdateStartWindow tag value that fails to parse (typo / wrong syntax) -# - Step.6_apply-updates.yml cron entries that the advisor cannot reason about +# - Step.7_apply-updates.yml cron entries that the advisor cannot reason about # (DayOfMonth restrictions, step values) so you can audit them manually # # REPORTS GENERATED: @@ -43,14 +44,14 @@ schedules: parameters: - name: pipelinePath - displayName: 'Path (file or folder) to Step.6_apply-updates.yml to audit, repo-relative. REQUIRED so the Recommend view can diff its proposed crons against what is already present in Step.6 and only emit a snippet for the truly missing entries. Default `.azure-pipelines` matches the standard Azure DevOps consumer layout; for repos that keep Step.6_apply-updates.yml at the repo root pass `.`' + displayName: 'Path (file or folder) to Step.7_apply-updates.yml to audit, repo-relative. REQUIRED so the Recommend view can diff its proposed crons against what is already present in Step.7 and only emit a snippet for the truly missing entries. Default `.azure-pipelines` matches the standard Azure DevOps consumer layout; for repos that keep Step.7_apply-updates.yml at the repo root pass `.`' type: string default: '.azure-pipelines' - name: schedulePath displayName: 'Path to apply-updates-schedule.yml. When set, the audit also reports RingMissingFromSchedule (fleet ring with no schedule row) and RingOrphanedInSchedule (schedule ring no cluster carries). Leave empty to skip the schedule-file two-way diff.' type: string - default: './apply-updates-schedule.yml' + default: './config/apply-updates-schedule.yml' - name: leadTimeMinutes displayName: 'Minutes before UpdateStartWindow opens that the pipeline should fire (0-60)' @@ -86,9 +87,14 @@ parameters: default: false variables: - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}' reportsPath: '$(Build.ArtifactStagingDirectory)/reports' + # v0.8.7 sideload advisor defaults. Override at the pipeline / variable-group + # level (set SIDELOAD_UPDATES=true to enable the recommended-sideload-schedule + # section). Defaults keep the audit byte-identical to v0.8.6 when unset. + SIDELOAD_UPDATES: 'false' + SIDELOAD_LEAD_DAYS: '7' stages: - stage: ScheduleCoverage @@ -156,6 +162,10 @@ stages: INPUT_DEBUG: ${{ lower(parameters.debug) }} INSTALLED_MODULE_VERSION: $(moduleVersion.installed_module_version) REPORTS_PATH: $(reportsPath) + # v0.8.7 sideload advisor (opt-in). When SIDELOAD_UPDATES=true the audit + # appends a "Recommended sideload schedule" section. Inert when unset/false. + SIDELOAD_UPDATES: $(SIDELOAD_UPDATES) + SIDELOAD_LEAD_DAYS: $(SIDELOAD_LEAD_DAYS) inputs: # Replace with your service connection name azureSubscription: 'AzureLocal-ServiceConnection' # Update with your service connection name diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.6_apply-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.6_apply-updates.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml index d1479e36..1d3ac712 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.6_apply-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/apply-updates.yml @@ -1,9 +1,10 @@ -# Step.6 - Apply Updates to Azure Local Clusters +# AZLOCAL-PIPELINE-ID: apply-updates +# Step.7 - Apply Updates to Azure Local Clusters # This pipeline applies updates to clusters filtered by UpdateRing tag value # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers # PREFERRED PATTERN - drive automated update runs from a ring-aware -# schedule file (apply-updates-schedule.yml at the repo root by default), +# schedule file (config/apply-updates-schedule.yml by default), # not from per-window crons. # # How it works: @@ -20,7 +21,7 @@ # New-AzLocalApplyUpdatesScheduleConfig. # # Generate a STRAWMAN schedule file from your fleet via: -# New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\apply-updates-schedule.yml +# New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\config\apply-updates-schedule.yml # Every row is commented out by default - review and uncomment to activate. # # Concurrency: ADO has no first-class YAML concurrency block; use Pipeline @@ -114,7 +115,7 @@ variables: # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - name: GENERATED_AGAINST_MODULE_VERSION - value: '0.8.6' + value: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -122,10 +123,12 @@ variables: value: '${{ parameters.moduleVersion }}' # Path to the ring-aware apply-updates-schedule.yml (relative to repo # root). The 'Resolve UpdateRing from schedule' task reads this file when - # $(Build.Reason)='Schedule'. Override at queue time if you keep the schedule - # outside the repo root. + # $(Build.Reason)='Schedule'. Default 'config/apply-updates-schedule.yml' + # matches where Copy-AzLocalPipelineExample drops the starter, alongside + # config/ClusterUpdateRings.csv. Override at queue time if you keep the + # schedule elsewhere. - name: APPLY_UPDATES_SCHEDULE_PATH - value: './apply-updates-schedule.yml' + value: './config/apply-updates-schedule.yml' # ITSM connector. Uncomment the group reference once you # have created a variable group named 'AzureLocal-ITSM-Secrets' containing: # ITSM_SN_INSTANCE_URL, ITSM_SN_CLIENT_ID, ITSM_SN_CLIENT_SECRET @@ -230,7 +233,7 @@ stages: displayName: 'Publish Readiness Report' inputs: PathtoPublish: '$(Build.ArtifactStagingDirectory)' - ArtifactName: 'azlocal-step.6-apply-updates-readiness-report_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.7-apply-updates-readiness-report_$(stamp.artifactStamp)' publishLocation: 'Container' # Stage 2: Apply Updates (with approval gate for production) @@ -324,7 +327,7 @@ stages: source: 'current' # artifact name now carries a UTC timestamp - rebuild via the # readinessArtifactStamp variable mapped from the CheckReadiness stage. - artifact: 'azlocal-step.6-apply-updates-readiness-report_$(readinessArtifactStamp)' + artifact: 'azlocal-step.7-apply-updates-readiness-report_$(readinessArtifactStamp)' path: '$(Build.ArtifactStagingDirectory)' # v0.8.5 thin-YAML - readiness-gated apply via shared helper. Validates @@ -379,7 +382,7 @@ stages: displayName: 'Publish Update Logs' inputs: PathtoPublish: '$(Build.ArtifactStagingDirectory)' - ArtifactName: 'azlocal-step.6-apply-updates-logs_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.7-apply-updates-logs_$(stamp.artifactStamp)' publishLocation: 'Container' - task: PublishTestResults@2 @@ -447,7 +450,7 @@ stages: condition: and(always(), eq('${{ parameters.raiseItsmTicket }}', 'true')) inputs: PathtoPublish: '$(Build.ArtifactStagingDirectory)' - ArtifactName: 'azlocal-step.6-apply-updates-itsm-results_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.7-apply-updates-itsm-results_$(stamp.artifactStamp)' publishLocation: 'Container' continueOnError: true diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.5_assess-update-readiness.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.5_assess-update-readiness.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml index 6a166dd3..736a07b7 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.5_assess-update-readiness.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/assess-update-readiness.yml @@ -1,7 +1,8 @@ +# AZLOCAL-PIPELINE-ID: assess-update-readiness # Step.5 - Assess Update Readiness (Pre-flight go/no-go gate) # -------------------------------------------------- # Runs Get-AzLocalClusterUpdateReadiness and Test-AzLocalClusterHealth -BlockingOnly -# against a target UpdateRing (or the whole fleet) BEFORE Step.6_apply-updates.yml runs. +# against a target UpdateRing (or the whole fleet) BEFORE Step.7_apply-updates.yml runs. # # Publishes two JUnit XML diagnostic results to the Azure DevOps Tests tab: # - readiness.xml (one test per cluster; fails if ReadyForUpdate = $false) @@ -14,7 +15,7 @@ # does NOT block downstream runs. In large fleets, day-to-day environmental issues # (transient storage noise, a single node out, etc.) routinely affect a small subset of # clusters; blocking the entire wave for one unhealthy cluster is rarely the desired -# behavior. Step.6_apply-updates.yml is itself per-cluster scoped, so clusters that are not +# behavior. Step.7_apply-updates.yml is itself per-cluster scoped, so clusters that are not # ready will simply no-op there too. # # If you want a hard pass/fail signal for a chained gate, read the output variables @@ -69,7 +70,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.0_authentication-test.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/authentication-test.yml similarity index 99% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.0_authentication-test.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/authentication-test.yml index 972546a8..43950bca 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.0_authentication-test.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/authentication-test.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: authentication-test # Step.0 - Authentication Validation and Subscription Scope Report # # PURPOSE: @@ -59,7 +60,7 @@ variables: # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - name: GENERATED_AGAINST_MODULE_VERSION - value: '0.8.6' + value: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.4_fleet-connectivity-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.4_fleet-connectivity-status.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml index 146758ef..3ca3d5f7 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.4_fleet-connectivity-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: fleet-connectivity-status # Step.4 - Fleet Connectivity Status Monitoring Pipeline # This pipeline surfaces network-connectivity issues across every Azure Local # cluster the service connection can read - four scopes in one pipeline: @@ -12,7 +13,7 @@ # - Find clusters where Arc agent or ARB has dropped off and updates / management calls will fail # - Catch Disconnected physical NICs that still have a real IP (likely cable / driver / port issues - # Disconnected adapters with no IP or APIPA self-assignment are intentionally excluded as noise) -# - Triage hand-off into ITSM (wired identically to Step.8 / Step.9 - opt-in via parameter) +# - Triage hand-off into ITSM (wired identically to Step.9 / Step.10 - opt-in via parameter) # # REPORTS GENERATED: # - Pipeline run summary (uploaded via ##vso[task.uploadsummary]): one section per @@ -56,7 +57,7 @@ name: Step.4 - Fleet Connectivity Status trigger: none # Manual or scheduled only schedules: - # Daily at 05:30 UTC - offset 30 min ahead of Step.8 (06:00) and Step.9 (07:00) + # Daily at 05:30 UTC - offset 30 min ahead of Step.9 (06:00) and Step.10 (07:00) # so the three fleet-scope read pipelines stagger their ARG calls. - cron: '30 5 * * *' displayName: 'Daily Fleet Connectivity Check' @@ -109,7 +110,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -121,7 +122,7 @@ variables: # named 'AzureLocal-ITSM-Secrets'. ADO does not allow mixing mapping-form # variables with '- group:' entries in the same block, so to enable the # group reference convert this entire 'variables:' block to list form (see - # Step.9 for the conversion recipe). When raiseItsmTicket=false the Raise + # Step.10 for the conversion recipe). When raiseItsmTicket=false the Raise # ITSM task is skipped and the secrets are not consulted. stages: @@ -228,7 +229,7 @@ stages: # ---------------------------------------------------------------------- # ITSM Connector (ServiceNow auto-raise on connectivity failures) # Opt-in (gated on parameters.raiseItsmTicket == true). Mirrors the - # Step.8 / Step.9 wiring - reads the JUnit file produced above, uses + # Step.9 / Step.10 wiring - reads the JUnit file produced above, uses # the same azurelocal-itsm.yml trigger matrix, same dedupe key. # ---------------------------------------------------------------------- - task: PowerShell@2 diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.9_fleet-health-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.9_fleet-health-status.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml index a04c783d..23bc87b2 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.9_fleet-health-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-health-status.yml @@ -1,4 +1,5 @@ -# Step.9 - Fleet Health Status Monitoring Pipeline +# AZLOCAL-PIPELINE-ID: fleet-health-status +# Step.10 - Fleet Health Status Monitoring Pipeline # This pipeline surfaces 24-hour system health-check failures across every Azure Local # cluster the service connection can read - including clusters that are already # "up to date" with no available updates. The 24-hour health checks continue to run @@ -104,7 +105,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -217,7 +218,7 @@ stages: condition: always() inputs: PathtoPublish: '$(reportsPath)' - ArtifactName: 'azlocal-step.9-fleet-health-status-report_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.10-fleet-health-status-report_$(stamp.artifactStamp)' publishLocation: 'Container' # Publish JUnit test results LAST so the Test Results tab is populated but the @@ -321,7 +322,7 @@ stages: condition: and(always(), eq('${{ parameters.raiseItsmTicket }}', 'true')) inputs: PathtoPublish: '$(reportsPath)' - ArtifactName: 'azlocal-step.9-fleet-health-status-itsm-results_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.10-fleet-health-status-itsm-results_$(stamp.artifactStamp)' publishLocation: 'Container' continueOnError: true @@ -331,5 +332,5 @@ stages: inputs: testResultsFormat: 'JUnit' testResultsFiles: '$(reportsPath)/itsm-results.xml' - testRunTitle: 'ITSM Tickets (Step.9) - $(Build.BuildNumber)' + testRunTitle: 'ITSM Tickets (Step.10) - $(Build.BuildNumber)' failTaskOnFailedTests: false diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.8_fleet-update-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.8_fleet-update-status.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml index a6ca0ea8..e8ba7f33 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.8_fleet-update-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/fleet-update-status.yml @@ -1,4 +1,5 @@ -# Step.8 - Fleet Update Status Monitoring Pipeline +# AZLOCAL-PIPELINE-ID: fleet-update-status +# Step.9 - Fleet Update Status Monitoring Pipeline # This pipeline monitors update status across all Azure Local clusters and generates reports # # USE CASES: @@ -65,7 +66,7 @@ parameters: # Failed update run reported by the 'Update Run History and Error Details' # testsuite (and any failed AzureLocalFleetUpdateStatus row). Default is false # so existing schedules stay byte-identical until you opt in. The connector - # reads the JUnit file Step.6 already produces (readiness-status.xml) and + # reads the JUnit file Step.7 already produces (readiness-status.xml) and # the trigger matrix in ./.itsm/azurelocal-itsm.yml. - name: raiseItsmTicket displayName: 'Open ITSM tickets (ServiceNow) for fleet-update-status failures' @@ -92,7 +93,7 @@ variables: # the version actually installed and to the latest on PSGallery, and emits a warning # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). @@ -203,7 +204,7 @@ stages: condition: always() inputs: PathtoPublish: '$(reportsPath)' - ArtifactName: 'azlocal-step.8-fleet-update-status-report_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.9-fleet-update-status-report_$(stamp.artifactStamp)' publishLocation: 'Container' # Publish JUnit test results LAST so the Test Results tab is populated @@ -310,7 +311,7 @@ stages: condition: and(always(), eq('${{ parameters.raiseItsmTicket }}', 'true')) inputs: PathtoPublish: '$(reportsPath)' - ArtifactName: 'azlocal-step.8-fleet-update-status-itsm-results_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.9-fleet-update-status-itsm-results_$(stamp.artifactStamp)' publishLocation: 'Container' continueOnError: true @@ -320,5 +321,5 @@ stages: inputs: testResultsFormat: 'JUnit' testResultsFiles: '$(reportsPath)/itsm-results.xml' - testRunTitle: 'ITSM Tickets (Step.6) - $(Build.BuildNumber)' + testRunTitle: 'ITSM Tickets (Step.9) - $(Build.BuildNumber)' failTaskOnFailedTests: false diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.1_inventory-clusters.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/inventory-clusters.yml similarity index 99% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.1_inventory-clusters.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/inventory-clusters.yml index 112369fa..f604cfde 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.1_inventory-clusters.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/inventory-clusters.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: inventory-clusters # Step.1 - Inventory Azure Local Clusters # This pipeline queries all Azure Local clusters and exports inventory with UpdateRing tag status # @@ -42,7 +43,7 @@ variables: # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - name: GENERATED_AGAINST_MODULE_VERSION - value: '0.8.6' + value: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.2_manage-updatering-tags.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml similarity index 99% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.2_manage-updatering-tags.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml index d6f47b1e..2d6b4712 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.2_manage-updatering-tags.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/manage-updatering-tags.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: manage-updatering-tags # Step.2 - Manage UpdateRing Tags # This pipeline creates or updates UpdateRing tags on Azure Local clusters from a CSV file # @@ -45,7 +46,7 @@ variables: # log if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - name: GENERATED_AGAINST_MODULE_VERSION - value: '0.8.6' + value: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.7_monitor-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml similarity index 96% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.7_monitor-updates.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml index 528c5fb8..8d40f7f6 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/Step.7_monitor-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/monitor-updates.yml @@ -1,9 +1,10 @@ -# Step.7 - Monitor In-Flight Updates (Azure DevOps) +# AZLOCAL-PIPELINE-ID: monitor-updates +# Step.8 - Monitor In-Flight Updates (Azure DevOps) # ----------------------------------------------------------- # Reports clusters whose latest update run is currently in flight, with the # CURRENT STEP each cluster is on and the ELAPSED DURATION of the run. Built # for operators who want a 30-minute-cadence "what is happening right now" -# view, distinct from the daily Step.8 fleet-update-status snapshot which +# view, distinct from the daily Step.9 fleet-update-status snapshot which # answers "is each cluster up-to-date?". # # This pipeline is REPORT-ONLY and always succeeds. Long-running runs are @@ -83,7 +84,7 @@ parameters: default: '' variables: - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}' reportsPath: '$(Build.ArtifactStagingDirectory)/reports' @@ -173,7 +174,7 @@ stages: condition: always() inputs: PathtoPublish: '$(reportsPath)' - ArtifactName: 'azlocal-step.7-update-monitor_$(stamp.artifactStamp)' + ArtifactName: 'azlocal-step.8-update-monitor_$(stamp.artifactStamp)' publishLocation: 'Container' - task: PublishTestResults@2 diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml new file mode 100644 index 00000000..72253097 --- /dev/null +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/azure-devops/sideload-updates.yml @@ -0,0 +1,249 @@ +# AZLOCAL-PIPELINE-ID: sideload-updates +# Step.6 - Sideload Updates (Opt-in, on-prem self-hosted agent) +# +# OPT-IN, OFF BY DEFAULT. This pipeline automates the on-prem sideloading of +# Azure Local solution-update media for clusters that cannot pull updates from +# Azure directly. It is the NEW Step.6 introduced in v0.8.7; the former Step.6 +# (Apply Updates) is now Step.7. +# +# WHY A SELF-HOSTED AGENT: +# The agent must sit on the SAME network as the target Azure Local clusters so +# it can (a) Robocopy the CombinedSolutionBundle to each cluster's +# infrastructure `import` SMB share and (b) PowerShell-remote (WinRM) into a +# cluster node to verify the SHA256 and run Add-SolutionUpdate. Microsoft-hosted +# agents cannot reach the on-prem fabric, so this pipeline targets a self-hosted +# AGENT in an agent POOL, selected by the `azlocal-sideload` demand. +# +# RE-ENTRANT STATE MACHINE: +# Each run advances every in-scope cluster by ONE state transition and exits. +# The multi-hour copy itself runs in a detached Windows Scheduled Task, so no +# pipeline run is ever long-lived. Drive this on a frequent CRON (e.g. every +# 30 minutes) so the state machine progresses Copying -> Copied -> Verified -> +# Imported across successive short runs. Re-running is always safe. +# +# GATE: the whole job is skipped unless the pipeline variable +# SIDELOAD_UPDATES == 'true'. When unset/false this file is inert. +# +# AUTHENTICATION: +# - Azure CLI context (ARG fleet read via 'az graph query' + UpdateSideloaded +# tag flip via 'az rest') is provided by the AzureCLI@2 task's Workload +# Identity Federation service connection. +# - Az PowerShell context (Get-AzKeyVaultSecret for the WinRM credential) is +# established inside the script from the federated token that +# addSpnToEnvironment exposes. For a secret-based service principal or a +# self-hosted agent managed identity, replace that Connect-AzAccount line per +# the SIDELOAD_KV_AUTH variable (see inline comments). +# - Cluster WinRM remoting uses an Active Directory [pscredential] built from +# two Key Vault secrets named in the sideload auth-map row (NOT the pipeline +# identity). + +# BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers +# Edits inside this block are preserved across Update-AzLocalPipelineExample +# module upgrades. Add or modify `schedules:` blocks below. +trigger: none # Manual or scheduled only + +# NO default schedule ships out-of-the-box (this pipeline requires an on-prem +# self-hosted agent that most projects do not have). Once your agent is online in +# a pool that satisfies the `azlocal-sideload` demand, UNCOMMENT the schedule +# below to drive the re-entrant state machine. A frequent poll (every 30 min) +# lets a single short run advance the multi-hour copy without staying alive. +# +# schedules: +# - cron: '*/30 * * * *' # every 30 minutes - advance/report sideload state +# displayName: 'Sideload state machine poll' +# branches: +# include: +# - main +# always: true +# END-AZLOCAL-CUSTOMIZE:schedule-triggers + +parameters: + - name: updateRing + # accepts a single ring, 'Prod;Ring2' list, or '***' wildcard (three stars - deliberate). + displayName: "UpdateRing tag value to scope the plan ('Wave1', 'Prod;Ring2', or '***' for ALL rings)." + type: string + default: '***' + + - name: dryRun + displayName: 'Preview the plan + transitions without staging media, registering tasks, or flipping tags (WhatIf).' + type: boolean + default: true + + - name: moduleVersion + displayName: 'Pin AzLocal.UpdateManagement version (empty = latest from PSGallery). See Automation-Pipeline-Examples/README.md section 5 "Optional configuration".' + type: string + default: '' + +variables: + # Module version this YAML was generated against. The install step compares this to + # the version actually installed and to the latest on PSGallery, and emits a warning + # log if the YAML appears stale - prompting you to refresh via + # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' + # Resolution order for the module version pin (leave all unset to install the latest, + # which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable + # 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest). + REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}' + reportsPath: '$(Build.ArtifactStagingDirectory)/reports' + + # ---- Sideload configuration (set these as PIPELINE VARIABLES or via a variable group) ---- + # Master gate. The stage is skipped unless this is the literal string 'true'. + # SIDELOAD_UPDATES: 'true' + # Shared UNC root that ALL agents can read+write (state\, logs\, cache\): + # SIDELOAD_STATE_ROOT: '\\\\fileserver\\azlocal-sideload' + # SIDELOAD_CACHE_ROOT: '' # defaults (in the cmdlet) to \cache + # SIDELOAD_AUTH_MAP_PATH: './config/sideload-auth-map.csv' + # SIDELOAD_CATALOG_PATH: './config/sideload-catalog.yml' + # SIDELOAD_LEAD_DAYS: '7' + # SIDELOAD_ROBOCOPY_SWITCHES: '/R:5 /W:30' + # SIDELOAD_HEARTBEAT_STALE_MINUTES: '60' + # SIDELOAD_REMOTING_FQDN_SUFFIX: '' # e.g. '.corp.contoso.com' + # APPLY_UPDATES_SCHEDULE_PATH: './config/apply-updates-schedule.yml' + # SIDELOAD_KV_AUTH: 'oidc' # oidc | managedidentity | serviceprincipal + +stages: +- stage: Sideload + displayName: 'Step.6 - Sideload Updates (Opt-in)' + # Master opt-in gate: do nothing unless SIDELOAD_UPDATES is explicitly 'true'. + condition: eq(variables['SIDELOAD_UPDATES'], 'true') + jobs: + - job: AdvanceSideload + displayName: 'Advance Sideload State Machine' + # Requires an on-prem self-hosted AGENT (Azure DevOps terminology) on the same + # network as the clusters, in a POOL whose agents advertise the `azlocal-sideload` + # capability. Replace '' with your pool name. + pool: + name: '' + demands: + - azlocal-sideload + + steps: + - checkout: self + displayName: 'Checkout repository' + + - task: PowerShell@2 + displayName: 'Install AzLocal.UpdateManagement from PSGallery' + name: moduleVersion + env: + REQUIRED_MODULE_VERSION: $(REQUIRED_MODULE_VERSION) + GENERATED_AGAINST_MODULE_VERSION: $(GENERATED_AGAINST_MODULE_VERSION) + inputs: + targetType: 'inline' + pwsh: true + script: | + $ErrorActionPreference = 'Stop' + $installArgs = @{ Name = 'AzLocal.UpdateManagement'; Scope = 'CurrentUser'; Force = $true; AllowClobber = $true } + if ($env:REQUIRED_MODULE_VERSION) { + $installArgs.RequiredVersion = $env:REQUIRED_MODULE_VERSION + Write-Host "REQUIRED_MODULE_VERSION is set - pinning install to v$($env:REQUIRED_MODULE_VERSION)." + } else { + Write-Host "REQUIRED_MODULE_VERSION is empty - installing the latest version from PSGallery (default fix-forward behaviour)." + } + Install-Module @installArgs + Import-Module AzLocal.UpdateManagement -Force + Add-AzLocalPipelineVersionBanner ` + -GeneratedAgainstVersion $env:GENERATED_AGAINST_MODULE_VERSION ` + -PinnedVersion $env:REQUIRED_MODULE_VERSION + + - task: AzureCLI@2 + displayName: 'Advance Sideload State Machine' + env: + INPUT_UPDATE_RING: ${{ parameters.updateRing }} + INPUT_DRY_RUN: ${{ parameters.dryRun }} + APPLY_UPDATES_SCHEDULE_PATH: $(APPLY_UPDATES_SCHEDULE_PATH) + SIDELOAD_STATE_ROOT: $(SIDELOAD_STATE_ROOT) + SIDELOAD_CACHE_ROOT: $(SIDELOAD_CACHE_ROOT) + SIDELOAD_AUTH_MAP_PATH: $(SIDELOAD_AUTH_MAP_PATH) + SIDELOAD_CATALOG_PATH: $(SIDELOAD_CATALOG_PATH) + SIDELOAD_LEAD_DAYS: $(SIDELOAD_LEAD_DAYS) + SIDELOAD_ROBOCOPY_SWITCHES: $(SIDELOAD_ROBOCOPY_SWITCHES) + SIDELOAD_HEARTBEAT_STALE_MINUTES: $(SIDELOAD_HEARTBEAT_STALE_MINUTES) + SIDELOAD_REMOTING_FQDN_SUFFIX: $(SIDELOAD_REMOTING_FQDN_SUFFIX) + reportsPath: $(reportsPath) + inputs: + azureSubscription: 'AzureLocal-ServiceConnection' # Update with your service connection name + scriptType: 'pscore' + scriptLocation: 'inlineScript' + # addSpnToEnvironment exposes $env:servicePrincipalId / $env:idToken / + # $env:tenantId so we can also light up an Az PowerShell context for the + # Key Vault secret read (Get-AzKeyVaultSecret). + addSpnToEnvironment: true + inlineScript: | + $ErrorActionPreference = 'Stop' + Import-Module AzLocal.UpdateManagement -Force + + if ([string]::IsNullOrWhiteSpace($env:SIDELOAD_STATE_ROOT)) { + throw "SIDELOAD_STATE_ROOT is not set. Configure the shared UNC state root pipeline variable before running this pipeline." + } + + # Establish an Az PowerShell context for the Key Vault credential read. + # WIF service connection -> federated token. For a secret-based service + # principal use -ServicePrincipal -Credential; for a self-hosted agent + # managed identity use Connect-AzAccount -Identity (per SIDELOAD_KV_AUTH). + if ($env:idToken -and $env:servicePrincipalId -and $env:tenantId) { + Connect-AzAccount -ServicePrincipal ` + -ApplicationId $env:servicePrincipalId ` + -Tenant $env:tenantId ` + -FederatedToken $env:idToken | Out-Null + } + + # 1. Resolve the per-cluster plan (read-only). + $planParams = @{ + SchedulePath = $env:APPLY_UPDATES_SCHEDULE_PATH + AuthMapPath = $env:SIDELOAD_AUTH_MAP_PATH + CatalogPath = $env:SIDELOAD_CATALOG_PATH + LeadDays = [int]$env:SIDELOAD_LEAD_DAYS + } + if ($env:INPUT_UPDATE_RING) { $planParams['UpdateRingValue'] = $env:INPUT_UPDATE_RING } + if ($env:SIDELOAD_REMOTING_FQDN_SUFFIX) { $planParams['FqdnSuffix'] = $env:SIDELOAD_REMOTING_FQDN_SUFFIX } + $plan = @(Resolve-AzLocalSideloadPlan @planParams) + Write-Host ("Resolved {0} cluster plan row(s)." -f $plan.Count) + + # 2. Advance the re-entrant state machine by one transition per cluster. + $applyParams = @{ + Plan = $plan + StateRoot = $env:SIDELOAD_STATE_ROOT + ErrorAction = 'Stop' + } + if ($env:SIDELOAD_CACHE_ROOT) { $applyParams['CacheRoot'] = $env:SIDELOAD_CACHE_ROOT } + if ($env:SIDELOAD_ROBOCOPY_SWITCHES) { $applyParams['RobocopySwitches'] = $env:SIDELOAD_ROBOCOPY_SWITCHES } + if ($env:SIDELOAD_HEARTBEAT_STALE_MINUTES) { $applyParams['HeartbeatStaleMinutes'] = [int]$env:SIDELOAD_HEARTBEAT_STALE_MINUTES } + if ($env:INPUT_DRY_RUN -eq 'true' -or $env:INPUT_DRY_RUN -eq 'True') { + $applyParams['WhatIf'] = $true + Write-Host 'DRY RUN MODE - no media staged, no tasks registered, no tags flipped.' + } + $results = @(Invoke-AzLocalSideloadUpdate @applyParams) + Write-Host ("Processed {0} cluster(s)." -f $results.Count) + + # 3. Render the status report (markdown step summary + JUnit XML + sideload-status.md). + New-Item -ItemType Directory -Path $env:reportsPath -Force | Out-Null + $report = Add-AzLocalSideloadStepSummary -StateRoot $env:SIDELOAD_STATE_ROOT -Plan $plan -OutputPath $env:reportsPath + + if ($report.HasFailures) { + Write-Host '##vso[task.logissue type=warning]One or more clusters are in a Failed state or have plan errors - see the Sideload status report.' + } + + - pwsh: | + $stamp = (Get-Date).ToUniversalTime().ToString('yyyyMMdd_HHmmss') + Write-Host "##vso[task.setvariable variable=artifactStamp;isOutput=true]$stamp" + Write-Host "Artifact timestamp: $stamp" + displayName: 'Compute Artifact Timestamp' + condition: always() + name: stamp + + - task: PublishBuildArtifacts@1 + displayName: 'Publish Sideload Reports' + condition: always() + inputs: + PathtoPublish: '$(reportsPath)' + ArtifactName: 'azlocal-step.6-sideload-updates-report_$(stamp.artifactStamp)' + publishLocation: 'Container' + + - task: PublishTestResults@2 + displayName: 'Publish Sideload Status as Test Results' + condition: always() + inputs: + testResultsFormat: 'JUnit' + testResultsFiles: '$(reportsPath)/sideload-junit.xml' + testRunTitle: 'Sideload Update - $(Build.BuildNumber)' + failTaskOnFailedTests: false diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md index 9b834854..34bdd002 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/appendix-pipelines.md @@ -17,17 +17,17 @@ The table below is the ground truth for what each shipped YAML does **out of the | Step 2 - `manage-updatering-tags` | `workflow_dispatch` only | `trigger: none` (manual only) | Runs on-demand whenever you edit the CSV. | | Step 3 - `apply-updates-schedule-audit` | `workflow_dispatch` + `schedule: cron '0 5 * * 1'` (Mondays 05:00 UTC) | `trigger: none` + `schedules: cron '0 5 * * 1'` | Weekly read-only drift advisor: compares apply-updates cron(s) to `UpdateStartWindow` tags. Runs before the daily fleet pipelines so its annotations are first on Monday mornings. | | Step 4 - `fleet-connectivity-status` | `workflow_dispatch` + `schedule: cron '30 5 * * *'` (daily 05:30 UTC) | `trigger: none` + `schedules: cron '30 5 * * *'` | Daily fleet connectivity / Arc / NIC / Resource Bridge snapshot with bidirectional node-coverage reconciliation. Runs 30 min before `fleet-update-status` so connectivity issues are visible upstream of update reporting. | -| Step 5 - `assess-update-readiness` | `workflow_dispatch` only | `trigger: none` (manual only) | **Recommended customisation** - schedule per ring 12-72h before the matching Step 6 `UpdateStartWindow` (see [Step 5 - Assess Update Readiness](#step-5---assess-update-readiness) below for the per-ring pattern + lead-time table). Always-green at pipeline level; per-cluster readiness gaps surface as JUnit `` entries. | -| Step 6 - `apply-updates` | `workflow_dispatch` only | `trigger: none` (manual only) | **No schedule shipped** - see the warning in [Step 6 - Apply Updates](#step-6---apply-updates) below. The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags only gate updates *while the pipeline is running*; they do **not** start the pipeline. | -| Step 7 - `monitor-updates` | `workflow_dispatch` only (commented-out `schedule: cron '*/30 * * * *'` sample shipped) | `trigger: none` only (commented-out `schedules: cron '*/30 * * * *'` sample shipped) | In-flight update progress / long-running-run flag. Disabled by default to avoid 48 idle runs/day between waves. Uncomment the cron when a wave is active. | -| Step 8 - `fleet-update-status` | `workflow_dispatch` + `schedule: cron '0 6 * * *'` (daily 06:00 UTC) | `trigger: none` + `schedules: cron '0 6 * * *'` | Daily fleet update snapshot. | -| Step 9 - `fleet-health-status` | `workflow_dispatch` + `schedule: cron '0 7 * * *'` (daily 07:00 UTC) | `trigger: none` + `schedules: cron '0 7 * * *'` | Daily 24-hour health-check snapshot. Offset by one hour from `fleet-update-status` to avoid contention. | +| Step 5 - `assess-update-readiness` | `workflow_dispatch` only | `trigger: none` (manual only) | **Recommended customisation** - schedule per ring 12-72h before the matching Step 7 `UpdateStartWindow` (see [Step 5 - Assess Update Readiness](#step-5---assess-update-readiness) below for the per-ring pattern + lead-time table). Always-green at pipeline level; per-cluster readiness gaps surface as JUnit `` entries. | +| Step 7 - `apply-updates` | `workflow_dispatch` only | `trigger: none` (manual only) | **No schedule shipped** - see the warning in [Step 7 - Apply Updates](#step-7---apply-updates) below. The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags only gate updates *while the pipeline is running*; they do **not** start the pipeline. | +| Step 8 - `monitor-updates` | `workflow_dispatch` only (commented-out `schedule: cron '*/30 * * * *'` sample shipped) | `trigger: none` only (commented-out `schedules: cron '*/30 * * * *'` sample shipped) | In-flight update progress / long-running-run flag. Disabled by default to avoid 48 idle runs/day between waves. Uncomment the cron when a wave is active. | +| Step 9 - `fleet-update-status` | `workflow_dispatch` + `schedule: cron '0 6 * * *'` (daily 06:00 UTC) | `trigger: none` + `schedules: cron '0 6 * * *'` | Daily fleet update snapshot. | +| Step 10 - `fleet-health-status` | `workflow_dispatch` + `schedule: cron '0 7 * * *'` (daily 07:00 UTC) | `trigger: none` + `schedules: cron '0 7 * * *'` | Daily 24-hour health-check snapshot. Offset by one hour from `fleet-update-status` to avoid contention. | > **All times are UTC.** GitHub Actions and Azure DevOps schedules both run on UTC; convert from your local timezone when picking cron values. Both platforms can delay scheduled runs by several minutes during high-load periods - do not rely on second-precision alignment. > > **GitHub Actions only**: scheduled workflows are **automatically disabled after 60 days of repository inactivity** (no commits, PRs, or issue activity). Re-enable them via the Actions UI or run any push. Azure DevOps schedules do not have this auto-disable behaviour. -> **Common conventions for every row below**: `Cmdlets invoked` lists only AzLocal.UpdateManagement cmdlets the YAML calls directly (helper cmdlets they themselves call transitively are documented in each cmdlet's section of the module-level [`README.md`](../../README.md)). `Depends on` lists upstream pipelines whose output is consumed or whose work must be in place before this pipeline runs - `None` means the pipeline is self-contained. `Exit conditions` describes what a non-success YAML run state means in practice; per-cluster issues are always surfaced as JUnit `` entries rather than failing the whole pipeline unless noted otherwise. `ITSM` indicates whether the bundled YAMLs ship with the opt-in ServiceNow auto-raise connector (`Get-AzLocalItsmConfig` + per-failure ticket creation gated on the `raise_itsm_ticket` / `raiseItsmTicket` input); only Step 4, Step 6, Step 8, and Step 9 ship it - the matrix config lives at `./.itsm/azurelocal-itsm.yml` (see [`Automation-Pipeline-Examples/ITSM/`](../ITSM/) for the full recipe). +> **Common conventions for every row below**: `Cmdlets invoked` lists only AzLocal.UpdateManagement cmdlets the YAML calls directly (helper cmdlets they themselves call transitively are documented in each cmdlet's section of the module-level [`README.md`](../../README.md)). `Depends on` lists upstream pipelines whose output is consumed or whose work must be in place before this pipeline runs - `None` means the pipeline is self-contained. `Exit conditions` describes what a non-success YAML run state means in practice; per-cluster issues are always surfaced as JUnit `` entries rather than failing the whole pipeline unless noted otherwise. `ITSM` indicates whether the bundled YAMLs ship with the opt-in ServiceNow auto-raise connector (`Get-AzLocalItsmConfig` + per-failure ticket creation gated on the `raise_itsm_ticket` / `raiseItsmTicket` input); only Step 4, Step 7, Step 9, and Step 10 ship it - the matrix config lives at `./.itsm/azurelocal-itsm.yml` (see [`Automation-Pipeline-Examples/ITSM/`](../ITSM/) for the full recipe). --- @@ -71,7 +71,7 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Trigger** | Manual only (`workflow_dispatch` / **Run pipeline** button). No schedule shipped - this is a deliberate change-controlled operation that should follow a CSV edit + review. Add a `schedule:` / `schedules:` block if your CSV is auto-generated and you want periodic re-application. | | **Cmdlets invoked** | `Set-AzLocalClusterUpdateRingTag`. | | **Depends on** | Step 1 produces the input CSV (the operator typically downloads `cluster-inventory.csv`, edits ring / window / exclusion values in a PR, then re-uploads it as the input to this pipeline). | -| **Artefacts** | `UpdateRingTag_YYYYMMDD_HHmmss.csv` (per-cluster CSV log: ClusterName, ResourceGroup, SubscriptionId, ResourceId, Action, PreviousTagValue, NewTagValue, Status, Message) and `UpdateRingTag_Results.json` (the same rows in JSON form, written from `-PassThru`) inside the published log artifact. Since v0.8.0 the pipeline job summary tab also renders a **result breakdown table** (Created / Updated / Already in sync / Skipped / WhatIf / Failed) plus a collapsible **per-cluster details** block, at parity with Step 3 / Step 5 / Step 6 / Step 8. | +| **Artefacts** | `UpdateRingTag_YYYYMMDD_HHmmss.csv` (per-cluster CSV log: ClusterName, ResourceGroup, SubscriptionId, ResourceId, Action, PreviousTagValue, NewTagValue, Status, Message) and `UpdateRingTag_Results.json` (the same rows in JSON form, written from `-PassThru`) inside the published log artifact. Since v0.8.0 the pipeline job summary tab also renders a **result breakdown table** (Created / Updated / Already in sync / Skipped / WhatIf / Failed) plus a collapsible **per-cluster details** block, at parity with Step 3 / Step 5 / Step 7 / Step 9. | | **When to run** | After editing the inventory CSV; whenever ring membership or maintenance windows change. | | **RBAC** | Write to tags only. The built-in **Tag Contributor** role on the cluster scope is sufficient (`Microsoft.Resources/tags/*`). Since v0.7.65, `Set-AzLocalClusterUpdateRingTag` writes via `Microsoft.Resources/tags/default` PATCH, so the broader `Microsoft.AzureStackHCI/clusters/write` is not required. | | **Exit conditions** | Pipeline run is green when every CSV row is processed (added / updated / unchanged). Per-cluster tag-write failures surface in the run log; the pipeline does not currently fail on per-row errors. Re-run after triaging is safe (the cmdlet is idempotent). | @@ -81,8 +81,8 @@ The table below is the ground truth for what each shipped YAML does **out of the | Aspect | Value | |---|---| -| **Purpose** | Read-only advisor that compares the cron schedule(s) in your `Step.6_apply-updates.yml` to the `UpdateStartWindow` tag values present on your clusters, and flags any `(UpdateRing, UpdateStartWindow)` pair that no cron in `Step.6_apply-updates.yml` will ever reach. Never edits tags or YAML. | -| **Inputs** | `pipeline_path` (file or folder; default `.github/workflows` on GitHub Actions, `.azure-pipelines` on Azure DevOps - the standard consumer locations for the bundled `Step.6_apply-updates.yml` sample), `lead_time_minutes` (0-60, default 5), `include_untagged` (default false), `module_version` (optional). | +| **Purpose** | Read-only advisor that compares the cron schedule(s) in your `apply-updates.yml` to the `UpdateStartWindow` tag values present on your clusters, and flags any `(UpdateRing, UpdateStartWindow)` pair that no cron in `apply-updates.yml` will ever reach. Never edits tags or YAML. | +| **Inputs** | `pipeline_path` (file or folder; default `.github/workflows` on GitHub Actions, `.azure-pipelines` on Azure DevOps - the standard consumer locations for the bundled `apply-updates.yml` sample), `lead_time_minutes` (0-60, default 5), `include_untagged` (default false), `module_version` (optional). | | **Trigger** | Manual (`workflow_dispatch` / **Run pipeline** button) **plus** scheduled weekly on Mondays at 05:00 UTC (`cron '0 5 * * 1'`). Deliberately runs before the daily `fleet-connectivity-status` (05:30 UTC), `fleet-update-status` (06:00 UTC), and `fleet-health-status` (07:00 UTC) pipelines so its drift annotations land at the top of the operator's Monday-morning inbox. Edit the cron in the YAML to change cadence. | | **Cmdlets invoked** | `Test-AzLocalApplyUpdatesScheduleCoverage`, `Get-AzLocalApplyUpdatesScheduleConfig` (when a `schedule.yml` is present in the repo). | | **Depends on** | Step 1 (cluster inventory + `UpdateStartWindow` tags), Step 2 (tags applied) for non-trivial output. Runs fine on an empty fleet but the audit is meaningless. | @@ -119,24 +119,42 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Cmdlets invoked** | `Get-AzLocalClusterInventory`, `Get-AzLocalClusterUpdateReadiness`, `Test-AzLocalClusterHealth`. | | **Depends on** | Step 1 (an `UpdateRing` tag must exist on at least one cluster to scope the readiness query) and Step 2 (to apply that tag at scale). | | **Artefacts** | `readiness.xml`, `readiness.csv`, `health-blocking.xml`, `health-blocking.csv`. | -| **When to run** | 12-72 hours before each Step 6 wave for the same ring (12-24h for small rings, 48-72h for large rings of 50+ clusters) - long enough for an operator to triage any `` entries before the maintenance window opens, short enough that the readiness signal is still fresh. See the **recommended customisation note below** for the cron pattern. | +| **When to run** | 12-72 hours before each Step 7 wave for the same ring (12-24h for small rings, 48-72h for large rings of 50+ clusters) - long enough for an operator to triage any `` entries before the maintenance window opens, short enough that the readiness signal is still fresh. See the **recommended customisation note below** for the cron pattern. | | **RBAC** | Read-only. Covered by the `Azure Stack HCI Update Operator (custom)` custom role. | | **Exit conditions** | Pipeline run is always green - per-cluster readiness gaps and blocking health checks surface as JUnit `` entries in the Tests tab. **Caveat**: because the pipeline never goes red, a silently-empty `readiness.xml` (e.g. ring tag typo, zero clusters in scope) will not generate a red email - check the Tests tab / `readiness.xml` artefact after each run, or wire the JUnit reporter into your existing CI status surface. | -| **ITSM** | Not supported - this is a pre-flight gate intended for operator review before the Step 6 wave; readiness gaps surface as JUnit `` entries and feed the Step 6 ITSM dispatch downstream rather than raising tickets here. | +| **ITSM** | Not supported - this is a pre-flight gate intended for operator review before the Step 7 wave; readiness gaps surface as JUnit `` entries and feed the Step 7 ITSM dispatch downstream rather than raising tickets here. | -> **RECOMMENDED CUSTOMISATION: schedule Step 5 per ring, 12-72h ahead of the matching Step 6 cron.** Unlike Step 6 (which silently does nothing if you forget to schedule it), Step 5 is *recommended* rather than *mandatory* - Step 6 does its own internal `Get-AzLocalClusterUpdateReadiness` per-cluster pre-flight, so skipping Step 5 will not break the apply step. What you lose by skipping it is the **human review window** - the chance to see "12 clusters in Ring2 have blocking health checks" and intervene before the maintenance window starts. +> **RECOMMENDED CUSTOMISATION: schedule Step 5 per ring, 12-72h ahead of the matching Step 7 cron.** Unlike Step 7 (which silently does nothing if you forget to schedule it), Step 5 is *recommended* rather than *mandatory* - Step 7 does its own internal `Get-AzLocalClusterUpdateReadiness` per-cluster pre-flight, so skipping Step 5 will not break the apply step. What you lose by skipping it is the **human review window** - the chance to see "12 clusters in Ring2 have blocking health checks" and intervene before the maintenance window starts. > -> Because `update_ring` is a required input there is no single bundled cron - **add one `schedule:` (GitHub Actions) / `schedules:` (Azure DevOps) entry per ring you intend to patch**, each setting `update_ring:` via the workflow inputs (GH Actions `with:` / ADO `parameters:`). Anchor each Step 5 cron to the matching Step 6 cron (same days, earlier by the lead time): +> Because `update_ring` is a required input there is no single bundled cron - **add one `schedule:` (GitHub Actions) / `schedules:` (Azure DevOps) entry per ring you intend to patch**, each setting `update_ring:` via the workflow inputs (GH Actions `with:` / ADO `parameters:`). Anchor each Step 5 cron to the matching Step 7 cron (same days, earlier by the lead time): > -> | Ring size | Recommended Step 5 lead time | Step 5 cron (UTC, anchoring on a Step 6 Sat 02:00 window) | Rationale | +> | Ring size | Recommended Step 5 lead time | Step 5 cron (UTC, anchoring on a Step 7 Sat 02:00 window) | Rationale | > |---|---|---|---| -> | Small (<= 10 clusters) | 12-24 hours ahead | `'0 2 * * 5'` (Fri 02:00 ahead of a Sat 02:00 Step 6) | Enough to triage a handful of `` entries; fresh enough that the health signal is still meaningful at apply time. | -> | Medium (10-50 clusters) | 24-48 hours ahead | `'0 2 * * 4'` (Thu 02:00 ahead of a Sat 02:00 Step 6) | Lets you raise tickets / engage cluster owners on per-cluster issues before the weekend. | -> | Large (50+ clusters) | 48-72 hours ahead | `'0 2 * * 3'` (Wed 02:00 ahead of a Sat 02:00 Step 6) | Allows time to escalate, swap clusters into a deferral ring (`UpdateExcluded=True`), or stage parallel mitigation. | +> | Small (<= 10 clusters) | 12-24 hours ahead | `'0 2 * * 5'` (Fri 02:00 ahead of a Sat 02:00 Step 7) | Enough to triage a handful of `` entries; fresh enough that the health signal is still meaningful at apply time. | +> | Medium (10-50 clusters) | 24-48 hours ahead | `'0 2 * * 4'` (Thu 02:00 ahead of a Sat 02:00 Step 7) | Lets you raise tickets / engage cluster owners on per-cluster issues before the weekend. | +> | Large (50+ clusters) | 48-72 hours ahead | `'0 2 * * 3'` (Wed 02:00 ahead of a Sat 02:00 Step 7) | Allows time to escalate, swap clusters into a deferral ring (`UpdateExcluded=True`), or stage parallel mitigation. | > -> **Known gap**: the Step 3 schedule-coverage audit currently validates Step 6 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Step 5 cron is correctly anchored to a Step 6 cron. Pair Step 5 and Step 6 cron edits in the same PR so the lead-time relationship is reviewable. The end-to-end runbook in the parent README's [section 8.1.1](../README.md#811-recommended-step5-pre-flight-schedule-per-ring) has worked examples for the most common ring layouts. +> **Known gap**: the Step 3 schedule-coverage audit currently validates Step 7 cron-to-`UpdateStartWindow` coverage only - it does **not** audit whether each Step 5 cron is correctly anchored to a Step 7 cron. Pair Step 5 and Step 7 cron edits in the same PR so the lead-time relationship is reviewable. The end-to-end runbook in the parent README's [section 8.1.1](../README.md#811-recommended-step5-pre-flight-schedule-per-ring) has worked examples for the most common ring layouts. -## Step 6 - Apply Updates +## Step 6 - Sideload Updates + +> **OPT-IN, OFF BY DEFAULT, self-hosted runner required.** Introduced in v0.8.7. This pipeline is inert unless the repository variable `SIDELOAD_UPDATES` is the literal string `'true'`. Full architecture, the scheduled-task survival model, the shared-state / multi-runner contract, the auth-map, and the catalog format are in [sideload.md](sideload.md); robocopy throttling guidance is in [sideload-robocopy.md](sideload-robocopy.md). + +| Aspect | Value | +|---|---| +| **Purpose** | Pre-stage Azure Local solution-update media onto clusters that **cannot pull updates from Azure directly** (dark / air-gapped / restricted-egress fabrics). It Robocopies the `CombinedSolutionBundle` (or OEM SBE package) to each cluster's infrastructure `import` SMB share, verifies the SHA256 over WinRM, runs `Add-SolutionUpdate`, and flips `UpdateSideloaded=True` so the downstream Step 7 apply can proceed. | +| **Inputs** | `update_ring` (optional - a single ring, a `;`-delimited list, or `***` for every tagged cluster; default `***`), `dry_run` (optional, default `true` - previews the plan + transitions with no staging / task / tag changes), `module_version` (optional pin). | +| **Trigger** | **Manual only by default** (`workflow_dispatch` / **Run pipeline** button). **No schedule ships** because the pipeline requires an on-prem self-hosted runner labelled `azlocal-sideload` (GH) / a self-hosted agent in a pool with the `azlocal-sideload` demand (ADO) that most repos do not have. Once the runner/agent is online, uncomment the bundled `*/30 * * * *` cron inside the `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` block to drive the state machine. | +| **Cmdlets invoked** | `Resolve-AzLocalSideloadPlan` (selects clusters whose next apply window is within `SIDELOAD_LEAD_DAYS`), `Invoke-AzLocalSideloadUpdate` (the re-entrant state machine), `Export-AzLocalSideloadStatusReport` + `Add-AzLocalSideloadStepSummary` (JUnit + Markdown summary). | +| **Re-entrant state machine** | Each run advances every in-scope cluster by **one** transition and exits: `Planned -> Copying -> Copied -> Verified -> Imported -> SideloadFlagged`. The multi-hour copy itself runs in a **detached Windows Scheduled Task** (driven by `Tools/Invoke-AzLocalSideloadCopyTask.ps1`) so no pipeline run is ever long-lived. Re-running is always safe; a stale `Copying` heartbeat (> `SIDELOAD_HEARTBEAT_STALE_MINUTES`) is re-driven. | +| **Depends on** | Step 1 (`UpdateRing` + sideload tags present, including `UpdateAuthAccountId`), Step 2 (to apply those tags at scale), and a populated `sideload-auth-map.csv` + `sideload-catalog.yml`. The downstream Step 7 apply consumes the `UpdateSideloaded=True` flag this pipeline sets. | +| **Artefacts** | `sideload-status.xml` (JUnit, one cluster per test), `sideload-status.csv`, plus per-run copy/verify logs under the shared `SIDELOAD_STATE_ROOT\logs`. | +| **Shared state** | `SIDELOAD_STATE_ROOT` must be a UNC path that every runner/agent can read+write (`state\`, `logs\`, `cache\`). Verified media is cached under `SIDELOAD_CACHE_ROOT` (defaults to `\cache`) so a bundle is downloaded + hashed once and reused across clusters. | +| **RBAC** | ARG fleet read + Key Vault secret read + `UpdateSideloaded` tag write for the pipeline identity; a separate Active Directory `[pscredential]` (built from two Key Vault secrets named in the auth-map row) is used for the cluster WinRM remoting and `Add-SolutionUpdate`. | +| **Exit conditions** | Pipeline run is green when every in-scope cluster either advances cleanly or is correctly classified as a skip. Per-cluster copy / verify / import failures surface as JUnit `` entries. | +| **ITSM** | Not supported in v0.8.7 (staging is a pre-apply preparation step; failures surface as JUnit `` entries for operator review). | + +## Step 7 - Apply Updates | Aspect | Value | |---|---| @@ -148,16 +166,16 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Artefacts** | `update-results.xml` (JUnit, one cluster per test), `update-logs/*` (CSV + detail). When ITSM is enabled: `itsm-results.csv`, `itsm-results.xml`. | | **When to run** | During the maintenance window for each ring, after the readiness assessment is reviewed. | | **RBAC** | Write to clusters required. Covered by the `Azure Stack HCI Update Operator (custom)` custom role (`Microsoft.AzureStackHCI/clusters/updates/apply/action` + cluster-update reads). | -| **Exit conditions** | Pipeline run is green when every in-scope cluster either succeeds or is correctly classified as `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` (these are skips, not failures). Per-cluster update failures surface as JUnit `` entries; long-running runs are tracked by Step 7. | +| **Exit conditions** | Pipeline run is green when every in-scope cluster either succeeds or is correctly classified as `ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag` (these are skips, not failures). Per-cluster update failures surface as JUnit `` entries; long-running runs are tracked by Step 8. | | **ITSM** | Supported (opt-in via `raise_itsm_ticket=true` / `raiseItsmTicket=true`). When enabled, one ServiceNow incident is raised per cluster whose update run finished with a failure state; classification skips (`ScheduleBlocked` / `SideloadedBlocked` / `ExcludedByTag`) do not generate tickets. `itsm_dry_run` builds payloads without creating tickets; `itsm_force_create` bypasses dedupe. | -> **MANDATORY CUSTOMISATION: the Apply Updates pipeline does not ship with a schedule.** The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags **only gate updates *while the pipeline is already running***; they do **not** start the pipeline. If you (a) use `UpdateStartWindow` tags to define when updates may be installed and (b) leave the shipped `Step.6_apply-updates.yml` with `workflow_dispatch` only (GH) / `trigger: none` (ADO), **no updates will ever be applied automatically** - the pipeline will simply never start during the window. +> **MANDATORY CUSTOMISATION: the Apply Updates pipeline does not ship with a schedule.** The cluster `UpdateStartWindow` / `UpdateExclusionsWindow` tags **only gate updates *while the pipeline is already running***; they do **not** start the pipeline. If you (a) use `UpdateStartWindow` tags to define when updates may be installed and (b) leave the shipped `apply-updates.yml` with `workflow_dispatch` only (GH) / `trigger: none` (ADO), **no updates will ever be applied automatically** - the pipeline will simply never start during the window. > -> Add a `schedule:` (GitHub Actions) / `schedules:` (Azure DevOps) block to `Step.6_apply-updates.yml` that fires at (or a few minutes before) the start of every `UpdateStartWindow` you have tagged. One cron entry per distinct window value. Worked examples and the per-cluster scheduling model are in [section 8](../README.md#8-scheduling-maintenance-windows-and-change-freeze-periods). +> Add a `schedule:` (GitHub Actions) / `schedules:` (Azure DevOps) block to `apply-updates.yml` that fires at (or a few minutes before) the start of every `UpdateStartWindow` you have tagged. One cron entry per distinct window value. Worked examples and the per-cluster scheduling model are in [section 8](../README.md#8-scheduling-maintenance-windows-and-change-freeze-periods). > -> Pipeline summaries (Step 6 GH Actions + Azure DevOps) classify per-cluster skips as `ScheduleBlocked` (outside `UpdateStartWindow` or inside `UpdateExclusionsWindow`), `SideloadedBlocked` (`UpdateSideloaded=False` on a sideloaded-workflow cluster) and, new in v0.7.90, `ExcludedByTag` (`UpdateExcluded=True` operator hard override). The Actions Required callout points operators at the `UpdateExcluded` tag when one or more clusters are intentionally held out of automation. +> Pipeline summaries (Step 7 GH Actions + Azure DevOps) classify per-cluster skips as `ScheduleBlocked` (outside `UpdateStartWindow` or inside `UpdateExclusionsWindow`), `SideloadedBlocked` (`UpdateSideloaded=False` on a sideloaded-workflow cluster) and, new in v0.7.90, `ExcludedByTag` (`UpdateExcluded=True` operator hard override). The Actions Required callout points operators at the `UpdateExcluded` tag when one or more clusters are intentionally held out of automation. -## Step 7 - Monitor In-Flight Updates +## Step 8 - Monitor In-Flight Updates | Aspect | Value | |---|---| @@ -165,19 +183,19 @@ The table below is the ground truth for what each shipped YAML does **out of the | **Inputs** | `scope` (`all-clusters` (default) or `by-update-ring`), `update_ring` (only used when `scope=by-update-ring`; single ring, `Prod;Ring2` semicolon-list, or `***` wildcard), `long_running_threshold_hours` (0-48, default 6), `module_version` (optional). | | **Trigger** | **Manual only by default** (`workflow_dispatch` / **Run pipeline** button). A commented-out `schedule: cron '*/30 * * * *'` (GH) / `schedules: cron '*/30 * * * *'` (ADO) sample is shipped inside the `# BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers` block - uncomment it for the duration of an active wave, then re-comment it once the wave drains, so you do not generate 48 empty runs per day between waves. | | **Cmdlets invoked** | `Get-AzLocalUpdateRuns`, `Get-AzLocalClusterInventory` (used for ring-scope mode). | -| **Depends on** | Step 6 must be active (at least one `InProgress` run) for the output to be non-empty. Runs cleanly on an empty fleet (reports "no in-flight runs"). | +| **Depends on** | Step 7 must be active (at least one `InProgress` run) for the output to be non-empty. Runs cleanly on an empty fleet (reports "no in-flight runs"). | | **Artefacts** | `update-monitor.xml` (JUnit, one `` per in-flight cluster - the failure message contains current step + progress + elapsed duration when the run is over `long_running_threshold_hours`, or the deepest-step error detail when `failureType='StepError'` fires under v0.7.96), `update-monitor.csv` (full per-cluster rows including the new `Status` + `ErrorMessage` columns), markdown job summary with the in-flight table (cluster + update names as portal links from v0.7.96) and long-running flag column. | | **Exit conditions** | Pipeline run is green when the snapshot completes. Long-running runs (over `long_running_threshold_hours`) AND stuck-step errors (`failureType='StepError'`, v0.7.96+) surface as JUnit `` entries so they appear in the Tests / Checks tab without failing the whole pipeline. v0.7.96 also adds two new `GITHUB_OUTPUT` values: `STEP_ERRORED` (count of runs that hit a step error inside the threshold) and `UNRESOLVED_FAILURES` (count of Failed runs surfaced from `Get-AzLocalUpdateRunFailures`). | -| **When to run** | While a wave is active (manually, or via the commented cron). The two daily snapshot pipelines (Step 8 and Step 9) remain the steady-state daily reports; this pipeline is purpose-built for the "is the apply-updates run I started two hours ago still progressing?" question that those daily snapshots cannot answer between runs. | +| **When to run** | While a wave is active (manually, or via the commented cron). The two daily snapshot pipelines (Step 9 and Step 10) remain the steady-state daily reports; this pipeline is purpose-built for the "is the apply-updates run I started two hours ago still progressing?" question that those daily snapshots cannot answer between runs. | | **RBAC** | Read-only - `Reader` on the cluster scope, plus the cluster-update API read paths already enumerated in the `Azure Stack HCI Update Operator (custom)` custom role. | -| **ITSM** | Not supported - in-flight monitor exists to be a fast feedback loop during an active wave; long-running runs surface as JUnit `` entries for the operator on call, and post-wave failures are picked up + ticketed by the daily Step 8 snapshot. | +| **ITSM** | Not supported - in-flight monitor exists to be a fast feedback loop during an active wave; long-running runs surface as JUnit `` entries for the operator on call, and post-wave failures are picked up + ticketed by the daily Step 9 snapshot. | | **Introduced** | v0.7.90. Enhanced in v0.7.96 with the `Status` + `ErrorMessage` columns, the new `StepError` JUnit failure type, the always-shown Failed-runs block, portal-linked Cluster Name / Update Name cells, and the new `STEP_ERRORED` / `UNRESOLVED_FAILURES` `GITHUB_OUTPUT` values. **Behaviour change in v0.7.98:** UX overhaul of the in-flight table - rows now sort by composite `SeverityScore` (`StepError severity x 1000 + RunSeverity x 100 + elapsed-hours bucket`) so stuck step errors and runs over 14 days bubble to the top regardless of cluster alphabetical order; each row carries per-cell `StateIcon` + `StatusIcon` icons and a horizontal chip stack of flags (`STEP-STUCK`, `RUN-STUCK`, `UNRESOLVED`, `RECENT-FAIL`); the job summary opens with a single `CRITICAL / WARN / OK` fleet status badge that collapses the worst row across the fleet into one line (e.g. `CRITICAL - 1 stuck step error(s), 1 run(s) > 14d, 1 step(s) > 4h`); each failed step's `errorMessage` is now wrapped in a collapsible `
Verbose error...
` block; and JUnit `` + `` are populated with real per-run elapsed seconds (was `time="0"` before, which made GitHub Test Reporter render `"5 tests were completed in 0ms"`). | -## Step 8 - Fleet Update Status +## Step 9 - Fleet Update Status | Aspect | Value | |---|---| -| **Purpose** | Daily fleet-wide snapshot of cluster update state. Read-only. **v0.7.90 pivots the Version Distribution markdown table by YYMM** (leading column `Version` = `2511`, `2604`, ...; new `Update Versions` column lists each distinct full version installed within that YYMM as ` x ` separated by `
`; rows sorted ascending by YYMM so the oldest YYMM is at the top). The underlying `` JUnit XML is unchanged - still one `` per distinct full `CurrentVersion` - so machine-readable consumers and CI test-reporters are unaffected. **v0.7.96 reworks the Primary Status bucket cascade to align with the Azure portal Update Manager `state` filter:** `NeedsAttention` is now promoted into the **Update Failed** bucket; `PreparationFailed` lands in a new **Action Required** bucket (with its own actionable guidance in the Actions Required callout because the run never started, so re-arming is required); `PreparationInProgress` is folded into **Update In Progress**. The new bucket surfaces via (1) a new `` JUnit attribute on the primary testsuite, (2) per-testcase `failureType='PreparationFailed'` instead of the generic `UpdateFailure`, (3) a new `ACTION_REQUIRED` `GITHUB_OUTPUT` (`actionRequired` pipeline variable in ADO), (4) `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and (5) a new "Action Required (PreparationFailed)" row in the Primary Status markdown table with separate actionable prose in the Actions Required callout. The `📜 Update Run History and Error Details` markdown table renders Cluster Name and Update Name cells as `` deep-links (cluster's Updates blade + the single-instance update-run history view) - same linking as the Step.7 in-flight table. | +| **Purpose** | Daily fleet-wide snapshot of cluster update state. Read-only. **v0.7.90 pivots the Version Distribution markdown table by YYMM** (leading column `Version` = `2511`, `2604`, ...; new `Update Versions` column lists each distinct full version installed within that YYMM as ` x ` separated by `
`; rows sorted ascending by YYMM so the oldest YYMM is at the top). The underlying `` JUnit XML is unchanged - still one `` per distinct full `CurrentVersion` - so machine-readable consumers and CI test-reporters are unaffected. **v0.7.96 reworks the Primary Status bucket cascade to align with the Azure portal Update Manager `state` filter:** `NeedsAttention` is now promoted into the **Update Failed** bucket; `PreparationFailed` lands in a new **Action Required** bucket (with its own actionable guidance in the Actions Required callout because the run never started, so re-arming is required); `PreparationInProgress` is folded into **Update In Progress**. The new bucket surfaces via (1) a new `` JUnit attribute on the primary testsuite, (2) per-testcase `failureType='PreparationFailed'` instead of the generic `UpdateFailure`, (3) a new `ACTION_REQUIRED` `GITHUB_OUTPUT` (`actionRequired` pipeline variable in ADO), (4) `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and (5) a new "Action Required (PreparationFailed)" row in the Primary Status markdown table with separate actionable prose in the Actions Required callout. The `📜 Update Run History and Error Details` markdown table renders Cluster Name and Update Name cells as `` deep-links (cluster's Updates blade + the single-instance update-run history view) - same linking as the Step.8 in-flight table. | | **Inputs** | Scope (`-AllClusters` or `-ScopeByUpdateRingTag`), `throttle_limit` (optional). v0.7.4-style ITSM toggles are also exposed: `raise_itsm_ticket`, `itsm_config_path`, `itsm_dry_run`, `itsm_force_create` (all optional, default off). | | **Trigger** | Manual (`workflow_dispatch` / **Run pipeline** button) **plus** scheduled daily at 06:00 UTC (`cron '0 6 * * *'`). Edit the cron in the YAML to change cadence. | | **Cmdlets invoked** | `Get-AzLocalClusterInventory`, `Get-AzLocalClusterUpdateReadiness`, `Get-AzLocalUpdateSummary`, `Get-AzLocalAvailableUpdates`, `Get-AzLocalUpdateRuns`, `Get-AzLocalUpdateRunFailures`, `Get-AzLocalLatestSolutionVersion`. When ITSM is enabled: `Get-AzLocalItsmConfig`. | @@ -187,9 +205,9 @@ The table below is the ground truth for what each shipped YAML does **out of the | **RBAC** | Read-only. Covered by the `Azure Stack HCI Update Operator (custom)` custom role. | | **Exit conditions** | Pipeline run is green when the snapshot completes. Per-cluster issues (outdated version, failed run history) surface as JUnit `` entries. v0.7.96 distinguishes `failureType='UpdateFailure'` (run executed and failed - landed in `NeedsAttention`) from `failureType='PreparationFailed'` (run never started - landed in the new `Action Required` bucket) so downstream automation can split paging behaviour per failure mode. | | **ITSM** | Supported (opt-in via `raise_itsm_ticket=true` / `raiseItsmTicket=true`). When enabled, one ServiceNow incident is raised per unresolved failed update run picked up by the daily snapshot; dedupe key combines cluster + update version + failure reason via `./.itsm/azurelocal-itsm.yml`. `itsm_dry_run` builds payloads without creating tickets; `itsm_force_create` bypasses dedupe. | -| **Introduced** | Originally Step 7 in v0.7.4; renumbered to Step 8 in v0.7.90 when the in-flight monitor moved into Step 7. v0.7.90 added the YYMM-pivoted Version Distribution table. v0.7.96 promoted `NeedsAttention` into the `Update Failed` bucket, added the new `Action Required` bucket for `PreparationFailed`, folded `PreparationInProgress` into `Update In Progress`, added the `primaryActionRequired` JUnit attribute, the `ACTION_REQUIRED` `GITHUB_OUTPUT`, `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and portal-linked Cluster Name / Update Name cells in the `Update Run History and Error Details` markdown table. **Behaviour change in v0.7.98:** each `` inside the `📜 Update Run History and Error Details` testsuite now emits `time` = `DurationMinutes * 60` (`DurationMinutes` from KQL `datetime_diff('minute', EndTime, StartTime)` via `Get-AzLocalUpdateRunFailures`), and the testsuite-level `time=` is the sum of those per-row seconds. The other two testsuites (`FleetVersionDistribution`, `AzureLocalFleetUpdateStatus`) intentionally stay at `time="0"` - they are instantaneous snapshot projections and a synthetic duration would be misleading. | +| **Introduced** | Originally Step 7 in v0.7.4; renumbered to Step 8 in v0.7.90 when the in-flight monitor moved into Step 7; renumbered to Step 9 in v0.8.7 when the on-prem sideload pipeline took Step 6. v0.7.90 added the YYMM-pivoted Version Distribution table. v0.7.96 promoted `NeedsAttention` into the `Update Failed` bucket, added the new `Action Required` bucket for `PreparationFailed`, folded `PreparationInProgress` into `Update In Progress`, added the `primaryActionRequired` JUnit attribute, the `ACTION_REQUIRED` `GITHUB_OUTPUT`, `Summary.UpdateFailures` + `Summary.ActionRequired` in `readiness-status.json`, and portal-linked Cluster Name / Update Name cells in the `Update Run History and Error Details` markdown table. **Behaviour change in v0.7.98:** each `` inside the `📜 Update Run History and Error Details` testsuite now emits `time` = `DurationMinutes * 60` (`DurationMinutes` from KQL `datetime_diff('minute', EndTime, StartTime)` via `Get-AzLocalUpdateRunFailures`), and the testsuite-level `time=` is the sum of those per-row seconds. The other two testsuites (`FleetVersionDistribution`, `AzureLocalFleetUpdateStatus`) intentionally stay at `time="0"` - they are instantaneous snapshot projections and a synthetic duration would be misleading. | -## Step 9 - Fleet Health Status +## Step 10 - Fleet Health Status | Aspect | Value | |---|---| diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload-robocopy.md b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload-robocopy.md new file mode 100644 index 00000000..6f070b39 --- /dev/null +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload-robocopy.md @@ -0,0 +1,70 @@ +# Sideload copy throttling (robocopy) + +> Companion to [sideload.md](sideload.md). Covers the `SIDELOAD_ROBOCOPY_SWITCHES` +> repository variable that tunes the detached copy worker. + +The Step.6 sideload pipeline copies the `CombinedSolutionBundle` (or staged OEM SBE +package) to each cluster's infrastructure `import` SMB share using **robocopy**, run +inside a detached Windows Scheduled Task (`Tools/Invoke-AzLocalSideloadCopyTask.ps1`). +On a constrained on-prem link a full bundle can be tens of GB, so the copy is the longest +single operation in the whole workflow. + +The worker always supplies a safe baseline set of switches. Anything you put in the +`SIDELOAD_ROBOCOPY_SWITCHES` repository variable is **appended** to that baseline, so use +it to add **retry, wait, and throttling** controls appropriate to your network. + +## Default + +``` +/R:5 /W:30 +``` + +- `/R:5` - retry a failed file up to **5** times. +- `/W:30` - wait **30 seconds** between retries. + +This keeps a transient blip (a brief share hiccup, a momentary auth glitch) from failing +the whole copy, without retrying forever. + +## Recommended switches for constrained links + +| Switch | Effect | When to use | +|---|---|---| +| `/IPG:n` | **Inter-Packet Gap** - insert `n` milliseconds between packets to cap effective bandwidth. | The single most useful throttle. On a shared WAN/MPLS link, `/IPG:30`-`/IPG:75` keeps the copy from saturating the pipe and starving production traffic. Higher `n` = slower copy, less contention. | +| `/R:n` | Retry count. | Raise on a flaky link (`/R:10`); the default 5 is fine for most. | +| `/W:n` | Wait seconds between retries. | Raise (`/W:60`) when retries are usually due to a share that recovers slowly. | +| `/Z` | Restartable mode - resume a partially-copied large file after an interruption. | Large bundles over an unreliable link. Slightly slower but survives mid-file drops. | +| `/J` | Unbuffered I/O. | Very large files on a fast, reliable LAN - improves throughput. **Do not** combine with `/Z`. | +| `/MT[:n]` | Multi-threaded copy (`n` threads, default 8). | A bundle is one large file, so `/MT` gives little benefit here and can increase contention - usually leave it off. | + +## Examples + +Throttle a shared WAN link to leave headroom for production traffic, with generous +retries: + +``` +/IPG:50 /R:10 /W:60 +``` + +Restartable copy over an unreliable link: + +``` +/Z /R:10 /W:60 +``` + +Fast, reliable LAN (no throttle needed): + +``` +/R:5 /W:30 /J +``` + +## Notes + +- Logging (`/LOG`, `/TEE`), source/destination paths, and the file selection are managed + by the worker - **do not** set those in `SIDELOAD_ROBOCOPY_SWITCHES`; only add the + tuning switches above. +- A long copy is **expected** and does not hold a pipeline run open - the copy runs in + the detached Scheduled Task while short, frequent pipeline runs report `Copying` + progress via the shared-state heartbeat (see [sideload.md section 2](sideload.md#2-re-entrant-state-machine--scheduled-task-survival-model)). +- If a copy stalls past `SIDELOAD_HEARTBEAT_STALE_MINUTES`, the state machine re-drives it + on the next live host (bounded by `MaxRetries`). Tune the stale window up if a slow but + healthy `/IPG`-throttled copy is being re-driven prematurely. diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload.md b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload.md new file mode 100644 index 00000000..7cd7a6e4 --- /dev/null +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/docs/sideload.md @@ -0,0 +1,220 @@ +# Sideload Updates (on-prem, opt-in) - Step 6 + +> **Introduced in v0.8.7.** Opt-in, off by default. This pipeline is inert unless the +> repository variable `SIDELOAD_UPDATES` is the literal string `'true'`. + +The **Sideload Updates** pipeline (`sideload-updates.yml`, logical pipeline id +`sideload-updates`, displayed as **Step.6**) pre-stages Azure Local solution-update +media onto clusters that **cannot pull updates from Azure directly** - dark, +air-gapped, or restricted-egress fabrics. Once the media is staged, verified, and +imported, the pipeline flips the `UpdateSideloaded=True` gate so the downstream +**Step.7 - Apply Updates** pipeline can proceed exactly as it does for +internet-connected clusters. + +For the per-pipeline reference card (inputs, artefacts, RBAC, exit conditions) see +[appendix-pipelines.md - Step 6](appendix-pipelines.md#step-6---sideload-updates). +For robocopy throttling guidance see [sideload-robocopy.md](sideload-robocopy.md). + +--- + +## 1. Why a self-hosted runner / agent + +The runner (GitHub Actions) or agent (Azure DevOps) must sit on the **same network as +the target clusters** so it can: + +1. **Robocopy** the `CombinedSolutionBundle` (or OEM SBE package) to each cluster's + infrastructure `import` SMB share. +2. **PowerShell-remote (WinRM)** into a cluster node to verify the SHA256 and run + `Add-SolutionUpdate`. + +Microsoft-hosted / cloud-hosted runners cannot reach the on-prem fabric, so this +pipeline targets: + +- **GitHub Actions** - a self-hosted **runner** labelled `azlocal-sideload` + (`runs-on: [self-hosted, azlocal-sideload]`). +- **Azure DevOps** - a self-hosted **agent** in a pool that satisfies the + `azlocal-sideload` demand (`pool: { name: , demands: azlocal-sideload }`). + +> Terminology: GitHub uses **runners** (in runner groups); Azure DevOps uses **agents** +> (in agent pools). This is never an AKS "node pool". + +--- + +## 2. Re-entrant state machine + scheduled-task survival model + +A solution-bundle copy can take **hours** over a constrained on-prem link. A pipeline +run that blocked for that long would burn agent time, hit job timeouts, and lose all +progress if the agent restarted. Instead, the pipeline is a **re-entrant state +machine**: + +- Each pipeline run advances **every in-scope cluster by exactly one transition**, then + exits. No run is ever long-lived. +- The multi-hour copy itself runs in a **detached Windows Scheduled Task** (driven by + the bundled `Tools/Invoke-AzLocalSideloadCopyTask.ps1` worker) that **survives** the + pipeline run ending and the agent process recycling. +- Progress is persisted in **shared-UNC state JSON**, so any runner/agent can read the + state and advance/report without cross-agent remoting. + +Drive the pipeline on a **frequent CRON (every 30 minutes)** so successive short runs +walk each cluster through the states: + +``` +Planned -> Copying -> Copied -> Verified -> Imported -> SideloadFlagged +``` + +Per cluster, the current shared state determines the action taken by +`Invoke-AzLocalSideloadUpdate`: + +| Current state | Action | +|---|---| +| (no state) + due now | Set `UpdateSideloaded=False`, ensure the verified bundle is in the shared cache, register + start the detached copy Scheduled Task, write `Copying` state. | +| `Copying` + fresh heartbeat | Report progress, leave the task running. | +| `Copying` + **stale** heartbeat (> `SIDELOAD_HEARTBEAT_STALE_MINUTES`) | Treat the host as dead and re-drive on the current (live) host, up to `MaxRetries`. | +| `Copied` | Open a WinRM session, verify the remote SHA256, run `Add-SolutionUpdate` + discovery, flip `UpdateSideloaded=True` + stamp `UpdateVersionInProgress`, mark `Imported`, remove the task. | +| `Failed` | Bounded retry, else surface the error as a JUnit ``. | +| `Imported` | Done - nothing to do. | + +Re-running is **always safe** - the state machine is idempotent. + +--- + +## 3. Shared state and multi-runner / multi-agent contract + +`SIDELOAD_STATE_ROOT` must be a **UNC path that every runner/agent can read and write**. +It holds three subfolders: + +- `state\` - one JSON document per cluster tracking the current transition + heartbeat. +- `logs\` - per-run copy / verify / import logs. +- `cache\` - the verified media cache (overridable via `SIDELOAD_CACHE_ROOT`; defaults + to `\cache`). + +Because a bundle is downloaded and hashed **once** into the shared cache and then reused +across every cluster that needs that version, only the first cluster pays the download + +hash cost. The pipeline's `concurrency` / queueing settings serialize overlapping runs so +they do not race on the same shared state. + +--- + +## 4. Authentication + +Two **distinct** identities are used - do not conflate them: + +1. **Pipeline identity** (Azure plane) - reads the fleet via Azure Resource Graph, reads + the Key Vault secrets, and writes the `UpdateSideloaded` / `UpdateVersionInProgress` + tags. Defaults to `azure/login` OIDC; `enable-AzPSSession=true` is required because + the Key Vault secrets are read via `Get-AzKeyVaultSecret` (Az PowerShell). For on-prem + runners where OIDC is not viable, switch via the `SIDELOAD_KV_AUTH` variable + (`oidc | managedidentity | serviceprincipal`). Tag writes need only **Tag Contributor** + (they reuse `Set-AzLocalClusterTagsMerge`). +2. **Cluster WinRM credential** (fabric plane) - an Active Directory `[pscredential]` + built at run time from **two Key Vault secrets** named in the matching sideload + auth-map row (a username secret + a password secret). This credential - **not** the + pipeline identity - is used for the WinRM session and `Add-SolutionUpdate`. The + detached copy Scheduled Task runs as the runner service account, which needs UNC + rights to the shared cache and the cluster import share. + +### 4.1 Sideload auth-map CSV (`SIDELOAD_AUTH_MAP_PATH`) + +Maps the numeric `UpdateAuthAccountId` tag (written onto clusters by Step.2) to the Key +Vault + secret names that hold the AD credential: + +```csv +UpdateAuthAccountId,KeyVaultName,UsernameSecretName,PasswordSecretName +1,kv-fabric-east,sideload-user,sideload-pass +2,kv-fabric-west,sideload-user,sideload-pass +``` + +- `UpdateAuthAccountId` must match `^\d{1,3}$` (numeric, 1-3 digits) and be **unique** (a + duplicate is a hard error). +- All four columns are required. + +--- + +## 5. Catalog (`SIDELOAD_CATALOG_PATH`) + +A source-controlled YAML describing the media available to the automation. Two package +classes are supported via `packageType`: + +- **Solution** - a Microsoft `CombinedSolutionBundle..zip` downloadable from a + direct `downloadUri` (published in the Microsoft Learn "import and discover updates + offline" table). `sha256` is **required** so the download / pre-staged copy can be + verified. `Update-AzLocalSideloadCatalog` can auto-populate these rows by parsing the + Learn table. +- **SBE** - an OEM Solution Builder Extension package that Microsoft does **not** host. + The operator stages the OEM files manually and records a `sourceFolder` (local or UNC + path). `downloadUri` is not applicable; `sha256` is optional (verified only when + supplied). These rows are added **manually**. + +```yaml +schemaVersion: 1 +packages: + - version: '12.2605.1003.210' + packageType: Solution + buildNumber: '12.2605.1003.210' + osBuild: '26100.4061' + downloadUri: 'https://.../CombinedSolutionBundle.12.2605.1003.210.zip' + sha256: 'ABCD...' # required for Solution; ^[0-9A-Fa-f]{64}$ + availabilityDate: '2026-05-13' + localPath: '' + - version: 'DellSBE-4.1.2412.1' + packageType: SBE + sourceFolder: '\\fileserver\sbe\Dell\4.1.2412.1' + sha256: '' # optional for SBE + availabilityDate: '2026-05-20' + notes: 'Dell OEM SBE package, staged manually' +``` + +--- + +## 6. Configuration (repository variables) + +| Variable | Default | Purpose | +|---|---|---| +| `SIDELOAD_UPDATES` | `false` | **Master gate.** The job is skipped unless this is the literal `'true'`. | +| `SIDELOAD_STATE_ROOT` | (none) | Shared UNC root holding `state\`, `logs\`, `cache\`. **Required** when enabled. | +| `SIDELOAD_CACHE_ROOT` | `\cache` | Shared verified media cache. | +| `SIDELOAD_AUTH_MAP_PATH` | `./config/sideload-auth-map.csv` | Auth-map CSV (see 4.1). `Copy-AzLocalPipelineExample` drops a header-only starter here (same `config/` folder on GitHub and Azure DevOps). | +| `SIDELOAD_CATALOG_PATH` | `./config/sideload-catalog.yml` | Catalog YAML (see 5). `Copy-AzLocalPipelineExample` drops an empty skeleton starter here. | +| `SIDELOAD_LEAD_DAYS` | `7` | Days before a cluster's next apply window that media should be sideloaded. | +| `SIDELOAD_ROBOCOPY_SWITCHES` | `/R:5 /W:30` | Extra robocopy switches for the detached worker (see [sideload-robocopy.md](sideload-robocopy.md)). | +| `SIDELOAD_HEARTBEAT_STALE_MINUTES` | `60` | Minutes after which a `Copying` heartbeat is considered stale and re-driven. | +| `SIDELOAD_REMOTING_FQDN_SUFFIX` | (empty) | Global FQDN suffix appended to a cluster name to form the WinRM host when the auth-map row does not override it. | +| `SIDELOAD_KV_AUTH` | `oidc` | Key Vault auth mode for the on-prem runner. | +| `APPLY_UPDATES_SCHEDULE_PATH` | `./config/apply-updates-schedule.yml` | Ring-aware apply-updates schedule; the planner reads it to find each cluster's next apply window. | + +--- + +## 7. Cmdlets + +| Cmdlet | Role | +|---|---| +| `Resolve-AzLocalSideloadPlan` | Read-only planner. Reads the fleet (ARG), apply schedule, auth-map, and catalog; emits one plan row per `UpdateAuthAccountId`-tagged cluster (plus error rows for misconfigurations) and marks which clusters are due within `LeadDays`. Reuses the same "next update" selection as the apply path. | +| `Invoke-AzLocalSideloadUpdate` | The re-entrant state machine (section 2). `SupportsShouldProcess` - `-WhatIf` previews transitions with no staging / task / tag changes. | +| `Export-AzLocalSideloadStatusReport` / `Add-AzLocalSideloadStepSummary` | JUnit XML + Markdown step-summary emitters. | +| `Update-AzLocalSideloadCatalog` | Auto-populates Solution rows by parsing the Microsoft Learn offline-updates table. SBE rows are added manually. | +| `Reset-AzLocalSideloadedTag` | Operator escape hatch to clear a stuck `UpdateSideloaded` tag. | + +--- + +## 8. End-to-end runbook + +1. **Stand up the runner/agent** on the cluster fabric network and label it + `azlocal-sideload` (GH) / give the pool the `azlocal-sideload` demand (ADO). +2. **Create the shared UNC root** (`SIDELOAD_STATE_ROOT`) readable + writable by the + runner service account, and grant that account rights to each cluster's import share. +3. **Populate Key Vault** with the per-fabric AD username/password secrets and author + `sideload-auth-map.csv`. +4. **Author `sideload-catalog.yml`** - run `Update-AzLocalSideloadCatalog` to fill the + Microsoft Solution rows, then add any OEM SBE rows manually. +5. **Tag the fleet** (Step.1 / Step.2): set `UpdateRing`, `UpdateStartWindow`, and + `UpdateAuthAccountId` on each sideloaded cluster. +6. **Set the repository variables** (section 6), starting with `SIDELOAD_UPDATES=true`. +7. **Dry run**: trigger the pipeline manually with `dry_run=true` and review the planned + transitions + the `sideload-status` artefacts. +8. **Enable the CRON**: uncomment the bundled `*/30 * * * *` schedule inside the + `BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers` block (preserved across + `Update-AzLocalPipelineExample` upgrades). The Step.3 schedule-coverage audit can + recommend a lead-time-aware cron (apply window minus `SIDELOAD_LEAD_DAYS`). +9. The state machine advances each cluster to `Imported` / `SideloadFlagged`; the + downstream **Step.7 - Apply Updates** wave then applies the staged update during the + cluster's `UpdateStartWindow`. diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.3_apply-updates-schedule-audit.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml similarity index 93% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.3_apply-updates-schedule-audit.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml index 49891ebf..1b5ad85f 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.3_apply-updates-schedule-audit.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates-schedule-audit.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: apply-updates-schedule-audit # Apply-Updates Schedule Coverage Audit # This workflow runs the read-only Test-AzLocalApplyUpdatesScheduleCoverage advisor # weekly to detect drift between the cron schedule(s) in your apply-updates pipeline @@ -5,13 +6,13 @@ # # WHY THIS PIPELINE EXISTS # ------------------------ -# Step.6_apply-updates.yml ships with NO default schedule (workflow_dispatch only) on purpose, +# Step.7_apply-updates.yml ships with NO default schedule (workflow_dispatch only) on purpose, # so customers must consciously choose when updates fire. After you have configured -# UpdateStartWindow tags on your clusters and added cron entries to Step.6_apply-updates.yml, this +# UpdateStartWindow tags on your clusters and added cron entries to Step.7_apply-updates.yml, this # pipeline is the safety net that catches: -# - a new ring tagged with UpdateStartWindow that no cron in Step.6_apply-updates.yml will reach +# - a new ring tagged with UpdateStartWindow that no cron in Step.7_apply-updates.yml will reach # - an UpdateStartWindow tag value that fails to parse (typo / wrong syntax) -# - Step.6_apply-updates.yml cron entries that the advisor cannot reason about +# - Step.7_apply-updates.yml cron entries that the advisor cannot reason about # (DayOfMonth restrictions, step values) so you can audit them manually # # REPORTS GENERATED @@ -46,13 +47,13 @@ on: workflow_dispatch: inputs: pipeline_path: - description: 'Path (file or folder) to Step.6_apply-updates.yml to audit, repo-relative. Required so the Recommend view can diff its proposed crons against what is already present in Step.6 and only emit a snippet for the truly missing entries. Default `.github/workflows` matches the standard GitHub Actions consumer layout where you paste the bundled `Step.6_apply-updates.yml` sample.' + description: 'Path (file or folder) to Step.7_apply-updates.yml to audit, repo-relative. Required so the Recommend view can diff its proposed crons against what is already present in Step.7 and only emit a snippet for the truly missing entries. Default `.github/workflows` matches the standard GitHub Actions consumer layout where you paste the bundled `Step.7_apply-updates.yml` sample.' required: true default: '.github/workflows' schedule_path: description: 'Path to apply-updates-schedule.yml. When set, the audit also reports RingMissingFromSchedule (fleet ring with no schedule row) and RingOrphanedInSchedule (schedule ring no cluster carries). Leave empty to skip the schedule-file two-way diff.' required: false - default: '.github/apply-updates-schedule.yml' + default: 'config/apply-updates-schedule.yml' lead_time_minutes: description: 'Minutes before UpdateStartWindow opens that the pipeline should fire (0-60). Default 5.' required: false @@ -95,7 +96,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' REQUIRED_MODULE_VERSION: ${{ github.event.inputs.module_version || vars.REQUIRED_MODULE_VERSION || '' }} # v0.8.4 - opt this workflow into Node.js 24 for all JavaScript actions # (actions/checkout, actions/download-artifact, actions/upload-artifact, @@ -201,13 +202,17 @@ jobs: id: audit env: INPUT_PIPELINE_PATH: ${{ github.event.inputs.pipeline_path || '.github/workflows' }} - INPUT_SCHEDULE_PATH: ${{ github.event.inputs.schedule_path || '.github/apply-updates-schedule.yml' }} + INPUT_SCHEDULE_PATH: ${{ github.event.inputs.schedule_path || 'config/apply-updates-schedule.yml' }} INPUT_LEAD_TIME_MINUTES: ${{ github.event.inputs.lead_time_minutes || '5' }} INPUT_FIRES_PER_WINDOW: ${{ github.event.inputs.fires_per_window || '2' }} INPUT_INCLUDE_UNTAGGED: ${{ github.event.inputs.include_untagged || 'false' }} INPUT_CLUSTER_CSV_PATH: ${{ github.event.inputs.cluster_csv_path || 'config/ClusterUpdateRings.csv' }} INPUT_DEBUG: ${{ github.event.inputs.debug || 'false' }} INSTALLED_MODULE_VERSION: ${{ steps.module-version.outputs.installed_module_version }} + # v0.8.7 sideload advisor (opt-in). When SIDELOAD_UPDATES=true the audit + # appends a "Recommended sideload schedule" section. Inert when unset/false. + SIDELOAD_UPDATES: ${{ vars.SIDELOAD_UPDATES || 'false' }} + SIDELOAD_LEAD_DAYS: ${{ vars.SIDELOAD_LEAD_DAYS || '7' }} run: | $ErrorActionPreference = 'Stop' Import-Module AzLocal.UpdateManagement -Force diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.6_apply-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.6_apply-updates.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates.yml index e86a6b8e..54be6089 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.6_apply-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/apply-updates.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: apply-updates # Apply Updates to Azure Local Clusters # This workflow applies updates to clusters filtered by UpdateRing tag value # @@ -11,12 +12,12 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: Step.6 - Apply Updates +name: Step.7 - Apply Updates on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers # PREFERRED PATTERN - drive automated update runs from a ring-aware - # schedule file (apply-updates-schedule.yml at the repo root by default), + # schedule file (config/apply-updates-schedule.yml by default), # not from per-window crons. # # How it works: @@ -31,7 +32,7 @@ on: # at New-AzLocalApplyUpdatesScheduleConfig. # # Generate a STRAWMAN schedule file from your fleet via: - # New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\.github\apply-updates-schedule.yml + # New-AzLocalApplyUpdatesScheduleConfig -OutputPath .\config\apply-updates-schedule.yml # Every row is commented out by default - review and uncomment to activate. # # Recommended cron: once per day at a low-traffic time (the resolver is @@ -130,7 +131,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). @@ -139,7 +140,7 @@ env: # root). The resolver step in check-readiness reads this file when the trigger # is `schedule:`. Override via repository variable APPLY_UPDATES_SCHEDULE_PATH # if you keep the schedule outside `.github/`. - APPLY_UPDATES_SCHEDULE_PATH: ${{ vars.APPLY_UPDATES_SCHEDULE_PATH || './.github/apply-updates-schedule.yml' }} + APPLY_UPDATES_SCHEDULE_PATH: ${{ vars.APPLY_UPDATES_SCHEDULE_PATH || './config/apply-updates-schedule.yml' }} # v0.8.4 - opt this workflow into Node.js 24 for all JavaScript actions # (actions/checkout, actions/download-artifact, actions/upload-artifact, # azure/login, dorny/test-reporter, etc). Per GitHub's 2025-09-19 deprecation @@ -174,7 +175,7 @@ jobs: # surface the timestamped readiness-artifact name so the downstream # apply-updates job can download it (cross-job step outputs are not visible - # everything passed to the next job has to go through the job 'outputs' map). - readiness_artifact: azlocal-step.6-apply-updates-readiness-report_${{ steps.artifact-stamp.outputs.timestamp }} + readiness_artifact: azlocal-step.7-apply-updates-readiness-report_${{ steps.artifact-stamp.outputs.timestamp }} # the ring(s) the schedule resolver picked for this firing. # Empty when the trigger was workflow_dispatch (downstream falls back to # the manual `update_ring` input). When the trigger was `schedule:`, this @@ -273,7 +274,7 @@ jobs: - name: Upload Readiness Report uses: actions/upload-artifact@v6 with: - name: azlocal-step.6-apply-updates-readiness-report_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.7-apply-updates-readiness-report_${{ steps.artifact-stamp.outputs.timestamp }} path: ./artifacts/readiness-report.csv retention-days: 30 @@ -396,7 +397,7 @@ jobs: - name: Upload Update Logs uses: actions/upload-artifact@v6 with: - name: azlocal-step.6-apply-updates-logs_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.7-apply-updates-logs_${{ steps.artifact-stamp.outputs.timestamp }} path: ./artifacts/* retention-days: 30 @@ -466,7 +467,7 @@ jobs: if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} uses: actions/upload-artifact@v6 with: - name: azlocal-step.6-apply-updates-itsm-results_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.7-apply-updates-itsm-results_${{ steps.artifact-stamp.outputs.timestamp }} path: ./artifacts/itsm-*.* retention-days: 30 continue-on-error: true diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.5_assess-update-readiness.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/assess-update-readiness.yml similarity index 98% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.5_assess-update-readiness.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/assess-update-readiness.yml index 58344219..cefa5a32 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.5_assess-update-readiness.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/assess-update-readiness.yml @@ -1,7 +1,8 @@ +# AZLOCAL-PIPELINE-ID: assess-update-readiness # Assess Update Readiness (Pre-flight go/no-go gate) # -------------------------------------------------- # Runs Get-AzLocalClusterUpdateReadiness and Test-AzLocalClusterHealth -BlockingOnly -# against a target UpdateRing (or the whole fleet) BEFORE Step.6_apply-updates.yml runs. +# against a target UpdateRing (or the whole fleet) BEFORE Step.7_apply-updates.yml runs. # # Outputs two JUnit XML diagnostic artifacts consumed by the Actions test reporter: # - readiness.xml (one test per cluster; fails if ReadyForUpdate = $false) @@ -16,7 +17,7 @@ # summary - but it does NOT block downstream runs. In large fleets, day-to-day # environmental issues (transient storage noise, a single node out, etc.) routinely affect # a small subset of clusters; blocking the entire wave for one unhealthy cluster is rarely -# the desired behavior. Step.6_apply-updates.yml is itself per-cluster scoped, so clusters that +# the desired behavior. Step.7_apply-updates.yml is itself per-cluster scoped, so clusters that # are not ready will simply no-op there too. # # If you want a hard pass/fail signal for a chained gate, read the job outputs @@ -71,7 +72,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.0_authentication-test.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/authentication-test.yml similarity index 99% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.0_authentication-test.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/authentication-test.yml index 2a0f6ac6..834d8d97 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.0_authentication-test.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/authentication-test.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: authentication-test # Step.0 - Authentication Validation and Subscription Scope Report # # PURPOSE: @@ -78,7 +79,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.4_fleet-connectivity-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.4_fleet-connectivity-status.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml index f4119703..a1a07faa 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.4_fleet-connectivity-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: fleet-connectivity-status # Fleet Connectivity Status Monitoring # This workflow surfaces network-connectivity issues across every Azure Local # cluster the service principal can read - four scopes in one pipeline: @@ -12,7 +13,7 @@ # - Find clusters where Arc agent or ARB has dropped off and updates / management calls will fail # - Catch Disconnected physical NICs that still have a real IP (likely cable / driver / port issues - # Disconnected adapters with no IP or APIPA self-assignment are intentionally excluded as noise) -# - Triage hand-off into ITSM (wired identically to Step.8 / Step.9 - opt-in via input) +# - Triage hand-off into ITSM (wired identically to Step.9 / Step.10 - opt-in via input) # # REPORTS GENERATED: # - Markdown step summary: one section per scope (cluster, ARB, Arc agents, NICs) @@ -55,7 +56,7 @@ on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers # Edits inside this block are preserved across Update-AzLocalPipelineExample # module upgrades. Add or modify `schedule:` cron blocks below. - # Daily at 05:30 UTC - offset 30 min ahead of Step.8 (06:00) and Step.9 (07:00) + # Daily at 05:30 UTC - offset 30 min ahead of Step.9 (06:00) and Step.10 (07:00) # so the three fleet-scope read pipelines stagger their ARG calls. schedule: - cron: '30 5 * * *' # Daily at 05:30 UTC @@ -115,7 +116,7 @@ env: # PSGallery, and emits a ::notice annotation if the YAML appears stale - # prompting you to refresh via Copy-AzLocalPipelineExample -Update. See # Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install # the latest, the default "fix-forward" behaviour): manual workflow_dispatch # input > repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). @@ -262,7 +263,7 @@ jobs: # ---------------------------------------------------------------------- # ITSM Connector (ServiceNow auto-raise on connectivity failures) # Opt-in (gated on inputs.raise_itsm_ticket == 'true'). Mirrors the - # Step.8 / Step.9 wiring - reads the JUnit file produced above, uses + # Step.9 / Step.10 wiring - reads the JUnit file produced above, uses # the same azurelocal-itsm.yml trigger matrix, same dedupe key. # ---------------------------------------------------------------------- - name: Install powershell-yaml (ITSM config parser) diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.9_fleet-health-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-health-status.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.9_fleet-health-status.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-health-status.yml index 11665452..2a41339b 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.9_fleet-health-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-health-status.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: fleet-health-status # Fleet Health Status Monitoring # This workflow surfaces 24-hour system health-check failures across every Azure Local # cluster the service principal can read - including clusters that are already @@ -31,7 +32,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: Step.9 - Fleet Health Status +name: Step.10 - Fleet Health Status on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -115,7 +116,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). @@ -225,7 +226,7 @@ jobs: - name: Upload Fleet Health Reports uses: actions/upload-artifact@v6 with: - name: azlocal-step.9-fleet-health-status-report_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.10-fleet-health-status-report_${{ steps.artifact-stamp.outputs.timestamp }} path: ./reports/ retention-days: 90 @@ -327,7 +328,7 @@ jobs: if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} uses: actions/upload-artifact@v6 with: - name: azlocal-step.9-fleet-health-status-itsm-results_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.10-fleet-health-status-itsm-results_${{ steps.artifact-stamp.outputs.timestamp }} path: ./reports/itsm-*.* retention-days: 30 continue-on-error: true @@ -336,7 +337,7 @@ jobs: if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} uses: dorny/test-reporter@v3 with: - name: ITSM Tickets (Step.9) + name: ITSM Tickets (Step.10) path: ./reports/itsm-results.xml reporter: java-junit continue-on-error: true diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.8_fleet-update-status.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-update-status.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.8_fleet-update-status.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-update-status.yml index 819d3b81..c1fe95a8 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.8_fleet-update-status.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/fleet-update-status.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: fleet-update-status # Fleet Update Status Monitoring # This workflow monitors update status across all Azure Local clusters and generates reports # @@ -23,7 +24,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the eight pipelines in execution order. -name: Step.8 - Fleet Update Status +name: Step.9 - Fleet Update Status on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -68,7 +69,7 @@ on: # Failed update run reported by the '📜 Update Run History and Error Details' # testsuite (and any failed AzureLocalFleetUpdateStatus row). Default is false # so existing schedules stay byte-identical until you opt in. The connector reads - # the JUnit file Step.6 already produces (./reports/readiness-status.xml) and + # the JUnit file Step.7 already produces (./reports/readiness-status.xml) and # the trigger matrix in ./.itsm/azurelocal-itsm.yml. raise_itsm_ticket: description: 'Open ITSM tickets (ServiceNow) for fleet-update-status failures' @@ -104,7 +105,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). @@ -205,7 +206,7 @@ jobs: id: artifact-stamp shell: pwsh # every downloadable artifact gets a UTC timestamp suffix so multiple runs on - # the same day produce distinct zip names (e.g. azlocal-step.8-fleet-update-status-report_20260518_143217.zip). + # the same day produce distinct zip names (e.g. azlocal-step.9-fleet-update-status-report_20260518_143217.zip). run: | $stamp = (Get-Date).ToUniversalTime().ToString('yyyyMMdd_HHmmss') "timestamp=$stamp" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append @@ -214,7 +215,7 @@ jobs: - name: Upload Fleet Status Reports uses: actions/upload-artifact@v6 with: - name: azlocal-step.8-fleet-update-status-report_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.9-fleet-update-status-report_${{ steps.artifact-stamp.outputs.timestamp }} path: ./reports/ retention-days: 90 @@ -317,7 +318,7 @@ jobs: if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} uses: actions/upload-artifact@v6 with: - name: azlocal-step.8-fleet-update-status-itsm-results_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.9-fleet-update-status-itsm-results_${{ steps.artifact-stamp.outputs.timestamp }} path: ./reports/itsm-*.* retention-days: 30 continue-on-error: true @@ -326,7 +327,7 @@ jobs: if: ${{ github.event.inputs.raise_itsm_ticket == 'true' }} uses: dorny/test-reporter@v3 with: - name: ITSM Tickets (Step.6) + name: ITSM Tickets (Step.9) path: ./reports/itsm-results.xml reporter: java-junit continue-on-error: true diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.1_inventory-clusters.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/inventory-clusters.yml similarity index 98% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.1_inventory-clusters.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/inventory-clusters.yml index e8024e88..912a4e5f 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.1_inventory-clusters.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/inventory-clusters.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: inventory-clusters # Inventory Azure Local Clusters # This workflow queries all Azure Local clusters and exports inventory with UpdateRing tag status # @@ -47,7 +48,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.2_manage-updatering-tags.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/manage-updatering-tags.yml similarity index 99% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.2_manage-updatering-tags.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/manage-updatering-tags.yml index 8cdd6600..c4c2fd53 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.2_manage-updatering-tags.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/manage-updatering-tags.yml @@ -1,3 +1,4 @@ +# AZLOCAL-PIPELINE-ID: manage-updatering-tags # Manage UpdateRing Tags # This workflow creates or updates UpdateRing tags on Azure Local clusters from a CSV file # @@ -64,7 +65,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.7_monitor-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/monitor-updates.yml similarity index 97% rename from AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.7_monitor-updates.yml rename to AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/monitor-updates.yml index b6129512..c62e0136 100644 --- a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/Step.7_monitor-updates.yml +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/monitor-updates.yml @@ -1,9 +1,10 @@ +# AZLOCAL-PIPELINE-ID: monitor-updates # Monitor In-Flight Updates (operational monitoring snapshot) # ----------------------------------------------------------- # Reports clusters whose latest update run is currently in flight, with the # CURRENT STEP each cluster is on and the ELAPSED DURATION of the run. Built # for operators who want a 30-minute-cadence "what is happening right now" -# view, distinct from the daily Step.8 fleet-update-status snapshot which +# view, distinct from the daily Step.9 fleet-update-status snapshot which # answers "is each cluster up-to-date?". # # This workflow is REPORT-ONLY and always succeeds. Long-running runs are @@ -22,7 +23,7 @@ # Workflow name carries the same Step.N - prefix as the filename so the GitHub # Actions sidebar (which sorts workflows alphabetically by this `name:` field) # lists the pipelines in execution order. -name: Step.7 - Monitor In-Flight Updates +name: Step.8 - Monitor In-Flight Updates on: # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers @@ -85,7 +86,7 @@ env: # this to the version actually installed and to the latest on PSGallery, and emits a # ::notice annotation if the YAML appears stale - prompting you to refresh via # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. - GENERATED_AGAINST_MODULE_VERSION: '0.8.6' + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' # Resolution order for the module version pin (leave all unset to install the latest, # which is the default "fix-forward" behaviour): manual workflow_dispatch input > # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). @@ -214,7 +215,7 @@ jobs: if: always() uses: actions/upload-artifact@v6 with: - name: azlocal-step.7-update-monitor_${{ steps.artifact-stamp.outputs.timestamp }} + name: azlocal-step.8-update-monitor_${{ steps.artifact-stamp.outputs.timestamp }} path: ./reports/* retention-days: 14 diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/sideload-updates.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/sideload-updates.yml new file mode 100644 index 00000000..3f3d6537 --- /dev/null +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/github-actions/sideload-updates.yml @@ -0,0 +1,273 @@ +# AZLOCAL-PIPELINE-ID: sideload-updates +# Sideload Updates (Opt-in, on-prem self-hosted runner) +# +# OPT-IN, OFF BY DEFAULT. This workflow automates the on-prem sideloading of +# Azure Local solution-update media for clusters that cannot pull updates from +# Azure directly. It is the NEW Step.6 introduced in v0.8.7; the former Step.6 +# (Apply Updates) is now Step.7. +# +# WHY A SELF-HOSTED RUNNER: +# The runner must sit on the SAME network as the target Azure Local clusters +# so it can (a) Robocopy the CombinedSolutionBundle to each cluster's +# infrastructure `import` SMB share and (b) PowerShell-remote (WinRM) into a +# cluster node to verify the SHA256 and run Add-SolutionUpdate. GitHub-hosted +# runners cannot reach the on-prem fabric, so this pipeline targets a +# self-hosted runner labelled `azlocal-sideload`. +# +# RE-ENTRANT STATE MACHINE: +# Each run advances every in-scope cluster by ONE state transition and exits. +# The multi-hour copy itself runs in a detached Windows Scheduled Task, so no +# pipeline run is ever long-lived. Drive this on a frequent CRON (e.g. every +# 30 minutes) so the state machine progresses Copying -> Copied -> Verified -> +# Imported across successive short runs. Re-running is always safe. +# +# GATE: the whole job is skipped unless the repository variable +# SIDELOAD_UPDATES == 'true'. When unset/false this file is inert. +# +# AUTHENTICATION: +# - Azure (ARG fleet read + Key Vault secret read + UpdateSideloaded tag flip): +# azure/login OIDC by default. enable-AzPSSession=true is REQUIRED because +# the Key Vault secrets are read via Get-AzKeyVaultSecret (Az PowerShell). +# For on-prem runners where OIDC is not viable, switch to managed identity +# or a service principal per the SIDELOAD_KV_AUTH variable (see comments). +# - Cluster WinRM remoting: an Active Directory [pscredential] built from two +# Key Vault secrets named in the sideload auth-map row (NOT the pipeline +# identity). The detached copy Scheduled Task runs as the runner service +# account (needs UNC rights to the shared cache + cluster import share). + +# Workflow name carries the same Step.N - prefix as the filename so the GitHub +# Actions sidebar (which sorts workflows alphabetically by this `name:` field) +# lists the pipelines in execution order. +name: Step.6 - Sideload Updates (Opt-in) + +on: + # BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers + # Edits inside this block are preserved across Update-AzLocalPipelineExample + # module upgrades. + # + # NO default schedule ships out-of-the-box (this pipeline requires an on-prem + # self-hosted runner that most repos do not have). Once your `azlocal-sideload` + # runner is online, UNCOMMENT the cron below to drive the re-entrant state + # machine. A frequent poll (every 30 min) lets a single short run advance the + # multi-hour copy without staying alive. The Step.3 schedule-coverage audit can + # recommend a lead-time-aware cron (apply window minus SIDELOAD_LEAD_DAYS). + # + # schedule: + # - cron: '*/30 * * * *' # every 30 minutes - advance/report sideload state + # END-AZLOCAL-CUSTOMIZE:schedule-triggers + + workflow_dispatch: + inputs: + update_ring: + # accepts a single ring (Wave1), a semicolon-delimited list (Prod;Ring2), + # or '***' (three stars - deliberate, not a typo) to match every cluster + # that HAS the UpdateRing tag set. + description: "UpdateRing tag value to scope the plan ('Wave1', 'Prod;Ring2', or '***' for ALL rings)." + required: false + default: '***' + dry_run: + description: 'Preview the plan + transitions without staging media, registering tasks, or flipping tags (WhatIf).' + required: false + default: 'true' + type: choice + options: + - 'true' + - 'false' + module_version: + description: 'Pin AzLocal.UpdateManagement version (empty = latest from PSGallery). See Automation-Pipeline-Examples/README.md section 5 "Optional configuration".' + required: false + default: '' + +env: + # Module version this workflow YAML was generated against. The install step compares + # this to the version actually installed and to the latest on PSGallery, and emits a + # ::notice annotation if the YAML appears stale - prompting you to refresh via + # Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5. + GENERATED_AGAINST_MODULE_VERSION: '0.8.7' + # Resolution order for the module version pin (leave all unset to install the latest, + # which is the default "fix-forward" behaviour): manual workflow_dispatch input > + # repository variable 'REQUIRED_MODULE_VERSION' > empty (latest). + REQUIRED_MODULE_VERSION: ${{ github.event.inputs.module_version || vars.REQUIRED_MODULE_VERSION || '' }} + # v0.8.4 - opt this workflow into Node.js 24 for all JavaScript actions + # (actions/checkout, actions/download-artifact, actions/upload-artifact, + # azure/login, dorny/test-reporter, etc). Per GitHub's 2025-09-19 deprecation + # notice Node 20 is forced off by default on 2026-06-16 and removed from the + # runner on 2026-09-16. Setting this env var silences the deprecation warnings + # and exercises Node 24 ahead of the cut-over. To temporarily opt back out, + # set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. + # https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/ + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true + # Path to the ring-aware apply-updates-schedule.yml (relative to repo root). The + # planner reads it to find each cluster's next apply window. Override via repository + # variable APPLY_UPDATES_SCHEDULE_PATH if you keep the schedule outside `.github/`. + APPLY_UPDATES_SCHEDULE_PATH: ${{ vars.APPLY_UPDATES_SCHEDULE_PATH || './.github/apply-updates-schedule.yml' }} + + # ---- Sideload configuration (repository Variables, vars.*) ---- + # Master gate. The job is skipped unless this is the literal string 'true'. + SIDELOAD_UPDATES: ${{ vars.SIDELOAD_UPDATES || 'false' }} + # Shared UNC root that ALL runners/agents can read+write: holds state\, logs\, cache\. + SIDELOAD_STATE_ROOT: ${{ vars.SIDELOAD_STATE_ROOT || '' }} + # Shared verified media cache. Defaults (in the cmdlet) to \cache. + SIDELOAD_CACHE_ROOT: ${{ vars.SIDELOAD_CACHE_ROOT || '' }} + # Path to the sideload auth-map CSV (UpdateAuthAccountId -> Key Vault + secret names). + # Defaults to `config/` so it sits alongside apply-updates-schedule.yml and + # ClusterUpdateRings.csv (this is where Copy-AzLocalPipelineExample drops the + # starter). Override the variable to relocate it. + SIDELOAD_AUTH_MAP_PATH: ${{ vars.SIDELOAD_AUTH_MAP_PATH || './config/sideload-auth-map.csv' }} + # Path to the committed sideload catalog YAML (Version -> DownloadUri/Sha256). + # Defaults to `config/` alongside the auth-map + schedule. Override to relocate. + SIDELOAD_CATALOG_PATH: ${{ vars.SIDELOAD_CATALOG_PATH || './config/sideload-catalog.yml' }} + # Days before a cluster's next apply window that media should be sideloaded. + SIDELOAD_LEAD_DAYS: ${{ vars.SIDELOAD_LEAD_DAYS || '7' }} + # Extra robocopy switches for the detached copy worker (throttling / retries). + SIDELOAD_ROBOCOPY_SWITCHES: ${{ vars.SIDELOAD_ROBOCOPY_SWITCHES || '/R:5 /W:30' }} + # Minutes after which a Copying heartbeat is considered stale (dead runner) and re-driven. + SIDELOAD_HEARTBEAT_STALE_MINUTES: ${{ vars.SIDELOAD_HEARTBEAT_STALE_MINUTES || '60' }} + # Global FQDN suffix appended to a cluster name to form the WinRM host when the + # auth-map row does not override it (e.g. '.corp.contoso.com'). Empty = bare name. + SIDELOAD_REMOTING_FQDN_SUFFIX: ${{ vars.SIDELOAD_REMOTING_FQDN_SUFFIX || '' }} + # Key Vault auth mode for the on-prem runner: oidc | managedidentity | serviceprincipal. + SIDELOAD_KV_AUTH: ${{ vars.SIDELOAD_KV_AUTH || 'oidc' }} + +# Prevent overlapping sideload runs from racing on the same shared state. +# cancel-in-progress=false queues a newly-triggered run behind an in-flight one. +concurrency: + group: sideload-updates-${{ github.workflow }} + cancel-in-progress: false + +jobs: + sideload: + name: Advance Sideload State Machine + # Requires an on-prem self-hosted RUNNER (GitHub Actions terminology) on the + # same network as the clusters, registered with the `azlocal-sideload` label + # (use a dedicated runner group to restrict which repos can target it). + runs-on: [self-hosted, azlocal-sideload] + # Master opt-in gate: do nothing unless SIDELOAD_UPDATES is explicitly 'true'. + if: ${{ vars.SIDELOAD_UPDATES == 'true' }} + permissions: + id-token: write + contents: read + checks: write # Required for the test reporter + + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + # Azure auth. enable-AzPSSession=true sets up BOTH the az CLI context (ARG + + # az rest tag writes) AND the Az PowerShell context (Get-AzKeyVaultSecret). + # + # SIDELOAD_KV_AUTH alternatives for on-prem runners where OIDC is not viable: + # - managedidentity: replace this step with + # run: Connect-AzAccount -Identity + # - serviceprincipal: store AZURE_CLIENT_SECRET as a repo secret and use + # azure/login with creds, or Connect-AzAccount -ServicePrincipal -Credential ... + - name: Azure Login (OIDC + Az PowerShell session) + uses: azure/login@v3 + with: + client-id: ${{ secrets.AZURE_CLIENT_ID }} + tenant-id: ${{ vars.AZURE_TENANT_ID }} + subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} + enable-AzPSSession: true + + - name: Install Azure CLI Resource Graph Extension + shell: pwsh + run: | + az extension add --name resource-graph --yes + + - name: Install AzLocal.UpdateManagement from PSGallery + shell: pwsh + id: module-version + run: | + $ErrorActionPreference = 'Stop' + $installArgs = @{ Name = 'AzLocal.UpdateManagement'; Scope = 'CurrentUser'; Force = $true; AllowClobber = $true } + if ($env:REQUIRED_MODULE_VERSION) { + $installArgs.RequiredVersion = $env:REQUIRED_MODULE_VERSION + Write-Host "REQUIRED_MODULE_VERSION is set - pinning install to v$($env:REQUIRED_MODULE_VERSION)." + } else { + Write-Host "REQUIRED_MODULE_VERSION is empty - installing the latest version from PSGallery (default fix-forward behaviour)." + } + Install-Module @installArgs + Import-Module AzLocal.UpdateManagement -Force + Add-AzLocalPipelineVersionBanner ` + -GeneratedAgainstVersion $env:GENERATED_AGAINST_MODULE_VERSION ` + -PinnedVersion $env:REQUIRED_MODULE_VERSION + + - name: Advance Sideload State Machine + id: sideload + shell: pwsh + env: + INPUT_UPDATE_RING: ${{ github.event.inputs.update_ring || '***' }} + INPUT_DRY_RUN: ${{ github.event.inputs.dry_run || 'true' }} + run: | + $ErrorActionPreference = 'Stop' + Import-Module AzLocal.UpdateManagement -Force + + if ([string]::IsNullOrWhiteSpace($env:SIDELOAD_STATE_ROOT)) { + throw "SIDELOAD_STATE_ROOT is not set. Configure the shared UNC state root repository variable before running this pipeline." + } + + # 1. Resolve the per-cluster plan (read-only). + $planParams = @{ + SchedulePath = $env:APPLY_UPDATES_SCHEDULE_PATH + AuthMapPath = $env:SIDELOAD_AUTH_MAP_PATH + CatalogPath = $env:SIDELOAD_CATALOG_PATH + LeadDays = [int]$env:SIDELOAD_LEAD_DAYS + } + if ($env:INPUT_UPDATE_RING) { $planParams['UpdateRingValue'] = $env:INPUT_UPDATE_RING } + if ($env:SIDELOAD_REMOTING_FQDN_SUFFIX) { $planParams['FqdnSuffix'] = $env:SIDELOAD_REMOTING_FQDN_SUFFIX } + $plan = @(Resolve-AzLocalSideloadPlan @planParams) + Write-Host ("Resolved {0} cluster plan row(s)." -f $plan.Count) + + # 2. Advance the re-entrant state machine by one transition per cluster. + $applyParams = @{ + Plan = $plan + StateRoot = $env:SIDELOAD_STATE_ROOT + ErrorAction = 'Stop' + } + if ($env:SIDELOAD_CACHE_ROOT) { $applyParams['CacheRoot'] = $env:SIDELOAD_CACHE_ROOT } + if ($env:SIDELOAD_ROBOCOPY_SWITCHES) { $applyParams['RobocopySwitches'] = $env:SIDELOAD_ROBOCOPY_SWITCHES } + if ($env:SIDELOAD_HEARTBEAT_STALE_MINUTES) { $applyParams['HeartbeatStaleMinutes'] = [int]$env:SIDELOAD_HEARTBEAT_STALE_MINUTES } + if ($env:INPUT_DRY_RUN -eq 'true') { + $applyParams['WhatIf'] = $true + Write-Host 'DRY RUN MODE - no media staged, no tasks registered, no tags flipped.' + } + $results = @(Invoke-AzLocalSideloadUpdate @applyParams) + Write-Host ("Processed {0} cluster(s)." -f $results.Count) + + # 3. Render the status report (markdown step summary + JUnit XML + sideload-status.md). + New-Item -ItemType Directory -Path ./reports -Force | Out-Null + $report = Add-AzLocalSideloadStepSummary -StateRoot $env:SIDELOAD_STATE_ROOT -Plan $plan -OutputPath ./reports + + if ($report.HasFailures) { + Write-Host '::warning::One or more clusters are in a Failed state or have plan errors - see the Sideload status report.' + } + + - name: Compute Artifact Timestamp + if: always() + id: artifact-stamp + shell: pwsh + # every downloadable artifact gets a UTC timestamp suffix so multiple runs on + # the same day produce distinct zip names. + run: | + $stamp = (Get-Date).ToUniversalTime().ToString('yyyyMMdd_HHmmss') + "timestamp=$stamp" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append + Write-Host "Artifact timestamp: $stamp" + + - name: Upload Sideload Reports + if: always() + uses: actions/upload-artifact@v6 + with: + name: azlocal-step.6-sideload-updates-report_${{ steps.artifact-stamp.outputs.timestamp }} + path: ./reports/ + retention-days: 90 + + - name: Publish Test Results + uses: dorny/test-reporter@v3 + if: always() + with: + name: Sideload Update Report + path: ./reports/sideload-junit.xml + reporter: java-junit + fail-on-error: false + list-suites: failed + list-tests: failed diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/sideload-auth-map.example.csv b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/sideload-auth-map.example.csv new file mode 100644 index 00000000..d5c367bc --- /dev/null +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/sideload-auth-map.example.csv @@ -0,0 +1,30 @@ +# --------------------------------------------------------------------------- +# Sideload auth-map (v0.8.7 on-prem solution-update sideloading automation) +# --------------------------------------------------------------------------- +# Maps the per-cluster numeric 'UpdateAuthAccountId' tag to the Key Vault that +# holds the Active Directory credential (stored as TWO separate secrets: +# username + password) used to PowerShell-remote into the cluster, plus +# optional remoting overrides. +# +# Columns: +# UpdateAuthAccountId REQUIRED numeric 1-3 digits (e.g. 001). MUST match the +# cluster 'UpdateAuthAccountId' tag exactly. +# KeyVaultName REQUIRED Key Vault holding the credential secrets. +# UsernameSecretName REQUIRED KV secret with the username. UPN form +# ('svc-sideload@corp.contoso.com') recommended; +# 'CORP\svc-sideload' also works. +# PasswordSecretName REQUIRED KV secret with the password. +# RemotingTargetFqdn optional explicit FQDN to remote to (overrides the +# cluster-name + suffix resolution). +# FqdnSuffix optional per-row DNS suffix appended to the cluster +# name (e.g. .corp.contoso.com). +# AuthMechanism optional WinRM auth: Kerberos | Negotiate | Default. +# ImportSharePath optional override for the cluster infrastructure +# 'import' UNC share. +# +# DUPLICATE UpdateAuthAccountId values are a HARD ERROR in the sideload pipeline. +# Lines beginning with '#' are header documentation only; remove before use or +# keep the single real header row below. +UpdateAuthAccountId,KeyVaultName,UsernameSecretName,PasswordSecretName,RemotingTargetFqdn,FqdnSuffix,AuthMechanism,ImportSharePath +001,kv-azlocal-prod,sideload-username-site1,sideload-password-site1,,.corp.contoso.com,Kerberos, +002,kv-azlocal-prod,sideload-username-site2,sideload-password-site2,cluster02.dmz.contoso.com,,Negotiate, diff --git a/AzLocal.UpdateManagement/Automation-Pipeline-Examples/sideload-catalog.example.yml b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/sideload-catalog.example.yml new file mode 100644 index 00000000..c274278b --- /dev/null +++ b/AzLocal.UpdateManagement/Automation-Pipeline-Examples/sideload-catalog.example.yml @@ -0,0 +1,34 @@ +# --------------------------------------------------------------------------- +# Sideload catalog (v0.8.7 on-prem solution-update sideloading automation) +# --------------------------------------------------------------------------- +# Source-controlled list of the Azure Local solution-update media (and OEM SBE +# packages) the sideloading automation is allowed to stage. The runtime copy +# path reads ONLY this committed file - it never live-fetches at run time, so +# the behaviour is deterministic and air-gap friendly. +# +# Two package classes (packageType): +# Solution : Microsoft CombinedSolutionBundle..zip, downloadable from a +# direct downloadUri (published in the Microsoft Learn +# 'import and discover updates offline' table). sha256 REQUIRED. +# Refresh these rows with: Update-AzLocalSideloadCatalog +# SBE : OEM Solution Builder Extension package that Microsoft does NOT +# host. Stage the OEM files yourself and record a sourceFolder +# (local or UNC path). downloadUri is not applicable; sha256 is +# optional (verified only when supplied). Add these rows MANUALLY - +# Update-AzLocalSideloadCatalog will not populate them. +schemaVersion: 1 +packages: + - version: '12.2605.1003.210' + packageType: Solution + buildNumber: '12.2605.1003.210' + osBuild: '26100.4061' + downloadUri: 'https://azurestackreleases.download.prss.microsoft.com/dbazure/AzureLocal/CombinedSolutionBundle/12.2605.1003.210/CombinedSolutionBundle.12.2605.1003.210.zip' + sha256: '0000000000000000000000000000000000000000000000000000000000000000' + availabilityDate: '2026-05-13' + localPath: '' + - version: 'DellSBE-4.1.2412.1' + packageType: SBE + sourceFolder: '\\fileserver\sbe\Dell\4.1.2412.1' + sha256: '' + availabilityDate: '2026-05-20' + notes: 'Dell OEM SBE package - staged manually by the operator' diff --git a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 index 7caff5b8..6f04c367 100644 --- a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 +++ b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psd1 @@ -3,7 +3,7 @@ RootModule = 'AzLocal.UpdateManagement.psm1' # Version number of this module. - ModuleVersion = '0.8.6' + ModuleVersion = '0.8.7' # Supported PSEditions CompatiblePSEditions = @('Desktop', 'Core') @@ -51,6 +51,8 @@ 'Private/Get-AzLocalItsmTriggerDecision.ps1', 'Private/Get-AzLocalModuleRootManifestPath.ps1', 'Private/Get-AzLocalPipelineCustomiseMarkers.ps1', + 'Private/Get-AzLocalPipelineId.ps1', + 'Private/Get-AzLocalPipelineManifest.ps1', 'Private/Get-AzLocalRunEndTime.ps1', 'Private/Get-CurrentStepPath.ps1', 'Private/Get-DeepestActiveStep.ps1', @@ -79,6 +81,19 @@ 'Private/Resolve-WildcardDate.ps1', 'Private/Resolve-WildcardDateRange.ps1', 'Private/Set-AzLocalClusterTagsMerge.ps1', + # On-prem solution-update sideloading automation (v0.8.7) + 'Private/Get-AzLocalSideloadAuthMap.ps1', + 'Private/Get-AzLocalSideloadCatalog.ps1', + 'Private/Convert-AzLocalSideloadCatalogSchemaVersion.ps1', + 'Private/Select-AzLocalNextUpdateForCluster.ps1', + 'Private/Resolve-AzLocalSideloadCredential.ps1', + 'Private/Get-AzLocalSolutionUpdateDownload.ps1', + 'Private/Get-AzLocalSideloadState.ps1', + 'Private/Resolve-AzLocalSideloadTargetPath.ps1', + 'Private/Register-AzLocalSideloadCopyTask.ps1', + 'Private/New-AzLocalPSRemotingSession.ps1', + 'Private/Test-AzLocalRemoteFileHash.ps1', + 'Private/Invoke-AzLocalRemoteSolutionImport.ps1', 'Private/Test-AzCliAvailable.ps1', 'Private/Test-AzLocalAllowedUpdateVersionsString.ps1', 'Private/Test-AzLocalUpdateExclusion.ps1', @@ -139,6 +154,12 @@ 'Public/Update-AzLocalPipelineExample.ps1', 'Public/Get-AzLocalFleetConnectivityStatus.ps1', 'Public/New-AzLocalFleetConnectivityStatusSummary.ps1', + # On-prem solution-update sideloading automation (v0.8.7) - catalog maintenance + 'Public/Update-AzLocalSideloadCatalog.ps1', + # On-prem solution-update sideloading automation (v0.8.7) - planner, orchestrator, reporting + 'Public/Resolve-AzLocalSideloadPlan.ps1', + 'Public/Invoke-AzLocalSideloadUpdate.ps1', + 'Public/Export-AzLocalSideloadStatusReport.ps1', # Thin-YAML pipeline foundation (v0.8.5) 'Public/Add-AzLocalPipelineVersionBanner.ps1', # Thin-YAML Step.0 (v0.8.5) - Authentication validation + subscription scope + cluster reachability @@ -251,7 +272,13 @@ 'Invoke-AzLocalReadinessGatedClusterUpdate', 'Add-AzLocalApplyUpdatesStepSummary', 'Add-AzLocalNoReadyClustersStepSummary', - 'Invoke-AzLocalItsmTicketingFromArtifact' + 'Invoke-AzLocalItsmTicketingFromArtifact', + # On-prem solution-update sideloading automation (v0.8.7) + 'Update-AzLocalSideloadCatalog', + 'Resolve-AzLocalSideloadPlan', + 'Invoke-AzLocalSideloadUpdate', + 'Export-AzLocalSideloadStatusReport', + 'Add-AzLocalSideloadStepSummary' ) # Cmdlets to export from this module, for best performance, do not use wildcards and do not delete the entry, use an empty array if there are no cmdlets to export. @@ -280,27 +307,19 @@ # ReleaseNotes of this module ReleaseNotes = @' -## Version 0.8.6 - Step.3 cycle-calendar enrichment (per-day CRON + UpdateStartWindow tag coverage) + six v0.8.5 pipeline regression fixes (Step.0/3/4/6/9) + new Pester static-audit guards - -Adds two opt-in render-time columns to `Get-AzLocalApplyUpdatesScheduleCycleCalendar` (auto-wired by `Export-AzLocalApplyUpdatesScheduleAudit`) AND fixes six production regressions introduced by the v0.8.5 thin-YAML port. New static-source Pester guards block the underlying anti-patterns from shipping again. No public API removed; no parameter changes on existing cmdlets; same module export count as v0.8.5 (55). - -- **FIX Step.0 (`Export-AzLocalAuthValidationReport`)**: `Resource Graph reachability` reported "1 cluster(s) visible" no matter the real fleet size. `Invoke-AzResourceGraphQuery` uses unary-comma `return , $arr.ToArray()`; wrapping it with `@()` collected the outer wrapper and dropped every row past the first. Switched to direct assignment + `$null` -> `@()` guard, plus a defensive `$clusterRows = @($clusterRows)` coerce on the variable so `.Count` is always a valid integer property under strict mode. -- **FIX Step.3 (`Test-AzLocalApplyUpdatesScheduleCoverage`)**: `$allMatched = @($segmentStatuses.MatchingCrons | Select-Object -Unique)` threw "The property 'MatchingCrons' cannot be found on this object." for any audit row where `$r.RequiredCrons` was empty - `$segmentStatuses` stayed `@()` and member-enumeration on an empty array trips strict mode. Guarded the access with `if ($segmentStatuses.Count -gt 0)`. -- **FIX Step.4 (`Get-AzLocalFleetConnectivityStatus`)**: ARB row rendering crashed under `Set-StrictMode -Version Latest` on any RG holding exactly one cluster. The if-as-expression silently unwrapped the single-element `@()` to a scalar PSCustomObject; `$matched.Count` then threw. The v0.8.6-fix1 `[object[]]$matched = if (...)` cast was insufficient on the production fleet (still scalar at runtime). The reliable fix: assign the raw list (or `$null`) to `$matchedList`, then `$matched = @($matchedList)` to coerce scalar -> Object[1] on the *variable* (idempotent for Object[N]). -- **FIX Step.6 (`Get-AzLocalClusterUpdateReadiness`)**: `if ($cluster.NotFound)` at line ~337 threw "The property 'NotFound' cannot be found on this object" because three of the four `$clustersToProcess += @{ ... }` builder paths omitted the key. Added `NotFound = $false` to all three missing-key paths so the strict-mode iteration is always safe. -- **FIX Step.9 (`Export-AzLocalFleetHealthStatusReport`)**: BOTH `$detail = @(Get-AzLocalFleetHealthFailures ...)` and `$overview = @(Get-AzLocalFleetHealthOverview ...)` collapsed the row-set to `Object[1]`; "Found 81 failing check(s)" from the helper became "Found 1" two lines later and Group-Object then crashed with exit code 1. Switched both to direct assignment + `$null` -> `@()` guards. -- **FIX Node.js 20 deprecation**: bumped `actions/upload-artifact@v4` -> `@v6` in `Step.0_authentication-test.yml` (every other Step YAML was already on `@v6`). -- **FIX Step.2 per-cluster `Message` now names the tags that actually changed.** When only a schedule tag (e.g. `UpdateExcluded`) differs between cluster and CSV, the summary used to read "UpdateRing tag updated successfully" - misleading because `UpdateRing` was unchanged. `Set-AzLocalClusterUpdateRingTag` now computes per-tag deltas from `$tagsToMerge.Keys` vs `$currentTags` and writes `Tags updated: UpdateExcluded: 'False' -> 'True'` (same shape for `-WhatIf`). -- **FIX Step.2 summary table split into outcome buckets.** `Set-AzLocalClusterUpdateRingTagFromCsv` previously rendered all clusters (created/updated/no-op/skipped/failed) into one `
` block. The summary now renders three independent collapsible sections, each shown only when non-empty: **"Clusters with Tag Updates Applied (N rows)"** (expanded), **"Clusters Skipped or Failed (N rows)"** (expanded), **"Clusters with No Tag Updates (no-op) (N rows)"** (collapsed). -- **FIX Step.2 `Total clusters processed` count was inflated (`44` vs `20` on a 20-row CSV).** `Set-AzLocalClusterUpdateRingTag` ended with `$results | Format-Table -AutoSize` (no `| Out-Host`), so the formatter wrapper objects leaked into the function's pipeline output and mixed with `return $results` under `-PassThru`. Fixed by piping Format-Table to `Out-Host` + defence-in-depth filter in `Set-AzLocalClusterUpdateRingTagFromCsv` that drops any object without a string `ClusterName` property. -- **NEW Pester regression guards** (11 It blocks across 7 Describe blocks): static-source scans for `@()` wrap on unary-comma helpers (`Invoke-AzResourceGraphQuery`, `Get-AzLocalFleetHealth{Failures,Overview}`, `Read-AzLocalApplyUpdatesYamlCrons`); Step.3 `.MatchingCrons` strict-mode crash; Step.4 `$matched if/else { @() }` branch; Step.6 `NotFound` key on every `$clustersToProcess` builder; Step.0 `$clusterRows` variable-level `@()` coerce; Step.2 `Format-Table | Out-Host`; GHA Node.js 20 artifact-action version deprecation; plus functional spy tests for Export-AzLocalAuthValidationReport + Export-AzLocalFleetHealthStatusReport. -- **NEW `-CronFiringsByDate`** (`[hashtable]`, keys = `yyyy-MM-dd` UTC, values = `[string[]]` of `HH:mm` UTC firing times). Adds centered `Ring CRON Start Time (UTC)
(Step 6 pipeline)` column between `Date (UTC)` and `Day`. Cell rendering: 0 firings -> `_(none)_`; 1-2 -> comma-joined; 3+ -> first 2 + `" (+N)"`; dead day -> `_(none - dead day)_`; missing date key -> blank. -- **NEW `-WindowMatchByRingAndDate`** (`[hashtable[string,hashtable]]`, ring -> date -> `@{ Matching=; Total= }`). Adds `Tag Start Window Match (>=95%)` column AFTER `Eligible rings`. Per-ring line: `` `Ring`: True/False mat/tot (pct%) `` (True iff `Matching/Total >= 0.95`). Rendering: `_(n/a)_` no entry; `_(0 clusters)_` Total=0; `_(n/a - dead day)_` on dead days. -- **Pure render-time contract preserved.** Cmdlet still does zero Azure / file I/O. Both columns opt-in. -- **`Export-AzLocalApplyUpdatesScheduleAudit` auto-wires both.** Cron firings via existing `Read-AzLocalApplyUpdatesYamlCrons` + `ConvertFrom-AzLocalCronExpression` (invalid crons -> no firings for that day). Window-match via `ConvertFrom-AzLocalUpdateWindow` per cluster (when `-ClusterCsvPath` supplied); overnight windows match firings in either the late-evening or early-morning portion via two-case `DayOfWeek` projection. -- **Failure mode is non-fatal.** Enrichment errors degrade to the v0.8.5 calendar; `Write-Warning` surfaces the cause but Step.3 summary continues to render. -- **Pester**: drift bumped to `'0.8.6'`; 12 new It blocks cover backwards compat, single-param paths, both-params 7-column header, `(+N)` suffix at 3+ firings, 95% threshold boundary, `_(0 clusters)_` / `_(n/a)_` rendering, ring + date case-insensitivity, and coexistence with `-ClusterRingCounts`. 2 new Export-* spy tests verify the auto-wire path. PLUS 11 new regression-guard It blocks across 7 Describe blocks. -- **All 20 bundled `Step.{0..9}.yml` templates** bump `GENERATED_AGAINST_MODULE_VERSION` from `'0.8.5'` to `'0.8.6'`. +## Version 0.8.7 - On-prem solution-update sideloading automation (new Step.6 pipeline + 5 new Public cmdlets) + pipeline filenames de-numbered (rename-aware Update) + breaking display-step renumber. Module export count grows 55 -> 60. + +Adds opt-in, off-by-default on-prem sideloading for Azure Local clusters that cannot pull solution updates from Azure directly. A new self-hosted Step.6 pipeline (`sideload-updates.yml`) robocopies update media to each cluster's import share, verifies the SHA256 over WinRM, runs Add-SolutionUpdate, and flips `UpdateSideloaded=True` for the downstream apply. It is driven as a re-entrant state machine on a frequent CRON; the multi-hour copy runs in a detached Windows Scheduled Task so no single pipeline run is long-lived. + +- **NEW Public cmdlets (5)**: `Update-AzLocalSideloadCatalog`, `Resolve-AzLocalSideloadPlan`, `Invoke-AzLocalSideloadUpdate`, `Export-AzLocalSideloadStatusReport`, `Add-AzLocalSideloadStepSummary`. Export count grows 55 -> 60. +- **NEW Step.6 sideload pipeline** (`sideload-updates.yml`, GitHub Actions + Azure DevOps), opt-in via the `SIDELOAD_UPDATES` repository variable; targets a self-hosted runner (GH) / self-hosted agent pool (ADO) labelled `azlocal-sideload`. Catalog (Solution + OEM SBE packageType), shared-UNC state, Key Vault-sourced AD credential for WinRM, and robocopy throttling are documented in Automation-Pipeline-Examples/docs/sideload.md + sideload-robocopy.md. +- **BREAKING - pipeline files de-numbered.** The `Step.N_` filename prefix is removed (e.g. `Step.7_apply-updates.yml` -> `apply-updates.yml`); the in-pipeline `Step.N - ` display names are unchanged. Stable filenames mean future reorders are pure display-name edits. `Update-AzLocalPipelineExample` now matches each destination pipeline by a stable logical id (embedded `# AZLOCAL-PIPELINE-ID:` marker, with legacy-filename aliases) and AUTO-RENAMES any older `Step.N_*.yml` to the new name while preserving your `BEGIN/END-AZLOCAL-CUSTOMIZE` CRON edits (emits a `RenamedFrom` result + a GH/ADO required-check warning). +- **BREAKING - display-step renumber** to make room for sideload at Step.6: apply-updates 6->7, monitor-updates 7->8, fleet-update-status 8->9, fleet-health-status 9->10. +- **Step.1 / Step.2 sideload-aware**: inventory can emit the `UpdateAuthAccountId` column (`-IncludeSideloadColumns`, auto-enabled from `SIDELOAD_UPDATES`); tag management can set `UpdateAuthAccountId` from CSV. Byte-identical output when sideload is off. +- **Step.3 advisor** can emit a recommended sideload CRON (apply window minus `SIDELOAD_LEAD_DAYS`) when sideloading is enabled. +- **All bundled pipeline templates** bump `GENERATED_AGAINST_MODULE_VERSION` from `'0.8.6'` to `'0.8.7'`. + +## Version 0.8.6 - Step.3 cycle-calendar enrichment (per-day CRON + UpdateStartWindow tag coverage) + six v0.8.5 pipeline regression fixes (Step.0/3/4/6/9) + new Pester static-audit guards. See CHANGELOG for the full v0.8.6 entry. ## Version 0.8.5 - New Public cmdlet `Get-AzLocalApplyUpdatesScheduleCycleCalendar` + Step.6 manual schedule-file inputs + Step.3 cycle-calendar regression fix + per-ring cluster-count column + full thin-YAML port of all 10 Step pipelines (14 new Public cmdlets). Module export count grows 35 -> 55. See CHANGELOG for the full v0.8.5 entry. diff --git a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 index 0a428f22..422c3b0a 100644 --- a/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 +++ b/AzLocal.UpdateManagement/AzLocal.UpdateManagement.psm1 @@ -151,7 +151,7 @@ Set-StrictMode -Version 1.0 # bumps to one but not the other are caught before release. Two consumers: # - Start-AzLocalClusterUpdate emits this in the run log header. # - Get-AzLocalFleetStatusData stamps it into exported fleet-state JSON. -$script:ModuleVersion = '0.8.6' +$script:ModuleVersion = '0.8.7' $script:DefaultApiVersion = '2025-10-01' $script:DefaultLogFolder = Join-Path -Path $env:ProgramData -ChildPath 'AzLocal.UpdateManagement' @@ -218,11 +218,17 @@ $script:UpdateSideloadedTagName = 'UpdateSideloaded' $script:UpdateVersionInProgressTagName = 'UpdateVersionInProgress' +# v0.8.7: numeric account id (1-3 digits, e.g. 001) that maps a cluster to a +# row in the sideload auth-map CSV (Key Vault + remoting settings) used by the +# on-prem solution-update sideloading automation. Written/exported by Step.1 +# and Step.2 only when SIDELOAD_UPDATES is enabled; absent otherwise. +$script:UpdateAuthAccountIdTagName = 'UpdateAuthAccountId' + # Current apply-updates-schedule.yml schema version produced + consumed by # this module. Incremented when a non-additive change to the schedule file -# format ships. Customer files on a LOWER version are auto-migrated by -# Update-AzLocalPipelineExample via the per-hop recipes registered in -# Private/Convert-AzLocalScheduleSchemaVersion.ps1. Customer files on a +# format ships. Customer files on a LOWER version are migrated by +# Update-AzLocalApplyUpdatesScheduleConfig via the per-hop recipes registered +# in Private/Convert-AzLocalScheduleSchemaVersion.ps1. Customer files on a # HIGHER version cause the migrator to refuse with a remediation message # pointing at PSGallery. # @@ -235,6 +241,17 @@ $script:UpdateVersionInProgressTagName = 'UpdateVersionInProgress' # silently absent. $script:ScheduleSchemaCurrentVersion = 2 +# Current sideload-catalog.yml schema version produced + consumed by this +# module (v0.8.7 on-prem sideloading automation). Mirrors the schedule schema +# framework: customer catalog files on a LOWER version are migrated by +# Update-AzLocalSideloadCatalog -SchemaMigrate via the per-hop recipes +# registered in Private/Convert-AzLocalSideloadCatalogSchemaVersion.ps1. +# Get-AzLocalSideloadCatalog refuses to read a catalog on a HIGHER version +# (it points the operator at PSGallery). The recipe table ships EMPTY at v1 +# - the framework is in place so a future non-additive catalog format change +# (e.g. v1 -> v2) is a small, comment-preserving, idempotent recipe addition. +$script:SideloadCatalogSchemaCurrentVersion = 1 + $script:DayMap = [ordered]@{ 'Mon' = [DayOfWeek]::Monday 'Tue' = [DayOfWeek]::Tuesday @@ -343,5 +360,11 @@ Export-ModuleMember -Function @( 'Invoke-AzLocalReadinessGatedClusterUpdate', 'Add-AzLocalApplyUpdatesStepSummary', 'Add-AzLocalNoReadyClustersStepSummary', - 'Invoke-AzLocalItsmTicketingFromArtifact' + 'Invoke-AzLocalItsmTicketingFromArtifact', + # On-prem solution-update sideloading automation (v0.8.7) + 'Update-AzLocalSideloadCatalog', + 'Resolve-AzLocalSideloadPlan', + 'Invoke-AzLocalSideloadUpdate', + 'Export-AzLocalSideloadStatusReport', + 'Add-AzLocalSideloadStepSummary' ) \ No newline at end of file diff --git a/AzLocal.UpdateManagement/CHANGELOG.md b/AzLocal.UpdateManagement/CHANGELOG.md index c33c6081..fd6dfc8a 100644 --- a/AzLocal.UpdateManagement/CHANGELOG.md +++ b/AzLocal.UpdateManagement/CHANGELOG.md @@ -5,6 +5,69 @@ All notable changes to the AzLocal.UpdateManagement module (renamed from AzStack The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.8.7] - 2026-06-11 + +On-prem solution-update sideloading automation release. Adds an opt-in, off-by-default workflow for Azure Local clusters that cannot pull solution updates from Azure directly: a new self-hosted Step.6 pipeline (`sideload-updates.yml`) robocopies update media to each cluster's import share, verifies the SHA256 over WinRM, runs `Add-SolutionUpdate`, and flips the `UpdateSideloaded=True` cluster tag so the downstream apply (now Step.7) picks it up. The long-running copy executes in a detached Windows Scheduled Task and is driven as a re-entrant state machine on a frequent CRON, so no individual pipeline run is long-lived. **This release de-numbers all bundled pipeline filenames and renumbers four display steps (BREAKING for any consumer that pins on the old `Step.N_*.yml` filenames).** Module export count grows 55 -> 60. + +### New Public cmdlets (5) + +- **`Update-AzLocalSideloadCatalog`** - validates / refreshes the sideload catalog (Solution and OEM SBE `packageType` entries) used to resolve which update media to stage. +- **`Resolve-AzLocalSideloadPlan`** - resolves the per-cluster sideload plan (target media, import share, current state) from inventory + catalog. +- **`Invoke-AzLocalSideloadUpdate`** - the re-entrant state-machine driver (Planned -> Copying -> Copied -> Verified -> Imported -> SideloadFlagged); spawns / re-attaches the detached robocopy Scheduled Task, verifies SHA256 over WinRM, runs `Add-SolutionUpdate`, and sets `UpdateSideloaded=True`. +- **`Export-AzLocalSideloadStatusReport`** - emits the per-cluster sideload status CSV + JUnit XML + step outputs for the Step.6 pipeline. +- **`Add-AzLocalSideloadStepSummary`** - renders the Step.6 markdown step summary (per-state table + shared-state contract + exit conditions). + +### New Step.6 sideload pipeline + +- **`sideload-updates.yml`** (GitHub Actions + Azure DevOps), opt-in via the `SIDELOAD_UPDATES` repository variable. Targets a self-hosted runner (`runs-on: [self-hosted, azlocal-sideload]`) on GitHub / a self-hosted agent pool (`pool: { name, demands: azlocal-sideload }`) on Azure DevOps. Manual `workflow_dispatch` plus a `*/30` CRON poll so the state machine can advance between runs. +- Catalog (Solution + OEM SBE), shared-UNC state for multi-runner coordination, a Key Vault-sourced AD credential for cluster WinRM, and robocopy throttling (`SIDELOAD_ROBOCOPY_SWITCHES`) are documented in `Automation-Pipeline-Examples/docs/sideload.md` and `sideload-robocopy.md`. + +### BREAKING - pipeline filenames de-numbered + +- The `Step.N_` filename prefix has been removed from every bundled pipeline (e.g. `Step.7_apply-updates.yml` -> `apply-updates.yml`). The in-pipeline `Step.N - ` **display** names are unchanged, so the operator-facing step ordering is preserved. Stable filenames mean future reorders become pure display-name edits. +- **`Update-AzLocalPipelineExample` is now rename-aware.** It matches each destination pipeline by a stable logical id (embedded `# AZLOCAL-PIPELINE-ID:` marker on line 1, with legacy-filename aliases) rather than by filename, and AUTO-RENAMES any older `Step.N_*.yml` on disk to the new de-numbered name while preserving the consumer's `BEGIN/END-AZLOCAL-CUSTOMIZE` CRON edits. It emits a `RenamedFrom` result field and a GitHub/Azure DevOps required-check warning when a rename occurs. + +### BREAKING - display-step renumber + +To make room for sideload at Step.6, four display steps shift up by one: apply-updates 6 -> 7, monitor-updates 7 -> 8, fleet-update-status 8 -> 9, fleet-health-status 9 -> 10. + +### Sideload-aware existing steps + +- **Step.1 inventory** can emit the `UpdateAuthAccountId` column (`-IncludeSideloadColumns`, auto-enabled when `SIDELOAD_UPDATES` is set). Byte-identical output when sideload is off. +- **Step.2 tag management** can set the `UpdateAuthAccountId` tag from CSV. +- **Step.3 schedule advisor** can emit a recommended sideload CRON (the apply window minus `SIDELOAD_LEAD_DAYS`) when sideloading is enabled. + +### BREAKING - all operator config relocated to a repo-root `config/` folder + +- Every operator-authored configuration file now lives in a single **`config/` folder at the repo root**, with the **identical path on GitHub Actions and Azure DevOps**. This consolidates around the convention already established for the ring CSV (`config/ClusterUpdateRings.csv`) and eliminates the previous `.github/`-vs-repo-root divergence. The four files are: `config/ClusterUpdateRings.csv`, `config/apply-updates-schedule.yml`, `config/sideload-auth-map.csv`, and `config/sideload-catalog.yml`. +- **The starter `apply-updates-schedule.yml` has moved** from its v0.7.92 location (`.github/apply-updates-schedule.yml` on GitHub / repo root on Azure DevOps) to `config/apply-updates-schedule.yml` on both platforms. The `apply-updates.yml` (Step.7) and `apply-updates-schedule-audit.yml` (Step.3) defaults for `APPLY_UPDATES_SCHEDULE_PATH` / `schedule_path` / `schedulePath` now point at `config/apply-updates-schedule.yml`. + +### Scaffolding - starter sideload config drop + +- **`Copy-AzLocalPipelineExample`** now drops two header-only starter config files alongside the starter `apply-updates-schedule.yml` in the repo-root **`config/`** folder when `-Platform GitHub` or `-Platform AzureDevOps` is used: `sideload-auth-map.csv` (CSV header row only) and `sideload-catalog.yml` (`schemaVersion: 1` + empty `packages:`). They land at `config/sideload-auth-map.csv` and `config/sideload-catalog.yml` - the same path on both platforms. Both are skeleton-only (no demo credential rows or media entries), so they parse to an empty auth-map / empty catalog and remain inert even if `SIDELOAD_UPDATES` is flipped on before they are populated. An existing file is **never** overwritten. Suppress with the new `-SkipStarterSideloadConfig` switch. The `sideload-updates.yml` `SIDELOAD_AUTH_MAP_PATH` / `SIDELOAD_CATALOG_PATH` defaults (both platforms) now point at `./config/` to match where the starters land. +- **`Update-AzLocalPipelineExample` does not touch `config/`.** It only refreshes the bundled pipeline YAMLs (matched by logical `# AZLOCAL-PIPELINE-ID:` marker), so all operator-owned config in `config/` is preserved across module upgrades. + +### Schema migration - sideload catalog framework + +- **New `sideload-catalog.yml` schema-migration framework**, symmetric with the existing `apply-updates-schedule.yml` one. Adds `$script:SideloadCatalogSchemaCurrentVersion` (currently `1`), a private text-surgery migration engine (`Private/Convert-AzLocalSideloadCatalogSchemaVersion.ps1`) with an EMPTY per-hop recipe table at v1, and a `-SchemaMigrate` parameter set on `Update-AzLocalSideloadCatalog` that migrates an existing catalog in place (backing the original up as `.v.old.yml`, preserving operator comments + OEM `SBE` entries verbatim, honouring `-WhatIf`/`-Confirm`, returning a `{ Action, FromVersion, ToVersion, BackupPath, Hops[] }` result). `Get-AzLocalSideloadCatalog` now reads the top-level `schemaVersion` and REFUSES to parse a catalog written by a newer module (missing `schemaVersion` is tolerated as v1 for back-compat). No behaviour change at v1 - the framework is wired up so the first non-additive catalog format change is a small, well-tested recipe addition. +- **Doc fix**: corrected stale references in `Convert-AzLocalScheduleSchemaVersion.ps1` and the `.psm1` that named `Update-AzLocalPipelineExample` as the schedule schema migrator; the actual migrator is `Update-AzLocalApplyUpdatesScheduleConfig`. + + +### Testing - sideload smoke coverage + wall-clock timing ledger + +- **New Step.6 sideload smoke coverage.** Added `Tools/smoke-test-sideload-plan.ps1` (a dedicated, sequence-faithful harness) plus a `Resolve-AzLocalSideloadPlan` section in the unified `Tools/validate-arg-queries.ps1`. A single `Resolve-AzLocalSideloadPlan` call exercises the whole Step.6 read path - the schedule parser, the private auth-map + catalog parsers, and the live cluster ARG query - so any ARG-query regression, parser drift, or plan-row schema change is caught locally before the `sideload-updates.yml` pipeline runs. The auth-map and catalog are operator-authored config (no bundled live example), so the harness synthesises minimal valid temp files and points the planner at the bundled `apply-updates-schedule.example.yml`. On a fleet with no `UpdateAuthAccountId`-tagged clusters the plan is legitimately empty (recorded as `PASS-EMPTY`) - still validating that the ARG query parsed/executed and all three config parsers ran without throwing. `Tools/SMOKE-COVERAGE.md` gains the sideload row and a "no fleet tags required" note. +- **Wall-clock timing is now recorded to a git-tracked ledger.** A new shared helper `Tests/Add-PesterRunTiming.ps1` appends one row per full-suite run to `Tests/test-run-timings.csv` (timestamp, module version, total/passed/failed/skipped, wall-clock seconds, Pester execution seconds, discovery seconds, runner label, PS version). `Tests/Invoke-Tests.ps1` wraps the run in a stopwatch and calls the helper at the end. Tracking suite duration over time makes wall-clock regressions visible and informs whether the eventual test-file split for parallelism is worth it. + +### Fixed + +- **Step.6 Apply Updates logged "Failed to export results: The property 'CurrentState' cannot be found on this object" after a successful update start.** The JUnit XML exporter (`Export-ResultsToJUnitXml`) used bare property access (`if ($result.CurrentState)` / `if ($result.Progress)` / `if ($result.UpdateName)`) in both the failure and success branches. The `UpdateStarted` success-row shape emitted by `Start-AzLocalClusterUpdate` has no `CurrentState` or `Progress` property, so under `Set-StrictMode -Version Latest` the success branch threw when exporting to `.xml` - the update itself started fine, only the results export failed. All three accesses in both branches now guard with `$result.PSObject.Properties['...']` (matching the existing `StartTime`/`EndTime` pattern). Adds a regression test that exports a `CurrentState`-less `UpdateStarted` row and asserts no throw. +- **In-flight monitor "Current Step" column always showed the top-level wrapper step (e.g. "Start update").** `Format-AzLocalUpdateRun` (and the duplicate logic in `Get-AzLocalFleetStatusData`) selected `CurrentStep` from the **top-level** `properties.progress.steps` array only. Azure Local nests a coarse wrapper step that stays `InProgress` for the entire run, so the Step.8 monitor (`Export-AzLocalUpdateRunMonitorReport`) and fleet-status report always rendered the wrapper name instead of the real current activity. Both paths now reuse `Get-DeepestActiveStep` to walk to the deepest `InProgress`/`Error`/`Failed` leaf (preserving the `(FAILED)` suffix), making `CurrentStep` consistent with `CurrentStepDetail` / `Get-CurrentStepPath` and the standard update-progress output. Adds two `Format-AzLocalUpdateRun` regression tests. + +### Versioning / tests + +- `ModuleVersion` 0.8.6 -> 0.8.7 (`.psd1` + `.psm1`). Export count assertion 55 -> 60. +- All bundled pipeline templates bump `GENERATED_AGAINST_MODULE_VERSION` from `'0.8.6'` to `'0.8.7'`. + ## [0.8.6] - 2026-06-10 Step.3 cycle-calendar enrichment release. Adds two opt-in columns to the per-day `## Cycle calendar` markdown table produced by `Get-AzLocalApplyUpdatesScheduleCycleCalendar` and auto-wires them from `Export-AzLocalApplyUpdatesScheduleAudit` so operators can see (a) which Step.6 cron firing times will fire on each calendar day and (b) what fraction of clusters in the ring(s) eligible on that day have an `UpdateStartWindow` tag that actually covers at least one of those firings. **Also fixes six production regressions introduced by the v0.8.5 thin-YAML port (Step.0, Step.3, Step.4 x2, Step.6, Step.9) and adds Pester guards so the same anti-patterns cannot ship again.** No public API removed; no existing parameter changed; no behavioural change on callers that do NOT supply the new dictionaries. Same module export count as v0.8.5 (55). diff --git a/AzLocal.UpdateManagement/Private/Convert-AzLocalScheduleSchemaVersion.ps1 b/AzLocal.UpdateManagement/Private/Convert-AzLocalScheduleSchemaVersion.ps1 index 781dc24e..5fd6baff 100644 --- a/AzLocal.UpdateManagement/Private/Convert-AzLocalScheduleSchemaVersion.ps1 +++ b/AzLocal.UpdateManagement/Private/Convert-AzLocalScheduleSchemaVersion.ps1 @@ -27,8 +27,8 @@ function Convert-AzLocalScheduleSchemaVersion { missing from the table, throw. Backup-on-write is performed by the caller - (Update-AzLocalPipelineExample) - this function only computes - the new text and reports what changed. + (Update-AzLocalApplyUpdatesScheduleConfig) - this function only + computes the new text and reports what changed. .PARAMETER Text Raw YAML text of the customer's schedule file. @@ -73,7 +73,7 @@ function Convert-AzLocalScheduleSchemaVersion { $current = [int]$cfg.SchemaVersion if ($current -gt $TargetSchemaVersion) { - throw "Convert-AzLocalScheduleSchemaVersion: '$SourcePath' is on schemaVersion=$current but this module only supports up to $TargetSchemaVersion. Upgrade the AzLocal.UpdateManagement module, then re-run Update-AzLocalPipelineExample." + throw "Convert-AzLocalScheduleSchemaVersion: '$SourcePath' is on schemaVersion=$current but this module only supports up to $TargetSchemaVersion. Upgrade the AzLocal.UpdateManagement module, then re-run Update-AzLocalApplyUpdatesScheduleConfig." } if ($current -eq $TargetSchemaVersion) { diff --git a/AzLocal.UpdateManagement/Private/Convert-AzLocalSideloadCatalogSchemaVersion.ps1 b/AzLocal.UpdateManagement/Private/Convert-AzLocalSideloadCatalogSchemaVersion.ps1 new file mode 100644 index 00000000..7805eab3 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Convert-AzLocalSideloadCatalogSchemaVersion.ps1 @@ -0,0 +1,156 @@ +function Convert-AzLocalSideloadCatalogSchemaVersion { + <# + .SYNOPSIS + Migrates a sideload-catalog.yml file from an older schema version to + the module's current schema version (or any chosen target). Text-surgery + based - operator comments, SBE (OEM) entries, and row order are + preserved verbatim. + + .DESCRIPTION + Sibling of Convert-AzLocalScheduleSchemaVersion.ps1 (the schedule-file + migrator) and follows the IDENTICAL design so the two schema frameworks + behave the same way. + + The function walks the per-hop recipe table registered in this file + (see $script:SideloadCatalogSchemaRecipes below). Each recipe is a + ScriptBlock with signature: + + param([string]$Text) -> @{ Text = ; Changes = @() } + + Recipes operate on RAW TEXT, not on the parsed structure. This is + deliberate so operator-authored YAML comments, manually-staged SBE + (OEM) package entries, and download/TODO annotations survive every + migration hop. Each recipe is expected to be IDEMPOTENT: running it + twice on the same input must produce the same output, and MUST update + the top-level 'schemaVersion:' line itself. + + Walker behaviour (identical to the schedule migrator): + * If $current == $target -> return Migrated=$false (no-op). + * If $current > $target -> throw (downgrade requested; bad). + * If $current < $target -> walk recipes $current -> $current+1 + -> ... -> $target. If any hop is + missing from the table, throw. + + The recipe table ships EMPTY at schema v1: there is no hop to run yet, + so a v1 catalog is always a no-op. The framework exists so the FIRST + non-additive catalog format change is a small, well-tested recipe + addition rather than a from-scratch build. + + Backup-on-write is performed by the caller + (Update-AzLocalSideloadCatalog -SchemaMigrate) - this function only + computes the new text and reports what changed. + + .PARAMETER Text + Raw YAML text of the customer's catalog file. + + .PARAMETER TargetSchemaVersion + Schema version to migrate TO. Default: this module's current + ($script:SideloadCatalogSchemaCurrentVersion). Tests can override to + exercise specific hops. + + .PARAMETER SourcePath + Optional path used in error messages. + + .OUTPUTS + [PSCustomObject] with: + Migrated [bool] + FromVersion [int] + ToVersion [int] + NewText [string] - migrated YAML text (or original if no-op) + Hops [object[]] - one row per executed recipe: + { FromVersion, ToVersion, Changes[] } + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + [Parameter(Mandatory = $true)] + [AllowEmptyString()] + [string]$Text, + + [Parameter(Mandatory = $false)] + [int]$TargetSchemaVersion = $script:SideloadCatalogSchemaCurrentVersion, + + [Parameter(Mandatory = $false)] + [string]$SourcePath = '' + ) + + # Read the current schemaVersion straight from the text. The catalog has + # no struct parser that surfaces schemaVersion, so a narrow regex is used. + # A catalog with NO schemaVersion line is treated as v1 (back-compat: the + # v1 reader ignored the field, so early files may omit it). + $current = 1 + $svMatch = [regex]::Match($Text, '(?m)^\s*schemaVersion\s*:\s*(\d+)\b') + if ($svMatch.Success) { + $current = [int]$svMatch.Groups[1].Value + } + + if ($current -gt $TargetSchemaVersion) { + throw "Convert-AzLocalSideloadCatalogSchemaVersion: '$SourcePath' is on schemaVersion=$current but this module only supports up to $TargetSchemaVersion. Upgrade the AzLocal.UpdateManagement module, then re-run Update-AzLocalSideloadCatalog -SchemaMigrate." + } + + if ($current -eq $TargetSchemaVersion) { + return [pscustomobject]@{ + Migrated = $false + FromVersion = $current + ToVersion = $TargetSchemaVersion + NewText = $Text + Hops = @() + } + } + + $hops = New-Object System.Collections.Generic.List[object] + $workingText = $Text + for ($v = $current; $v -lt $TargetSchemaVersion; $v++) { + $key = "$v->$($v + 1)" + # $script:SideloadCatalogSchemaRecipes is an [ordered] dictionary, + # which exposes .Contains() but NOT .ContainsKey() - the latter throws + # MethodNotFound on System.Collections.Specialized.OrderedDictionary. + if (-not $script:SideloadCatalogSchemaRecipes.Contains($key)) { + throw "Convert-AzLocalSideloadCatalogSchemaVersion: no migration recipe registered for '$key'. The module is missing a hop - this is a bug; file at https://github.com/NeilBird/Azure-Local/issues." + } + $recipe = $script:SideloadCatalogSchemaRecipes[$key] + $hopResult = & $recipe $workingText + if (-not $hopResult.ContainsKey('Text') -or -not $hopResult.ContainsKey('Changes')) { + throw "Convert-AzLocalSideloadCatalogSchemaVersion: recipe '$key' did not return the expected @{ Text=...; Changes=... } shape." + } + $workingText = [string]$hopResult.Text + $hops.Add([pscustomobject]@{ + FromVersion = $v + ToVersion = $v + 1 + Changes = @($hopResult.Changes) + }) | Out-Null + } + + return [pscustomobject]@{ + Migrated = $true + FromVersion = $current + ToVersion = $TargetSchemaVersion + NewText = $workingText + Hops = $hops.ToArray() + } +} + +# ===================================================================== +# Sideload-catalog schema migration recipes - dispatch table +# ===================================================================== +# Each value is a ScriptBlock with signature: +# param([string]$Text) -> @{ Text = ; Changes = @() } +# +# Recipes MUST be idempotent (running twice = same output as running once) +# and MUST update the top-level 'schemaVersion:' line themselves. +# +# The table ships EMPTY at schema v1 - there is no migration hop to run yet. +# This is intentional: the framework is wired up (engine + version var + +# reader guard + -SchemaMigrate entry point) so that the FIRST non-additive +# catalog format change is a small, well-tested recipe addition. +# +# To add the first hop in a future module version: +# 1. Bump $script:SideloadCatalogSchemaCurrentVersion in the .psm1. +# 2. Append a recipe here with the new 'N->N+1' key (text-surgery, +# idempotent, marker-keyed - see the schedule recipe table in +# Private/Convert-AzLocalScheduleSchemaVersion.ps1 for a worked +# example that preserves operator comments and rows verbatim). +# 3. Add tests in Tests/AzLocal.UpdateManagement.Tests.ps1 for the new +# hop in isolation AND chained from version 1. +# ===================================================================== +$script:SideloadCatalogSchemaRecipes = [ordered]@{} diff --git a/AzLocal.UpdateManagement/Private/ConvertFrom-AzLocalCronExpression.ps1 b/AzLocal.UpdateManagement/Private/ConvertFrom-AzLocalCronExpression.ps1 index 5e84eb3b..0b0c48c6 100644 --- a/AzLocal.UpdateManagement/Private/ConvertFrom-AzLocalCronExpression.ps1 +++ b/AzLocal.UpdateManagement/Private/ConvertFrom-AzLocalCronExpression.ps1 @@ -5,7 +5,7 @@ function ConvertFrom-AzLocalCronExpression { reference week (Sunday 00:00 -> Saturday 23:59 UTC). .DESCRIPTION Used by Test-AzLocalApplyUpdatesScheduleCoverage to decide whether a - cron entry from Step.6_apply-updates.yml covers any of the maintenance windows + cron entry from apply-updates.yml covers any of the maintenance windows derived from cluster UpdateStartWindow tags. Supports the subset of cron syntax that GitHub Actions and Azure DevOps diff --git a/AzLocal.UpdateManagement/Private/Export-ResultsToJUnitXml.ps1 b/AzLocal.UpdateManagement/Private/Export-ResultsToJUnitXml.ps1 index df32b835..add228ea 100644 --- a/AzLocal.UpdateManagement/Private/Export-ResultsToJUnitXml.ps1 +++ b/AzLocal.UpdateManagement/Private/Export-ResultsToJUnitXml.ps1 @@ -117,13 +117,13 @@ function Export-ResultsToJUnitXml { [void]$xmlBuilder.AppendLine("Cluster: $clusterName") [void]$xmlBuilder.AppendLine("Status: $($result.Status)") [void]$xmlBuilder.AppendLine("Message: $message") - if ($result.UpdateName) { + if ($result.PSObject.Properties['UpdateName'] -and $result.UpdateName) { [void]$xmlBuilder.AppendLine("Update: $(ConvertTo-XmlSafeString $result.UpdateName)") } - if ($result.CurrentState) { + if ($result.PSObject.Properties['CurrentState'] -and $result.CurrentState) { [void]$xmlBuilder.AppendLine("Current State: $(ConvertTo-XmlSafeString $result.CurrentState)") } - if ($result.Progress) { + if ($result.PSObject.Properties['Progress'] -and $result.Progress) { [void]$xmlBuilder.AppendLine("Progress: $($result.Progress)") } if ($result.PSObject.Properties['StartTime'] -and $result.StartTime) { @@ -168,13 +168,13 @@ function Export-ResultsToJUnitXml { if ($result.Message) { [void]$xmlBuilder.AppendLine("Message: $(ConvertTo-XmlSafeString $result.Message)") } - if ($result.UpdateName) { + if ($result.PSObject.Properties['UpdateName'] -and $result.UpdateName) { [void]$xmlBuilder.AppendLine("Update: $(ConvertTo-XmlSafeString $result.UpdateName)") } - if ($result.CurrentState) { + if ($result.PSObject.Properties['CurrentState'] -and $result.CurrentState) { [void]$xmlBuilder.AppendLine("Final State: $(ConvertTo-XmlSafeString $result.CurrentState)") } - if ($result.Progress) { + if ($result.PSObject.Properties['Progress'] -and $result.Progress) { [void]$xmlBuilder.AppendLine("Progress: $($result.Progress)") } if ($result.PSObject.Properties['StartTime'] -and $result.StartTime) { diff --git a/AzLocal.UpdateManagement/Private/Format-AzLocalUpdateRun.ps1 b/AzLocal.UpdateManagement/Private/Format-AzLocalUpdateRun.ps1 index 3c7744aa..e2904441 100644 --- a/AzLocal.UpdateManagement/Private/Format-AzLocalUpdateRun.ps1 +++ b/AzLocal.UpdateManagement/Private/Format-AzLocalUpdateRun.ps1 @@ -49,14 +49,21 @@ function Format-AzLocalUpdateRun { $totalSteps = @($steps).Count $progress = "$completedSteps/$totalSteps steps" - $inProgressStep = $steps | Where-Object { $_.status -eq "InProgress" } | Select-Object -First 1 - $failedStep = $steps | Where-Object { $_.status -in @("Error", "Failed") } | Select-Object -First 1 - - if ($inProgressStep) { - $currentStep = $inProgressStep.name - } - elseif ($failedStep) { - $currentStep = "$($failedStep.name) (FAILED)" + # Resolve the deepest InProgress/Error/Failed step in the nested progress tree. + # The top-level steps array only exposes a coarse wrapper step (e.g. "Start update") + # that stays InProgress for the entire run, so selecting the first InProgress/Failed + # step from that level alone always reported the wrapper as the "current step". + # Walking to the deepest active step keeps CurrentStep consistent with + # CurrentStepDetail and the standard update-progress output, both of which already + # traverse the full tree. + $deepestActive = Get-DeepestActiveStep -Steps $steps + if ($deepestActive) { + if ($deepestActive.status -in @("Error", "Failed")) { + $currentStep = "$($deepestActive.name) (FAILED)" + } + else { + $currentStep = $deepestActive.name + } } $currentStepDetail = Get-CurrentStepPath -Steps $steps -IncludeErrorMessage @@ -69,11 +76,11 @@ function Format-AzLocalUpdateRun { } } - # Per-step elapsed (v0.7.96): walk to the deepest InProgress/Failed step and surface its - # startTimeUtc + computed elapsed. This is the primary "is something stuck?" signal for - # large-node clusters where overall-run duration alone is unreliable (a 16-node legitimate - # run can easily exceed 8h while every individual step ticks along normally). - $deepestActive = Get-DeepestActiveStep -Steps $steps + # Per-step elapsed (v0.7.96): use the deepest InProgress/Failed step (resolved above) + # and surface its startTimeUtc + computed elapsed. This is the primary "is something + # stuck?" signal for large-node clusters where overall-run duration alone is unreliable + # (a 16-node legitimate run can easily exceed 8h while every individual step ticks along + # normally). # Deepest non-empty errorMessage from any Error/Failed step in the tree (v0.7.96). # Uses a coalesce(e8Msg..e1Msg) recursion so operators see the actual leaf failure diff --git a/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineId.ps1 b/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineId.ps1 new file mode 100644 index 00000000..b044c3d4 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineId.ps1 @@ -0,0 +1,57 @@ +function Get-AzLocalPipelineId { + <# + .SYNOPSIS + Extracts the stable logical pipeline ID from a bundled pipeline YAML's + '# AZLOCAL-PIPELINE-ID:' header comment. + + .DESCRIPTION + Private helper introduced in v0.8.7 supporting the rename-aware + behaviour of Update-AzLocalPipelineExample. Every bundled YAML carries + a single header comment of the form: + + # AZLOCAL-PIPELINE-ID: apply-updates + + The marker is a plain YAML comment (leading '#') and therefore has + zero runtime effect on GitHub Actions or Azure DevOps. The ID is the + de-numbered base name and is the STABLE identity that survives both + filename renames and display-step renumbering. + + Matching is case-insensitive on the 'AZLOCAL-PIPELINE-ID' token; any + leading whitespace and one-or-more '#' characters are tolerated. Only + the FIRST occurrence is returned. The captured ID is trimmed. + + .PARAMETER Text + The full YAML text to scan. Use Get-Content -Raw or + [IO.File]::ReadAllText to obtain it. + + .OUTPUTS + [string] - the logical pipeline ID, or $null if no marker is present + (e.g. a pre-v0.8.7 copy that predates the ID convention). + + .NOTES + Author : Neil Bird, Microsoft + Module : AzLocal.UpdateManagement + Added in : v0.8.7 + #> + [CmdletBinding()] + [OutputType([string])] + param( + [Parameter(Mandatory = $true)] + [AllowEmptyString()] + [string]$Text + ) + + if ([string]::IsNullOrEmpty($Text)) { + return $null + } + + # (?m) multiline so ^ matches each line start. Tolerate leading spaces, + # one-or-more '#', spaces around the colon. Capture the identifier + # ([A-Za-z0-9_-]+). + $match = [regex]::Match($Text, '(?im)^\s*#+\s*AZLOCAL-PIPELINE-ID\s*:\s*([A-Za-z0-9_-]+)\s*$') + if ($match.Success) { + return $match.Groups[1].Value.Trim() + } + + return $null +} diff --git a/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineManifest.ps1 b/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineManifest.ps1 new file mode 100644 index 00000000..091fca7a --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Get-AzLocalPipelineManifest.ps1 @@ -0,0 +1,95 @@ +function Get-AzLocalPipelineManifest { + <# + .SYNOPSIS + Returns the canonical pipeline manifest mapping each logical pipeline + ID to its current bundled filename, display step number and the + historical filenames it may have shipped under. + + .DESCRIPTION + Private data helper introduced in v0.8.7. It is the single source of + truth that decouples a pipeline's STABLE IDENTITY (the logical ID, + e.g. 'apply-updates') from its volatile PRESENTATION (the display step + number 'Step.7 - ...' and, historically, the filename prefix + 'Step.7_apply-updates.yml'). + + Why this exists + --------------- + Up to and including v0.8.6 the bundled CI/CD YAMLs encoded their + ordering in BOTH the filename ('Step.7_apply-updates.yml') and the + display name ('Step.7 - Apply Updates ...'). Update-AzLocalPipelineExample + matched a customer's destination file to a bundled source file by + EXACT FILENAME. That made any renumbering a breaking change: when + v0.8.7 moved apply-updates from Step.6 to Step.7, a customer's + 'Step.6_apply-updates.yml' (carrying their CRON triggers inside the + BEGIN/END-AZLOCAL-CUSTOMIZE:schedule-triggers markers) became an + orphan, and the new 'Step.7_apply-updates.yml' was treated as a + brand-new file - silently stranding the customer's schedule. + + v0.8.7 fixes this two ways: + 1. The bundled filenames lose their 'Step.N_' prefix entirely and + become stable forever (e.g. 'apply-updates.yml'). The step + number now lives ONLY in the display name and artifact names, + which flow through the marker-aware merge automatically. + 2. Every bundled YAML carries a '# AZLOCAL-PIPELINE-ID: ' + comment, and this manifest records each ID's canonical filename + plus the legacy filenames it used to ship under (Aliases). + Update-AzLocalPipelineExample uses the ID/alias mapping to find + a customer's file even when it is still named with an old + 'Step.N_' prefix, then renames it on disk while carrying the + customer's marker bodies (CRONs) into the new file. + + Both platforms (GitHub Actions and Azure DevOps) share the same + de-numbered base name + '.yml' for a given logical ID, so a single + FileName/Aliases set covers both - the only difference is the + platform subfolder the file lives in. + + .OUTPUTS + PSCustomObject[] - one row per logical pipeline, in display order: + Id - stable logical identifier (de-numbered base name) + FileName - canonical bundled filename ('.yml') + DisplayStep - current Step.N number shown in the display name / + artifact names (presentation only) + IntroducedIn - module version the pipeline first shipped in + Aliases - [string[]] historical filenames this pipeline has + shipped under (most-recent first). Empty for + pipelines that are net-new in this release. + + .NOTES + Author : Neil Bird, Microsoft + Module : AzLocal.UpdateManagement + Added in : v0.8.7 + + MAINTENANCE: when a pipeline is renamed, renumbered or added, update + ONLY this manifest and the '# AZLOCAL-PIPELINE-ID' comment in the YAML. + - Renumber (display only): bump DisplayStep, leave FileName/Id alone. + Nothing else needs to change - Update merges the new display name in. + - Rename a file: change FileName, PREPEND the old filename to Aliases. + - Add a pipeline: add a new row with the next DisplayStep, a fresh Id, + IntroducedIn = the new module version and Aliases = @(). + #> + [CmdletBinding()] + [OutputType([PSCustomObject[]])] + param() + + # Display order follows the Step.N numbering shown in the rendered + # GitHub Actions / Azure DevOps pipeline list. Aliases list the historical + # 'Step.N_.yml' filenames most-recent first. The four pipelines that + # were renumbered in v0.8.7 Group R (apply 6->7, monitor 7->8, + # fleet-update 8->9, fleet-health 9->10) carry BOTH the post-renumber and + # the pre-renumber legacy filename so Update can recover a customer's CRONs + # regardless of which copy they started from. sideload-updates is net-new + # in v0.8.7 and therefore has no aliases. + @( + [PSCustomObject]@{ Id = 'authentication-test'; FileName = 'authentication-test.yml'; DisplayStep = 0; IntroducedIn = '0.7.0'; Aliases = @('Step.0_authentication-test.yml') } + [PSCustomObject]@{ Id = 'inventory-clusters'; FileName = 'inventory-clusters.yml'; DisplayStep = 1; IntroducedIn = '0.7.0'; Aliases = @('Step.1_inventory-clusters.yml') } + [PSCustomObject]@{ Id = 'manage-updatering-tags'; FileName = 'manage-updatering-tags.yml'; DisplayStep = 2; IntroducedIn = '0.7.0'; Aliases = @('Step.2_manage-updatering-tags.yml') } + [PSCustomObject]@{ Id = 'apply-updates-schedule-audit'; FileName = 'apply-updates-schedule-audit.yml'; DisplayStep = 3; IntroducedIn = '0.7.0'; Aliases = @('Step.3_apply-updates-schedule-audit.yml') } + [PSCustomObject]@{ Id = 'fleet-connectivity-status'; FileName = 'fleet-connectivity-status.yml'; DisplayStep = 4; IntroducedIn = '0.7.0'; Aliases = @('Step.4_fleet-connectivity-status.yml') } + [PSCustomObject]@{ Id = 'assess-update-readiness'; FileName = 'assess-update-readiness.yml'; DisplayStep = 5; IntroducedIn = '0.7.0'; Aliases = @('Step.5_assess-update-readiness.yml') } + [PSCustomObject]@{ Id = 'sideload-updates'; FileName = 'sideload-updates.yml'; DisplayStep = 6; IntroducedIn = '0.8.7'; Aliases = @() } + [PSCustomObject]@{ Id = 'apply-updates'; FileName = 'apply-updates.yml'; DisplayStep = 7; IntroducedIn = '0.7.0'; Aliases = @('Step.7_apply-updates.yml', 'Step.6_apply-updates.yml') } + [PSCustomObject]@{ Id = 'monitor-updates'; FileName = 'monitor-updates.yml'; DisplayStep = 8; IntroducedIn = '0.7.0'; Aliases = @('Step.8_monitor-updates.yml', 'Step.7_monitor-updates.yml') } + [PSCustomObject]@{ Id = 'fleet-update-status'; FileName = 'fleet-update-status.yml'; DisplayStep = 9; IntroducedIn = '0.7.0'; Aliases = @('Step.9_fleet-update-status.yml', 'Step.8_fleet-update-status.yml') } + [PSCustomObject]@{ Id = 'fleet-health-status'; FileName = 'fleet-health-status.yml'; DisplayStep = 10; IntroducedIn = '0.7.0'; Aliases = @('Step.10_fleet-health-status.yml', 'Step.9_fleet-health-status.yml') } + ) +} diff --git a/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadAuthMap.ps1 b/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadAuthMap.ps1 new file mode 100644 index 00000000..3b8d76e7 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadAuthMap.ps1 @@ -0,0 +1,138 @@ +function Get-AzLocalSideloadAuthMap { + <# + .SYNOPSIS + Parses the sideload auth-map CSV that maps the numeric UpdateAuthAccountId + cluster tag to the Key Vault secrets + remoting settings used to stage a + sideloaded solution update on an on-premises Azure Local cluster. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation feature. + + Reads a CSV with these columns: + - UpdateAuthAccountId (REQUIRED) numeric account id, 1-3 digits + (e.g. 001). Matches the per-cluster + 'UpdateAuthAccountId' tag written by Step.2. + - KeyVaultName (REQUIRED) Key Vault holding the AD credential. + - UsernameSecretName (REQUIRED) KV secret holding the username + (UPN 'user@contoso.com' recommended; + 'DOMAIN\user' also accepted). + - PasswordSecretName (REQUIRED) KV secret holding the password. + - RemotingTargetFqdn (optional) explicit FQDN to remote to; when + set it overrides the cluster-name + suffix + resolution. + - FqdnSuffix (optional) per-row DNS suffix appended to the + cluster name (e.g. '.corp.contoso.com'). + - AuthMechanism (optional) WinRM auth mechanism + (Kerberos|Negotiate|Default). Defaults to + Default at the remoting layer. + - ImportSharePath (optional) override for the cluster + infrastructure 'import' UNC share. + + Validation: + - UpdateAuthAccountId must match ^\d{1,3}$ (numeric only). The raw + (trimmed) string is used verbatim as the lookup key so it must + match the tag value exactly. + - KeyVaultName, UsernameSecretName, PasswordSecretName must be + non-empty. + - A DUPLICATE UpdateAuthAccountId is a HARD ERROR (the mapping would + be ambiguous). This is a pre-requisite gate for the sideload + pipeline. + + Returns a hashtable keyed by the trimmed UpdateAuthAccountId string, + whose values are [PSCustomObject] rows. An empty CSV returns an empty + hashtable. + + .PARAMETER Path + Path to the auth-map CSV file. + + .OUTPUTS + [hashtable] keyed by UpdateAuthAccountId -> [PSCustomObject]. + #> + [CmdletBinding()] + [OutputType([hashtable])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$Path + ) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Sideload auth-map CSV not found at '$Path'. Set SIDELOAD_AUTH_MAP_PATH (or pass -Path) to a CSV with columns UpdateAuthAccountId,KeyVaultName,UsernameSecretName,PasswordSecretName." + } + + try { + # Tolerate inline documentation: drop blank lines and '#'-prefixed + # comment lines (Import-Csv has no native comment support) before + # parsing. The real header is the first non-comment line. + $rawLines = @(Get-Content -LiteralPath $Path -ErrorAction Stop) + $dataLines = @($rawLines | Where-Object { + -not [string]::IsNullOrWhiteSpace($_) -and ($_.TrimStart())[0] -ne '#' + }) + $rows = if ($dataLines.Count -gt 0) { @($dataLines | ConvertFrom-Csv) } else { @() } + } + catch { + throw "Failed to read sideload auth-map CSV '$Path': $($_.Exception.Message)" + } + + $authMap = @{} + if ($rows.Count -eq 0) { + return $authMap + } + + # Column presence is validated from the first row's properties so a + # misspelled header surfaces clearly instead of as silent $null fields. + $firstRow = $rows[0] + $required = @('UpdateAuthAccountId', 'KeyVaultName', 'UsernameSecretName', 'PasswordSecretName') + $columns = @($firstRow.PSObject.Properties.Name) + $missing = @($required | Where-Object { $columns -notcontains $_ }) + if ($missing.Count -gt 0) { + throw "Sideload auth-map CSV '$Path' is missing required column(s): $($missing -join ', '). Required columns: $($required -join ', ')." + } + + $rowIndex = 0 + foreach ($row in $rows) { + $rowIndex++ + $accountId = ([string]$row.UpdateAuthAccountId).Trim() + if ([string]::IsNullOrWhiteSpace($accountId)) { + throw "Sideload auth-map CSV '$Path' row $rowIndex has an empty UpdateAuthAccountId." + } + if ($accountId -notmatch '^\d{1,3}$') { + throw "Sideload auth-map CSV '$Path' row $rowIndex has an invalid UpdateAuthAccountId '$accountId'. Must be numeric (1-3 digits, e.g. 001)." + } + if ($authMap.ContainsKey($accountId)) { + throw "Sideload auth-map CSV '$Path' contains a DUPLICATE UpdateAuthAccountId '$accountId' (row $rowIndex). Each account id must be unique." + } + + $keyVaultName = ([string]$row.KeyVaultName).Trim() + $usernameSecret = ([string]$row.UsernameSecretName).Trim() + $passwordSecret = ([string]$row.PasswordSecretName).Trim() + if ([string]::IsNullOrWhiteSpace($keyVaultName)) { + throw "Sideload auth-map CSV '$Path' row $rowIndex (UpdateAuthAccountId '$accountId') has an empty KeyVaultName." + } + if ([string]::IsNullOrWhiteSpace($usernameSecret)) { + throw "Sideload auth-map CSV '$Path' row $rowIndex (UpdateAuthAccountId '$accountId') has an empty UsernameSecretName." + } + if ([string]::IsNullOrWhiteSpace($passwordSecret)) { + throw "Sideload auth-map CSV '$Path' row $rowIndex (UpdateAuthAccountId '$accountId') has an empty PasswordSecretName." + } + + # Optional columns are tolerated as absent properties. + $remotingTargetFqdn = if ($columns -contains 'RemotingTargetFqdn') { ([string]$row.RemotingTargetFqdn).Trim() } else { '' } + $fqdnSuffix = if ($columns -contains 'FqdnSuffix') { ([string]$row.FqdnSuffix).Trim() } else { '' } + $authMechanism = if ($columns -contains 'AuthMechanism') { ([string]$row.AuthMechanism).Trim() } else { '' } + $importSharePath = if ($columns -contains 'ImportSharePath') { ([string]$row.ImportSharePath).Trim() } else { '' } + + $authMap[$accountId] = [PSCustomObject]@{ + UpdateAuthAccountId = $accountId + KeyVaultName = $keyVaultName + UsernameSecretName = $usernameSecret + PasswordSecretName = $passwordSecret + RemotingTargetFqdn = $remotingTargetFqdn + FqdnSuffix = $fqdnSuffix + AuthMechanism = $authMechanism + ImportSharePath = $importSharePath + } + } + + return $authMap +} diff --git a/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadCatalog.ps1 b/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadCatalog.ps1 new file mode 100644 index 00000000..8c2ac39b --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadCatalog.ps1 @@ -0,0 +1,208 @@ +function Get-AzLocalSideloadCatalog { + <# + .SYNOPSIS + Parses the source-controlled sideload catalog YAML into package entries + describing the Azure Local solution-update media (and OEM SBE packages) + available to the on-prem sideloading automation. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation feature. + Zero external YAML dependency - parses only the narrow shape below. + + Two package classes are supported (PackageType): + + - Solution : a Microsoft CombinedSolutionBundle..zip that is + downloadable from a direct DownloadUri (published in the Microsoft + Learn 'import and discover updates offline' table). Sha256 is + REQUIRED so the download / pre-staged copy can be verified. + + - SBE : an OEM Solution Builder Extension package that Microsoft does + NOT host. The operator stages the OEM files manually and records a + SourceFolder (local or UNC path). DownloadUri is not applicable; + Sha256 is optional (verified only when supplied). + + Expected YAML shape (2-space indentation): + + schemaVersion: 1 + packages: + - version: '12.2605.1003.210' + packageType: Solution + buildNumber: '12.2605.1003.210' + osBuild: '26100.4061' + downloadUri: 'https://.../CombinedSolutionBundle.12.2605.1003.210.zip' + sha256: 'ABCD...' + availabilityDate: '2026-05-13' + localPath: '' + - version: 'DellSBE-4.1.2412.1' + packageType: SBE + sourceFolder: '\\fileserver\sbe\Dell\4.1.2412.1' + sha256: '' + availabilityDate: '2026-05-20' + notes: 'Dell OEM SBE package, staged manually' + + Validation: + - At least the 'version' key is required per package; duplicates are a + hard error. + - packageType defaults to 'Solution' when omitted; must be + Solution or SBE (case-insensitive). + - Solution entries require a non-empty DownloadUri OR LocalPath, plus a + Sha256 matching ^[0-9A-Fa-f]{64}$. + - SBE entries require a non-empty SourceFolder. Sha256, when present, + must match ^[0-9A-Fa-f]{64}$. + + Returns an array of [PSCustomObject] package entries (empty array when + the catalog has no packages). + + .PARAMETER Path + Path to the catalog YAML file. + + .OUTPUTS + [PSCustomObject[]] one entry per package. + #> + [CmdletBinding()] + [OutputType([PSCustomObject[]])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$Path + ) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Sideload catalog YAML not found at '$Path'. Set SIDELOAD_CATALOG_PATH (or pass -Path) to a catalog file. Generate / refresh one with Update-AzLocalSideloadCatalog." + } + + $lines = @(Get-Content -LiteralPath $Path -ErrorAction Stop) + $packages = New-Object System.Collections.Generic.List[PSCustomObject] + $seenVersions = New-Object System.Collections.Generic.HashSet[string] ([System.StringComparer]::OrdinalIgnoreCase) + + $inPackages = $false + $current = $null + $lineNo = 0 + + # Strips one matching pair of surrounding quotes and a trailing inline + # comment. Bare values are returned trimmed. + $parseScalar = { + param([string]$raw) + $v = $raw.Trim() + if ($v.Length -ge 2) { + $first = $v[0]; $last = $v[$v.Length - 1] + if (($first -eq "'" -and $last -eq "'") -or ($first -eq '"' -and $last -eq '"')) { + return $v.Substring(1, $v.Length - 2) + } + } + # Strip trailing ' # comment' only on unquoted scalars. + $hashIndex = $v.IndexOf(' #') + if ($hashIndex -ge 0) { $v = $v.Substring(0, $hashIndex).Trim() } + return $v + } + + $commit = { + param($entry) + if ($null -eq $entry) { return } + $version = ([string]$entry['version']).Trim() + if ([string]::IsNullOrWhiteSpace($version)) { + throw "Sideload catalog '$Path' has a package entry with no 'version'." + } + if (-not $seenVersions.Add($version)) { + throw "Sideload catalog '$Path' contains a DUPLICATE package version '$version'. Each version must be unique." + } + + $packageType = ([string]$entry['packageType']).Trim() + if ([string]::IsNullOrWhiteSpace($packageType)) { $packageType = 'Solution' } + if ($packageType -notmatch '^(?i:Solution|SBE)$') { + throw "Sideload catalog '$Path' package '$version' has an invalid packageType '$packageType'. Must be 'Solution' or 'SBE'." + } + $packageType = if ($packageType -match '^(?i:SBE)$') { 'SBE' } else { 'Solution' } + + $downloadUri = ([string]$entry['downloadUri']).Trim() + $localPath = ([string]$entry['localPath']).Trim() + $sourceFolder = ([string]$entry['sourceFolder']).Trim() + $sha256 = ([string]$entry['sha256']).Trim() + + if ($packageType -eq 'Solution') { + if ([string]::IsNullOrWhiteSpace($downloadUri) -and [string]::IsNullOrWhiteSpace($localPath)) { + throw "Sideload catalog '$Path' Solution package '$version' must have a non-empty downloadUri or localPath." + } + if ($sha256 -notmatch '^[0-9A-Fa-f]{64}$') { + throw "Sideload catalog '$Path' Solution package '$version' must have a valid SHA256 (64 hex chars). Got: '$sha256'." + } + } + else { + if ([string]::IsNullOrWhiteSpace($sourceFolder)) { + throw "Sideload catalog '$Path' SBE package '$version' must have a non-empty sourceFolder (the operator-staged OEM content path)." + } + if (-not [string]::IsNullOrWhiteSpace($sha256) -and $sha256 -notmatch '^[0-9A-Fa-f]{64}$') { + throw "Sideload catalog '$Path' SBE package '$version' has an invalid SHA256 (must be 64 hex chars or empty). Got: '$sha256'." + } + } + + $packages.Add([PSCustomObject]@{ + Version = $version + PackageType = $packageType + BuildNumber = ([string]$entry['buildNumber']).Trim() + OsBuild = ([string]$entry['osBuild']).Trim() + DownloadUri = $downloadUri + Sha256 = $sha256 + AvailabilityDate = ([string]$entry['availabilityDate']).Trim() + LocalPath = $localPath + SourceFolder = $sourceFolder + Notes = ([string]$entry['notes']).Trim() + }) + } + + foreach ($rawLine in $lines) { + $lineNo++ + $line = $rawLine.TrimEnd() + if ([string]::IsNullOrWhiteSpace($line)) { continue } + $trimmed = $line.Trim() + if ($trimmed.StartsWith('#')) { continue } + + if (-not $inPackages) { + if ($trimmed -match '^packages:\s*$') { + $inPackages = $true + } + elseif ($trimmed -match '^schemaVersion:\s*(.+)$') { + # Guard against a catalog written by a NEWER module. A higher + # schemaVersion may carry fields/semantics this reader does + # not understand, so refuse rather than silently mis-parse. + # Mirrors the schedule-file behaviour. Missing schemaVersion + # is tolerated (treated as v1) for back-compat. + $svRaw = (& $parseScalar $Matches[1]) + $sv = 0 + if (-not [int]::TryParse($svRaw, [ref]$sv)) { + throw "Sideload catalog '$Path' has a non-integer schemaVersion '$svRaw'." + } + if ($sv -gt $script:SideloadCatalogSchemaCurrentVersion) { + throw "Sideload catalog '$Path' is on schemaVersion=$sv but this module only supports up to $($script:SideloadCatalogSchemaCurrentVersion). Upgrade the AzLocal.UpdateManagement module (Update-Module AzLocal.UpdateManagement), then re-read the catalog." + } + } + # Other top-level scalars are ignored - only the packages list + # is consumed. + continue + } + + if ($trimmed -match '^-\s*(.*)$') { + # New list item - commit the previous entry first. + & $commit $current + $current = @{} + $rest = $Matches[1].Trim() + if (-not [string]::IsNullOrWhiteSpace($rest)) { + if ($rest -match '^([A-Za-z0-9_]+):\s*(.*)$') { + $current[$Matches[1]] = (& $parseScalar $Matches[2]) + } + } + continue + } + + if ($trimmed -match '^([A-Za-z0-9_]+):\s*(.*)$') { + if ($null -eq $current) { + throw "Sideload catalog '$Path' line $lineNo has a key outside a package list item: '$trimmed'." + } + $current[$Matches[1]] = (& $parseScalar $Matches[2]) + continue + } + } + & $commit $current + + return $packages.ToArray() +} diff --git a/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadState.ps1 b/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadState.ps1 new file mode 100644 index 00000000..5f5c2121 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Get-AzLocalSideloadState.ps1 @@ -0,0 +1,162 @@ +function New-AzLocalSideloadState { + <# + .SYNOPSIS + Builds a fresh sideload state object for a cluster. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. Produces the + canonical state record persisted (one JSON per cluster) under the SHARED + UNC state root so that ANY runner/agent can read it without cross-agent + remoting. The detached copy Scheduled Task and each re-entrant pipeline run + read/update this record to advance the per-cluster state machine. + + .OUTPUTS + [PSCustomObject] the state record. + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + [Parameter(Mandatory = $true)][string]$ClusterName, + [Parameter(Mandatory = $true)][string]$Version, + [string]$TaskName = '', + [string]$MediaPath = '', + [string]$TargetPath = '', + [string]$LogPath = '', + [ValidateSet('Queued', 'Copying', 'Copied', 'Verified', 'Imported', 'Failed', 'Stale')] + [string]$State = 'Queued' + ) + + $nowUtc = [DateTime]::UtcNow.ToString('o') + return [PSCustomObject]@{ + ClusterName = $ClusterName + Version = $Version + State = $State + OwningMachine = $env:COMPUTERNAME + TaskName = $TaskName + MediaPath = $MediaPath + TargetPath = $TargetPath + LogPath = $LogPath + StartUtc = $nowUtc + LastHeartbeatUtc = $nowUtc + TotalBytes = [long]0 + CopiedBytes = [long]0 + Mbps = [double]0 + EtaUtc = '' + ExitCode = $null + Retries = [int]0 + Message = '' + } +} + +function Get-AzLocalSideloadStatePath { + <# + .SYNOPSIS + Returns the shared-state JSON path for a cluster, creating the state + directory if needed. + .OUTPUTS + [string] + #> + [CmdletBinding()] + [OutputType([string])] + param( + [Parameter(Mandatory = $true)][string]$StateRoot, + [Parameter(Mandatory = $true)][string]$ClusterName + ) + + $stateDir = Join-Path -Path $StateRoot -ChildPath 'state' + if (-not (Test-Path -LiteralPath $stateDir)) { + New-Item -ItemType Directory -Path $stateDir -Force | Out-Null + } + # Sanitize cluster name for use as a file name. + $safe = ($ClusterName -replace '[^A-Za-z0-9._-]', '_') + return (Join-Path -Path $stateDir -ChildPath ("{0}.json" -f $safe)) +} + +function Get-AzLocalSideloadState { + <# + .SYNOPSIS + Reads the shared sideload state record for a cluster; returns $null when + no state exists yet. + .OUTPUTS + [PSCustomObject] or $null + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + [Parameter(Mandatory = $true)][string]$StateRoot, + [Parameter(Mandatory = $true)][string]$ClusterName + ) + + $path = Get-AzLocalSideloadStatePath -StateRoot $StateRoot -ClusterName $ClusterName + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + return $null + } + try { + $raw = Get-Content -LiteralPath $path -Raw -ErrorAction Stop + if ([string]::IsNullOrWhiteSpace($raw)) { return $null } + return ($raw | ConvertFrom-Json -ErrorAction Stop) + } + catch { + throw "Failed to read sideload state for cluster '$ClusterName' from '$path': $($_.Exception.Message)" + } +} + +function Set-AzLocalSideloadState { + <# + .SYNOPSIS + Persists a sideload state record to the shared UNC state directory using an + atomic temp-write + move so concurrent readers never see a partial file. + .OUTPUTS + [void] + #> + [CmdletBinding(SupportsShouldProcess = $true)] + [OutputType([void])] + param( + [Parameter(Mandatory = $true)][string]$StateRoot, + [Parameter(Mandatory = $true)][ValidateNotNull()][PSCustomObject]$State + ) + + if ([string]::IsNullOrWhiteSpace([string]$State.ClusterName)) { + throw "Set-AzLocalSideloadState: State object has no ClusterName." + } + + $path = Get-AzLocalSideloadStatePath -StateRoot $StateRoot -ClusterName $State.ClusterName + if (-not $PSCmdlet.ShouldProcess($path, 'Write sideload state')) { return } + + $json = $State | ConvertTo-Json -Depth 6 + $temp = "{0}.{1}.partial" -f $path, ([guid]::NewGuid().ToString('N')) + try { + Write-Utf8NoBomFile -Path $temp -Content $json + Move-Item -LiteralPath $temp -Destination $path -Force + } + finally { + if (Test-Path -LiteralPath $temp -PathType Leaf) { + Remove-Item -LiteralPath $temp -Force -ErrorAction SilentlyContinue + } + } +} + +function Test-AzLocalSideloadHeartbeatStale { + <# + .SYNOPSIS + Returns $true when a Copying state's heartbeat is older than the stale + threshold (the owning runner/agent likely died), so the next pipeline run + can re-drive the copy on a live host. + .OUTPUTS + [bool] + #> + [CmdletBinding()] + [OutputType([bool])] + param( + [Parameter(Mandatory = $true)][ValidateNotNull()][PSCustomObject]$State, + [Parameter(Mandatory = $true)][int]$StaleMinutes + ) + + if ($State.State -ne 'Copying') { return $false } + [DateTime]$last = [DateTime]::MinValue + if (-not [DateTime]::TryParse([string]$State.LastHeartbeatUtc, [ref]$last)) { + return $true + } + $ageMinutes = ([DateTime]::UtcNow - $last.ToUniversalTime()).TotalMinutes + return ($ageMinutes -gt $StaleMinutes) +} diff --git a/AzLocal.UpdateManagement/Private/Get-AzLocalSolutionUpdateDownload.ps1 b/AzLocal.UpdateManagement/Private/Get-AzLocalSolutionUpdateDownload.ps1 new file mode 100644 index 00000000..26c638a2 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Get-AzLocalSolutionUpdateDownload.ps1 @@ -0,0 +1,211 @@ +function Get-AzLocalSolutionUpdateDownload { + <# + .SYNOPSIS + Resolves a sideload catalog entry to a verified, locally available media + path - serving from a shared cache, downloading the Microsoft + CombinedSolutionBundle when needed, or returning the operator-staged OEM + SBE source folder. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. + + For a 'Solution' catalog entry: + 1. If LocalPath is set and exists, it is the pre-staged bundle - the + SHA256 is still verified. + 2. Else, if the shared cache already holds the bundle and its + Get-FileHash matches the catalog SHA256, it is served from cache. + 3. Else the bundle is downloaded from DownloadUri to a per-process temp + file in the cache directory and atomically moved into place + (concurrent-agent safe), then SHA256-verified. + + For an 'SBE' catalog entry: Microsoft does not host the package. The + operator-staged SourceFolder is validated to exist; its SHA256 is + verified only when the catalog supplies one (folder hashing is not + attempted). No download occurs. + + Network + hashing calls are isolated in Invoke-AzLocalFileDownload and + Get-AzLocalFileHashText so they can be mocked in unit tests. + + .PARAMETER CatalogEntry + A catalog entry [PSCustomObject] from Get-AzLocalSideloadCatalog. + + .PARAMETER CacheRoot + Directory used as the shared verified cache for downloaded Solution + bundles. Created if missing. + + .OUTPUTS + [PSCustomObject] with: + Version the catalog version + PackageType 'Solution' | 'SBE' + MediaPath the verified bundle .zip (Solution) or staged folder (SBE) + Sha256 the verified hash (when applicable) + FromCache [bool] served from the shared cache without downloading + Downloaded [bool] a download occurred this call + Verified [bool] a SHA256 comparison succeeded this call + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNull()] + [PSCustomObject]$CatalogEntry, + + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$CacheRoot + ) + + $version = [string]$CatalogEntry.Version + $packageType = if ($CatalogEntry.PackageType) { [string]$CatalogEntry.PackageType } else { 'Solution' } + + # ---- SBE (operator-staged, no download) ---------------------------- + if ($packageType -eq 'SBE') { + $sourceFolder = [string]$CatalogEntry.SourceFolder + if ([string]::IsNullOrWhiteSpace($sourceFolder)) { + throw "Sideload SBE package '$version' has no SourceFolder. Stage the OEM files and set SourceFolder in the catalog." + } + if (-not (Test-Path -LiteralPath $sourceFolder)) { + throw "Sideload SBE package '$version' SourceFolder '$sourceFolder' does not exist or is not reachable from this runner/agent." + } + $verified = $false + # Folder-level hashing is not performed; only a single-file SourceFolder + # (pointing at a .zip) is hash-verified when a SHA256 is supplied. + if (-not [string]::IsNullOrWhiteSpace([string]$CatalogEntry.Sha256) -and (Test-Path -LiteralPath $sourceFolder -PathType Leaf)) { + $actual = Get-AzLocalFileHashText -Path $sourceFolder + if (-not [string]::Equals($actual, [string]$CatalogEntry.Sha256, [System.StringComparison]::OrdinalIgnoreCase)) { + throw "Sideload SBE package '$version' SHA256 mismatch. Expected $($CatalogEntry.Sha256), got $actual." + } + $verified = $true + } + return [PSCustomObject]@{ + Version = $version + PackageType = 'SBE' + MediaPath = $sourceFolder + Sha256 = [string]$CatalogEntry.Sha256 + FromCache = $false + Downloaded = $false + Verified = $verified + } + } + + # ---- Solution (cache / pre-staged / download) ---------------------- + $expectedSha = [string]$CatalogEntry.Sha256 + if ([string]::IsNullOrWhiteSpace($expectedSha)) { + throw "Sideload Solution package '$version' has no SHA256 in the catalog; refusing to stage unverifiable media." + } + + # 1. Pre-staged LocalPath. + $localPath = [string]$CatalogEntry.LocalPath + if (-not [string]::IsNullOrWhiteSpace($localPath) -and (Test-Path -LiteralPath $localPath -PathType Leaf)) { + $actual = Get-AzLocalFileHashText -Path $localPath + if (-not [string]::Equals($actual, $expectedSha, [System.StringComparison]::OrdinalIgnoreCase)) { + throw "Sideload Solution package '$version' pre-staged LocalPath '$localPath' SHA256 mismatch. Expected $expectedSha, got $actual." + } + return [PSCustomObject]@{ + Version = $version + PackageType = 'Solution' + MediaPath = $localPath + Sha256 = $expectedSha + FromCache = $false + Downloaded = $false + Verified = $true + } + } + + if (-not (Test-Path -LiteralPath $CacheRoot)) { + New-Item -ItemType Directory -Path $CacheRoot -Force | Out-Null + } + + $fileName = "CombinedSolutionBundle.$version.zip" + if ($CatalogEntry.BuildNumber) { $fileName = "CombinedSolutionBundle.$($CatalogEntry.BuildNumber).zip" } + $cachePath = Join-Path -Path $CacheRoot -ChildPath $fileName + + # 2. Shared cache hit. + if (Test-Path -LiteralPath $cachePath -PathType Leaf) { + $actual = Get-AzLocalFileHashText -Path $cachePath + if ([string]::Equals($actual, $expectedSha, [System.StringComparison]::OrdinalIgnoreCase)) { + return [PSCustomObject]@{ + Version = $version + PackageType = 'Solution' + MediaPath = $cachePath + Sha256 = $expectedSha + FromCache = $true + Downloaded = $false + Verified = $true + } + } + Write-Log -Message "Sideload cache file '$cachePath' failed SHA256 verification (expected $expectedSha, got $actual); re-downloading." -Level Warning + } + + # 3. Download. Write to a per-process temp file then atomically move into + # place so concurrent runners/agents never read a partial file. + $downloadUri = [string]$CatalogEntry.DownloadUri + if ([string]::IsNullOrWhiteSpace($downloadUri)) { + throw "Sideload Solution package '$version' has no DownloadUri and is not present (verified) in the cache or a pre-staged LocalPath." + } + $tempPath = Join-Path -Path $CacheRoot -ChildPath ("{0}.{1}.partial" -f $fileName, ([guid]::NewGuid().ToString('N'))) + try { + Write-Log -Message "Downloading sideload bundle '$version' from '$downloadUri'." -Level Info + Invoke-AzLocalFileDownload -Uri $downloadUri -OutFile $tempPath + + $actual = Get-AzLocalFileHashText -Path $tempPath + if (-not [string]::Equals($actual, $expectedSha, [System.StringComparison]::OrdinalIgnoreCase)) { + throw "Sideload Solution package '$version' downloaded SHA256 mismatch. Expected $expectedSha, got $actual." + } + Move-Item -LiteralPath $tempPath -Destination $cachePath -Force + } + finally { + if (Test-Path -LiteralPath $tempPath -PathType Leaf) { + Remove-Item -LiteralPath $tempPath -Force -ErrorAction SilentlyContinue + } + } + + return [PSCustomObject]@{ + Version = $version + PackageType = 'Solution' + MediaPath = $cachePath + Sha256 = $expectedSha + FromCache = $false + Downloaded = $true + Verified = $true + } +} + +function Invoke-AzLocalFileDownload { + <# + .SYNOPSIS + Thin wrapper around Invoke-WebRequest -OutFile, isolated for mocking. + .OUTPUTS + [void] + #> + [CmdletBinding()] + [OutputType([void])] + param( + [Parameter(Mandatory = $true)][string]$Uri, + [Parameter(Mandatory = $true)][string]$OutFile + ) + + try { + Invoke-WebRequest -Uri $Uri -OutFile $OutFile -UseBasicParsing -ErrorAction Stop + } + catch { + throw "Failed to download '$Uri': $($_.Exception.Message)" + } +} + +function Get-AzLocalFileHashText { + <# + .SYNOPSIS + Thin wrapper around Get-FileHash (SHA256), isolated for mocking. + .OUTPUTS + [string] the uppercase SHA256 hex string. + #> + [CmdletBinding()] + [OutputType([string])] + param( + [Parameter(Mandatory = $true)][string]$Path + ) + + $h = Get-FileHash -LiteralPath $Path -Algorithm SHA256 -ErrorAction Stop + return ([string]$h.Hash).ToUpperInvariant() +} diff --git a/AzLocal.UpdateManagement/Private/Invoke-AzLocalRemoteSolutionImport.ps1 b/AzLocal.UpdateManagement/Private/Invoke-AzLocalRemoteSolutionImport.ps1 new file mode 100644 index 00000000..e58279c8 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Invoke-AzLocalRemoteSolutionImport.ps1 @@ -0,0 +1,176 @@ +function Invoke-AzLocalRemoteSolutionImport { + <# + .SYNOPSIS + Imports (stages + discovers) a sideloaded solution update on an Azure Local + cluster node over WinRM so it becomes available to the apply pipeline. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. Runs the + documented offline import sequence on the target node: + + 1. Expand the CombinedSolutionBundle .zip (already robocopied into the + import share) into the import\Solution folder. + 2. Run Add-SolutionUpdate against the import folder to register it. + 3. Poll Get-SolutionUpdate until the matching version is discovered, or + until it reports AdditionalContentRequired (an OEM SBE package must be + staged alongside it), or the discovery timeout elapses. + + This does NOT apply/install the update - that remains the job of the apply + pipeline (Step.7) once UpdateSideloaded is flipped True. The function only + confirms the media is discoverable so the gate can be opened. + + For SBE packages the operator-staged SBE content is already robocopied into + the import share; the same Add-SolutionUpdate / Get-SolutionUpdate discovery + applies. When a Solution reports AdditionalContentRequired and no SBE has + been provided, the result is 'NeedsSbe' so the orchestrator can surface it. + + The remote work is isolated in a single Invoke-Command so it can be mocked + in unit tests. + + .PARAMETER Session + Open PSSession to the cluster node. + + .PARAMETER ImportRoot + The import folder path AS SEEN ON THE NODE (e.g. + C:\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\import). + + .PARAMETER MediaFileName + For a Solution package: the .zip file name within ImportRoot to expand. + For an SBE package: the staged SBE subfolder name within ImportRoot. + + .PARAMETER PackageType + 'Solution' or 'SBE'. + + .PARAMETER Version + The version expected to appear in Get-SolutionUpdate. + + .PARAMETER DiscoveryTimeoutMinutes + Max minutes to poll for discovery. Default 30. + + .PARAMETER PollSeconds + Seconds between discovery polls. Default 30. + + .OUTPUTS + [PSCustomObject] with: + ImportState 'Imported' | 'NeedsSbe' | 'Discovering' | 'Failed' + DiscoveredName the discovered update name (when found) + Version echoed version + Message human-readable detail + #> + [CmdletBinding(SupportsShouldProcess = $true)] + [OutputType([PSCustomObject])] + param( + # Untyped (ValidateNotNull) so the function is unit-testable by passing a + # session double and mocking Invoke-Command; callers pass a real PSSession. + [Parameter(Mandatory = $true)] + [ValidateNotNull()] + $Session, + + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$ImportRoot, + + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$MediaFileName, + + [Parameter(Mandatory = $true)] + [ValidateSet('Solution', 'SBE')] + [string]$PackageType, + + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$Version, + + [int]$DiscoveryTimeoutMinutes = 30, + [int]$PollSeconds = 30 + ) + + if (-not $PSCmdlet.ShouldProcess($Session.ComputerName, "Import solution update '$Version'")) { + return [PSCustomObject]@{ + ImportState = 'Discovering' + DiscoveredName = '' + Version = $Version + Message = 'WhatIf - import not performed.' + } + } + + $remote = Invoke-Command -Session $Session -ScriptBlock { + param($ImportRoot, $MediaFileName, $PackageType, $Version, $DiscoveryTimeoutMinutes, $PollSeconds) + + $result = @{ + ImportState = 'Failed' + DiscoveredName = '' + Version = $Version + Message = '' + } + + try { + if ($PackageType -eq 'Solution') { + $zip = Join-Path -Path $ImportRoot -ChildPath $MediaFileName + if (-not (Test-Path -LiteralPath $zip -PathType Leaf)) { + $result.Message = "Solution bundle '$zip' not found on node." + return $result + } + $solutionFolder = Join-Path -Path $ImportRoot -ChildPath 'Solution' + if (-not (Test-Path -LiteralPath $solutionFolder)) { + New-Item -ItemType Directory -Path $solutionFolder -Force | Out-Null + } + Expand-Archive -LiteralPath $zip -DestinationPath $solutionFolder -Force + } + else { + $solutionFolder = Join-Path -Path $ImportRoot -ChildPath $MediaFileName + if (-not (Test-Path -LiteralPath $solutionFolder)) { + $result.Message = "SBE source folder '$solutionFolder' not found on node." + return $result + } + } + + if (-not (Get-Command Add-SolutionUpdate -ErrorAction SilentlyContinue)) { + $result.Message = 'Add-SolutionUpdate is not available on the node (Azure Local update cmdlets missing).' + return $result + } + + Add-SolutionUpdate -SolutionUpdateContentFolderPath $solutionFolder -ErrorAction Stop + + $deadline = (Get-Date).AddMinutes($DiscoveryTimeoutMinutes) + do { + Start-Sleep -Seconds $PollSeconds + $updates = @(Get-SolutionUpdate -ErrorAction SilentlyContinue) + $match = $updates | Where-Object { + ($_.Version -eq $Version) -or ($_.DisplayName -like "*$Version*") -or ($_.Name -like "*$Version*") + } | Select-Object -First 1 + + if ($null -ne $match) { + $state = [string]$match.State + if ($state -match 'AdditionalContentRequired') { + $result.ImportState = 'NeedsSbe' + $result.DiscoveredName = [string]$match.Name + $result.Message = "Discovered '$($match.Name)' but it reports AdditionalContentRequired (OEM SBE needed)." + return $result + } + $result.ImportState = 'Imported' + $result.DiscoveredName = [string]$match.Name + $result.Message = "Discovered '$($match.Name)' (State=$state)." + return $result + } + } while ((Get-Date) -lt $deadline) + + $result.ImportState = 'Discovering' + $result.Message = "Discovery still in progress after $DiscoveryTimeoutMinutes minute(s); will re-check on next run." + return $result + } + catch { + $result.ImportState = 'Failed' + $result.Message = "Import failed on node: $($_.Exception.Message)" + return $result + } + } -ArgumentList $ImportRoot, $MediaFileName, $PackageType, $Version, $DiscoveryTimeoutMinutes, $PollSeconds + + return [PSCustomObject]@{ + ImportState = [string]$remote.ImportState + DiscoveredName = [string]$remote.DiscoveredName + Version = [string]$remote.Version + Message = [string]$remote.Message + } +} diff --git a/AzLocal.UpdateManagement/Private/New-AzLocalPSRemotingSession.ps1 b/AzLocal.UpdateManagement/Private/New-AzLocalPSRemotingSession.ps1 new file mode 100644 index 00000000..deb7ca79 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/New-AzLocalPSRemotingSession.ps1 @@ -0,0 +1,86 @@ +function New-AzLocalPSRemotingSession { + <# + .SYNOPSIS + Opens a PowerShell remoting (WinRM) PSSession to an Azure Local cluster + node using the AD credential resolved from Key Vault. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. + + Transport defaults to WinRM over HTTPS (port 5986) with Kerberos/Negotiate + authentication - the recommended posture for cross-forest service accounts. + HTTP (5985) is supported as a documented fallback by passing -UseSsl:$false. + CredSSP is intentionally NOT used: there is no second-hop (robocopy runs as + the runner/agent over UNC, and Add-SolutionUpdate executes locally on the + target node), so basic Kerberos/Negotiate is sufficient. + + The caller is responsible for Remove-PSSession in a finally block. + + .PARAMETER ComputerName + The remoting target (cluster name, override FQDN, or name+suffix). + + .PARAMETER Credential + The [pscredential] built by Resolve-AzLocalSideloadCredential. + + .PARAMETER UseSsl + Use WinRM HTTPS (5986). Default $true. Set $false for the HTTP (5985) + fallback. + + .PARAMETER Port + Explicit WinRM port. When 0 (default), 5986 is used for SSL and 5985 for + non-SSL. + + .PARAMETER Authentication + WinRM authentication mechanism. Default 'Negotiate'. From the auth-map + AuthMechanism column when supplied. + + .PARAMETER SkipCertificateCheck + When using SSL, skip CA/CN/revocation checks on the node's WinRM HTTPS + listener certificate (lab/self-signed only - not recommended for prod). + + .OUTPUTS + [System.Management.Automation.Runspaces.PSSession] + #> + [CmdletBinding()] + [OutputType([System.Management.Automation.Runspaces.PSSession])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$ComputerName, + + [Parameter(Mandatory = $true)] + [ValidateNotNull()] + [System.Management.Automation.PSCredential]$Credential, + + [bool]$UseSsl = $true, + + [int]$Port = 0, + + [ValidateSet('Default', 'Negotiate', 'Kerberos', 'Basic', 'CredSSP')] + [string]$Authentication = 'Negotiate', + + [switch]$SkipCertificateCheck + ) + + $effectivePort = if ($Port -gt 0) { $Port } elseif ($UseSsl) { 5986 } else { 5985 } + + $newSessionParams = @{ + ComputerName = $ComputerName + Credential = $Credential + Port = $effectivePort + Authentication = $Authentication + ErrorAction = 'Stop' + } + if ($UseSsl) { $newSessionParams['UseSSL'] = $true } + + if ($SkipCertificateCheck) { + $newSessionParams['SessionOption'] = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck + } + + try { + return New-PSSession @newSessionParams + } + catch { + throw "Failed to open WinRM session to '$ComputerName' (port $effectivePort, SSL=$UseSsl, auth=$Authentication): $($_.Exception.Message)" + } +} diff --git a/AzLocal.UpdateManagement/Private/Read-AzLocalApplyUpdatesYamlCrons.ps1 b/AzLocal.UpdateManagement/Private/Read-AzLocalApplyUpdatesYamlCrons.ps1 index c14e906e..66418f81 100644 --- a/AzLocal.UpdateManagement/Private/Read-AzLocalApplyUpdatesYamlCrons.ps1 +++ b/AzLocal.UpdateManagement/Private/Read-AzLocalApplyUpdatesYamlCrons.ps1 @@ -9,11 +9,15 @@ function Read-AzLocalApplyUpdatesYamlCrons { Discovery rules: - If Path is a file, scan that file. - - If Path is a directory, recursively find files matching any of - 'Step.6_apply-updates*.yml', 'Step.6_apply-updates*.yaml', - 'apply-updates*.yml', or 'apply-updates*.yaml'. - (The 'Step.5_' prefix is the v0.7.68+ shipped name; the un-prefixed - form is the legacy name still supported for backwards compatibility.) + - If Path is a directory, recursively scan every *.yml / *.yaml and + keep only the apply-updates pipeline, identified by its stable + '# AZLOCAL-PIPELINE-ID: apply-updates' header comment (v0.8.7+). + Files that lack the ID comment (pre-v0.8.7 copies) fall back to a + filename match: 'apply-updates*.yml/.yaml', optionally carrying a + legacy 'Step.N_' prefix - but the de-numbered sibling + 'apply-updates-schedule-audit.yml' (the Step.3 audit pipeline, + which has its own poll crons) is explicitly EXCLUDED so its crons + are never mistaken for apply-updates windows. Platform is inferred from the parent directory name when the YAML is under .../github-actions/ or .../azure-devops/. Falls back to the @@ -57,24 +61,30 @@ function Read-AzLocalApplyUpdatesYamlCrons { $item = Get-Item -LiteralPath $Path $files = if ($item.PSIsContainer) { - # NOTE: Get-ChildItem -LiteralPath -Recurse -Include silently ignores the - # -Include filter and returns every recursed file (confirmed in PS 5.1). - # That caused v0.7.68 to pick up every Step.N_*.yml sibling (Step.1, Step.3, - # Step.4, Step.7, Step.8, Step.9 all carry their own schedule crons) and treat - # their crons as apply-updates crons - garbage in the audit, and on PS 7 the - # binder surfaced it as 'Cannot bind argument to parameter Expression because - # it is an empty string' once any unparseable capture was reached. - # Use -Filter (which is honoured under -Recurse) one pattern at a time, - # then dedupe by FullName. - $patterns = @( - 'Step.6_apply-updates*.yml', - 'Step.6_apply-updates*.yaml', - 'apply-updates*.yml', - 'apply-updates*.yaml' - ) - $hits = @() - foreach ($pattern in $patterns) { - $hits += @(Get-ChildItem -Path $item.FullName -Recurse -File -Filter $pattern -ErrorAction SilentlyContinue) + # v0.8.7: identify the apply-updates pipeline by its stable + # AZLOCAL-PIPELINE-ID header comment rather than a filename glob. The + # previous 'apply-updates*.yml' wildcard now collides with the + # de-numbered Step.3 audit pipeline ('apply-updates-schedule-audit.yml') + # and would wrongly ingest its poll crons. Recurse all YAML, then keep + # only ID == 'apply-updates'; for pre-ID copies fall back to a filename + # match that explicitly excludes the schedule-audit sibling. + $allYaml = @(Get-ChildItem -Path $item.FullName -Recurse -File -ErrorAction SilentlyContinue | + Where-Object { $_.Extension -in @('.yml', '.yaml') }) + $hits = New-Object System.Collections.Generic.List[System.IO.FileInfo] + foreach ($yf in $allYaml) { + $ymlText = $null + try { $ymlText = [System.IO.File]::ReadAllText($yf.FullName, [System.Text.UTF8Encoding]::new($false)) } catch { $ymlText = $null } + $ymlId = if ($ymlText) { Get-AzLocalPipelineId -Text $ymlText } else { $null } + if ($ymlId) { + if ($ymlId -eq 'apply-updates') { $hits.Add($yf) | Out-Null } + # ID present but not apply-updates -> deliberately skipped. + } + elseif ($yf.Name -match '(?i)^(Step\.\d+_)?apply-updates.*\.(yml|yaml)$' -and + $yf.Name -notmatch '(?i)apply-updates-schedule-audit') { + # Pre-v0.8.7 copy with no ID comment: name-based fallback, + # excluding the schedule-audit pipeline. + $hits.Add($yf) | Out-Null + } } @($hits | Sort-Object FullName -Unique) } diff --git a/AzLocal.UpdateManagement/Private/Register-AzLocalSideloadCopyTask.ps1 b/AzLocal.UpdateManagement/Private/Register-AzLocalSideloadCopyTask.ps1 new file mode 100644 index 00000000..a155259a --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Register-AzLocalSideloadCopyTask.ps1 @@ -0,0 +1,159 @@ +function Register-AzLocalSideloadCopyTask { + <# + .SYNOPSIS + Registers and starts a Windows Scheduled Task that runs the detached + sideload copy worker (Tools/Invoke-AzLocalSideloadCopyTask.ps1). + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. + + The Scheduled Task is the mechanism that lets a multi-hour robocopy + OUTLIVE the short-lived CI/CD job (and even a host reboot). The task runs + as the runner/agent service account (or a supplied principal) which must + have UNC read rights to the shared cache and write rights to the target + cluster import share. + + Network (UNC) access requires a logon type that carries network + credentials: a group Managed Service Account (gMSA) or a stored password. + The default S4U logon does NOT carry network credentials; supply + -PrincipalUserId with -LogonType Password (and -Password) or a gMSA via + -LogonType ServiceAccount for production UNC copies. + + .PARAMETER TaskName + Name to register the task under. + + .PARAMETER ClusterName + Cluster name (passed to the worker). + + .PARAMETER Version + Solution-update version (passed to the worker). + + .PARAMETER SourcePath + Verified media path (.zip or staged SBE folder). + + .PARAMETER TargetPath + Destination UNC import-share folder. + + .PARAMETER StateRoot + Shared UNC state root. + + .PARAMETER RobocopySwitches + Extra robocopy switches passed through to the worker. + + .PARAMETER HeartbeatSeconds + Heartbeat interval passed to the worker. + + .PARAMETER PrincipalUserId + Optional account to run the task as (e.g. 'CONTOSO\svc-azl' or a gMSA + 'CONTOSO\gmsa-azl$'). Defaults to the current user. + + .PARAMETER LogonType + Scheduled-task logon type. Default S4U. Use Password (with -Password) or + ServiceAccount (gMSA) for network/UNC access. + + .PARAMETER Password + Secure password when -LogonType is Password. + + .OUTPUTS + [string] the registered task name. + #> + [CmdletBinding(SupportsShouldProcess = $true)] + [OutputType([string])] + param( + [Parameter(Mandatory = $true)][string]$TaskName, + [Parameter(Mandatory = $true)][string]$ClusterName, + [Parameter(Mandatory = $true)][string]$Version, + [Parameter(Mandatory = $true)][string]$SourcePath, + [Parameter(Mandatory = $true)][string]$TargetPath, + [Parameter(Mandatory = $true)][string]$StateRoot, + [string]$RobocopySwitches = '/R:5 /W:30', + [int]$HeartbeatSeconds = 30, + [string]$PrincipalUserId = ("{0}\{1}" -f $env:USERDOMAIN, $env:USERNAME), + [ValidateSet('S4U', 'Password', 'ServiceAccount', 'Interactive')] + [string]$LogonType = 'S4U', + [System.Security.SecureString]$Password + ) + + if (-not (Get-Command Register-ScheduledTask -ErrorAction SilentlyContinue)) { + throw "ScheduledTasks module is not available on this host. On-prem sideloading requires a Windows runner/agent with the ScheduledTasks module." + } + + # $PSScriptRoot here is the module's Private folder; the worker ships under Tools\. + $moduleRoot = Split-Path -Path $PSScriptRoot -Parent + $worker = Join-Path -Path $moduleRoot -ChildPath 'Tools\Invoke-AzLocalSideloadCopyTask.ps1' + if (-not (Test-Path -LiteralPath $worker -PathType Leaf)) { + throw "Sideload copy worker script not found at '$worker'." + } + + $argLine = @( + '-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Bypass', + '-File', ('"{0}"' -f $worker), + '-ClusterName', ('"{0}"' -f $ClusterName), + '-Version', ('"{0}"' -f $Version), + '-SourcePath', ('"{0}"' -f $SourcePath), + '-TargetPath', ('"{0}"' -f $TargetPath), + '-StateRoot', ('"{0}"' -f $StateRoot), + '-RobocopySwitches', ('"{0}"' -f $RobocopySwitches), + '-HeartbeatSeconds', $HeartbeatSeconds + ) -join ' ' + + $action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument $argLine + $settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -ExecutionTimeLimit ([TimeSpan]::Zero) + + switch ($LogonType) { + 'Password' { + if (-not $Password) { throw "LogonType 'Password' requires -Password." } + $plain = [System.Net.NetworkCredential]::new('', $Password).Password + $principal = $null # password principal passed to Register-ScheduledTask directly + } + 'ServiceAccount' { + $principal = New-ScheduledTaskPrincipal -UserId $PrincipalUserId -LogonType ServiceAccount -RunLevel Highest + } + 'Interactive' { + $principal = New-ScheduledTaskPrincipal -UserId $PrincipalUserId -LogonType Interactive -RunLevel Highest + } + default { + $principal = New-ScheduledTaskPrincipal -UserId $PrincipalUserId -LogonType S4U -RunLevel Highest + } + } + + if (-not $PSCmdlet.ShouldProcess($TaskName, 'Register and start sideload copy Scheduled Task')) { + return $TaskName + } + + # Replace any pre-existing task of the same name. + Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false -ErrorAction SilentlyContinue + + try { + if ($LogonType -eq 'Password') { + Register-ScheduledTask -TaskName $TaskName -Action $action -Settings $settings -User $PrincipalUserId -Password $plain -RunLevel Highest -Force | Out-Null + } + else { + Register-ScheduledTask -TaskName $TaskName -Action $action -Settings $settings -Principal $principal -Force | Out-Null + } + Start-ScheduledTask -TaskName $TaskName + } + finally { + if ($LogonType -eq 'Password') { $plain = $null; Remove-Variable -Name plain -ErrorAction SilentlyContinue } + } + + return $TaskName +} + +function Remove-AzLocalSideloadCopyTask { + <# + .SYNOPSIS + Unregisters a completed/failed sideload copy Scheduled Task. + .OUTPUTS + [void] + #> + [CmdletBinding(SupportsShouldProcess = $true)] + [OutputType([void])] + param( + [Parameter(Mandatory = $true)][string]$TaskName + ) + + if (-not (Get-Command Unregister-ScheduledTask -ErrorAction SilentlyContinue)) { return } + if (-not $PSCmdlet.ShouldProcess($TaskName, 'Unregister sideload copy Scheduled Task')) { return } + Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false -ErrorAction SilentlyContinue +} diff --git a/AzLocal.UpdateManagement/Private/Resolve-AzLocalSideloadCredential.ps1 b/AzLocal.UpdateManagement/Private/Resolve-AzLocalSideloadCredential.ps1 new file mode 100644 index 00000000..580da15b --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Resolve-AzLocalSideloadCredential.ps1 @@ -0,0 +1,94 @@ +function Resolve-AzLocalSideloadCredential { + <# + .SYNOPSIS + Builds the Active Directory [pscredential] used to PowerShell-remote into + an Azure Local cluster, from the two Key Vault secrets named in the + sideload auth-map row for the cluster's UpdateAuthAccountId. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. Given an + auth-map row (from Get-AzLocalSideloadAuthMap), reads the username and + password secrets from the row's Key Vault and returns a [pscredential]. + + Key Vault authentication is assumed to be already established by the + pipeline (azure/login OIDC, managed identity, or service principal) per + the SIDELOAD_KV_AUTH variable - this helper only calls + Get-AzKeyVaultSecret. The actual KV read is isolated in the thin wrapper + Get-AzLocalKeyVaultSecretText so it can be mocked in unit tests. + + The username may be a UPN ('svc@contoso.com') or DOMAIN\user form; both + are accepted by [pscredential]. The plaintext password is converted to a + SecureString and the plaintext copy is cleared from memory before return. + + .PARAMETER AuthRow + A single auth-map row [PSCustomObject] with KeyVaultName, + UsernameSecretName, PasswordSecretName. + + .OUTPUTS + [System.Management.Automation.PSCredential] + #> + [CmdletBinding()] + [OutputType([System.Management.Automation.PSCredential])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNull()] + [PSCustomObject]$AuthRow + ) + + foreach ($field in @('KeyVaultName', 'UsernameSecretName', 'PasswordSecretName')) { + if ([string]::IsNullOrWhiteSpace([string]$AuthRow.$field)) { + throw "Resolve-AzLocalSideloadCredential: auth-map row (UpdateAuthAccountId '$($AuthRow.UpdateAuthAccountId)') has an empty $field." + } + } + + $username = $null + $password = $null + try { + $username = Get-AzLocalKeyVaultSecretText -VaultName $AuthRow.KeyVaultName -SecretName $AuthRow.UsernameSecretName + $password = Get-AzLocalKeyVaultSecretText -VaultName $AuthRow.KeyVaultName -SecretName $AuthRow.PasswordSecretName + + if ([string]::IsNullOrWhiteSpace($username)) { + throw "Username secret '$($AuthRow.UsernameSecretName)' in vault '$($AuthRow.KeyVaultName)' resolved to an empty value." + } + if ([string]::IsNullOrEmpty($password)) { + throw "Password secret '$($AuthRow.PasswordSecretName)' in vault '$($AuthRow.KeyVaultName)' resolved to an empty value." + } + + $securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force + return New-Object System.Management.Automation.PSCredential ($username.Trim(), $securePassword) + } + finally { + # Clear the plaintext secret copies from memory as soon as possible. + if ($null -ne $password) { $password = $null } + Remove-Variable -Name password -ErrorAction SilentlyContinue + } +} + +function Get-AzLocalKeyVaultSecretText { + <# + .SYNOPSIS + Thin wrapper around Get-AzKeyVaultSecret -AsPlainText so the parent + function can be unit-tested by mocking just this call. + .OUTPUTS + [string] the plaintext secret value. + #> + [CmdletBinding()] + [OutputType([string])] + param( + [Parameter(Mandatory = $true)][string]$VaultName, + [Parameter(Mandatory = $true)][string]$SecretName + ) + + if (-not (Get-Command Get-AzKeyVaultSecret -ErrorAction SilentlyContinue)) { + throw "Az.KeyVault module is not loaded. Install with: Install-Module Az.KeyVault -Scope CurrentUser" + } + + try { + $value = Get-AzKeyVaultSecret -VaultName $VaultName -Name $SecretName -AsPlainText -ErrorAction Stop + } + catch { + throw "Failed to read Key Vault secret '$SecretName' from vault '$VaultName': $($_.Exception.Message)" + } + + return $value +} diff --git a/AzLocal.UpdateManagement/Private/Resolve-AzLocalSideloadTargetPath.ps1 b/AzLocal.UpdateManagement/Private/Resolve-AzLocalSideloadTargetPath.ps1 new file mode 100644 index 00000000..851b41c0 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Resolve-AzLocalSideloadTargetPath.ps1 @@ -0,0 +1,50 @@ +function Resolve-AzLocalSideloadTargetPath { + <# + .SYNOPSIS + Resolves the UNC path of an Azure Local cluster's infrastructure 'import' + share, where the CombinedSolutionBundle (and any staged OEM SBE content) + is copied prior to running Add-SolutionUpdate on a cluster node. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. + + Precedence: + 1. An explicit ImportSharePath supplied on the auth-map row (operator + override; may be a full UNC such as + \\cluster\ClusterStorage$\Infrastructure_1\Shares\SU1_Infrastructure_1\import). + 2. The conventional Azure Local layout derived from the remoting host: + \\\C$\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\import + + The on-prem documented mechanism copies the bundle into the import share, + then expands it to an 'import\Solution' subfolder for Add-SolutionUpdate. + This helper returns the 'import' root; the caller chooses the Solution / + SBE subfolder. + + .PARAMETER RemotingHost + The host name / FQDN used to reach the cluster (cluster name, override + FQDN, or name + suffix). Used to build the default UNC. + + .PARAMETER ImportSharePath + Optional operator override (auth-map ImportSharePath column). + + .OUTPUTS + [string] the UNC path of the import share root. + #> + [CmdletBinding()] + [OutputType([string])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$RemotingHost, + + [string]$ImportSharePath + ) + + if (-not [string]::IsNullOrWhiteSpace($ImportSharePath)) { + return $ImportSharePath.TrimEnd('\') + } + + # Default conventional Azure Local infrastructure volume layout. + $relative = 'C$\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\import' + return ('\\{0}\{1}' -f $RemotingHost, $relative) +} diff --git a/AzLocal.UpdateManagement/Private/Select-AzLocalNextUpdateForCluster.ps1 b/AzLocal.UpdateManagement/Private/Select-AzLocalNextUpdateForCluster.ps1 new file mode 100644 index 00000000..56905002 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Select-AzLocalNextUpdateForCluster.ps1 @@ -0,0 +1,153 @@ +function Select-AzLocalNextUpdateForCluster { + <# + .SYNOPSIS + Selects the next Azure Local solution update to install for a cluster, + applying the optional AllowedUpdateVersions allow-list and the + latest-by-YYMM auto-pick rule. + + .DESCRIPTION + Pure decision helper extracted from Start-AzLocalClusterUpdate (v0.8.7) + so the apply path AND the on-prem sideloading automation agree on which + update is "next" for a given cluster. No side effects beyond the + Get-LatestUpdateByYYMM warning - callers own all logging, CSV, and + result-object emission. + + Selection rules (identical to the historic Start-AzLocalClusterUpdate + behaviour): + - If -UpdateName is supplied, that explicit choice wins. The named + update must be present in -ReadyUpdates (matched on 'name'), + otherwise Reason = 'UpdateNotFound'. + - Otherwise, when -AllowedUpdateVersions is supplied and is NOT the + 'Latest' no-constraint sentinel, the Ready pool is filtered to + updates whose 'name' OR 'properties.version' is an EXACT + (case-insensitive) match for a supplied entry. An empty result = + Reason 'NotInAllowList' (strict no-op, never falls back to latest). + - 'Latest' (case-insensitive) appearing ALONE means "no constraint"; + the filter is skipped. + - The winner is the latest Ready update by YYMM (Get-LatestUpdateByYYMM). + - An empty -ReadyUpdates pool yields Reason 'NoneReady'. + + .PARAMETER ReadyUpdates + The updates already filtered to a Ready state (Ready / ReadyToInstall). + + .PARAMETER AllowedUpdateVersions + Optional allow-list of update names / version strings. 'Latest' (alone) + disables filtering. Empty / whitespace entries are ignored. + + .PARAMETER UpdateName + Optional explicit update name that overrides the allow-list. + + .OUTPUTS + [PSCustomObject] with: + Reason 'Selected' | 'NotInAllowList' | 'UpdateNotFound' | 'NoneReady' + SelectedUpdate the chosen update object, or $null + FilteredUpdates the Ready updates passing the allow-list (array) + AllowOnlyLatest [bool] the allow-list was the 'Latest' sentinel + AllowListEffective the cleaned non-empty allow-list entries (array) + AllowDisplay comma-joined allow-list for messaging + ReadyDisplay 'name (vVersion)' list of Ready updates for messaging + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + [Parameter(Mandatory = $true)] + [AllowNull()] + [AllowEmptyCollection()] + [array]$ReadyUpdates, + + [Parameter(Mandatory = $false)] + [AllowNull()] + [AllowEmptyCollection()] + [string[]]$AllowedUpdateVersions, + + [Parameter(Mandatory = $false)] + [string]$UpdateName + ) + + $readyArr = @($ReadyUpdates | Where-Object { $null -ne $_ }) + $readyDisplay = (@($readyArr | ForEach-Object { "$($_.name) (v$($_.properties.version))" }) -join '; ') + + $allowListEffective = @($AllowedUpdateVersions | Where-Object { + -not [string]::IsNullOrWhiteSpace([string]$_) + }) + $allowDisplay = (@($allowListEffective) -join ', ') + $allowOnlyLatest = ($allowListEffective.Count -gt 0) -and ( + -not (@($allowListEffective | Where-Object { + -not [string]::Equals([string]$_, 'Latest', [System.StringComparison]::OrdinalIgnoreCase) + }).Count -gt 0) + ) + + if ($readyArr.Count -eq 0) { + return [PSCustomObject]@{ + Reason = 'NoneReady' + SelectedUpdate = $null + FilteredUpdates = @() + AllowOnlyLatest = $allowOnlyLatest + AllowListEffective = $allowListEffective + AllowDisplay = $allowDisplay + ReadyDisplay = $readyDisplay + } + } + + # Explicit -UpdateName wins over the allow-list. + if ($UpdateName) { + $named = @($readyArr | Where-Object { $_.name -eq $UpdateName }) + if ($named.Count -eq 0) { + return [PSCustomObject]@{ + Reason = 'UpdateNotFound' + SelectedUpdate = $null + FilteredUpdates = $readyArr + AllowOnlyLatest = $allowOnlyLatest + AllowListEffective = $allowListEffective + AllowDisplay = $allowDisplay + ReadyDisplay = $readyDisplay + } + } + return [PSCustomObject]@{ + Reason = 'Selected' + SelectedUpdate = $named[0] + FilteredUpdates = $named + AllowOnlyLatest = $allowOnlyLatest + AllowListEffective = $allowListEffective + AllowDisplay = $allowDisplay + ReadyDisplay = $readyDisplay + } + } + + # Allow-list filter (skipped when only 'Latest' or no list supplied). + $pool = $readyArr + if ($allowListEffective.Count -gt 0 -and -not $allowOnlyLatest) { + $allowSet = New-Object System.Collections.Generic.HashSet[string] ([System.StringComparer]::OrdinalIgnoreCase) + foreach ($a in $allowListEffective) { + $sa = [string]$a + if (-not [string]::IsNullOrWhiteSpace($sa)) { [void]$allowSet.Add($sa.Trim()) } + } + $filtered = @($readyArr | Where-Object { + ($_.name -and $allowSet.Contains([string]$_.name)) -or + ($_.properties -and $_.properties.version -and $allowSet.Contains([string]$_.properties.version)) + }) + if ($filtered.Count -eq 0) { + return [PSCustomObject]@{ + Reason = 'NotInAllowList' + SelectedUpdate = $null + FilteredUpdates = @() + AllowOnlyLatest = $allowOnlyLatest + AllowListEffective = $allowListEffective + AllowDisplay = $allowDisplay + ReadyDisplay = $readyDisplay + } + } + $pool = $filtered + } + + $selected = Get-LatestUpdateByYYMM -Updates $pool + return [PSCustomObject]@{ + Reason = 'Selected' + SelectedUpdate = $selected + FilteredUpdates = $pool + AllowOnlyLatest = $allowOnlyLatest + AllowListEffective = $allowListEffective + AllowDisplay = $allowDisplay + ReadyDisplay = $readyDisplay + } +} diff --git a/AzLocal.UpdateManagement/Private/Test-AzLocalRemoteFileHash.ps1 b/AzLocal.UpdateManagement/Private/Test-AzLocalRemoteFileHash.ps1 new file mode 100644 index 00000000..89c26c96 --- /dev/null +++ b/AzLocal.UpdateManagement/Private/Test-AzLocalRemoteFileHash.ps1 @@ -0,0 +1,62 @@ +function Test-AzLocalRemoteFileHash { + <# + .SYNOPSIS + Verifies the SHA256 of a file on a remote Azure Local cluster node against + the catalog-published hash. + + .DESCRIPTION + Private helper for the v0.8.7 on-prem sideloading automation. After the + detached robocopy lands the bundle on the cluster's import share, this + confirms integrity on the node itself (over WinRM) before Add-SolutionUpdate + is run, guarding against a partial/corrupt copy. + + .PARAMETER Session + An open PSSession to the cluster node (from New-AzLocalPSRemotingSession). + + .PARAMETER RemotePath + The path to the file AS SEEN ON THE NODE (local path on the node, not the + UNC the runner/agent used). + + .PARAMETER ExpectedSha256 + The catalog SHA256 to compare against (case-insensitive). + + .OUTPUTS + [PSCustomObject] with: Match [bool], ExpectedSha256, ActualSha256, RemotePath. + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + # Untyped (ValidateNotNull) so the function is unit-testable by passing a + # session double and mocking Invoke-Command; callers pass a real PSSession. + [Parameter(Mandatory = $true)] + [ValidateNotNull()] + $Session, + + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$RemotePath, + + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$ExpectedSha256 + ) + + $actual = Invoke-Command -Session $Session -ScriptBlock { + param($Path) + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + return $null + } + (Get-FileHash -LiteralPath $Path -Algorithm SHA256).Hash + } -ArgumentList $RemotePath + + $actualText = if ($null -ne $actual) { ([string]$actual).ToUpperInvariant() } else { '' } + $match = (-not [string]::IsNullOrEmpty($actualText)) -and + [string]::Equals($actualText, $ExpectedSha256.ToUpperInvariant(), [System.StringComparison]::OrdinalIgnoreCase) + + return [PSCustomObject]@{ + Match = $match + ExpectedSha256 = $ExpectedSha256.ToUpperInvariant() + ActualSha256 = $actualText + RemotePath = $RemotePath + } +} diff --git a/AzLocal.UpdateManagement/Public/Copy-AzLocalItsmSample.ps1 b/AzLocal.UpdateManagement/Public/Copy-AzLocalItsmSample.ps1 index 25605954..1e99d691 100644 --- a/AzLocal.UpdateManagement/Public/Copy-AzLocalItsmSample.ps1 +++ b/AzLocal.UpdateManagement/Public/Copy-AzLocalItsmSample.ps1 @@ -36,7 +36,7 @@ function Copy-AzLocalItsmSample { committed to a repository alongside the workflow / pipeline YAMLs copied by `Copy-AzLocalPipelineExample`. The default `-Destination` is `.\.itsm` - the relative path that both - `Step.6_apply-updates.yml` workflows default `itsm_config_path` / + `apply-updates.yml` workflows default `itsm_config_path` / `itsmConfigPath` to (resolved relative to the repo root at job runtime). diff --git a/AzLocal.UpdateManagement/Public/Copy-AzLocalPipelineExample.ps1 b/AzLocal.UpdateManagement/Public/Copy-AzLocalPipelineExample.ps1 index eca2021e..04dad689 100644 --- a/AzLocal.UpdateManagement/Public/Copy-AzLocalPipelineExample.ps1 +++ b/AzLocal.UpdateManagement/Public/Copy-AzLocalPipelineExample.ps1 @@ -70,15 +70,20 @@ function Copy-AzLocalPipelineExample { .PARAMETER SkipStarterSchedule Only meaningful with `-Platform GitHub` or `-Platform AzureDevOps`. - By default (v0.7.92+) the function ALSO drops a starter - `apply-updates-schedule.yml` one level UP from `-Destination` (i.e. - the parent of `.github\workflows\` for GitHub, or the parent of - the pipelines folder you chose for ADO) when no file already - exists at that path. The starter is a verbatim copy of the bundled + By default (v0.7.92+; relocated to `config\` in v0.8.7) the + function ALSO drops a starter `apply-updates-schedule.yml` into a + `config\` folder at the REPO ROOT - the same folder as the + documented `config/ClusterUpdateRings.csv` that Step.2 consumes, so + all operator-authored config sits together and the path is + IDENTICAL on GitHub and Azure DevOps. The repo root is resolved from + `-Destination`: two levels up for the canonical GitHub layout + (`.github\workflows`), one level up otherwise. The starter is only + written when no file already exists at `config\apply-updates-schedule.yml`. + The starter is a verbatim copy of the bundled `apply-updates-schedule.example.yml` and is safe to land alongside - a freshly copied Step.6 pipeline (the bundled Step.6 ships with - every `cron:` line COMMENTED OUT, so the schedule file cannot - cause Step.6 to fire on a `schedule:` trigger until the operator + a freshly copied `apply-updates` pipeline (the bundled pipeline ships + with every `cron:` line COMMENTED OUT, so the schedule file cannot + cause `apply-updates` to fire on a `schedule:` trigger until the operator explicitly adds at least one cron entry). Two safety rails apply: @@ -101,6 +106,29 @@ function Copy-AzLocalPipelineExample { Has no effect when `-Platform All` is in use (the starter is only relevant for actively-installed CI YAMLs). + .PARAMETER SkipStarterSideloadConfig + Only meaningful with `-Platform GitHub` or `-Platform AzureDevOps`. + + By default (v0.8.7+) the function ALSO drops two starter on-prem + sideloading config files into the SAME `config\` folder as the + starter schedule and the ring CSV (one path, both platforms): + - `sideload-auth-map.csv` - header row + guidance comments only. + - `sideload-catalog.yml` - `schemaVersion` + an empty `packages:` + list. + + Both starters are HEADER / SKELETON ONLY (no demo data rows), so they + are inert even if `SIDELOAD_UPDATES` is flipped to `true` before they + are populated: an empty auth-map produces an `UnknownAuthAccountId` + status and an empty catalog produces `NoCatalogEntry` - neither + performs a Key Vault lookup or stages any media. The Step.6 sideload + pipeline is itself hard-gated OFF unless `SIDELOAD_UPDATES == 'true'`, + so the files have zero effect for operators who never opt in. + + As with the starter schedule, existing files are NEVER overwritten. + + Pass `-SkipStarterSideloadConfig` to suppress the drop entirely. Has + no effect when `-Platform All` is in use. + .PARAMETER Update Allow overwriting destination files that already exist. Without this switch the function aborts with a list of conflicting files. With @@ -188,6 +216,21 @@ function Copy-AzLocalPipelineExample { points at the live-fleet regenerator `New-AzLocalApplyUpdatesScheduleConfig -OutputPath -Force` for the operator's next move. + Changed in : v0.8.7 - for `-Platform GitHub|AzureDevOps` ALL starter + operator config now lands in a `config\` folder at the + repo root (one path on both platforms, matching the + documented `config/ClusterUpdateRings.csv`). The starter + `apply-updates-schedule.yml` MOVED here from its v0.7.92 + location (parent of `.github\workflows\` / pipelines + folder). The function now ALSO drops two starter on-prem + sideloading config files (`sideload-auth-map.csv` + + `sideload-catalog.yml`, header/skeleton only) into the + same `config\` folder. Existing files are NEVER + overwritten. Pass `-SkipStarterSchedule` / + `-SkipStarterSideloadConfig` to suppress. The sideload + starters are inert until populated AND + `SIDELOAD_UPDATES=true`, so they are safe for operators + who never sideload. #> [CmdletBinding(SupportsShouldProcess = $true, ConfirmImpact = 'Low')] [OutputType([System.IO.DirectoryInfo])] @@ -211,7 +254,13 @@ function Copy-AzLocalPipelineExample { # Default OFF, i.e. the starter IS copied unless one already # exists at the destination (in which case the existing file is # always preserved, regardless of this switch). - [switch]$SkipStarterSchedule + [switch]$SkipStarterSchedule, + + # v0.8.7: when set, suppress the default starter sideload-config + # drop (sideload-auth-map.csv + sideload-catalog.yml) for + # Platform=GitHub|AzureDevOps. Default OFF. Existing files are + # always preserved regardless of this switch. + [switch]$SkipStarterSideloadConfig ) # ------------------------------------------------------------------ @@ -402,28 +451,48 @@ function Copy-AzLocalPipelineExample { Write-Verbose "Copied $copiedCount file(s) from '$sourceRoot' to '$targetRoot' (skipped: $skippedCount)." # ------------------------------------------------------------------ - # 6 (v0.7.92). Starter apply-updates-schedule.yml drop. + # 6 (v0.7.92, relocated to config\ in v0.8.7). Starter + # apply-updates-schedule.yml drop. # Default-on for -Platform GitHub|AzureDevOps; suppressed by - # -SkipStarterSchedule. The starter lands ONE LEVEL UP from the - # pipelines folder (sibling of .github\workflows\ for GitHub, or - # sibling of the ADO pipelines folder) so the schedule file is - # not co-mingled with CI YAMLs. NEVER overwrites an existing - # file - the existing file is always preserved and reported in - # the Next-steps summary. Safe to land alongside Step.6 because - # Step.6 ships with every `cron:` line commented out (verified - # in github-actions/Step.6_apply-updates.yml). + # -SkipStarterSchedule. The starter lands in a `config\` folder at + # the REPO ROOT (the same location as the documented + # `config/ClusterUpdateRings.csv` that Step.2 consumes), so ALL + # operator-authored config (ring CSV, schedule, sideload auth-map + + # catalog) sits together in one folder and the path is IDENTICAL on + # GitHub and Azure DevOps. NEVER overwrites an existing file - the + # existing file is always preserved and reported in the Next-steps + # summary. Safe to land alongside Step.6 because Step.6 ships with + # every `cron:` line commented out. + # + # Repo-root resolution: for the canonical GitHub layout + # (`-Destination .github\workflows`) the repo root is TWO levels up + # (so config\ lands next to .github\, not inside it). For Azure + # DevOps (and any other layout) it is ONE level up from the + # pipelines folder. In both cases config\ ends up at the repo root. # ------------------------------------------------------------------ + $configDir = $null + if ($Platform -in @('GitHub', 'AzureDevOps')) { + $trimmedTarget = $targetRoot.TrimEnd('\', '/') + $oneLevelUp = Split-Path -Parent $trimmedTarget + if ($Platform -eq 'GitHub' -and ($trimmedTarget -match '[\\/]\.github[\\/]workflows$')) { + # .github\workflows -> step up past .github to the repo root. + $repoRoot = Split-Path -Parent $oneLevelUp + } + else { + $repoRoot = $oneLevelUp + } + if ([string]::IsNullOrWhiteSpace($repoRoot)) { + # Defensive: drive root or no parent - fall back to the target. + $repoRoot = $trimmedTarget + } + $configDir = Join-Path -Path $repoRoot -ChildPath 'config' + } + $scheduleSrc = Join-Path -Path $sourceRoot -ChildPath 'apply-updates-schedule.example.yml' $scheduleDest = $null $scheduleAction = $null # 'Copied' | 'Preserved' | 'Skipped' | 'Missing' if ($Platform -in @('GitHub', 'AzureDevOps') -and -not $SkipStarterSchedule.IsPresent) { - $scheduleParent = Split-Path -Parent ($targetRoot.TrimEnd('\','/')) - if ([string]::IsNullOrWhiteSpace($scheduleParent)) { - # Defensive: if $targetRoot has no parent (e.g. a drive root), - # fall back to dropping the schedule alongside the YAMLs. - $scheduleParent = $targetRoot - } - $scheduleDest = Join-Path -Path $scheduleParent -ChildPath 'apply-updates-schedule.yml' + $scheduleDest = Join-Path -Path $configDir -ChildPath 'apply-updates-schedule.yml' if (-not (Test-Path -LiteralPath $scheduleSrc -PathType Leaf)) { # Source missing - should never happen in a healthy install but @@ -437,8 +506,7 @@ function Copy-AzLocalPipelineExample { Write-Verbose ("Copy-AzLocalPipelineExample: starter schedule preserved (already exists at '{0}'); not copied." -f $scheduleDest) } elseif ($PSCmdlet.ShouldProcess($scheduleDest, "Copy starter apply-updates-schedule.yml from '$scheduleSrc'")) { - # Create parent on demand (e.g. .github\ may not exist when the - # caller created .github\workflows\ via a single -Force New-Item). + # Create the config\ folder on demand. $newScheduleParent = Split-Path -Parent $scheduleDest if (-not (Test-Path -LiteralPath $newScheduleParent)) { $null = New-Item -ItemType Directory -Path $newScheduleParent -Force -ErrorAction Stop @@ -454,14 +522,84 @@ function Copy-AzLocalPipelineExample { elseif ($Platform -in @('GitHub', 'AzureDevOps')) { # -SkipStarterSchedule was explicitly set. $scheduleAction = 'SkippedBySwitch' - $scheduleParent = Split-Path -Parent ($targetRoot.TrimEnd('\','/')) - if (-not [string]::IsNullOrWhiteSpace($scheduleParent)) { - $scheduleDest = Join-Path -Path $scheduleParent -ChildPath 'apply-updates-schedule.yml' + if ($configDir) { + $scheduleDest = Join-Path -Path $configDir -ChildPath 'apply-updates-schedule.yml' + } + } + + # ------------------------------------------------------------------ + # 6b (v0.8.7). Starter on-prem sideloading config drop + # (sideload-auth-map.csv + sideload-catalog.yml). Default-on for + # -Platform GitHub|AzureDevOps; suppressed by + # -SkipStarterSideloadConfig. Lands in the SAME `config\` folder as + # the starter schedule and the ring CSV, so all operator config files + # sit together at the repo root. Both files are HEADER / SKELETON + # ONLY (no demo data rows), so they are inert even if SIDELOAD_UPDATES + # is flipped on before they are populated: an empty auth-map yields + # UnknownAuthAccountId and an empty catalog yields NoCatalogEntry - + # neither performs a Key Vault lookup nor stages media. The Step.6 + # sideload pipeline is itself hard-gated OFF unless + # SIDELOAD_UPDATES == 'true'. NEVER overwrites an existing file. + # ------------------------------------------------------------------ + $sideloadAuthMapDest = $null + $sideloadCatalogDest = $null + $sideloadConfigAction = $null # 'Copied' | 'Preserved' | 'Skipped' | 'SkippedBySwitch' + if ($Platform -in @('GitHub', 'AzureDevOps')) { + $sideloadAuthMapDest = Join-Path -Path $configDir -ChildPath 'sideload-auth-map.csv' + $sideloadCatalogDest = Join-Path -Path $configDir -ChildPath 'sideload-catalog.yml' + + if ($SkipStarterSideloadConfig.IsPresent) { + $sideloadConfigAction = 'SkippedBySwitch' + } + else { + # Header / skeleton starter content (NOT a copy of the worked + # .example files, which carry demo rows). ASCII-only so + # Set-Content -Encoding ASCII writes no BOM. + $authMapStarter = @( + '# Sideload auth-map starter (v0.8.7 on-prem sideloading). Worked example:' + '# Automation-Pipeline-Examples/sideload-auth-map.example.csv' + "# Lines starting with '#' are stripped at runtime by Get-AzLocalSideloadAuthMap;" + '# the first non-comment line is the CSV header. UpdateAuthAccountId must be' + '# numeric (1-3 digits) and unique. The first four columns are required.' + 'UpdateAuthAccountId,KeyVaultName,UsernameSecretName,PasswordSecretName,RemotingTargetFqdn,FqdnSuffix,AuthMechanism,ImportSharePath' + ) + $catalogStarter = @( + '# Sideload catalog starter (v0.8.7 on-prem sideloading). Worked example:' + '# Automation-Pipeline-Examples/sideload-catalog.example.yml' + '# Run Update-AzLocalSideloadCatalog to populate Solution rows from the' + '# Microsoft Learn offline-updates table; add OEM SBE rows manually.' + 'schemaVersion: 1' + 'packages:' + ) + + $sideloadWroteAny = $false + $sideloadPreservedAny = $false + foreach ($drop in @( + @{ Dest = $sideloadAuthMapDest; Content = $authMapStarter; Label = 'sideload-auth-map.csv' } + @{ Dest = $sideloadCatalogDest; Content = $catalogStarter; Label = 'sideload-catalog.yml' } + )) { + if (Test-Path -LiteralPath $drop.Dest -PathType Leaf) { + $sideloadPreservedAny = $true + Write-Verbose ("Copy-AzLocalPipelineExample: starter {0} preserved (already exists at '{1}')." -f $drop.Label, $drop.Dest) + } + elseif ($PSCmdlet.ShouldProcess($drop.Dest, ("Write starter {0}" -f $drop.Label))) { + $dropParent = Split-Path -Parent $drop.Dest + if (-not (Test-Path -LiteralPath $dropParent)) { + $null = New-Item -ItemType Directory -Path $dropParent -Force -ErrorAction Stop + } + Set-Content -LiteralPath $drop.Dest -Value $drop.Content -Encoding ASCII -ErrorAction Stop + $sideloadWroteAny = $true + } + } + if ($sideloadWroteAny) { $sideloadConfigAction = 'Copied' } + elseif ($sideloadPreservedAny) { $sideloadConfigAction = 'Preserved' } + else { $sideloadConfigAction = 'Skipped' } } } # ------------------------------------------------------------------ # 7. Friendly "what now" summary so the user does not have to open + # the README first to know what they just copied. Uses Write-Host # (intentional - this is operator-facing UI text, not pipeline # output; per Microsoft guidance Write-Host is appropriate for @@ -482,6 +620,13 @@ function Copy-AzLocalPipelineExample { elseif ($scheduleAction -eq 'Preserved') { Write-Host (" Starter schedule preserved (existing file): {0}" -f $scheduleDest) -ForegroundColor Yellow } + if ($sideloadConfigAction -eq 'Copied') { + Write-Host (" Starter sideload config dropped at: {0}" -f $sideloadAuthMapDest) -ForegroundColor Green + Write-Host (" {0}" -f $sideloadCatalogDest) -ForegroundColor Green + } + elseif ($sideloadConfigAction -eq 'Preserved') { + Write-Host " Starter sideload config preserved (existing sideload-auth-map.csv / sideload-catalog.yml left untouched)" -ForegroundColor Yellow + } Write-Host "" Write-Host "Next steps:" -ForegroundColor Cyan # Helper for step 4/5/6 messaging: the schedule sub-message branches @@ -523,6 +668,16 @@ function Copy-AzLocalPipelineExample { } } } + # Optional sideload (Step.6) sub-message - only emitted when a starter + # sideload config was dropped or already exists. Sideloading is OFF by + # default (SIDELOAD_UPDATES gate), so this is purely informational. + $sideloadHintLines = @() + if ($sideloadConfigAction -in @('Copied', 'Preserved')) { + $sideloadHintLines += " *. OPTIONAL on-prem sideloading (Step.6, OFF by default): two starter config files are in place:" + if ($sideloadAuthMapDest) { $sideloadHintLines += (" {0}" -f $sideloadAuthMapDest) } + if ($sideloadCatalogDest) { $sideloadHintLines += (" {0}" -f $sideloadCatalogDest) } + $sideloadHintLines += " Populate both, set repository variable SIDELOAD_UPDATES=true, then dry-run the sideload-updates pipeline. Until then they are inert. See Automation-Pipeline-Examples/docs/sideload.md." + } switch ($Platform) { 'GitHub' { # Detect the canonical .github\workflows\ destination so we can @@ -542,6 +697,7 @@ function Copy-AzLocalPipelineExample { foreach ($line in $scheduleHintLines) { Write-Host $line } Write-Host " See section 5.1 step 5 + section 8 of the README for the full schema, multi-stage rollouts, and the allowedUpdateVersions allow-list." Write-Host " 5. Optional: enable the ITSM connector by setting 'raise_itsm_ticket=true' (setup in ITSM/README.md)." + foreach ($line in $sideloadHintLines) { Write-Host $line } } 'AzureDevOps' { Write-Host (" 1. Commit the YAML files from '{0}' to your Azure Repo." -f $targetRoot) @@ -550,8 +706,9 @@ function Copy-AzLocalPipelineExample { Write-Host " 4. Each pipeline references service connection 'AzureLocal-ServiceConnection' - either name yours to match or edit 'azureSubscription:' in each YAML." Write-Host " 5. SCHEDULED Step.6 (apply-updates) requires apply-updates-schedule.yml:" -ForegroundColor Yellow foreach ($line in $scheduleHintLines) { Write-Host $line } - Write-Host " Step.6 reads APPLY_UPDATES_SCHEDULE_PATH (default './apply-updates-schedule.yml' at repo root). Override the variable in the pipeline if you keep the schedule elsewhere. See section 5.2 step 6 + section 8 of the README." + Write-Host " Step.7 reads APPLY_UPDATES_SCHEDULE_PATH (default './config/apply-updates-schedule.yml'). Override the variable in the pipeline if you keep the schedule elsewhere. See section 5.2 step 6 + section 8 of the README." Write-Host " 6. Optional: enable the ITSM connector by setting 'raise_itsm_ticket=true' (setup in ITSM/README.md)." + foreach ($line in $sideloadHintLines) { Write-Host $line } } default { $readmePath = Join-Path -Path $targetRoot -ChildPath 'README.md' diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 index ab2446fb..634c5148 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalApplyUpdatesScheduleAudit.ps1 @@ -73,7 +73,7 @@ function Export-AzLocalApplyUpdatesScheduleAudit { `$env:BUILD_ARTIFACTSTAGINGDIRECTORY` (Azure DevOps). .PARAMETER PipelineYamlPath - Path (file or folder) to Step.6_apply-updates.yml. REQUIRED so + Path (file or folder) to Step.7_apply-updates.yml. REQUIRED so the Recommend view can diff its proposed crons against what is already in Step.6 and only emit a snippet for the truly missing entries. @@ -124,6 +124,18 @@ function Export-AzLocalApplyUpdatesScheduleAudit { `Add-AzLocalPipelineStepSummary` on Azure DevOps and Local hosts. Default 'schedule-coverage-summary.md'. + .PARAMETER SideloadEnabled + Opt-in switch for the v0.8.7 "Recommended sideload schedule" section. + When not explicitly bound, it is resolved from the SIDELOAD_UPDATES + environment variable (true/1/yes/on => enabled). When disabled the + section is omitted entirely and the audit is byte-identical to v0.8.6. + + .PARAMETER SideloadLeadDays + How many days before a ring's apply window the media should already be + sideloaded (drives the recommended sideload kickoff cron = apply firing + shifted back this many days). When not explicitly bound it is resolved + from the SIDELOAD_LEAD_DAYS environment variable, defaulting to 7. + .PARAMETER InstalledModuleVersion Optional [string] used in the markdown footer ('Generated by AzLocal.UpdateManagement v'). @@ -206,6 +218,13 @@ function Export-AzLocalApplyUpdatesScheduleAudit { [ValidateNotNullOrEmpty()] [string]$SummaryFileName = 'schedule-coverage-summary.md', + [Parameter(Mandatory = $false)] + [bool]$SideloadEnabled, + + [Parameter(Mandatory = $false)] + [ValidateRange(0, 365)] + [int]$SideloadLeadDays = 7, + [Parameter(Mandatory = $false)] [AllowEmptyString()] [AllowNull()] @@ -233,7 +252,7 @@ function Export-AzLocalApplyUpdatesScheduleAudit { } if (-not (Test-Path -LiteralPath $PipelineYamlPath)) { - throw "PipelineYamlPath '$PipelineYamlPath' does not exist. Set it to the folder containing Step.6_apply-updates.yml (e.g. '.github/workflows' or '.azure-pipelines') so the Recommend view can diff its proposed crons against the existing file." + throw "PipelineYamlPath '$PipelineYamlPath' does not exist. Set it to the folder containing Step.7_apply-updates.yml (e.g. '.github/workflows' or '.azure-pipelines') so the Recommend view can diff its proposed crons against the existing file." } $haveSchedule = $false @@ -548,7 +567,7 @@ function Export-AzLocalApplyUpdatesScheduleAudit { & $addDetailTable '### Audit Detail - Cron coverage (Uncovered / Partial / Malformed first)' $cronRows 'No tagged clusters found - nothing to audit.' if (-not $hasIssues -and $recoContent) { - [void]$md.Add('### Reference - Recommended schedule (copy into Step.6_apply-updates.yml)') + [void]$md.Add('### Reference - Recommended schedule (copy into Step.7_apply-updates.yml)') [void]$md.Add('') [void]$md.Add($recoContent) [void]$md.Add('') @@ -559,7 +578,7 @@ function Export-AzLocalApplyUpdatesScheduleAudit { # calendar silently disappeared from clean-fleet runs. # v0.8.6 enrichment: when PipelineYamlPath (always) and ClusterCsvPath # (optional) are available, build two extra columns: - # * "Ring CRON Start Time (Step 6 pipeline)" - per-day UTC firing + # * "Ring CRON Start Time (apply-updates pipeline)" - per-day UTC firing # times projected from Step.6 cron triggers. # * "Tag Start Window Match (>=95%)" - per (ring, date) pair: do # >=95% of clusters in the ring have an UpdateStartWindow tag @@ -714,6 +733,106 @@ function Export-AzLocalApplyUpdatesScheduleAudit { } } + # ---- Recommended sideload schedule (Step.6, opt-in v0.8.7) ------------ + # Gated on SIDELOAD_UPDATES (parameter override -> env var). When disabled + # the section is omitted and the audit output is byte-identical to v0.8.6. + # The on-prem Step.6 sideload pipeline is re-entrant and driven by a + # frequent poll cron; its planner uses SIDELOAD_LEAD_DAYS to decide when a + # cluster is "due". This section reuses the apply-window crons already read + # from PipelineYamlPath to recommend, per apply firing, the EARLIEST weekly + # cron at which staging should begin (apply firing shifted back LeadDays). + # + # NOTE: the local accumulator is deliberately NOT named $sideloadEnabled - + # PowerShell variable names are CASE-INSENSITIVE, so $sideloadEnabled would + # alias the [bool]$SideloadEnabled parameter and clobber the bound value to + # $false on the first assignment (silent: -SideloadEnabled $true then + # rendered nothing while only the env-var path worked). + $emitSideloadSection = $false + if ($PSBoundParameters.ContainsKey('SideloadEnabled')) { + $emitSideloadSection = $SideloadEnabled + } + elseif ($env:SIDELOAD_UPDATES) { + $emitSideloadSection = (([string]$env:SIDELOAD_UPDATES).Trim().ToLowerInvariant() -in @('true', '1', 'yes', 'on')) + } + $sideloadScheduleIncluded = $false + if ($emitSideloadSection) { + $leadDays = $SideloadLeadDays + if (-not $PSBoundParameters.ContainsKey('SideloadLeadDays') -and $env:SIDELOAD_LEAD_DAYS) { + $parsedLead = 0 + if ([int]::TryParse((([string]$env:SIDELOAD_LEAD_DAYS).Trim()), [ref]$parsedLead) -and $parsedLead -ge 0 -and $parsedLead -le 365) { + $leadDays = $parsedLead + } + } + + # Distinct apply-window weekly firings (DayOfWeek + TimeOfDay) from the + # apply pipeline crons. Read-AzLocalApplyUpdatesYamlCrons uses a + # unary-comma return - direct assignment only (NEVER @() wrap). + $applyFirings = New-Object 'System.Collections.Generic.List[psobject]' + try { + $sideloadCronTriggers = Read-AzLocalApplyUpdatesYamlCrons -Path $PipelineYamlPath -ErrorAction Stop + foreach ($sct in $sideloadCronTriggers) { + try { + $sparsed = ConvertFrom-AzLocalCronExpression -Expression $sct.CronExpression -ErrorAction Stop + if ($sparsed.IsValid -and -not $sparsed.IsComplex) { + foreach ($sft in @($sparsed.FireTimes)) { + $applyFirings.Add([pscustomobject]@{ + DayOfWeek = [int]$sft.DayOfWeek + TimeOfDay = $sft.TimeOfDay + }) | Out-Null + } + } + } + catch { + Write-Verbose "Sideload section: cron parse failed for '$($sct.CronExpression)': $($_.Exception.Message)" + } + } + } + catch { + Write-Warning "Failed to read apply-updates crons for the sideload schedule recommendation: $($_.Exception.Message)" + } + + [void]$md.Add('### Recommended sideload schedule (Step.6 - opt-in)') + [void]$md.Add('') + [void]$md.Add("`SIDELOAD_UPDATES` is enabled, so media must be pre-staged on each cluster **$leadDays day(s)** before its apply window opens. The on-prem **Step.6 - Sideload Update** pipeline is re-entrant: drive it on a frequent poll cron and its planner uses `SIDELOAD_LEAD_DAYS=$leadDays` to decide when each cluster is due.") + [void]$md.Add('') + [void]$md.Add('Recommended Step.6 poll cron (every 30 minutes) - paste into the `schedule:`/`schedules:` block of `sideload-updates.yml`:') + [void]$md.Add('') + [void]$md.Add('```') + [void]$md.Add('*/30 * * * *') + [void]$md.Add('```') + [void]$md.Add('') + + # Build a distinct, ordered list of apply firings. + $distinctFirings = @( + $applyFirings | + Sort-Object DayOfWeek, TimeOfDay -Unique + ) + if ($distinctFirings.Count -gt 0) { + $dayNames = @('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat') + [void]$md.Add("Per apply-window firing, the EARLIEST weekly cron at which staging should begin (apply firing shifted back $leadDays day(s)):") + [void]$md.Add('') + [void]$md.Add('| Apply firing (UTC) | Lead (days) | Begin-staging (UTC) | Sideload kickoff cron |') + [void]$md.Add('|--------------------|------------:|---------------------|-----------------------|') + foreach ($f in $distinctFirings) { + $applyDow = [int]$f.DayOfWeek + $hhmm = ([timespan]$f.TimeOfDay).ToString('hh\:mm') + $minute = ([timespan]$f.TimeOfDay).Minutes + $hour = ([timespan]$f.TimeOfDay).Hours + $shiftedDow = ((($applyDow - ($leadDays % 7)) % 7) + 7) % 7 + $kickoffCron = ('{0} {1} * * {2}' -f $minute, $hour, $shiftedDow) + [void]$md.Add(('| {0} {1} | {2} | {3} {1} | `{4}` |' -f $dayNames[$applyDow], $hhmm, $leadDays, $dayNames[$shiftedDow], $kickoffCron)) + } + [void]$md.Add('') + [void]$md.Add('> The kickoff cron is the EARLIEST point staging could start for that window. The re-entrant Step.6 pipeline (frequent poll) advances the multi-hour copy from that point; you do NOT need a separate cron per ring.') + [void]$md.Add('') + } + else { + [void]$md.Add('> No simple weekly apply-window firings were found in the pipeline YAML, so a per-window kickoff cron could not be derived. Drive Step.6 on the recommended poll cron above; its planner will stage each cluster `SIDELOAD_LEAD_DAYS` days before its apply window.') + [void]$md.Add('') + } + $sideloadScheduleIncluded = $true + } + [void]$md.Add('### Reports Available') [void]$md.Add("- ``$AuditCsvFileName`` - one row per (UpdateRing, UpdateStartWindow) pair") [void]$md.Add("- ``$MatrixCsvFileName`` - full inventory + required cron per row") diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalClusterReadinessGateReport.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalClusterReadinessGateReport.ps1 index 6fd83722..51869ac6 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalClusterReadinessGateReport.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalClusterReadinessGateReport.ps1 @@ -7,7 +7,7 @@ function Export-AzLocalClusterReadinessGateReport { per-cluster readiness markdown table to the run summary. .DESCRIPTION v0.8.5 Step.6 thin-YAML helper. Replaces the ~80-line inline `run:` - block that lived in both Step.6_apply-updates.yml pipelines. + block that lived in both Step.7_apply-updates.yml pipelines. Behaviour matches the prior inline block byte-for-byte: - When -UpdateRing is empty/whitespace (no schedule row matched diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalClusterUpdateReadinessReport.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalClusterUpdateReadinessReport.ps1 index be916c1c..648bf74c 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalClusterUpdateReadinessReport.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalClusterUpdateReadinessReport.ps1 @@ -355,10 +355,10 @@ function Export-AzLocalClusterUpdateReadinessReport { # 2. Action banner if ($notReady -gt 0 -or $clustersWithCritical -gt 0) { - [void]$md.Add("> **Action required**: $notReady cluster(s) not ready and/or $clustersWithCritical cluster(s) with Critical health failures. Review the **Not-Ready** and **Critical-health** sections below first; the CSV artifacts in ``azlocal-step.5-readiness-assessment-report_*`` carry the full per-finding detail. Remediate (hardware vendor SBE / firmware / cluster health) before or alongside the next apply-updates run. **The healthy clusters are safe to proceed** - Step.6_apply-updates.yml is per-cluster scoped.") + [void]$md.Add("> **Action required**: $notReady cluster(s) not ready and/or $clustersWithCritical cluster(s) with Critical health failures. Review the **Not-Ready** and **Critical-health** sections below first; the CSV artifacts in ``azlocal-step.5-readiness-assessment-report_*`` carry the full per-finding detail. Remediate (hardware vendor SBE / firmware / cluster health) before or alongside the next apply-updates run. **The healthy clusters are safe to proceed** - Step.7_apply-updates.yml is per-cluster scoped.") } else { - [void]$md.Add('> **All clear**: every cluster in scope is ready for update. Safe to proceed with Step.6_apply-updates.yml for this ring.') + [void]$md.Add('> **All clear**: every cluster in scope is ready for update. Safe to proceed with Step.7_apply-updates.yml for this ring.') } [void]$md.Add('') @@ -396,7 +396,7 @@ function Export-AzLocalClusterUpdateReadinessReport { if ($criticalRows.Count -gt 0) { [void]$md.Add('### Critical-health clusters') [void]$md.Add('') - [void]$md.Add('_Cross-link: see **Step.4_fleet-connectivity-status** for connectivity-class failures and **Step.9_fleet-health-status** for the broader Critical/Warning catalog._') + [void]$md.Add('_Cross-link: see **Step.4_fleet-connectivity-status** for connectivity-class failures and **Step.10_fleet-health-status** for the broader Critical/Warning catalog._') [void]$md.Add('') [void]$md.Add('| Cluster | UpdateRing | Health state | Critical | Warning |') [void]$md.Add('|---------|------------|--------------|----------|---------|') @@ -448,9 +448,9 @@ function Export-AzLocalClusterUpdateReadinessReport { [void]$md.Add('### Cross-link to other pipelines') [void]$md.Add('') [void]$md.Add('- **Step.4_fleet-connectivity-status** - root-cause Disconnected / Offline / partial-connectivity findings on the Not-Ready and Critical-health rows above.') - [void]$md.Add('- **Step.6_apply-updates** - apply updates to the Ready clusters in this ring (manual workflow_dispatch, or wait for the scheduled cron firing).') - [void]$md.Add('- **Step.7_monitor-updates** - tail in-flight runs once Step.6 has started (auto-trigger on Step.6 completion, or manual).') - [void]$md.Add('- **Step.9_fleet-health-status** - broader Critical / Warning health catalog across the whole fleet (not just blocking-only).') + [void]$md.Add('- **Step.7_apply-updates** - apply updates to the Ready clusters in this ring (manual workflow_dispatch, or wait for the scheduled cron firing).') + [void]$md.Add('- **Step.8_monitor-updates** - tail in-flight runs once Step.7 has started (auto-trigger on Step.7 completion, or manual).') + [void]$md.Add('- **Step.10_fleet-health-status** - broader Critical / Warning health catalog across the whole fleet (not just blocking-only).') [void]$md.Add('') [void]$md.Add('_Note: the **Update Readiness Assessment** entry in the Checks tab is the merged combined view; the [JUnit Debug] entries are diagnostic mirrors for CI/test tooling._') diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalFleetHealthStatusReport.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalFleetHealthStatusReport.ps1 index c80915ea..f3e6a4b6 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalFleetHealthStatusReport.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalFleetHealthStatusReport.ps1 @@ -7,7 +7,7 @@ function Export-AzLocalFleetHealthStatusReport { .DESCRIPTION Public entry-point for the v0.8.5 thin-YAML refactor of the Step.9 fleet-health-status pipeline. Before v0.8.5 the GitHub Actions and - Azure DevOps Step.9_fleet-health-status.yml files each carried + Azure DevOps Step.10_fleet-health-status.yml files each carried ~600 lines of inline PowerShell (failure collection + summary roll-up + overview collection + JUnit XML construction + collapsible markdown rendering + step-output emission). This @@ -155,7 +155,7 @@ function Export-AzLocalFleetHealthStatusReport { # snapshot object for downstream PowerShell use. .EXAMPLE - # Used by Step.9_fleet-health-status.yml (GitHub Actions + Azure DevOps): + # Used by Step.10_fleet-health-status.yml (GitHub Actions + Azure DevOps): Export-AzLocalFleetHealthStatusReport ` -Scope $env:INPUT_SCOPE ` -UpdateRing $env:INPUT_UPDATE_RING ` diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalFleetUpdateStatusReport.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalFleetUpdateStatusReport.ps1 index a8e7612a..c3b61b05 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalFleetUpdateStatusReport.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalFleetUpdateStatusReport.ps1 @@ -148,7 +148,7 @@ function Export-AzLocalFleetUpdateStatusReport { # snapshot object for downstream PowerShell use. .EXAMPLE - # Used by Step.8_fleet-update-status.yml (GitHub Actions + Azure DevOps): + # Used by Step.9_fleet-update-status.yml (GitHub Actions + Azure DevOps): Export-AzLocalFleetUpdateStatusReport ` -Scope $env:INPUT_SCOPE ` -UpdateRing $env:INPUT_UPDATE_RING ` diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalSideloadStatusReport.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalSideloadStatusReport.ps1 new file mode 100644 index 00000000..1d0f33cc --- /dev/null +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalSideloadStatusReport.ps1 @@ -0,0 +1,139 @@ +function Export-AzLocalSideloadStatusReport { + <# + .SYNOPSIS + Builds a sideload status report (markdown + JUnit XML) from the shared + state records and optional plan error rows. + + .DESCRIPTION + Public reporting helper for the v0.8.7 on-prem sideloading automation. Reads + every per-cluster state JSON under StateRoot\state, summarises progress by + state (Queued / Copying / Copied / Verified / Imported / Failed / Stale) + with throughput + ETA, and appends any plan error rows (NotInAllowList, + UnknownAuthAccountId, NoCatalogEntry). Optionally writes the markdown and a + JUnit XML (one test case per cluster; Failed/error rows fail) to disk. + + .PARAMETER StateRoot + Shared UNC root containing state\. + + .PARAMETER Plan + Optional plan rows from Resolve-AzLocalSideloadPlan, used to surface + configuration/selection error rows in the report. + + .PARAMETER OutputPath + Optional directory to write sideload-status.md and sideload-junit.xml. + + .OUTPUTS + [PSCustomObject] with Markdown, JUnitXml, Counts, HasFailures. + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()][string]$StateRoot, + [Parameter(Mandatory = $false)][PSCustomObject[]]$Plan, + [Parameter(Mandatory = $false)][string]$OutputPath + ) + + $stateDir = Join-Path -Path $StateRoot -ChildPath 'state' + $states = @() + if (Test-Path -LiteralPath $stateDir) { + $states = @( + Get-ChildItem -LiteralPath $stateDir -Filter '*.json' -File -ErrorAction SilentlyContinue | ForEach-Object { + try { Get-Content -LiteralPath $_.FullName -Raw | ConvertFrom-Json } catch { $null } + } | Where-Object { $null -ne $_ } + ) + } + + $lines = New-Object System.Collections.Generic.List[string] + $lines.Add('## Sideload status') + $lines.Add('') + + if ($states.Count -gt 0) { + $lines.Add('| Cluster | Version | State | Progress | Mbps | ETA (UTC) | Owner | Retries | Message |') + $lines.Add('| --- | --- | --- | --- | --- | --- | --- | --- | --- |') + foreach ($s in ($states | Sort-Object ClusterName)) { + $pct = if ([long]$s.TotalBytes -gt 0) { ('{0}%' -f [math]::Round(([double]$s.CopiedBytes / [double]$s.TotalBytes) * 100, 1)) } else { '-' } + $rowText = ('| {0} | {1} | {2} | {3} | {4} | {5} | {6} | {7} | {8} |' -f ` + $s.ClusterName, $s.Version, $s.State, $pct, $s.Mbps, $s.EtaUtc, $s.OwningMachine, $s.Retries, $s.Message) + $lines.Add($rowText) + } + $lines.Add('') + } + else { + $lines.Add('_No sideload state records found._') + $lines.Add('') + } + + $errorRows = @() + if ($Plan) { + $errorRows = @($Plan | Where-Object { @('NotInAllowList', 'UnknownAuthAccountId', 'NoCatalogEntry', 'NoneReady') -contains [string]$_.Status }) + } + if ($errorRows.Count -gt 0) { + $lines.Add('### Plan warnings / errors') + $lines.Add('') + $lines.Add('| Cluster | Status | Message |') + $lines.Add('| --- | --- | --- |') + foreach ($e in $errorRows) { + $lines.Add(('| {0} | {1} | {2} |' -f $e.ClusterName, $e.Status, $e.Message)) + } + $lines.Add('') + } + + $markdown = ($lines -join [Environment]::NewLine) + + # JUnit: one case per cluster state; Failed (and plan errors) fail. + $cases = New-Object System.Collections.Generic.List[object] + foreach ($s in $states) { + if ([string]$s.State -eq 'Failed') { + $cases.Add(@{ Name = ('{0} ({1})' -f $s.ClusterName, $s.Version); Failure = @{ Message = [string]$s.Message; Type = 'SideloadFailed' } }) + } + else { + $cases.Add(@{ Name = ('{0} ({1}) [{2}]' -f $s.ClusterName, $s.Version, $s.State) }) + } + } + foreach ($e in $errorRows) { + $cases.Add(@{ Name = ('{0} [{1}]' -f $e.ClusterName, $e.Status); Failure = @{ Message = [string]$e.Message; Type = [string]$e.Status } }) + } + + $suites = @(@{ Name = 'Sideload'; TestCases = $cases.ToArray() }) + $junit = New-AzLocalPipelineJUnitXml -TestSuitesName 'AzLocalSideload' -Suites $suites + + $counts = $states | Group-Object State | ForEach-Object { [PSCustomObject]@{ State = $_.Name; Count = $_.Count } } + $hasFailures = (@($states | Where-Object { [string]$_.State -eq 'Failed' }).Count -gt 0) -or ($errorRows.Count -gt 0) + + if (-not [string]::IsNullOrWhiteSpace($OutputPath)) { + if (-not (Test-Path -LiteralPath $OutputPath)) { New-Item -ItemType Directory -Path $OutputPath -Force | Out-Null } + Write-Utf8NoBomFile -Path (Join-Path $OutputPath 'sideload-status.md') -Content $markdown + Write-Utf8NoBomFile -Path (Join-Path $OutputPath 'sideload-junit.xml') -Content $junit + } + + return [PSCustomObject]@{ + Markdown = $markdown + JUnitXml = $junit + Counts = @($counts) + HasFailures = $hasFailures + } +} + +function Add-AzLocalSideloadStepSummary { + <# + .SYNOPSIS + Renders the sideload status report into the CI/CD step summary. + .DESCRIPTION + Thin wrapper that calls Export-AzLocalSideloadStatusReport and appends the + markdown to the pipeline step summary (GitHub/ADO/Local handled internally + by Add-AzLocalPipelineStepSummary). + .OUTPUTS + [PSCustomObject] the report object from Export-AzLocalSideloadStatusReport. + #> + [CmdletBinding()] + [OutputType([PSCustomObject])] + param( + [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()][string]$StateRoot, + [Parameter(Mandatory = $false)][PSCustomObject[]]$Plan, + [Parameter(Mandatory = $false)][string]$OutputPath + ) + + $report = Export-AzLocalSideloadStatusReport -StateRoot $StateRoot -Plan $Plan -OutputPath $OutputPath + Add-AzLocalPipelineStepSummary -Markdown $report.Markdown | Out-Null + return $report +} diff --git a/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 b/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 index b4b2b8a6..b65dfd16 100644 --- a/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 +++ b/AzLocal.UpdateManagement/Public/Export-AzLocalUpdateRunMonitorReport.ps1 @@ -9,7 +9,7 @@ function Export-AzLocalUpdateRunMonitorReport { .DESCRIPTION Phase 1 (v0.8.5) of the thin-YAML refactor. Condenses the inline - `run: |` body of the v0.8.4 Step.7_monitor-updates.yml (GitHub + `run: |` body of the v0.8.4 Step.8_monitor-updates.yml (GitHub Actions + Azure DevOps) into a single cmdlet call so the per-platform yml shrinks to a few lines and the workload becomes unit-testable against a synthetic Get-AzLocalUpdateRuns result. diff --git a/AzLocal.UpdateManagement/Public/Get-AzLocalApplyUpdatesScheduleCycleCalendar.ps1 b/AzLocal.UpdateManagement/Public/Get-AzLocalApplyUpdatesScheduleCycleCalendar.ps1 index 4d2c2693..c16cd503 100644 --- a/AzLocal.UpdateManagement/Public/Get-AzLocalApplyUpdatesScheduleCycleCalendar.ps1 +++ b/AzLocal.UpdateManagement/Public/Get-AzLocalApplyUpdatesScheduleCycleCalendar.ps1 @@ -65,9 +65,9 @@ function Get-AzLocalApplyUpdatesScheduleCycleCalendar { .PARAMETER CronFiringsByDate Optional hashtable mapping 'yyyy-MM-dd' (UTC) date strings to a sorted, deduplicated [string[]] of 'HH:mm' UTC firing times that - the Step.6 apply-updates pipeline cron(s) will produce on that + the apply-updates pipeline cron(s) will produce on that date. When supplied AND -AsMarkdown is set, the per-day calendar - table gains a centered "Ring CRON Start Time (Step 6 pipeline)" + table gains a centered "Ring CRON Start Time (apply-updates pipeline)" column immediately after "Date (UTC)". The cell renders up to 2 firing times verbatim; any additional firings beyond the first 2 are summarised as "(+N)" (e.g. "02:00, 04:00 (+1)" when there @@ -340,7 +340,7 @@ function Get-AzLocalApplyUpdatesScheduleCycleCalendar { $alignCells = New-Object System.Collections.Generic.List[string] [void]$headerCells.Add('Date (UTC)'); [void]$alignCells.Add('---') if ($hasCronFirings) { - [void]$headerCells.Add('Ring CRON Start Time
(Step 6 pipeline)') + [void]$headerCells.Add('Ring CRON Start Time
(apply-updates pipeline)') [void]$alignCells.Add(':---:') } [void]$headerCells.Add('Day'); [void]$alignCells.Add('---') diff --git a/AzLocal.UpdateManagement/Public/Get-AzLocalClusterInventory.ps1 b/AzLocal.UpdateManagement/Public/Get-AzLocalClusterInventory.ps1 index 0b4d2ea5..93ff24c3 100644 --- a/AzLocal.UpdateManagement/Public/Get-AzLocalClusterInventory.ps1 +++ b/AzLocal.UpdateManagement/Public/Get-AzLocalClusterInventory.ps1 @@ -43,6 +43,14 @@ function Get-AzLocalClusterInventory { Format is auto-detected from file extension (.csv or .json). CSV is useful for editing in Excel; JSON for CI/CD and API integrations. + .PARAMETER IncludeSideloadColumns + Optional. When set, adds the numeric 'UpdateAuthAccountId' tag value as + an extra column (placed just before ResourceId) in both the returned + objects and the CSV/JSON export. Used by the opt-in on-prem sideload + workflow (Step.6) so Step.1 / Step.2 can round-trip the per-cluster + auth-map account id. When NOT set, the column is omitted entirely and + the output is byte-identical to prior versions. + .EXAMPLE # Get inventory of all clusters across all subscriptions Get-AzLocalClusterInventory @@ -112,7 +120,10 @@ function Get-AzLocalClusterInventory { [string]$ExportPath, [Parameter(Mandatory = $false)] - [switch]$PassThru + [switch]$PassThru, + + [Parameter(Mandatory = $false)] + [switch]$IncludeSideloadColumns ) # Pre-flight: Validate export path is writable before expensive operations @@ -303,20 +314,29 @@ function Get-AzLocalClusterInventory { $sideloadedTagValue = Get-TagValue -Tags $cluster.tags -Name $script:UpdateSideloadedTagName $versionInProgressTagValue = Get-TagValue -Tags $cluster.tags -Name $script:UpdateVersionInProgressTagName - $inventoryItem = [PSCustomObject]@{ + # Build the per-cluster inventory item via an ordered hashtable so the + # optional sideload UpdateAuthAccountId column can be inserted just + # before ResourceId without disturbing the column order/output of the + # default (sideload-off) path - which must remain byte-identical. + $itemProps = [ordered]@{ ClusterName = $cluster.name ResourceGroup = $cluster.resourceGroup SubscriptionId = $cluster.subscriptionId SubscriptionName = $subscriptionMap[$cluster.subscriptionId] UpdateRing = if ($ringTagValue) { $ringTagValue } else { "" } HasUpdateRingTag = if ($ringTagValue) { "Yes" } else { "No" } - UpdateStartWindow = if ($windowTagValue) { $windowTagValue } else { "" } + UpdateStartWindow = if ($windowTagValue) { $windowTagValue } else { "" } UpdateExclusionsWindow = if ($exclusionsTagValue) { $exclusionsTagValue } else { "" } UpdateExcluded = if ($excludedTagValue) { $excludedTagValue } else { "" } UpdateSideloaded = if ($sideloadedTagValue) { $sideloadedTagValue } else { "" } UpdateVersionInProgress = if ($versionInProgressTagValue) { $versionInProgressTagValue } else { "" } - ResourceId = $cluster.id } + if ($IncludeSideloadColumns) { + $authIdTagValue = Get-TagValue -Tags $cluster.tags -Name $script:UpdateAuthAccountIdTagName + $itemProps['UpdateAuthAccountId'] = if ($authIdTagValue) { $authIdTagValue } else { "" } + } + $itemProps['ResourceId'] = $cluster.id + $inventoryItem = [PSCustomObject]$itemProps $inventory += $inventoryItem } @@ -337,7 +357,10 @@ function Get-AzLocalClusterInventory { # Determine export format from file extension $extension = [System.IO.Path]::GetExtension($ExportPath).ToLower() - $exportData = $inventory | Select-Object ClusterName, ResourceGroup, SubscriptionId, SubscriptionName, UpdateRing, HasUpdateRingTag, UpdateStartWindow, UpdateExclusionsWindow, UpdateExcluded, UpdateSideloaded, UpdateVersionInProgress, ResourceId + $selectColumns = @('ClusterName', 'ResourceGroup', 'SubscriptionId', 'SubscriptionName', 'UpdateRing', 'HasUpdateRingTag', 'UpdateStartWindow', 'UpdateExclusionsWindow', 'UpdateExcluded', 'UpdateSideloaded', 'UpdateVersionInProgress') + if ($IncludeSideloadColumns) { $selectColumns += 'UpdateAuthAccountId' } + $selectColumns += 'ResourceId' + $exportData = $inventory | Select-Object $selectColumns switch ($extension) { '.json' { diff --git a/AzLocal.UpdateManagement/Public/Get-AzLocalFleetStatusData.ps1 b/AzLocal.UpdateManagement/Public/Get-AzLocalFleetStatusData.ps1 index e256b675..b3baf620 100644 --- a/AzLocal.UpdateManagement/Public/Get-AzLocalFleetStatusData.ps1 +++ b/AzLocal.UpdateManagement/Public/Get-AzLocalFleetStatusData.ps1 @@ -406,9 +406,14 @@ function Get-AzLocalFleetStatusData { if ($latestProps.progress -and $latestProps.progress.steps) { $steps = $latestProps.progress.steps $runProgress = "$(@($steps | Where-Object { $_.status -eq 'Success' }).Count)/$(@($steps).Count) steps" - $ipStep = $steps | Where-Object { $_.status -eq 'InProgress' } | Select-Object -First 1 - $fStep = $steps | Where-Object { $_.status -in @('Error','Failed') } | Select-Object -First 1 - if ($ipStep) { $currentStep = $ipStep.name } elseif ($fStep) { $currentStep = "$($fStep.name) (FAILED)" } + # Walk to the deepest InProgress/Error/Failed step rather than the + # coarse top-level wrapper (e.g. "Start update"), so CurrentStep stays + # consistent with CurrentStepDetail / the standard update-progress output. + $deepestActive = Get-DeepestActiveStep -Steps $steps + if ($deepestActive) { + if ($deepestActive.status -in @('Error','Failed')) { $currentStep = "$($deepestActive.name) (FAILED)" } + else { $currentStep = $deepestActive.name } + } $currentStepDetail = Get-CurrentStepPath -Steps $steps -IncludeErrorMessage if ([string]::IsNullOrWhiteSpace($currentStepDetail)) { $currentStepDetail = $currentStep } } diff --git a/AzLocal.UpdateManagement/Public/Get-AzLocalLatestSolutionVersion.ps1 b/AzLocal.UpdateManagement/Public/Get-AzLocalLatestSolutionVersion.ps1 index 71a871ef..233945c8 100644 --- a/AzLocal.UpdateManagement/Public/Get-AzLocalLatestSolutionVersion.ps1 +++ b/AzLocal.UpdateManagement/Public/Get-AzLocalLatestSolutionVersion.ps1 @@ -61,7 +61,7 @@ function Get-AzLocalLatestSolutionVersion { if ($manifest.SupportedYYMMs -contains $clusterYymm) { 'Supported' } else { 'Unsupported' } .EXAMPLE - # Used by Step.8_fleet-update-status pipeline (Step.7 prior to v0.7.90 renumber); + # Used by Step.9_fleet-update-status pipeline (Step.8 before the v0.8.7 sideload renumber; Step.7 before v0.7.90); # falls back to fleet-observed top-6 if the manifest is unreachable try { $m = Get-AzLocalLatestSolutionVersion -ErrorAction Stop diff --git a/AzLocal.UpdateManagement/Public/Invoke-AzLocalClusterInventory.ps1 b/AzLocal.UpdateManagement/Public/Invoke-AzLocalClusterInventory.ps1 index ecac9431..97cc1f0f 100644 --- a/AzLocal.UpdateManagement/Public/Invoke-AzLocalClusterInventory.ps1 +++ b/AzLocal.UpdateManagement/Public/Invoke-AzLocalClusterInventory.ps1 @@ -76,6 +76,15 @@ function Invoke-AzLocalClusterInventory { on Azure DevOps and Local hosts. Default `cluster-inventory-summary.md`. + .PARAMETER IncludeSideloadColumns + Opt-in switch for the on-prem sideload workflow (Step.6). When set, + the exported CSV/JSON gains an extra `UpdateAuthAccountId` column + (placed before `ResourceId`) carrying each cluster's numeric + sideload auth-map account id tag. When NOT explicitly passed, the + switch is resolved from the `SIDELOAD_UPDATES` repo variable + (true/1/yes/on, case-insensitive). When unset/false the artifacts + are byte-identical to prior versions. + .PARAMETER PassThru When set, returns a single PSCustomObject summarising the run (ClusterCount, WithTagCount, WithoutTagCount, CsvPath, JsonPath, @@ -139,12 +148,25 @@ function Invoke-AzLocalClusterInventory { [ValidateNotNullOrEmpty()] [string]$SummaryFileName = 'cluster-inventory-summary.md', + [Parameter(Mandatory = $false)] + [switch]$IncludeSideloadColumns, + [Parameter(Mandatory = $false)] [switch]$PassThru ) $pipelineHost = Get-AzLocalPipelineHost + # Resolve the opt-in sideload gate. When the caller did not explicitly pass + # -IncludeSideloadColumns, fall back to the SIDELOAD_UPDATES repo variable + # (the same bool gate the Step.6 sideload pipeline keys off). Accepts + # true/1/yes/on (case-insensitive). When unset/false the inventory output + # is byte-identical to prior versions (no UpdateAuthAccountId column). + if (-not $PSBoundParameters.ContainsKey('IncludeSideloadColumns')) { + $sideloadFlag = ([string]$env:SIDELOAD_UPDATES).Trim().ToLowerInvariant() + $IncludeSideloadColumns = $sideloadFlag -in @('true', '1', 'yes', 'on') + } + if (-not $OutputDirectory) { if ($pipelineHost -eq 'AzureDevOps' -and $env:BUILD_ARTIFACTSTAGINGDIRECTORY) { $OutputDirectory = $env:BUILD_ARTIFACTSTAGINGDIRECTORY @@ -173,7 +195,13 @@ function Invoke-AzLocalClusterInventory { } Write-Host '--- Running cluster inventory ---' - $inventory = Get-AzLocalClusterInventory @invParams -ExportPath $csvPath -PassThru + $invParams['ExportPath'] = $csvPath + $invParams['PassThru'] = $true + if ($IncludeSideloadColumns) { + $invParams['IncludeSideloadColumns'] = $true + Write-Host 'Sideload mode (SIDELOAD_UPDATES) enabled - including UpdateAuthAccountId column' + } + $inventory = Get-AzLocalClusterInventory @invParams if ($null -eq $inventory) { $inventory = @() } # Force array shape so .Count is reliable even when a single row comes back as a bare PSCustomObject. $inventory = @($inventory) @@ -189,7 +217,12 @@ function Invoke-AzLocalClusterInventory { # Empty fleet: Get-AzLocalClusterInventory does not write a CSV when there # are zero rows. Synthesise an empty canonical CSV so Step.2 has a file # to read (a header-only CSV is treated as 'no work to do'). - $emptyHeader = 'ClusterName,ResourceGroup,SubscriptionId,SubscriptionName,UpdateRing,HasUpdateRingTag,UpdateStartWindow,UpdateExclusions,UpdateSideloaded,UpdateVersionInProgress,ResourceId' + if ($IncludeSideloadColumns) { + $emptyHeader = 'ClusterName,ResourceGroup,SubscriptionId,SubscriptionName,UpdateRing,HasUpdateRingTag,UpdateStartWindow,UpdateExclusions,UpdateSideloaded,UpdateVersionInProgress,UpdateAuthAccountId,ResourceId' + } + else { + $emptyHeader = 'ClusterName,ResourceGroup,SubscriptionId,SubscriptionName,UpdateRing,HasUpdateRingTag,UpdateStartWindow,UpdateExclusions,UpdateSideloaded,UpdateVersionInProgress,ResourceId' + } Set-Content -LiteralPath $canonicalCsvPath -Value $emptyHeader -Encoding UTF8 Set-Content -LiteralPath $csvPath -Value $emptyHeader -Encoding UTF8 Write-Host "No cluster rows returned - wrote header-only CSV to $canonicalCsvPath." diff --git a/AzLocal.UpdateManagement/Public/Invoke-AzLocalReadinessGatedClusterUpdate.ps1 b/AzLocal.UpdateManagement/Public/Invoke-AzLocalReadinessGatedClusterUpdate.ps1 index 54c4d205..bfd668f0 100644 --- a/AzLocal.UpdateManagement/Public/Invoke-AzLocalReadinessGatedClusterUpdate.ps1 +++ b/AzLocal.UpdateManagement/Public/Invoke-AzLocalReadinessGatedClusterUpdate.ps1 @@ -8,7 +8,7 @@ function Invoke-AzLocalReadinessGatedClusterUpdate { per-cluster apply results to apply-results.json. .DESCRIPTION v0.8.5 Step.6 thin-YAML helper. Replaces the ~110-line inline `run:` - block that lived in both Step.6_apply-updates.yml pipelines. + block that lived in both Step.7_apply-updates.yml pipelines. Behaviour matches the prior inline block byte-for-byte: - Loads readiness-report.csv, validates the ClusterResourceId diff --git a/AzLocal.UpdateManagement/Public/Invoke-AzLocalSideloadUpdate.ps1 b/AzLocal.UpdateManagement/Public/Invoke-AzLocalSideloadUpdate.ps1 new file mode 100644 index 00000000..2bc1727d --- /dev/null +++ b/AzLocal.UpdateManagement/Public/Invoke-AzLocalSideloadUpdate.ps1 @@ -0,0 +1,346 @@ +function Invoke-AzLocalSideloadUpdate { + <# + .SYNOPSIS + Re-entrant Step.6 state machine that drives on-prem sideloading of Azure + Local solution updates: stage media, detached robocopy to the cluster + import share, remote SHA256 verify, Add-SolutionUpdate import, then flip the + UpdateSideloaded gate True. + + .DESCRIPTION + Public entry point for the v0.8.7 on-prem sideloading automation, designed + to be invoked repeatedly on a CRON by a single short-lived pipeline run. + Each invocation advances every in-scope cluster by ONE state transition and + exits; the multi-hour copy itself runs in a detached Windows Scheduled Task + (Tools/Invoke-AzLocalSideloadCopyTask.ps1) so no pipeline run is ever + long-lived. Progress is tracked in shared-UNC state JSON so any runner/agent + can advance/report without cross-agent remoting. + + Per cluster, the current shared state determines the action: + - (no state) + due-now -> set UpdateSideloaded=False, ensure media in + the shared cache, register+start the copy + Scheduled Task, write Copying state. + - Copying + fresh heartbeat -> report progress, leave running. + - Copying + stale heartbeat -> re-drive on this (live) host (bounded). + - Copied -> open WinRM session, verify remote hash, + import (Add-SolutionUpdate + discovery), + flip UpdateSideloaded=True + + UpdateVersionInProgress, mark Imported, + remove the task. + - Failed -> bounded retry, else surface error. + - Imported -> done. + + All Azure tag writes reuse Set-AzLocalClusterTagsMerge (Tag Contributor + RBAC only). The KV-derived AD credential is used solely for WinRM. + + .PARAMETER Plan + Plan rows from Resolve-AzLocalSideloadPlan. Only rows whose Status is + 'Planned' (due now) or which have existing in-flight state are advanced. + + .PARAMETER StateRoot + Shared UNC root for state\ and logs\. + + .PARAMETER CacheRoot + Shared verified media cache. Defaults to StateRoot\cache. + + .PARAMETER RobocopySwitches + Extra robocopy switches for the copy worker. + + .PARAMETER HeartbeatStaleMinutes + Minutes after which a Copying heartbeat is considered stale (dead host). + + .PARAMETER MaxRetries + Maximum copy re-drive attempts per cluster. + + .PARAMETER UseSsl + Use WinRM HTTPS (5986). Default $true. + + .PARAMETER TaskLogonType / TaskPrincipalUserId / TaskPassword + Scheduled-task identity controls (see Register-AzLocalSideloadCopyTask). + + .OUTPUTS + [PSCustomObject[]] one result row per processed cluster. + #> + [CmdletBinding(SupportsShouldProcess = $true)] + [OutputType([PSCustomObject[]])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNull()] + [PSCustomObject[]]$Plan, + + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$StateRoot, + + [Parameter(Mandatory = $false)] + [string]$CacheRoot, + + [Parameter(Mandatory = $false)] + [string]$RobocopySwitches = '/R:5 /W:30', + + [Parameter(Mandatory = $false)] + [ValidateRange(1, 1440)] + [int]$HeartbeatStaleMinutes = 60, + + [Parameter(Mandatory = $false)] + [ValidateRange(0, 20)] + [int]$MaxRetries = 3, + + [Parameter(Mandatory = $false)] + [bool]$UseSsl = $true, + + [Parameter(Mandatory = $false)] + [ValidateSet('S4U', 'Password', 'ServiceAccount', 'Interactive')] + [string]$TaskLogonType = 'S4U', + + [Parameter(Mandatory = $false)] + [string]$TaskPrincipalUserId, + + [Parameter(Mandatory = $false)] + [System.Security.SecureString]$TaskPassword + ) + + if ([string]::IsNullOrWhiteSpace($CacheRoot)) { + $CacheRoot = Join-Path -Path $StateRoot -ChildPath 'cache' + } + + $results = New-Object System.Collections.Generic.List[object] + + foreach ($p in $Plan) { + $clusterName = [string]$p.ClusterName + $state = Get-AzLocalSideloadState -StateRoot $StateRoot -ClusterName $clusterName + + # Skip clusters that are neither due now nor already in flight. + if ($null -eq $state -and $p.Status -ne 'Planned') { + $results.Add([PSCustomObject]@{ + ClusterName = $clusterName; Action = 'Skip'; State = $p.Status; Version = [string]$p.SelectedVersion; Message = [string]$p.Message + }) + continue + } + + try { + # ---------------- Terminal / in-flight state handling ---------------- + # NOTE: 'continue' inside a PowerShell switch targets the switch, not the + # enclosing foreach, so each case uses 'break' and the kickoff is guarded + # by 'else' to avoid falling through after handling existing state. + if ($null -ne $state) { + switch ([string]$state.State) { + 'Imported' { + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'None'; State = 'Imported'; Version = [string]$state.Version; Message = 'Already imported.' }) + break + } + 'Copying' { + if (Test-AzLocalSideloadHeartbeatStale -State $state -StaleMinutes $HeartbeatStaleMinutes) { + if ([int]$state.Retries -ge $MaxRetries) { + $state.State = 'Failed'; $state.Message = "Copy heartbeat stale and max retries ($MaxRetries) reached." + if ($PSCmdlet.ShouldProcess($clusterName, 'Persist Failed state (stale, retries exhausted)')) { Set-AzLocalSideloadState -StateRoot $StateRoot -State $state } + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'Failed'; State = 'Failed'; Version = [string]$state.Version; Message = $state.Message }) + } + else { + $advanced = Invoke-SideloadCopyStart -Plan $p -StateRoot $StateRoot -CacheRoot $CacheRoot -RobocopySwitches $RobocopySwitches -TaskLogonType $TaskLogonType -TaskPrincipalUserId $TaskPrincipalUserId -TaskPassword $TaskPassword -Retries ([int]$state.Retries + 1) -SkipGateFlip + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'ReDrive'; State = 'Copying'; Version = [string]$p.SelectedVersion; Message = "Stale heartbeat; re-driven (retry $($advanced.Retries))." }) + } + } + else { + $pct = if ([long]$state.TotalBytes -gt 0) { [math]::Round(([double]$state.CopiedBytes / [double]$state.TotalBytes) * 100, 1) } else { 0 } + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'InProgress'; State = 'Copying'; Version = [string]$state.Version; Message = "Copy in progress ($pct`%, $($state.Mbps) Mbps, ETA $($state.EtaUtc))." }) + } + break + } + 'Copied' { + $imp = Complete-SideloadImport -Plan $p -State $state -StateRoot $StateRoot -UseSsl $UseSsl + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'Import'; State = $imp.State; Version = [string]$state.Version; Message = $imp.Message }) + break + } + 'Failed' { + if ([int]$state.Retries -lt $MaxRetries) { + $advanced = Invoke-SideloadCopyStart -Plan $p -StateRoot $StateRoot -CacheRoot $CacheRoot -RobocopySwitches $RobocopySwitches -TaskLogonType $TaskLogonType -TaskPrincipalUserId $TaskPrincipalUserId -TaskPassword $TaskPassword -Retries ([int]$state.Retries + 1) -SkipGateFlip + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'Retry'; State = 'Copying'; Version = [string]$p.SelectedVersion; Message = "Retrying copy (retry $($advanced.Retries))." }) + } + else { + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'Failed'; State = 'Failed'; Version = [string]$state.Version; Message = "Copy failed; max retries ($MaxRetries) reached. $($state.Message)" }) + } + break + } + default { + # Queued / Verified / Stale - report and let next run advance. + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'Observe'; State = [string]$state.State; Version = [string]$state.Version; Message = [string]$state.Message }) + break + } + } + } + else { + # ---------------- No state + due now: kick off the copy ---------------- + $null = Invoke-SideloadCopyStart -Plan $p -StateRoot $StateRoot -CacheRoot $CacheRoot -RobocopySwitches $RobocopySwitches -TaskLogonType $TaskLogonType -TaskPrincipalUserId $TaskPrincipalUserId -TaskPassword $TaskPassword -Retries 0 + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'Start'; State = 'Copying'; Version = [string]$p.SelectedVersion; Message = "Sideload started; media staged and copy task launched." }) + } + } + catch { + $results.Add([PSCustomObject]@{ ClusterName = $clusterName; Action = 'Error'; State = 'Failed'; Version = [string]$p.SelectedVersion; Message = $_.Exception.Message }) + } + } + + return $results.ToArray() +} + +function Invoke-SideloadCopyStart { + # Module-private helper (NOT exported): stage media + register the copy task. + [CmdletBinding(SupportsShouldProcess = $true)] + [OutputType([PSCustomObject])] + param( + [PSCustomObject]$Plan, + [string]$StateRoot, + [string]$CacheRoot, + [string]$RobocopySwitches, + [string]$TaskLogonType, + [string]$TaskPrincipalUserId, + [System.Security.SecureString]$TaskPassword, + [int]$Retries = 0, + [switch]$SkipGateFlip + ) + + $clusterName = [string]$Plan.ClusterName + if (-not $PSCmdlet.ShouldProcess($clusterName, "Stage media and start sideload copy ($($Plan.SelectedVersion))")) { + return [PSCustomObject]@{ Retries = $Retries } + } + + # 1. Close the apply gate so Step.7 cannot apply mid-stage. + if (-not $SkipGateFlip) { + Set-AzLocalClusterTagsMerge -ClusterResourceId ([string]$Plan.ClusterResourceId) -Tags @{ $script:UpdateSideloadedTagName = 'False' } | Out-Null + } + + # 2. Ensure verified media in the shared cache. + $download = Get-AzLocalSolutionUpdateDownload -CatalogEntry $Plan.CatalogEntry -CacheRoot $CacheRoot + + # 3. Compute source + destination. SBE content lands in a named subfolder. + if ([string]$download.PackageType -eq 'SBE') { + $leaf = Split-Path -Path $download.MediaPath -Leaf + $dest = Join-Path -Path ([string]$Plan.TargetPath) -ChildPath $leaf + $mediaFileName = $leaf + } + else { + $dest = [string]$Plan.TargetPath + $mediaFileName = Split-Path -Path $download.MediaPath -Leaf + } + + # 4. Register + start the detached copy Scheduled Task. + $taskName = ('AzLocalSideload_{0}' -f ($clusterName -replace '[^A-Za-z0-9._-]', '_')) + $regParams = @{ + TaskName = $taskName + ClusterName = $clusterName + Version = [string]$Plan.SelectedVersion + SourcePath = [string]$download.MediaPath + TargetPath = $dest + StateRoot = $StateRoot + RobocopySwitches = $RobocopySwitches + LogonType = $TaskLogonType + } + if ($TaskPrincipalUserId) { $regParams['PrincipalUserId'] = $TaskPrincipalUserId } + if ($TaskPassword) { $regParams['Password'] = $TaskPassword } + Register-AzLocalSideloadCopyTask @regParams | Out-Null + + # 5. Write initial Copying state (carrying media/dest details + retries). + $state = New-AzLocalSideloadState -ClusterName $clusterName -Version ([string]$Plan.SelectedVersion) -State 'Copying' -TaskName $taskName -MediaPath ([string]$download.MediaPath) -TargetPath $dest + $state.Retries = $Retries + $state.Message = if ($Retries -gt 0) { "Copy re-driven (retry $Retries)." } else { 'Copy task launched.' } + Set-AzLocalSideloadState -StateRoot $StateRoot -State $state + + return [PSCustomObject]@{ Retries = $Retries; MediaFileName = $mediaFileName } +} + +function Complete-SideloadImport { + # Module-private helper (NOT exported): verify remote hash + import + flip gate. + [CmdletBinding(SupportsShouldProcess = $true)] + [OutputType([PSCustomObject])] + param( + [PSCustomObject]$Plan, + [PSCustomObject]$State, + [string]$StateRoot, + [bool]$UseSsl = $true + ) + + $clusterName = [string]$Plan.ClusterName + if (-not $PSCmdlet.ShouldProcess($clusterName, "Verify + import solution update '$($State.Version)'")) { + return [PSCustomObject]@{ State = 'Copied'; Message = 'WhatIf - import not performed.' } + } + + $authRow = $Plan.AuthRow + $credential = Resolve-AzLocalSideloadCredential -AuthRow $authRow + $authMech = if (-not [string]::IsNullOrWhiteSpace([string]$authRow.AuthMechanism)) { [string]$authRow.AuthMechanism } else { 'Negotiate' } + + $session = $null + try { + $session = New-AzLocalPSRemotingSession -ComputerName ([string]$Plan.RemotingHost) -Credential $credential -UseSsl $UseSsl -Authentication $authMech + + # Convert the UNC import target to the node-local path + media file path. + $nodeImportRoot = ConvertTo-AzLocalNodeLocalPath -UncPath ([string]$Plan.TargetPath) + $isSbe = ([string]$Plan.PackageType -eq 'SBE') + $mediaLeaf = Split-Path -Path ([string]$State.TargetPath) -Leaf + + if ($isSbe) { + # SBE content lives under the node import root in its named subfolder. + $importState = Invoke-AzLocalRemoteSolutionImport -Session $session -ImportRoot $nodeImportRoot -MediaFileName $mediaLeaf -PackageType 'SBE' -Version ([string]$State.Version) + } + else { + $mediaFileName = Split-Path -Path ([string]$State.MediaPath) -Leaf + $remoteFile = Join-Path -Path $nodeImportRoot -ChildPath $mediaFileName + $expectedSha = [string]$Plan.CatalogEntry.Sha256 + if (-not [string]::IsNullOrWhiteSpace($expectedSha)) { + $verify = Test-AzLocalRemoteFileHash -Session $session -RemotePath $remoteFile -ExpectedSha256 $expectedSha + if (-not $verify.Match) { + $State.State = 'Failed'; $State.Message = "Remote SHA256 mismatch (expected $expectedSha, got $($verify.ActualSha256))." + Set-AzLocalSideloadState -StateRoot $StateRoot -State $State + return [PSCustomObject]@{ State = 'Failed'; Message = $State.Message } + } + } + $importState = Invoke-AzLocalRemoteSolutionImport -Session $session -ImportRoot $nodeImportRoot -MediaFileName $mediaFileName -PackageType 'Solution' -Version ([string]$State.Version) + } + + switch ($importState.ImportState) { + 'Imported' { + # Flip the gate: UpdateSideloaded=True + record the staged version. + Set-AzLocalClusterTagsMerge -ClusterResourceId ([string]$Plan.ClusterResourceId) -Tags @{ + $script:UpdateSideloadedTagName = 'True' + $script:UpdateVersionInProgressTagName = [string]$State.Version + } | Out-Null + $State.State = 'Imported'; $State.Message = "Imported '$($importState.DiscoveredName)'; UpdateSideloaded=True." + Set-AzLocalSideloadState -StateRoot $StateRoot -State $State + if (-not [string]::IsNullOrWhiteSpace([string]$State.TaskName)) { + Remove-AzLocalSideloadCopyTask -TaskName ([string]$State.TaskName) + } + return [PSCustomObject]@{ State = 'Imported'; Message = $State.Message } + } + 'NeedsSbe' { + $State.State = 'Verified'; $State.Message = "Solution discovered but requires OEM SBE content (AdditionalContentRequired). Stage SBE and re-run." + Set-AzLocalSideloadState -StateRoot $StateRoot -State $State + return [PSCustomObject]@{ State = 'NeedsSbe'; Message = $State.Message } + } + 'Discovering' { + $State.Message = $importState.Message + Set-AzLocalSideloadState -StateRoot $StateRoot -State $State + return [PSCustomObject]@{ State = 'Discovering'; Message = $importState.Message } + } + default { + $State.State = 'Failed'; $State.Message = $importState.Message + Set-AzLocalSideloadState -StateRoot $StateRoot -State $State + return [PSCustomObject]@{ State = 'Failed'; Message = $importState.Message } + } + } + } + finally { + if ($null -ne $session) { Remove-PSSession -Session $session -ErrorAction SilentlyContinue } + $credential = $null + } +} + +function ConvertTo-AzLocalNodeLocalPath { + # Module-private helper: \\host\C$\rest -> C:\rest ; passthrough otherwise. + [CmdletBinding()] + [OutputType([string])] + param([Parameter(Mandatory = $true)][string]$UncPath) + + $m = [regex]::Match($UncPath, '^\\\\[^\\]+\\([A-Za-z])\$\\(.*)$') + if ($m.Success) { + return ('{0}:\{1}' -f $m.Groups[1].Value, $m.Groups[2].Value) + } + return $UncPath +} diff --git a/AzLocal.UpdateManagement/Public/New-AzLocalApplyUpdatesScheduleConfig.ps1 b/AzLocal.UpdateManagement/Public/New-AzLocalApplyUpdatesScheduleConfig.ps1 index 549535aa..6fec94e0 100644 --- a/AzLocal.UpdateManagement/Public/New-AzLocalApplyUpdatesScheduleConfig.ps1 +++ b/AzLocal.UpdateManagement/Public/New-AzLocalApplyUpdatesScheduleConfig.ps1 @@ -196,7 +196,7 @@ resources [void]$sb.AppendLine('# This is the single source of truth for "which UpdateRing(s) is/are') [void]$sb.AppendLine('# eligible to apply updates on a given UTC date". It is consumed by:') [void]$sb.AppendLine('#') - [void]$sb.AppendLine('# * Step.6_apply-updates.yml - reads it at every cron firing and') + [void]$sb.AppendLine('# * Step.7_apply-updates.yml - reads it at every cron firing and') [void]$sb.AppendLine('# resolves the UpdateRingValue to use') [void]$sb.AppendLine('# for that run.') [void]$sb.AppendLine('# * Step.3_apply-updates-schedule-audit.yml') diff --git a/AzLocal.UpdateManagement/Public/Resolve-AzLocalPipelineUpdateRing.ps1 b/AzLocal.UpdateManagement/Public/Resolve-AzLocalPipelineUpdateRing.ps1 index 45531f3b..0669f2e9 100644 --- a/AzLocal.UpdateManagement/Public/Resolve-AzLocalPipelineUpdateRing.ps1 +++ b/AzLocal.UpdateManagement/Public/Resolve-AzLocalPipelineUpdateRing.ps1 @@ -6,7 +6,7 @@ function Resolve-AzLocalPipelineUpdateRing { manual input or apply-updates-schedule.yml. .DESCRIPTION v0.8.5 Step.6 thin-YAML helper. Replaces the ~80-line inline `run:` - block that lived in both Step.6_apply-updates.yml pipelines (GitHub + block that lived in both Step.7_apply-updates.yml pipelines (GitHub Actions + Azure DevOps). Three resolution paths: diff --git a/AzLocal.UpdateManagement/Public/Resolve-AzLocalSideloadPlan.ps1 b/AzLocal.UpdateManagement/Public/Resolve-AzLocalSideloadPlan.ps1 new file mode 100644 index 00000000..7c6ac5be --- /dev/null +++ b/AzLocal.UpdateManagement/Public/Resolve-AzLocalSideloadPlan.ps1 @@ -0,0 +1,200 @@ +function Resolve-AzLocalSideloadPlan { + <# + .SYNOPSIS + Builds the per-cluster sideload plan: which solution-update version each + UpdateAuthAccountId-tagged Azure Local cluster should have sideloaded, and + whether it is due within the configured lead window. + + .DESCRIPTION + Public planner for the v0.8.7 on-prem sideloading automation. It does NOT + change any state - it only reads the fleet (Azure Resource Graph), the + apply-updates schedule, the auth map and the catalog, then emits one plan + object per cluster (plus error rows for misconfigurations such as an + unknown auth account id, a version not in the allow-list, or a missing + catalog entry). + + The plan objects feed Invoke-AzLocalSideloadUpdate (the re-entrant Step.6 + state machine). Selection reuses Select-AzLocalNextUpdateForCluster so the + sideload path and the apply path agree on which update is "next". + + .PARAMETER SchedulePath + Path to the apply-updates schedule YAML (Get-AzLocalApplyUpdatesScheduleConfig). + + .PARAMETER AuthMapPath + Path to the sideload auth-map CSV (Get-AzLocalSideloadAuthMap). + + .PARAMETER CatalogPath + Path to the sideload catalog YAML (Get-AzLocalSideloadCatalog). + + .PARAMETER LeadDays + How many days before a cluster's next apply window the media should be + sideloaded. A cluster is "due" when (nextWindow - LeadDays) <= Now. + + .PARAMETER UpdateRingValue + Optional UpdateRing tag filter (single, list, or '***' wildcard for all + rings). When omitted, all clusters carrying an UpdateAuthAccountId tag are + considered. + + .PARAMETER FqdnSuffix + Global default FQDN suffix appended to a cluster name to form the remoting + host when the auth-map row does not override it (SIDELOAD_REMOTING_FQDN_SUFFIX). + + .PARAMETER Now + The reference time (UTC). Defaults to now. + + .PARAMETER SubscriptionId + Optional subscription scope for the Resource Graph query. + + .OUTPUTS + [PSCustomObject[]] plan + error rows. + #> + [CmdletBinding()] + [OutputType([PSCustomObject[]])] + param( + [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()][string]$SchedulePath, + [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()][string]$AuthMapPath, + [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()][string]$CatalogPath, + [Parameter(Mandatory = $false)][ValidateRange(0, 365)][int]$LeadDays = 7, + [Parameter(Mandatory = $false)][string]$UpdateRingValue = '***', + [Parameter(Mandatory = $false)][string]$FqdnSuffix = '', + [Parameter(Mandatory = $false)][datetime]$Now = [datetime]::UtcNow, + [Parameter(Mandatory = $false)][string]$SubscriptionId + ) + + $config = Get-AzLocalApplyUpdatesScheduleConfig -Path $SchedulePath + $authMap = Get-AzLocalSideloadAuthMap -Path $AuthMapPath + $catalog = @(Get-AzLocalSideloadCatalog -Path $CatalogPath) + + $ringFilter = ConvertTo-AzLocalUpdateRingKqlFilter -UpdateRingValue $UpdateRingValue + $argQuery = "resources | where type =~ 'microsoft.azurestackhci/clusters' | where isnotempty(tags['UpdateAuthAccountId']) $ringFilter | project id, name, resourceGroup, subscriptionId, tags" + + $clusterRows = if ($PSBoundParameters.ContainsKey('SubscriptionId') -and $SubscriptionId) { + Invoke-AzResourceGraphQuery -Query $argQuery -SubscriptionId $SubscriptionId + } + else { + Invoke-AzResourceGraphQuery -Query $argQuery + } + + # Pre-compute the next 366 days of firings so we can find each cluster's next window. + $firings = @(Get-AzLocalApplyUpdatesScheduleNextFirings -Schedule $config -StartDate $Now.Date -Days 366) + + $results = New-Object System.Collections.Generic.List[object] + + foreach ($cluster in $clusterRows) { + $clusterName = [string]$cluster.name + $accountId = [string]$cluster.tags.UpdateAuthAccountId + $ringTag = [string]$cluster.tags.UpdateRing + + $row = [ordered]@{ + ClusterName = $clusterName + ClusterResourceId = [string]$cluster.id + ResourceGroup = [string]$cluster.resourceGroup + SubscriptionId = [string]$cluster.subscriptionId + UpdateAuthAccountId = $accountId + Ring = $ringTag + NextWindowUtc = $null + LeadDays = $LeadDays + DueNow = $false + SelectedVersion = '' + SelectedUpdateName = '' + PackageType = '' + CatalogEntry = $null + RemotingHost = '' + TargetPath = '' + AuthRow = $null + Status = 'Planned' + Message = '' + } + + # Auth-map row. + if (-not $authMap.ContainsKey($accountId)) { + $row.Status = 'UnknownAuthAccountId' + $row.Message = "Cluster '$clusterName' has UpdateAuthAccountId '$accountId' which is not present in the auth map." + $results.Add([PSCustomObject]$row); continue + } + $authRow = $authMap[$accountId] + $row.AuthRow = $authRow + + # Remoting host + target path. + $remotingHost = if (-not [string]::IsNullOrWhiteSpace([string]$authRow.RemotingTargetFqdn)) { + [string]$authRow.RemotingTargetFqdn + } + else { + $suffix = if (-not [string]::IsNullOrWhiteSpace([string]$authRow.FqdnSuffix)) { [string]$authRow.FqdnSuffix } else { $FqdnSuffix } + if (-not [string]::IsNullOrWhiteSpace($suffix)) { ('{0}{1}' -f $clusterName, $suffix) } else { $clusterName } + } + $row.RemotingHost = $remotingHost + $row.TargetPath = Resolve-AzLocalSideloadTargetPath -RemotingHost $remotingHost -ImportSharePath ([string]$authRow.ImportSharePath) + + # Next apply window for this cluster's ring. + $nextFiring = $firings | Where-Object { + $_.Rings -and ($_.Rings -contains $ringTag) -and ($_.DateUtc -ge $Now.Date) + } | Sort-Object DateUtc | Select-Object -First 1 + if ($nextFiring) { + $row.NextWindowUtc = $nextFiring.DateUtc + $dueDate = ([datetime]$nextFiring.DateUtc).AddDays(-$LeadDays) + $row.DueNow = ($Now -ge $dueDate) + } + + # Resolve allow-list for the cluster (today's resolution gives the list). + $ringInfo = Resolve-AzLocalCurrentUpdateRing -Schedule $config -Now $Now + $allowed = @($ringInfo.AllowedUpdateVersions) + + # Available Ready updates -> adapter shape for Select-AzLocalNextUpdateForCluster. + $available = @(Get-AzLocalAvailableUpdates -ClusterResourceId ([string]$cluster.id) -ErrorAction SilentlyContinue) + $readyAdapters = @( + $available | Where-Object { [string]$_.UpdateState -eq 'Ready' } | ForEach-Object { + [PSCustomObject]@{ + name = [string]$_.UpdateName + PackageType = [string]$_.PackageType + properties = [PSCustomObject]@{ version = [string]$_.Version; state = [string]$_.UpdateState } + } + } + ) + + $selection = Select-AzLocalNextUpdateForCluster -ReadyUpdates $readyAdapters -AllowedUpdateVersions $allowed + switch ($selection.Reason) { + 'Selected' { + $sel = $selection.SelectedUpdate + $version = [string]$sel.properties.version + if ([string]::IsNullOrWhiteSpace($version)) { $version = [string]$sel.name } + $row.SelectedVersion = $version + $row.SelectedUpdateName = [string]$sel.name + $row.PackageType = [string]$sel.PackageType + + $entry = $catalog | Where-Object { [string]$_.Version -eq $version } | Select-Object -First 1 + if ($null -eq $entry) { + $row.Status = 'NoCatalogEntry' + $row.Message = "Selected version '$version' for cluster '$clusterName' has no entry in the sideload catalog." + } + else { + $row.CatalogEntry = $entry + if (-not $row.PackageType) { $row.PackageType = [string]$entry.PackageType } + $row.Status = if ($row.DueNow) { 'Planned' } else { 'NotDue' } + $row.Message = if ($row.DueNow) { + "Cluster '$clusterName' due for sideload of '$version' (window $($row.NextWindowUtc))." + } + else { + "Cluster '$clusterName' not yet within the $LeadDays-day lead window for '$version'." + } + } + } + 'NotInAllowList' { + $row.Status = 'NotInAllowList' + $row.Message = "No Ready update for cluster '$clusterName' matches the allow-list [$($selection.AllowDisplay)]. Ready: [$($selection.ReadyDisplay)]." + } + 'NoneReady' { + $row.Status = 'NoneReady' + $row.Message = "Cluster '$clusterName' has no Ready updates available." + } + default { + $row.Status = [string]$selection.Reason + $row.Message = "Cluster '$clusterName' selection reason: $($selection.Reason)." + } + } + + $results.Add([PSCustomObject]$row) + } + + return $results.ToArray() +} diff --git a/AzLocal.UpdateManagement/Public/Set-AzLocalClusterUpdateRingTag.ps1 b/AzLocal.UpdateManagement/Public/Set-AzLocalClusterUpdateRingTag.ps1 index e1e732aa..0061be62 100644 --- a/AzLocal.UpdateManagement/Public/Set-AzLocalClusterUpdateRingTag.ps1 +++ b/AzLocal.UpdateManagement/Public/Set-AzLocalClusterUpdateRingTag.ps1 @@ -53,6 +53,14 @@ function Set-AzLocalClusterUpdateRingTag { Note: regardless of whether this parameter is supplied, this function ALWAYS stamps UpdateExcluded='False' on any cluster that does not already carry the tag, so the tag is discoverable in the Azure portal and ready for an operator to flip to 'True'. + + .PARAMETER UpdateAuthAccountIdValue + Optional. Numeric (1-3 digit) value to assign to the "UpdateAuthAccountId" tag when + using -ClusterResourceIds. This tag maps the cluster to a row in the on-prem sideload + auth-map (sideload-auth-map.csv) used by the opt-in Step.6 sideload workflow. Must + match ^\d{1,3}$ (e.g. '001') - non-numeric values are rejected. Not used with + -InputCsvPath (values come from the optional UpdateAuthAccountId column). When the + column/value is absent, no UpdateAuthAccountId tag is written. New in v0.8.7. .PARAMETER Force If specified, will overwrite existing "UpdateRing" tags. Without this switch, @@ -129,6 +137,10 @@ function Set-AzLocalClusterUpdateRingTag { [ValidateSet('True', 'False', '1', '0', 'true', 'false', '', IgnoreCase = $true)] [string]$UpdateExcludedValue, + [Parameter(Mandatory = $false, ParameterSetName = 'ByResourceId')] + [ValidatePattern('^(\d{1,3})?$')] + [string]$UpdateAuthAccountIdValue, + [Parameter(Mandatory = $false)] [switch]$Force, @@ -203,11 +215,13 @@ function Set-AzLocalClusterUpdateRingTag { $hasUpdateStartWindowCol = 'UpdateStartWindow' -in $csvColumns $hasUpdateExclusionsWindowCol = 'UpdateExclusionsWindow' -in $csvColumns $hasUpdateExcludedCol = 'UpdateExcluded' -in $csvColumns - if ($hasUpdateStartWindowCol -or $hasUpdateExclusionsWindowCol -or $hasUpdateExcludedCol) { + $hasUpdateAuthAccountIdCol = 'UpdateAuthAccountId' -in $csvColumns + if ($hasUpdateStartWindowCol -or $hasUpdateExclusionsWindowCol -or $hasUpdateExcludedCol -or $hasUpdateAuthAccountIdCol) { $scheduleColumns = @() if ($hasUpdateStartWindowCol) { $scheduleColumns += 'UpdateStartWindow' } if ($hasUpdateExclusionsWindowCol) { $scheduleColumns += 'UpdateExclusionsWindow' } if ($hasUpdateExcludedCol) { $scheduleColumns += 'UpdateExcluded' } + if ($hasUpdateAuthAccountIdCol) { $scheduleColumns += 'UpdateAuthAccountId' } Write-Log -Message "CSV includes schedule tag columns: $($scheduleColumns -join ', ')" -Level Info } @@ -226,6 +240,9 @@ function Set-AzLocalClusterUpdateRingTag { if ($hasUpdateExcludedCol -and $row.UpdateExcluded -and $row.UpdateExcluded.Trim() -ne '') { $entry['UpdateExcludedValue'] = $row.UpdateExcluded.Trim() } + if ($hasUpdateAuthAccountIdCol -and $row.UpdateAuthAccountId -and $row.UpdateAuthAccountId.Trim() -ne '') { + $entry['UpdateAuthAccountIdValue'] = $row.UpdateAuthAccountId.Trim() + } $clustersToTag += $entry } } @@ -247,6 +264,9 @@ function Set-AzLocalClusterUpdateRingTag { if ($UpdateExcludedValue) { Write-Log -Message "UpdateExcluded value to set: $UpdateExcludedValue" -Level Info } + if ($UpdateAuthAccountIdValue) { + Write-Log -Message "UpdateAuthAccountId value to set: $UpdateAuthAccountIdValue" -Level Info + } foreach ($resourceId in $ClusterResourceIds) { $entry = @{ @@ -262,6 +282,9 @@ function Set-AzLocalClusterUpdateRingTag { if ($UpdateExcludedValue) { $entry['UpdateExcludedValue'] = $UpdateExcludedValue } + if ($UpdateAuthAccountIdValue) { + $entry['UpdateAuthAccountIdValue'] = $UpdateAuthAccountIdValue + } $clustersToTag += $entry } } @@ -466,6 +489,7 @@ function Set-AzLocalClusterUpdateRingTag { $hasNewScheduleTags = ($clusterEntry.UpdateStartWindowValue -and (-not $currentTags.PSObject.Properties[$script:UpdateStartWindowTagName] -or $currentTags.$($script:UpdateStartWindowTagName) -ne $clusterEntry.UpdateStartWindowValue)) -or ($clusterEntry.UpdateExclusionsWindowValue -and (-not $currentTags.PSObject.Properties[$script:UpdateExclusionsWindowTagName] -or $currentTags.$($script:UpdateExclusionsWindowTagName) -ne $clusterEntry.UpdateExclusionsWindowValue)) -or ($clusterEntry.UpdateExcludedValue -and (-not $currentTags.PSObject.Properties[$script:UpdateExcludedTagName] -or $currentTags.$($script:UpdateExcludedTagName) -ne $clusterEntry.UpdateExcludedValue)) -or + ($clusterEntry.UpdateAuthAccountIdValue -and (-not $currentTags.PSObject.Properties[$script:UpdateAuthAccountIdTagName] -or $currentTags.$($script:UpdateAuthAccountIdTagName) -ne $clusterEntry.UpdateAuthAccountIdValue)) -or $needsExcludedDefaultStamp if (-not $Force -and -not $hasNewScheduleTags) { @@ -576,6 +600,19 @@ function Set-AzLocalClusterUpdateRingTag { Write-Log -Message " Will default-stamp $($script:UpdateExcludedTagName) tag: 'False' (tag absent on cluster)" -Level Info } + # Sideload auth-map account id (opt-in Step.6 workflow). Validate it is + # numeric (1-3 digits) before writing - a non-numeric value would never + # match a sideload-auth-map.csv row. An invalid value throws and is + # surfaced as a per-cluster Failed result by the surrounding catch. + if ($clusterEntry.UpdateAuthAccountIdValue) { + $authIdToWrite = ([string]$clusterEntry.UpdateAuthAccountIdValue).Trim() + if ($authIdToWrite -notmatch '^\d{1,3}$') { + throw "Invalid UpdateAuthAccountId '$authIdToWrite' for cluster '$clusterName' - must be numeric (1-3 digits, e.g. 001)." + } + $tagsToMerge[$script:UpdateAuthAccountIdTagName] = $authIdToWrite + Write-Log -Message " Will also set $($script:UpdateAuthAccountIdTagName) tag: $authIdToWrite" -Level Info + } + # Compute the actual per-tag deltas so the per-cluster Message in the # summary names the tags that genuinely changed (not always "UpdateRing"). # When only a schedule tag (e.g. UpdateExcluded) differs we want the diff --git a/AzLocal.UpdateManagement/Public/Start-AzLocalClusterUpdate.ps1 b/AzLocal.UpdateManagement/Public/Start-AzLocalClusterUpdate.ps1 index 43a3986b..c3817296 100644 --- a/AzLocal.UpdateManagement/Public/Start-AzLocalClusterUpdate.ps1 +++ b/AzLocal.UpdateManagement/Public/Start-AzLocalClusterUpdate.ps1 @@ -1007,86 +1007,73 @@ function Start-AzLocalClusterUpdate { # strips 'Latest' before calling this cmdlet (emits empty # to the env var), but be defensive in case an operator # invokes -AllowedUpdateVersions Latest manually. - $allowListEffective = @($AllowedUpdateVersions | Where-Object { - -not [string]::IsNullOrWhiteSpace([string]$_) - }) - $allowOnlyLatest = ($allowListEffective.Count -gt 0) -and ( - -not (@($allowListEffective | Where-Object { - -not [string]::Equals([string]$_, 'Latest', [System.StringComparison]::OrdinalIgnoreCase) - }).Count -gt 0) - ) - if ($allowOnlyLatest) { + # + # v0.8.7: the allow-list filter + latest-by-YYMM auto-pick is + # centralised in the Private helper Select-AzLocalNextUpdateForCluster + # so the on-prem sideloading automation resolves the SAME "next + # update". All logging / CSV / result-object side effects below + # are preserved byte-for-byte from the original inline block. + $selection = Select-AzLocalNextUpdateForCluster -ReadyUpdates $readyUpdates -AllowedUpdateVersions $AllowedUpdateVersions -UpdateName $UpdateName + + if ($selection.AllowOnlyLatest) { Write-Log -Message "AllowedUpdateVersions = 'Latest' (no-constraint sentinel) - skipping allow-list filter for cluster '$clusterName'; will install the latest Ready update." -Level Info } - if ($allowListEffective.Count -gt 0 -and -not $allowOnlyLatest) { - if ($UpdateName) { - Write-Log -Message ("Both -UpdateName ('$UpdateName') and -AllowedUpdateVersions ({0} entries) were supplied for cluster '$clusterName'. -UpdateName takes precedence; allow-list is logged-and-ignored for this cluster." -f @($AllowedUpdateVersions).Count) -Level Warning - } - else { - $allowSet = New-Object System.Collections.Generic.HashSet[string] ([System.StringComparer]::OrdinalIgnoreCase) - foreach ($a in @($AllowedUpdateVersions)) { - $sa = [string]$a - if (-not [string]::IsNullOrWhiteSpace($sa)) { [void]$allowSet.Add($sa.Trim()) } - } - $filtered = @($readyUpdates | Where-Object { - ($_.name -and $allowSet.Contains([string]$_.name)) -or - ($_.properties -and $_.properties.version -and $allowSet.Contains([string]$_.properties.version)) - }) - $allowDisplay = (@($allowSet) -join ', ') - if ($filtered.Count -eq 0) { - $readyDisplay = (@($readyUpdates | ForEach-Object { "$($_.name) (v$($_.properties.version))" }) -join '; ') - Write-Log -Message "Cluster '$clusterName': none of the $($readyUpdates.Count) Ready update(s) match the AllowedUpdateVersions allow-list. Allow-list: [$allowDisplay]. Available Ready: [$readyDisplay]. Skipping cluster (status='NotInAllowList')." -Level Warning + $allowListActive = ($selection.AllowListEffective.Count -gt 0 -and -not $selection.AllowOnlyLatest) + if ($allowListActive -and $UpdateName) { + Write-Log -Message ("Both -UpdateName ('$UpdateName') and -AllowedUpdateVersions ({0} entries) were supplied for cluster '$clusterName'. -UpdateName takes precedence; allow-list is logged-and-ignored for this cluster." -f @($AllowedUpdateVersions).Count) -Level Warning + } - # Parse Resource Group and Subscription ID from cluster resource ID - $clusterRgName = ($clusterInfo.id -split '/resourceGroups/')[1] -split '/' | Select-Object -First 1 - $clusterSubId = ($clusterInfo.id -split '/subscriptions/')[1] -split '/' | Select-Object -First 1 - $healthState = if ($updateSummary.properties.healthState) { $updateSummary.properties.healthState } else { "Unknown" } - Write-UpdateCsvLog -LogType Skipped ` - -ClusterName $clusterName ` - -ResourceGroup $clusterRgName ` - -SubscriptionId $clusterSubId ` - -Message "Update skipped - no Ready update matches AllowedUpdateVersions allow-list [$allowDisplay]. Available Ready: $readyDisplay" ` - -UpdateState $updateSummary.properties.state ` - -HealthState $healthState + if ($selection.Reason -eq 'NotInAllowList') { + $allowDisplay = $selection.AllowDisplay + $readyDisplay = $selection.ReadyDisplay + Write-Log -Message "Cluster '$clusterName': none of the $($readyUpdates.Count) Ready update(s) match the AllowedUpdateVersions allow-list. Allow-list: [$allowDisplay]. Available Ready: [$readyDisplay]. Skipping cluster (status='NotInAllowList')." -Level Warning - $results.Add([PSCustomObject]@{ - ClusterName = $clusterName - Status = "NotInAllowList" - Message = "No Ready update matches AllowedUpdateVersions allow-list [$allowDisplay]" - UpdateName = $null - StartTime = $clusterStartTime - EndTime = Get-Date - Duration = $null - }) | Out-Null - continue - } - Write-Log -Message "AllowedUpdateVersions filter kept $($filtered.Count)/$($readyUpdates.Count) Ready update(s) for cluster '$clusterName'. Allow-list: [$allowDisplay]." -Level Info - $readyUpdates = $filtered - } + # Parse Resource Group and Subscription ID from cluster resource ID + $clusterRgName = ($clusterInfo.id -split '/resourceGroups/')[1] -split '/' | Select-Object -First 1 + $clusterSubId = ($clusterInfo.id -split '/subscriptions/')[1] -split '/' | Select-Object -First 1 + $healthState = if ($updateSummary.properties.healthState) { $updateSummary.properties.healthState } else { "Unknown" } + Write-UpdateCsvLog -LogType Skipped ` + -ClusterName $clusterName ` + -ResourceGroup $clusterRgName ` + -SubscriptionId $clusterSubId ` + -Message "Update skipped - no Ready update matches AllowedUpdateVersions allow-list [$allowDisplay]. Available Ready: $readyDisplay" ` + -UpdateState $updateSummary.properties.state ` + -HealthState $healthState + + $results.Add([PSCustomObject]@{ + ClusterName = $clusterName + Status = "NotInAllowList" + Message = "No Ready update matches AllowedUpdateVersions allow-list [$allowDisplay]" + UpdateName = $null + StartTime = $clusterStartTime + EndTime = Get-Date + Duration = $null + }) | Out-Null + continue + } + + if ($allowListActive -and -not $UpdateName) { + Write-Log -Message "AllowedUpdateVersions filter kept $($selection.FilteredUpdates.Count)/$($readyUpdates.Count) Ready update(s) for cluster '$clusterName'. Allow-list: [$($selection.AllowDisplay)]." -Level Info } # Step 5: Select update to apply Write-Log -Message "Step 5: Selecting update to apply..." -Level Info - $selectedUpdate = $null - if ($UpdateName) { - $selectedUpdate = $readyUpdates | Where-Object { $_.name -eq $UpdateName } - if (-not $selectedUpdate) { - Write-Log -Message "Specified update '$UpdateName' not found or not in Ready state for cluster '$clusterName'." -Level Warning - $results.Add([PSCustomObject]@{ - ClusterName = $clusterName - Status = "UpdateNotFound" - Message = "Specified update '$UpdateName' not found or not ready" - UpdateName = $UpdateName - StartTime = $clusterStartTime - EndTime = Get-Date - Duration = $null - }) | Out-Null - continue - } + if ($selection.Reason -eq 'UpdateNotFound') { + Write-Log -Message "Specified update '$UpdateName' not found or not in Ready state for cluster '$clusterName'." -Level Warning + $results.Add([PSCustomObject]@{ + ClusterName = $clusterName + Status = "UpdateNotFound" + Message = "Specified update '$UpdateName' not found or not ready" + UpdateName = $UpdateName + StartTime = $clusterStartTime + EndTime = Get-Date + Duration = $null + }) | Out-Null + continue } - else { + $selectedUpdate = $selection.SelectedUpdate + if (-not $UpdateName) { # Select the latest ready update by YYMM version from the update name - $selectedUpdate = Get-LatestUpdateByYYMM -Updates $readyUpdates Write-Log -Message "Auto-selected latest update: $($selectedUpdate.name)" -Level Info } diff --git a/AzLocal.UpdateManagement/Public/Test-AzLocalApplyUpdatesScheduleCoverage.ps1 b/AzLocal.UpdateManagement/Public/Test-AzLocalApplyUpdatesScheduleCoverage.ps1 index 055505ff..fd9219b9 100644 --- a/AzLocal.UpdateManagement/Public/Test-AzLocalApplyUpdatesScheduleCoverage.ps1 +++ b/AzLocal.UpdateManagement/Public/Test-AzLocalApplyUpdatesScheduleCoverage.ps1 @@ -93,7 +93,7 @@ function Test-AzLocalApplyUpdatesScheduleCoverage { older CSVs that pre-date the ResourceId column. .PARAMETER PipelineYamlPath - Optional for -View Audit. Path to a single Step.6_apply-updates.yml file, or to + Optional for -View Audit. Path to a single Step.7_apply-updates.yml file, or to a folder that contains apply-updates*.yml files (typically the Automation-Pipeline-Examples folder of your forked module). Drives the cron-vs-UpdateStartWindow coverage check. May be supplied together with @@ -236,7 +236,7 @@ function Test-AzLocalApplyUpdatesScheduleCoverage { if ($View -eq 'Audit' -and [string]::IsNullOrWhiteSpace($PipelineYamlPath) -and [string]::IsNullOrWhiteSpace($SchedulePath)) { - throw "-View 'Audit' requires at least one of -PipelineYamlPath or -SchedulePath. Point -PipelineYamlPath at Step.6_apply-updates.yml (or the Automation-Pipeline-Examples folder) and/or -SchedulePath at your apply-updates-schedule.yml." + throw "-View 'Audit' requires at least one of -PipelineYamlPath or -SchedulePath. Point -PipelineYamlPath at Step.7_apply-updates.yml (or the Automation-Pipeline-Examples folder) and/or -SchedulePath at your apply-updates-schedule.yml." } if ($PipelineYamlPath -and -not (Test-Path -LiteralPath $PipelineYamlPath)) { throw "PipelineYamlPath not found: $PipelineYamlPath" @@ -609,12 +609,12 @@ resources if ($emitGh) { if ($emitAdo) { - [void]$cronSb.AppendLine('### GitHub Actions - paste under the existing `on:` key in Step.6_apply-updates.yml') + [void]$cronSb.AppendLine('### GitHub Actions - paste under the existing `on:` key in Step.7_apply-updates.yml') [void]$cronSb.AppendLine() } # v0.8.1: emit ONLY the `schedule:` block (no surrounding `on:`/`workflow_dispatch:` # lines) so the snippet can be pasted as-is under the existing `on:` key of - # Step.6_apply-updates.yml. Step.6 already declares `workflow_dispatch:` with a + # Step.7_apply-updates.yml. Step.7 already declares `workflow_dispatch:` with a # rich `inputs:` block (update_ring, dry_run, ITSM, module_version) - emitting a # second bare `workflow_dispatch:` here produced a duplicate top-level key and # GH rejected the workflow ("'workflow_dispatch' is already defined"). The @@ -639,7 +639,7 @@ resources } if ($emitAdo) { if ($emitGh) { - [void]$cronSb.AppendLine('### Azure DevOps - paste at the top level of Step.6_apply-updates.yml') + [void]$cronSb.AppendLine('### Azure DevOps - paste at the top level of Step.7_apply-updates.yml') [void]$cronSb.AppendLine() } [void]$cronSb.AppendLine('```yaml') @@ -696,9 +696,9 @@ resources } } $step6FileLabel = switch ($Platform) { - 'GitHubActions' { '.github/workflows/Step.6_apply-updates.yml' } - 'AzureDevOps' { '.azuredevops/Step.6_apply-updates.yml' } - default { 'Step.6_apply-updates.yml' } + 'GitHubActions' { '.github/workflows/Step.7_apply-updates.yml' } + 'AzureDevOps' { '.azuredevops/Step.7_apply-updates.yml' } + default { 'Step.7_apply-updates.yml' } } $fullSb = New-Object System.Text.StringBuilder @@ -830,11 +830,11 @@ resources [void]$fullSb.AppendLine("### How to fix - edit ``$step6FileLabel``") [void]$fullSb.AppendLine() if ($emitGh -and -not $emitAdo) { - [void]$fullSb.AppendLine('Add (or merge with) the following `schedule:` block under the existing `on:` key. Place it inside the `# BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers` / `# END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers so it survives `Update-AzLocalPipelineExample` refreshes. **Do NOT add a second `workflow_dispatch:` line** - Step.6 already declares one with the `update_ring` / `dry_run` / ITSM / `module_version` inputs that the manual `Run workflow` button needs:') + [void]$fullSb.AppendLine('Add (or merge with) the following `schedule:` block under the existing `on:` key. Place it inside the `# BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers` / `# END-AZLOCAL-CUSTOMIZE:schedule-triggers` markers so it survives `Update-AzLocalPipelineExample` refreshes. **Do NOT add a second `workflow_dispatch:` line** - Step.7 already declares one with the `update_ring` / `dry_run` / ITSM / `module_version` inputs that the manual `Run workflow` button needs:') } elseif ($emitAdo -and -not $emitGh) { [void]$fullSb.AppendLine('Add (or merge with) a top-level `schedules:` block:') } else { - [void]$fullSb.AppendLine('Choose the snippet matching your CI platform and paste/merge into your Step.6 pipeline file. For GitHub Actions, paste the `schedule:` block under the existing `on:` key (do NOT add a second `workflow_dispatch:` - Step.6 already declares one). For Azure DevOps, paste the `schedules:` block at the top level.') + [void]$fullSb.AppendLine('Choose the snippet matching your CI platform and paste/merge into your Step.6 pipeline file. For GitHub Actions, paste the `schedule:` block under the existing `on:` key (do NOT add a second `workflow_dispatch:` - Step.7 already declares one). For Azure DevOps, paste the `schedules:` block at the top level.') } [void]$fullSb.AppendLine() # v0.8.2: paste-tip - the snippet below is at 2-space indent (sibling of diff --git a/AzLocal.UpdateManagement/Public/Update-AzLocalPipelineExample.ps1 b/AzLocal.UpdateManagement/Public/Update-AzLocalPipelineExample.ps1 index 6634f9de..78cd3dad 100644 --- a/AzLocal.UpdateManagement/Public/Update-AzLocalPipelineExample.ps1 +++ b/AzLocal.UpdateManagement/Public/Update-AzLocalPipelineExample.ps1 @@ -24,14 +24,26 @@ function Update-AzLocalPipelineExample { currently shipped: schedule-triggers (every main pipeline) - itsm-secrets (Step.6_apply-updates.yml only) + itsm-secrets (Step.7_apply-updates.yml only) Per source YAML the cmdlet: - 1. Locates the matching destination file under -Destination. + 1. Locates the matching destination file under -Destination BY + STABLE LOGICAL ID (v0.8.7), not by filename. Each bundled YAML + carries a '# AZLOCAL-PIPELINE-ID: ' header comment; the + cmdlet matches a destination file by (a) the canonical filename, + else (b) a destination YAML whose embedded ID equals the source + ID, else (c) a legacy 'Step.N_.yml' alias filename recorded + in the bundled pipeline manifest. A (b)/(c) match whose filename + differs from the canonical name is RENAMED on disk to the + canonical name as part of the merge, carrying the customer's + marker bodies (e.g. schedule CRONs) forward. This means a module + release that renames or renumbers a pipeline is no longer a + breaking change - the customer's CRONs follow the pipeline. - Net-new files in the source set are CREATED (full copy). - - Files present at -Destination but not in the source set are - left untouched (orphaned customer-only files survive). + - Files present at -Destination but not in the source set (and + not matched by ID/alias) are left untouched (orphaned + customer-only files survive). 2. Parses both files for marker pairs. For every marker name found in BOTH files the destination body (the lines between BEGIN and @@ -97,10 +109,18 @@ function Update-AzLocalPipelineExample { .OUTPUTS PSCustomObject[] (with -PassThru) - one row per source file with: - File - destination path + File - destination path (always the CANONICAL, + de-numbered filename) Action - 'Created' | 'Updated' | 'Updated-PinOnly' - | 'Unchanged' | 'Overwritten' + | 'Renamed' | 'Unchanged' | 'Overwritten' | 'Skipped-NeedsForce' | 'Skipped-NoChange' + RenamedFrom - legacy filename the destination file was + matched under and renamed FROM (e.g. + 'Step.7_apply-updates.yml'), or $null when no + rename occurred. A non-null value means the + file was physically renamed to the canonical + name; if Action is also 'Updated'/'Overwritten' + the content was refreshed in the same pass. PreservedMarkers - [string[]] marker names whose body was preserved from the destination NewMarkers - [string[]] marker names introduced in this @@ -193,8 +213,23 @@ function Update-AzLocalPipelineExample { Write-Log -Message "Update-AzLocalPipelineExample: comparing bundled $Platform samples in '$platformSrc' against destination '$destResolved'." -Level Info # ------------------------------------------------------------------ - # 3. Build the list of source YAMLs. Match on exact filename at the - # destination so renames stay the caller's responsibility. + # 3. Build the list of source YAMLs and the rename-aware matching maps. + # + # v0.8.7: pipelines are matched to their destination counterpart by + # STABLE LOGICAL ID (the '# AZLOCAL-PIPELINE-ID:' header comment), not + # by filename. This lets the cmdlet follow a pipeline across a filename + # rename or a Step.N renumber WITHOUT stranding the customer's + # BEGIN/END-AZLOCAL-CUSTOMIZE marker bodies (e.g. their schedule CRONs). + # + # Resolution order for each source file's destination match: + # (a) canonical filename present at -Destination, else + # (b) a destination *.yml whose embedded AZLOCAL-PIPELINE-ID equals + # the source ID (the customer renamed it, or it is a newer copy), + # else + # (c) a legacy 'Step.N_.yml' alias filename listed in the + # manifest (a pre-v0.8.7 copy that predates the ID comment). + # A (b)/(c) match whose filename differs from the canonical name is + # RENAMED on disk to the canonical name after the marker-aware merge. # ------------------------------------------------------------------ $srcFiles = @(Get-ChildItem -LiteralPath $platformSrc -Filter '*.yml' -File -ErrorAction Stop) if ($srcFiles.Count -eq 0) { @@ -202,23 +237,97 @@ function Update-AzLocalPipelineExample { return } + # Manifest: logical ID -> canonical filename + legacy alias filenames. + $manifest = @(Get-AzLocalPipelineManifest) + + # Pre-scan the destination folder once, indexing any *.yml that carries an + # AZLOCAL-PIPELINE-ID comment by that ID (first occurrence wins). Used for + # resolution path (b). + $destIdMap = @{} + foreach ($destYml in @(Get-ChildItem -LiteralPath $destResolved -Filter '*.yml' -File -ErrorAction SilentlyContinue)) { + try { + $destYmlText = [System.IO.File]::ReadAllText($destYml.FullName, [System.Text.UTF8Encoding]::new($false)) + $destYmlId = Get-AzLocalPipelineId -Text $destYmlText + if ($destYmlId -and -not $destIdMap.ContainsKey($destYmlId)) { + $destIdMap[$destYmlId] = $destYml.FullName + } + } + catch { + Write-Verbose "Update-AzLocalPipelineExample: could not read '$($destYml.FullName)' during ID pre-scan: $($_.Exception.Message)" + } + } + $results = New-Object System.Collections.Generic.List[pscustomobject] foreach ($srcFile in $srcFiles) { + # Canonical destination path = the bundled (already de-numbered) + # filename dropped into -Destination. This is always the WRITE target. $destFile = Join-Path -Path $destResolved -ChildPath $srcFile.Name + # Resolve the source pipeline's logical ID. Prefer the embedded + # AZLOCAL-PIPELINE-ID comment; fall back to the de-numbered base name. + $srcText = [System.IO.File]::ReadAllText($srcFile.FullName, [System.Text.UTF8Encoding]::new($false)) + $srcId = Get-AzLocalPipelineId -Text $srcText + if (-not $srcId) { + $srcId = [System.IO.Path]::GetFileNameWithoutExtension($srcFile.Name) + } + $row = [PSCustomObject]@{ File = $destFile Action = '' + RenamedFrom = $null PreservedMarkers = @() NewMarkers = @() RemovedMarkers = @() } + # Resolve the EXISTING destination representation to merge FROM. + # (a) canonical filename, (b) ID match, (c) alias filename. + # $existingDestFile is the READ source; $destFile stays the WRITE + # target. When they differ, this is a rename. + $existingDestFile = $null + $renamedFrom = $null + if (Test-Path -LiteralPath $destFile) { + $existingDestFile = $destFile + # If a canonical file exists AND a legacy alias also lingers, the + # alias is a stale orphan from a prior layout. Flag it (do not + # touch it - the canonical file is authoritative). + foreach ($mEntry in @($manifest | Where-Object { $_.Id -eq $srcId })) { + foreach ($aliasName in $mEntry.Aliases) { + $aliasPath = Join-Path -Path $destResolved -ChildPath $aliasName + if ((Test-Path -LiteralPath $aliasPath) -and ($aliasPath -ne $destFile)) { + Write-Log -Message " Note : '$aliasName' is a stale orphan (canonical '$($srcFile.Name)' already present). Safe to delete." -Level Warning + } + } + } + } + elseif ($destIdMap.ContainsKey($srcId)) { + $existingDestFile = $destIdMap[$srcId] + $renamedFrom = Split-Path -Leaf $existingDestFile + } + else { + foreach ($mEntry in @($manifest | Where-Object { $_.Id -eq $srcId })) { + foreach ($aliasName in $mEntry.Aliases) { + $aliasPath = Join-Path -Path $destResolved -ChildPath $aliasName + if (Test-Path -LiteralPath $aliasPath) { + $existingDestFile = $aliasPath + $renamedFrom = $aliasName + break + } + } + if ($existingDestFile) { break } + } + } + + if ($renamedFrom) { + $row.RenamedFrom = $renamedFrom + Write-Log -Message " Rename : '$renamedFrom' -> '$($srcFile.Name)' (matched by pipeline ID '$srcId'). Marker bodies (e.g. schedule CRONs) will be carried over." -Level Warning + Write-Log -Message " Platform note: renaming a workflow/pipeline file resets its run history grouping and can break branch-protection required status checks (GitHub) or orphan the pipeline definition until you re-point it at the new YAML path (Azure DevOps). Re-point/re-register after this run." -Level Warning + } + # 3a. Net-new file: simple copy. ------------------------------- - if (-not (Test-Path -LiteralPath $destFile)) { + if (-not $existingDestFile) { if ($PSCmdlet.ShouldProcess($destFile, "Create new file from bundled sample")) { - $srcText = [System.IO.File]::ReadAllText($srcFile.FullName, [System.Text.UTF8Encoding]::new($false)) Write-Utf8NoBomFile -Path $destFile -Content $srcText $row.Action = 'Created' $srcMarkers = Get-AzLocalPipelineCustomiseMarkers -Text $srcText @@ -232,9 +341,9 @@ function Update-AzLocalPipelineExample { continue } - # 3b. File exists. Read both ---------------------------------- - $srcText = [System.IO.File]::ReadAllText($srcFile.FullName, [System.Text.UTF8Encoding]::new($false)) - $destText = [System.IO.File]::ReadAllText($destFile, [System.Text.UTF8Encoding]::new($false)) + # 3b. File exists. Read the existing (possibly aliased) destination + # representation. $srcText was already read above. + $destText = [System.IO.File]::ReadAllText($existingDestFile, [System.Text.UTF8Encoding]::new($false)) $srcMarkers = Get-AzLocalPipelineCustomiseMarkers -Text $srcText $destMarkers = Get-AzLocalPipelineCustomiseMarkers -Text $destText @@ -245,7 +354,18 @@ function Update-AzLocalPipelineExample { # 3c. Both files marker-free -> straight diff/overwrite path. - if (-not $hasSrcMarkers -and -not $hasDestMarkers) { if ($srcText -eq $destText) { - $row.Action = 'Unchanged' + if ($renamedFrom) { + # Content identical but the file is at a legacy name -> rename to canonical. + if ($PSCmdlet.ShouldProcess($destFile, "Rename '$renamedFrom' -> '$($srcFile.Name)' (content identical)")) { + Write-Utf8NoBomFile -Path $destFile -Content $srcText + if (($existingDestFile -ne $destFile) -and (Test-Path -LiteralPath $existingDestFile)) { Remove-Item -LiteralPath $existingDestFile -Force } + Write-Log -Message " Renamed : '$renamedFrom' -> '$($srcFile.Name)' (content identical)" -Level Success + } + $row.Action = 'Renamed' + } + else { + $row.Action = 'Unchanged' + } [void]$results.Add($row) continue } @@ -271,6 +391,7 @@ function Update-AzLocalPipelineExample { $destPinValue = if ($destPinMatch.Success) { ([regex]::Match($destPinMatch.Value, "'([^']+)'")).Groups[1].Value } else { '?' } if ($PSCmdlet.ShouldProcess($destFile, "Bump GENERATED_AGAINST_MODULE_VERSION from '$destPinValue' to '$srcPinValue' (pin-only diff)")) { Write-Utf8NoBomFile -Path $destFile -Content $srcText + if ($renamedFrom -and ($existingDestFile -ne $destFile) -and (Test-Path -LiteralPath $existingDestFile)) { Remove-Item -LiteralPath $existingDestFile -Force } Write-Log -Message " Updated : $($srcFile.Name), pin-only ('$destPinValue' -> '$srcPinValue')" -Level Success } $row.Action = 'Updated-PinOnly' @@ -285,6 +406,7 @@ function Update-AzLocalPipelineExample { } if ($PSCmdlet.ShouldProcess($destFile, "Overwrite (no markers, -Force supplied)")) { Write-Utf8NoBomFile -Path $destFile -Content $srcText + if ($renamedFrom -and ($existingDestFile -ne $destFile) -and (Test-Path -LiteralPath $existingDestFile)) { Remove-Item -LiteralPath $existingDestFile -Force } Write-Log -Message " Overwritten (forced): $($srcFile.Name)" -Level Warning } $row.Action = 'Overwritten' @@ -304,6 +426,7 @@ function Update-AzLocalPipelineExample { } if ($PSCmdlet.ShouldProcess($destFile, "First-migration overwrite (destination has no markers, -Force supplied)")) { Write-Utf8NoBomFile -Path $destFile -Content $srcText + if ($renamedFrom -and ($existingDestFile -ne $destFile) -and (Test-Path -LiteralPath $existingDestFile)) { Remove-Item -LiteralPath $existingDestFile -Force } Write-Log -Message " Overwritten (first migration): $($srcFile.Name) - re-apply any customisations now." -Level Warning } $row.Action = 'Overwritten' @@ -325,6 +448,7 @@ function Update-AzLocalPipelineExample { } if ($PSCmdlet.ShouldProcess($destFile, "Overwrite (markers removed by upgrade, -Force supplied)")) { Write-Utf8NoBomFile -Path $destFile -Content $srcText + if ($renamedFrom -and ($existingDestFile -ne $destFile) -and (Test-Path -LiteralPath $existingDestFile)) { Remove-Item -LiteralPath $existingDestFile -Force } Write-Log -Message " Overwritten (markers removed): $($srcFile.Name) - destination marker bodies discarded." -Level Warning } $row.Action = 'Overwritten' @@ -363,17 +487,31 @@ function Update-AzLocalPipelineExample { $row.RemovedMarkers = @($destMarkers.Keys | Where-Object { -not $srcMarkers.ContainsKey($_) }) if ($merged -eq $destText) { - $row.Action = 'Unchanged' + if ($renamedFrom) { + # Merged content identical to the destination but the file is + # at a legacy name -> rename to canonical (carries markers). + if ($PSCmdlet.ShouldProcess($destFile, "Rename '$renamedFrom' -> '$($srcFile.Name)' (merged content identical)")) { + Write-Utf8NoBomFile -Path $destFile -Content $merged + if (($existingDestFile -ne $destFile) -and (Test-Path -LiteralPath $existingDestFile)) { Remove-Item -LiteralPath $existingDestFile -Force } + Write-Log -Message " Renamed : '$renamedFrom' -> '$($srcFile.Name)' (merged content identical, markers preserved)" -Level Success + } + $row.Action = 'Renamed' + } + else { + $row.Action = 'Unchanged' + } [void]$results.Add($row) continue } if ($PSCmdlet.ShouldProcess($destFile, "Update YAML (preserve $($preserved.Count) marker block(s))")) { Write-Utf8NoBomFile -Path $destFile -Content $merged + if ($renamedFrom -and ($existingDestFile -ne $destFile) -and (Test-Path -LiteralPath $existingDestFile)) { Remove-Item -LiteralPath $existingDestFile -Force } $kept = if ($preserved.Count -gt 0) { ", preserved=$([string]::Join(',', $preserved))" } else { '' } $added = if ($row.NewMarkers.Count -gt 0) { ", new=$([string]::Join(',', $row.NewMarkers))" } else { '' } $removed = if ($row.RemovedMarkers.Count -gt 0) { ", removed=$([string]::Join(',', $row.RemovedMarkers))" } else { '' } - Write-Log -Message " Updated : $($srcFile.Name)${kept}${added}${removed}" -Level Success + $renamed = if ($renamedFrom) { ", renamedFrom=$renamedFrom" } else { '' } + Write-Log -Message " Updated : $($srcFile.Name)${kept}${added}${removed}${renamed}" -Level Success } $row.Action = 'Updated' [void]$results.Add($row) diff --git a/AzLocal.UpdateManagement/Public/Update-AzLocalSideloadCatalog.ps1 b/AzLocal.UpdateManagement/Public/Update-AzLocalSideloadCatalog.ps1 new file mode 100644 index 00000000..2c85170a --- /dev/null +++ b/AzLocal.UpdateManagement/Public/Update-AzLocalSideloadCatalog.ps1 @@ -0,0 +1,357 @@ +function Update-AzLocalSideloadCatalog { + <# + .SYNOPSIS + Refreshes or scaffolds the sideload catalog YAML from the Microsoft Learn + 'import and discover updates offline' download table. + + .DESCRIPTION + Operator-run helper for the v0.8.7 on-prem sideloading automation. Fetches + the published Microsoft Learn page that lists the Azure Local + CombinedSolutionBundle releases, parses each table row, and merges the + discovered Microsoft 'Solution' packages into the catalog YAML for the + operator to REVIEW and commit. + + Parsing contract (per table row): + - The version anchor TEXT is the solution version (e.g. 12.2605.1003.210). + - The version anchor HREF is the direct download URI. + - The notes column carries the SHA256 (64 hex chars) and an + 'Availability date: YYYY-MM-DD' string. + - The OS build column (e.g. 26100.4061) is captured when present. + + Rules: + - Only 'Solution' (Microsoft) packages are touched. Existing 'SBE' + (OEM) entries and any operator-authored fields are PRESERVED verbatim. + - A discovered version that already exists as a Solution entry is + updated in place (download URI / SHA256 / OS build / availability + date); new versions are appended. + - A discovered row with NO resolvable download URI is still written but + FLAGGED (a warning is emitted and a '# TODO: fill downloadUri' + comment is added) so the operator can complete it manually. + - The runtime copy path never calls this function; it reads only the + committed catalog. + + The catalog file is written WITHOUT a BOM. This function honours + -WhatIf / -Confirm and does not overwrite the file under -WhatIf. + + SCHEMA MIGRATION (-SchemaMigrate): + Passing -SchemaMigrate switches the cmdlet from content refresh to a + schema-version migration of an EXISTING catalog file (no web fetch). + It mirrors Update-AzLocalApplyUpdatesScheduleConfig -SchemaMigrate: + the per-hop recipes in Private/Convert-AzLocalSideloadCatalogSchemaVersion.ps1 + are walked from the file's current schemaVersion up to the module's + current version, operator comments + SBE (OEM) entries + row order are + preserved verbatim, the original is backed up as .v.old.yml, + and a structured result object is returned. At schema v1 the recipe + table is empty, so this is a no-op until the first format change ships. + + .PARAMETER Path + Path to the catalog YAML to create or update. + + .PARAMETER SourceUri + The Microsoft Learn page URL to parse. Defaults to the documented + 'import and discover updates offline' article. + + .PARAMETER Html + Optional raw HTML to parse instead of fetching SourceUri (used for + offline / air-gapped refresh and for unit testing). + + .PARAMETER SchemaMigrate + Migrate an existing catalog file to the module's current schema + version instead of refreshing its content. Non-destructive: backs the + original up as .v.old.yml and returns a result object + ({ Action, Path, FromVersion, ToVersion, BackupPath, Hops[] }). + + .OUTPUTS + Content refresh (default): [PSCustomObject[]] the merged package + entries that were written. + -SchemaMigrate: a single [PSCustomObject] describing the migration + (Action = 'Migrated' | 'Unchanged-SchemaCurrent' | 'WhatIf'). + #> + [CmdletBinding(SupportsShouldProcess = $true, DefaultParameterSetName = 'ContentRefresh')] + [OutputType([PSCustomObject[]])] + param( + [Parameter(Mandatory = $true)] + [ValidateNotNullOrEmpty()] + [string]$Path, + + [Parameter(Mandatory = $false, ParameterSetName = 'ContentRefresh')] + [ValidateNotNullOrEmpty()] + [string]$SourceUri = 'https://learn.microsoft.com/en-us/azure/azure-local/manage/import-discover-updates-offline-23h2', + + [Parameter(Mandatory = $false, ParameterSetName = 'ContentRefresh')] + [string]$Html, + + [Parameter(Mandatory = $true, ParameterSetName = 'SchemaMigrate')] + [switch]$SchemaMigrate + ) + + # ---- SchemaMigrate mode: text-surgery schema migration ------------- + # Bypasses the web fetch entirely. Mirrors the schedule-file migrator + # (Update-AzLocalApplyUpdatesScheduleConfig -SchemaMigrate). + if ($PSCmdlet.ParameterSetName -eq 'SchemaMigrate') { + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Update-AzLocalSideloadCatalog: catalog file not found: '$Path'. Create a starter with Copy-AzLocalPipelineExample, or refresh content with Update-AzLocalSideloadCatalog -Path ." + } + $full = (Resolve-Path -LiteralPath $Path).Path + $text = Get-Content -LiteralPath $full -Raw -ErrorAction Stop + + Write-Log -Message "Update-AzLocalSideloadCatalog: computing schema migration for '$full' to version $($script:SideloadCatalogSchemaCurrentVersion)..." -Level Info + $result = Convert-AzLocalSideloadCatalogSchemaVersion -Text $text -TargetSchemaVersion $script:SideloadCatalogSchemaCurrentVersion -SourcePath $full + + if (-not $result.Migrated) { + Write-Log -Message "Catalog file is already on schemaVersion=$($result.ToVersion). No changes required." -Level Info + return [pscustomobject]@{ + Action = 'Unchanged-SchemaCurrent' + Path = $full + FromVersion = $result.FromVersion + ToVersion = $result.ToVersion + BackupPath = $null + Hops = @() + } + } + + # Backup naming mirrors the schedule migrator: + # .v.old.yml in the same directory. + $dir = [System.IO.Path]::GetDirectoryName($full) + $base = [System.IO.Path]::GetFileNameWithoutExtension($full) + $backupName = "$base.v$($result.FromVersion).old.yml" + $backupPath = if ($dir) { Join-Path $dir $backupName } else { $backupName } + + if ((Test-Path -LiteralPath $backupPath) -and -not $WhatIfPreference) { + throw "Update-AzLocalSideloadCatalog: backup target '$backupPath' already exists. A previous migration from version $($result.FromVersion) was not cleaned up. Review/commit/delete it, then re-run." + } + + $changeSummary = ($result.Hops | ForEach-Object { "v$($_.FromVersion)->v$($_.ToVersion): $(($_.Changes -join '; '))" }) -join ' | ' + $shouldMsg = "Migrate schemaVersion $($result.FromVersion) -> $($result.ToVersion). Backup '$([IO.Path]::GetFileName($full))' as '$backupName'. Changes: $changeSummary" + if (-not $PSCmdlet.ShouldProcess($full, $shouldMsg)) { + Write-Log -Message "WhatIf/Confirm declined: catalog file NOT modified. Computed migration was: $shouldMsg" -Level Info + return [pscustomobject]@{ + Action = 'WhatIf' + Path = $full + FromVersion = $result.FromVersion + ToVersion = $result.ToVersion + BackupPath = $backupPath + Hops = $result.Hops + } + } + + # Rename original first so we never have two valid copies at the + # canonical path; roll back the rename if the write fails. + Rename-Item -LiteralPath $full -NewName $backupName -ErrorAction Stop + Write-Log -Message "Renamed original to: $backupPath" -Level Info + try { + Write-Utf8NoBomFile -Path $full -Content $result.NewText + } + catch { + Write-Log -Message "Write of migrated content FAILED: $($_.Exception.Message). Rolling back the rename so the original is restored." -Level Error + try { Rename-Item -LiteralPath $backupPath -NewName ([System.IO.Path]::GetFileName($full)) -ErrorAction Stop } + catch { Write-Log -Message "ROLLBACK ALSO FAILED. Manual recovery needed: rename '$backupPath' back to '$full' by hand." -Level Error } + throw + } + + Write-Log -Message "Migrated $full to schemaVersion=$($result.ToVersion)." -Level Success + foreach ($hop in $result.Hops) { + Write-Log -Message " v$($hop.FromVersion) -> v$($hop.ToVersion):" -Level Info + foreach ($c in $hop.Changes) { Write-Log -Message " + $c" -Level Info } + } + Write-Log -Message "Review the migration with: git diff -- ""$([IO.Path]::GetFileName($full))""" -Level Info + Write-Log -Message "Once you have committed the new file, the backup '$backupName' can be removed." -Level Info + return [pscustomobject]@{ + Action = 'Migrated' + Path = $full + FromVersion = $result.FromVersion + ToVersion = $result.ToVersion + BackupPath = $backupPath + Hops = $result.Hops + } + } + + # ---- Acquire HTML -------------------------------------------------- + if ([string]::IsNullOrWhiteSpace($Html)) { + Write-Log -Message "Update-AzLocalSideloadCatalog: fetching catalog source '$SourceUri'." -Level Info + try { + $response = Invoke-WebRequest -Uri $SourceUri -UseBasicParsing -ErrorAction Stop + $Html = [string]$response.Content + } + catch { + throw "Failed to fetch sideload catalog source '$SourceUri': $($_.Exception.Message)" + } + } + + $discovered = Get-AzLocalSideloadCatalogRowFromHtml -Html $Html + + if ($discovered.Count -eq 0) { + Write-Log -Message "Update-AzLocalSideloadCatalog: no solution-version rows were parsed from the source. The page layout may have changed; review the catalog manually." -Level Warning + } + + # ---- Load existing catalog (preserve SBE + manual entries) --------- + $existing = @() + if (Test-Path -LiteralPath $Path -PathType Leaf) { + $existing = @(Get-AzLocalSideloadCatalog -Path $Path) + } + + $merged = New-Object System.Collections.Generic.List[PSCustomObject] + $solutionByVersion = @{} + foreach ($entry in $existing) { + $merged.Add($entry) + if ($entry.PackageType -eq 'Solution') { + $solutionByVersion[$entry.Version] = $entry + } + } + + foreach ($row in $discovered) { + if ([string]::IsNullOrWhiteSpace($row.DownloadUri)) { + Write-Log -Message "Update-AzLocalSideloadCatalog: discovered version '$($row.Version)' has no resolvable download URI - written with a TODO flag for manual completion." -Level Warning + } + if ($solutionByVersion.ContainsKey($row.Version)) { + $target = $solutionByVersion[$row.Version] + $target.BuildNumber = $row.BuildNumber + $target.OsBuild = $row.OsBuild + $target.DownloadUri = $row.DownloadUri + $target.Sha256 = $row.Sha256 + $target.AvailabilityDate = $row.AvailabilityDate + } + else { + $newEntry = [PSCustomObject]@{ + Version = $row.Version + PackageType = 'Solution' + BuildNumber = $row.BuildNumber + OsBuild = $row.OsBuild + DownloadUri = $row.DownloadUri + Sha256 = $row.Sha256 + AvailabilityDate = $row.AvailabilityDate + LocalPath = '' + SourceFolder = '' + Notes = '' + } + $merged.Add($newEntry) + $solutionByVersion[$row.Version] = $newEntry + } + } + + $yaml = ConvertTo-AzLocalSideloadCatalogYaml -Packages $merged.ToArray() -SourceUri $SourceUri + + if ($PSCmdlet.ShouldProcess($Path, "Write sideload catalog with $($merged.Count) package(s)")) { + $parent = Split-Path -Parent $Path + if ($parent -and -not (Test-Path -LiteralPath $parent)) { + New-Item -ItemType Directory -Path $parent -Force | Out-Null + } + Write-Utf8NoBomFile -Path $Path -Content $yaml + Write-Log -Message "Update-AzLocalSideloadCatalog: wrote $($merged.Count) package(s) to '$Path'. Review and commit." -Level Success + } + + return $merged.ToArray() +} + +function Get-AzLocalSideloadCatalogRowFromHtml { + <# + .SYNOPSIS + Parses CombinedSolutionBundle rows out of the Microsoft Learn offline-update + table HTML. Isolated so it can be unit-tested without a network call. + .OUTPUTS + [PSCustomObject[]] with Version, BuildNumber, OsBuild, DownloadUri, + Sha256, AvailabilityDate. + #> + [CmdletBinding()] + [OutputType([PSCustomObject[]])] + param( + [Parameter(Mandatory = $true)] + [AllowEmptyString()] + [string]$Html + ) + + $rows = New-Object System.Collections.Generic.List[PSCustomObject] + if ([string]::IsNullOrWhiteSpace($Html)) { return $rows.ToArray() } + + # Find every CombinedSolutionBundle download anchor. The href carries the + # version + build inside the path/filename; the anchor text is the version. + $anchorPattern = '(?is)]*?href\s*=\s*"(?[^"]*CombinedSolutionBundle[^"]*\.zip)"[^>]*>(?.*?)' + $anchorMatches = [regex]::Matches($Html, $anchorPattern) + + $seen = New-Object System.Collections.Generic.HashSet[string] ([System.StringComparer]::OrdinalIgnoreCase) + foreach ($m in $anchorMatches) { + $href = $m.Groups['href'].Value.Trim() + $text = ([regex]::Replace($m.Groups['text'].Value, '<[^>]+>', '')).Trim() + + # Prefer an explicit version in the anchor text; otherwise derive it + # from the filename CombinedSolutionBundle..zip. + $version = $null + if ($text -match '\d+\.\d+\.\d+\.\d+') { $version = $Matches[0] } + elseif ($href -match 'CombinedSolutionBundle\.(?\d+\.\d+\.\d+\.\d+)\.zip') { $version = $Matches['v'] } + if (-not $version) { continue } + if (-not $seen.Add($version)) { continue } + + # Look at a window of HTML following the anchor for the SHA256, the + # availability date, and the OS build that accompany this row. + $tail = $Html.Substring($m.Index, [Math]::Min(2000, $Html.Length - $m.Index)) + $sha256 = if ($tail -match '\b([0-9A-Fa-f]{64})\b') { $Matches[1].ToUpperInvariant() } else { '' } + $availabilityDate = if ($tail -match '(?i)Availability date[^0-9]*([0-9]{4}-[0-9]{2}-[0-9]{2})') { $Matches[1] } else { '' } + $osBuild = if ($tail -match '\b(26\d{3}\.\d+)\b') { $Matches[1] } else { '' } + + $rows.Add([PSCustomObject]@{ + Version = $version + BuildNumber = $version + OsBuild = $osBuild + DownloadUri = $href + Sha256 = $sha256 + AvailabilityDate = $availabilityDate + }) + } + + return $rows.ToArray() +} + +function ConvertTo-AzLocalSideloadCatalogYaml { + <# + .SYNOPSIS + Serialises sideload catalog package entries to the narrow YAML shape + consumed by Get-AzLocalSideloadCatalog. + .OUTPUTS + [string] the YAML document. + #> + [CmdletBinding()] + [OutputType([string])] + param( + [Parameter(Mandatory = $true)] + [AllowEmptyCollection()] + [PSCustomObject[]]$Packages, + + [Parameter(Mandatory = $false)] + [string]$SourceUri = '' + ) + + $sb = New-Object System.Text.StringBuilder + [void]$sb.AppendLine('# Sideload catalog - generated/refreshed by Update-AzLocalSideloadCatalog.') + [void]$sb.AppendLine('# Review and commit. SBE (OEM) entries are operator-maintained and preserved.') + if (-not [string]::IsNullOrWhiteSpace($SourceUri)) { + [void]$sb.AppendLine(('# Source: {0}' -f $SourceUri)) + } + [void]$sb.AppendLine(('schemaVersion: {0}' -f $script:SideloadCatalogSchemaCurrentVersion)) + [void]$sb.AppendLine('packages:') + + foreach ($pkg in $Packages) { + $packageType = if ($pkg.PackageType) { $pkg.PackageType } else { 'Solution' } + [void]$sb.AppendLine((" - version: '{0}'" -f $pkg.Version)) + [void]$sb.AppendLine((" packageType: {0}" -f $packageType)) + if ($pkg.BuildNumber) { [void]$sb.AppendLine((" buildNumber: '{0}'" -f $pkg.BuildNumber)) } + if ($pkg.OsBuild) { [void]$sb.AppendLine((" osBuild: '{0}'" -f $pkg.OsBuild)) } + if ($packageType -eq 'Solution') { + if ([string]::IsNullOrWhiteSpace([string]$pkg.DownloadUri)) { + [void]$sb.AppendLine(" # TODO: fill downloadUri - not resolved automatically") + [void]$sb.AppendLine(" downloadUri: ''") + } + else { + [void]$sb.AppendLine((" downloadUri: '{0}'" -f $pkg.DownloadUri)) + } + if ($pkg.LocalPath) { [void]$sb.AppendLine((" localPath: '{0}'" -f $pkg.LocalPath)) } + } + else { + [void]$sb.AppendLine((" sourceFolder: '{0}'" -f $pkg.SourceFolder)) + } + [void]$sb.AppendLine((" sha256: '{0}'" -f $pkg.Sha256)) + if ($pkg.AvailabilityDate) { [void]$sb.AppendLine((" availabilityDate: '{0}'" -f $pkg.AvailabilityDate)) } + if ($pkg.Notes) { [void]$sb.AppendLine((" notes: '{0}'" -f $pkg.Notes)) } + } + + return $sb.ToString() +} diff --git a/AzLocal.UpdateManagement/README.md b/AzLocal.UpdateManagement/README.md index 0c17913d..fb3ec98e 100644 --- a/AzLocal.UpdateManagement/README.md +++ b/AzLocal.UpdateManagement/README.md @@ -2,7 +2,7 @@ > ⚠️ **Disclaimer**: This module is **NOT** a Microsoft supported service offering or product. It is provided as example code only, with no warranty or official support. Refer to the [MIT license](https://github.com/NeilBird/Azure-Local/blob/main/LICENSE) for further information. -**Latest Version:** v0.8.6 - [Published in PowerShell Gallery](https://www.powershellgallery.com/packages/AzLocal.UpdateManagement/0.8.6) +**Latest Version:** v0.8.7 - [Published in PowerShell Gallery](https://www.powershellgallery.com/packages/AzLocal.UpdateManagement/0.8.7) > 📢 **Renamed in v0.7.3**: this module was previously published as `AzStackHci.ManageUpdates`. The new module name aligns with the Azure Local product name (_Microsoft retired the *Azure Stack HCI* brand in late 2024_). The module GUID is preserved across the rename. If you have the old name installed, run: > @@ -23,7 +23,7 @@ Azure Local REST API specification (includes update management endpoints): https **This README (overview + most-recent release notes):** - [Where to Start](#where-to-start) -- [What's New in v0.8.6](#whats-new-in-v086) +- [What's New in v0.8.7](#whats-new-in-v087) - [What's New in v0.8.4](#whats-new-in-v084) - [Files](#files) - [Prerequisites](#prerequisites) @@ -69,7 +69,7 @@ If you are new to this module, work through these in order from a regular PowerS | 5 | Apply the update | [`Start-AzLocalClusterUpdate`](docs/cmdlet-reference.md#start-azlocalclusterupdate) (single cluster or `-ScopeByUpdateRingTag` for a wave) | | 6 | Monitor and report | [`Get-AzLocalUpdateRuns`](docs/cmdlet-reference.md#get-azlocalupdateruns), [`Get-AzLocalFleetProgress`](docs/cmdlet-reference.md#get-azlocalfleetprogress), [`New-AzLocalFleetStatusHtmlReport`](docs/cmdlet-reference.md#new-azlocalfleetstatushtmlreport) | -> **For CI/CD?** Skip this table and go straight to [Automation-Pipeline-Examples/README.md](./Automation-Pipeline-Examples/README.md) - it covers OIDC / Managed Identity / Service Principal setup, federated credentials, nine GitHub Actions workflows, and nine Azure DevOps pipelines (the v0.7.90 set includes `Step.7_monitor-updates`, `Step.8_fleet-update-status`, and `Step.9_fleet-health-status`). +> **For CI/CD?** Skip this table and go straight to [Automation-Pipeline-Examples/README.md](./Automation-Pipeline-Examples/README.md) - it covers OIDC / Managed Identity / Service Principal setup, federated credentials, eleven GitHub Actions workflows, and eleven Azure DevOps pipelines (including the on-prem `sideload-updates` pipeline plus `apply-updates`, `monitor-updates`, `fleet-update-status`, and `fleet-health-status`). Pipeline files are no longer prefixed with `Step.N_` - the in-pipeline display names still carry the `Step.N` ordering, and `Update-AzLocalPipelineExample` migrates any older `Step.N_*.yml` files to the new names automatically while preserving your customizations. ### Common workflows (function-invocation order) @@ -79,7 +79,7 @@ If you are new to this module, work through these in order from a regular PowerS | **Staged wave deployment** | `Get-AzLocalClusterInventory` -> `Set-AzLocalClusterUpdateRingTag` -> `Get-AzLocalClusterUpdateReadiness -ScopeByUpdateRingTag` -> `Start-AzLocalClusterUpdate -ScopeByUpdateRingTag` -> `Get-AzLocalFleetProgress` -> `New-AzLocalFleetStatusHtmlReport` | | **Daily fleet status report** | `Get-AzLocalFleetStatusData -AllClusters -IncludeUpdateRuns -IncludeHealthDetails -ExportPath ...` -> `New-AzLocalFleetStatusHtmlReport -StatusData $data -OutputPath ...` | | **Daily fleet health audit (v0.7.65)** | `Get-AzLocalFleetHealthFailures -View Summary -ExportPath fleet-health-summary.csv` -> review top failure reasons by cluster impact -> drill into [`Get-AzLocalFleetHealthFailures -View Detail`](docs/cmdlet-reference.md#get-azlocalfleethealthfailures) for per-cluster remediation | -| **Schedule coverage drift audit (v0.7.65)** | `Test-AzLocalApplyUpdatesScheduleCoverage -View Audit -PipelineYamlPath .\.github\workflows` -> for any `Uncovered` rows, copy the `RequiredCronUTC` value and paste it into `Step.6_apply-updates.yml` -> re-run `-View Audit` to confirm `Covered` -> wire the bundled `Step.3_apply-updates-schedule-audit.yml` pipeline (weekly Mon 05:00 UTC) so future tag drift is caught automatically. Full runbook: [`Automation-Pipeline-Examples/README.md` section 8.3](./Automation-Pipeline-Examples/README.md#83-end-to-end-runbook-apply-updates-schedule-coverage-audit) | +| **Schedule coverage drift audit (v0.7.65)** | `Test-AzLocalApplyUpdatesScheduleCoverage -View Audit -PipelineYamlPath .\.github\workflows` -> for any `Uncovered` rows, copy the `RequiredCronUTC` value and paste it into `apply-updates.yml` -> re-run `-View Audit` to confirm `Covered` -> wire the bundled `apply-updates-schedule-audit.yml` pipeline (weekly Mon 05:00 UTC) so future tag drift is caught automatically. Full runbook: [`Automation-Pipeline-Examples/README.md` section 8.3](./Automation-Pipeline-Examples/README.md#83-end-to-end-runbook-apply-updates-schedule-coverage-audit) | | **Pre-update health gate (CI/CD)** | `Test-AzLocalClusterHealth -BlockingOnly` -> `Test-AzLocalUpdateScheduleAllowed` -> `Test-AzLocalFleetHealthGate` -> proceed only on pass | | **Sideloaded payload (v0.7.1)** | Operator sets `UpdateSideloaded=False` -> stage payload out-of-band -> operator flips `UpdateSideloaded=True` -> `Start-AzLocalClusterUpdate` (auto-stamps `UpdateVersionInProgress`) -> `Get-AzLocalUpdateRuns` (auto-resets tags on success) -> `Reset-AzLocalSideloadedTag -Force` only if a tag gets stuck | | **Pause / resume long fleet run** | `Stop-AzLocalFleetUpdate -SaveState` -> ... -> `Resume-AzLocalFleetUpdate -StateFilePath ...` | @@ -87,21 +87,21 @@ If you are new to this module, work through these in order from a regular PowerS > Most CI/CD pipelines in [Automation-Pipeline-Examples/](Automation-Pipeline-Examples/) are direct implementations of one of these workflows. Start there if you want a copy-pasteable end-to-end pipeline. -## What's New in v0.8.6 +## What's New in v0.8.7 -**Step.3 cycle calendar enrichment: per-day Step.6 CRON firing times + per-(ring, date) `UpdateStartWindow` tag-coverage check (>=95% threshold).** Adds two new optional render-time columns to `Get-AzLocalApplyUpdatesScheduleCycleCalendar` and auto-wires them from `Export-AzLocalApplyUpdatesScheduleAudit` so operators can see, in one table: (a) which Step.6 cron firing times will fire on each calendar day, and (b) what fraction of clusters in the ring(s) eligible on that day have an `UpdateStartWindow` tag that actually covers at least one of those firings. **No public API removed; no existing parameter changed; no behavioural change on callers that do NOT supply the new dictionaries.** Same module export count as v0.8.5 (55). +**On-prem solution-update sideloading automation: new self-hosted Step.6 pipeline + 5 new Public cmdlets + de-numbered pipeline filenames.** Adds an opt-in, off-by-default workflow for Azure Local clusters that cannot pull solution updates from Azure directly. The new Step.6 pipeline (`sideload-updates.yml`) robocopies update media to each cluster's import share, verifies the SHA256 over WinRM, runs `Add-SolutionUpdate`, and flips the `UpdateSideloaded=True` tag so the downstream apply (now Step.7) picks it up. The multi-hour copy runs in a detached Windows Scheduled Task and the pipeline is driven as a re-entrant state machine on a frequent CRON, so no individual run is long-lived. **Module export count grows 55 -> 60.** -1. **New `Get-AzLocalApplyUpdatesScheduleCycleCalendar -CronFiringsByDate`** (`[hashtable]`, keys = `yyyy-MM-dd` UTC, values = `[string[]]` of `HH:mm` UTC firing times). When supplied, the per-day markdown table gains a centered `Ring CRON Start Time (UTC)
(Step 6 pipeline)` column between `Date (UTC)` and `Day`. Cell rendering: 0 firings -> `_(none)_`; 1-2 firings -> comma-joined (e.g. `02:00, 22:00`); 3+ firings -> first 2 + `" (+N)"` (e.g. `02:00, 10:00 (+2)`); dead day (`IsDeadDay`) -> `_(none - dead day)_`; missing date key -> blank cell. -2. **New `Get-AzLocalApplyUpdatesScheduleCycleCalendar -WindowMatchByRingAndDate`** (`[hashtable[string,hashtable]]` keyed first by ring name then by `yyyy-MM-dd` UTC, leaf = `@{ Matching=; Total= }`). When supplied, the per-day table gains a `Tag Start Window Match (>=95%)` column AFTER `Eligible rings`. Each cell lists one line per eligible ring: `` `Ring`: True/False mat/tot (pct%) `` (True iff `Matching/Total >= 0.95`). `_(n/a)_` when a ring has no entry for the date; `_(0 clusters)_` when `Total=0`; `_(n/a - dead day)_` on dead days. -3. **Pure render-time contract preserved.** `Get-AzLocalApplyUpdatesScheduleCycleCalendar` still does zero Azure / file I/O. Both columns are opt-in; omit either or both and the v0.8.5 column layout is bit-identical. -4. **`Export-AzLocalApplyUpdatesScheduleAudit` auto-wires both columns.** Cron firings are derived by parsing `Step.6_apply-updates*.yml` from `-PipelineYamlPath` (uses existing `Read-AzLocalApplyUpdatesYamlCrons` + `ConvertFrom-AzLocalCronExpression`; invalid/complex crons degrade gracefully to no firings for that day). The window-match dict is derived when `-ClusterCsvPath` is supplied, by parsing each cluster's `UpdateStartWindow` tag (via `ConvertFrom-AzLocalUpdateWindow`) against each day's cron firings; overnight windows (e.g. `Mon-Sun_22:00-04:00`) match firings that fall in either the late-evening or early-morning portion correctly via a two-case `DayOfWeek` projection. -5. **95% threshold.** A ring on a given day is True only when `Matching / Total >= 0.95` exact. Below threshold -> False (operators see the actual `mat/tot (pct%)` numbers and can act). -6. **Failure mode is non-fatal.** Any error in the enrichment block degrades gracefully to a calendar without the new columns (matching v0.8.5 behaviour); a `Write-Warning` surfaces the cause but the Step.3 summary continues to render. -7. **Pester suite updates**: drift-sync test bumped to `'0.8.6'`. 12 new It blocks under `Describe 'v0.8.6 Apply-Updates Schedule: Get-AzLocalApplyUpdatesScheduleCycleCalendar - CronFiringsByDate and WindowMatchByRingAndDate columns (markdown)'` cover: backwards compat, single-param paths, both-params path with 7-column header, `(+N)` suffix at 3+ firings, `_(none)_` cell, 95% threshold boundary (95/100 -> True, 94/100 -> False), `_(0 clusters)_` rendering, `_(n/a)_` rendering, case-insensitivity on ring + date keys, and three-optional-columns coexistence with `-ClusterRingCounts`. 2 new Export-* spy tests verify the auto-wire path. +1. **5 new Public cmdlets**: `Update-AzLocalSideloadCatalog`, `Resolve-AzLocalSideloadPlan`, `Invoke-AzLocalSideloadUpdate`, `Export-AzLocalSideloadStatusReport`, `Add-AzLocalSideloadStepSummary`. +2. **New Step.6 sideload pipeline** (`sideload-updates.yml`, GitHub Actions + Azure DevOps), opt-in via the `SIDELOAD_UPDATES` repository variable. Targets a self-hosted runner (`runs-on: [self-hosted, azlocal-sideload]`) on GitHub / a self-hosted agent pool (`pool: { name, demands: azlocal-sideload }`) on Azure DevOps. Manual dispatch plus a `*/30` CRON poll advances the state machine (Planned -> Copying -> Copied -> Verified -> Imported -> SideloadFlagged). See [Automation-Pipeline-Examples/docs/sideload.md](Automation-Pipeline-Examples/docs/sideload.md) + [sideload-robocopy.md](Automation-Pipeline-Examples/docs/sideload-robocopy.md). +3. **BREAKING - pipeline filenames de-numbered.** The `Step.N_` filename prefix is removed (e.g. `Step.7_apply-updates.yml` -> `apply-updates.yml`); the in-pipeline `Step.N - ` display names are unchanged. `Update-AzLocalPipelineExample` is now rename-aware: it matches each destination pipeline by a stable logical id (embedded `# AZLOCAL-PIPELINE-ID:` marker, with legacy-filename aliases) and AUTO-RENAMES any older `Step.N_*.yml` to the new name while preserving your `BEGIN/END-AZLOCAL-CUSTOMIZE` CRON edits (emits a `RenamedFrom` result + a required-check warning). +4. **BREAKING - display-step renumber** to make room for sideload at Step.6: apply-updates 6 -> 7, monitor-updates 7 -> 8, fleet-update-status 8 -> 9, fleet-health-status 9 -> 10. +5. **Sideload-aware existing steps**: Step.1 inventory can emit the `UpdateAuthAccountId` column (`-IncludeSideloadColumns`, auto-enabled from `SIDELOAD_UPDATES`); Step.2 tag management can set `UpdateAuthAccountId` from CSV; Step.3 advisor can emit a recommended sideload CRON (apply window minus `SIDELOAD_LEAD_DAYS`). Byte-identical output when sideload is off. +6. **Fixed**: the in-flight monitor "Current Step" column always showed the top-level wrapper step ("Start update"). `Format-AzLocalUpdateRun` and `Get-AzLocalFleetStatusData` now walk to the deepest active step (`Get-DeepestActiveStep`), matching the standard update-progress output. +7. **BREAKING - all operator config relocated to a repo-root `config/` folder.** Every operator-authored config file - `ClusterUpdateRings.csv`, `apply-updates-schedule.yml`, `sideload-auth-map.csv`, `sideload-catalog.yml` - now lives in a single `config/` folder at the repo root, with the **identical path on GitHub Actions and Azure DevOps**. The starter `apply-updates-schedule.yml` moves here from its v0.7.92 location (`.github/` / repo root), and every pipeline default (`APPLY_UPDATES_SCHEDULE_PATH`, `SIDELOAD_AUTH_MAP_PATH`, `SIDELOAD_CATALOG_PATH`, ring-CSV paths) now points at `config/`. `Copy-AzLocalPipelineExample` seeds these starters into `config/` (never overwriting); `Update-AzLocalPipelineExample` refreshes only the pipeline YAMLs and leaves `config/` untouched. -`GENERATED_AGAINST_MODULE_VERSION` bumped from `0.8.5` to `0.8.6` across all 20 bundled `Step.{0..9}.yml` templates. +`GENERATED_AGAINST_MODULE_VERSION` bumped from `0.8.6` to `0.8.7` across all bundled pipeline templates. -See [CHANGELOG.md](CHANGELOG.md#086---2026-06-10) for the full v0.8.6 entry. See [CHANGELOG.md](CHANGELOG.md#085---2026-06-09) for the v0.8.5 entry. +See [CHANGELOG.md](CHANGELOG.md#087---2026-06-11) for the full v0.8.7 entry. See [CHANGELOG.md](CHANGELOG.md#086---2026-06-10) for the v0.8.6 entry. ## Files @@ -221,7 +221,7 @@ Copy-AzLocalPipelineExample -Destination C:\repos\my-fleet -Platform GitHub The function prints a short "next steps" summary pointing at the copied README and the platform-specific YAML folder. See [`Automation-Pipeline-Examples/README.md`](Automation-Pipeline-Examples/README.md) for the full step-by-step setup guide. -> 🔄 **Refreshing pipelines after a module upgrade?** Use `Update-AzLocalPipelineExample` instead of `Copy-AzLocalPipelineExample`. It is a marker-aware merge that refreshes everything **outside** the `# BEGIN-AZLOCAL-CUSTOMIZE:` / `# END-AZLOCAL-CUSTOMIZE:` blocks in each YAML and **preserves** everything inside them - so your custom cron schedules (`schedule-triggers`) and ITSM secret bindings (`itsm-secrets` in Step.6) survive the upgrade. +> 🔄 **Refreshing pipelines after a module upgrade?** Use `Update-AzLocalPipelineExample` instead of `Copy-AzLocalPipelineExample`. It is a marker-aware merge that refreshes everything **outside** the `# BEGIN-AZLOCAL-CUSTOMIZE:` / `# END-AZLOCAL-CUSTOMIZE:` blocks in each YAML and **preserves** everything inside them - so your custom cron schedules (`schedule-triggers`) and ITSM secret bindings (`itsm-secrets` in Step.7) survive the upgrade. > > ```powershell > # Preview what would change @@ -531,7 +531,7 @@ New-AzLocalFleetStatusHtmlReport ` -IncludeHealthDetails -IncludeUpdateRuns ``` -> 💡 **CI/CD**: this same assess -> remediate -> apply flow is wired into the pipeline examples under `Automation-Pipeline-Examples/`: see the `Step.5_assess-update-readiness.yml` pipeline (report-only) and the `check-readiness` job inside `Step.6_apply-updates.yml`. +> 💡 **CI/CD**: this same assess -> remediate -> apply flow is wired into the pipeline examples under `Automation-Pipeline-Examples/`: see the `assess-update-readiness.yml` pipeline (report-only) and the `check-readiness` job inside `apply-updates.yml`. ## Available Functions @@ -546,7 +546,7 @@ The module exports **36 cmdlets**. Full detail (parameters, ARM API surface, RBA | **Fleet reads** | Daily fleet status reports, fleet health audits, version distribution | `Get-AzLocalFleetStatusData`, `New-AzLocalFleetStatusHtmlReport`, `Get-AzLocalFleetHealthOverview`, `Get-AzLocalFleetHealthFailures`, `Get-AzLocalFleetProgress` | | **Fleet gates** | Schedule coverage audit, fleet-wide health gate before a wave | `Test-AzLocalApplyUpdatesScheduleCoverage`, `Test-AzLocalFleetHealthGate`, `Test-AzLocalUpdateScheduleAllowed` | | **Fleet writes** | Wave-scoped update launcher with pause/resume state file | `Invoke-AzLocalFleetOperation`, `Stop-AzLocalFleetUpdate`, `Resume-AzLocalFleetUpdate`, `Export-AzLocalFleetState` | -| **Pipeline support** | Refresh bundled `Step.*.yml` workflow templates while preserving operator edits | `Update-AzLocalPipelineExample` | +| **Pipeline support** | Refresh bundled `*.yml` workflow templates while preserving operator edits | `Update-AzLocalPipelineExample` | | **Diagnostics** | Resolve effective ring for a cluster, latest solution version from the public catalog | `Resolve-AzLocalCurrentUpdateRing`, `Get-AzLocalLatestSolutionVersion`, `Get-AzLocalUpdateRunFailures` | Full signatures, ARM endpoints, and worked examples: **[docs/cmdlet-reference.md](docs/cmdlet-reference.md)**. @@ -562,7 +562,7 @@ The module's gating cmdlets (`Get-AzLocalClusterUpdateReadiness`, `Test-AzLocalC Most common issues fall into one of these buckets: -- **`az login` succeeds but `Get-AzLocalClusterInventory` returns nothing** - the identity has tenant-level `Reader` but not subscription `Reader` on the subscriptions where clusters live. Run the **`Step.0_authentication-test`** pipeline to enumerate the subscriptions the identity actually sees. +- **`az login` succeeds but `Get-AzLocalClusterInventory` returns nothing** - the identity has tenant-level `Reader` but not subscription `Reader` on the subscriptions where clusters live. Run the **`authentication-test`** pipeline to enumerate the subscriptions the identity actually sees. - **`Start-AzLocalClusterUpdate` returns `Unauthorized`** - the identity has `Azure Stack HCI Reader` instead of `Azure Stack HCI Administrator`. See [docs/rbac.md](docs/rbac.md). - **`Get-AzLocalFleetHealthOverview` returns `ParserFailure: token=`** - the underlying ARG query exceeded the `az graph query -q` Windows argument-truncation threshold (~2.8 KB). Fixed in v0.7.74; refresh your pipeline pins to v0.7.74+. - **`Test-AzLocalClusterHealth` reports duplicate findings** - ARM upstream sometimes emits byte-identical `healthCheckResult` rows; fixed in v0.7.76 via row-tuple dedup. @@ -580,7 +580,13 @@ This code is provided as-is for educational and reference purposes. The full What's-New history (v0.7.81 and earlier) has moved to [docs/release-history.md](docs/release-history.md). -The most recent release notes for **v0.8.6** stay above under [`What's New in v0.8.6`](#whats-new-in-v086). +The most recent release notes for **v0.8.7** stay above under [`What's New in v0.8.7`](#whats-new-in-v087). + +### What's New in v0.8.6 + +**Step.3 cycle calendar enrichment: per-day Step.6 CRON firing times + per-(ring, date) `UpdateStartWindow` tag-coverage check (>=95% threshold).** Adds two opt-in render-time columns to `Get-AzLocalApplyUpdatesScheduleCycleCalendar` (auto-wired from `Export-AzLocalApplyUpdatesScheduleAudit`) so operators see in one table which Step.6 cron firing times fire on each calendar day and what fraction of eligible clusters have an `UpdateStartWindow` tag that covers a firing. Also fixes six v0.8.5 thin-YAML port regressions (Step.0/3/4/6/9) and adds Pester static-audit guards. Same module export count as v0.8.5 (55). + +See [CHANGELOG.md](CHANGELOG.md#086---2026-06-10) for the full v0.8.6 entry. ### What's New in v0.8.4 diff --git a/AzLocal.UpdateManagement/Tests/Add-PesterRunTiming.ps1 b/AzLocal.UpdateManagement/Tests/Add-PesterRunTiming.ps1 new file mode 100644 index 00000000..62f5b766 --- /dev/null +++ b/AzLocal.UpdateManagement/Tests/Add-PesterRunTiming.ps1 @@ -0,0 +1,88 @@ +# --------------------------------------------------------------------------- +# Add-PesterRunTiming - shared helper that appends one wall-clock timing row +# to the git-tracked Tests/test-run-timings.csv ledger. +# +# Called by BOTH the canonical runner (Invoke-Tests.ps1) and the safe detached +# summary runner used from the VS Code chat harness, so every full-suite run we +# do is recorded in source control. Tracking the numbers in git lets us spot +# wall-clock regressions over time and decide whether the eventual test-file +# split for parallelism is worth it (see Tools/SMOKE-COVERAGE.md discussion). +# +# The ledger is intentionally a flat CSV (one row per run) so it diffs cleanly +# and can be charted with Import-Csv | ... in a one-liner. ASCII, no BOM, to +# satisfy the repo's encoding convention. +# --------------------------------------------------------------------------- +[CmdletBinding()] +param( + # The Pester run result object ([Pester.Run]) from Invoke-Pester -PassThru. + [Parameter(Mandatory = $true)] + $Result, + + # Total wall-clock seconds for the run as measured by the CALLER (module + # import + discovery + execution + teardown), not just Pester's own timer. + [Parameter(Mandatory = $true)] + [double]$WallClockSeconds, + + # Short label identifying which harness produced the row, e.g. + # 'Invoke-Tests.ps1' or 'chat-detached'. + [Parameter(Mandatory = $true)] + [string]$Runner, + + # Optional override for the ledger path. Defaults to the CSV next to this + # script (Tests/test-run-timings.csv). + [Parameter(Mandatory = $false)] + [string]$TimingsPath = (Join-Path -Path $PSScriptRoot -ChildPath 'test-run-timings.csv'), + + # Optional explicit module version (falls back to the loaded module). + [Parameter(Mandatory = $false)] + [string]$ModuleVersion +) + +if (-not $ModuleVersion) { + $ModuleVersion = (Get-Module AzLocal.UpdateManagement -ErrorAction SilentlyContinue | + Sort-Object Version -Descending | + Select-Object -First 1).Version +} +# Fallback: after Invoke-Pester completes the module may no longer be visible +# via Get-Module at this scope (it was imported inside Pester's container), so +# read ModuleVersion straight from the manifest next to the Tests folder. +if (-not $ModuleVersion) { + $manifestPath = Join-Path -Path $PSScriptRoot -ChildPath '..\AzLocal.UpdateManagement.psd1' + if (Test-Path -LiteralPath $manifestPath) { + try { + $ModuleVersion = (Import-PowerShellDataFile -Path $manifestPath).ModuleVersion + } + catch { } + } +} + +# Pester v5 exposes both the test-execution Duration and a separate +# DiscoveryDuration; record each so we can see how much of the wall clock is +# discovery (the part a file-split would parallelise). +$pesterSeconds = 0.0 +$discoverySeconds = 0.0 +if ($Result.Duration) { $pesterSeconds = [math]::Round($Result.Duration.TotalSeconds, 2) } +if ($Result.PSObject.Properties.Name -contains 'DiscoveryDuration' -and $Result.DiscoveryDuration) { + $discoverySeconds = [math]::Round($Result.DiscoveryDuration.TotalSeconds, 2) +} + +$row = [PSCustomObject][ordered]@{ + TimestampUtc = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ') + ModuleVersion = [string]$ModuleVersion + Total = [int]$Result.TotalCount + Passed = [int]$Result.PassedCount + Failed = [int]$Result.FailedCount + Skipped = [int]$Result.SkippedCount + WallClockSeconds = [math]::Round($WallClockSeconds, 2) + PesterDurationSeconds = $pesterSeconds + DiscoverySeconds = $discoverySeconds + Runner = $Runner + PSVersion = $PSVersionTable.PSVersion.ToString() +} + +# Export-Csv -Append writes the header automatically only when the file does +# not yet exist; on subsequent runs it appends data rows without a duplicate +# header. -Encoding ASCII keeps the file BOM-free. +$row | Export-Csv -Path $TimingsPath -Append -NoTypeInformation -Encoding ASCII + +return $row diff --git a/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 b/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 index 6e112454..6e8bc186 100644 --- a/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 +++ b/AzLocal.UpdateManagement/Tests/AzLocal.UpdateManagement.Tests.ps1 @@ -34,8 +34,8 @@ Describe 'Module: AzLocal.UpdateManagement' { $script:ModuleInfo | Should -Not -BeNullOrEmpty } - It 'Should have version 0.8.6' { - $script:ModuleInfo.Version | Should -Be '0.8.6' + It 'Should have version 0.8.7' { + $script:ModuleInfo.Version | Should -Be '0.8.7' } It 'Module version constants are in sync between .psm1 and .psd1' { @@ -143,7 +143,7 @@ Describe 'Module: AzLocal.UpdateManagement' { $match.Groups['v'].Value | Should -Be $manifestVersion -Because 'the first "## Version X.Y.Z" heading in psd1 ReleaseNotes must match the manifest ModuleVersion - did you forget to prepend the new release section?' } - It 'All bundled Step.[1-7]_*.yml pipeline samples pin GENERATED_AGAINST_MODULE_VERSION to manifest ModuleVersion' -ForEach @( + It 'All bundled *.yml pipeline samples pin GENERATED_AGAINST_MODULE_VERSION to manifest ModuleVersion' -ForEach @( @{ Platform = 'github-actions' } @{ Platform = 'azure-devops' } ) { @@ -153,17 +153,16 @@ Describe 'Module: AzLocal.UpdateManagement' { # GENERATED_AGAINST_MODULE_VERSION constant. Operators who run # Update-AzLocalPipelineExample after the upgrade get YAMLs # that disagree with the module they just installed. Every - # Step.[1-7]_*.yml sample (both platforms) must pin to the - # current manifest ModuleVersion. Step.0_authentication-test.yml - # is exempt - it does not consume the module so the pin is - # intentionally absent. + # bundled *.yml sample (both platforms) must pin to the current + # manifest ModuleVersion. v0.8.7 de-numbered the filenames and + # authentication-test.yml now also consumes the module, so every + # sample (no exemptions) carries the pin. $manifestPath = Join-Path -Path $PSScriptRoot -ChildPath '..\AzLocal.UpdateManagement.psd1' $manifestVersion = (Import-PowerShellDataFile -Path $manifestPath).ModuleVersion $samplesDir = Join-Path -Path $PSScriptRoot -ChildPath "..\Automation-Pipeline-Examples\$Platform" Test-Path $samplesDir | Should -BeTrue -Because "the $Platform sample folder must exist at $samplesDir" - $yamls = @(Get-ChildItem -Path $samplesDir -Filter 'Step.*.yml' -File | - Where-Object { $_.Name -notlike 'Step.0_*' }) - $yamls.Count | Should -BeGreaterThan 0 -Because "$samplesDir must ship at least one Step.[1-7]_*.yml sample" + $yamls = @(Get-ChildItem -Path $samplesDir -Filter '*.yml' -File) + $yamls.Count | Should -BeGreaterThan 0 -Because "$samplesDir must ship at least one *.yml sample" # GitHub Actions form (single line): # GENERATED_AGAINST_MODULE_VERSION: '0.7.71' # Azure DevOps form (two lines under a `variables:` list entry): @@ -180,12 +179,12 @@ Describe 'Module: AzLocal.UpdateManagement' { } } - It 'Step.4 pipeline templates call Export-AzLocalFleetConnectivityStatusReport (v0.8.5 thin-YAML)' -ForEach @( - @{ Platform = 'github-actions'; Path = '..\Automation-Pipeline-Examples\github-actions\Step.4_fleet-connectivity-status.yml' } - @{ Platform = 'azure-devops'; Path = '..\Automation-Pipeline-Examples\azure-devops\Step.4_fleet-connectivity-status.yml' } + It 'fleet-connectivity-status pipeline templates call Export-AzLocalFleetConnectivityStatusReport (v0.8.5 thin-YAML)' -ForEach @( + @{ Platform = 'github-actions'; Path = '..\Automation-Pipeline-Examples\github-actions\fleet-connectivity-status.yml' } + @{ Platform = 'azure-devops'; Path = '..\Automation-Pipeline-Examples\azure-devops\fleet-connectivity-status.yml' } ) { $yamlPath = Join-Path -Path $PSScriptRoot -ChildPath $Path - Test-Path $yamlPath | Should -BeTrue -Because "Step.4 template for $Platform must exist at $yamlPath" + Test-Path $yamlPath | Should -BeTrue -Because "fleet-connectivity-status template for $Platform must exist at $yamlPath" $content = Get-Content -Path $yamlPath -Raw # v0.8.5 thin-YAML guard: the ~255-line inline collection block @@ -195,12 +194,12 @@ Describe 'Module: AzLocal.UpdateManagement' { $content | Should -Not -Match '\$data\.ClusterRows' -Because "Step.4 $Platform must not still wire raw cmdlet rowsets into the yml - the cmdlet emits step outputs instead (v0.8.5)" } - It 'Step.3 pipeline templates call Export-AzLocalApplyUpdatesScheduleAudit (v0.8.5 thin-YAML)' -ForEach @( - @{ Platform = 'github-actions'; Path = '..\Automation-Pipeline-Examples\github-actions\Step.3_apply-updates-schedule-audit.yml' } - @{ Platform = 'azure-devops'; Path = '..\Automation-Pipeline-Examples\azure-devops\Step.3_apply-updates-schedule-audit.yml' } + It 'apply-updates-schedule-audit pipeline templates call Export-AzLocalApplyUpdatesScheduleAudit (v0.8.5 thin-YAML)' -ForEach @( + @{ Platform = 'github-actions'; Path = '..\Automation-Pipeline-Examples\github-actions\apply-updates-schedule-audit.yml' } + @{ Platform = 'azure-devops'; Path = '..\Automation-Pipeline-Examples\azure-devops\apply-updates-schedule-audit.yml' } ) { $yamlPath = Join-Path -Path $PSScriptRoot -ChildPath $Path - Test-Path $yamlPath | Should -BeTrue -Because "Step.3 template for $Platform must exist at $yamlPath" + Test-Path $yamlPath | Should -BeTrue -Because "apply-updates-schedule-audit template for $Platform must exist at $yamlPath" $content = Get-Content -Path $yamlPath -Raw # v0.8.5 thin-YAML guard: the ~220-line inline 'Run Schedule @@ -212,8 +211,8 @@ Describe 'Module: AzLocal.UpdateManagement' { $content | Should -Not -Match 'function\s+Convert-ScheduleRow' -Because "Step.3 $Platform must not contain inline schedule-row helpers (removed in v0.8.5)" } - It 'Should export exactly 55 functions' { - $script:ModuleInfo.ExportedFunctions.Count | Should -Be 55 + It 'Should export exactly 60 functions' { + $script:ModuleInfo.ExportedFunctions.Count | Should -Be 60 } It 'Should export the expected functions' { @@ -300,7 +299,13 @@ Describe 'Module: AzLocal.UpdateManagement' { 'Invoke-AzLocalReadinessGatedClusterUpdate', 'Add-AzLocalApplyUpdatesStepSummary', 'Add-AzLocalNoReadyClustersStepSummary', - 'Invoke-AzLocalItsmTicketingFromArtifact' + 'Invoke-AzLocalItsmTicketingFromArtifact', + # On-Prem Sideloading Automation (v0.8.7) - catalog refresh + plan resolution + readiness-gated sideload apply + status report + step summary + 'Update-AzLocalSideloadCatalog', + 'Resolve-AzLocalSideloadPlan', + 'Invoke-AzLocalSideloadUpdate', + 'Export-AzLocalSideloadStatusReport', + 'Add-AzLocalSideloadStepSummary' ) foreach ($func in $expectedFunctions) { @@ -444,23 +449,23 @@ Describe 'Module: AzLocal.UpdateManagement' { } Context 'Schedule-audit pipeline_path default is consumer-friendly (v0.7.66 regression)' { - # v0.7.66 regression guard: Step.3_apply-updates-schedule-audit.yml (GH + ADO) + # v0.7.66 regression guard: apply-updates-schedule-audit.yml (GH + ADO) # shipped with a default pipeline_path of # 'AzLocal.UpdateManagement/Automation-Pipeline-Examples' - a path that # only exists in this module's source repo. In a consumer repo (where - # Step.6_apply-updates.yml lives under .github/workflows or .azure-pipelines), + # apply-updates.yml lives under .github/workflows or .azure-pipelines), # the default-trigger run failed with # PipelineYamlPath '...' does not exist on the runner # before the schedule advisor could emit JUnit XML. The defaults are # now '.github/workflows' on GH and '.azure-pipelines' on ADO. # This test guards both files from ever regressing to the in-source path. - It 'Neither Step.3_apply-updates-schedule-audit.yml defaults to the in-source examples folder' { + It 'Neither apply-updates-schedule-audit.yml defaults to the in-source examples folder' { $examplesRoot = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples' $examplesRoot = (Resolve-Path -Path $examplesRoot).Path $auditYamls = @( - Join-Path $examplesRoot 'github-actions\Step.3_apply-updates-schedule-audit.yml' - Join-Path $examplesRoot 'azure-devops\Step.3_apply-updates-schedule-audit.yml' + Join-Path $examplesRoot 'github-actions\apply-updates-schedule-audit.yml' + Join-Path $examplesRoot 'azure-devops\apply-updates-schedule-audit.yml' ) $offenders = New-Object System.Collections.Generic.List[string] @@ -539,8 +544,8 @@ Describe 'Module: AzLocal.UpdateManagement' { $ghRoot = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions' $ghRoot = (Resolve-Path -Path $ghRoot).Path - $ymlFiles = Get-ChildItem -Path $ghRoot -Filter 'Step.*.yml' -File - $ymlFiles.Count | Should -BeGreaterOrEqual 10 -Because 'all 10 Step.{0..9}.yml templates are expected under Automation-Pipeline-Examples/github-actions/' + $ymlFiles = Get-ChildItem -Path $ghRoot -Filter '*.yml' -File + $ymlFiles.Count | Should -BeGreaterOrEqual 11 -Because 'all 11 de-numbered pipeline templates are expected under Automation-Pipeline-Examples/github-actions/' $offenders = New-Object System.Collections.Generic.List[string] foreach ($yml in $ymlFiles) { @@ -569,7 +574,7 @@ Describe 'Module: AzLocal.UpdateManagement' { It 'Every GitHub Actions install step contains the version banner string' { $ghRoot = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions' $ghRoot = (Resolve-Path -Path $ghRoot).Path - $ymlFiles = Get-ChildItem -Path $ghRoot -Filter 'Step.*.yml' -File + $ymlFiles = Get-ChildItem -Path $ghRoot -Filter '*.yml' -File $bannerLiteral = [regex]::Escape('Pipeline YAML v$generated | Module v$installed installed') $cmdletCall = [regex]::Escape('Add-AzLocalPipelineVersionBanner') @@ -588,7 +593,7 @@ Describe 'Module: AzLocal.UpdateManagement' { It 'Every Azure DevOps yml install site contains the version banner string (Step.6 second install deliberately skips upload to avoid duplicate banner)' { $adoRoot = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\azure-devops' $adoRoot = (Resolve-Path -Path $adoRoot).Path - $ymlFiles = Get-ChildItem -Path $adoRoot -Filter 'Step.*.yml' -File + $ymlFiles = Get-ChildItem -Path $adoRoot -Filter '*.yml' -File $bannerPattern = [regex]::Escape('Pipeline YAML v$generated | Module v$installed installed') $cmdletPattern = [regex]::Escape('Add-AzLocalPipelineVersionBanner') @@ -601,16 +606,17 @@ Describe 'Module: AzLocal.UpdateManagement' { # with one Add-AzLocalPipelineVersionBanner call - count either # form as a banner emit. $perFileExpect = @{ - 'Step.0_authentication-test.yml' = 1 - 'Step.1_inventory-clusters.yml' = 1 - 'Step.2_manage-updatering-tags.yml' = 1 - 'Step.3_apply-updates-schedule-audit.yml' = 1 - 'Step.4_fleet-connectivity-status.yml' = 1 - 'Step.5_assess-update-readiness.yml' = 1 - 'Step.6_apply-updates.yml' = 1 - 'Step.7_monitor-updates.yml' = 1 - 'Step.8_fleet-update-status.yml' = 1 - 'Step.9_fleet-health-status.yml' = 1 + 'authentication-test.yml' = 1 + 'inventory-clusters.yml' = 1 + 'manage-updatering-tags.yml' = 1 + 'apply-updates-schedule-audit.yml' = 1 + 'fleet-connectivity-status.yml' = 1 + 'assess-update-readiness.yml' = 1 + 'apply-updates.yml' = 1 + 'monitor-updates.yml' = 1 + 'fleet-update-status.yml' = 1 + 'fleet-health-status.yml' = 1 + 'sideload-updates.yml' = 1 } $offenders = New-Object System.Collections.Generic.List[string] foreach ($yml in $ymlFiles) { @@ -636,7 +642,7 @@ Describe 'Module: AzLocal.UpdateManagement' { $detail = if ($offenders.Count -gt 0) { $offenders -join [Environment]::NewLine } else { '(no mismatches)' } $offenders.Count | Should -Be 0 -Because "every ADO yml uploads exactly 1 version banner to the build Summary - via ##vso[task.uploadsummary] for the v0.8.4 inline form, or via Add-AzLocalPipelineVersionBanner for the v0.8.5 thin-YAML form. Step.6 has 2 install steps but the second (ApplyUpdates stage) deliberately skips the upload so the banner is not duplicated. Findings:$([Environment]::NewLine)$detail" - $totalSites | Should -Be 10 -Because "total ADO Summary banner emits should be 10 (1 per yml; Step.6's second install step skips upload)" + $totalSites | Should -Be 11 -Because "total ADO Summary banner emits should be 11 (1 per yml; the apply-updates pipeline's second install step skips upload)" } } @@ -652,7 +658,7 @@ Describe 'Module: AzLocal.UpdateManagement' { # the cmdlets that produce that output (the per-cluster table # content is asserted in the cmdlet-level Pester suite). It 'GitHub Actions Step.6 yml invokes Invoke-AzLocalReadinessGatedClusterUpdate (writes apply-results.json) and Add-AzLocalApplyUpdatesStepSummary (renders both per-cluster tables)' { - $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions\Step.6_apply-updates.yml' + $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions\apply-updates.yml' $yml = (Resolve-Path -Path $yml).Path $content = Get-Content -LiteralPath $yml -Raw $content | Should -Match 'Invoke-AzLocalReadinessGatedClusterUpdate' -Because 'Step.6 GH apply-updates step must call the readiness-gated apply cmdlet (which writes apply-results.json)' @@ -662,7 +668,7 @@ Describe 'Module: AzLocal.UpdateManagement' { } It 'Azure DevOps Step.6 yml invokes Invoke-AzLocalReadinessGatedClusterUpdate (writes apply-results.json) and Add-AzLocalApplyUpdatesStepSummary (renders both per-cluster tables)' { - $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\azure-devops\Step.6_apply-updates.yml' + $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\azure-devops\apply-updates.yml' $yml = (Resolve-Path -Path $yml).Path $content = Get-Content -LiteralPath $yml -Raw $content | Should -Match 'Invoke-AzLocalReadinessGatedClusterUpdate' -Because 'ADO Step.6 apply-updates task must call the readiness-gated apply cmdlet' @@ -672,7 +678,7 @@ Describe 'Module: AzLocal.UpdateManagement' { } It 'GitHub Actions Step.6 yml invokes the schedule-resolver, readiness-gate, no-ready, and ITSM cmdlets in their respective steps' { - $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions\Step.6_apply-updates.yml' + $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions\apply-updates.yml' $yml = (Resolve-Path -Path $yml).Path $content = Get-Content -LiteralPath $yml -Raw $content | Should -Match 'Resolve-AzLocalPipelineUpdateRing' -Because 'Step.6 GH resolve-ring step must call Resolve-AzLocalPipelineUpdateRing (replaces the ~80-line inline resolver script)' @@ -682,7 +688,7 @@ Describe 'Module: AzLocal.UpdateManagement' { } It 'Azure DevOps Step.6 yml invokes the schedule-resolver, readiness-gate, no-ready, and ITSM cmdlets in their respective tasks' { - $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\azure-devops\Step.6_apply-updates.yml' + $yml = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\azure-devops\apply-updates.yml' $yml = (Resolve-Path -Path $yml).Path $content = Get-Content -LiteralPath $yml -Raw $content | Should -Match 'Resolve-AzLocalPipelineUpdateRing' @@ -702,8 +708,8 @@ Describe 'Module: AzLocal.UpdateManagement' { # path must still work verbatim when the new input is false (back-compat), # and the resolver must throw a helpful error when BOTH paths are empty. BeforeAll { - $script:S6Gh = (Resolve-Path (Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions\Step.6_apply-updates.yml')).Path - $script:S6Ado = (Resolve-Path (Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\azure-devops\Step.6_apply-updates.yml')).Path + $script:S6Gh = (Resolve-Path (Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\github-actions\apply-updates.yml')).Path + $script:S6Ado = (Resolve-Path (Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\azure-devops\apply-updates.yml')).Path $script:S6GhContent = Get-Content -LiteralPath $script:S6Gh -Raw $script:S6AdoContent = Get-Content -LiteralPath $script:S6Ado -Raw } @@ -1627,6 +1633,80 @@ Describe 'Helper Function: Format-AzLocalUpdateRun (Internal)' { $f.ErrorMessage | Should -BeNullOrEmpty } } + + It 'CurrentStep reflects the deepest InProgress step, not the top-level wrapper (Step.7 monitor fix)' { + InModuleScope AzLocal.UpdateManagement { + # Real Azure Local progress trees nest a coarse top-level wrapper step + # (e.g. "Start update") that stays InProgress for the whole run. CurrentStep + # must dig to the deepest active leaf so the in-flight monitor shows the real + # current activity instead of always showing the wrapper name. + $run = [PSCustomObject]@{ + id = '/subscriptions/x/resourceGroups/rg/providers/Microsoft.AzureStackHCI/clusters/c1/updates/Solution12.2604/updateRuns/rDeep' + name = 'rDeep' + properties = [PSCustomObject]@{ + state = 'InProgress' + timeStarted = (Get-Date).AddMinutes(-30).ToUniversalTime().ToString('o') + lastUpdatedTime = $null + duration = $null + progress = [PSCustomObject]@{ + status = 'InProgress' + steps = @( + [PSCustomObject]@{ + name = 'Start update' + status = 'InProgress' + steps = @( + [PSCustomObject]@{ + name = 'Update Cluster' + status = 'InProgress' + steps = @( + [PSCustomObject]@{ name = 'Update Node 2'; status = 'InProgress' } + ) + } + ) + } + ) + } + location = 'eastus' + } + } + $f = Format-AzLocalUpdateRun -run $run -clusterName 'c1' + $f.CurrentStep | Should -Be 'Update Node 2' + } + } + + It 'CurrentStep reflects the deepest Failed step with (FAILED) suffix, not the top-level wrapper (Step.7 monitor fix)' { + InModuleScope AzLocal.UpdateManagement { + $run = [PSCustomObject]@{ + id = '/subscriptions/x/resourceGroups/rg/providers/Microsoft.AzureStackHCI/clusters/c1/updates/Solution12.2604/updateRuns/rDeepFail' + name = 'rDeepFail' + properties = [PSCustomObject]@{ + state = 'Failed' + timeStarted = '2026-04-24T16:10:24Z' + lastUpdatedTime = '2026-04-24T17:10:24Z' + duration = 'PT1H' + progress = [PSCustomObject]@{ + status = 'Error' + steps = @( + [PSCustomObject]@{ + name = 'Start update' + status = 'Error' + steps = @( + [PSCustomObject]@{ + name = 'Run Pre Update Validation' + status = 'Error' + errorMessage = 'validation failed' + } + ) + } + ) + } + location = 'eastus' + } + } + $f = Format-AzLocalUpdateRun -run $run -clusterName 'c1' + $f.CurrentStep | Should -Be 'Run Pre Update Validation (FAILED)' + } + } } Describe 'Helper Function: Get-DeepestErrorMessage (Internal)' { @@ -2786,6 +2866,45 @@ Describe 'Integration: Start-AzLocalClusterUpdate Schedule Status' { if (Test-Path $outputPath) { Remove-Item $outputPath -Force } } } + + It 'Export-ResultsToJUnitXml should not throw on an UpdateStarted row lacking CurrentState/Progress (v0.8.7)' { + # Regression: the UpdateStarted success shape emitted by + # Start-AzLocalClusterUpdate has no CurrentState or Progress + # property. Under Set-StrictMode -Version Latest the success + # (default) branch's bare 'if ($result.CurrentState)' / + # 'if ($result.Progress)' threw "The property 'CurrentState' + # cannot be found on this object", surfacing as + # "Failed to export results: ..." in the Apply Updates step. + $testResult = [PSCustomObject]@{ + ClusterName = 'London' + Status = 'UpdateStarted' + Message = 'Update initiated successfully' + UpdateName = 'Solution12.2605.1003.210' + StartTime = Get-Date + EndTime = Get-Date + Duration = '00:00:14' + } + $outputPath = Join-Path $env:TEMP "pester-junit-updatestarted-$([Guid]::NewGuid()).xml" + try { + { + & (Get-Module 'AzLocal.UpdateManagement') { + param($results, $path) + Export-ResultsToJUnitXml -Results $results -OutputPath $path -TestSuiteName 'Test' -OperationType 'StartUpdate' + } @($testResult) $outputPath + } | Should -Not -Throw + + $outputPath | Should -Exist + $xml = [xml](Get-Content $outputPath -Raw) + $testCase = $xml.SelectSingleNode('//testcase') + $testCase | Should -Not -BeNullOrEmpty + # Success row renders as , not //. + $testCase.SelectSingleNode('system-out') | Should -Not -BeNullOrEmpty + $testCase.SelectSingleNode('failure') | Should -BeNullOrEmpty + } + finally { + if (Test-Path $outputPath) { Remove-Item $outputPath -Force } + } + } } Context 'JUnit XML export handles non-failure non-passing statuses (v0.7.62)' { @@ -6268,7 +6387,7 @@ Describe 'Function: Copy-AzLocalPipelineExample' { # YAMLs landed directly in $dest $yamls = @(Get-ChildItem -LiteralPath $dest -Filter '*.yml' -File) $yamls.Count | Should -BeGreaterThan 0 - $yamls.Name | Should -Contain 'Step.0_authentication-test.yml' + $yamls.Name | Should -Contain 'authentication-test.yml' # No platform-named subfolder, no Automation-Pipeline-Examples wrapper Test-Path (Join-Path $dest 'github-actions') | Should -BeFalse @@ -6290,7 +6409,7 @@ Describe 'Function: Copy-AzLocalPipelineExample' { $yamls = @(Get-ChildItem -LiteralPath $dest -Filter '*.yml' -File) $yamls.Count | Should -BeGreaterThan 0 - $yamls.Name | Should -Contain 'Step.0_authentication-test.yml' + $yamls.Name | Should -Contain 'authentication-test.yml' Test-Path (Join-Path $dest 'azure-devops') | Should -BeFalse Test-Path (Join-Path $dest 'github-actions') | Should -BeFalse @@ -6338,7 +6457,7 @@ Describe 'Function: Copy-AzLocalPipelineExample' { Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub 6>$null | Out-Null # Mutate one destination file so we can prove it gets overwritten - $target = Join-Path $dest 'Step.0_authentication-test.yml' + $target = Join-Path $dest 'authentication-test.yml' $sentinel = '# SENTINEL - if this comment survives, -Update did not overwrite' Set-Content -LiteralPath $target -Value $sentinel -Encoding ASCII (Get-Content -LiteralPath $target -Raw) | Should -Match 'SENTINEL' @@ -6380,7 +6499,7 @@ Describe 'Function: Copy-AzLocalPipelineExample' { # Seed and then mutate Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub 6>$null | Out-Null - $target = Join-Path $dest 'Step.0_authentication-test.yml' + $target = Join-Path $dest 'authentication-test.yml' $sentinel = '# WHATIF SENTINEL - -WhatIf must preserve this' Set-Content -LiteralPath $target -Value $sentinel -Encoding ASCII @@ -6436,9 +6555,9 @@ Describe 'Function: Copy-AzLocalPipelineExample' { # v0.7.92: starter apply-updates-schedule.yml drop (default-on for # -Platform GitHub|AzureDevOps; suppressed by -SkipStarterSchedule). - # The starter lands ONE LEVEL UP from -Destination so the schedule - # file sits beside .github\workflows\ (GitHub) or the pipelines - # folder (ADO) rather than inside it. + # v0.8.7 relocates the starter into a repo-root `config\` folder so the + # schedule sits beside config\ClusterUpdateRings.csv - the IDENTICAL + # path on GitHub and Azure DevOps. It 'v0.7.92: -SkipStarterSchedule is exposed as a [switch] parameter' { $cmd = Get-Command -Name 'Copy-AzLocalPipelineExample' -ErrorAction Stop @@ -6446,14 +6565,14 @@ Describe 'Function: Copy-AzLocalPipelineExample' { $cmd.Parameters['SkipStarterSchedule'].ParameterType | Should -Be ([switch]) } - It 'v0.7.92: -Platform GitHub default drops starter apply-updates-schedule.yml ONE LEVEL UP from -Destination' { + It 'v0.8.7: -Platform GitHub default drops starter apply-updates-schedule.yml into repo-root config folder' { $repoRoot = Join-Path $script:cpDestRoot 'gh-starter-fresh' $dest = Join-Path $repoRoot '.github\workflows' New-Item -Path $dest -ItemType Directory -Force | Out-Null Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub 6>$null | Out-Null - $scheduleDest = Join-Path $repoRoot '.github\apply-updates-schedule.yml' + $scheduleDest = Join-Path $repoRoot 'config\apply-updates-schedule.yml' Test-Path $scheduleDest | Should -BeTrue # File content must match the bundled example verbatim (starter copy # is byte-for-byte, no templating). @@ -6462,14 +6581,14 @@ Describe 'Function: Copy-AzLocalPipelineExample' { (Get-FileHash -LiteralPath $scheduleDest).Hash | Should -Be (Get-FileHash -LiteralPath $source).Hash } - It 'v0.7.92: -Platform AzureDevOps default drops starter apply-updates-schedule.yml ONE LEVEL UP from -Destination' { + It 'v0.8.7: -Platform AzureDevOps default drops starter apply-updates-schedule.yml into repo-root config folder' { $repoRoot = Join-Path $script:cpDestRoot 'ado-starter-fresh' $dest = Join-Path $repoRoot 'pipelines' New-Item -Path $dest -ItemType Directory -Force | Out-Null Copy-AzLocalPipelineExample -Destination $dest -Platform AzureDevOps 6>$null | Out-Null - $scheduleDest = Join-Path $repoRoot 'apply-updates-schedule.yml' + $scheduleDest = Join-Path $repoRoot 'config\apply-updates-schedule.yml' Test-Path $scheduleDest | Should -BeTrue } @@ -6479,7 +6598,8 @@ Describe 'Function: Copy-AzLocalPipelineExample' { New-Item -Path $dest -ItemType Directory -Force | Out-Null # Pre-seed an operator-tailored schedule file - $scheduleDest = Join-Path $repoRoot '.github\apply-updates-schedule.yml' + $scheduleDest = Join-Path $repoRoot 'config\apply-updates-schedule.yml' + New-Item -Path (Split-Path -Parent $scheduleDest) -ItemType Directory -Force | Out-Null $sentinel = '# OPERATOR SENTINEL - must never be overwritten by Copy-AzLocalPipelineExample' Set-Content -LiteralPath $scheduleDest -Value $sentinel -Encoding ASCII @@ -6496,7 +6616,7 @@ Describe 'Function: Copy-AzLocalPipelineExample' { Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub -SkipStarterSchedule 6>$null | Out-Null - $scheduleDest = Join-Path $repoRoot '.github\apply-updates-schedule.yml' + $scheduleDest = Join-Path $repoRoot 'config\apply-updates-schedule.yml' Test-Path $scheduleDest | Should -BeFalse } @@ -6525,9 +6645,123 @@ Describe 'Function: Copy-AzLocalPipelineExample' { Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub -WhatIf 6>$null | Out-Null - $scheduleDest = Join-Path $repoRoot '.github\apply-updates-schedule.yml' + $scheduleDest = Join-Path $repoRoot 'config\apply-updates-schedule.yml' Test-Path $scheduleDest | Should -BeFalse } + + # v0.8.7: starter on-prem sideloading config drop (sideload-auth-map.csv + # + sideload-catalog.yml). Default-on for -Platform GitHub|AzureDevOps; + # suppressed by -SkipStarterSideloadConfig. Lands in the SAME repo-root + # `config\` folder as the starter schedule and the ring CSV - the + # IDENTICAL path on GitHub and Azure DevOps. Both files are + # header/skeleton only (no demo data rows), so they parse to an empty + # auth-map / empty catalog. + + It 'v0.8.7: -SkipStarterSideloadConfig is exposed as a [switch] parameter' { + $cmd = Get-Command -Name 'Copy-AzLocalPipelineExample' -ErrorAction Stop + $cmd.Parameters.ContainsKey('SkipStarterSideloadConfig') | Should -BeTrue + $cmd.Parameters['SkipStarterSideloadConfig'].ParameterType | Should -Be ([switch]) + } + + It 'v0.8.7: -Platform GitHub default drops starter sideload config into repo-root config folder' { + $repoRoot = Join-Path $script:cpDestRoot 'gh-sideload-fresh' + $dest = Join-Path $repoRoot '.github\workflows' + New-Item -Path $dest -ItemType Directory -Force | Out-Null + + Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub 6>$null | Out-Null + + Test-Path (Join-Path $repoRoot 'config\sideload-auth-map.csv') | Should -BeTrue + Test-Path (Join-Path $repoRoot 'config\sideload-catalog.yml') | Should -BeTrue + } + + It 'v0.8.7: -Platform AzureDevOps default drops starter sideload config into repo-root config folder' { + $repoRoot = Join-Path $script:cpDestRoot 'ado-sideload-fresh' + $dest = Join-Path $repoRoot 'pipelines' + New-Item -Path $dest -ItemType Directory -Force | Out-Null + + Copy-AzLocalPipelineExample -Destination $dest -Platform AzureDevOps 6>$null | Out-Null + + Test-Path (Join-Path $repoRoot 'config\sideload-auth-map.csv') | Should -BeTrue + Test-Path (Join-Path $repoRoot 'config\sideload-catalog.yml') | Should -BeTrue + } + + It 'v0.8.7: starter auth-map is header-only and parses to an EMPTY auth-map (no demo rows)' { + $repoRoot = Join-Path $script:cpDestRoot 'gh-sideload-parse' + $dest = Join-Path $repoRoot '.github\workflows' + New-Item -Path $dest -ItemType Directory -Force | Out-Null + + Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub 6>$null | Out-Null + + $authMapPath = Join-Path $repoRoot 'config\sideload-auth-map.csv' + $catalogPath = Join-Path $repoRoot 'config\sideload-catalog.yml' + + # The starter must carry NO demo credential data (so it is inert even + # if SIDELOAD_UPDATES is flipped on before the operator fills it in). + $authRaw = Get-Content -Raw -LiteralPath $authMapPath + $authRaw | Should -Match 'UpdateAuthAccountId,KeyVaultName' + $authRaw | Should -Not -Match 'kv-azlocal' + + $map = InModuleScope AzLocal.UpdateManagement -Parameters @{ P = $authMapPath } { + param($P) + Get-AzLocalSideloadAuthMap -Path $P + } + $map.Keys.Count | Should -Be 0 + + $catalog = InModuleScope AzLocal.UpdateManagement -Parameters @{ P = $catalogPath } { + param($P) + @(Get-AzLocalSideloadCatalog -Path $P) + } + $catalog.Count | Should -Be 0 + } + + It 'v0.8.7: existing sideload config at the target is NEVER overwritten by the starter drop' { + $repoRoot = Join-Path $script:cpDestRoot 'gh-sideload-preserve' + $dest = Join-Path $repoRoot '.github\workflows' + New-Item -Path $dest -ItemType Directory -Force | Out-Null + New-Item -Path (Join-Path $repoRoot 'config') -ItemType Directory -Force | Out-Null + + $authMapPath = Join-Path $repoRoot 'config\sideload-auth-map.csv' + $sentinel = '# OPERATOR SENTINEL - must never be overwritten' + Set-Content -LiteralPath $authMapPath -Value $sentinel -Encoding ASCII + + Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub 6>$null | Out-Null + + (Get-Content -LiteralPath $authMapPath -Raw) | Should -Match 'OPERATOR SENTINEL' + } + + It 'v0.8.7: -SkipStarterSideloadConfig suppresses the sideload starter drop entirely' { + $repoRoot = Join-Path $script:cpDestRoot 'gh-sideload-skip' + $dest = Join-Path $repoRoot '.github\workflows' + New-Item -Path $dest -ItemType Directory -Force | Out-Null + + Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub -SkipStarterSideloadConfig 6>$null | Out-Null + + Test-Path (Join-Path $repoRoot 'config\sideload-auth-map.csv') | Should -BeFalse + Test-Path (Join-Path $repoRoot 'config\sideload-catalog.yml') | Should -BeFalse + } + + It 'v0.8.7: -Platform All does NOT drop sideload starter config (only GitHub|AzureDevOps do)' { + $parent = Join-Path $script:cpDestRoot 'all-no-sideload-parent' + New-Item -Path $parent -ItemType Directory -Force | Out-Null + $dest = Join-Path $parent 'all-no-sideload' + New-Item -Path $dest -ItemType Directory -Force | Out-Null + + Copy-AzLocalPipelineExample -Destination $dest 6>$null | Out-Null + + Test-Path (Join-Path $parent 'sideload-auth-map.csv') | Should -BeFalse + Test-Path (Join-Path $dest 'sideload-auth-map.csv') | Should -BeFalse + } + + It 'v0.8.7: -WhatIf does not write the sideload starter config' { + $repoRoot = Join-Path $script:cpDestRoot 'gh-sideload-whatif' + $dest = Join-Path $repoRoot '.github\workflows' + New-Item -Path $dest -ItemType Directory -Force | Out-Null + + Copy-AzLocalPipelineExample -Destination $dest -Platform GitHub -WhatIf 6>$null | Out-Null + + Test-Path (Join-Path $repoRoot 'config\sideload-auth-map.csv') | Should -BeFalse + Test-Path (Join-Path $repoRoot 'config\sideload-catalog.yml') | Should -BeFalse + } } #endregion Copy-AzLocalPipelineExample (v0.7.4, updated in v0.7.50, v0.7.92) @@ -6970,7 +7204,7 @@ on: schedule: - cron: '55 1 * * 6,0' - cron: "0 22 * * 5" -"@ | Set-Content -Path (Join-Path $script:tmpYamlDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:tmpYamlDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII @" trigger: none schedules: @@ -6978,7 +7212,7 @@ schedules: displayName: Weekday early-morning branches: include: [ main ] -"@ | Set-Content -Path (Join-Path $script:tmpYamlDir 'azure-devops\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:tmpYamlDir 'azure-devops\Step.7_apply-updates.yml') -Encoding ASCII } AfterAll { Remove-Item -Path $script:tmpYamlDir -Recurse -Force -ErrorAction SilentlyContinue @@ -7029,7 +7263,7 @@ on: on: schedule: - cron: '0 7 * * *' -"@ | Set-Content -Path (Join-Path $regressionDir 'github-actions\Step.9_fleet-health-status.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $regressionDir 'github-actions\Step.10_fleet-health-status.yml') -Encoding ASCII # Apply-updates file: ships with only commented-out cron examples, # so the reader should return ZERO crons for this folder overall. @" @@ -7037,7 +7271,7 @@ on: workflow_dispatch: # schedule: # - cron: '0 22 * * 6' -"@ | Set-Content -Path (Join-Path $regressionDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $regressionDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII InModuleScope AzLocal.UpdateManagement -Parameters @{ dir = $regressionDir } { param($dir) @@ -7063,7 +7297,7 @@ on: schedule: - cron: ' ' - cron: '0 22 * * 6' -"@ | Set-Content -Path (Join-Path $emptyDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $emptyDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII InModuleScope AzLocal.UpdateManagement -Parameters @{ dir = $emptyDir } { param($dir) @@ -7087,7 +7321,7 @@ on: on: schedule: - cron: '50 1 * * 6,0' -"@ | Set-Content -Path (Join-Path $script:tmpYamlDir2 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:tmpYamlDir2 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII } AfterAll { Remove-Item -Path $script:tmpYamlDir2 -Recurse -Force -ErrorAction SilentlyContinue @@ -7239,7 +7473,7 @@ on: on: schedule: - cron: '55 1 * * 6,0' -"@ | Set-Content -Path (Join-Path $script:beltYamlDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:beltYamlDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII } AfterAll { Remove-Item -Path $script:beltYamlDir -Recurse -Force -ErrorAction SilentlyContinue @@ -7331,7 +7565,7 @@ on: schedule: - cron: '50 1 * * 6,0' - cron: '55 21 * * 1-5' -"@ | Set-Content -Path (Join-Path $script:multiYamlDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:multiYamlDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII } AfterAll { Remove-Item -Path $script:multiYamlDir -Recurse -Force -ErrorAction SilentlyContinue @@ -7434,7 +7668,7 @@ on: on: schedule: - cron: '55 1 * * 6,0' -"@ | Set-Content -Path (Join-Path $script:pruneYamlDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:pruneYamlDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII } AfterAll { Remove-Item -Path $script:pruneYamlDir -Recurse -Force -ErrorAction SilentlyContinue @@ -7501,7 +7735,7 @@ on: @" on: workflow_dispatch: -"@ | Set-Content -Path (Join-Path $script:nwtDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:nwtDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII # CSV WITH ResourceId column: 4 clusters. One peer (cA) in Ring1 # with a real UpdateStartWindow; one orphan target (cB) in Ring1 @@ -7631,7 +7865,7 @@ cB,rg1,s1,Ring1, @" on: workflow_dispatch: -"@ | Set-Content -Path (Join-Path $script:calDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:calDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII # Minimal schema-v2 schedule: 2-week cycle, Ring1 in week 1 # (Mon-Fri), Ring2 in week 2 (Mon-Fri). No allowedUpdateVersions. @@ -7706,7 +7940,7 @@ schedule: @" on: workflow_dispatch: -"@ | Set-Content -Path (Join-Path $script:exclDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:exclDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII } AfterAll { Remove-Item -Path $script:exclDir -Recurse -Force -ErrorAction SilentlyContinue @@ -7752,7 +7986,7 @@ on: # End-to-end smoke against the actual YAMLs published to PSGallery as # part of the module. This is the gap that let v0.7.68 ship with the # -LiteralPath/-Include glob regression: unit tests only fed the reader - # synthetic Step.6_apply-updates.yml files in isolation, never a real + # synthetic Step.7_apply-updates.yml files in isolation, never a real # multi-pipeline folder. Running the audit against the bundle now # guarantees that no future change re-introduces a glob that also picks # up sibling Step.N_*.yml schedule triggers. @@ -7777,7 +8011,7 @@ on: InModuleScope AzLocal.UpdateManagement -Parameters @{ bundleGhDir = $script:bundleGhDir } { param($bundleGhDir) $r = Read-AzLocalApplyUpdatesYamlCrons -Path $bundleGhDir - @($r).Count | Should -Be 0 -Because 'the shipped Step.6_apply-updates.yml ships with only commented cron examples, and the reader must NOT pick up crons from sibling Step.N pipelines' + @($r).Count | Should -Be 0 -Because 'the shipped Step.7_apply-updates.yml ships with only commented cron examples, and the reader must NOT pick up crons from sibling Step.N pipelines' } } } @@ -7966,7 +8200,7 @@ Describe 'v0.7.66 Artifact download names carry a UTC timestamp suffix' { } # Accept either the in-stage step-output form (`$(stamp.artifactStamp)`) # OR a cross-stage variable that ends in `ArtifactStamp)`, which is - # how `Step.6_apply-updates.yml` consumes the CheckReadiness stage's stamp + # how `Step.7_apply-updates.yml` consumes the CheckReadiness stage's stamp # via the `readinessArtifactStamp` mapped variable. if ($name -notmatch '\$\(.+?[Aa]rtifactStamp\)') { $offenders.Add("$($yml.Name): ${key} '$name' missing a `$(...artifactStamp) suffix") @@ -8360,14 +8594,14 @@ Describe 'Function: Update-AzLocalPipelineExample' { } Context 'Marker-aware merge preserves destination customisations' { - It 'Replaces the schedule-triggers body with the destination body in Step.6_apply-updates.yml' { + It 'Replaces the schedule-triggers body with the destination body in apply-updates.yml' { $temp = Join-Path $env:TEMP "upe-merge-$([guid]::NewGuid())" New-Item -ItemType Directory -Path $temp -Force | Out-Null try { - # 1. Copy the bundled Step.6_apply-updates.yml to the destination. - $src = Join-Path $script:UpePlatformSrcGh 'Step.6_apply-updates.yml' + # 1. Copy the bundled apply-updates.yml to the destination. + $src = Join-Path $script:UpePlatformSrcGh 'apply-updates.yml' Copy-Item -Path $src -Destination $temp - $destFile = Join-Path $temp 'Step.6_apply-updates.yml' + $destFile = Join-Path $temp 'apply-updates.yml' # 2. Inject a customer cron INSIDE the schedule-triggers marker block. $customerBody = "`r`n schedule:`r`n - cron: '0 22 * * 6' # Wave1 SatNight22UTC`r`n " @@ -8384,7 +8618,7 @@ Describe 'Function: Update-AzLocalPipelineExample' { # 3. Run the cmdlet. Source body should be REPLACED with customer body. $r = Update-AzLocalPipelineExample -Destination $temp -Platform GitHub -PassThru -Confirm:$false - $row = $r | Where-Object { $_.File -like '*Step.6_apply-updates.yml' } + $row = $r | Where-Object { $_.File -like '*apply-updates.yml' } $row.Action | Should -Match 'Updated|Unchanged' # Customer cron MUST survive $newText = [System.IO.File]::ReadAllText($destFile, [System.Text.UTF8Encoding]::new($false)) @@ -8413,11 +8647,11 @@ Describe 'Function: Update-AzLocalPipelineExample' { New-Item -ItemType Directory -Path $temp -Force | Out-Null try { # Place a stripped-down YAML at dest with NO markers, same filename as a bundled file. - $destFile = Join-Path $temp 'Step.6_apply-updates.yml' + $destFile = Join-Path $temp 'apply-updates.yml' 'name: legacy file with no markers' | Set-Content -LiteralPath $destFile -Encoding utf8 $r = Update-AzLocalPipelineExample -Destination $temp -Platform GitHub -PassThru -Confirm:$false 3>$null - $row = $r | Where-Object { $_.File -like '*Step.6_apply-updates.yml' } + $row = $r | Where-Object { $_.File -like '*apply-updates.yml' } $row.Action | Should -Be 'Skipped-NeedsForce' # Dest file unchanged (Get-Content -Raw -LiteralPath $destFile) | Should -Match 'legacy file with no markers' @@ -8428,11 +8662,11 @@ Describe 'Function: Update-AzLocalPipelineExample' { $temp = Join-Path $env:TEMP "upe-firstmig-force-$([guid]::NewGuid())" New-Item -ItemType Directory -Path $temp -Force | Out-Null try { - $destFile = Join-Path $temp 'Step.6_apply-updates.yml' + $destFile = Join-Path $temp 'apply-updates.yml' 'name: legacy file with no markers' | Set-Content -LiteralPath $destFile -Encoding utf8 $r = Update-AzLocalPipelineExample -Destination $temp -Platform GitHub -PassThru -Force -Confirm:$false 3>$null - $row = $r | Where-Object { $_.File -like '*Step.6_apply-updates.yml' } + $row = $r | Where-Object { $_.File -like '*apply-updates.yml' } $row.Action | Should -Be 'Overwritten' $row.NewMarkers | Should -Contain 'schedule-triggers' (Get-Content -Raw -LiteralPath $destFile) | Should -Match 'BEGIN-AZLOCAL-CUSTOMIZE:schedule-triggers' @@ -9283,7 +9517,7 @@ Describe 'Function: Test-AzLocalApplyUpdatesScheduleCoverage - v0.7.70 Section d @" on: workflow_dispatch: -"@ | Set-Content -Path (Join-Path $script:v7_70_yamlDir 'github-actions\Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $script:v7_70_yamlDir 'github-actions\Step.7_apply-updates.yml') -Encoding ASCII # Schedule file that knows about Pilot + Wave1 but is MISSING 'Production' # (which is tagged on c3) and ORPHANS 'Pilot' (which no cluster carries). @@ -10252,8 +10486,8 @@ Describe 'Function: Get-AzLocalFleetHealthOverview - v0.7.70 (ARG-first fleet he $cmd.CommandType | Should -Be 'Function' } - It 'BS7: Module exports exactly 55 functions (was 49 after Step.9 thin-YAML port; Step.6 thin-YAML port adds 6 apply-updates pipeline cmdlets)' { - (Get-Module AzLocal.UpdateManagement).ExportedFunctions.Count | Should -Be 55 + It 'BS7: Module exports exactly 60 functions (was 55 after Step.6 thin-YAML port; v0.8.7 sideload automation adds 5 cmdlets)' { + (Get-Module AzLocal.UpdateManagement).ExportedFunctions.Count | Should -Be 60 } } @@ -10642,8 +10876,8 @@ Describe 'Smoke test: Step.4 pipeline cmdlet migration (v0.7.79)' { BeforeAll { $script:repoRoot = Split-Path -Path $PSScriptRoot -Parent $script:step4Files = @( - Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/github-actions/Step.4_fleet-connectivity-status.yml' - Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/azure-devops/Step.4_fleet-connectivity-status.yml' + Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml' + Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml' ) } @@ -10689,8 +10923,8 @@ Describe 'Regression v0.7.83: Step.4 ARB inline script handles single-cluster Cl BeforeAll { $script:repoRoot = Split-Path -Path $PSScriptRoot -Parent $script:step4Cases = @( - @{ Platform = 'github-actions'; Path = (Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/github-actions/Step.4_fleet-connectivity-status.yml') } - @{ Platform = 'azure-devops'; Path = (Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/azure-devops/Step.4_fleet-connectivity-status.yml') } + @{ Platform = 'github-actions'; Path = (Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/github-actions/fleet-connectivity-status.yml') } + @{ Platform = 'azure-devops'; Path = (Join-Path -Path $script:repoRoot -ChildPath 'Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml') } ) } @@ -11912,7 +12146,7 @@ Describe 'Regression v0.7.87: bundled GitHub Actions YAML run: blocks stay under } It 'Step.4 GH YAML uses the new renderer function (no inline 22 KB markdown)' { - $step4 = Join-Path -Path $script:ghActionsDir -ChildPath 'Step.4_fleet-connectivity-status.yml' + $step4 = Join-Path -Path $script:ghActionsDir -ChildPath 'fleet-connectivity-status.yml' $raw = Get-Content -LiteralPath $step4 -Raw $raw | Should -Match 'New-AzLocalFleetConnectivityStatusSummary' ` -Because 'Step.4 must call the v0.7.87 renderer function rather than inlining the markdown summary' @@ -11921,7 +12155,7 @@ Describe 'Regression v0.7.87: bundled GitHub Actions YAML run: blocks stay under } It 'Step.4 ADO YAML uses the new renderer function (no inline 22 KB markdown)' { - $step4 = Join-Path -Path $script:repoRootForCap -ChildPath 'Automation-Pipeline-Examples/azure-devops/Step.4_fleet-connectivity-status.yml' + $step4 = Join-Path -Path $script:repoRootForCap -ChildPath 'Automation-Pipeline-Examples/azure-devops/fleet-connectivity-status.yml' $raw = Get-Content -LiteralPath $step4 -Raw $raw | Should -Match 'New-AzLocalFleetConnectivityStatusSummary' ` -Because 'Step.4 ADO must call the same v0.7.87 renderer function as the GH twin' @@ -12791,7 +13025,7 @@ schedule: $cron[$d] = @('02:00') } $md = Get-AzLocalApplyUpdatesScheduleCycleCalendar -Schedule $script:cccol_cfg -StartDate $script:cccol_wk1Mon -Days 7 -AsMarkdown -CronFiringsByDate $cron - $md | Should -Match '\| Date \(UTC\) \| Ring CRON Start Time
\(Step 6 pipeline\) \| Day \| CycleWeek \| Eligible rings \| AllowedUpdateVersions \|' + $md | Should -Match '\| Date \(UTC\) \| Ring CRON Start Time
\(apply-updates pipeline\) \| Day \| CycleWeek \| Eligible rings \| AllowedUpdateVersions \|' $md | Should -Match '\|---\|:---:\|---\|' $md | Should -Not -Match 'Tag Start Window Match' } @@ -12829,7 +13063,7 @@ schedule: } } $md = Get-AzLocalApplyUpdatesScheduleCycleCalendar -Schedule $script:cccol_cfg -StartDate $script:cccol_wk1Mon -Days 7 -AsMarkdown -CronFiringsByDate $cron -WindowMatchByRingAndDate $wm - $md | Should -Match '\| Date \(UTC\) \| Ring CRON Start Time
\(Step 6 pipeline\) \| Day \| CycleWeek \| Eligible rings \| Tag Start Window Match \(>=95%\) \| AllowedUpdateVersions \|' + $md | Should -Match '\| Date \(UTC\) \| Ring CRON Start Time
\(apply-updates pipeline\) \| Day \| CycleWeek \| Eligible rings \| Tag Start Window Match \(>=95%\) \| AllowedUpdateVersions \|' $md | Should -Match '`Cdn`: True 30/30 \(100%\)' $md | Should -Match '`Prod`: True 40/40 \(100%\)' } @@ -12899,7 +13133,7 @@ schedule: $wm = @{ Cdn = @{ ($script:cccol_wk1Mon.ToString('yyyy-MM-dd')) = @{ Matching = 30; Total = 30 } } } $md = Get-AzLocalApplyUpdatesScheduleCycleCalendar -Schedule $script:cccol_cfg -StartDate $script:cccol_wk1Mon -Days 1 -AsMarkdown ` -CronFiringsByDate $cron -ClusterRingCounts $rc -WindowMatchByRingAndDate $wm - $md | Should -Match '\| Date \(UTC\) \| Ring CRON Start Time
\(Step 6 pipeline\) \| Day \| CycleWeek \| Eligible rings \| Tag Start Window Match \(>=95%\) \| Clusters in ring\(s\) \| AllowedUpdateVersions \|' + $md | Should -Match '\| Date \(UTC\) \| Ring CRON Start Time
\(apply-updates pipeline\) \| Day \| CycleWeek \| Eligible rings \| Tag Start Window Match \(>=95%\) \| Clusters in ring\(s\) \| AllowedUpdateVersions \|' } } @@ -15777,7 +16011,7 @@ Describe 'Thin-YAML Step.3: Export-AzLocalApplyUpdatesScheduleAudit' { $env:_S3_OUTDIR = $script:_s3_outDir $env:_S3_PIPELINEDIR = $script:_s3_pipelineDir - $global:_s3_auditRows = @( (New-S3AuditRow -Status 'Uncovered' -Issue 'No cron entry matches Sat 01:00 +60' -Recommendation 'Add 55 0 * * 6 to Step.6_apply-updates.yml') ) + $global:_s3_auditRows = @( (New-S3AuditRow -Status 'Uncovered' -Issue 'No cron entry matches Sat 01:00 +60' -Recommendation 'Add 55 0 * * 6 to Step.7_apply-updates.yml') ) $result = InModuleScope AzLocal.UpdateManagement { Mock Test-AzLocalApplyUpdatesScheduleCoverage { $global:_s3_auditRows } -ParameterFilter { $View -eq 'Audit' } @@ -16027,7 +16261,7 @@ Describe 'Thin-YAML Step.3: Export-AzLocalApplyUpdatesScheduleAudit' { $env:_S3_OUTDIR = $script:_s3_outDir $env:_S3_SCHEDULE = $script:_s3_scheduleFile - # Drop a real Step.6_apply-updates.yml with two cron triggers so + # Drop a real Step.7_apply-updates.yml with two cron triggers so # Read-AzLocalApplyUpdatesYamlCrons + ConvertFrom-AzLocalCronExpression # produce non-empty FireTimes for the calendar block. $ghDir = Join-Path $script:_s3_pipelineDir 'github-actions' @@ -16038,7 +16272,7 @@ on: - cron: '0 2 * * *' - cron: '30 4 * * 1-5' workflow_dispatch: -"@ | Set-Content -Path (Join-Path $ghDir 'Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $ghDir 'Step.7_apply-updates.yml') -Encoding ASCII $env:_S3_PIPELINEDIR = $script:_s3_pipelineDir # Schedule has cycleWeeks=1 so the calendar default horizon is 7 days. @@ -16107,7 +16341,7 @@ on: on: schedule: - cron: '0 2 * * *' -"@ | Set-Content -Path (Join-Path $ghDir 'Step.6_apply-updates.yml') -Encoding ASCII +"@ | Set-Content -Path (Join-Path $ghDir 'Step.7_apply-updates.yml') -Encoding ASCII $env:_S3_PIPELINEDIR = $script:_s3_pipelineDir # Synthesise a tiny cluster CSV: 2 clusters in 'Cdn' ring, one with @@ -16776,3 +17010,509 @@ Describe 'v0.8.6 regression guard: Step.0 Export-AzLocalAuthValidationReport def ) } } + +#region On-Prem Sideloading Automation (v0.8.7) - behaviour tests for the 5 new public cmdlets + +Describe 'Sideload (v0.8.7): Update-AzLocalSideloadCatalog' { + + Context 'Offline HTML parsing + catalog merge' { + + It 'Parses CombinedSolutionBundle rows from offline HTML and writes a reviewable catalog' { + $catalogPath = Join-Path $env:TEMP ("azlocal-sideload-catalog-{0}.yml" -f ([Guid]::NewGuid())) + try { + $html = @' + + + + +
12.2605.1003.21026100.4061SHA256: ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789. Availability date: 2026-05-15
+'@ + $result = Update-AzLocalSideloadCatalog -Path $catalogPath -Html $html -Confirm:$false + + $result | Should -Not -BeNullOrEmpty + @($result).Count | Should -Be 1 + $result[0].Version | Should -Be '12.2605.1003.210' + $result[0].PackageType | Should -Be 'Solution' + $result[0].OsBuild | Should -Be '26100.4061' + $result[0].DownloadUri | Should -Be 'https://download.contoso.com/CombinedSolutionBundle.12.2605.1003.210.zip' + $result[0].Sha256 | Should -Be 'ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789' + $result[0].AvailabilityDate | Should -Be '2026-05-15' + + Test-Path -LiteralPath $catalogPath | Should -BeTrue + $written = Get-Content -LiteralPath $catalogPath -Raw + $written | Should -Match "version: '12\.2605\.1003\.210'" + $written | Should -Match 'packageType: Solution' + } + finally { + Remove-Item -LiteralPath $catalogPath -Force -ErrorAction SilentlyContinue + } + } + + It 'Preserves existing SBE (OEM) entries while merging Solution rows' { + $catalogPath = Join-Path $env:TEMP ("azlocal-sideload-catalog-{0}.yml" -f ([Guid]::NewGuid())) + try { + # Seed an SBE entry the operator authored by hand. + $seed = @' +schemaVersion: 1 +packages: + - version: '12.2605.1003.210' + packageType: SBE + sourceFolder: '\\fileserver\sbe\contoso' + sha256: '1111111111111111111111111111111111111111111111111111111111111111' +'@ + Set-Content -LiteralPath $catalogPath -Value $seed -Encoding ASCII + + $html = @' +12.2606.0.99 +SHA256 2222222222222222222222222222222222222222222222222222222222222222 Availability date: 2026-06-01 +'@ + $result = Update-AzLocalSideloadCatalog -Path $catalogPath -Html $html -Confirm:$false + + # SBE entry preserved + new Solution entry appended. + $sbe = @($result | Where-Object { $_.PackageType -eq 'SBE' }) + $sbe.Count | Should -Be 1 + $sbe[0].Version | Should -Be '12.2605.1003.210' + + $sol = @($result | Where-Object { $_.PackageType -eq 'Solution' -and $_.Version -eq '12.2606.0.99' }) + $sol.Count | Should -Be 1 + $sol[0].DownloadUri | Should -Match 'CombinedSolutionBundle\.12\.2606\.0\.99\.zip' + } + finally { + Remove-Item -LiteralPath $catalogPath -Force -ErrorAction SilentlyContinue + } + } + + It 'Does not write the catalog under -WhatIf' { + $catalogPath = Join-Path $env:TEMP ("azlocal-sideload-catalog-{0}.yml" -f ([Guid]::NewGuid())) + try { + $html = '12.2605.1003.210' + $null = Update-AzLocalSideloadCatalog -Path $catalogPath -Html $html -WhatIf + Test-Path -LiteralPath $catalogPath | Should -BeFalse -Because '-WhatIf must not write the catalog file' + } + finally { + Remove-Item -LiteralPath $catalogPath -Force -ErrorAction SilentlyContinue + } + } + } + + Context 'Schema migration (-SchemaMigrate)' { + + It '-SchemaMigrate is exposed as a [switch] parameter' { + $cmd = Get-Command -Name 'Update-AzLocalSideloadCatalog' -ErrorAction Stop + $cmd.Parameters.ContainsKey('SchemaMigrate') | Should -BeTrue + $cmd.Parameters['SchemaMigrate'].ParameterType | Should -Be ([switch]) + } + + It 'Migrating a v1 catalog is a no-op (already on the current schema) and leaves the file untouched' { + $catalogPath = Join-Path $env:TEMP ("azlocal-sideload-catalog-{0}.yml" -f ([Guid]::NewGuid())) + try { + $seed = @' +# Operator comment that must survive +schemaVersion: 1 +packages: + - version: 'DellSBE-4.1.2412.1' + packageType: SBE + sourceFolder: '\\fileserver\sbe\Dell\4.1.2412.1' +'@ + Set-Content -LiteralPath $catalogPath -Value $seed -Encoding ASCII + $before = Get-Content -LiteralPath $catalogPath -Raw + + $result = Update-AzLocalSideloadCatalog -Path $catalogPath -SchemaMigrate -Confirm:$false + + $result.Action | Should -Be 'Unchanged-SchemaCurrent' + $result.FromVersion | Should -Be 1 + $result.ToVersion | Should -Be 1 + $result.BackupPath | Should -BeNullOrEmpty + # File content unchanged; no backup written. + (Get-Content -LiteralPath $catalogPath -Raw) | Should -Be $before + $base = [System.IO.Path]::GetFileNameWithoutExtension($catalogPath) + Test-Path -LiteralPath (Join-Path (Split-Path -Parent $catalogPath) ($base + '.v1.old.yml')) | Should -BeFalse + } + finally { + Remove-Item -LiteralPath $catalogPath -Force -ErrorAction SilentlyContinue + } + } + + It '-SchemaMigrate on a missing file throws a clear error' { + $missing = Join-Path $env:TEMP ("azlocal-sideload-missing-{0}.yml" -f ([Guid]::NewGuid())) + { Update-AzLocalSideloadCatalog -Path $missing -SchemaMigrate -Confirm:$false } | + Should -Throw -ExpectedMessage '*catalog file not found*' + } + } +} + +Describe 'Sideload (v0.8.7): catalog schema framework' { + + It 'Get-AzLocalSideloadCatalog REFUSES a catalog written by a newer module (higher schemaVersion)' { + $catalogPath = Join-Path $env:TEMP ("azlocal-sideload-future-{0}.yml" -f ([Guid]::NewGuid())) + try { + $seed = @' +schemaVersion: 99 +packages: + - version: '12.2605.1003.210' + packageType: Solution + downloadUri: 'https://contoso/CombinedSolutionBundle.zip' + sha256: 'ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789' +'@ + Set-Content -LiteralPath $catalogPath -Value $seed -Encoding ASCII + { + InModuleScope AzLocal.UpdateManagement -Parameters @{ P = $catalogPath } { + param($P) + Get-AzLocalSideloadCatalog -Path $P + } + } | Should -Throw -ExpectedMessage '*only supports up to*' + } + finally { + Remove-Item -LiteralPath $catalogPath -Force -ErrorAction SilentlyContinue + } + } + + It 'Get-AzLocalSideloadCatalog tolerates a MISSING schemaVersion (treated as v1) and parses packages' { + $catalogPath = Join-Path $env:TEMP ("azlocal-sideload-nosv-{0}.yml" -f ([Guid]::NewGuid())) + try { + $seed = @' +packages: + - version: 'DellSBE-4.1.2412.1' + packageType: SBE + sourceFolder: '\\fileserver\sbe\Dell\4.1.2412.1' +'@ + Set-Content -LiteralPath $catalogPath -Value $seed -Encoding ASCII + $pkgs = @(InModuleScope AzLocal.UpdateManagement -Parameters @{ P = $catalogPath } { + param($P) + Get-AzLocalSideloadCatalog -Path $P + }) + $pkgs.Count | Should -Be 1 + $pkgs[0].Version | Should -Be 'DellSBE-4.1.2412.1' + } + finally { + Remove-Item -LiteralPath $catalogPath -Force -ErrorAction SilentlyContinue + } + } + + It 'Convert-AzLocalSideloadCatalogSchemaVersion is a no-op at the current schema version' { + $r = InModuleScope AzLocal.UpdateManagement { + $text = "schemaVersion: $($script:SideloadCatalogSchemaCurrentVersion)`r`npackages:`r`n" + Convert-AzLocalSideloadCatalogSchemaVersion -Text $text + } + $r.Migrated | Should -BeFalse + $r.Hops.Count | Should -Be 0 + } + + It 'Convert-AzLocalSideloadCatalogSchemaVersion throws a wired-up error when a hop is missing (empty recipe table at v1)' { + { + InModuleScope AzLocal.UpdateManagement { + # Force a target one above current; the v1 recipe table is empty + # so the engine must report the missing '1->2' hop. + Convert-AzLocalSideloadCatalogSchemaVersion -Text "schemaVersion: 1`r`npackages:`r`n" -TargetSchemaVersion 2 + } + } | Should -Throw -ExpectedMessage "*no migration recipe registered for '1->2'*" + } +} + +Describe 'Sideload (v0.8.7): Resolve-AzLocalSideloadPlan' { + + It 'Flags clusters whose UpdateAuthAccountId is not in the auth map' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalApplyUpdatesScheduleConfig { [PSCustomObject]@{} } + Mock Get-AzLocalSideloadAuthMap { @{} } # empty -> any account id is unknown + Mock Get-AzLocalSideloadCatalog { @() } + Mock ConvertTo-AzLocalUpdateRingKqlFilter { '' } + Mock Get-AzLocalApplyUpdatesScheduleNextFirings { @() } + Mock Invoke-AzResourceGraphQuery { + @([PSCustomObject]@{ + id = '/subscriptions/s/clusters/c1' + name = 'cluster1' + resourceGroup = 'rg1' + subscriptionId = 's' + tags = [PSCustomObject]@{ UpdateAuthAccountId = 'missing-acct'; UpdateRing = 'Ring1' } + }) + } + + $plan = Resolve-AzLocalSideloadPlan -SchedulePath 'sched.yml' -AuthMapPath 'auth.csv' -CatalogPath 'cat.yml' + + @($plan).Count | Should -Be 1 + $plan[0].Status | Should -Be 'UnknownAuthAccountId' + $plan[0].ClusterName | Should -Be 'cluster1' + } + } + + It 'Emits a Planned row when the cluster is due and the selected version is in the catalog' { + InModuleScope AzLocal.UpdateManagement { + $script:PlanNow = [datetime]'2026-06-11T00:00:00Z' + + Mock Get-AzLocalApplyUpdatesScheduleConfig { [PSCustomObject]@{} } + Mock Get-AzLocalSideloadAuthMap { + @{ 'acct1' = [PSCustomObject]@{ RemotingTargetFqdn = 'host1.contoso.com'; ImportSharePath = '\\host1\import'; AuthMechanism = 'Negotiate' } } + } + Mock Get-AzLocalSideloadCatalog { + @([PSCustomObject]@{ Version = '12.2605.1003.210'; PackageType = 'Solution' }) + } + Mock ConvertTo-AzLocalUpdateRingKqlFilter { '' } + Mock Resolve-AzLocalSideloadTargetPath { '\\host1\import' } + Mock Resolve-AzLocalCurrentUpdateRing { [PSCustomObject]@{ AllowedUpdateVersions = @('12.2605.1003.210') } } + Mock Get-AzLocalAvailableUpdates { + @([PSCustomObject]@{ UpdateState = 'Ready'; UpdateName = 'Solution12'; PackageType = 'Solution'; Version = '12.2605.1003.210' }) + } + Mock Select-AzLocalNextUpdateForCluster { + [PSCustomObject]@{ + Reason = 'Selected' + SelectedUpdate = [PSCustomObject]@{ name = 'Solution12'; PackageType = 'Solution'; properties = [PSCustomObject]@{ version = '12.2605.1003.210'; state = 'Ready' } } + } + } + Mock Get-AzLocalApplyUpdatesScheduleNextFirings { + @([PSCustomObject]@{ Rings = @('Ring1'); DateUtc = $script:PlanNow.AddDays(2) }) + } + Mock Invoke-AzResourceGraphQuery { + @([PSCustomObject]@{ + id = '/subscriptions/s/clusters/c1' + name = 'cluster1' + resourceGroup = 'rg1' + subscriptionId = 's' + tags = [PSCustomObject]@{ UpdateAuthAccountId = 'acct1'; UpdateRing = 'Ring1' } + }) + } + + # NextWindow = Now+2, LeadDays=7 -> dueDate = Now-5 <= Now -> DueNow -> Planned + $plan = Resolve-AzLocalSideloadPlan -SchedulePath 'sched.yml' -AuthMapPath 'auth.csv' -CatalogPath 'cat.yml' -LeadDays 7 -Now $script:PlanNow + + @($plan).Count | Should -Be 1 + $plan[0].Status | Should -Be 'Planned' + $plan[0].DueNow | Should -BeTrue + $plan[0].SelectedVersion | Should -Be '12.2605.1003.210' + $plan[0].CatalogEntry | Should -Not -BeNullOrEmpty + } + } + + It 'Flags NoCatalogEntry when the selected version is absent from the catalog' { + InModuleScope AzLocal.UpdateManagement { + $script:PlanNow2 = [datetime]'2026-06-11T00:00:00Z' + + Mock Get-AzLocalApplyUpdatesScheduleConfig { [PSCustomObject]@{} } + Mock Get-AzLocalSideloadAuthMap { + @{ 'acct1' = [PSCustomObject]@{ RemotingTargetFqdn = 'host1.contoso.com'; ImportSharePath = '\\host1\import' } } + } + Mock Get-AzLocalSideloadCatalog { @() } # empty catalog + Mock ConvertTo-AzLocalUpdateRingKqlFilter { '' } + Mock Resolve-AzLocalSideloadTargetPath { '\\host1\import' } + Mock Resolve-AzLocalCurrentUpdateRing { [PSCustomObject]@{ AllowedUpdateVersions = @('12.2605.1003.210') } } + Mock Get-AzLocalAvailableUpdates { + @([PSCustomObject]@{ UpdateState = 'Ready'; UpdateName = 'Solution12'; PackageType = 'Solution'; Version = '12.2605.1003.210' }) + } + Mock Select-AzLocalNextUpdateForCluster { + [PSCustomObject]@{ + Reason = 'Selected' + SelectedUpdate = [PSCustomObject]@{ name = 'Solution12'; PackageType = 'Solution'; properties = [PSCustomObject]@{ version = '12.2605.1003.210'; state = 'Ready' } } + } + } + Mock Get-AzLocalApplyUpdatesScheduleNextFirings { + @([PSCustomObject]@{ Rings = @('Ring1'); DateUtc = $script:PlanNow2.AddDays(2) }) + } + Mock Invoke-AzResourceGraphQuery { + @([PSCustomObject]@{ + id = '/subscriptions/s/clusters/c1' + name = 'cluster1' + resourceGroup = 'rg1' + subscriptionId = 's' + tags = [PSCustomObject]@{ UpdateAuthAccountId = 'acct1'; UpdateRing = 'Ring1' } + }) + } + + $plan = Resolve-AzLocalSideloadPlan -SchedulePath 'sched.yml' -AuthMapPath 'auth.csv' -CatalogPath 'cat.yml' -LeadDays 7 -Now $script:PlanNow2 + + @($plan).Count | Should -Be 1 + $plan[0].Status | Should -Be 'NoCatalogEntry' + } + } +} + +Describe 'Sideload (v0.8.7): Invoke-AzLocalSideloadUpdate state machine' { + + It 'Skips a cluster that has no state and is not due (Status != Planned)' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalSideloadState { $null } + Mock Invoke-SideloadCopyStart { [PSCustomObject]@{ Retries = 0 } } + + $plan = @([PSCustomObject]@{ ClusterName = 'c1'; Status = 'NotDue'; SelectedVersion = '12.0'; Message = 'not yet' }) + $res = Invoke-AzLocalSideloadUpdate -Plan $plan -StateRoot 'TestDrive:\state' -Confirm:$false + + @($res).Count | Should -Be 1 + $res[0].Action | Should -Be 'Skip' + Assert-MockCalled Invoke-SideloadCopyStart -Times 0 -Scope It + } + } + + It 'Starts the copy for a due cluster with no existing state' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalSideloadState { $null } + Mock Invoke-SideloadCopyStart { [PSCustomObject]@{ Retries = 0; MediaFileName = 'media.zip' } } + + $plan = @([PSCustomObject]@{ ClusterName = 'c1'; Status = 'Planned'; SelectedVersion = '12.2605.1003.210'; Message = 'due' }) + $res = Invoke-AzLocalSideloadUpdate -Plan $plan -StateRoot 'TestDrive:\state' -Confirm:$false + + $res[0].Action | Should -Be 'Start' + $res[0].State | Should -Be 'Copying' + Assert-MockCalled Invoke-SideloadCopyStart -Times 1 -Scope It + } + } + + It 'Reports None when the cluster is already Imported' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalSideloadState { + [PSCustomObject]@{ ClusterName = 'c1'; State = 'Imported'; Version = '12.0'; Message = 'done' } + } + + $plan = @([PSCustomObject]@{ ClusterName = 'c1'; Status = 'Planned'; SelectedVersion = '12.0' }) + $res = Invoke-AzLocalSideloadUpdate -Plan $plan -StateRoot 'TestDrive:\state' -Confirm:$false + + $res[0].Action | Should -Be 'None' + $res[0].State | Should -Be 'Imported' + } + } + + It 'Reports InProgress for a Copying state with a fresh heartbeat' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalSideloadState { + [PSCustomObject]@{ ClusterName = 'c1'; State = 'Copying'; Version = '12.0'; TotalBytes = [long]100; CopiedBytes = [long]50; Mbps = 42; EtaUtc = '2026-06-11T01:00:00Z'; Retries = 0; Message = 'copying' } + } + Mock Test-AzLocalSideloadHeartbeatStale { $false } + + $plan = @([PSCustomObject]@{ ClusterName = 'c1'; Status = 'Planned'; SelectedVersion = '12.0' }) + $res = Invoke-AzLocalSideloadUpdate -Plan $plan -StateRoot 'TestDrive:\state' -Confirm:$false + + $res[0].Action | Should -Be 'InProgress' + $res[0].State | Should -Be 'Copying' + $res[0].Message | Should -Match '50' + } + } + + It 'Marks Failed when a Copying heartbeat is stale and retries are exhausted' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalSideloadState { + [PSCustomObject]@{ ClusterName = 'c1'; State = 'Copying'; Version = '12.0'; Retries = [int]3; Message = '' } + } + Mock Test-AzLocalSideloadHeartbeatStale { $true } + Mock Set-AzLocalSideloadState { } + + $plan = @([PSCustomObject]@{ ClusterName = 'c1'; Status = 'Planned'; SelectedVersion = '12.0' }) + $res = Invoke-AzLocalSideloadUpdate -Plan $plan -StateRoot 'TestDrive:\state' -MaxRetries 3 -Confirm:$false + + $res[0].Action | Should -Be 'Failed' + $res[0].State | Should -Be 'Failed' + Assert-MockCalled Set-AzLocalSideloadState -Times 1 -Scope It + } + } + + It 'Advances a Copied state through the import + gate-flip helper' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalSideloadState { + [PSCustomObject]@{ ClusterName = 'c1'; State = 'Copied'; Version = '12.0' } + } + Mock Complete-SideloadImport { [PSCustomObject]@{ State = 'Imported'; Message = 'Imported; UpdateSideloaded=True.' } } + + $plan = @([PSCustomObject]@{ ClusterName = 'c1'; Status = 'Planned'; SelectedVersion = '12.0' }) + $res = Invoke-AzLocalSideloadUpdate -Plan $plan -StateRoot 'TestDrive:\state' -Confirm:$false + + $res[0].Action | Should -Be 'Import' + $res[0].State | Should -Be 'Imported' + Assert-MockCalled Complete-SideloadImport -Times 1 -Scope It + } + } + + It 'Surfaces an Error row when a helper throws' { + InModuleScope AzLocal.UpdateManagement { + Mock Get-AzLocalSideloadState { $null } + Mock Invoke-SideloadCopyStart { throw 'media staging failed' } + + $plan = @([PSCustomObject]@{ ClusterName = 'c1'; Status = 'Planned'; SelectedVersion = '12.0' }) + $res = Invoke-AzLocalSideloadUpdate -Plan $plan -StateRoot 'TestDrive:\state' -Confirm:$false + + $res[0].Action | Should -Be 'Error' + $res[0].State | Should -Be 'Failed' + $res[0].Message | Should -Match 'media staging failed' + } + } +} + +Describe 'Sideload (v0.8.7): Export-AzLocalSideloadStatusReport' { + + BeforeEach { + $script:SideloadStateRoot = Join-Path $env:TEMP ("azlocal-sideload-state-{0}" -f ([Guid]::NewGuid())) + $script:SideloadStateDir = Join-Path $script:SideloadStateRoot 'state' + New-Item -ItemType Directory -Path $script:SideloadStateDir -Force | Out-Null + } + + AfterEach { + Remove-Item -LiteralPath $script:SideloadStateRoot -Recurse -Force -ErrorAction SilentlyContinue + } + + It 'Builds a markdown table and JUnit XML from shared state records' { + $copying = [PSCustomObject]@{ ClusterName = 'clusterA'; Version = '12.0'; State = 'Copying'; OwningMachine = 'agent1'; TotalBytes = [long]100; CopiedBytes = [long]25; Mbps = 30; EtaUtc = '2026-06-11T02:00:00Z'; Retries = 0; Message = 'in progress' } + $imported = [PSCustomObject]@{ ClusterName = 'clusterB'; Version = '12.0'; State = 'Imported'; OwningMachine = 'agent2'; TotalBytes = [long]0; CopiedBytes = [long]0; Mbps = 0; EtaUtc = ''; Retries = 0; Message = 'done' } + ($copying | ConvertTo-Json) | Set-Content -LiteralPath (Join-Path $script:SideloadStateDir 'clusterA.json') -Encoding ASCII + ($imported | ConvertTo-Json) | Set-Content -LiteralPath (Join-Path $script:SideloadStateDir 'clusterB.json') -Encoding ASCII + + $report = Export-AzLocalSideloadStatusReport -StateRoot $script:SideloadStateRoot + + $report.Markdown | Should -Match '## Sideload status' + $report.Markdown | Should -Match 'clusterA' + $report.Markdown | Should -Match 'clusterB' + $report.JUnitXml | Should -Match '= 8 is failure. + +.PARAMETER ClusterName + The Azure Local cluster name (used to name the shared-state JSON file). + +.PARAMETER Version + The solution-update version being copied (recorded in state). + +.PARAMETER SourcePath + The verified media path: a .zip file (Solution) or a folder (SBE). + +.PARAMETER TargetPath + The destination UNC folder on the cluster's import share. + +.PARAMETER StateRoot + The SHARED UNC state root. state\.json and logs\ live beneath it. + +.PARAMETER RobocopySwitches + Extra robocopy switches (e.g. '/R:5 /W:30 /IPG:50'); space-separated. + +.PARAMETER HeartbeatSeconds + Interval between heartbeat JSON updates. Default 30. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$ClusterName, + [Parameter(Mandatory = $true)][string]$Version, + [Parameter(Mandatory = $true)][string]$SourcePath, + [Parameter(Mandatory = $true)][string]$TargetPath, + [Parameter(Mandatory = $true)][string]$StateRoot, + [string]$RobocopySwitches = '/R:5 /W:30', + [int]$HeartbeatSeconds = 30 +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Get-StateFilePath { + param([string]$Root, [string]$Cluster) + $stateDir = Join-Path -Path $Root -ChildPath 'state' + if (-not (Test-Path -LiteralPath $stateDir)) { New-Item -ItemType Directory -Path $stateDir -Force | Out-Null } + $safe = ($Cluster -replace '[^A-Za-z0-9._-]', '_') + return (Join-Path -Path $stateDir -ChildPath ("{0}.json" -f $safe)) +} + +function Write-StateAtomic { + param([string]$Path, [PSCustomObject]$State) + $json = $State | ConvertTo-Json -Depth 6 + $temp = "{0}.{1}.partial" -f $Path, ([guid]::NewGuid().ToString('N')) + try { + [System.IO.File]::WriteAllText($temp, $json, [System.Text.UTF8Encoding]::new($false)) + Move-Item -LiteralPath $temp -Destination $Path -Force + } + finally { + if (Test-Path -LiteralPath $temp -PathType Leaf) { Remove-Item -LiteralPath $temp -Force -ErrorAction SilentlyContinue } + } +} + +function Get-FolderSize { + param([string]$Path) + if (-not (Test-Path -LiteralPath $Path)) { return [long]0 } + if (Test-Path -LiteralPath $Path -PathType Leaf) { + return [long]((Get-Item -LiteralPath $Path).Length) + } + $sum = (Get-ChildItem -LiteralPath $Path -Recurse -File -ErrorAction SilentlyContinue | Measure-Object -Property Length -Sum).Sum + if ($null -eq $sum) { return [long]0 } + return [long]$sum +} + +$statePath = Get-StateFilePath -Root $StateRoot -Cluster $ClusterName +$logDir = Join-Path -Path $StateRoot -ChildPath 'logs' +if (-not (Test-Path -LiteralPath $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null } +$safeName = ($ClusterName -replace '[^A-Za-z0-9._-]', '_') +$logPath = Join-Path -Path $logDir -ChildPath ("{0}.{1}.robocopy.log" -f $safeName, ([DateTime]::UtcNow.ToString('yyyyMMddHHmmss'))) + +# Carry forward existing fields (retries, taskname) when present. +$existing = $null +if (Test-Path -LiteralPath $statePath -PathType Leaf) { + try { $existing = (Get-Content -LiteralPath $statePath -Raw | ConvertFrom-Json) } catch { $existing = $null } +} + +$startUtc = [DateTime]::UtcNow +$totalBytes = Get-FolderSize -Path $SourcePath + +$state = [PSCustomObject]@{ + ClusterName = $ClusterName + Version = $Version + State = 'Copying' + OwningMachine = $env:COMPUTERNAME + TaskName = if ($existing) { [string]$existing.TaskName } else { '' } + MediaPath = $SourcePath + TargetPath = $TargetPath + LogPath = $logPath + StartUtc = $startUtc.ToString('o') + LastHeartbeatUtc = $startUtc.ToString('o') + TotalBytes = $totalBytes + CopiedBytes = [long]0 + Mbps = [double]0 + EtaUtc = '' + ExitCode = $null + Retries = if ($existing) { [int]$existing.Retries } else { [int]0 } + Message = 'Copy started.' +} +Write-StateAtomic -Path $statePath -State $state + +# Ensure destination exists. +if (-not (Test-Path -LiteralPath $TargetPath)) { New-Item -ItemType Directory -Path $TargetPath -Force | Out-Null } + +# Build the robocopy argument list. For a single-file source, robocopy copies +# from the parent directory with the file name as a selector; for a folder it +# mirrors the tree. +if (Test-Path -LiteralPath $SourcePath -PathType Leaf) { + $srcDir = Split-Path -Path $SourcePath -Parent + $srcFile = Split-Path -Path $SourcePath -Leaf + $roboArgs = @($srcDir, $TargetPath, $srcFile) +} +else { + $roboArgs = @($SourcePath, $TargetPath, '/E') +} +$roboArgs += ($RobocopySwitches -split '\s+' | Where-Object { $_ }) +$roboArgs += @('/NP', '/NJH', '/NJS', ("/LOG:{0}" -f $logPath)) + +$proc = Start-Process -FilePath 'robocopy.exe' -ArgumentList $roboArgs -PassThru +while (-not $proc.HasExited) { + Start-Sleep -Seconds $HeartbeatSeconds + $copied = Get-FolderSize -Path $TargetPath + $elapsed = ([DateTime]::UtcNow - $startUtc).TotalSeconds + $mbps = if ($elapsed -gt 0) { [math]::Round((($copied * 8) / 1MB) / $elapsed, 2) } else { 0 } + $eta = '' + if ($copied -gt 0 -and $totalBytes -gt $copied -and $elapsed -gt 0) { + $bytesPerSec = $copied / $elapsed + if ($bytesPerSec -gt 0) { + $remainingSec = ($totalBytes - $copied) / $bytesPerSec + $eta = [DateTime]::UtcNow.AddSeconds($remainingSec).ToString('o') + } + } + $state.CopiedBytes = [long]$copied + $state.Mbps = [double]$mbps + $state.EtaUtc = $eta + $state.LastHeartbeatUtc = [DateTime]::UtcNow.ToString('o') + $state.Message = 'Copy in progress.' + Write-StateAtomic -Path $statePath -State $state +} + +$exit = $proc.ExitCode +$state.ExitCode = $exit +$state.CopiedBytes = Get-FolderSize -Path $TargetPath +$state.LastHeartbeatUtc = [DateTime]::UtcNow.ToString('o') +if ($exit -lt 8) { + $state.State = 'Copied' + $state.Message = ("Copy completed (robocopy exit {0})." -f $exit) +} +else { + $state.State = 'Failed' + $state.Message = ("Copy FAILED (robocopy exit {0}); see log {1}." -f $exit, $logPath) +} +Write-StateAtomic -Path $statePath -State $state + +# Surface a non-zero process exit only on real failure (>= 8). +if ($exit -ge 8) { exit 1 } else { exit 0 } diff --git a/AzLocal.UpdateManagement/Tools/SMOKE-COVERAGE.md b/AzLocal.UpdateManagement/Tools/SMOKE-COVERAGE.md index c4dc7b3c..56045bd2 100644 --- a/AzLocal.UpdateManagement/Tools/SMOKE-COVERAGE.md +++ b/AzLocal.UpdateManagement/Tools/SMOKE-COVERAGE.md @@ -19,6 +19,7 @@ exercised in a real GHA / ADO run. | `Step.4_fleet-connectivity-status.yml` | `Get-AzLocalFleetConnectivityStatus` | [validate-arg-queries.ps1](validate-arg-queries.ps1) | [smoke-test-connectivity-status.ps1](smoke-test-connectivity-status.ps1) | | `Step.5_assess-update-readiness.yml` | `Get-AzLocalClusterInventory`, `Get-AzLocalClusterUpdateReadiness`, `Test-AzLocalClusterHealth -BlockingOnly` | [validate-arg-queries.ps1](validate-arg-queries.ps1) | [smoke-test-assess-readiness.ps1](smoke-test-assess-readiness.ps1) | | `Step.6_apply-updates.yml` | `Get-AzLocalClusterUpdateReadiness` (read), `Start-AzStackHciUpdate` (write) | [validate-arg-queries.ps1](validate-arg-queries.ps1) covers the read | n/a (write path; readiness already covered) | +| `sideload-updates.yml` (v0.8.7 Step.6) | `Resolve-AzLocalSideloadPlan` (read; wraps the schedule, auth-map + catalog parsers and the cluster ARG query), `Invoke-AzLocalSideloadUpdate` (write) | [validate-arg-queries.ps1](validate-arg-queries.ps1) covers the planner read | [smoke-test-sideload-plan.ps1](smoke-test-sideload-plan.ps1) | | `Step.7_monitor-updates.yml` (v0.7.90) | `Get-AzLocalClusterInventory`, `Get-AzLocalUpdateRuns -Latest` | [validate-arg-queries.ps1](validate-arg-queries.ps1) | [smoke-test-monitor-updates.ps1](smoke-test-monitor-updates.ps1) | | `Step.8_fleet-update-status.yml` | `Get-AzLocalClusterInventory`, `Get-AzLocalClusterUpdateReadiness`, `Get-AzLocalLatestSolutionVersion`, `Get-AzLocalUpdateRunFailures`, `Get-AzLocalUpdateSummary`, `Get-AzLocalAvailableUpdates`, `Get-AzLocalUpdateRuns -Latest` | [validate-arg-queries.ps1](validate-arg-queries.ps1) | [smoke-test-fleet-update-status.ps1](smoke-test-fleet-update-status.ps1) | | `Step.9_fleet-health-status.yml` | `Get-AzLocalFleetHealthOverview`, `Get-AzLocalFleetHealthFailures -View Detail` | [validate-arg-queries.ps1](validate-arg-queries.ps1) | [smoke-test-fleet-health-status.ps1](smoke-test-fleet-health-status.ps1) | @@ -57,6 +58,23 @@ Exit code is `1` if any section is `FAIL-SCHEMA` or `ERROR`, `0` otherwise. - Pass `-SubscriptionId` to override the default subscription (or use `az account set --subscription ` beforehand) +### Sideloading (Step.6) - no fleet tags required + +`smoke-test-sideload-plan.ps1` and the `Resolve-AzLocalSideloadPlan` section in +`validate-arg-queries.ps1` synthesise their own minimal auth-map + catalog temp +files (the auth-map and catalog are operator-authored config, not ARG sources, +so there is no bundled live example). They point the planner at the bundled +`apply-updates-schedule.example.yml`. + +If the fleet has **no clusters tagged with `UpdateAuthAccountId`** (the normal +state before an operator opts a cluster in to sideloading), the planner returns +**zero rows**, which is recorded as **`PASS-EMPTY`**. That is the expected, +healthy result and still validates the full Step.6 read path: the cluster ARG +query parsed + executed, and the schedule / auth-map / catalog parsers ran +without throwing. A `PASS` (rows + required columns present) only appears once +real `UpdateAuthAccountId` tags exist in the fleet. + + ## Adding a new pipeline When introducing a new `Step.N_*.yml` pipeline: diff --git a/AzLocal.UpdateManagement/Tools/smoke-test-sideload-plan.ps1 b/AzLocal.UpdateManagement/Tools/smoke-test-sideload-plan.ps1 new file mode 100644 index 00000000..fbb25016 --- /dev/null +++ b/AzLocal.UpdateManagement/Tools/smoke-test-sideload-plan.ps1 @@ -0,0 +1,186 @@ +# --------------------------------------------------------------------------- +# Smoke test for the Step.6 sideload-updates pipeline (v0.8.7). +# +# Validates the EXPORTED planner cmdlet Resolve-AzLocalSideloadPlan, which is +# the read-only data source the sideload-updates.yml pipeline consumes before +# Invoke-AzLocalSideloadUpdate mutates any cluster. A single call exercises the +# whole Step.6 read path end to end: +# +# 1. Get-AzLocalApplyUpdatesScheduleConfig - parses the apply-updates schedule +# 2. Get-AzLocalSideloadAuthMap (private) - parses the auth-map CSV +# 3. Get-AzLocalSideloadCatalog (private) - parses the catalog YAML +# 4. Invoke-AzResourceGraphQuery - the live ARG cluster query +# ("resources | where type =~ 'microsoft.azurestackhci/clusters' +# | where isnotempty(tags['UpdateAuthAccountId']) ...") +# +# So any ARG-query regression, schedule/auth/catalog parser drift, or plan-row +# schema change is caught here before the Step.6 YAML is exercised in a real +# GHA / ADO run - matching the coverage the other Step.N smoke tests provide. +# +# The auth-map and catalog are OPERATOR-authored (not ARG sources) and have no +# bundled live example, so this harness synthesises minimal valid temp files +# (an empty pair AND a populated pair) and points the cmdlet at the bundled +# apply-updates-schedule.example.yml. On an empty / un-tagged fleet the plan is +# legitimately empty (PASS-EMPTY) - that still validates the ARG query parsed +# and executed and the three config parsers did not throw. +# +# Requires: `az login`; signed-in identity has Reader on the target +# subscription(s). Resolve-AzLocalSideloadPlan handles ARG pagination + retry. +# --------------------------------------------------------------------------- +[CmdletBinding()] +param( + [string]$SubscriptionId, + [string]$ModulePath, + [string]$SchedulePath, + [string]$AuthMapPath, + [string]$CatalogPath +) +$ErrorActionPreference = 'Stop' + +if (-not $ModulePath) { + $ModulePath = Join-Path -Path $PSScriptRoot -ChildPath '..\AzLocal.UpdateManagement.psd1' +} +if (-not (Test-Path $ModulePath)) { throw "Module manifest not found at: $ModulePath" } +if (-not $SchedulePath) { + $SchedulePath = Join-Path -Path $PSScriptRoot -ChildPath '..\Automation-Pipeline-Examples\apply-updates-schedule.example.yml' +} +if (-not (Test-Path $SchedulePath)) { + throw "Schedule example file not found at: $SchedulePath - pass -SchedulePath to override" +} + +Write-Host "Importing module: $ModulePath" -ForegroundColor Cyan +Get-Module AzLocal.UpdateManagement -All | Remove-Module -Force -ErrorAction SilentlyContinue +Import-Module $ModulePath -Force -ErrorAction Stop +$moduleVersion = (Get-Module AzLocal.UpdateManagement | Sort-Object Version -Descending | Select-Object -First 1).Version +Write-Host "Module version: $moduleVersion" -ForegroundColor Cyan +Write-Host "Schedule file : $SchedulePath" -ForegroundColor Cyan + +# Resolve-AzLocalSideloadPlan runs an ARG query, so az is REQUIRED here. +try { $null = Get-Command az -ErrorAction Stop } +catch { throw 'az CLI is not on PATH. Install Azure CLI and run `az login` before running this smoke test.' } +$accountJson = & az account show -o json 2>$null +if ($LASTEXITCODE -ne 0 -or -not $accountJson) { + throw 'az account show failed. Run `az login` and try again.' +} +$account = $accountJson | ConvertFrom-Json +Write-Host "Signed-in subscription: $($account.name) ($($account.id))" -ForegroundColor Cyan +if ($SubscriptionId -and ($account.id -ne $SubscriptionId)) { + Write-Host "Switching to subscription $SubscriptionId ..." -ForegroundColor Yellow + & az account set --subscription $SubscriptionId +} + +# Plan-row schema the Step.6 pipeline consumes (Resolve-AzLocalSideloadPlan.ps1 +# ~L88 [ordered]@{...}). Keep in lock-step with that builder. +$planColumns = @( + 'ClusterName', 'ClusterResourceId', 'ResourceGroup', 'SubscriptionId', + 'UpdateAuthAccountId', 'Ring', 'NextWindowUtc', 'LeadDays', 'DueNow', + 'SelectedVersion', 'SelectedUpdateName', 'PackageType', 'CatalogEntry', + 'RemotingHost', 'TargetPath', 'AuthRow', 'Status', 'Message' +) + +$results = [System.Collections.Generic.List[object]]::new() +function Test-Cmdlet { + param([string]$Name, [scriptblock]$Invoke, [string[]]$RequiredColumns) + Write-Host "`n=== $Name ===" -ForegroundColor Cyan + try { + $rows = & $Invoke + $rowCount = @($rows).Count + Write-Host "Returned $rowCount row(s)" -ForegroundColor Yellow + if ($rowCount -gt 0 -and $RequiredColumns -and $RequiredColumns.Count -gt 0) { + $cols = $rows[0].PSObject.Properties.Name + $missing = @($RequiredColumns | Where-Object { $cols -notcontains $_ }) + if ($missing.Count -eq 0) { + Write-Host "All required columns present ($($RequiredColumns.Count))" -ForegroundColor Green + $results.Add([PSCustomObject]@{ Cmdlet = $Name; Status = 'PASS'; Rows = $rowCount; Missing = ''; Error = '' }) + } + else { + Write-Host "MISSING columns: $($missing -join ', ')" -ForegroundColor Red + Write-Host " Actual columns: $($cols -join ', ')" -ForegroundColor DarkGray + $results.Add([PSCustomObject]@{ Cmdlet = $Name; Status = 'FAIL-SCHEMA'; Rows = $rowCount; Missing = ($missing -join ','); Error = '' }) + } + } + elseif ($rowCount -gt 0) { + Write-Host "Returned $rowCount row(s) (no required-column assertion)" -ForegroundColor Green + $results.Add([PSCustomObject]@{ Cmdlet = $Name; Status = 'PASS'; Rows = $rowCount; Missing = ''; Error = '' }) + } + else { + Write-Host "Returned 0 rows (still validates ARG query parse + config parsers)" -ForegroundColor Yellow + $results.Add([PSCustomObject]@{ Cmdlet = $Name; Status = 'PASS-EMPTY'; Rows = 0; Missing = ''; Error = '' }) + } + } + catch { + Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red + $results.Add([PSCustomObject]@{ Cmdlet = $Name; Status = 'ERROR'; Rows = 0; Missing = ''; Error = $_.Exception.Message }) + } +} + +# Synthesise temp auth-map + catalog files when the caller did not supply them. +# These are operator-authored config (not ARG), so there is no live example to +# point at - minimal valid content is enough to validate the parser wiring. +$tmpDir = New-Item -ItemType Directory -Path (Join-Path $env:TEMP "smoke-step6-$(Get-Random)") -Force +$cleanupTemp = $true +try { + if (-not $AuthMapPath) { + $AuthMapPath = Join-Path $tmpDir.FullName 'sideload-auth-map.csv' + @( + 'UpdateAuthAccountId,KeyVaultName,UsernameSecretName,PasswordSecretName,RemotingTargetFqdn,FqdnSuffix,AuthMechanism,ImportSharePath' + '001,kv-smoke,sideload-user,sideload-pass,,.smoke.contoso.com,Negotiate,' + ) | Set-Content -LiteralPath $AuthMapPath -Encoding ASCII + } + if (-not $CatalogPath) { + $CatalogPath = Join-Path $tmpDir.FullName 'sideload-catalog.yml' + @( + 'schemaVersion: 1' + 'packages:' + " - version: '12.2605.1003.210'" + ' packageType: Solution' + ' osBuild: ''26100.4061''' + " downloadUri: 'https://download.contoso.com/CombinedSolutionBundle.12.2605.1003.210.zip'" + " sha256: 'ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789'" + ) | Set-Content -LiteralPath $CatalogPath -Encoding ASCII + } + else { + # Caller supplied a catalog path; do not delete it on cleanup. + } + Write-Host "Auth-map file : $AuthMapPath" -ForegroundColor Cyan + Write-Host "Catalog file : $CatalogPath" -ForegroundColor Cyan + + $commonArgs = @{ SchedulePath = $SchedulePath; AuthMapPath = $AuthMapPath; CatalogPath = $CatalogPath } + if ($SubscriptionId) { $commonArgs.SubscriptionId = $SubscriptionId } + + # 1. Default plan resolve (all rings) - the primary Step.6 read call. + Test-Cmdlet -Name 'Resolve-AzLocalSideloadPlan (all rings)' ` + -Invoke { Resolve-AzLocalSideloadPlan @commonArgs } ` + -RequiredColumns $planColumns + + # 2. Plan resolve with an explicit lead window - exercises the + # next-firing / DueNow date arithmetic against the live schedule. + Test-Cmdlet -Name 'Resolve-AzLocalSideloadPlan -LeadDays 14' ` + -Invoke { Resolve-AzLocalSideloadPlan @commonArgs -LeadDays 14 } ` + -RequiredColumns $planColumns +} +finally { + if ($cleanupTemp) { + Remove-Item -Path $tmpDir.FullName -Recurse -Force -ErrorAction SilentlyContinue + } +} + +# --------------------------------------------------------------------------- +# Final report. +# --------------------------------------------------------------------------- +Write-Host "`n========================================" -ForegroundColor Cyan +Write-Host "Smoke Test Results - Resolve-AzLocalSideloadPlan (Step.6)" -ForegroundColor Cyan +Write-Host "Module version: $moduleVersion" -ForegroundColor Cyan +Write-Host "Subscription: $($account.name) ($($account.id))" -ForegroundColor Cyan +Write-Host "========================================`n" -ForegroundColor Cyan +$results | Format-Table -AutoSize Cmdlet, Status, Rows, Missing, Error | Out-String -Width 200 | Write-Host + +$fail = @($results | Where-Object { $_.Status -in @('FAIL-SCHEMA', 'ERROR') }) +if ($fail.Count -gt 0) { + Write-Host "FAILED: $($fail.Count) section(s) did not pass." -ForegroundColor Red + exit 1 +} +else { + Write-Host "All sections passed." -ForegroundColor Green + exit 0 +} diff --git a/AzLocal.UpdateManagement/Tools/validate-arg-queries.ps1 b/AzLocal.UpdateManagement/Tools/validate-arg-queries.ps1 index 56189211..ff05a150 100644 --- a/AzLocal.UpdateManagement/Tools/validate-arg-queries.ps1 +++ b/AzLocal.UpdateManagement/Tools/validate-arg-queries.ps1 @@ -173,6 +173,45 @@ Test-Cmdlet -Name 'Test-AzLocalApplyUpdatesScheduleCoverage' ` } ` -RequiredColumns @() +# 13. Resolve-AzLocalSideloadPlan (Step.6 sideload-updates; v0.8.7 NEW) +# Read-only planner. Its ARG query selects clusters carrying an +# UpdateAuthAccountId tag; the auth-map + catalog are operator-authored config +# with no bundled live example, so we synthesise minimal valid temp files. On a +# fleet with no UpdateAuthAccountId-tagged clusters the plan is legitimately +# empty (PASS-EMPTY) - that still validates the ARG query parsed/executed and +# the schedule/auth/catalog parsers did not throw. Dedicated, sequence-faithful +# coverage lives in Tools/smoke-test-sideload-plan.ps1. +Test-Cmdlet -Name 'Resolve-AzLocalSideloadPlan' ` + -Invoke { + $scheduleFile = 'C:\Users\nebird\Repos\Azure-Local\AzLocal.UpdateManagement\Automation-Pipeline-Examples\apply-updates-schedule.example.yml' + if (-not (Test-Path $scheduleFile)) { + Write-Host " (no example schedule file; skipping)" -ForegroundColor DarkYellow + return @() + } + $tmp = New-Item -ItemType Directory -Path (Join-Path $env:TEMP "argv-step6-$(Get-Random)") -Force + try { + $authMap = Join-Path $tmp.FullName 'sideload-auth-map.csv' + @( + 'UpdateAuthAccountId,KeyVaultName,UsernameSecretName,PasswordSecretName,RemotingTargetFqdn,FqdnSuffix,AuthMechanism,ImportSharePath' + '001,kv-smoke,sideload-user,sideload-pass,,.smoke.contoso.com,Negotiate,' + ) | Set-Content -LiteralPath $authMap -Encoding ASCII + $catalog = Join-Path $tmp.FullName 'sideload-catalog.yml' + @( + 'schemaVersion: 1' + 'packages:' + " - version: '12.2605.1003.210'" + ' packageType: Solution' + " downloadUri: 'https://download.contoso.com/CombinedSolutionBundle.12.2605.1003.210.zip'" + " sha256: 'ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789'" + ) | Set-Content -LiteralPath $catalog -Encoding ASCII + Resolve-AzLocalSideloadPlan -SchedulePath $scheduleFile -AuthMapPath $authMap -CatalogPath $catalog + } + finally { + Remove-Item -Path $tmp.FullName -Recurse -Force -ErrorAction SilentlyContinue + } + } ` + -RequiredColumns @('ClusterName','ClusterResourceId','UpdateAuthAccountId','Ring','DueNow','SelectedVersion','PackageType','RemotingHost','TargetPath','Status','Message') + Write-Host "`n========================================" -ForegroundColor Cyan Write-Host " ARG / Pipeline-driver Validation Summary" -ForegroundColor Cyan Write-Host "========================================" -ForegroundColor Cyan diff --git a/AzLocal.UpdateManagement/docs/cmdlet-reference.md b/AzLocal.UpdateManagement/docs/cmdlet-reference.md index f0f26969..cab40041 100644 --- a/AzLocal.UpdateManagement/docs/cmdlet-reference.md +++ b/AzLocal.UpdateManagement/docs/cmdlet-reference.md @@ -1286,7 +1286,7 @@ $detail = Get-AzLocalFleetHealthFailures -View Detail -ExportPath .\reports\fl $summary = Get-AzLocalFleetHealthFailures -View Summary -ExportPath .\reports\fleet-health-summary.csv -PassThru ``` -> **CI/CD**: the bundled `Step.9_fleet-health-status.yml` pipeline samples (GitHub Actions and Azure DevOps) wire this cmdlet into a daily-scheduled run that emits JUnit XML, CSV/JSON exports, and a Markdown step summary. See [Automation-Pipeline-Examples/README.md](./Automation-Pipeline-Examples/README.md). +> **CI/CD**: the bundled `fleet-health-status.yml` pipeline samples (GitHub Actions and Azure DevOps) wire this cmdlet into a daily-scheduled run that emits JUnit XML, CSV/JSON exports, and a Markdown step summary. See [Automation-Pipeline-Examples/README.md](./Automation-Pipeline-Examples/README.md). **Required permissions** (read-only): - `Microsoft.AzureStackHCI/clusters/read` @@ -1301,7 +1301,7 @@ $summary = Get-AzLocalFleetHealthFailures -View Summary -ExportPath .\reports\fl *Added in v0.7.65.* -Read-only **schedule-coverage advisor**. Compares the cron schedule(s) declared in your `Step.6_apply-updates.yml` pipeline (GitHub Actions and/or Azure DevOps) to the `UpdateStartWindow` tag values actually present on your clusters, and flags every `(UpdateRing, UpdateStartWindow)` pair that no cron in the pipeline will ever reach. Never edits cluster tags. Never edits pipeline YAML. It is the safety net that closes the loop between section 8 of [`Automation-Pipeline-Examples/README.md`](./Automation-Pipeline-Examples/README.md) (the `UpdateStartWindow` tag is a *gate*, not a *trigger*) and `Test-AzLocalUpdateScheduleAllowed` (the runtime per-cluster gate inside `Start-AzLocalClusterUpdate`). +Read-only **schedule-coverage advisor**. Compares the cron schedule(s) declared in your `apply-updates.yml` pipeline (GitHub Actions and/or Azure DevOps) to the `UpdateStartWindow` tag values actually present on your clusters, and flags every `(UpdateRing, UpdateStartWindow)` pair that no cron in the pipeline will ever reach. Never edits cluster tags. Never edits pipeline YAML. It is the safety net that closes the loop between section 8 of [`Automation-Pipeline-Examples/README.md`](./Automation-Pipeline-Examples/README.md) (the `UpdateStartWindow` tag is a *gate*, not a *trigger*) and `Test-AzLocalUpdateScheduleAllowed` (the runtime per-cluster gate inside `Start-AzLocalClusterUpdate`). Under the covers it pre-scans the pipeline YAML file(s) with a regex (no `powershell-yaml` dependency), runs a single Azure Resource Graph query against `resources` for clusters with `UpdateStartWindow` / `UpdateRing` tags, parses each tag value with the same `ConvertFrom-AzLocalUpdateWindow` helper used by the runtime gate, then enumerates every cron fire time over a reference week and compares it to each parsed window (with a configurable lead-time buffer). @@ -1311,7 +1311,7 @@ Under the covers it pre-scans the pipeline YAML file(s) with a regex (no `powers |---|---|---|---|---| | `-SubscriptionId` | String | No | All accessible | Optional. Limit the ARG query to a single subscription. | | `-View` | String | No | `Audit` | `Audit` (one row per `(Ring, Window)` pair with `Covered` / `Uncovered` / `PartiallyCovered` / `MalformedTag` / `NoWindowTag` / `UnparseableCron` status + remediation), `Matrix` (every distinct `(Ring, Window)` pair with the cron expression the advisor would generate for it), or `Recommend` (ready-to-paste GH Actions + Azure DevOps cron blocks). | -| `-PipelineYamlPath` | String | Audit only | - | Path to `Step.6_apply-updates.yml` file(s) or a folder containing them. Required when `-View Audit`. | +| `-PipelineYamlPath` | String | Audit only | - | Path to `apply-updates.yml` file(s) or a folder containing them. Required when `-View Audit`. | | `-Platform` | String | No | `Both` | `GitHubActions`, `AzureDevOps`, or `Both`. Filters which YAML files are scanned and which cron blocks the Recommend view emits. | | `-LeadTimeMinutes` | Int | No | `5` | Range 0-60. How many minutes the cron should fire **before** the window opens (so cluster enumeration + auth completes before `Test-AzLocalUpdateScheduleAllowed` evaluates). | | `-UpdateRingTag` | String[] | No | - | Optional. Narrow the audit to one or more `UpdateRing` tag values. | @@ -1334,7 +1334,7 @@ Test-AzLocalApplyUpdatesScheduleCoverage ` # Audit only the GitHub Actions sample, with a 10-minute lead time Test-AzLocalApplyUpdatesScheduleCoverage ` - -PipelineYamlPath .\.github\workflows\Step.6_apply-updates.yml ` + -PipelineYamlPath .\.github\workflows\apply-updates.yml ` -Platform GitHubActions ` -LeadTimeMinutes 10 @@ -1350,7 +1350,7 @@ $matrix = Test-AzLocalApplyUpdatesScheduleCoverage -View Matrix -ExportPath .\sc $rec = Test-AzLocalApplyUpdatesScheduleCoverage -View Recommend -ExportPath .\schedule-coverage-recommend.md -PassThru ``` -> **CI/CD**: the bundled `Step.3_apply-updates-schedule-audit.yml` pipeline samples (GitHub Actions and Azure DevOps) wire this cmdlet into a weekly-scheduled run (Mon 05:00 UTC) that emits JUnit XML, three CSV/MD exports, and a Markdown step summary. Full end-to-end runbook in [`Automation-Pipeline-Examples/README.md` section 8.3](./Automation-Pipeline-Examples/README.md#83-end-to-end-runbook-apply-updates-schedule-coverage-audit). +> **CI/CD**: the bundled `apply-updates-schedule-audit.yml` pipeline samples (GitHub Actions and Azure DevOps) wire this cmdlet into a weekly-scheduled run (Mon 05:00 UTC) that emits JUnit XML, three CSV/MD exports, and a Markdown step summary. Full end-to-end runbook in [`Automation-Pipeline-Examples/README.md` section 8.3](./Automation-Pipeline-Examples/README.md#83-end-to-end-runbook-apply-updates-schedule-coverage-audit). **Required permissions** (read-only): - `Microsoft.Resources/subscriptions/resourceGroups/read` diff --git a/AzLocal.UpdateManagement/docs/rbac.md b/AzLocal.UpdateManagement/docs/rbac.md index c991eb38..4c5693f0 100644 --- a/AzLocal.UpdateManagement/docs/rbac.md +++ b/AzLocal.UpdateManagement/docs/rbac.md @@ -35,7 +35,7 @@ The following permissions are required for update + fleet-connectivity operation | Read physical NIC inventory via edge devices (Step.4) | `Microsoft.AzureStackHCI/edgeDevices/read` | | Read Azure Resource Bridge appliance status (Step.4) | `Microsoft.ResourceConnector/appliances/read` | -> **v0.7.80 note:** The last three rows above were added in v0.7.80. They are required by `Get-AzLocalFleetConnectivityStatus` (introduced in v0.7.79) and therefore by the `Step.4_fleet-connectivity-status.yml` pipeline. Without them, the cmdlet still returns the cluster connectivity section but every other section (Arc agents, physical NICs, Azure Resource Bridges) silently returns zero rows because ARG yields an empty `.data` array for resource types the caller cannot read. Pipelines that were created against the v0.7.79-or-earlier custom-role JSON will see 0 Arc agents / 0 NICs / 0 ARBs until the role is updated. +> **v0.7.80 note:** The last three rows above were added in v0.7.80. They are required by `Get-AzLocalFleetConnectivityStatus` (introduced in v0.7.79) and therefore by the `fleet-connectivity-status.yml` pipeline. Without them, the cmdlet still returns the cluster connectivity section but every other section (Arc agents, physical NICs, Azure Resource Bridges) silently returns zero rows because ARG yields an empty `.data` array for resource types the caller cannot read. Pipelines that were created against the v0.7.79-or-earlier custom-role JSON will see 0 Arc agents / 0 NICs / 0 ARBs until the role is updated. ### Roles That Do NOT Have Update Permissions diff --git a/AzLocal.UpdateManagement/docs/release-history.md b/AzLocal.UpdateManagement/docs/release-history.md index b51a759f..20bd2250 100644 --- a/AzLocal.UpdateManagement/docs/release-history.md +++ b/AzLocal.UpdateManagement/docs/release-history.md @@ -4,7 +4,7 @@ > > **For older releases**, this is the canonical reference; the main README intentionally stays slim so the most recent block is easy to find. > -> **For v0.8.4 (the current release)**, see the main [README.md](../README.md#whats-new-in-v084) `What's New in v0.8.4` section. +> **For v0.8.7 (the current release)**, see the main [README.md](../README.md#whats-new-in-v087) `What's New in v0.8.7` section. ---