Skip to content

Commit 61a1ca1

Browse files
design: ADR 0012 AMW/LAW metrics topology + Tier B shortlist refinement + missing Tier A signals
ADR 0011 (Tier B shortlist): - Removed HCI cluster provisioning state from Tier B (local registry is a valid proxy; ARM is a cross-check, not primary) - Removed Arc Machine authoritative status from Tier B (Tier A already covers this via local registry + event log; ARM view lags reality) - Split DCR check: Tier A = AMA local config registry probe; Tier B = ARM-side dataCollectionRuleAssociations (only this half needs ARM) - Trimmed Tier B to 8 genuinely ARM-only signals: KV, Storage Account, RBAC, SPN expiry, Custom Location, Logical Networks, Update Manager ARM linkage, DCR ARM association - Added 'Additional Tier A signals' section: HCI registration local proxy, LCM/SolutionUpdate state, Environment Checker, cloud witness reachability, Hyper-V host health, QoS policy conformance, S2D cache dirty %, storage rebuild job progress + ETA, DCR local AMA config probe ADR 0012 (new): - Documents Azure Monitor Workspace (AMW) vs Log Analytics Workspace (LAW) metrics routing for the Azure Monitor health model track only - Defines two topologies: A (legacy, metrics in LAW Perf table) and B (modern, metrics split to AMW via monitoringAccounts DCR destination) - Health model Bicep detects topology at deploy time via ARG DCR scan - AMW preferred when available; LAW Perf fallback always supported - DCMA native metrics unaffected by topology (always native metrics store) - Dual query variants required for each performance signal (KQL + PromQL) signal-catalog.md: - Cluster: added Quorum.Witness.Reachable, EnvironmentChecker.Pass, LCM.Update.PendingCritical, LCM.Environment.Health, LCM.Update.LastRunResult - Node: added HyperV.State, HyperV.vSwitch.Binding - StorageTier: added Cache.DirtyPages.Percent (perf counter) - StoragePool: added RebuildJob.PercentComplete, RebuildJob.ETA.Hours - NIC: added QoS.Policy.Conformance, DCBX.Willing prerequisites.md: - Row 10: added AMW to managed identity Monitoring Reader grant - Row 16: added AMW as Recommended prerequisite with topology-detection note
1 parent d6c172f commit 61a1ca1

6 files changed

Lines changed: 297 additions & 12 deletions

File tree

PLAN.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -336,6 +336,7 @@ Every threshold, alert severity, and behavior is **parameterized** so customers
336336
- [x] ADR 0009 — Alert vs health-state separation policy — Accepted
337337
- [x] ADR 0010 — Cloud-side prerequisites contract (HCI Insights, AMA, DCMA, Service Group, RBAC, networking) — Accepted
338338
- [x] ADR 0011 — L3 Azure-side scope: agent-local Arc health checks (Tier A) vs. management server ARM probes (Tier B) — Accepted
339+
- [x] ADR 0012 — Azure Monitor Workspace vs Log Analytics Workspace: metrics routing for the health model (dual-topology support, LAW Perf fallback) — Accepted
339340
- [x] Build full structural inventory tables in `docs/design/`
340341
- [x] Component inventory (~27 entities across 3 layers — updated with `AzureLocal.PhysicalDisk`, `AzureLocal.NetworkAdapter`, Arc agent Tier A group) — `scope-topology.md`
341342
- [x] Signal inventory (~90+ signals × dimensions × thresholds × source — expanded with physical disk health, storage pool size/reserve/retired, volume size detail, physical NIC/RDMA, Arc agent local checks, extension installation probes, cluster network state) — `signal-catalog.md`

docs/azure-monitor/prerequisites.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,12 +30,13 @@ Authoritative prerequisites for building an Azure Monitor **Health Model** (prev
3030
| 7 | **Cluster managed identity** ("enhanced management") enabled | Insights "Get Started" button is disabled until cluster identity is on; AMA on each node uses *server* identity for log collection (mismatch is the #1 cause of empty workbooks). | Blocking | [Insights prerequisites](https://learn.microsoft.com/en-us/previous-versions/azure/azure-local/manage/monitor-hci-single?view=azloc-2604&tabs=22h2-and-later#prerequisites) |
3131
| 8 | **Azure Service Group** scoped to the cluster's resources | Health models are built *on top of* a service group; entities are auto-discovered from members every 5 min. | Blocking | [Service Groups overview](https://learn.microsoft.com/en-us/azure/governance/service-groups/overview), [Create a health model](https://learn.microsoft.com/en-us/azure/azure-monitor/health-models/create) |
3232
| 9 | **RBAC for health-model authoring** — Contributor on host RG, Service Group Contributor on the SG, Monitoring Contributor on the model | Required to create / edit / discover. | Blocking | [Permissions required](https://learn.microsoft.com/en-us/azure/azure-monitor/health-models/create#permissions-required) |
33-
| 10 | **Managed identity for the health model** with Service Group Reader + Monitoring Reader on members + Monitoring Reader on workspace(s) | Identity the model uses to enumerate the SG and pull telemetry/log data. | Blocking | [Permissions required](https://learn.microsoft.com/en-us/azure/azure-monitor/health-models/create#permissions-required) |
33+
| 10 | **Managed identity for the health model** with Service Group Reader + Monitoring Reader on members + Monitoring Reader on LAW + Monitoring Reader on AMW (if AMW topology in use) | Identity the model uses to enumerate the SG and pull telemetry/log data from both workspace types. | Blocking | [Permissions required](https://learn.microsoft.com/en-us/azure/azure-monitor/health-models/create#permissions-required) |
3434
| 11 | **Outbound 80/443** to required Azure URLs from every node and ARB; **no HTTPS inspection**; **no Arc Private Link Scopes** | Without it AMA, DCMA, Arc agent, and ARB cannot push data. | Blocking | [Firewall requirements for Azure Local](https://learn.microsoft.com/en-us/azure/azure-local/concepts/firewall-requirements?view=azloc-2604) |
3535
| 12 | **VM Insights** for ARB and management VMs | Adds processes/dependency map and OS-level perf for the management plane VMs. | Recommended | [VM Insights overview](https://learn.microsoft.com/en-us/azure/azure-monitor/vm/vminsights-overview) |
3636
| 13 | **Activity Log + Resource Health** (auto, no setup) | Free Azure-resource-level signals — useful as health-model signals on the cluster, Key Vault, Storage Account, etc. | Auto / no action | [Activity log](https://learn.microsoft.com/en-us/azure/azure-monitor/platform/activity-log) |
3737
| 14 | **Azure Resource Graph access** (Reader on the SG/subscription) | The model's discovery and many operator queries (inventory of nodes, extensions, Arc state) rely on ARG. | Blocking for discovery queries | [Service Groups / Health Model discovery](https://learn.microsoft.com/en-us/azure/azure-monitor/health-models/concepts#azure-resource-entity) |
3838
| 15 | **Azure Update Manager onboarding** for the cluster | Provides update-status signals (assessment age, missing critical updates) you can wire into the health model. | Recommended | [Monitoring capabilities](https://learn.microsoft.com/en-us/azure/azure-local/concepts/monitoring-overview?view=azloc-2604) |
39+
| 16 | **Azure Monitor Workspace (AMW)** with a supplementary DCR routing AMA performance counter metrics to `monitoringAccounts` destination | Preferred target for performance metrics (PromQL, lower cost). **Not required** — the health model auto-detects topology and falls back to `Perf` table queries on the LAW when no AMW DCR destination is found. Most current Azure Local deployments are still on the LAW-only (legacy) topology. | Recommended | [Azure Monitor Workspace overview](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/azure-monitor-workspace-overview), [ADR 0012](../design/decisions/0012-azure-monitor-workspace-vs-law-metrics.md) |
3940

4041
---
4142

docs/design/decisions/0011-l3-azure-scope-and-connectivity.md

Lines changed: 38 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -70,19 +70,28 @@ ever calling Azure.
7070

7171
### "Hard L3" — signals that genuinely require ARM access (management server only)
7272

73-
These signals require calling Azure management APIs. They run from the designated
74-
management server only, targeting `Class="Microsoft.SystemCenter.ManagementServer"`:
73+
Only signals with **no on-box representation at all** belong here. If a local proxy
74+
exists (registry, service state, event log), the signal moves to Tier A. This list
75+
was reviewed against all known local data sources and trimmed accordingly.
7576

76-
| Signal | ARM endpoint | SCOM class | Why it can't be done locally |
77+
These run from the designated management server only, targeting
78+
`Class="Microsoft.SystemCenter.ManagementServer"`:
79+
80+
| Signal | ARM / Graph endpoint | SCOM class | Why it cannot be done locally |
7781
|---|---|---|---|
78-
| HCI cluster resource provisioning state | `microsoft.azurestackhci/clusters` ARM property | `AzureLocal.Azure.HCICluster` | ARM-only; not in local registry |
79-
| Arc Machine connection status (authoritative) | `microsoft.hybridcompute/machines` ARM property | `AzureLocal.Azure.ArcMachine` | Local check is self-reported; ARM is authoritative |
80-
| Key Vault Resource Health + secret expiry | `microsoft.keyvault/vaults` + Key Vault data plane | `AzureLocal.Azure.KeyVault` | KV is a pure Azure resource |
81-
| Storage Account Resource Health | `microsoft.storage/storageaccounts` | `AzureLocal.Azure.StorageAccount` | Storage Account is a pure Azure resource |
82-
| Required RBAC role assignments | ARM Authorization API | `AzureLocal.Azure.RBAC` | Role assignments live in ARM only |
83-
| SPN credential expiry | Microsoft Graph `servicePrincipals/{id}/passwordCredentials` | *(management server monitor)* | Microsoft Graph only |
84-
| Update Manager linkage freshness | `microsoft.maintenance/maintenanceconfigurations` | `AzureLocal.Azure.HCICluster` | ARM-only |
85-
| DCR existence and association | ARM Resource Graph | `AzureLocal.Azure.HCICluster` | ARM-only |
82+
| Key Vault Resource Health + secret expiry | `microsoft.keyvault/vaults` + KV data plane | `AzureLocal.Azure.KeyVault` | Pure Azure resource — no local representation |
83+
| Storage Account Resource Health | `microsoft.storage/storageaccounts` | `AzureLocal.Azure.StorageAccount` | Pure Azure resource — no local representation |
84+
| Required RBAC role assignments (SPN, MI, cluster identity) | ARM Authorization `roleAssignments` | `AzureLocal.Azure.RBAC` | Role assignments exist only in ARM — no local copy |
85+
| SPN credential expiry | Microsoft Graph `servicePrincipals/{id}/passwordCredentials` | *(management server monitor)* | Microsoft Graph only — no local representation |
86+
| Custom Location provisioning state | `microsoft.extendedlocation/customlocations` | `AzureLocal.Azure.CustomLocation` | ARM-only resource created by deployment stack |
87+
| Logical Network provisioning state + IP pool usage | `microsoft.azurestackhci/logicalnetworks` | `AzureLocal.Azure.LogicalNetwork` | ARM-only resource managed by MOC |
88+
| Update Manager maintenance config ARM linkage | `microsoft.maintenance/maintenanceconfigurations` | `AzureLocal.Azure.HCICluster` | Whether the ARM maintenance config is *linked* to the cluster is ARM-only; update state itself is available locally via `Get-SolutionUpdate` |
89+
| DCR ARM-side association (supplementary) | ARM `dataCollectionRuleAssociations` via Resource Graph | `AzureLocal.Azure.HCICluster` | Whether the ARM-side association record still exists — the local AMA config (Tier A) tells you what AMA is doing, but can't detect if the ARM association was deleted |
90+
91+
**Signals removed from Tier B compared to the original draft:**
92+
93+
- **HCI cluster provisioning state** → moved to Tier A: `HKLM:\SOFTWARE\Microsoft\AzureStackHCI``RegistrationStatus` + `ConnectionIdentifier` + `LastConnectedTime` are valid local proxies. ARM is a cross-check, not the primary.
94+
- **Arc Machine connection status (authoritative)** → already covered by Tier A `ArcAgent.ConnectionStatus.Local`. The ARM view *lags* the node's reality by up to one sync interval — the local registry is actually *more* timely. The ARM version was removed from Tier B to avoid false health-state discrepancy between Tier A and Tier B.
8695

8796
## Decision
8897

@@ -148,6 +157,24 @@ which propagates to the Cluster. **No ARM call required.**
148157
This is acceptable for configuration drift, not acceptable for availability signals (which
149158
is why availability lives in Tier A).
150159

160+
## Additional Tier A signals — locally checkable, previously missing
161+
162+
These signals were identified during Tier B review as locally checkable but not
163+
previously included in the Tier A list. All run on the SCOM agent on each node
164+
or on a designated cluster node (cookdown target).
165+
166+
| What to check | How to check it locally | SCOM class |
167+
|---|---|---|
168+
| **HCI cluster registration status (local proxy)** | Registry `HKLM:\SOFTWARE\Microsoft\AzureStackHCI``RegistrationStatus`, `LastConnectedTime`, `ConnectionIdentifier` | `AzureLocal.HCIRegistration` |
169+
| **LCM / SolutionUpdate state** | `Get-SolutionUpdate` (update pending/failed state) + `Get-SolutionUpdateEnvironment` (LCM health) | `AzureLocal.LCMState` |
170+
| **Environment Checker results** | `Invoke-AzureStackHCIEnvironmentChecker` — ships on every cluster; returns pass/fail per check category (network, storage, Arc, update) | `AzureLocal.Cluster` (run at discovery cadence) |
171+
| **Cloud Witness reachability** | `Get-ClusterQuorum` → witness type; if cloud witness: `Test-NetConnection <account>.blob.core.windows.net -Port 443` | `AzureLocal.Cluster` |
172+
| **Hyper-V host health** | `Get-VMHost` health state + `Get-VMSwitch` NIC binding presence | `AzureLocal.Node` |
173+
| **DCB / QoS policy conformance** | `Get-NetQosPolicy`, `Get-NetQosFlowControl`, `Get-NetQosTrafficClass` — verify PFC enabled on correct priorities, ETS bandwidth allocation per Network ATC intent | `AzureLocal.NetworkAdapter` |
174+
| **S2D cache dirty pages percent** | Perf counter `\Cluster Storage Cache Stores(*)\Cache Dirty Pages %` | `AzureLocal.StorageTier` |
175+
| **Storage rebuild job progress** | `Get-StorageJob | Where-Object JobState -eq 'Running'``PercentComplete` + elapsed time | `AzureLocal.StoragePool` |
176+
| **DCR local AMA config present** | Registry `HKLM:\SOFTWARE\Microsoft\Azure Monitor Agent\` → active DCR list present and non-empty | `AzureLocal.Node` (supplements Tier B ARM association check) |
177+
151178
## Required extensions to monitor (Tier A — locally checkable)
152179

153180
These extensions are required for a supported Azure Local deployment. The MP checks each one:

0 commit comments

Comments
 (0)