|
| 1 | +# ADR 0011 — L3 Azure-side scope: agent-local checks vs. management server ARM probes |
| 2 | + |
| 3 | +- **Status**: Accepted |
| 4 | +- **Date**: 2026-05-05 |
| 5 | +- **Deciders**: @AzureLocal/azurelocal-scom-mp-maintainers |
| 6 | + |
| 7 | +## Context |
| 8 | + |
| 9 | +[ADR 0001](0001-scope-and-topology.md) defines three layers. [ADR 0004](0004-scom-discovery-strategy.md) |
| 10 | +says L3 is discovered from a designated management server using `Az` PowerShell modules. But it |
| 11 | +doesn't answer the harder question this project must resolve before Phase 3 authoring: |
| 12 | + |
| 13 | +> **Which "Azure-side" signals actually require connecting to Azure, and which can be |
| 14 | +> checked locally from the SCOM agent on the cluster node?** |
| 15 | +
|
| 16 | +This matters because: |
| 17 | + |
| 18 | +- The SCOM agent on each Azure Local node is a standard Windows agent. It **can** make |
| 19 | + outbound HTTPS calls, but doing so from every node on every monitoring interval |
| 20 | + contradicts the SCOM architecture (agents monitor local state; management servers |
| 21 | + proxy external calls). |
| 22 | +- Azure Local nodes typically operate in hardened network environments where |
| 23 | + `management.azure.com` is only reachable through a designated proxy or from a specific |
| 24 | + management host — not from every cluster node. |
| 25 | +- Some "Azure connectivity" signals are actually surfaced **locally** by the Arc Connected |
| 26 | + Machine Agent and the DCMA without needing a separate outbound call. |
| 27 | +- Some L3 signals (Key Vault expiry, RBAC drift, SPN credential) require ARM API access |
| 28 | + and **cannot** be surfaced from agent scripts on the cluster node. |
| 29 | + |
| 30 | +Reference: |
| 31 | +- [Azure Local — firewall requirements](https://learn.microsoft.com/en-us/azure/azure-local/concepts/firewall-requirements?view=azloc-2604) |
| 32 | + — outbound HTTPS-inspection prohibited; Arc Private Link Scopes unsupported; specific |
| 33 | + endpoints listed for each service (Arc, AMA, DCMA, Update Manager, etc.) |
| 34 | +- [Azure Arc-enabled servers overview](https://learn.microsoft.com/en-us/azure/azure-arc/servers/overview) |
| 35 | + — the `HIMDS` (Hybrid Instance Metadata Service) and `AzureConnectedMachineAgent` services |
| 36 | + manage and report Arc connectivity state locally. |
| 37 | +- [Azure Monitor Agent overview](https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-overview) |
| 38 | + — AMA manages its own connection lifecycle; its service state and last-write timestamp |
| 39 | + are available locally. |
| 40 | + |
| 41 | +## The core question answered |
| 42 | + |
| 43 | +### SCOM agents NEVER make direct ARM calls |
| 44 | + |
| 45 | +This is the governing rule. No monitor script targeting `AzureLocal.Cluster`, |
| 46 | +`AzureLocal.Node`, or any other L1/L2 class shall call `management.azure.com`. The |
| 47 | +SCOM agent's job is to observe local state. ARM calls belong on the management server. |
| 48 | + |
| 49 | +### "Soft L3" — locally observable Arc state (agent-side) |
| 50 | + |
| 51 | +A significant amount of what looks like "Azure connectivity" is actually **surfaced |
| 52 | +locally** by the agents Microsoft already ships with every Azure Local deployment. These |
| 53 | +are checkable from the SCOM agent without any outbound ARM call: |
| 54 | + |
| 55 | +| What to check | How to check it locally | SCOM class | Collection host | |
| 56 | +|---|---|---|---| |
| 57 | +| Arc Connected Machine Agent service state | `Get-Service "HIMDS"` + `Get-Service "GCArcService"` | `AzureLocal.Node` | Each cluster node | |
| 58 | +| Arc agent connection status | Registry: `HKLM:\SOFTWARE\Microsoft\Azure Connected Machine Agent\Config` → `Status` | `AzureLocal.Node` | Each cluster node | |
| 59 | +| Arc agent last-heartbeat age | Event log: `Microsoft-AzureArc-HybridAgent/Operational` channel — Event ID 50 (heartbeat sent) / Event ID 70 (connection error) | `AzureLocal.Node` | Each cluster node | |
| 60 | +| AMA (AzureMonitorWindowsAgent) extension state | `Get-Service "AzureMonitorWindowsAgent"` + extension health event log `Microsoft-AzureMonitorWindowsAgent/Operational` | `AzureLocal.DCMA` | Each cluster node | |
| 61 | +| DCMA (diagnostics extension) service state | `Get-Service "AzHCSvc"` (Azure HCI Service) | `AzureLocal.DCMA` | Each cluster node | |
| 62 | +| DCMA last telemetry upload | Registry: `HKLM:\SOFTWARE\Microsoft\AzureStackHCI` → `LastConnectedTime` | `AzureLocal.DCMA` | Each cluster node | |
| 63 | +| Required Arc extension installed | Registry: `HKLM:\SOFTWARE\Microsoft\Azure Connected Machine Agent\Extensions\<ExtensionName>` → `Status` | `AzureLocal.Node` | Each cluster node | |
| 64 | +| Required Arc extension version currency | Same registry path → `Version` | `AzureLocal.Node` | Each cluster node | |
| 65 | +| HCI Registration status | Registry: `HKLM:\SOFTWARE\Microsoft\AzureStackHCI` → `RegistrationStatus` + `DiagnosticLevel` | `AzureLocal.HCIRegistration` | Each cluster node | |
| 66 | + |
| 67 | +All of the above are **L1/L2 signals on L1/L2 classes**, collected by the SCOM agent |
| 68 | +on each node. They tell the operator "is this node's Azure connectivity healthy?" without |
| 69 | +ever calling Azure. |
| 70 | + |
| 71 | +### "Hard L3" — signals that genuinely require ARM access (management server only) |
| 72 | + |
| 73 | +These signals require calling Azure management APIs. They run from the designated |
| 74 | +management server only, targeting `Class="Microsoft.SystemCenter.ManagementServer"`: |
| 75 | + |
| 76 | +| Signal | ARM endpoint | SCOM class | Why it can't be done locally | |
| 77 | +|---|---|---|---| |
| 78 | +| HCI cluster resource provisioning state | `microsoft.azurestackhci/clusters` ARM property | `AzureLocal.Azure.HCICluster` | ARM-only; not in local registry | |
| 79 | +| Arc Machine connection status (authoritative) | `microsoft.hybridcompute/machines` ARM property | `AzureLocal.Azure.ArcMachine` | Local check is self-reported; ARM is authoritative | |
| 80 | +| Key Vault Resource Health + secret expiry | `microsoft.keyvault/vaults` + Key Vault data plane | `AzureLocal.Azure.KeyVault` | KV is a pure Azure resource | |
| 81 | +| Storage Account Resource Health | `microsoft.storage/storageaccounts` | `AzureLocal.Azure.StorageAccount` | Storage Account is a pure Azure resource | |
| 82 | +| Required RBAC role assignments | ARM Authorization API | `AzureLocal.Azure.RBAC` | Role assignments live in ARM only | |
| 83 | +| SPN credential expiry | Microsoft Graph `servicePrincipals/{id}/passwordCredentials` | *(management server monitor)* | Microsoft Graph only | |
| 84 | +| Update Manager linkage freshness | `microsoft.maintenance/maintenanceconfigurations` | `AzureLocal.Azure.HCICluster` | ARM-only | |
| 85 | +| DCR existence and association | ARM Resource Graph | `AzureLocal.Azure.HCICluster` | ARM-only | |
| 86 | + |
| 87 | +## Decision |
| 88 | + |
| 89 | +We adopt a **two-tier L3 strategy**: |
| 90 | + |
| 91 | +### Tier A — "Soft L3": Agent-local Arc health checks (required, blocking) |
| 92 | + |
| 93 | +Every signal in the "soft L3" table above is implemented as a standard SCOM unit monitor |
| 94 | +on the L1/L2 class it belongs to. These run on every cluster node, use the SCOM agent's |
| 95 | +standard PowerShell/registry/event-log data sources, and have **Standard** health impact. |
| 96 | +They directly answer "is this cluster's Azure connectivity working?" from on-prem. |
| 97 | + |
| 98 | +Because these checks run locally, they: |
| 99 | +- Work even if the management server has no network path to Azure |
| 100 | +- Respond immediately when the Arc agent drops (service stops → monitor fires within one |
| 101 | + polling interval, default 5 min) |
| 102 | +- Don't require any Azure credentials on the cluster nodes |
| 103 | + |
| 104 | +### Tier B — "Hard L3": Management server ARM probes (optional by default, configurable to Standard) |
| 105 | + |
| 106 | +L3 ARM probes run from the management server. The management server requires: |
| 107 | +- `Az.Accounts`, `Az.Resources`, `Az.ResourceGraph` PowerShell modules |
| 108 | +- A SCOM Run As account backed by a service principal with the role assignments defined |
| 109 | + in [ADR 0010](0010-cloud-prerequisites-contract.md) |
| 110 | +- Outbound HTTPS to `management.azure.com`, `login.microsoftonline.com`, `graph.microsoft.com` |
| 111 | + |
| 112 | +By default, all Tier B monitors ship with **`Impact = Limited`** (health visible, does not |
| 113 | +propagate to the cluster's top-level health state). Operators can upgrade individual monitors |
| 114 | +to `Impact = Standard` via the override pack. Rationale: |
| 115 | + |
| 116 | +1. Not every SCOM deployment has a management server with Azure network access |
| 117 | +2. KV secret expiry, RBAC drift, and SPN expiry are configuration warnings — important |
| 118 | + to know, but a missing KV secret doesn't stop the cluster from running |
| 119 | +3. The SCOM MP should not require Azure credentials to function for operators who only |
| 120 | + want on-prem monitoring |
| 121 | + |
| 122 | +### The "is the cluster connected to Azure?" answer lives in Tier A |
| 123 | + |
| 124 | +The canonical SCOM health state for Azure connectivity comes from the agent-local checks: |
| 125 | + |
| 126 | +- `ArcAgent.HimdsService.State` — is the HIMDS service running? |
| 127 | +- `ArcAgent.ConnectionStatus.Local` — does the local registry report Connected? |
| 128 | +- `ArcAgent.Heartbeat.EventLog.Age` — how old is the most recent heartbeat event? |
| 129 | +- `DCMA.Service.State` + `DCMA.LastUpload.Age` — is telemetry flowing? |
| 130 | + |
| 131 | +If any of these are unhealthy, the Node's Availability aggregate goes Degraded/Unhealthy, |
| 132 | +which propagates to the Cluster. **No ARM call required.** |
| 133 | + |
| 134 | +## Consequences |
| 135 | + |
| 136 | +- **Positive**: SCOM MP works fully on deployments with no management-server-to-Azure |
| 137 | + connectivity. Tier A is fully self-contained. |
| 138 | +- **Positive**: Tier A provides immediate response to Arc agent failures — no Azure round-trip. |
| 139 | +- **Positive**: Tier B gives rich configuration drift detection (KV expiry, RBAC) for |
| 140 | + operators who do have management server Azure access. |
| 141 | +- **Positive**: Operators can choose their L3 depth by flipping impact from Limited to |
| 142 | + Standard on the Tier B monitors — no MP fork required. |
| 143 | +- **Negative**: Tier B requires `Az` modules + SPN credentials on the management server. |
| 144 | + Some operators will not configure this. |
| 145 | +- **Negative**: The management server becomes a single point of collection for all L3 ARM |
| 146 | + data. If it goes offline, L3 Tier B monitors show Unknown. |
| 147 | +- **Neutral**: Tier B checks run on a 1-hour polling interval (see ADR 0004) — not real-time. |
| 148 | + This is acceptable for configuration drift, not acceptable for availability signals (which |
| 149 | + is why availability lives in Tier A). |
| 150 | + |
| 151 | +## Required extensions to monitor (Tier A — locally checkable) |
| 152 | + |
| 153 | +These extensions are required for a supported Azure Local deployment. The MP checks each one: |
| 154 | + |
| 155 | +| Extension | Service / Registry key | Signal | |
| 156 | +|---|---|---| |
| 157 | +| `AzureMonitorWindowsAgent` | `AzureMonitorWindowsAgent` service + `Microsoft-AzureMonitorWindowsAgent/Operational` channel | `Extension.AMA.State` | |
| 158 | +| `AzureEdgeTelemetryAndDiagnostics` (DCMA) | `AzHCSvc` service | `Extension.DCMA.State` | |
| 159 | +| `AzureConnectedMachineAgent` (Arc for Servers) | `HIMDS` + `GCArcService` services | `ArcAgent.HimdsService.State`, `ArcAgent.GCArcService.State` | |
| 160 | +| `WindowsOSSettings` (Arc extension) | Registry probe | `Extension.OsSettings.Installed` | |
| 161 | + |
| 162 | +References: |
| 163 | +- [Azure Local — firewall requirements](https://learn.microsoft.com/en-us/azure/azure-local/concepts/firewall-requirements?view=azloc-2604) |
| 164 | +- [Azure Monitor Agent network / DCE settings](https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-data-collection-endpoint?tabs=PowerShellWindows) |
| 165 | +- [Azure Local Insights — single cluster](https://learn.microsoft.com/en-us/previous-versions/azure/azure-local/manage/monitor-hci-single?view=azloc-2604&tabs=22h2-and-later) |
| 166 | +- [Azure Local — required permissions](https://learn.microsoft.com/en-us/azure/azure-local/deploy/deployment-arc-register-server-permissions?view=azloc-2604) |
| 167 | + |
| 168 | +## Alternatives considered |
| 169 | + |
| 170 | +- **No L3 monitoring at all** — rejected: Tier A local checks are low cost and directly |
| 171 | + answer the "is Arc healthy?" question operators ask every day. Not implementing them |
| 172 | + leaves a real gap. |
| 173 | +- **All L3 from management server** — rejected: forces every operator to provision Azure |
| 174 | + credentials, breaks deployments without management-server-to-Azure connectivity, and |
| 175 | + creates ARM rate-limit exposure from many agents. |
| 176 | +- **L3 via ARM from each cluster node** — rejected: violates SCOM architecture (agents |
| 177 | + observe local state), adds Azure credential management to 4–16 nodes, creates ARM |
| 178 | + throttling risk, and violates most customer firewall policies. |
| 179 | +- **L3 Tier B with Standard impact by default** — rejected: would make the MP health state |
| 180 | + "Unknown" for operators who haven't provisioned management server Azure credentials, even |
| 181 | + though on-prem monitoring is fully functional. |
| 182 | + |
| 183 | +## References |
| 184 | + |
| 185 | +- ADR 0001 — [Scope & topology](0001-scope-and-topology.md) |
| 186 | +- ADR 0002 — [Primary signal source](0002-signal-source.md) |
| 187 | +- ADR 0004 — [SCOM discovery strategy](0004-scom-discovery-strategy.md) |
| 188 | +- ADR 0010 — [Cloud prerequisites contract](0010-cloud-prerequisites-contract.md) |
| 189 | +- [Azure Local — firewall requirements](https://learn.microsoft.com/en-us/azure/azure-local/concepts/firewall-requirements?view=azloc-2604) |
| 190 | +- [Azure Arc-enabled servers overview](https://learn.microsoft.com/en-us/azure/azure-arc/servers/overview) |
| 191 | +- [Azure Monitor Agent overview](https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-overview) |
| 192 | +- [SCOM — Heartbeat overview](https://learn.microsoft.com/en-us/system-center/scom/manage-agent-heartbeat-overview?view=sc-om-2025) |
| 193 | +- [Kevin Holman — SCOM 2025 Security Account Matrix](https://kevinholman.com/2024/11/25/scom-2025-security-account-matrix/) |
0 commit comments