Skip to content

Commit 990f2eb

Browse files
design+structure: full plan audit, 6 new ADRs, drop dead weight
Plan audit follow-up addressing every gap raised in the design review. PLAN.md cleanup (cut dead weight): - Drop Org Standards section detail referencing private AzureLocal/platform (file contents, badge URLs, version pins moved to private repo memory) - Drop "Phase 0 platform compliance checklist" (Phase 0 is complete; checklist was stale) - Drop "Blog & New Repo Reminders" block (project-management noise, not design) - Drop stale MkDocs nav example (live mkdocs.yml is source of truth) - Fix Override pack name in Key Decisions table: AzureLocal.SCOM.MP.Overrides.xml -> AzureLocal.SCOM.Override.xml (per ADR 0007) - Fix Org standards row in Key Decisions table (no longer references public badge) - Fix repo structure tree: squaredup/ shown at root -> moved under src/ - Reconcile SquaredUp appendix: drop "Evaluate" language, point to committed Phase 3 Step 4 / Phase 4 Step 5 work Mermaid diagrams moved into docs/ (so MkDocs renders them): - diagrams/mermaid/scom-health-rollup.md -> docs/scom-mp/diagrams/health-tree.md - diagrams/mermaid/azure-monitor-entity-graph.md -> docs/azure-monitor/diagrams/entity-graph.md - diagrams/mermaid/health-state-flow.md -> docs/design/diagrams/health-state-flow.md - diagrams/mermaid/ removed (drawio/ stays as editable originals) - mkdocs.yml nav: add Diagrams sub-section to SCOM MP, Azure Monitor, and Design Six new ADRs (all Accepted) filling design gaps: ADR 0014 - CI/CD Pipeline Strategy GitHub Actions only; one workflow per artifact stream (mp-build, bicep-validate, pwsh-test, kql-validate, parity-check, docs-build, docs-deploy, release-please). OIDC-only auth to Azure. release-please in Conventional Commits mode. parity-check.yml mechanically enforces ADR 0007 cross-track naming. ADR 0015 - Testing Strategy 5-layer test pyramid: static/lint, unit (Pester + KustoLoco), integration (bicep what-if + LAW seed), cross-track parity (THE keystone gate for ADR 0007), and behavioral acceptance at release. tests/Test-Parity.ps1 walks signal-catalog.md and asserts every threshold name appears in BOTH the SCOM override pack AND a bicepparam tier. ADR 0016 - Signing & Secrets Management Two-key SCOM MP signing model: test key (free dev access) vs release key (HSM-backed Key Vault, OIDC-only, manual-approval release env). Three OIDC-federated SPNs (pr/lab/release) - zero long-lived secrets in repo. gitleaks pre-commit + push-protection. Release key never rotated (sealed-MP version contract). ADR 0017 - Versioning & Release Policy Single repo SemVer version covers all three artifact streams (sealed MPs, Bicep, docs). Detailed bump rules table (add class = minor, rename param = major, etc). Library MP coupling rule: any Library version change is Minor minimum, never Patch. Conventional Commits -> release-please. mike-versioned docs site with version selector. ADR 0018 - Self-Observability ("Monitor the Monitor") 9 new AzureLocal.Monitoring.* entities + 12 signals covering SCOM agent, AMA extension, DCR association, DCMA, LAW ingestion, alert-rule health, and signal-query health. Rolls up under a parallel root branch (NOT under cluster) so a dead pipeline turns the root red even when cluster shows stale Healthy. Dedicated Sev2 action group, own dashboard panel. ADR 0019 - Cost, Scale, and Data Retention Per-tier ingestion envelope: Lab ~250 MB/day, Standard ~600 MB/day, Strict ~1.2 GB/day per 4-node cluster. Hard scale cap: 50 clusters per service group; >50 = shard via per-region service groups. Retention by data class: HealthState 90d, Perf 30d interactive + 365d archive, Self-Observability 30d. 5-step cost downgrade playbook. PLAN.md updates from new ADRs: - Phase 2 ADR list: add 0014-0019 (was 0001-0013) - Phase 2 sign-off: 13 ADRs -> 19 ADRs - Phase 2 Definition of Done: explicit exit criteria (signal coverage per entity per dimension, parity test implementable from catalog, mkdocs build --strict passes, Self-Observability branch reflected in topology) - Phase 3 Step 1: wire CI/CD scaffolding (ADR 0014), wire signing+identity (ADR 0016), wire test harness (ADR 0015) before authoring starts - Phase 3 (added): Self-Observability monitors (ADR 0018), tuning runbooks (docs/scom-mp/runbooks/), DB sizing guidance (ADR 0019) - Phase 4 Step 4 (added): sizing exercise to validate ADR 0019 numbers, Self-Observability KQL signals (ADR 0018), cost-tuning playbook (ADR 0019) - References note: REFERENCES.md is authoritative; docs/references.md is generated via include-markdown plugin (no hand-edited duplicate) mkdocs.yml: nav adds ADRs 0014-0019 + diagram pages under each section ADR index: full table updated through 0019
1 parent db54cd0 commit 990f2eb

13 files changed

Lines changed: 866 additions & 163 deletions

PLAN.md

Lines changed: 66 additions & 163 deletions
Large diffs are not rendered by default.

diagrams/mermaid/.gitkeep

Whitespace-only changes.
File renamed without changes.
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# ADR 0014 — CI/CD Pipeline Strategy
2+
3+
- **Status**: Accepted
4+
- **Date**: 2026-05-05
5+
- **Deciders**: @kturner
6+
7+
## Context
8+
9+
Phase 3 (SCOM MP authoring) and Phase 4 (Azure Monitor Bicep) cannot start safely without a defined
10+
CI/CD pipeline. We need build-and-validate gates for three independent artifact streams:
11+
12+
1. **SCOM MPs** — sealed `AzureLocal.SCOM.Library.mp` + sealed `AzureLocal.SCOM.Monitoring.mp` +
13+
unsealed `AzureLocal.SCOM.Override.xml` (per ADR 0007)
14+
2. **Azure Monitor Bicep**`src/azure-monitor/bicep/main.bicep` + modules + `.bicepparam` tiers
15+
(per ADR 0013)
16+
3. **MkDocs documentation site** — published to GitHub Pages with `mike` versioning
17+
18+
There is no pipeline yet. Without one, the three artifacts drift from each other and from the
19+
design docs, signing keys leak into hand-runs, and there is no reproducible release artifact.
20+
21+
## Decision
22+
23+
Adopt a **GitHub Actions–only** CI/CD model with one workflow per artifact stream, all triggered
24+
on PR and on push to `main`. Releases are cut by [release-please](https://github.com/googleapis/release-please)
25+
in Conventional Commits mode.
26+
27+
### Workflows (under `.github/workflows/`)
28+
29+
| Workflow | Trigger | Job summary |
30+
|---|---|---|
31+
| `mp-build.yml` | PR + main; tags `v*` | Restore VSAE solution → `MPSeal` against test key (PR) or prod key from KeyVault (release) → `MPVerify``MPBPA` → upload `*.mp` + `*.xml` as artifact |
32+
| `bicep-validate.yml` | PR + main | `bicep build` all `.bicep``bicep lint``az deployment sub what-if` against the lab subscription using `lab.bicepparam` → publish ARM JSON to `dist/` artifact |
33+
| `pwsh-test.yml` | PR + main | `Invoke-ScriptAnalyzer -Settings PSGallery -Severity Warning,Error` → run `Pester` against `src/scom-mp/discovery/*.Tests.ps1` and `src/azure-monitor/scripts/*.Tests.ps1` |
34+
| `kql-validate.yml` | PR + main | Validate every `kql/signals/*.kql` file parses (`Kusto.Language` parser) and runs against the lab LAW with mock seed data; assert expected health state |
35+
| `parity-check.yml` | PR + main | Custom step (PowerShell) that walks `signal-catalog.md` and asserts every threshold name appears in **both** the SCOM override pack AND a `bicepparam` tier (cross-track parity per ADR 0007) |
36+
| `docs-build.yml` | PR | `mkdocs build --strict` → run `lychee` link-check across rendered site |
37+
| `docs-deploy.yml` | push main; release tag | `mike deploy --update-aliases <version> latest``mike set-default latest` → push to `gh-pages` |
38+
| `release-please.yml` | push main | Standard release-please action — opens/maintains release PR, on merge tags `vX.Y.Z` and creates GitHub Release |
39+
40+
### Branch protection on `main`
41+
42+
- All seven workflows above must pass
43+
- Linear history required; squash-merge only
44+
- One review required, code-owner required for `src/scom-mp/`, `src/azure-monitor/bicep/`, and `docs/design/decisions/`
45+
46+
### Identity model
47+
48+
- **Cloud-side jobs** (`bicep-validate`, `kql-validate`) authenticate to Azure via **GitHub OIDC**
49+
federated to a dedicated lab-only service principal — no long-lived secrets in repo settings.
50+
- **MP signing key** for release builds is fetched from Azure Key Vault using the same OIDC SPN;
51+
scoped to a release-only environment that requires manual approval (see ADR 0016).
52+
53+
### Out of scope (deliberately)
54+
55+
- Self-hosted runners — start on GitHub-hosted, revisit only if KQL or Bicep eval cost becomes
56+
a real problem.
57+
- Container builds — no containers in this repo.
58+
- Third-party CI (Azure Pipelines, Jenkins) — explicitly rejected; the org standard is GH Actions.
59+
60+
## Consequences
61+
62+
- **Positive**:
63+
- Every artifact has a green-build gate before it can land on `main`.
64+
- `parity-check.yml` mechanically enforces the cross-track naming guarantee from ADR 0007 —
65+
so the design promise survives long-term drift.
66+
- Release-please gives clean SemVer tags and an auto-generated changelog matching the
67+
org-standard format.
68+
- OIDC eliminates the "rotate the SPN secret" toil and the leak-in-logs risk.
69+
- **Negative**:
70+
- More workflow YAML to maintain than a single monolith would need.
71+
- `kql-validate` requires a long-lived lab LAW with seeded fixtures — small but ongoing cost.
72+
- **Neutral**:
73+
- Pipeline lives in `.github/workflows/` only; no Azure DevOps mirror.
74+
- **Affected components / owners**: All Phase 3 and Phase 4 deliverables, plus the Phase 6
75+
release process. Workflow files added by the first commit of Phase 3 work.
76+
77+
## Alternatives considered
78+
79+
- **Single monolithic CI workflow** — rejected; couples MP, Bicep, KQL, and docs cycle times,
80+
and surfaces unrelated failures on every PR.
81+
- **Self-hosted runners on a SCOM management server** — rejected for now; GitHub-hosted Windows
82+
runners can install VSAE/MPVerify on demand, and self-hosted adds opex without a clear win at
83+
Phase 3.
84+
- **Azure DevOps Pipelines** — rejected; not the org standard, and OIDC + release-please are
85+
better-supported in GitHub Actions.
86+
- **No `parity-check`** — rejected; ADR 0007's cross-track naming promise is unenforceable
87+
without a mechanical gate, and would silently rot.
88+
- **`bicep what-if` only at release time** — rejected; what-if is the cheapest way to catch
89+
schema drift on Health Model preview API churn, so it runs every PR.
90+
91+
## References
92+
93+
- ADR 0007 — Naming convention (cross-track parity)
94+
- ADR 0013 — Azure Monitor deployment strategy (Bicep-first)
95+
- ADR 0016 — Signing & secrets management (sibling ADR for key handling detail)
96+
- ADR 0017 — Versioning & release policy (sibling ADR; release-please configuration)
97+
- [release-please](https://github.com/googleapis/release-please)
98+
- [Azure login GitHub Action with OIDC](https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure)
99+
- Brian Wren MP authoring guide — MPSeal / MPVerify usage
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# ADR 0015 — Testing Strategy
2+
3+
- **Status**: Accepted
4+
- **Date**: 2026-05-05
5+
- **Deciders**: @kturner
6+
7+
## Context
8+
9+
The repo ships three artifact streams (sealed SCOM MPs, Bicep modules, MkDocs site) that are
10+
authored independently but must remain semantically aligned (same logical thresholds, same
11+
entity model, same health propagation). Phase 3 and Phase 4 cannot have a "done" gate without a
12+
defined verification approach. ADR 0014 lists the *workflows* that run tests; this ADR defines
13+
*what* the tests are and *what coverage means* for each artifact.
14+
15+
## Decision
16+
17+
Adopt a **layered test pyramid** scoped to what is realistically testable per artifact, with one
18+
hard gate per layer. No artifact merges to `main` without its layer passing.
19+
20+
### Layer 1 — Static / lint (every PR, fast)
21+
22+
| Artifact | Tool | Gate |
23+
|---|---|---|
24+
| SCOM MP XML | `MPVerify.exe` | Zero errors; warnings reviewed |
25+
| SCOM MP XML | `MPBPA` (MP Best Practice Analyzer) | Zero criticals; warnings reviewed |
26+
| Bicep | `bicep build` + `bicep lint --diagnostics-format sarif` | Zero `error`-severity diagnostics |
27+
| PowerShell | `PSScriptAnalyzer` (Settings: `PSGallery`, Severity ≥ `Warning`) | Zero `Error`; `Warning` reviewed |
28+
| KQL | `Microsoft.Azure.Kusto.Language` parser | Every `*.kql` parses successfully |
29+
| Markdown | `markdownlint-cli2` (default rules + `MD013` off) | Zero errors |
30+
| Links | `lychee` against built MkDocs site | Zero broken internal links; external links cached + warn-only |
31+
32+
### Layer 2 — Unit (every PR, fast, hermetic)
33+
34+
| Artifact | Tool | What gets tested |
35+
|---|---|---|
36+
| PowerShell discovery scripts (SCOM) | `Pester 5.x` | Every discovery in `src/scom-mp/discovery/` has a sibling `*.Tests.ps1` that mocks `Get-Cluster`, `Get-PhysicalDisk`, `Get-StoragePool`, etc. and asserts the discovery emits the expected `<Instance>` elements |
37+
| KQL signals | `KustoLoco` (or LAW with seed table) — local Kusto evaluator | Each `kql/signals/<signal>.kql` has a sibling `<signal>.tests.json` listing `{seed_rows, expected_state}` cases; harness loads seed, runs query, asserts `HealthState` column matches |
38+
| Bicep modules | `bicep what-if` against an empty resource group | Each module deploys cleanly with `lab.bicepparam`; what-if returns expected resource set |
39+
| Health rollup logic | Pester | Synthetic entity tree fed through aggregate/dependency monitor logic; assert worst-state per ADR 0003 |
40+
41+
### Layer 3 — Integration (PR + nightly, against lab subscription)
42+
43+
| Artifact | Method | Gate |
44+
|---|---|---|
45+
| Bicep stack | `az deployment sub create` against lab subscription using `lab.bicepparam` | Deployment succeeds; smoke test queries `healthmodels` resource and asserts entities match `entities.bicep` count |
46+
| KQL signals (live) | Run every signal against lab LAW with seeded HCI cluster | Each signal returns at least one row of expected schema (`HealthState`, `EntityId`, `Timestamp`) |
47+
| MP load test | Import sealed MPs into pre-prod SCOM management group | All discoveries run within 10 minutes; no `Health Service Modules` event-log errors |
48+
49+
### Layer 4 — Cross-track parity (every PR, hard gate)
50+
51+
This is the keystone test that protects ADR 0007's naming-parity promise. A single PowerShell
52+
script (`tests/Test-Parity.ps1`) walks:
53+
54+
1. `docs/design/signal-catalog.md` — extract every `Threshold name` column entry
55+
2. `src/scom-mp/AzureLocal.SCOM.Override.xml` — extract every `<Override>` parameter name
56+
3. `src/azure-monitor/bicep/parameters/*.bicepparam` — extract every parameter assignment
57+
58+
Asserts: **every threshold name in the catalog appears in BOTH the SCOM override pack AND at least
59+
one bicepparam file**, and vice versa (no orphan parameters on either side). Fails the PR with a
60+
diff if any names diverge.
61+
62+
### Layer 5 — Behavioral / acceptance (release tag only)
63+
64+
| Artifact | Method |
65+
|---|---|
66+
| Full SCOM MP | Deploy to pre-prod SCOM, run for 24h, capture `OperationsManager` event log; assert no script-runtime errors, no orphaned discoveries |
67+
| Full Azure Monitor stack | Deploy `standard.bicepparam` to a real Azure Local lab cluster, drive a known fault (e.g. fail a physical disk), assert root-entity health flips to Degraded within 15 minutes |
68+
69+
### Test code location
70+
71+
- Pester tests: alongside the script under test (`Foo.ps1``Foo.Tests.ps1`)
72+
- KQL signal seed data: `src/azure-monitor/kql/signals/<signal>.tests.json`
73+
- Cross-track parity: `tests/Test-Parity.ps1` (top-level `tests/`)
74+
- Behavioral acceptance scripts: `tests/acceptance/` (excluded from PR runs; tagged `Acceptance`)
75+
76+
### What is **not** tested in CI
77+
78+
- SCOM-side end-to-end with real cluster — too expensive for PR runs; covered by Layer 5 release gate
79+
- Performance / scale (max cluster size, max signals) — captured as a separate Phase 6 perf-baseline exercise, not a gate
80+
- Localization — out of scope (English-only deliverable)
81+
82+
## Consequences
83+
84+
- **Positive**:
85+
- Every PR has a fast, hermetic gate (Layer 1 + 2) that catches structural and logic regressions before integration cost is paid.
86+
- Layer 4 mechanically enforces cross-track parity — the design promise from ADR 0007 cannot silently rot.
87+
- Layer 5 acceptance gate gives confidence to cut a SemVer release without manual smoke testing.
88+
- **Negative**:
89+
- Authoring overhead — every discovery script and every KQL signal requires a sibling test file. Will slow down Phase 3 day-1 velocity.
90+
- Layer 3 requires an always-on lab subscription with a seeded LAW, which has small but ongoing Azure cost.
91+
- **Neutral**:
92+
- Test runners are GitHub-hosted Windows for MP/Pester, ubuntu-latest for everything else.
93+
- **Affected components / owners**: All Phase 3 and Phase 4 deliverables. The first PR for Phase 3 must include `tests/Test-Parity.ps1` and at least one Pester test file as the pattern reference.
94+
95+
## Alternatives considered
96+
97+
- **Manual test plan, no automation** — rejected; cross-track parity cannot be sustained by manual review across ~250 signals.
98+
- **Pester for Bicep too** — rejected; `bicep what-if` is the right tool for IaC validation, Pester for PowerShell only.
99+
- **End-to-end test on every PR** — rejected; too expensive (Azure deployment time, lab quota), held to nightly + release.
100+
- **Test in production SCOM** — rejected; Layer 5 uses pre-prod SCOM only.
101+
- **Skip Layer 4 (parity)** — rejected; ADR 0007's value is gated entirely on this test existing.
102+
103+
## References
104+
105+
- ADR 0003 — Health rollup policy (drives Layer 2 rollup tests)
106+
- ADR 0007 — Naming convention (drives Layer 4 parity test)
107+
- ADR 0013 — Azure Monitor deployment strategy (Bicep what-if as Layer 1+3)
108+
- ADR 0014 — CI/CD pipeline (workflows that run these tests)
109+
- [Pester](https://pester.dev/)
110+
- [PSScriptAnalyzer](https://learn.microsoft.com/en-us/powershell/module/psscriptanalyzer/)
111+
- [KustoLoco](https://github.com/NeilMacMullen/KustoLoco) — local Kusto evaluator for offline KQL tests
112+
- [bicep what-if](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/deploy-what-if)
113+
- Brian Wren — MPVerify and MPBPA usage

0 commit comments

Comments
 (0)