Commit 54ce61e
feat(dashboards): add Validation Health Board renderer + daily refresh (microsoft-foundry#313)
* docs: add validation reporting decisions (Design Lock)
Captures Q1 (audience: PMs/rollup), Q3 (freshness: sync + cron), Q4 (grouping: sample-path tree), Q5 (untracked: surfaced distinctly), Q6 (pending: yellow), Q9 (out-of-scope guardrail). Q2 deferred to View Exploration; Q7/Q8 deferred to later phases.
Phase naming is binding: Health Board / Trend History / Team Routing. Do not use G/G1/G2/G3 in branches, PRs, or docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docs: lock Q2 (Health Board form factor) after View Exploration
Committed static markdown page under internal/dashboards/, Upptime-style. Job-summary form factor held as fallback. Accepts one-extra-validation-run-per-refresh tax.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docs: lock post-rubber-duck sub-decisions (Q3 cron-only, Q5 registry-based coverage)
- Q3 revised: dashboard refresh is cron-only + workflow_dispatch.
Drops sync-to-public piggyback to avoid (a) elevating that
workflow's contents:read permission and (b) a verify-sync race
where a dashboard commit between sync's push and verify's
checkout would cause verify to read statuses on the wrong SHA.
Both fixes would touch gate machinery (Q9 forbids).
- Q5 expanded: 'expected' set derives from the pipeline registry
in validation-results-contract.md, covering both ado-build
(sample.yaml) and hosted-agents-e2e (agent.manifest.yaml under
hosted-agents roots), even though hosted-agents-e2e isn't
posting yet.
- Q5b: two distinct coverage numbers (tracked pass rate +
coverage), never compressed into one '% healthy'.
- Q5c: multi-reporter state precedence (failure>pending>success).
- Working agreements: parser reuse via new --json mode on
parse-validation-statuses.sh, markdown escaping on
status-derived fields, push retry on non-FF.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat(scripts): add --json mode to parse-validation-statuses.sh
Adds a per-context structured output mode used by the validation
Health Board reporting layer. Existing gate callers (compute-blocklist.sh)
are untouched and continue to use the colon-list mode.
Output shape (--json):
[{path, pipeline_id, state, target_url, created_at, context}, ...]
- latest-write-wins per context (same dedup rule as blocked mode)
- all states retained (success/failure/error/pending), not just blocking
- target_url defaults to empty string when status payload omits it
- sorted by (path, pipeline_id) for stable rendering
Implementation:
- jq branch and python fallback both honour the new mode.
- --json switch is positional (first arg before the file path).
Tests:
J1-J7 added to test-status-parsing.sh covering counts, fields,
latest-write-wins, success retention, -- decoding, empty input,
and non-validation context filtering. All 16 (S+J) tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat(dashboards): add Validation Health Board renderer + daily refresh
Implements the Health Board phase of validation status reporting per
docs/validation-reporting-decisions.md (Q1-Q9 + sub-decisions).
What this adds:
- .github/scripts/render-validation-health.sh: reads combined commit
statuses for a SHA, parses via parse-validation-statuses.sh --json,
discovers the expected tracked-set from the on-disk registry
(sample.yaml + agent.manifest.yaml under samples/), applies
multi-reporter precedence (failure/error > pending > ungated >
success), and emits a single static markdown page with two
headline numbers (Tracked Pass Rate + Coverage), a per-sample
table, an Ungated section, and a footer.
- .github/workflows/validation-health-refresh.yml: daily cron
(06:30 UTC) + workflow_dispatch. Renders against current main HEAD,
commits the regenerated dashboard with a bounded non-FF retry,
and uses workflow-level concurrency to enforce single-writer.
- internal/dashboards/validation-health.md: seeded placeholder; the
first refresh run replaces it with the real board.
- .github/tests/test-render-validation-health.sh: 7 fixture tests
(R1-R7) covering headline math separation, failure/pending
precedence, the partial-ungated-stays-visible bug fix, evidence
column contents, ungated section presence, and URL pipe encoding.
Locked design points honored:
- Q3 (cron-only refresh, no sync piggyback): respected.
- Q5 (registry rule for ado-build vs hosted-agents-e2e): encoded
in render-validation-health.sh discovery.
- Q5b (two coverage numbers, never compressed): tracked pass rate
and coverage rendered separately.
- Q5c (multi-reporter precedence + evidence column): implemented;
partial-ungated samples stay visible per rubber-duck finding.
- Q9 (do not touch gate machinery): sync-to-public.yml and
verify-sync.yml are untouched.
Notes for review:
- The refresh workflow is intentionally one writer to one file under
internal/dashboards/. There is a known small window where a human
commit landing between checkout and push will cause the retry loop
to render against the new SHA, which may show a transient coverage
collapse if validation hasn't posted for that SHA yet. Accepted
per Q3 design intent (max ~24h staleness; visible noise rate
bounded by once/day cron + manual dispatch).
- Brandon cannot self-approve PRs in this repo. This commit lands on
branch brandom/validation-reporting; PR will need a co-reviewer.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Render: keep all sample rows visible (drop <details> collapse)
Per design revision: at-a-glance read should include both attention-needing and green rows. Removes the green-rows <details> block; rows now flow inline in the same table, sorted worst-state first.
Test R4 strengthened to assert renderer never emits a <details> collapse block.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: a <a@b>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>1 parent a3e107f commit 54ce61e
7 files changed
Lines changed: 1028 additions & 18 deletions
File tree
- .github
- scripts
- tests
- workflows
- docs
- internal/dashboards
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | | - | |
5 | | - | |
6 | | - | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
7 | 17 | | |
8 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
9 | 25 | | |
10 | 26 | | |
11 | 27 | | |
| |||
14 | 30 | | |
15 | 31 | | |
16 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
17 | 69 | | |
18 | 70 | | |
19 | 71 | | |
| |||
57 | 109 | | |
58 | 110 | | |
59 | 111 | | |
| 112 | + | |
60 | 113 | | |
61 | 114 | | |
62 | 115 | | |
63 | 116 | | |
64 | 117 | | |
65 | | - | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
66 | 121 | | |
67 | 122 | | |
68 | 123 | | |
| |||
87 | 142 | | |
88 | 143 | | |
89 | 144 | | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | 145 | | |
100 | 146 | | |
101 | 147 | | |
102 | | - | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
103 | 160 | | |
104 | | - | |
105 | | - | |
106 | | - | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
107 | 177 | | |
108 | 178 | | |
109 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
110 | 187 | | |
111 | 188 | | |
112 | 189 | | |
| |||
0 commit comments