Skip to content

Commit 54ce61e

Browse files
brandom-msftaCopilotgithub-actions[bot]
authored
feat(dashboards): add Validation Health Board renderer + daily refresh (microsoft-foundry#313)
* docs: add validation reporting decisions (Design Lock) Captures Q1 (audience: PMs/rollup), Q3 (freshness: sync + cron), Q4 (grouping: sample-path tree), Q5 (untracked: surfaced distinctly), Q6 (pending: yellow), Q9 (out-of-scope guardrail). Q2 deferred to View Exploration; Q7/Q8 deferred to later phases. Phase naming is binding: Health Board / Trend History / Team Routing. Do not use G/G1/G2/G3 in branches, PRs, or docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: lock Q2 (Health Board form factor) after View Exploration Committed static markdown page under internal/dashboards/, Upptime-style. Job-summary form factor held as fallback. Accepts one-extra-validation-run-per-refresh tax. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: lock post-rubber-duck sub-decisions (Q3 cron-only, Q5 registry-based coverage) - Q3 revised: dashboard refresh is cron-only + workflow_dispatch. Drops sync-to-public piggyback to avoid (a) elevating that workflow's contents:read permission and (b) a verify-sync race where a dashboard commit between sync's push and verify's checkout would cause verify to read statuses on the wrong SHA. Both fixes would touch gate machinery (Q9 forbids). - Q5 expanded: 'expected' set derives from the pipeline registry in validation-results-contract.md, covering both ado-build (sample.yaml) and hosted-agents-e2e (agent.manifest.yaml under hosted-agents roots), even though hosted-agents-e2e isn't posting yet. - Q5b: two distinct coverage numbers (tracked pass rate + coverage), never compressed into one '% healthy'. - Q5c: multi-reporter state precedence (failure>pending>success). - Working agreements: parser reuse via new --json mode on parse-validation-statuses.sh, markdown escaping on status-derived fields, push retry on non-FF. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(scripts): add --json mode to parse-validation-statuses.sh Adds a per-context structured output mode used by the validation Health Board reporting layer. Existing gate callers (compute-blocklist.sh) are untouched and continue to use the colon-list mode. Output shape (--json): [{path, pipeline_id, state, target_url, created_at, context}, ...] - latest-write-wins per context (same dedup rule as blocked mode) - all states retained (success/failure/error/pending), not just blocking - target_url defaults to empty string when status payload omits it - sorted by (path, pipeline_id) for stable rendering Implementation: - jq branch and python fallback both honour the new mode. - --json switch is positional (first arg before the file path). Tests: J1-J7 added to test-status-parsing.sh covering counts, fields, latest-write-wins, success retention, -- decoding, empty input, and non-validation context filtering. All 16 (S+J) tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(dashboards): add Validation Health Board renderer + daily refresh Implements the Health Board phase of validation status reporting per docs/validation-reporting-decisions.md (Q1-Q9 + sub-decisions). What this adds: - .github/scripts/render-validation-health.sh: reads combined commit statuses for a SHA, parses via parse-validation-statuses.sh --json, discovers the expected tracked-set from the on-disk registry (sample.yaml + agent.manifest.yaml under samples/), applies multi-reporter precedence (failure/error > pending > ungated > success), and emits a single static markdown page with two headline numbers (Tracked Pass Rate + Coverage), a per-sample table, an Ungated section, and a footer. - .github/workflows/validation-health-refresh.yml: daily cron (06:30 UTC) + workflow_dispatch. Renders against current main HEAD, commits the regenerated dashboard with a bounded non-FF retry, and uses workflow-level concurrency to enforce single-writer. - internal/dashboards/validation-health.md: seeded placeholder; the first refresh run replaces it with the real board. - .github/tests/test-render-validation-health.sh: 7 fixture tests (R1-R7) covering headline math separation, failure/pending precedence, the partial-ungated-stays-visible bug fix, evidence column contents, ungated section presence, and URL pipe encoding. Locked design points honored: - Q3 (cron-only refresh, no sync piggyback): respected. - Q5 (registry rule for ado-build vs hosted-agents-e2e): encoded in render-validation-health.sh discovery. - Q5b (two coverage numbers, never compressed): tracked pass rate and coverage rendered separately. - Q5c (multi-reporter precedence + evidence column): implemented; partial-ungated samples stay visible per rubber-duck finding. - Q9 (do not touch gate machinery): sync-to-public.yml and verify-sync.yml are untouched. Notes for review: - The refresh workflow is intentionally one writer to one file under internal/dashboards/. There is a known small window where a human commit landing between checkout and push will cause the retry loop to render against the new SHA, which may show a transient coverage collapse if validation hasn't posted for that SHA yet. Accepted per Q3 design intent (max ~24h staleness; visible noise rate bounded by once/day cron + manual dispatch). - Brandon cannot self-approve PRs in this repo. This commit lands on branch brandom/validation-reporting; PR will need a co-reviewer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Render: keep all sample rows visible (drop <details> collapse) Per design revision: at-a-glance read should include both attention-needing and green rows. Removes the green-rows <details> block; rows now flow inline in the same table, sorted worst-state first. Test R4 strengthened to assert renderer never emits a <details> collapse block. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: a <a@b> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent a3e107f commit 54ce61e

7 files changed

Lines changed: 1028 additions & 18 deletions

File tree

.github/scripts/parse-validation-statuses.sh

Lines changed: 95 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,27 @@
11
#!/usr/bin/env bash
2-
# Parse GitHub validation commit statuses into a SYNC_BLOCKED_PATHS value.
2+
# Parse GitHub validation commit statuses.
33
#
4-
# Input: GitHub statuses-list JSON, or an equivalent object with a `statuses` array.
5-
# Output: colon-separated repo-relative sample paths whose latest validation status
6-
# is failure, error, or pending. Empty output means nothing is blocked.
4+
# Default mode (no flag):
5+
# Output: colon-separated repo-relative sample paths whose latest validation
6+
# status is failure, error, or pending. Empty output means nothing
7+
# is blocked. Used by the sync gate (compute-blocklist.sh).
8+
#
9+
# --json mode:
10+
# Output: JSON array of latest-per-context entries with shape
11+
# {path, pipeline_id, state, target_url, created_at, context}.
12+
# All states retained (success/failure/error/pending). Used by
13+
# the validation Health Board reporting layer.
14+
#
15+
# Input: GitHub statuses-list JSON, or an equivalent object with a `statuses`
16+
# array. Provided as a file path or "-" for stdin.
717
set -euo pipefail
818

19+
mode="blocked"
20+
if [[ "${1:-}" == "--json" ]]; then
21+
mode="json"
22+
shift
23+
fi
24+
925
input="${1:--}"
1026

1127
if [[ "$input" != "-" && ! -f "$input" ]]; then
@@ -14,6 +30,42 @@ if [[ "$input" != "-" && ! -f "$input" ]]; then
1430
fi
1531

1632
if command -v jq >/dev/null 2>&1; then
33+
if [[ "$mode" == "json" ]]; then
34+
jq '
35+
def status_items:
36+
if type == "array" then .
37+
elif (.statuses? | type) == "array" then .statuses
38+
else []
39+
end;
40+
41+
def parse_context($ctx):
42+
($ctx | capture("^validation/(?<pipeline>[^/]+)/(?<path>.+)$"))
43+
| { pipeline_id: .pipeline, path: (.path | gsub("--"; "/")) };
44+
45+
[ status_items[]?
46+
| select((.context? | type) == "string")
47+
| select((.state? | type) == "string")
48+
| select(.state == "success" or .state == "failure" or .state == "error" or .state == "pending")
49+
| select(.context | startswith("validation/"))
50+
| select(.context | test("^validation/[^/]+/.+"))
51+
| (parse_context(.context)) as $parsed
52+
| {
53+
path: $parsed.path,
54+
pipeline_id: $parsed.pipeline_id,
55+
state: .state,
56+
target_url: ((.target_url? // "") | tostring),
57+
created_at: ((.created_at? // "") | tostring),
58+
context: .context
59+
}
60+
]
61+
| sort_by(.context, .created_at)
62+
| group_by(.context)
63+
| map(last)
64+
| sort_by(.path, .pipeline_id)
65+
' "$input"
66+
exit 0
67+
fi
68+
1769
jq -r '
1870
def status_items:
1971
if type == "array" then .
@@ -57,12 +109,15 @@ fi
57109

58110
PYTHON_SCRIPT=$(cat <<'PY'
59111
import json
112+
import os
60113
import re
61114
import sys
62115
63116
VALID_STATES = {"success", "failure", "error", "pending"}
64117
BLOCKING_STATES = {"failure", "error", "pending"}
65-
CONTEXT_RE = re.compile(r"^validation/[^/]+/(?P<path>.+)$")
118+
CONTEXT_RE = re.compile(r"^validation/(?P<pipeline>[^/]+)/(?P<path>.+)$")
119+
120+
MODE = os.environ.get("PARSER_MODE", "blocked")
66121
67122
try:
68123
payload = json.load(sys.stdin)
@@ -87,26 +142,48 @@ for item in statuses:
87142
continue
88143
if not context.startswith("validation/") or state not in VALID_STATES:
89144
continue
90-
created_at = str(item.get("created_at", ""))
91-
previous = latest_by_context.get(context)
92-
if previous is None or created_at >= previous["created_at"]:
93-
latest_by_context[context] = {"state": state, "created_at": created_at}
94-
95-
blocked_paths = set()
96-
for context, result in latest_by_context.items():
97-
if result["state"] not in BLOCKING_STATES:
98-
continue
99145
match = CONTEXT_RE.match(context)
100146
if not match:
101147
continue
102-
blocked_paths.add(match.group("path").replace("--", "/"))
148+
created_at = str(item.get("created_at", ""))
149+
target_url = str(item.get("target_url", "") or "")
150+
previous = latest_by_context.get(context)
151+
if previous is None or created_at >= previous["created_at"]:
152+
latest_by_context[context] = {
153+
"state": state,
154+
"created_at": created_at,
155+
"target_url": target_url,
156+
"pipeline_id": match.group("pipeline"),
157+
"path": match.group("path").replace("--", "/"),
158+
"context": context,
159+
}
103160
104-
output = ":".join(sorted(blocked_paths))
105-
if output:
106-
print(output)
161+
if MODE == "json":
162+
rows = sorted(
163+
latest_by_context.values(),
164+
key=lambda r: (r["path"], r["pipeline_id"]),
165+
)
166+
json.dump(rows, sys.stdout)
167+
sys.stdout.write("\n")
168+
else:
169+
blocked_paths = set()
170+
for result in latest_by_context.values():
171+
if result["state"] not in BLOCKING_STATES:
172+
continue
173+
blocked_paths.add(result["path"])
174+
output = ":".join(sorted(blocked_paths))
175+
if output:
176+
print(output)
107177
PY
108178
)
109179

180+
env_prefix=()
181+
if [[ "$mode" == "json" ]]; then
182+
export PARSER_MODE=json
183+
else
184+
export PARSER_MODE=blocked
185+
fi
186+
110187
if [[ "$input" == "-" ]]; then
111188
"$python_bin" -c "$PYTHON_SCRIPT"
112189
else

0 commit comments

Comments
 (0)