Skip to content

Commit a67dd82

Browse files
hyperpolymathhyperpolymath
authored
tooling(scripts): nested-path support — Git Tree helper + classifier consumers (#204)
## Summary Two-commit change adding nested-path support to the sweep-classifier pipeline: 1. **`scripts/sweep-classifiers/list-workflow-paths.sh`** — walks `gh repo list` and queries each repo's Git Tree API directly. Bypasses two compounding undercounts in `gh api /search/code`. 2. **All 4 `classify-*.sh` scripts updated** to consume the helper's TSV output and emit the sweep-target path as an explicit column. ## Why the helper exists — 3 layers of undercount 1. **Layer 1 — path-prefix filter:** `path:.github/workflows` matches the path PREFIX, excluding nested `<subdir>/.github/workflows/<file>.yml` paths outright. 2. **Layer 2 — org-scope truncation:** even broad `filename:<file>.yml org:<org>` queries hit internal caps. Validated against `scorecard.yml`: broad query saw 152 paths (all flagged top-level); per-repo enumeration found **626 additional nested copies** the broad query missed entirely. 3. **Layer 3 — nested workflows are inert:** GitHub Actions only runs `.github/workflows/` at the repo root. Nested copies are vendored templates / stale leftover. Security campaigns gain nothing from sweeping nested copies; single-source-of-truth campaigns still benefit. ## Helper output TSV, one row per matching workflow file: ``` <repo>\t<path>\t<blob-sha>\t<top-level|nested> ``` Cost: one Git Tree API call per repo (~300 calls), uses `core` bucket (5000/hr) not throttled `code_search` (10/min). ## Classifier extensions Each `classify-*.sh` now auto-detects input format from the first byte: - `{` → JSONL from `gh /search/code` (legacy path) - otherwise → TSV from `list-workflow-paths.sh` (preferred — handles nested) Output is unified to 7 columns: `repo \t path \t sha \t class \t reason \t lines \t details`. The new `path` column carries the file's location inside the repo, so sweeps can target nested copies as first-class wrapper sites. Shared `normalize_input` extracted into `_lib.sh`; each classifier sources it. ## Validation Smoke-tested both input paths: - TSV (helper): classify-mirror.sh on scorecard-tuples.tsv (287 repos × top-level + nested) — fetches blobs and emits per-(repo, path) rows. - JSONL (legacy): classify-mirror.sh on mirror-full.json — 267 TRIVIAL + 22 NEEDS_REVIEW, matching prior `/tmp/drift-survey/sweep-report.md`. ## Stacked on #194 `scripts/sweep-classifiers/` only exists once #194 merges. The diff against `main` includes #194's files transitively; once #194 lands, this PR narrows to just the helper + extensions. ## Standing follow-ups - Once this lands, re-survey each candidate with the helper for ground-truth wrapper-site counts before firing any sweep. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent a6c97f9 commit a67dd82

7 files changed

Lines changed: 517 additions & 0 deletions

File tree

‎scripts/sweep-classifiers/README.adoc‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
= Sweep classifiers
2+
:SPDX-License-Identifier: PMPL-1.0-or-later
3+
4+
Per-template classifiers used to triage the wrapper-sweep work that
5+
follows each of the foundational reusable PRs filed against
6+
`standards`:
7+
8+
[cols="2,1,3", options="header"]
9+
|===
10+
| Classifier | Reusable PR | Template
11+
| `classify-mirror.sh` | #187 | `mirror.yml` (7-forge mirror bundle)
12+
| `classify-secret-scanner.sh` | #190 | `secret-scanner.yml` (trufflehog/gitleaks/rust/shell)
13+
| `classify-codeql.sh` | #192 | `codeql.yml` (CodeQL security analysis)
14+
| `classify-hypatia-scan.sh` | #193 | `hypatia-scan.yml` (Hypatia neurosymbolic scan)
15+
|===
16+
17+
== Pipeline
18+
19+
. Paginate `gh api /search/code` for the template across the estate.
20+
. Group results by blob SHA; fetch each unique blob exactly once.
21+
. Classify each blob (job-set match / line-count band / language matrix).
22+
. Emit per-repo TSV: `<repo>\t<sha>\t<class>\t<reason>\t<lines>\t<details>`.
23+
24+
The expensive step (blob fetch) is cached in `$BLOBS_DIR`, so reruns
25+
are fast.
26+
27+
== Usage
28+
29+
[source,bash]
30+
----
31+
# 1. Paginate the search (one-time per template):
32+
gh api --paginate -X GET '/search/code' \
33+
-f q='filename:mirror.yml path:.github/workflows org:hyperpolymath' \
34+
--jq '.items[] | {repo: .repository.name, sha: .sha}' \
35+
> /tmp/mirror-full.json
36+
37+
# 2. Run the classifier:
38+
BLOBS_DIR=/tmp/mirror-blobs \
39+
./classify-mirror.sh /tmp/mirror-full.json > /tmp/mirror-classification.tsv
40+
41+
# 3. Summarise:
42+
awk -F'\t' '{print $3}' /tmp/mirror-classification.tsv | sort | uniq -c | sort -rn
43+
----
44+
45+
== Nested-path caveat — 3 layers of undercount
46+
47+
`gh api /search/code` undercounts monorepo-nested workflow files in
48+
three compounding ways:
49+
50+
. *Layer 1 — `path:` filter is a prefix match.* `path:.github/workflows`
51+
excludes paths like `a2ml/bindings/deno/.github/workflows/mirror.yml`
52+
outright.
53+
. *Layer 2 — even broad queries are org-scope-truncated.* Removing the
54+
`path:` filter and running `filename:codeql.yml org:hyperpolymath`
55+
returns ~190 paths, but per-repo enumeration of just
56+
`developer-ecosystem` finds 170 codeql.yml files. The endpoint
57+
silently caps results once it hits internal limits. Empirically
58+
validated against `scorecard.yml`: broad query saw 152 paths, all
59+
top-level; per-repo enumeration revealed 626 nested copies the
60+
broad query missed entirely.
61+
. *Layer 3 — nested workflows do not execute.* GitHub Actions only runs
62+
workflows from the repo-root `.github/workflows/` directory. Nested
63+
copies are inert vendored templates or stale leftover. Security-driven
64+
campaigns (e.g. propagating a missing guardrail) do NOT close
65+
additional attack surface via nested wrappers. Single-source-of-truth
66+
campaigns still benefit.
67+
68+
For accurate counts, use `list-workflow-paths.sh` (this directory),
69+
which walks `gh repo list` and queries each repo's Git Tree API
70+
directly — bypassing Layers 1 and 2.
71+
72+
[source,bash]
73+
----
74+
./list-workflow-paths.sh codeql.yml \
75+
> /tmp/drift-survey/codeql-all-tuples.tsv
76+
77+
# Top-level vs nested split:
78+
awk -F'\t' '{print $4}' /tmp/drift-survey/codeql-all-tuples.tsv \
79+
| sort | uniq -c
80+
----
81+
82+
Output format: `<repo>\t<path>\t<blob-sha>\t<top-level|nested>`.
83+
84+
The legacy `gh api /search/code` query is still useful as a quick first
85+
look, but `list-workflow-paths.sh` is the source of truth for any
86+
sweep-planning decision.
87+
88+
== Output classes (varies by classifier)
89+
90+
* `TRIVIAL` / `TRIVIAL_CURRENT` / `TRIVIAL_DEFAULT` — canonical match;
91+
pure mechanical wrapper conversion.
92+
* `MISSING_*` / `SLIM_*` / `OLDER_*` — propagation lag; same wrapper as
93+
TRIVIAL upgrades the workflow body on first run after merge.
94+
* `SINGLE_NON_DEFAULT` / `MULTI_LANGUAGE` — requires one or more input
95+
overrides at the call site.
96+
* `NEEDS_REVIEW` — extra jobs, line count out of any known band, or
97+
custom workflow body. Per-repo diff required before wrapper.
98+
99+
== Related
100+
101+
* `scripts/apply-baseline.sh` — Hypatia-finding baseline filter (paired
102+
with `scripts/tests/apply-baseline-test.sh`).
103+
* PRs #187, #190, #192, #193 — the reusables these classifiers triage.

‎scripts/sweep-classifiers/_lib.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: PMPL-1.0-or-later
3+
# _lib.sh — Shared helpers for classify-*.sh scripts.
4+
#
5+
# Sourced by every classifier. Not directly executable.
6+
#
7+
# Provides: normalize_input <file>
8+
#
9+
# Emits one TSV row per workflow file: "<repo>\t<path>\t<sha>".
10+
# Accepts two input shapes auto-detected from the first line:
11+
#
12+
# 1. JSONL from `gh api /search/code` (legacy):
13+
# {"repo":"foo","sha":"abc123"}
14+
# Emitted path is empty (no nested-copy info available).
15+
#
16+
# 2. TSV from `list-workflow-paths.sh` (preferred — handles nested):
17+
# foo<TAB><path><TAB>abc123<TAB>top-level|nested
18+
# Emitted path is the original `<path>` value; the scope column
19+
# is dropped.
20+
21+
normalize_input() {
22+
local file="$1"
23+
if [[ "$(head -c1 "$file")" == "{" ]]; then
24+
jq -r '[.repo, (.path // ""), .sha] | @tsv' "$file"
25+
else
26+
awk -F'\t' 'NF >= 3 { print $1 "\t" $2 "\t" $3 }' "$file"
27+
fi
28+
}

‎scripts/sweep-classifiers/classify-codeql.sh‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: PMPL-1.0-or-later
3+
# classify-codeql.sh — classify per-repo codeql.yml for #192 sweep.
4+
#
5+
# Canonical: 49 lines, single `analyze` job, matrix language=javascript-typescript
6+
# build-mode=none.
7+
#
8+
# Classes:
9+
# TRIVIAL_DEFAULT — single javascript-typescript language, default wrapper
10+
# SINGLE_NON_DEFAULT — single rust or actions language, override wrapper
11+
# MULTI_LANGUAGE — 2+ languages, multi-call wrapper
12+
# NEEDS_REVIEW — NONE language matrix, or large custom workflow
13+
14+
set -euo pipefail
15+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
16+
# shellcheck source=_lib.sh
17+
. "$SCRIPT_DIR/_lib.sh"
18+
19+
INPUT="${1:-/tmp/drift-survey/codeql-full.json}"
20+
BLOBS_DIR="${BLOBS_DIR:-/tmp/drift-survey/codeql-blobs}"
21+
mkdir -p "$BLOBS_DIR"
22+
23+
classify_blob() {
24+
local blob="$1" langs lines jobs
25+
jobs=$(awk '/^jobs:[[:space:]]*$/{in_jobs=1; next} in_jobs && /^[A-Za-z]/{exit} in_jobs && /^ [a-z][a-z0-9_-]*:[[:space:]]*$/{sub(/^ /,""); sub(/:.*/,""); print}' "$blob" 2>/dev/null | sort -u | paste -sd, -)
26+
langs=$(grep -E "^\s*- language:" "$blob" 2>/dev/null | sed 's/.*language: //; s/[[:space:]]*$//' | sort -u | paste -sd, -)
27+
lines=$(wc -l < "$blob")
28+
langs="${langs:-NONE}"
29+
30+
if [ "$lines" -gt 80 ]; then
31+
echo "NEEDS_REVIEW custom_workflow_${lines}_lines $lines $langs"
32+
return
33+
fi
34+
case "$langs" in
35+
javascript-typescript) echo "TRIVIAL_DEFAULT - $lines $langs" ;;
36+
rust|actions) echo "SINGLE_NON_DEFAULT +language=$langs $lines $langs" ;;
37+
NONE) echo "NEEDS_REVIEW no_language_matrix $lines $langs" ;;
38+
*,*) echo "MULTI_LANGUAGE +per-language-wrapper $lines $langs" ;;
39+
*) echo "NEEDS_REVIEW unknown_lang:$langs $lines $langs" ;;
40+
esac
41+
}
42+
43+
echo "[fetch] retrieving unique blobs..." >&2
44+
normalize_input "$INPUT" | awk -F'\t' '{print $3 "\t" $1}' | sort -u | while IFS=$'\t' read -r sha repo; do
45+
[ -z "$sha" ] || [ "$sha" = "null" ] && continue
46+
blob_file="$BLOBS_DIR/$sha.yml"
47+
[ -s "$blob_file" ] && continue
48+
gh api "/repos/hyperpolymath/$repo/git/blobs/$sha" --jq '.content' 2>/dev/null \
49+
| base64 -d > "$blob_file" || echo "::warn fetch failed for $sha ($repo)" >&2
50+
done
51+
52+
declare -A SHA_CLASS
53+
echo "[classify] classifying unique blobs..." >&2
54+
for blob in "$BLOBS_DIR"/*.yml; do
55+
[ -s "$blob" ] || continue
56+
sha=$(basename "$blob" .yml)
57+
SHA_CLASS[$sha]=$(classify_blob "$blob")
58+
done
59+
60+
normalize_input "$INPUT" | while IFS=$'\t' read -r repo path sha; do
61+
[ -z "$sha" ] || [ "$sha" = "null" ] && { printf '%s\t%s\t%s\tNEEDS_REVIEW\tnull_sha\t-\t-\n' "$repo" "$path" "$sha"; continue; }
62+
printf '%s\t%s\t%s\t%s\n' "$repo" "$path" "$sha" "${SHA_CLASS[$sha]:-NEEDS_REVIEW fetch_failed - -}"
63+
done

‎scripts/sweep-classifiers/classify-hypatia-scan.sh‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: PMPL-1.0-or-later
3+
# classify-hypatia-scan.sh — classify per-repo hypatia-scan.yml for #193 sweep.
4+
#
5+
# Canonical (standards/.github/workflows/hypatia-scan.yml): 416 lines,
6+
# single `scan` job. Every estate variant carries the same job set;
7+
# drift is pure propagation lag, manifested as line count.
8+
#
9+
# Classes:
10+
# TRIVIAL_CURRENT — 410-416 lines (current canonical-1)
11+
# SLIM_PROPAGATION_LAG — 245-260 lines (older slimmer version)
12+
# OLDER_SLIM — 205-235 lines (much older slim version)
13+
# NEEDS_REVIEW — anything else (multi-job, line count out of band,
14+
# or self-source like hypatia/standards repos)
15+
16+
set -euo pipefail
17+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
18+
# shellcheck source=_lib.sh
19+
. "$SCRIPT_DIR/_lib.sh"
20+
21+
INPUT="${1:-/tmp/drift-survey/hypatia-scan-full.json}"
22+
BLOBS_DIR="${BLOBS_DIR:-/tmp/drift-survey/hypatia-blobs}"
23+
mkdir -p "$BLOBS_DIR"
24+
25+
classify_blob() {
26+
local blob="$1" jobs lines
27+
jobs=$(awk '/^jobs:[[:space:]]*$/{in_jobs=1; next} in_jobs && /^[A-Za-z]/{exit} in_jobs && /^ [a-z][a-z0-9_-]*:[[:space:]]*$/{sub(/^ /,""); sub(/:.*/,""); print}' "$blob" 2>/dev/null | sort -u | paste -sd, -)
28+
jobs="${jobs:-NONE}"
29+
lines=$(wc -l < "$blob")
30+
if [ "$jobs" != "scan" ]; then
31+
echo "NEEDS_REVIEW job_set:$jobs $lines $jobs"
32+
return
33+
fi
34+
case "$lines" in
35+
41[0-9]|42[0-9]) echo "TRIVIAL_CURRENT - $lines scan" ;;
36+
24[0-9]|25[0-9]) echo "SLIM_PROPAGATION_LAG +upgrade-to-current $lines scan" ;;
37+
20[5-9]|21[0-9]|22[0-9]|23[0-9]) echo "OLDER_SLIM +upgrade-to-current $lines scan" ;;
38+
*) echo "NEEDS_REVIEW line_count_oob:$lines $lines scan" ;;
39+
esac
40+
}
41+
42+
echo "[fetch] retrieving unique blobs..." >&2
43+
normalize_input "$INPUT" | awk -F'\t' '{print $3 "\t" $1}' | sort -u | while IFS=$'\t' read -r sha repo; do
44+
[ -z "$sha" ] || [ "$sha" = "null" ] && continue
45+
blob_file="$BLOBS_DIR/$sha.yml"
46+
[ -s "$blob_file" ] && continue
47+
gh api "/repos/hyperpolymath/$repo/git/blobs/$sha" --jq '.content' 2>/dev/null \
48+
| base64 -d > "$blob_file" || echo "::warn fetch failed for $sha ($repo)" >&2
49+
done
50+
51+
declare -A SHA_CLASS
52+
echo "[classify] classifying unique blobs..." >&2
53+
for blob in "$BLOBS_DIR"/*.yml; do
54+
[ -s "$blob" ] || continue
55+
sha=$(basename "$blob" .yml)
56+
SHA_CLASS[$sha]=$(classify_blob "$blob")
57+
done
58+
59+
normalize_input "$INPUT" | while IFS=$'\t' read -r repo path sha; do
60+
[ -z "$sha" ] || [ "$sha" = "null" ] && { printf '%s\t%s\t%s\tNEEDS_REVIEW\tnull_sha\t-\t-\n' "$repo" "$path" "$sha"; continue; }
61+
printf '%s\t%s\t%s\t%s\n' "$repo" "$path" "$sha" "${SHA_CLASS[$sha]:-NEEDS_REVIEW fetch_failed - -}"
62+
done

‎scripts/sweep-classifiers/classify-mirror.sh‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: PMPL-1.0-or-later
3+
# classify-mirror.sh — classify per-repo mirror.yml for the #187 sweep.
4+
#
5+
# Reads /tmp/drift-survey/mirror-full.json (output of `gh api --paginate
6+
# /search/code` filtered to mirror.yml in .github/workflows). For each
7+
# unique blob SHA, fetches the blob once (cached in $BLOBS_DIR) and
8+
# classifies as:
9+
#
10+
# TRIVIAL — exactly the canonical 7 forges, line count in band,
11+
# no extra/missing top-level jobs.
12+
# NEEDS_REVIEW — anything else (extra forges, missing forges,
13+
# structural diff). Reported with reason.
14+
#
15+
# Output: TSV to stdout, one row per repo:
16+
# <repo>\t<sha>\t<classification>\t<reason>\t<line_count>\t<forges>
17+
18+
set -euo pipefail
19+
20+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
21+
# shellcheck source=_lib.sh
22+
. "$SCRIPT_DIR/_lib.sh"
23+
24+
INPUT="${1:-/tmp/drift-survey/mirror-full.json}"
25+
BLOBS_DIR="${BLOBS_DIR:-/tmp/drift-survey/blobs}"
26+
mkdir -p "$BLOBS_DIR"
27+
28+
# Canonical 7 forges (alphabetical for stable compare).
29+
CANON_FORGES="bitbucket,codeberg,disroot,gitea,gitlab,radicle,sourcehut"
30+
LINE_MIN=130
31+
LINE_MAX=160
32+
33+
classify_blob() {
34+
local blob="$1"
35+
local forges line_count extra_jobs
36+
37+
# Forges: top-level jobs under `jobs:` starting with mirror-.
38+
forges=$(awk '
39+
/^jobs:[[:space:]]*$/ { in_jobs=1; next }
40+
in_jobs && /^[A-Za-z]/ { exit }
41+
in_jobs && /^ mirror-[a-z]+:[[:space:]]*$/ {
42+
sub(/^ mirror-/, ""); sub(/:[[:space:]]*$/, ""); print
43+
}
44+
' "$blob" 2>/dev/null | sort -u | paste -sd, -)
45+
forges="${forges:-NONE}"
46+
line_count=$(wc -l < "$blob")
47+
48+
# Any non-mirror top-level jobs? Scope to the `jobs:` block (between
49+
# `^jobs:$` and the next column-0 key, EOF, or comment-only/blank).
50+
extra_jobs=$(awk '
51+
/^jobs:[[:space:]]*$/ { in_jobs=1; next }
52+
in_jobs && /^[A-Za-z]/ { exit } # next column-0 key ends the block
53+
in_jobs && /^ [a-z][a-z0-9_-]*:[[:space:]]*$/ {
54+
sub(/^ /, ""); sub(/:[[:space:]]*$/, ""); print
55+
}
56+
' "$blob" 2>/dev/null | grep -vE '^mirror-' | paste -sd, -)
57+
extra_jobs="${extra_jobs:-}"
58+
59+
if [ "$forges" != "$CANON_FORGES" ]; then
60+
echo "NEEDS_REVIEW forge_set:$forges $line_count $forges"
61+
return
62+
fi
63+
if [ -n "$extra_jobs" ]; then
64+
echo "NEEDS_REVIEW extra_jobs:$extra_jobs $line_count $forges"
65+
return
66+
fi
67+
if [ "$line_count" -lt "$LINE_MIN" ] || [ "$line_count" -gt "$LINE_MAX" ]; then
68+
echo "NEEDS_REVIEW line_count_oob:$line_count $line_count $forges"
69+
return
70+
fi
71+
echo "TRIVIAL - $line_count $forges"
72+
}
73+
74+
# 1. Fetch each unique SHA's blob (cached).
75+
echo "[fetch] unique SHAs to retrieve..." >&2
76+
normalize_input "$INPUT" | awk -F'\t' '{print $3 "\t" $1}' | sort -u | while IFS=$'\t' read -r sha repo; do
77+
[ -z "$sha" ] || [ "$sha" = "null" ] && continue
78+
blob_file="$BLOBS_DIR/$sha.yml"
79+
[ -s "$blob_file" ] && continue
80+
gh api "/repos/hyperpolymath/$repo/git/blobs/$sha" --jq '.content' 2>/dev/null \
81+
| base64 -d > "$blob_file" || echo "::warn fetch failed for $sha ($repo)" >&2
82+
done
83+
84+
# 2. Classify each unique SHA, cache in a map.
85+
declare -A SHA_CLASS
86+
echo "[classify] classifying unique blobs..." >&2
87+
for blob in "$BLOBS_DIR"/*.yml; do
88+
[ -s "$blob" ] || continue
89+
sha=$(basename "$blob" .yml)
90+
SHA_CLASS[$sha]=$(classify_blob "$blob")
91+
done
92+
93+
# 3. Emit per-(repo, path) TSV: repo \t path \t sha \t class \t reason \t lines \t forges
94+
normalize_input "$INPUT" | while IFS=$'\t' read -r repo path sha; do
95+
[ -z "$sha" ] || [ "$sha" = "null" ] && { printf '%s\t%s\t%s\tNEEDS_REVIEW\tnull_sha\t-\t-\n' "$repo" "$path" "$sha"; continue; }
96+
printf '%s\t%s\t%s\t%s\n' "$repo" "$path" "$sha" "${SHA_CLASS[$sha]:-NEEDS_REVIEW fetch_failed - -}"
97+
done

0 commit comments

Comments
 (0)