Build a generated data-audit dashboard (gh-pages) across the 8 Python-family repos

[mmcky]

Handover — build a generated data-audit dashboard for data-lectures (gh-pages)

Mission

Turn the one-time, hand-built dataset audit (a private claude.ai artifact, snapshot 2026-07-15) into a generated, self-updating data-audit dashboard hosted on GitHub Pages in QuantEcon/data-lectures, covering all data referenced by the 8 synced Python-family lecture repos. "Generated" is load-bearing: the dashboard must be rebuilt from live repo state by a script (clone + grep), so it stays correct as migrations proceed — not a hand-edited HTML file that rots.

Starting point — the prior artifact (reuse its design, distrust its data)

• URL: https://claude.ai/code/artifact/87174334-f9e0-4b5c-b4df-0739a5abcdc2 — "QuantEcon lecture datasets — registry & audit". Owned by the user; WebFetch it (claude.ai/code/artifact URLs are WebFetch-able; curl gets the SPA shell). 55 KB self-contained HTML, light/dark theme-aware, responsive.
• Its structure is the design template — keep it: stat tiles; §1 hosting-pattern taxonomy; §2 Registry A (static files, grouped by repo); §3 Registry B (%%file embedded); §4 Registry C (live-API); §5 cross-lecture/series reuse; §6 orphans (committed-but-unreferenced); §7 provenance classes (verbatim / constructed / dynamic-snapshot / author-assembled); §8 live-API pedagogy analysis + draft styleguide rules.
• Its DATA is stale (2026-07-15). Regenerate from current main of every repo. Do NOT copy its tables forward — several rows are now wrong (see "What changed since").

Scope — the 8 synced Python-family repos

lecture-python-intro, lecture-python-programming, lecture-python.myst, lecture-python-advanced.myst, lecture-jax, lecture-dp, lecture-wasm, continuous_time_mcs. (The prior artifact covered 7; add lecture-wasm, newly onboarded — it mirrors intro's .md sources and carries its own data reads.) They're already cloned under repos/<name>/ in the workspace (make sync / bin/foreach to refresh). Out of scope by decision: translations (.zh-cn/.fr/.fa) and the topic-based series (workspace-lectures#18).

Method — clone + grep, NEVER gh search code

gh search code cannot find URLs — it tokenizes on / and ., so URL queries return a clean zero (it already produced wrong counts: missed lecture-python.zh-cn's 9 refs, mis-counted lecture-stats 7-vs-6). Clone the repo and grep origin/main. Grep for: read_csv|read_excel|read_stata|read_parquet|read_pickle|read_json|\.npy|np\.load|loadmat|urlretrieve|requests\.get|wbgapi|pandas_datareader|DataReader|fredgraph|yfinance|%%file|%%writefile across *.md and *.ipynb. Classify each hit into the 7 hosting patterns (own-repo URL / local relative path / external data-repo / legacy-repo URL / %%file embedded / live API / data-lectures) and the provenance classes. For each: file/URL, consuming lecture(s), hosting pattern, provenance, flags (Colab-breaking local paths, LFS media-vs-raw, branch-pinned refs, orphans), and migration status (migrated to data-lectures / pending / live-API / embedded).

What changed since the 2026-07-15 snapshot (must be reflected)

• P1 migrated lingcod_msy_recovery.csv → data-lectures; intro msy_fishery.md now reads the interim URL (was local-path).
• P2 migrated the pandas_panel trio realwage.csv / countries.csv / employ.csv → data-lectures; both lecture-python-programming and lecture-python.myst pandas_panel.md now read data-lectures (were legacy-repo / own-repo copies, now deleted).
• SCF ..._no_weights.csv branch-pin fixed → reads high_dim_data main (was a critical branch-only ref).
• ols maketable .dta files repointed off the legacy repo to own-repo copies.
• Legacy QuantEcon/lecture-python renamed to lecture-python.rst and archived (workspace-lectures#17); its 8 legacy-repo URLs retired.
• Net: the "8 legacy-repo URLs" and several "branch-pinned"/"legacy" flags in the old artifact are now resolved; 4 files (lingcod + trio) moved to a new "migrated → data-lectures" state.

Architecture

1. Generator (Python, in data-lectures scripts/, e.g. build_audit.py): reads each cloned repo's main, greps + classifies, emits structured data (JSON) → renders the HTML dashboard. Keep the two stages separate so the data is reusable.
2. Reuse the manifests for the migrated subset. data-lectures already has scripts/build_catalog.py → CATALOG.md, generated from lectures/*.yml (the migrated-only registry, 4 datasets). The audit is the full-universe superset; for migrated files it should pull class/license/consumers/integrity straight from the manifests (single source of truth), and only grep the lecture repos for the not-yet-migrated rows. Don't duplicate manifest data.
3. gh-pages hosting. Deploy the dashboard as a page on data-lectures' Pages site. NOTE the Phase-4 gate: data.quantecon.org is a bare A record → a dead AWS-Sydney box (52.64.86.66); the custom domain isn't live yet and needs infra sign-off. So either ship the dashboard on the default quantecon.github.io/data-lectures/ Pages URL now, or coordinate with the Phase-4 Pages/DNS work. data-lectures has no .github/ yet — a Pages workflow is new (Phase 5).
4. Keep it current: a CI check (or the same Phase-5 workflow) regenerates and asserts no diff, so a migration that isn't reflected fails the build — same pattern as CATALOG.md.

Design bar (it's a dashboard, make it good)

Load the dataviz skill before writing chart/tile code. The prior artifact is already a solid, brand-neutral, theme-aware baseline (CSS variables for light/dark, stat tiles, pill taxonomy, responsive tables) — match or exceed it. Self-contained HTML (Pages CSP-friendly: inline CSS/JS, no external fetches). Keep pattern/provenance colour pills consistent. Add summary stat tiles (total files, migrated / pending / live-API / orphaned, legacy refs remaining → should now be ~0).

Relationship to existing artifacts (don't duplicate)

• CATALOG.md (data-lectures) — generated migrated-only registry; the audit's migrated rows should agree with it.
• data-lectures#15 — interim→final URL cutover tracker (issue).
• workspace-lectures#20 — licensing tracker (countries.csv WorldData all-rights-reserved).
• meta#336 (convention proposal), meta#337 (hosting risks / orphans), meta#338 (pilot), QuantEcon.manual#108 (draft styleguide/datasets.md). The audit's §8 draft rules should feed manual#108.
• PLAN.md Phase 2 (catalog — done) and Phase 9 (broad sweep — the audit is its driver).

Project state for context

• Pilot P1 (lingcod) and P2 (pandas_panel trio) are complete and merged. P3 (heavy_tails + SCF — the LFS case) and P4 (UNRATE — dynamic-snapshot case) remain.
• data-lectures main has branch protection (PRs required, zero-approval solo-merge OK; no force-push/delete). Work on a branch, open a PR.
• Cross-repo commit/PR hygiene: never put a closing keyword (fixes/closes/…) immediately before an owner/repo#N reference (auto-closes it). GitHub prose: don't hard-wrap; no prose in code fences.
• Model note: this work benefits from being run on Opus with 1M context (opus[1m]).

First moves for the fresh session

1. WebFetch the prior artifact; extract its section structure + taxonomy as the design spec.
2. make sync (or confirm repos/ current); clone+grep all 8 repos on main; build the structured dataset; reconcile the migrated subset against lectures/*.yml.
3. Diff your fresh numbers against the 2026-07-15 artifact to prove what changed (sanity check + a nice "what moved" story).
4. Write scripts/build_audit.py (data → HTML), generate the dashboard, add a Pages workflow, open a PR on data-lectures.