diff --git a/CATALOG.md b/CATALOG.md new file mode 100644 index 0000000..9980b97 --- /dev/null +++ b/CATALOG.md @@ -0,0 +1,20 @@ + + + + +# Dataset catalog — `QuantEcon/data-lectures` + +The migrated-dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest; files not yet migrated are tracked in [PLAN.md](PLAN.md) Phase 9. + +**4 datasets migrated** · 1.8 MB total · 3 permitted / 1 restricted redistribution + +| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)
WorldData.info country reference table | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | +| [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)
Eurostat employment in Europe — by age and sex, 2007–2016 | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | +| [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)
Pacific Coast lingcod — biomass and fishing pressure relative to MSY | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) | +| [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | + +--- + +**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9). diff --git a/PLAN.md b/PLAN.md index ac96525..5ed6f0b 100644 --- a/PLAN.md +++ b/PLAN.md @@ -53,11 +53,11 @@ Ordering note: phases 1–3 and 6 can proceed now; phase 4 needs the DNS questio - [x] Rename `QuantEcon/data` → `data-lectures` (GitHub redirects preserve all existing URLs, so this was non-breaking) - [x] Add repo description and topics (`quantecon`, `datasets`, `economics`, `open-data`, `teaching-materials`) -### Phase 2 — Layout (2026-07-16 — catalog outstanding) +### Phase 2 — Layout (2026-07-16) - [x] Restructure consumer-keyed tree → flat published tree — `lectures/` is the published root; no folder implies series ownership. Done while zero lectures referenced the repo, so no consumer could break; **that window is now closed** - [x] Decide where non-published assets live relative to the published tree — `scripts/` and `manifest-schema.yml` sit at the root, outside `lectures/`, and are never served. Manifests are the exception: they live *inside* `lectures/` as sidecars named `.yml`, so a dataset cannot be moved or removed without its metadata, and CI can assert the pairing with a glob -- [ ] Generate an index/catalog page from the manifests — doubles as the dataset registry +- [x] Generate an index/catalog page from the manifests — doubles as the dataset registry. `scripts/build_catalog.py` emits `CATALOG.md` (migrated-only registry) from `lectures/*.yml`; regenerate on any manifest change, and a Phase 5 CI check will assert it is current (`git diff --exit-code`). Feeds the Pages index at Phase 4 The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.yml`) because a stem-keyed sidecar collides when one dataset ships in two formats — exactly the `fig_3.xlsx` / `fig_3.ods` case this repo already had. Strawman until the pilot tests it; see `manifest-schema.yml`. diff --git a/requirements.txt b/requirements.txt index ee92909..0eb37a5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1,2 @@ wbgapi==1.0.12 +PyYAML==6.0.3 # scripts/build_catalog.py — parse the sidecar manifests diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py new file mode 100644 index 0000000..468b5a2 --- /dev/null +++ b/scripts/build_catalog.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +"""Generate CATALOG.md — the migrated-dataset registry — from the sidecar +manifests in lectures/*.yml. + +CATALOG.md is a GENERATED file: the per-dataset manifests are the source of +truth, and this script is the only thing that should write the catalog. Run it +after adding or editing any manifest: + + python scripts/build_catalog.py + +CI asserts the catalog is current with `git diff --exit-code CATALOG.md` after +regenerating, so a stale catalog fails the build (PLAN Phase 5). + +Scope: migrated-only. A dataset appears here once it has a manifest; files in +lectures/ without a manifest are not yet migrated and are tracked in PLAN +Phase 9, not here. +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import yaml + +REPO = Path(__file__).resolve().parent.parent +LECTURES = REPO / "lectures" +OUT = REPO / "CATALOG.md" + +# Interim URL form (AGENTS.md); swaps to data.quantecon.org/lectures/ at Phase 4. +RAW = "https://github.com/QuantEcon/data-lectures/raw/main/lectures" + + +def human_size(n: int) -> str: + size = float(n) + for unit in ("B", "KB", "MB", "GB"): + if size < 1024 or unit == "GB": + return f"{int(size)} {unit}" if unit == "B" else f"{size:.1f} {unit}" + size /= 1024 + return f"{size:.1f} GB" + + +def oneline(text) -> str: + """Collapse a folded/multiline scalar to a single trimmed line.""" + if text is None: + return "" + return " ".join(str(text).split()) + + +def load_manifests(): + manifests = [] + for path in sorted(LECTURES.glob("*.yml")): + with path.open(encoding="utf-8") as f: + m = yaml.safe_load(f) + if not isinstance(m, dict) or "filename" not in m: + print(f"skip (not a manifest): {path.name}", file=sys.stderr) + continue + m["_manifest_path"] = path + data_file = LECTURES / m["filename"] + m["_size"] = data_file.stat().st_size if data_file.exists() else None + manifests.append(m) + return manifests + + +def fmt_consumers(consumers) -> str: + if not consumers: + return "—" + parts = [] + for c in consumers: + repo = c.get("repo", "") + short = repo.split("/")[-1] if repo else "?" + file = c.get("file", "") + stem = file.split("/")[-1] if file else "" + if repo and file: + parts.append(f"[{short} · {stem}](https://github.com/{repo}/blob/main/{file})") + else: + parts.append(f"{short} · {stem}".strip(" ·")) + return "
".join(parts) + + +def fmt_source(source) -> str: + if not isinstance(source, dict): + return oneline(source) + name = oneline(source.get("name", "")) + url = source.get("url") + return f"[{name}]({url})" if url else name + + +def fmt_redist(license) -> str: + if not isinstance(license, dict): + return "?" + r = license.get("redistribution", "?") + return {"permitted": "✅ permitted", "restricted": "⚠️ restricted"}.get(r, str(r)) + + +def fmt_integrity(integrity) -> str: + if not isinstance(integrity, dict): + return "?" + up = integrity.get("upstream") or {} + status = up.get("status", "?") + mark = {"verified": "✅", "spot-checked": "◑", "unverifiable": "⚠️", + "unverified": "…", "failing": "❌"}.get(status, "") + return f"{mark} {status}".strip() + + +def fmt_builder(m) -> str: + if m.get("class") == "verbatim": + return "n/a (verbatim)" + bs = m.get("builder_status", "?") + return {"committed": "✅ committed", "unrecovered": "⚠️ unrecovered", + "not-applicable": "n/a"}.get(bs, str(bs)) + + +def build(manifests) -> str: + total = sum(m["_size"] or 0 for m in manifests) + permitted = sum(1 for m in manifests + if (m.get("license") or {}).get("redistribution") == "permitted") + restricted = len(manifests) - permitted + + lines = [] + lines.append("") + lines.append("") + lines.append("") + lines.append("") + lines.append("# Dataset catalog — `QuantEcon/data-lectures`") + lines.append("") + lines.append( + "The migrated-dataset registry, **auto-generated** from the sidecar " + "manifests (`lectures/*.yml`). Do not edit by hand — run " + "`python scripts/build_catalog.py`. A dataset appears here once it has a " + "manifest; files not yet migrated are tracked in " + "[PLAN.md](PLAN.md) Phase 9." + ) + lines.append("") + lines.append( + f"**{len(manifests)} datasets migrated** · {human_size(total)} total · " + f"{permitted} permitted / {restricted} restricted redistribution" + ) + lines.append("") + lines.append("| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by |") + lines.append("| --- | --- | --- | --- | --- | --- | --- | --- | --- |") + for m in manifests: + fn = m["filename"] + title = oneline(m.get("title", "")) + dataset = f"[**{fn}**]({RAW}/{fn})" + if title: + dataset += f"
{title}" + license = m.get("license") or {} + row = [ + dataset, + m.get("class", "?"), + fmt_source(m.get("source")), + oneline(license.get("name", "?")), + fmt_redist(license), + fmt_integrity(m.get("integrity")), + fmt_builder(m), + human_size(m["_size"]) if m["_size"] is not None else "—", + fmt_consumers(m.get("consumers")), + ] + lines.append("| " + " | ".join(row) + " |") + lines.append("") + lines.append("---") + lines.append("") + lines.append( + "**Legend** — *Integrity* is the `integrity.upstream.status` (is this " + "what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · " + "… unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as " + "inherited exposures and tracked for licence review " + "([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). " + "*Builder* ⚠️ unrecovered marks a constructed dataset whose builder was " + "never committed (PLAN Phase 9)." + ) + lines.append("") + return "\n".join(lines) + + +def main() -> int: + manifests = load_manifests() + if not manifests: + print("no manifests found under lectures/*.yml", file=sys.stderr) + return 1 + OUT.write_text(build(manifests), encoding="utf-8") + print(f"wrote {OUT.relative_to(REPO)} — {len(manifests)} datasets") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())