Add generated dataset catalog (CATALOG.md + build_catalog.py)

CATALOG.md

@@ -0,0 +1,20 @@
+<!-- GENERATED FILE — do not edit by hand. -->
+<!-- Regenerate: python scripts/build_catalog.py -->
+<!-- Source of truth: the per-dataset manifests at lectures/<file>.yml -->
+
+# Dataset catalog — `QuantEcon/data-lectures`
+
+The migrated-dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest; files not yet migrated are tracked in [PLAN.md](PLAN.md) Phase 9.
+
+**4 datasets migrated** · 1.8 MB total · 3 permitted / 1 restricted redistribution
+
+| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)<br><sub>WorldData.info country reference table</sub> | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
+| [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)<br><sub>Eurostat employment in Europe — by age and sex, 2007–2016</sub> | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
+| [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)<br><sub>Pacific Coast lingcod — biomass and fishing pressure relative to MSY</sub> | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) |
+| [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)<br><sub>OECD real minimum wages — 32 countries, 2006–2016</sub> | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
+
+---
+
+**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9).

PLAN.md

@@ -53,11 +53,11 @@ Ordering note: phases 1–3 and 6 can proceed now; phase 4 needs the DNS questio
 - [x] Rename `QuantEcon/data` → `data-lectures` (GitHub redirects preserve all existing URLs, so this was non-breaking)
 - [x] Add repo description and topics (`quantecon`, `datasets`, `economics`, `open-data`, `teaching-materials`)
 
-### Phase 2 — Layout (2026-07-16 — catalog outstanding)
+### Phase 2 — Layout (2026-07-16)
 
 - [x] Restructure consumer-keyed tree → flat published tree — `lectures/` is the published root; no folder implies series ownership. Done while zero lectures referenced the repo, so no consumer could break; **that window is now closed**
 - [x] Decide where non-published assets live relative to the published tree — `scripts/` and `manifest-schema.yml` sit at the root, outside `lectures/`, and are never served. Manifests are the exception: they live *inside* `lectures/` as sidecars named `<filename>.yml`, so a dataset cannot be moved or removed without its metadata, and CI can assert the pairing with a glob
-- [ ] Generate an index/catalog page from the manifests — doubles as the dataset registry
+- [x] Generate an index/catalog page from the manifests — doubles as the dataset registry. `scripts/build_catalog.py` emits `CATALOG.md` (migrated-only registry) from `lectures/*.yml`; regenerate on any manifest change, and a Phase 5 CI check will assert it is current (`git diff --exit-code`). Feeds the Pages index at Phase 4
 
 The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.yml`) because a stem-keyed sidecar collides when one dataset ships in two formats — exactly the `fig_3.xlsx` / `fig_3.ods` case this repo already had. Strawman until the pilot tests it; see `manifest-schema.yml`.
 

requirements.txt

@@ -1 +1,2 @@
 wbgapi==1.0.12
+PyYAML==6.0.3        # scripts/build_catalog.py — parse the sidecar manifests

scripts/build_catalog.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python3
+"""Generate CATALOG.md — the migrated-dataset registry — from the sidecar
+manifests in lectures/*.yml.
+
+CATALOG.md is a GENERATED file: the per-dataset manifests are the source of
+truth, and this script is the only thing that should write the catalog. Run it
+after adding or editing any manifest:
+
+    python scripts/build_catalog.py
+
+CI asserts the catalog is current with `git diff --exit-code CATALOG.md` after
+regenerating, so a stale catalog fails the build (PLAN Phase 5).
+
+Scope: migrated-only. A dataset appears here once it has a manifest; files in
+lectures/ without a manifest are not yet migrated and are tracked in PLAN
+Phase 9, not here.
+"""
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import yaml
+
+REPO = Path(__file__).resolve().parent.parent
+LECTURES = REPO / "lectures"
+OUT = REPO / "CATALOG.md"
+
+# Interim URL form (AGENTS.md); swaps to data.quantecon.org/lectures/ at Phase 4.
+RAW = "https://github.com/QuantEcon/data-lectures/raw/main/lectures"
+
+
+def human_size(n: int) -> str:
+    size = float(n)
+    for unit in ("B", "KB", "MB", "GB"):
+        if size < 1024 or unit == "GB":
+            return f"{int(size)} {unit}" if unit == "B" else f"{size:.1f} {unit}"
+        size /= 1024
+    return f"{size:.1f} GB"
+
+
+def oneline(text) -> str:
+    """Collapse a folded/multiline scalar to a single trimmed line."""
+    if text is None:
+        return ""
+    return " ".join(str(text).split())
+
+
+def load_manifests():
+    manifests = []
+    for path in sorted(LECTURES.glob("*.yml")):
+        with path.open(encoding="utf-8") as f:
+            m = yaml.safe_load(f)
+        if not isinstance(m, dict) or "filename" not in m:
+            print(f"skip (not a manifest): {path.name}", file=sys.stderr)
+            continue
+        m["_manifest_path"] = path
+        data_file = LECTURES / m["filename"]
+        m["_size"] = data_file.stat().st_size if data_file.exists() else None
+        manifests.append(m)
+    return manifests
+
+
+def fmt_consumers(consumers) -> str:
+    if not consumers:
+        return "—"
+    parts = []
+    for c in consumers:
+        repo = c.get("repo", "")
+        short = repo.split("/")[-1] if repo else "?"
+        file = c.get("file", "")
+        stem = file.split("/")[-1] if file else ""
+        if repo and file:
+            parts.append(f"[{short} · {stem}](https://github.com/{repo}/blob/main/{file})")
+        else:
+            parts.append(f"{short} · {stem}".strip(" ·"))
+    return "<br>".join(parts)
+
+
+def fmt_source(source) -> str:
+    if not isinstance(source, dict):
+        return oneline(source)
+    name = oneline(source.get("name", ""))
+    url = source.get("url")
+    return f"[{name}]({url})" if url else name
+
+
+def fmt_redist(license) -> str:
+    if not isinstance(license, dict):
+        return "?"
+    r = license.get("redistribution", "?")
+    return {"permitted": "✅ permitted", "restricted": "⚠️ restricted"}.get(r, str(r))
+
+
+def fmt_integrity(integrity) -> str:
+    if not isinstance(integrity, dict):
+        return "?"
+    up = integrity.get("upstream") or {}
+    status = up.get("status", "?")
+    mark = {"verified": "✅", "spot-checked": "◑", "unverifiable": "⚠️",
+            "unverified": "…", "failing": "❌"}.get(status, "")
+    return f"{mark} {status}".strip()
+
+
+def fmt_builder(m) -> str:
+    if m.get("class") == "verbatim":
+        return "n/a (verbatim)"
+    bs = m.get("builder_status", "?")
+    return {"committed": "✅ committed", "unrecovered": "⚠️ unrecovered",
+            "not-applicable": "n/a"}.get(bs, str(bs))
+
+
+def build(manifests) -> str:
+    total = sum(m["_size"] or 0 for m in manifests)
+    permitted = sum(1 for m in manifests
+                    if (m.get("license") or {}).get("redistribution") == "permitted")
+    restricted = len(manifests) - permitted
+
+    lines = []
+    lines.append("<!-- GENERATED FILE — do not edit by hand. -->")
+    lines.append("<!-- Regenerate: python scripts/build_catalog.py -->")
+    lines.append("<!-- Source of truth: the per-dataset manifests at lectures/<file>.yml -->")
+    lines.append("")
+    lines.append("# Dataset catalog — `QuantEcon/data-lectures`")
+    lines.append("")
+    lines.append(
+        "The migrated-dataset registry, **auto-generated** from the sidecar "
+        "manifests (`lectures/*.yml`). Do not edit by hand — run "
+        "`python scripts/build_catalog.py`. A dataset appears here once it has a "
+        "manifest; files not yet migrated are tracked in "
+        "[PLAN.md](PLAN.md) Phase 9."
+    )
+    lines.append("")
+    lines.append(
+        f"**{len(manifests)} datasets migrated** · {human_size(total)} total · "
+        f"{permitted} permitted / {restricted} restricted redistribution"
+    )
+    lines.append("")
+    lines.append("| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by |")
+    lines.append("| --- | --- | --- | --- | --- | --- | --- | --- | --- |")
+    for m in manifests:
+        fn = m["filename"]
+        title = oneline(m.get("title", ""))
+        dataset = f"[**{fn}**]({RAW}/{fn})"
+        if title:
+            dataset += f"<br><sub>{title}</sub>"
+        license = m.get("license") or {}
+        row = [
+            dataset,
+            m.get("class", "?"),
+            fmt_source(m.get("source")),
+            oneline(license.get("name", "?")),
+            fmt_redist(license),
+            fmt_integrity(m.get("integrity")),
+            fmt_builder(m),
+            human_size(m["_size"]) if m["_size"] is not None else "—",
+            fmt_consumers(m.get("consumers")),
+        ]
+        lines.append("| " + " | ".join(row) + " |")
+    lines.append("")
+    lines.append("---")
+    lines.append("")
+    lines.append(
+        "**Legend** — *Integrity* is the `integrity.upstream.status` (is this "
+        "what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · "
+        "… unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as "
+        "inherited exposures and tracked for licence review "
+        "([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). "
+        "*Builder* ⚠️ unrecovered marks a constructed dataset whose builder was "
+        "never committed (PLAN Phase 9)."
+    )
+    lines.append("")
+    return "\n".join(lines)
+
+
+def main() -> int:
+    manifests = load_manifests()
+    if not manifests:
+        print("no manifests found under lectures/*.yml", file=sys.stderr)
+        return 1
+    OUT.write_text(build(manifests), encoding="utf-8")
+    print(f"wrote {OUT.relative_to(REPO)} — {len(manifests)} datasets")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())