Builder architecture: four-stage template, traceback-clean failures, notebook policy (Phase 5)

[mmcky]

Design notes from a review discussion on PR #12, recorded here to be actioned in PLAN Phase 5 (automation). Nothing below is implemented yet; this is the supporting research for the builder architecture and a copy-able template.

Why builders are plain .py scripts, not notebooks

The question that prompted this: we could run a notebook to generate a dataset — but extracting tracebacks and failure states from a notebook run is difficult. That difficulty is structural, not incidental, and it settles the architecture:

1. Errors must propagate as exceptions with a full traceback and a non-zero exit. A plain script gives CI exactly that for free. Papermill/nbconvert can execute a notebook, but a failure surfaces as a truncated cell-execution error embedded in JSON output — the natural stack trace is lost, and the exit-code contract is murkier.
2. CI needs to distinguish two failure classes. A ValidationError (the data is wrong — needs a human) is not an infrastructure failure (network/credentials — retry). A script makes this a clean except boundary; a notebook blurs it.
3. The manifest's schema block is the validation spec. The builder's validate() stage reads the dataset's sidecar <filename>.yml and enforces its columns, row_count_floor, known_nulls and date_range. The builder and the PR-CI check then enforce the same contract from one source of truth — no drift between what the manifest claims and what the builder guarantees.
4. Atomic write + last-good guarantee. Validate the in-memory frame before writing; write to a temp file in the same directory, then os.replace() into place. A failed refresh or a crash mid-write leaves the previously-committed snapshot untouched — which is the AGENTS.md promise that an upstream outage may fail a refresh but must never break a lecture build.

Notebook-origin datasets

Common for QuantEcon: a dataset's construction starts life in a lecture notebook. The rule:

• If it needs to refresh (dynamic snapshot): port the notebook logic to a plain .py builder — or pair the notebook with jupytext, whose .py:percent twin is the runnable, diffable, traceback-clean artifact that becomes the committed builder. The notebook stays in the lecture repo as pedagogy.
• If it was a one-off construction that will not refresh: record the notebook as provenance in the manifest's builder field, with a builder_status that signals it is not a CI-runnable builder (open sub-question: a distinct status value such as notebook, vs reusing not-applicable with a note — decide when the first such dataset lands).
• Papermill-executing a notebook as the builder is the fallback only when the notebook itself must be the deliverable, accepting the worse failure ergonomics.

Template

The four-stage contract (AGENTS.md: fetch → pre-process → validate → write), with preprocess kept pure (no I/O) so it is unit-testable in isolation:

"""Builder for <output>.csv — <one-line description>.

Contract (AGENTS.md / PLAN Phase 5): fetch -> pre-process -> validate -> write.
Writes ONLY on validation pass, atomically, so a failed refresh leaves the
last-good snapshot untouched and never breaks a lecture build.

Run:  python scripts/<name>.py
Exit: 0 on success; non-zero WITH a traceback on any failure (CI-readable).
"""
from __future__ import annotations
import logging, os, tempfile
import pandas as pd, yaml
# import wbgapi as wb   # or the relevant upstream client

HERE = os.path.dirname(os.path.abspath(__file__))
LECTURES = os.path.join(os.path.dirname(HERE), "lectures")
OUTPUT = "example.csv"              # the published filename
MANIFEST = OUTPUT + ".yml"          # its sidecar — the validation spec

log = logging.getLogger(OUTPUT)


class ValidationError(Exception):
    """Data failed an invariant — distinct from an infrastructure failure, so CI
    can tell 'the data is wrong' (human) from 'the network was down' (retry)."""


def fetch():
    """Pull raw data from upstream. Network/credential failures surface here."""
    raise NotImplementedError


def preprocess(raw) -> pd.DataFrame:
    """Pure raw -> published frame. No I/O, so it's unit-testable in isolation."""
    raise NotImplementedError


def validate(df: pd.DataFrame, spec: dict, previous: pd.DataFrame | None) -> None:
    """Enforce the manifest's `schema` block — the single source of truth."""
    schema = spec["schema"]
    expected = [c["name"] for c in schema["columns"]]
    if list(df.columns) != expected:
        raise ValidationError(f"columns {list(df.columns)} != manifest {expected}")
    if len(df) < schema["row_count_floor"]:
        raise ValidationError(f"{len(df)} rows < floor {schema['row_count_floor']}")

    known = schema.get("known_nulls") or {}
    for col in df.columns:
        n = int(df[col].isna().sum())
        if n != known.get(col, 0):
            raise ValidationError(f"{col}: {n} nulls, manifest allows {known.get(col, 0)}")

    # Overlap window: history must not change vs the last-good vintage.
    if previous is not None:
        common = previous.index.intersection(df.index)
        a, b = df.loc[common], previous.loc[common]
        # NOTE: float columns need a tolerance — use np.isclose, not `!=`.
        changed = ((a != b) & a.notna() & b.notna()).to_numpy().any()
        if changed:
            raise ValidationError("values changed in the overlap window vs last-good")


def write(df: pd.DataFrame, path: str) -> None:
    """Atomic write: temp file in the same dir, then os.replace() (atomic on POSIX).
    A crash mid-write can never leave a partial file where a lecture would read it."""
    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), suffix=".tmp")
    os.close(fd)
    try:
        df.to_csv(tmp)
        os.replace(tmp, path)
    finally:
        if os.path.exists(tmp):
            os.remove(tmp)


def run() -> None:
    out = os.path.join(LECTURES, OUTPUT)
    spec = yaml.safe_load(open(os.path.join(LECTURES, MANIFEST)))
    previous = pd.read_csv(out, index_col=0) if os.path.exists(out) else None

    log.info("fetch");        raw = fetch()
    log.info("pre-process");  df = preprocess(raw)
    log.info("validate");     validate(df, spec, previous)   # raises -> no write
    log.info("write");        write(df, out)
    log.info("ok: %d rows", len(df))


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(name)s %(levelname)s %(message)s")
    run()   # uncaught exceptions -> full traceback + non-zero exit (CI-friendly)

Deliberately deferred

• A shared harness. atomic_write, ValidationError, and manifest-driven validate() are obviously shared machinery, but the repo has exactly one dynamic builder today (business_cycle.py). Keep builders self-contained; extract scripts/_builder.py when the second one lands.
• Manifest write-back — stamping retrieved / integrity on a successful refresh belongs to the refresh-as-PR wiring, i.e. Phase 5 proper, not the template.

When Phase 5 is actioned

1. Commit the template as scripts/_template_builder.py and add a "Builder architecture" section to scripts/README.md.
2. Retrofit business_cycle.py to the four-stage contract — it currently writes three files with no validate stage (AGENTS.md already flags this).
3. Decide the notebook-origin builder_status sub-question above when the first such dataset lands.

Part of #8
See QuantEcon/meta#338

[mmcky]

Schema implications of this architecture

Writing the template surfaced a consequence worth stating outright: the moment a builder's validate() reads the sidecar manifest and enforces it, the manifest's schema block stops being documentation and becomes an executable contract. Several fields are not yet precise enough to execute. None of this needs action now — the schema is a strawman and says so — but these are exactly what the next pilot and Phase 5 should test the schema against, so they are recorded here rather than lost.

1. Column patterns — a real gap our own flagship file already hits. The template checks list(df.columns) != expected, but the schema example for business_cycle_data.csv lists YR1960 with the comment "one column per year" — a single exemplar standing in for ~64 columns. A machine validator cannot read that comment; it would fail the file. The schema needs a way to express repeated columns before validation can be real, e.g.:

columns:
  - {name: economy, dtype: string, description: ISO3 country code}
  - {name: Country, dtype: string, description: Country name}
  - {pattern: 'YR\d{4}', dtype: float64, description: Annual GDP growth (%), one column per year}

2. known_nulls semantics need pinning: exact count or ceiling? The template enforces exact (n != known), the stronger invariant and right for frozen files like lingcod (F_over_Fmsy: 1, always). But for a dynamic snapshot, null counts can legitimately drift between refreshes — a new terminal assessment year appears and the null moves or multiplies. Likely answer: exact for static extracts, ceiling for dynamic snapshots — or a per-column form like {F_over_Fmsy: {max: 3}}. Needs an explicit decision, not a silent default.

3. dtype vocabulary is currently inconsistent. The lingcod manifest says int64 / float64 (pandas names); the schema sketch says string / float (loose names). A human reader doesn't care; validate() must compare against something canonical. Recommendation: standardise on pandas dtype strings, since that is what the checking code actually sees.

4. Float tolerance for the overlap window. "Values unchanged vs last-good" cannot be != on floats (the template already flags this — np.isclose needed). If tolerance ever needs to vary by dataset it becomes a schema field (overlap_tolerance); until then a builder-side default suffices. Awareness item, not a field yet.

5. builder_status for notebook-origin datasets — already an open sub-question in the issue body; noted here only so the schema-facing list is complete in one place.

Items 1–3 are decisions the schema has to make before Phase 5's validation can run; 4–5 are cheap to defer. The P2 pilot is the natural place to test 1–3 against a second real file.