Harden packaging: audit out-of-package runtime data + clean-install smoke test (#82)

* Harden packaging: audit out-of-package runtime data, add clean-install smoke test

Audit (no runtime change): traced every user-reachable CLI entry point
(permea, permea bench, permea bench diagnose, and the separate permea-drylab
script) for reads of data living outside the shipped package trees
(permea_core, permea_explain, permea_ui). Verdict: none. `permea bench
diagnose` reads only user-supplied dataset/clusters; its downstream
run()/diagnose()/warnings registry are pure Python. The many repo-root-relative
path constants (ROOT = parents[3], Path("docs/examples/generated"),
Path("benchmarks/*.yaml"), ...) all live in artifact-generation modules driven
by scripts/, not exposed through the CLI. A clean-venv sweep from outside the
repo confirmed no FileNotFoundError / ModuleNotFoundError / missing-resource on
any command. Because nothing broke, no files were moved and no package-data rule
was added -- moving unused data would ship files nothing needs.

Changes:
- pyproject.toml: add a `dev` optional-dependency group (build, twine, pytest)
  so a canonical sdist+wheel can be produced and CI can run the suite. No upload.
- pyproject.toml: register `slow` and `packaging` pytest markers.
- tests/test_packaging_smoke.py: one clean-install smoke test (marked
  slow+packaging) that builds a real wheel, installs it into a throwaway venv,
  and drives `permea bench diagnose` from outside the repo on files produced by
  permea_ui.fixtures.write_example, asserting exit 0 and a fired warning code.
  The existing 562 tests run against an editable install and structurally cannot
  catch a packaging regression; this closes that gap. It gates on a real
  `python -m build` capability probe (not `import build`) so it skips cleanly
  when the dev extra is absent and runs in CI where `.[dev]` is installed.

Hygiene:
- Remove stray src/permea_core/.DS_Store (untracked; .DS_Store already ignored).
- Delete finished untracked doc docs/pr/PR-permea-explain-and-packaging.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Add CI so the packaging smoke test actually runs

The prior commit's tests/test_packaging_smoke.py claimed "CI installs the dev
extra", but the repo had no .github/workflows. The test skips whenever
`python -m build` is absent, so on the maintainer machine it skipped and ran
nowhere else -- it guarded nothing and that comment was not verifiable. This
adds the CI that makes the claim true.

Audit (Phase A): the suite is fully offline -- no test makes a live LLM provider
call and none requires a secret. The provider tests clear/monkeypatch
PERMEA_LLM_* and use fakes; test_explain_provider_configured is "strictly
offline" by construction; test_explain_boundary asserts openai/anthropic/httpx
are NOT imported. So CI needs no secrets.

Findings while wiring CI (pip install ".[dev]" alone could not even COLLECT the
suite):
- The test suite imports feature-extra deps unconditionally (fastapi in
  test_ui_drylab, jsonschema in several test_explain_*), and PyYAML -- an
  OPTIONAL runtime dep of permea_core guarded behind `try: import yaml` -- is
  declared in no extra at all. With only `.[dev]` these produced 14 collection
  errors, not skips.
- Fix: make `dev` a self-referential superset -- `permea-core[explain,ui]` plus
  pyyaml plus build/twine/pytest -- so `pip install .[dev]` yields an
  environment that can run the ENTIRE suite in one command. Base `dependencies`
  are untouched: PyYAML stays optional for CLI users; the diagnose path never
  needs it.

Changes:
- .github/workflows/ci.yml: CI on pull_request + push to main, ubuntu-latest,
  matrix over Python 3.11/3.12/3.13 (per requires-python >=3.11). Steps:
  checkout, setup-python, `pip install ".[dev]"`, then two pytest steps --
  `-m "not packaging"` (fast) and `-m packaging` (the wheel build/install test,
  kept separate so a packaging regression is distinguishable in the log). No
  secrets; only actions/checkout and actions/setup-python.
- pyproject.toml: expand the `dev` extra as above.
- tests/test_packaging_smoke.py: the "CI installs the dev extra" statement is now
  true (workflow added); docstring now states plainly that the test SKIPS when
  `python -m build` is unavailable, so a green local run does not imply it ran,
  and points at the CI workflow where it actually executes. Test not weakened.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: Permea-lab-admin

.github/workflows/ci.yml

@@ -0,0 +1,41 @@
+name: CI
+
+# Runs the test suite on every PR and on pushes to main. No secrets are used: the suite is
+# fully offline (provider tests clear/monkeypatch PERMEA_LLM_* and never make a network call),
+# so nothing here needs repository secrets.
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # The versions declared supported by `requires-python = ">=3.11"`.
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      # `.[dev]` is self-referential: it pulls the explain+ui feature extras (the suite
+      # imports fastapi/jsonschema unconditionally), PyYAML (optional runtime dep exercised
+      # by the YAML tests), and build/twine/pytest. One command, whole suite runnable.
+      - name: Install
+        run: pip install ".[dev]"
+
+      # Fast lane: everything except the slow packaging build/install test.
+      - name: Fast tests
+        run: pytest -m "not packaging"
+
+      # Packaging lane, kept separate so a packaging regression is obvious in the CI log.
+      # This is the step that makes tests/test_packaging_smoke.py actually RUN (it builds a
+      # real wheel, installs it into a throwaway venv, and drives the console script): on the
+      # maintainer machine it skips for lack of `python -m build`, so CI is where it counts.
+      - name: Packaging tests
+        run: pytest -m packaging

pyproject.toml

@@ -41,6 +41,27 @@ ui = [
     "python-multipart>=0.0.9",
 ]
 
+# The full develop-and-test bundle: `pip install .[dev]` is the single command CI runs to
+# get an environment that can execute the ENTIRE suite. Nothing here is needed to *use* the
+# CLI.
+dev = [
+    # Self-referential: pull the feature extras, because the test suite imports their deps
+    # unconditionally (tests/test_ui_drylab.py imports fastapi; several test_explain_*.py
+    # import jsonschema). Without these, pytest fails at COLLECTION, not as skips.
+    "permea-core[explain,ui]",
+    # Release + test tooling. `build`/`twine` are what a release runs (no upload in-repo);
+    # `pytest` runs the suite, including the packaging smoke test that shells out to
+    # `python -m build`.
+    "build>=1.0",
+    "twine>=5.0",
+    "pytest>=8.0",
+    # PyYAML is an OPTIONAL runtime dep of permea_core: the registry/manifest/card loaders
+    # guard `import yaml` and raise a clear error when it is absent, so it is deliberately
+    # NOT in base `dependencies` (the diagnose CLI never needs it). The suite exercises those
+    # YAML paths, so it must be present to develop/test -- hence here.
+    "pyyaml>=6.0",
+]
+
 [project.scripts]
 permea = "permea_core.cli.main:main"
 permea-drylab = "permea_ui.__main__:main"
@@ -59,3 +80,12 @@ permea_explain = ["schemas/*.schema.json"]
 # The UI serves these from disk at request time; a wheel without them installs a server
 # that returns 404 for its own front end.
 permea_ui = ["static/*"]
+
+[tool.pytest.ini_options]
+# Registered so the packaging smoke test's markers are not "unknown". `packaging` (also
+# tagged `slow`) builds a wheel, spins up a throwaway venv, and drives the installed console
+# script -- deselect it in fast runs with:  pytest -m "not packaging"
+markers = [
+    "slow: test is slow (builds/install/subprocess); safe to skip in fast local runs",
+    "packaging: exercises a real wheel build + clean-venv install (implies slow)",
+]

tests/test_packaging_smoke.py

@@ -0,0 +1,101 @@
+"""Clean-install packaging smoke test.
+
+Every other test in this suite runs against an *editable* install, where the whole repo is
+on ``sys.path``. That layout structurally cannot catch a packaging regression: a file that
+exists in the repo but is never declared to ship (missing ``package-data``, or a data file
+that lives outside the three package trees) is still importable in-place, so an editable
+test sees it and passes -- while a real ``pip install`` user gets a wheel without it.
+
+This test closes that gap by taking the outside-user path exactly: build a real wheel,
+install it into a throwaway venv, then drive the public ``permea bench diagnose`` console
+script from a working directory OUTSIDE the repo, on inputs produced by the *installed*
+``permea_ui.fixtures.write_example``. If any runtime file failed to ship, this fails.
+
+Marked ``packaging`` (and ``slow``) so fast unit runs can deselect it::
+
+    pytest -m "not packaging"
+
+Needs no network beyond the PyPI dependency resolve that ``build`` (isolated backend) and
+``pip install`` perform -- i.e. it is CI-safe wherever PyPI is reachable.
+
+IMPORTANT -- this test SKIPS when ``python -m build`` is unavailable (no pypa-build on the
+interpreter). So a green *local* run does NOT mean this test ran: on a maintainer machine
+without the ``dev`` extra it is skipped, not executed. Where it actually runs is CI --
+``.github/workflows/ci.yml`` installs ``.[dev]`` (which provides ``build``) and invokes it
+via the dedicated ``pytest -m packaging`` step. If you want to run it locally, install the
+dev extra first: ``pip install ".[dev]"``.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+import venv
+from pathlib import Path
+
+import pytest
+
+pytestmark = [pytest.mark.slow, pytest.mark.packaging]
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+
+
+def _run(cmd: list[str], **kwargs: object) -> subprocess.CompletedProcess[str]:
+    """Run a subprocess, capturing text output; raise with full output on failure."""
+    proc = subprocess.run(cmd, capture_output=True, text=True, **kwargs)  # type: ignore[call-overload]
+    if proc.returncode != 0:
+        raise AssertionError(
+            f"command failed ({proc.returncode}): {' '.join(cmd)}\n"
+            f"--- stdout ---\n{proc.stdout}\n--- stderr ---\n{proc.stderr}"
+        )
+    return proc
+
+
+def test_wheel_installs_and_diagnose_runs_from_clean_venv(tmp_path: Path) -> None:
+    # `python -m build` is the canonical wheel builder; it ships in the `dev` extra. Gate on
+    # the ACTUAL capability, not an `import build`: a bare env may have an unrelated module
+    # also named `build` on sys.path, which imports fine yet has no runnable CLI. Probe the
+    # real invocation and skip (don't fail) when unavailable -- CI's `.[dev]` install provides
+    # it (.github/workflows/ci.yml), so the skip only ever happens off-CI.
+    probe = subprocess.run(
+        [sys.executable, "-m", "build", "--version"], capture_output=True, text=True
+    )
+    if probe.returncode != 0:
+        pytest.skip("`python -m build` is unavailable (install the `dev` extra) -- cannot build a wheel")
+
+    # 1. Build a wheel from the repo into a temp dir (isolated backend pulls setuptools).
+    dist = tmp_path / "dist"
+    _run([sys.executable, "-m", "build", "--wheel", "--outdir", str(dist), str(REPO_ROOT)])
+    wheels = list(dist.glob("*.whl"))
+    assert len(wheels) == 1, f"expected exactly one wheel, got {wheels}"
+
+    # 2. Fresh venv; install the wheel (this resolves numpy/scikit-learn from PyPI).
+    venv_dir = tmp_path / "venv"
+    venv.create(venv_dir, with_pip=True)
+    win = sys.platform == "win32"
+    bin_dir = venv_dir / ("Scripts" if win else "bin")
+    py = bin_dir / ("python.exe" if win else "python")
+    _run([str(py), "-m", "pip", "install", "--quiet", str(wheels[0])])
+
+    # 3. Materialize the synthetic demo fixture using the INSTALLED package, not the repo.
+    fixture_dir = tmp_path / "fixture"
+    fixture_dir.mkdir()
+    _run([
+        str(py), "-c",
+        "from pathlib import Path; from permea_ui.fixtures import write_example; "
+        f"write_example(Path({str(fixture_dir)!r}))",
+    ])
+    dataset = fixture_dir / "example_dataset.csv"
+    clusters = fixture_dir / "clusters_family.tsv"
+    assert dataset.exists() and clusters.exists(), "write_example did not produce the expected files"
+
+    # 4. Drive the console script from OUTSIDE the repo; assert clean exit + a fired code.
+    permea = bin_dir / ("permea.exe" if win else "permea")
+    proc = subprocess.run(
+        [str(permea), "bench", "diagnose", str(dataset), str(clusters)],
+        cwd=str(tmp_path),  # deliberately not the repo root
+        capture_output=True,
+        text=True,
+    )
+    assert proc.returncode == 0, f"diagnose exited {proc.returncode}\n{proc.stderr}"
+    assert "PERMEA-W" in proc.stdout, f"expected a warning code in output, got:\n{proc.stdout}"