refactor(package): stable lakeflow_framework import namespace with bundled config/schemas (#87)

* feat(package): restructure src/ into pip-installable lakeflow_framework package

- Introduce src/lakeflow_framework/ as a proper Python package with pyproject.toml
  (hatchling build); config/ and schemas/ bundled as package data
- Implement Strategy B (Workspace Files-first) resolver in config_resolver:
  load_framework_default_json resolves via Workspace Files → importlib.resources →
  local/config/ overlay; load_framework_schema returns an importlib.resources
  traversable for bundled JSON schemas
- Reduce src/*.py shims to thin re-exports from lakeflow_framework for backward compat
- Add contrib/ extension point with README and __init__ stub
- Add tests: test_package.py (import surface), test_strategy_b_resolver.py (resolver + schema)
- Update all internal imports across dataflow/, dataflow_spec_builder/, and support modules
- Add docs: ADR-0008 (package layout), ADR-0009 (Workspace Files-first resolver),
  deploy_wheel.rst, deploy_framework_overview.rst, contributor_contrib.rst;
  update all existing docs/ pages to reference lakeflow_framework imports
- Align operational metadata, templates, logging, config, validator, and test 
  docs with the packaged layout; extend the spelling wordlist; rephrase
  historized to history-tracked in samples README.

Author: rederik76

README.md

@@ -23,33 +23,41 @@ The framework supports centralized and domain-oriented operating models, and acc
 - Support for batch and streaming pipelines across Bronze/Silver/Gold, aligned to your chosen modelling pattern
 - Flexible for centralized and domain-oriented operating models
 
+## Prerequisites
+
+- Access to a Databricks workspace
+- Databricks CLI installed and authenticated (`databricks auth login` for your workspace, or a configured CLI profile)
+- Familiarity with Databricks Lakeflow Spark Declarative Pipelines concepts
+
 ## Quick start
 
+Deploy the framework:
+
 ```bash
 git clone https://github.com/databricks-solutions/lakeflow_framework.git
 cd lakeflow_framework
-pip install -r requirements-dev.txt
+databricks bundle deploy -t dev
 ```
 
-Then:
-
-1. Open the hosted docs: https://databricks-solutions.github.io/lakeflow_framework/
-2. Deploy the framework using the `Deploy Framework` guide
-3. Deploy samples from `samples/` using the documentation walkthroughs
-4. Build your first pipeline bundle using the `Build a Pipeline Bundle` guide
+Deploy samples (requires the framework above): see [samples/README.md](samples/README.md) for bundle descriptions and deploy scripts (`deploy.sh`, `deploy_feature_samples.sh`, `deploy_tpch.sh`, and others).
 
-## Prerequisites
+```bash
+cd samples
+./deploy.sh
+```
 
-- Access to a Databricks workspace
-- Databricks CLI installed and configured
-- Python environment with project dependencies installed
-- Familiarity with Databricks Lakeflow Spark Declarative Pipelines concepts
+Full deployment steps, configuration options, and walkthroughs are in the public docs [Getting Started](https://databricks-solutions.github.io/lakeflow_framework/current/getting_started.html) section.
 
 ## Repository structure
 
-- `docs/` - Sphinx documentation and versioned docs build tooling
-- `samples/` - example framework and pipeline bundles
-- `src/` - framework source code and runtime components
+- `docs/` — Sphinx documentation and versioned docs build tooling
+- `samples/` — example framework and pipeline bundles
+- `src/` — framework bundle root deployed to the workspace (`framework.sourcePath` in DAB)
+  - `lakeflow_framework/` — canonical Python package: runtime code, bundled default config (`config/default/`), and JSON schemas (`schemas/`). Prefer `from lakeflow_framework...` imports in new code.
+  - `*.py` at `src/` root — backward-compatibility shims for legacy bare imports (e.g. `from constants import ...`); removed at v1.0.0
+  - `local/` — customer-owned sparse config and extensions; never overwritten by upstream upgrades ([src/local/README.md](src/local/README.md))
+
+See [Import conventions](https://databricks-solutions.github.io/lakeflow_framework/contributor_imports.html)
 
 ## Version compatibility
 
@@ -69,10 +77,33 @@ The framework is actively maintained. Databricks support does not cover this rep
 Please refer to the [documentation](https://databricks-solutions.github.io/lakeflow_framework/) for further details and an explanation of the samples.
 The documentation needs to be deployed as HTML or Markdown within your org before it can be used.
 
+### Local development quick start (contributors)
+
+Clone the repository, then set up a Python 3.12 environment. This is a minimal local quick start — full contributor guidance (VS Code setup, lockfiles, deployment, and PR workflow) is in the documentation under **Framework Development & Contributors**.
+
+```bash
+git clone https://github.com/databricks-solutions/lakeflow_framework.git
+cd lakeflow_framework
+python -m venv .venv
+source .venv/bin/activate   # Windows: .venv\Scripts\activate
+pip install --require-hashes --no-deps -r requirements-dev.lock
+```
+
+For editable installs and IDE auto-complete: `pip install -e ".[contrib]"` (see [Development Environment Setup](https://databricks-solutions.github.io/lakeflow_framework/contributor_dev_env.html)).
+
+Run unit tests:
+
+```bash
+pytest tests/ -m "not integration and not spark"
+```
+
+See also `tests/README.md` and `docs/source/contributor_imports.rst` in the repository.
+
 ### Local docs development (optional)
 
+Requires dev dependencies from `requirements-dev.lock`:
+
 ```bash
-pip install -r requirements-docs.txt
 make -C docs html
 ```
 

VERSION

@@ -1 +1 @@
-v0.19.0
+v0.20.0

docs/decisions/0008-lakeflow-framework-package.md

@@ -0,0 +1,145 @@
+# ADR-0008: `lakeflow_framework` pip-installable package
+
+**Date:** 2026-05-23
+**Status:** Accepted
+**PR:** refactor/lakeflow-framework-package (v0.20.0)
+
+---
+
+## Context
+
+The framework shipped as a flat collection of `.py` files directly under `src/`.
+Consumers cloned the repository and added `src/` to `sys.path` via
+`framework.sourcePath` — effectively a "copy the source" distribution model.
+
+This had three structural problems:
+
+1. **No stable import namespace.** All modules were importable as bare names
+   (`from constants import ...`, `from bundle_loader import ...`). Any module
+   name in a customer's pipeline bundle could shadow a framework module.
+
+2. **No pip distribution path.** Teams that manage Python dependencies via PyPI,
+   a UC Volume, or Artifactory had no clean way to consume the framework. Every
+   consumer was forced to clone the repository and re-deploy it as a DAB bundle.
+
+3. **Data files (config, schemas) lived outside any package.** `src/config/default/`
+   and `src/schemas/` were loaded via hard-coded `os.path` joins relative to
+   `framework.sourcePath`. There was no mechanism to bundle these files with a
+   wheel and access them portably.
+
+The project's `pyproject.toml` did not exist; `setup.py` or any standard packaging
+entry point was absent.
+
+## Decision
+
+Introduce a proper Python package under `src/lakeflow_framework/` and a
+`pyproject.toml` at the repository root.
+
+### Package layout
+
+```
+src/
+  lakeflow_framework/
+    __init__.py                        ← public API + __version__
+    constants.py                       ← FrameworkPaths, enums
+    config_resolver.py
+    bundle_loader.py
+    logger.py
+    dlt_pipeline_builder.py
+    ...
+    config/
+      default/                         ← bundled default config (package data)
+    schemas/                           ← bundled JSON schemas (package data)
+    dataflow/
+      flows/
+    dataflow_spec_builder/
+    contrib/
+      __init__.py                      ← scaffold; empty until first module lands
+      README.rst
+```
+
+All imports inside the package use **absolute `lakeflow_framework.*` names**.
+No bare imports remain inside `src/lakeflow_framework/`.
+
+### Compat shims
+
+The old flat `src/*.py` locations (e.g. `src/constants.py`,
+`src/bundle_loader.py`) are replaced by thin re-export shims:
+
+```python
+# src/constants.py  — compat shim, remove at v1.0.0
+from lakeflow_framework.constants import *  # noqa: F401,F403
+```
+
+Existing pipeline notebooks and bundles that import bare names continue to work
+without change until `v1.0.0`.
+
+### `pyproject.toml`
+
+```toml
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.backends.legacy:build"
+
+[project]
+name = "lakeflow-framework"
+dynamic = ["version"]
+...
+
+[tool.setuptools.dynamic]
+version = { file = "VERSION" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+lakeflow_framework = [
+    "config/default/**/*",
+    "schemas/**/*",
+]
+```
+
+`VERSION` is the single source of truth for the release version; both
+`importlib.metadata` (wheel installs) and a direct file read (editable / flat
+deploy) converge on it.
+
+### `contrib` subpackage
+
+`src/lakeflow_framework/contrib/` is introduced as a scaffold with an empty
+`__init__.py` and a support-policy `README.rst`. No modules land in this PR.
+See `docs/decisions/0008-*` (this file) for the publish model and
+`docs/source/contributor_contrib.rst` for the contributor guide.
+
+### Optional extras
+
+```
+pip install lakeflow-framework          # core only
+pip install "lakeflow-framework[contrib]"   # + future contrib extras
+pip install "lakeflow-framework[all]"       # everything
+```
+
+The `contrib` extra is currently empty; installing it is a no-op until a contrib
+module is added.
+
+### Deprecation timeline
+
+| Version | Action |
+|---------|--------|
+| v0.20.0 | `src/lakeflow_framework/` package introduced; bare `src/` imports still work via shims |
+| v1.0.0  | Compat shims at old flat `src/*.py` paths removed |
+
+## Consequences
+
+- Teams can now `pip install lakeflow-framework` from PyPI, a UC Volume, or
+  Artifactory. The wheel bundles default config and schemas; no separate file
+  deployment is required.
+- The `lakeflow_framework.*` import namespace is globally unique; no shadow-import
+  risk with customer code.
+- Existing flat-deploy customers are unaffected: the `src/` shims preserve
+  backward compatibility, and `framework.sourcePath` + Workspace Files-first
+  resolution (ADR-0009) means behaviour is identical to pre-v0.20.0.
+- Editable installs (`pip install -e ".[contrib]"`) are supported for local
+  development; the `src/` layout ensures the installed package and the source
+  tree are the same directory.
+- `docs/conf.py` reads `release` from the `VERSION` file at build time — the
+  version shown in HTML headers is always current without manual edits.

docs/decisions/0009-strategy-b-workspace-files-first-resolver.md

@@ -0,0 +1,120 @@
+# ADR-0009: Strategy B — Workspace Files-first config and schema resolution
+
+**Date:** 2026-05-23
+**Status:** Accepted
+**PR:** refactor/lakeflow-framework-package (v0.20.0)
+
+---
+
+## Context
+
+With the introduction of the `lakeflow_framework` Python package (ADR-0008),
+default config files (`config/default/`) and JSON schemas (`schemas/`) are
+bundled inside the wheel via `importlib.resources`. Previously they lived only
+in Workspace Files at `{framework_path}/lakeflow_framework/config/default/`.
+
+Two deployments now co-exist:
+
+| Deploy mode | Where defaults live |
+|-------------|---------------------|
+| Flat DAB deploy | On workspace files under `framework.sourcePath` |
+| Wheel install | Bundled inside the wheel (`importlib.resources`) |
+| Wheel + local overlay | Wheel bundled, plus `src/local/config/` fragments |
+
+The resolver (`load_framework_default_json`) must decide which source to read
+first when both a Workspace Files path and the wheel package data are potentially available.
+
+### Options considered
+
+**Option A — Package-first (wheel-first)**
+- Always read from `importlib.resources`; treat Workspace Files as an optional
+  override layer.
+- `+` Simple: the wheel is always the canonical source of defaults.
+- `−` **Breaking for existing flat-deploy customers.** Their defaults live in
+  Workspace Files; if the wheel shadow-contains a newer (or different) version of a default
+  file, behaviour silently changes on upgrade.
+- `−` Contradicts the "explicit wins" principle: a customer who set
+  `framework.sourcePath` explicitly expects Workspace Files to be authoritative.
+
+**Option B — Workspace Files-first (chosen)**
+- When `framework.sourcePath` is set and the file exists in Workspace Files, use it.
+- Fall back to `importlib.resources` only when no Workspace Files path is configured or
+  the file is absent from Workspace Files.
+- Always deep-merge `src/local/config/` overlay on top (unchanged from
+  ADR-0006).
+
+**Option C — Explicit-only**
+- Require callers to pass either a Workspace Files path or `importlib.resources` traversable;
+  no automatic fallback.
+- `−` Forces every call site to be aware of the deploy mode; adds boilerplate
+  at all 15+ call sites.
+
+## Decision
+
+Adopt **Strategy B (Workspace Files-first)** via `load_framework_default_json`:
+
+```python
+def load_framework_default_json(
+    name: str,
+    framework_path: Optional[str] = None,
+) -> Dict:
+    """
+    Resolution order:
+    1. Workspace Files  — {framework_path}/lakeflow_framework/config/default/{name}
+               if framework_path is set and the file exists.
+    2. Package data — importlib.resources (wheel or src/lakeflow_framework/ on sys.path).
+    3. src/local/config/{name} overlay — deep-merged on top of (1) or (2)
+               when framework_path is set and the fragment exists.
+    """
+```
+
+### Resolution sequence in detail
+
+1. **Workspace Files (explicit):** if `framework_path` is provided and
+   `{framework_path}/lakeflow_framework/config/default/{name}` exists, load it.
+2. **Package data (fallback):** otherwise use
+   `importlib.resources.files("lakeflow_framework") / "config" / "default" / name`.
+   This works for both an installed wheel and an editable / flat-deploy install
+   (the `src/lakeflow_framework/` directory is on `sys.path` in both cases).
+3. **Local overlay:** if `framework_path` is set and
+   `{framework_path}/src/local/config/{name}` exists, deep-merge it on top of
+   whichever base was loaded (dict keys merged recursively; overlay wins on
+   conflict; non-dict values replaced wholesale). This preserves the ADR-0006
+   sparse-overlay guarantee regardless of deploy mode.
+
+The same logic applies to schema resolution via
+`load_framework_schema(name)` in `config_resolver`, which returns an
+`importlib.resources` traversable suitable for `jsonschema.RefResolver` and
+similar validators.
+
+### Rationale for Workspace Files-first
+
+- **Zero-change upgrade for flat-deploy customers.** Their `framework.sourcePath`
+  points to the deployed `src/` directory in Workspace Files; those files are
+  found at step 1 and the wheel is never consulted. Behaviour is identical to
+  pre-v0.20.0.
+- **Explicit wins.** Setting `framework.sourcePath` is a deliberate act.
+  Workspace Files-first honours that intent; package-first would silently override it.
+- **Wheel-install customers get clean defaults from the package.** They do not
+  set `framework.sourcePath` (or set it only for local overlays), so step 1
+  is skipped and `importlib.resources` is used — exactly as expected.
+- **Testability.** Workspace Files-first is trivially testable: point
+  `framework_path` at a temp directory and place a fixture file there.
+  Package-data fallback is also testable by omitting `framework_path`.
+
+## Consequences
+
+- **`load_framework_default_json`** is the single resolver for all default
+  config and schema reads; call sites that previously used
+  `os.path.join(framework_path, FrameworkPaths.CONFIG_PATH, name)` are
+  migrated to it.
+- Flat-deploy customers see **no behaviour change** on upgrade to v0.20.0.
+- Wheel-install customers get defaults from the wheel with no `framework.sourcePath`
+  required.
+- The `src/local/config/` sparse overlay (ADR-0006) continues to work in all
+  three deploy modes — it is applied at step 3 independent of which base was
+  used in steps 1–2.
+- If a future release ships an intentionally different default (e.g. a renamed
+  key), flat-deploy customers picking up a new wheel without re-deploying their
+  bundle will continue to use their Workspace Files defaults until they re-deploy.
+  This is safe and predictable: Workspace Files-first ensures the Workspace Files always win.

docs/source/concepts.rst

@@ -70,11 +70,12 @@ The Framework Bundle contains:
    * - Component
      - Description
    * - **Framework Source Code**
-     - The core framework source code under the ``src`` folder.
+     - The core framework source code under the ``src/lakeflow_framework/`` package.
    * - **Global Framework Configuration**
-     - The global framework configuration under the ``src/config`` folder.
-   * - **Data Flow Spec Schema Definition**
-     - The Data Flow Spec schema definition and validations under the ``src/schemas`` folder.
+     - The framework default configuration under ``src/lakeflow_framework/config/default/``. Customer
+       overrides go in ``src/local/config/`` as sparse files — only the keys to change are needed.
+   * - **Data Flow Spec Schema Definitions**
+     - The Data Flow Spec schema definitions and validations under ``src/lakeflow_framework/schemas/``.
    * - **Deployment YAML file**
      - The ``databricks.yml`` file which defines the bundle deployment settings.
 
@@ -86,7 +87,7 @@ The Framework Bundle is deployed to a given workspace files location from where
 Framework Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-The Framework and most of its features will have configuration settings that can be set in the Framework Bundle ``src/config`` folder. The configuration settings are explained in the section: :doc:`features`
+The Framework and most of its features will have configuration settings that can be set in the Framework Bundle ``src/lakeflow_framework/config/default/`` folder. Default values ship with the framework and are resolved automatically; individual keys can be overridden by placing sparse files in ``src/local/config/``. The configuration settings are explained in the section: :doc:`features`
 
 .. admonition:: Setting Precedence
   :class: note

docs/source/contributor.rst

@@ -6,6 +6,8 @@ Framework Development & Contributors
    :caption: Contents:
 
    contributor_dev_env
+   contributor_imports
    contributor_dev_git
    contributor_dev_steps
    contributor_dev_docs
+   contributor_contrib