Update docs and READMEs for lakeflow_framework package paths.

Align operational metadata, templates, logging, config, validator, and test
docs with the packaged layout; extend the spelling wordlist; rephrase
historized to history-tracked in samples README.

Author: rederik76

docs/source/contributor_dev_steps.rst

@@ -43,7 +43,7 @@ Development Process
         pytest tests/ -m "not integration and not spark"
 
    - See ``tests/README.md`` for layout, fixtures, markers, and conventions.
-   - Optional coverage: add ``--cov=src --cov-report=term-missing``.
+   - Optional coverage: add ``--cov=lakeflow_framework --cov-report=term-missing``.
    - CI runs the same pytest command on every pull request (``.github/workflows/ci.yml``).
 
 3. Integration Testing / Samples

docs/source/feature_framework_configuration.rst

@@ -126,7 +126,7 @@ Example — change one global setting without touching the rest of ``global.json
    }
 
 Save this as ``src/local/config/global.json``. All other keys from
-``src/config/default/global.json`` are kept unchanged.
+``src/lakeflow_framework/config/default/global.json`` are kept unchanged.
 
 For **directory-based config** (e.g. ``dataflow_spec_mapping/``), place the
 entire override directory in ``src/local/config/`` — the local directory takes

docs/source/feature_logging.rst

@@ -18,7 +18,7 @@ You can optionally plug in a **custom logger** via a dedicated ``logger.json`` c
 .. admonition:: Setting Precedence
   :class: warning
 
-  * Framework code and extensions should use the shared logger helper ``pipeline_config.get_logger()`` rather than creating their own loggers.
+  * Framework code and extensions should use the shared logger helper ``get_logger()`` from ``lakeflow_framework.pipeline_config`` rather than creating their own loggers.
 
 Overview
 --------
@@ -460,14 +460,14 @@ Using the Logger in Code
 
 .. code-block:: python
 
-   from pipeline_config import get_logger
+   from lakeflow_framework.pipeline_config import get_logger
 
    logger = get_logger()
    logger.info("Processing batch %s", batch_id)
 
-``utility.set_logger`` remains for backward compatibility and delegates to the framework default logger factory; new code should use ``pipeline_config.get_logger()``.
+``lakeflow_framework.utility.set_logger`` remains for backward compatibility and delegates to the framework default logger factory; new code should use ``get_logger()`` from ``lakeflow_framework.pipeline_config``.
 
-Custom loggers that support async flushing may require an explicit ``close()`` at the end of the pipeline. Consider a **post_init** hook under ``extensions/post_init/`` that calls ``close()`` on the primary logger when exposed.
+Custom loggers that support async flushing may require an explicit ``close()`` at the end of the pipeline. Consider a **post-init** hook under ``src/init/post/`` (pipeline bundle) or ``src/local/init/post/`` (framework bundle) that calls ``close()`` on the primary logger when exposed.
 
 Troubleshooting
 ^^^^^^^^^^^^^^^^

docs/source/feature_operational_metadata.rst

@@ -29,7 +29,7 @@ Configuration
 -------------
 
 | **Scope: Global**
-| In the Framework bundle, operational metadata columns are defined in JSON configuration files at Lakehouse layer level (e.g. bronze, silver, gold). The configuration files are locate at and must be named as follows: ``src/config/default/operational_metadata_<layer>.json`` 
+| In the Framework bundle, operational metadata columns are defined in JSON configuration files at Lakehouse layer level (e.g. bronze, silver, gold). The configuration files are locate at and must be named as follows: ``src/lakeflow_framework/config/default/operational_metadata_<layer>.json``
 
 .. admonition:: Layer Config
    :class: note

docs/source/feature_templates.rst

@@ -421,7 +421,7 @@ Validation
 
 Each expanded spec is validated using the existing schema validators to ensure correctness.
 
-Template usage specs are validated against the schema at ``src/schemas/spec_template.json``:
+Template usage specs are validated against the schema at ``src/lakeflow_framework/schemas/spec_template.json``:
 
 - ``template``: Required string (template name without .json extension)
 - ``params``: Required array with at least one parameter object

docs/source/spelling_wordlist.txt

@@ -4,6 +4,7 @@ Artifactory
 async
 autocomplete
 backfill
+backend
 behaviour
 cardinality
 cdc
@@ -14,8 +15,10 @@ cli
 Cmd
 config
 configs
+contrib
 css
 Ctrl
+customisation
 customisations
 dataFlowId
 Dataframe
@@ -27,6 +30,7 @@ deduplicate
 dedupes
 deduplication
 deployable
+deps
 Del
 Demultiplexing
 dependant
@@ -47,6 +51,7 @@ initialisation
 Inmon
 IntelliSense
 instantiation
+integrations
 json
 keystore
 kwargs
@@ -82,6 +87,7 @@ rst
 Schemas
 schemas
 sdist
+semver
 Serverless
 sinkOptions
 sourceDetails
@@ -95,6 +101,7 @@ stdout
 Subquery
 subdirectory
 subqueries
+subpackage
 tableMigrationDetails
 tableProperties
 targetFormat

samples/README.md

@@ -7,7 +7,7 @@ The Framework comes with extensive samples that demonstrate the use of the frame
 | **`feature-samples`** | Demonstrates every framework feature in isolation using a single `{namespace}_feature` schema. The simplest entry point. |
 | **`pattern-samples`** | End-to-end medallion architecture patterns (bronze → silver → gold) across multiple schemas. Includes multi-source streaming, stream-static joins, CDC from snapshot sources, and gold-layer materialized views. |
 | **`yaml_sample`** | Demonstrates that data flow specs can be written in YAML format instead of JSON. Contains YAML equivalents of basic specs. |
-| **`tpch_sample`** | The most comprehensive end-to-end reference, built on the TPC-H dataset in the UC samples catalog. Covers multi-source schema-on-read bronze, conformed/historized silver (SCD2 + SCD1 + append-only facts with DQ quarantine), and a governed gold star schema with surrogate keys, point-in-time joins, pre-aggregated MVs, and UC metric views. Uses template specs and a three-run incremental simulation. See its [README](tpch_sample/README.md). |
+| **`tpch_sample`** | The most comprehensive end-to-end reference, built on the TPC-H dataset in the UC samples catalog. Covers multi-source schema-on-read bronze, conformed/history-tracked silver (SCD2 + SCD1 + append-only facts with DQ quarantine), and a governed gold star schema with surrogate keys, point-in-time joins, pre-aggregated MVs, and UC metric views. Uses template specs and a three-run incremental simulation. See its [README](tpch_sample/README.md). |
 
 ## Deploying the Samples
 
@@ -146,12 +146,12 @@ Parameters:
 
 ## TPC-H Sample
 
-The TPC-H sample is the most comprehensive end-to-end example in the framework. Built on the TPC-H schema in the Databricks UC samples catalog, it turns a realistic, sample dataset into a fully streaming medallion data warehouse — from multi-source raw ingestion through conformed, historized silver to a governed gold star schema with value-add aggregations.
+The TPC-H sample is the most comprehensive end-to-end example in the framework. Built on the TPC-H schema in the Databricks UC samples catalog, it turns a realistic, sample dataset into a fully streaming medallion data warehouse — from multi-source raw ingestion through conformed, history-tracked silver to a governed gold star schema with value-add aggregations.
 
 It demonstrates a wide range of framework capabilities together, including:
 
 * **Schema-on-read bronze** that infers and evolves the schema, ingesting Parquet from eight simulated source systems via Auto Loader.
-* **Conformed, historized silver** with SCD2 dimensions, SCD1 reference data, append-only facts, and data-quality expectations with quarantine.
+* **Conformed, history-tracked silver** with SCD2 dimensions, SCD1 reference data, append-only facts, and data-quality expectations with quarantine.
 * **Governed gold star schema** with surrogate keys and point-in-time (as-of) joins, plus two metrics approaches side by side: pre-aggregated materialized views and UC metric views.
 * **Template specs** that collapse repetitive bronze and silver flows into reusable templates.
 * **A three-run incremental simulation** covering SCD changes, fact growth, a backdated out-of-order correction, and ongoing quarantine.

scripts/README.md

@@ -27,8 +27,8 @@ python scripts/validate_dataflows.py -v
 ```
 
 **What it validates:**
-- Searches for all `*_main.json` files in `dataflows/**/dataflowspec/` directories
-- Validates against `src/schemas/main.json` (which routes to appropriate sub-schemas)
+- Searches for all `*_main.json` files under `dataflows/` (including `dataflowspec/` subdirectories and flat `dataflows/*_main.json` layouts)
+- Validates against `src/lakeflow_framework/schemas/main.json` (which routes to appropriate sub-schemas)
 - Reports validation errors with clear messages
 
 **Version Mapping (enabled by default):**

src/local/config/README.md

@@ -4,7 +4,7 @@ Fork-safe sparse override directory for framework configuration files.
 
 ## How it works
 
-Files placed here are **deep-merged on top of their `src/config/default/`
+Files placed here are **deep-merged on top of their `src/lakeflow_framework/config/default/`
 counterparts** at runtime. You only need to include the keys you want to
 change — all other keys retain their default values.
 
@@ -25,7 +25,7 @@ key improvements:
 
 ## Supported files
 
-Any file present in `src/config/default/` can be partially overridden here
+Any file present in `src/lakeflow_framework/config/default/` can be partially overridden here
 using the same filename. Common candidates:
 
 | File | Purpose |
@@ -47,7 +47,7 @@ default.
 To migrate:
 
 1. Identify which keys in your `config/override/` files differ from
-   `config/default/`.
+   `lakeflow_framework/config/default/`.
 2. Create sparse files here containing only those differing keys.
 3. Delete or empty `config/override/`.
 

tests/README.md

@@ -14,7 +14,7 @@ pytest tests/ -m "not integration and not spark"
 pytest tests/ -m integration
 
 # Optional coverage report
-pytest tests/ -m "not integration and not spark" --cov=src --cov-report=term-missing
+pytest tests/ -m "not integration and not spark" --cov=lakeflow_framework --cov-report=term-missing
 ```
 
 Use **Python 3.12** for development; newer Python versions may break PySpark-related dependencies.
@@ -32,7 +32,7 @@ tests/
 ├── unit/                # Fast, isolated module tests
 └── integration/         # Sample-dependent tests (require samples/ checkout)
     ├── conftest.py      # validate_bundle(), bundle paths, feature-samples fixtures
-    ├── test_validate_dataflows.py      # Script helpers + all specs per JSON bundle
+    ├── test_validate_dataflows.py      # Per-bundle sample validation + CLI subprocess
     └── test_feature_samples_spec_builder.py  # DataflowSpecBuilder on feature-samples
 ```
 
@@ -50,7 +50,7 @@ Configure in `pytest.ini`. CI runs: `-m "not integration and not spark"`.
 
 1. **Use `pipeline_context`** — do not call `initialize_core()` inline in test modules.
 2. **Prefer `tests/fixtures/`** — unit tests should not depend on `samples/` paths.
-3. **Mirror `src/`** — `tests/unit/test_<module>.py` maps to `src/<module>.py`.
+3. **Mirror the package** — `tests/unit/test_<module>.py` maps to `src/lakeflow_framework/<module>.py`; use `lakeflow_framework.*` imports (not compat shims).
 4. **One concern per test** — name tests `test_<behavior>_when_<condition>`.
 5. **No DLT mocks** — tests that need `@dp.table` belong in samples/E2E, not unit tests.