feat(config): sparse fork-safe local config override with deep-merge and caller-owned override deprecation (#78)

Introduces src/config_resolver.py as the single source of truth for
framework config loading. load_framework_config() loads the authoritative
default from config/default/<name> and, when present, deep-merges a
partial override file from src/local/config/<name> on top — so users only
need to specify the keys they want to change.

- src/config_resolver.py: new module with load_framework_config,
  resolve_framework_config_dir, load_framework_global_config, and the
  deprecated resolve_framework_config_path shim (kept until v1.0.0)
- src/constants.py: add FrameworkPaths.LOCAL_CONFIG_PATH
- src/logger.py: load_framework_logger_config now delegates to the
  resolver; fix factory name mismatch (create_logger → get_logger)
- src/dlt_pipeline_builder.py: replace manual config path construction
  with config_resolver calls
- src/dataflow_spec_builder/spec_mapper.py: use
  resolve_framework_config_dir for mapping path
- src/logger.py: simplify and clarify logger config loading and overall 
  code readability
- src/utility.py: extract config resolution functions into
  config_resolver; keep deep_merge here
- src/local/config/logger.json + README.md: sample overlay and
  usage documentation
- Deprecate config/override/ with a runtime DeprecationWarning
- docs: rewrite feature_framework_configuration.rst; update
  feature_logging.rst for new config path; add ADR 0006

Author: rederik76

VERSION

@@ -1 +1 @@
-v0.14.0
+v0.15.0

docs/decisions/0003-pluggable-logger-architecture.md

@@ -43,8 +43,9 @@ Introduce a **factory-based pluggable logger** configured entirely via
 
 **Resolution sequence** (`src/logger.py`, `resolve_pipeline_logger`):
 
-1. Load `logger.json` from the framework bundle (`src/config/override/`) and the
-   pipeline bundle (`pipeline_configs/`); deep-merge per precedence rules
+1. Load `logger.json` from the framework bundle (`src/local/config/` overlay on top of
+   `src/config/default/`, or `src/config/override/` if still present — deprecated v0.13.0)
+   and the pipeline bundle (`pipeline_configs/`); deep-merge per precedence rules
    (framework wins by default; `allow_pipeline_logger_override: true` inverts).
 2. If `enabled` is `false`, return the default `lakeflowframework` stdout logger.
 3. If `library` is set and not importable, fall back to the default logger with a
@@ -53,8 +54,9 @@ Introduce a **factory-based pluggable logger** configured entirely via
    the returned object exposes `debug`, `info`, `warning`, `error`, `critical`,
    and `exception`.
 5. On any failure (import error, factory error, invalid return type), fall back to
-   the default logger with a warning — the pipeline never fails due to logging
-   misconfiguration.
+   the default logger, emit a `CustomLoggerConfigWarning` (Python `warnings` — visible
+   in stderr and catchable with `pytest.warns`), and log an `ERROR` record — the
+   pipeline never fails due to logging misconfiguration.
 6. Apply `mirror_to_stdout` and `CompositeLogger` wrapping (see ADR-0004).
 
 **Resolved log level injection:** before calling the factory the framework

docs/decisions/0006-local-config-sparse-overlay.md

@@ -0,0 +1,124 @@
+# ADR-0006: `src/local/config/` sparse overlay replaces `config/override/`
+
+**Date:** 2026-05-17
+**Status:** Accepted
+**PR:** feature/local-config-resolver-and-override (v0.15.0)
+
+---
+
+## Context
+
+The original framework config mechanism offered a binary choice: use
+`config/default/` (shipped defaults) **or** replace it entirely with
+`config/override/` (full-tree copy). This had three significant problems:
+
+1. **Whole-tree requirement.** A customer who wanted to change a single key in
+   `global.json` had to copy the entire `config/default/` tree into
+   `config/override/` and keep it in sync across framework upgrades. A missed
+   sync would silently hide new default values.
+
+2. **Coupling across config files.** The original resolver treated the presence
+   of any non-hidden file in `config/override/` as a signal to use that tree
+   exclusively for _all_ framework config reads. Adding a `logger.json` override
+   inadvertently activated the override tree for global config, requiring the
+   override tree to be complete.
+
+3. **Override logic scattered inside the loader.** The loader (`load_framework_config`)
+   was responsible for detecting whether `config/override/` was active, emitting
+   deprecation warnings, and loading the correct file. This mixed concerns and made
+   the detection logic duplicated at every call site (especially for multi-file
+   resolution like `GLOBAL_CONFIG`).
+
+The `src/local/` fork-safe area introduced in ADR-0001 provides a natural home
+for customer customisations. Config overrides should follow the same pattern.
+
+## Decision
+
+Introduce `src/local/config/` as a **sparse overlay directory** whose files are
+**deep-merged on top of their `src/config/default/` equivalents** at runtime.
+
+### Separation of concerns
+
+Override detection and deprecation warnings are the **caller's responsibility**.
+`load_framework_config` is a pure loader with no override logic:
+
+- **`DLTPipelineBuilder._load_framework_global_config`** checks
+  `config/override/` for global config files, emits the `DeprecationWarning` if
+  active, stores the resolved root in `self._active_config_path`, and passes it
+  to `load_framework_config`.  `self._active_config_path` is reused by
+  `_setup_operational_metadata` so the override detection only runs once per
+  pipeline initialisation.
+
+- **`load_framework_logger_config`** independently checks
+  `config/override/logger.json`, emits its own `DeprecationWarning` if active,
+  and passes the resolved root to `load_framework_config`.  Logger config is
+  detected separately from global config because a customer may have one but
+  not the other.
+
+### `load_framework_config` API
+
+```python
+load_framework_config(
+    name: str | Sequence[str],
+    framework_path: str,
+    config_path: str = FrameworkPaths.CONFIG_PATH,   # caller-resolved active root
+    fail_on_not_exists: bool = True,
+) -> Dict
+```
+
+Behaviour:
+
+1. When `name` is a **sequence** (e.g. `FrameworkPaths.GLOBAL_CONFIG =
+   ("global.json", "global.yaml", "global.yml")`):
+   - Find all matching files inside `config_path`.
+   - Raise `ValueError` if more than one match is found.
+   - Raise `FileNotFoundError` (or return `{}` if `fail_on_not_exists=False`)
+     if no match is found.
+   - Resolve to the single matching filename and proceed.
+2. Load the file from `os.path.join(framework_path, config_path, name)`.
+3. If `src/local/config/<name>` exists, **deep-merge** it on top:
+   - Dict values merged recursively; overlay wins on conflicts.
+   - Non-dict values and lists replaced wholesale.
+   - Keys absent from the overlay retain their default values.
+4. Return the merged dict.
+
+No override detection. No warnings. Callers pass the already-resolved `config_path`.
+
+### `resolve_framework_config_dir` (directory-based config)
+
+For directory-based config (e.g. `dataflow_spec_mapping/`):
+
+1. If `src/local/config/<subdir>/` exists, return it (local wins entirely).
+2. Otherwise return `src/config/default/<subdir>/`.
+
+### `resolve_framework_config_path` (legacy shim)
+
+Kept until v1.0.0 for any external callers. Emits a `DeprecationWarning` when
+`config/override/` is active. Framework internals no longer call this.
+
+### Deprecation timeline
+
+| Version | Action |
+|---------|--------|
+| v0.13.0 | `config/override/` deprecated; `DeprecationWarning` added |
+| v0.15.0 | `src/local/config/` introduced; all framework call sites migrated to `load_framework_config`; override detection moved to callers |
+| v1.0.0  | `config/override/` support and `resolve_framework_config_path` removed |
+
+## Consequences
+
+- A customer who wants to change one key in `global.json` creates a one-field
+  `src/local/config/global.json` — no need to maintain a full copy.
+- Framework upgrades that add new default keys automatically appear in the
+  merged config; only explicitly overridden keys are affected.
+- `config/override/` still works during the deprecation window; a
+  `DeprecationWarning` is emitted by the caller (not deep inside the loader),
+  giving an accurate stack frame in the pipeline logs.
+- The multi-file resolution check (preventing duplicate `global.json` and
+  `global.yaml` coexisting) now lives inside `load_framework_config` when a
+  sequence is passed — consistent with the pipeline bundle's equivalent check in
+  `_load_pipeline_bundle_global_config_file`.
+- The `logger.json` override (pluggable logger, ADR-0003) now lives in
+  `src/local/config/logger.json` rather than `config/override/`.
+- Substitutions and secrets files (which include workspace-environment prefixes)
+  are not yet covered by `load_framework_config` and remain on the legacy path
+  until a follow-up PR addresses the workspace-env prefix complexity.

docs/source/feature_framework_configuration.rst

@@ -11,21 +11,25 @@ Framework configuration
    * - **Databricks Docs:**
      - NA
 
-Framework-level settings (global JSON/YAML, substitutions, secrets, spec mappings, operational metadata) live under **one** active directory. The framework chooses between a **default** tree and an optional **override** tree; everything else reads paths relative to that choice.
+Framework-level settings (global JSON/YAML, substitutions, secrets, spec
+mappings, operational metadata) live under ``src/config/default/``. Individual
+values can be overridden per-deployment using sparse files in
+``src/local/config/`` — only the keys you want to change are needed.
 
 Configuration
 -------------
 
 | **Scope: Global (framework bundle)**
-| **Default:** ``./config/default/`` (for example ``src/config/default/`` when the framework root is ``src``).
-| **Override:** ``./config/override/`` (for example ``src/config/override/``). Optional; see **Override** below.
+| **Default:** ``src/config/default/`` — authoritative, always read.
+| **Override:** ``src/local/config/`` — sparse override; deep-merged on top of defaults.
 
-Under the active directory you normally have:
+Under ``src/config/default/`` you normally have:
 
 * exactly one global file: ``global.json``, ``global.yaml``, or ``global.yml``
 * a ``dataflow_spec_mapping/`` directory (see :doc:`feature_versioning_dataflow_spec`)
 * optional per-target substitution and secrets files (see :doc:`feature_substitutions`, :doc:`feature_secrets`)
 * optional ``operational_metadata_<layer>.json`` (see :doc:`feature_operational_metadata`)
+* optional ``logger.json`` (see :doc:`feature_logging`)
 
 Mandatory
 ---------
@@ -57,13 +61,51 @@ Inside the global file, all top-level keys are optional. Common ones:
    * - ``override_max_workers`` / ``pipeline_builder_disable_threading``
      - :doc:`feature_builder_parallelization`
 
-Override
---------
+Local override (``src/local/config/``)
+---------------------------------------
+
+Place sparse JSON/YAML files in ``src/local/config/`` to override individual
+keys without copying the entire default file. The framework **deep-merges** the
+overlay on top of the defaults at runtime:
+
+* Dict values are merged recursively — only the keys present in the overlay
+  are changed.
+* Non-dict values and lists are replaced wholesale.
+* Keys not present in the overlay retain their default values.
+
+Example — change one global setting without touching the rest of ``global.json``:
+
+.. code-block:: json
+
+   {
+     "dataflow_spec_version": "0.0.3"
+   }
+
+Save this as ``src/local/config/global.json``. All other keys from
+``src/config/default/global.json`` are kept unchanged.
+
+For **directory-based config** (e.g. ``dataflow_spec_mapping/``), place the
+entire override directory in ``src/local/config/`` — the local directory takes
+full precedence over the default.
+
+See ``src/local/config/README.md`` in the framework bundle for the full list of
+supported files and migration instructions.
+
+.. deprecated:: v0.14.0
+
+   ``config/override/``
+   ^^^^^^^^^^^^^^^^^^^^
+
+   The ``config/override/`` mechanism (whole-tree replacement) is deprecated as
+   of v0.14.0 and will be removed in v1.0.0. Migrate to ``src/local/config/``
+   sparse files instead.
 
-* If ``./config/override/`` has **no** non-hidden files (only names starting with ``.``, such as ``.gitkeep``), the framework uses ``./config/default/``.
-* If it has **any** non-hidden file or folder, the framework uses ``./config/override/`` instead—but then that directory must already contain **both** a valid global file and a ``dataflow_spec_mapping/`` directory. Otherwise startup fails with a message to copy the full layout from ``./config/default/``.
-* If **neither** directory has non-hidden content, startup fails: add configuration under ``./config/default/``.
+   **Migration steps:**
 
-.. tip::
+   1. Identify which keys in your ``config/override/`` files differ from
+      ``config/default/``.
+   2. Create sparse files in ``src/local/config/`` containing only those keys.
+   3. Remove all files from ``config/override/`` (leave the ``.gitkeep``).
 
-   Leave ``config/override`` empty until you can mirror the whole ``config/default`` tree.
+   A ``DeprecationWarning`` is emitted at pipeline startup when
+   ``config/override/`` contains non-hidden files.

docs/source/feature_logging.rst

@@ -374,6 +374,11 @@ Create ``src/local/libraries/structured_stdout_logger.py``:
        """Factory called by the framework as factory(dbutils, spark, **factory_args)."""
        return StructuredStdoutLogger(level=level, logger_name=logger_name)
 
+**Step 2 — Enable the logger via** ``src/local/config/logger.json``
+
+Add or update ``src/local/config/logger.json`` in the framework bundle
+(a sparse file is sufficient — only the keys you want to set are needed):
+
 **Step 2 — Enable the logger via** ``config/override/logger.json``
 
 Add or update ``src/config/override/logger.json`` in the framework bundle
@@ -602,6 +607,3 @@ Structured stdout logger output (JSON, via ``structured_stdout_logger``):
    {"timestamp": "2025-02-06T04:05:46.161000+00:00", "level": "INFO", "logger": "lakeflowframework", "message": "Initializing Pipeline..."}
    {"timestamp": "2025-02-06T04:05:48.254000+00:00", "level": "INFO", "logger": "lakeflowframework", "message": "Creating Flow: flow_name"}
    {"timestamp": "2025-02-06T04:06:26.527000+00:00", "level": "ERROR", "logger": "lakeflowframework", "message": "Failed to process Data Flow Spec: schema mismatch", "exc_info": "Traceback (most recent call last):\n  ..."}
-
-
-