feat(logging): pluggable custom logger support (#73)

Replaces the hardcoded utility.set_logger() call with a merge-driven resolution
pipeline that supports optional custom loggers configured via logger.json in both
the framework bundle (config/override/) and the pipeline bundle (pipeline_configs/).

Core changes:
- src/logger.py: new module — create_default_logger, load/merge logger.json,
  resolve_logger, CompositeLogger (custom primary + stdout mirror), and the
  top-level resolve_pipeline_logger entry point
- src/config/default/logger.json: shipped default (enabled: false)
- src/dlt_pipeline_builder.py: wire resolve_pipeline_logger; call
  register_bundle_sys_paths() before logger init so src/local/libraries/
  modules are importable when the factory is loaded
- src/bundle_loader.py: make logger optional to break the init chicken-and-egg
- src/utility.py: convert debug print() calls to logger.debug(); add optional
  logger param to get_data_from_files_parallel
- tests/test_logger.py: unit tests for merge, resolve, and CompositeLogger
- docs/source/feature_logging.rst: full rewrite; includes custom logger contract
  and step-by-step structured JSON logger example

Author: rederik76

VERSION

@@ -1 +1 @@
-v0.13.1
+v0.14.0

docs/decisions/0003-pluggable-logger-architecture.md

@@ -0,0 +1,88 @@
+# ADR-0003: Pluggable logger architecture (factory pattern + CompositeLogger)
+
+**Date:** 2026-05-17
+**Status:** Accepted
+**PR:** feature/pluggable-logger (v0.14.0)
+
+---
+
+## Context
+
+The framework previously used a single hard-coded `logging.Logger` instance
+(`lakeflowframework`) writing plain-text records to stdout. Customers with structured
+logging infrastructure (Application Insights, Splunk, Datadog, internal JSON
+pipelines) had no way to integrate the framework's log output without patching
+the framework itself.
+
+Requirements gathered from customer scenarios:
+
+- Zero-code integration for loggers whose factory signature is compatible with
+  `factory(dbutils, spark, **kwargs)` — config-only.
+- A thin wrapper path for loggers whose signature is incompatible.
+- Pipeline-level `logLevel` must propagate to the custom logger.
+- Misconfiguration must never crash the pipeline.
+- The default logger must remain the out-of-the-box experience — no opt-out
+  required from customers who do not need a custom logger.
+
+## Decision
+
+Introduce a **factory-based pluggable logger** configured entirely via
+`logger.json` files:
+
+```json
+{
+  "enabled": true,
+  "module": "my_logger_module",
+  "factory": "get_logger",
+  "level": "INFO",
+  "factory_args": {
+    "log_to_output": false
+  }
+}
+```
+
+**Resolution sequence** (`src/logger.py`, `resolve_pipeline_logger`):
+
+1. Load `logger.json` from the framework bundle (`src/config/override/`) and the
+   pipeline bundle (`pipeline_configs/`); deep-merge per precedence rules
+   (framework wins by default; `allow_pipeline_logger_override: true` inverts).
+2. If `enabled` is `false`, return the default `lakeflowframework` stdout logger.
+3. If `library` is set and not importable, fall back to the default logger with a
+   warning.
+4. Import `module`, call `factory(dbutils, spark, **factory_args)`, validate that
+   the returned object exposes `debug`, `info`, `warning`, `error`, `critical`,
+   and `exception`.
+5. On any failure (import error, factory error, invalid return type), fall back to
+   the default logger with a warning — the pipeline never fails due to logging
+   misconfiguration.
+6. Apply `mirror_to_stdout` and `CompositeLogger` wrapping (see ADR-0004).
+
+**Resolved log level injection:** before calling the factory the framework
+injects the resolved `level` (Spark `logLevel` pipeline setting wins over the
+JSON config value) as a `"level"` key in `factory_args`. Custom factories must
+declare `level: str = "INFO"` in their signature.
+
+**`sys.path` ordering:** `register_bundle_sys_paths` is called **before**
+`resolve_pipeline_logger` so that modules in `src/local/libraries/` are
+importable when the factory is loaded. See ADR-0005.
+
+**Custom logger contract:** the object returned by the factory must implement
+`debug`, `info`, `warning`, `error`, `critical`, and `exception`
+as `(message, *args, **kwargs)`. It must handle `exc_info`, `stacklevel`, and
+`stack_info` kwargs (pop and discard the latter two; resolve `exc_info=True` via
+`traceback.format_exc()`). Level checks must be applied before message
+formatting (lazy evaluation). See `docs/source/feature_logging.rst` for the
+full contract table and reference implementation.
+
+## Consequences
+
+- Customers can integrate any logger whose factory accepts
+  `(dbutils, spark, **kwargs)` using only `logger.json` — no code changes.
+- Customers whose logger requires adaptation write a thin wrapper module in
+  `src/local/libraries/` or as a cluster-installed package; `logger.json`
+  points to the wrapper.
+- Pipeline startup always succeeds regardless of logging misconfiguration.
+- The framework default logger (`lakeflowframework` / plain-text stdout) remains the
+  default; no action needed from customers who do not require a custom logger.
+- The `level` kwarg injection is part of the public factory contract — custom
+  logger authors must handle it.

docs/decisions/0004-mirror-to-stdout-default.md

@@ -0,0 +1,68 @@
+# ADR-0004: `mirror_to_stdout` defaults to `false`; default logger silenced when custom logger is active
+
+**Date:** 2026-05-17
+**Status:** Accepted
+**PR:** feature/pluggable-logger (v0.14.0)
+
+---
+
+## Context
+
+When a custom logger is active there are two potential sources of stdout output:
+
+1. The custom logger itself (e.g. structured JSON written via `print()`).
+2. The `lakeflowframework` Python `logging.Logger` which was already writing
+   plain-text lines to stdout before the custom logger existed.
+
+Without an explicit policy both sources emit simultaneously, producing duplicate
+or mixed-format output in the Databricks pipeline **Logs** UI. This is
+confusing and wastes log ingestion quota for customers routing logs to external
+systems.
+
+Two use-cases must be supported:
+
+- **Custom logger owns stdout** — structured JSON only; plain-text lines from
+  the default logger must be suppressed.
+- **Dual output** — custom logger forwards to an external system _and_ the
+  plain-text stdout mirror is retained for easy in-UI inspection
+  (`mirror_to_stdout: true`).
+
+## Decision
+
+`mirror_to_stdout` defaults to **`false`**.
+
+When a custom logger initialises successfully and `mirror_to_stdout` is `false`
+(default):
+
+- All handlers are removed from the `lakeflowframework` `logging.Logger` instance.
+- A `logging.NullHandler` is attached so any code holding a direct
+  `logging.getLogger("lakeflowframework")` reference stays quiet.
+- The custom logger is returned directly — it is the sole output path.
+
+When `mirror_to_stdout` is `true`:
+
+- The custom logger is wrapped in a **`CompositeLogger`** together with the
+  default `lakeflowframework` stdout handler.
+- Every log call is forwarded to the custom logger first (primary), then to the
+  default logger (mirror).
+- `CompositeLogger` passes `*args` and `**kwargs` through to the mirror logger
+  unchanged so formatting and metadata are not lost.
+- `CompositeLogger.close()` closes both loggers.
+
+When the custom logger fails to load, `mirror_to_stdout` has no effect — the
+default logger is returned unchanged.
+
+## Consequences
+
+- Customers whose custom logger writes to stdout will not see duplicate lines by
+  default.
+- Customers who want dual output set `mirror_to_stdout: true` and (if their
+  custom logger also writes to stdout) suppress that output via a factory arg
+  (e.g. `log_to_output: false`).
+- The `lakeflowframework` `logging.Logger` instance is modified in-place when
+  silenced. Any code that obtained a direct `logging.getLogger("lakeflowframework")`
+  reference before logger resolution will stop producing output — this is the
+  intended behaviour.
+- `mirror_to_stdout: false` is the breaking change relative to the previous
+  hard-coded stdout logger. Teams relying on plain-text stdout output and also
+  enabling a custom logger must set `mirror_to_stdout: true` explicitly.

docs/decisions/0005-syspath-registration-decoupled-from-logger.md

@@ -0,0 +1,63 @@
+# ADR-0005: `sys.path` registration decoupled from logger initialisation
+
+**Date:** 2026-05-17
+**Status:** Accepted
+**PR:** feature/pluggable-logger (v0.14.0)
+
+---
+
+## Context
+
+`DLTPipelineBuilder.__init__` must perform two steps in sequence:
+
+1. Register `src/local/libraries/` and other bundle paths on `sys.path`
+   (`register_bundle_sys_paths`) so that custom modules are importable.
+2. Resolve the pipeline logger (`resolve_pipeline_logger`), which may import a
+   custom logger module from one of those paths.
+
+The original implementation of `register_bundle_sys_paths` required a fully
+initialised logger to emit `INFO` messages about which paths were registered.
+This created a circular dependency:
+
+- The logger cannot be resolved until `sys.path` is registered (the custom
+  logger module may live in `src/local/libraries/`).
+- `sys.path` registration required a logger to report what it was doing.
+
+The temporary fix of passing a bare `logging.getLogger` instance worked but
+meant the framework always emitted plain-text `INFO` lines during bootstrap
+regardless of whether a custom logger was configured — the opposite of the
+intent.
+
+## Decision
+
+Make the `logger` parameter **optional** (`logger=None`) in
+`register_bundle_sys_paths`, `_add_if_dir`, and `_warn_legacy_extensions`
+in `bundle_loader.py`. All logging calls inside these functions are guarded by
+`if logger is not None:`.
+
+`DLTPipelineBuilder.__init__` calls `register_bundle_sys_paths` **silently**
+(without a logger) early in bootstrap, before `resolve_pipeline_logger` is
+called. The log messages about which paths were registered are therefore emitted
+only when the caller explicitly passes a logger — for example, during
+reregistration triggered by a reload.
+
+```python
+# bootstrap order in DLTPipelineBuilder.__init__
+self._load_mandatory_paths()
+register_bundle_sys_paths(self.framework_path, self.bundle_path)  # silent
+self.logger = pipeline_logger.resolve_pipeline_logger(...)        # custom module now importable
+self.logger.info("Initializing Pipeline...")
+```
+
+## Consequences
+
+- Custom logger modules placed in `src/local/libraries/` are guaranteed to be
+  on `sys.path` before the factory import is attempted.
+- Bootstrap `sys.path` registration produces no log output. This is acceptable
+  because any misconfiguration (missing path, permission error) raises an
+  exception rather than a log message.
+- Callers that want diagnostic output from `register_bundle_sys_paths` must pass
+  a logger explicitly — the function signature is backward-compatible (existing
+  callers passing a logger continue to work unchanged).
+- Future extensions to `bundle_loader` should follow the same optional-logger
+  pattern to avoid reintroducing the circular dependency.