fix(docs): fix Sphinx build warnings across documentation (#80)

- Add autosectionlabel_prefix_document = True to eliminate ~80 duplicate label warnings
- Fix broken :doc: and :ref: cross-references (Groups 4 & 5)
- Fix code-block language directives causing lexer failures (Group 6)
- Fix all "Title underline too short" warnings across 27 files
- Remove stale toctree entries and deduplicate files in multiple toctrees (Groups 1 & 3)
- Fix CRITICAL unexpected section title in feature_framework_configuration.rst
- Add patterns_streaming_flow_groups to toctree (Group 2)
- Remove deploy_enterprise.rst (empty orphan)
- Rename deployment anchors in framework and pipeline bundle docs to avoid
cross-page label collisions
- mark standalone reference/guide pages as :orphan: to suppress toc.not_included warnings when they are linked but not listed in a toctree.
- Enabled sphinx spell checking as make target - make spelling. Fixed spelling issues and added word list.
- fix docs dependency typo by replacing gitsphinx-autoapi==3.8.0 with
sphinx-autoapi==3.8.0 in requirements-docs.txt.

Author: rederik76

VERSION

@@ -1 +1 @@
-v0.16.0
+v0.16.1

docs/Makefile

@@ -13,7 +13,7 @@ IMAGEDIR      = source/images
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: help Makefile markdown
+.PHONY: help Makefile markdown spelling
 
 # Add custom markdown build command
 md:
@@ -26,6 +26,10 @@ md:
 	@cp -r "$(SOURCEDIR)/_static"/* "$(BUILDDIR)/markdown/_static"
 	@echo "Build finished. The markdown files are in $(BUILDDIR)/markdown."
 
+# Add explicit spelling target
+spelling:
+	@$(SPHINXBUILD) -M spelling "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile

docs/conf.py

@@ -23,9 +23,12 @@
     'sphinx_design',
     'myst_parser',
     'sphinx_tabs.tabs',
-    'custom_markdown_builder'
+    'custom_markdown_builder',
+    "sphinxcontrib.spelling"
 ]
 
+autosectionlabel_prefix_document = True
+
 templates_path = ['source/_templates']
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 

docs/source/build_pipeline_bundle_steps.rst

@@ -134,13 +134,14 @@ The databricks.yml needs to be adjusted to include the following configurations:
 Based on the Use Case and the standards defined in your Org, select the appropriate bundle structure. See the :doc:`build_pipeline_bundle_structure` section for guidance.
 
 4. Select your Data Flow Specification Language / Format
--------------------------------------------------------
+--------------------------------------------------------
 
 Based on the implementation and standards in your Org, you can select the appropriate specification language / format. See the :doc:`feature_spec_format` section details.
 
 Be aware that:
+
 - The default format is `JSON`.
-- The format may have alrteady been enforced globally at Framework level per you orgs standards.
+- The format may have already been enforced globally at Framework level per your org standards.
 - If enabled at Framework level, you can set the format at the Pipeline Bundle level.
 - You cannot mix and match formats in the same bundle, it's important to ensure consistency for engineers working on the same bundle.
 
@@ -149,7 +150,7 @@ Be aware that:
 
 If you haven't already done so, familiarize yourself with the :doc:`feature_substitutions` feature of the Framework.
 
-If you need to use substitutions and the substitutions you require have not been configure globally at the Framework level, you need to now setup your substitutions file. See the :doc:`substitutions` section for guidance.
+If you need to use substitutions and the substitutions you require have not been configure globally at the Framework level, you need to now setup your substitutions file. See the :doc:`feature_substitutions` section for guidance.
 
 .. note::
     This step is optional and only required if substitutions are required to deploy the same pipeline bundle to multiple environments with different resources names. This step can also be actioned later in the build process after the Data Flow Specs have been created.
@@ -182,8 +183,8 @@ Iterate over the following steps to create each individual Data Flow:
 
   If your pipeline requires Spark configuration, event hook registration, or any one-time setup that must run outside of Data Flow logic, add ``.py`` scripts to:
 
-  - ``src/init/pre/`` — run **before** SDP dataflow declarations
-  - ``src/init/post/`` — run **after** SDP dataflow declarations
+  - ``src/init/pre/`` — run **before** SDP data flow declarations
+  - ``src/init/post/`` — run **after** SDP data flow declarations
 
   Scripts are executed in sorted filename order. Files whose names begin with ``_`` are skipped. Use a numeric prefix (e.g. ``01_setup.py``) to control execution order.
 
@@ -199,6 +200,7 @@ Iterate over the following steps to create each individual Data Flow:
     If necessary, create a new folder in the ``src/dataflows`` directory based on your selected bundle strategy.
 
   b. Create Data Flow Spec file(s):
+
     * Refer to the :doc:`dataflow_spec_reference` section to build your Data Flow Spec
     * Refer to the :doc:`patterns` section for high level patterns and sample code.
     * Refer to the :doc:`deploy_samples` section on how to deploy the samples so you can reference the sample code.
@@ -208,7 +210,7 @@ Iterate over the following steps to create each individual Data Flow:
     Create your schema JSON / DDL files in the ``schema`` sub-directory of your Data Flow Spec's home folder:
 
       * You should in general always specify a schema for your source and target tables, unless you want schema evolution to happen automatically in Bronze.
-      * Schemas are optional for staging tables.
+      * Schema definitions are optional for staging tables.
       * Each schema must be defined in it's own individual file.
       * Each schema must be referenced by the appropriate object(s) in your Data Flow Spec JSON file(s).
 
@@ -280,7 +282,7 @@ To create a single Pipeline definition, follow these steps:
 
 3. **Add any required Data Flow filters:**
 
-  By default, if you don't specify any Data Flow filters, the pipleine will execute all Data Flows in you Pipeline Bundle.
+  By default, if you don't specify any Data Flow filters, the pipeline will execute all Data Flows in your Pipeline Bundle.
 
   If you are creating more than one Pipeline definition in your bundle, you may want your Pipeline(s) to only execute specific Data Flows. 
 
@@ -309,7 +311,7 @@ To create a single Pipeline definition, follow these steps:
   You can add the appropriate Data Flow filter options described above to the Pipeline definition, as show below:
 
   .. code-block:: yaml
-    :emphasize-lines: 20-23
+    :emphasize-lines: 19-22
 
       resources:
         pipelines:

docs/source/build_pipeline_bundle_structure.rst

@@ -15,7 +15,7 @@ Ultimately you will need to determine the best way to scope your Pipeline Bundle
 
 .. important::
 
-   Per the :ref:`concepts` section of this documentation:
+   Per the :doc:`concepts` section of this documentation:
 
    * A data flow, and its Data Flow Spec, defines the source(s) and logic required to generate a **single target table**.
    * A Pipeline Bundle can contain multiple Data Flow Specs, and a Pipeline deployed by the bundle may execute the logic for one or more Data Flow Specs.
@@ -36,7 +36,7 @@ Some of the most common groupings strategies are shown below:
    * - Logical Grouping
      - Description
    * - Monolithic
-     - A single Pipeline Bundle containing all ata flows and Pipeline definitions. Only suitable for smaller and simpler deployments.
+     - A single Pipeline Bundle containing all data flows and Pipeline definitions. Only suitable for smaller and simpler deployments.
    * - Bronze
      - * Source System - A Pipeline per Source System or application
    * - Silver / Enterprise Models
@@ -116,7 +116,7 @@ The ``src/`` directories serve distinct purposes:
 
 It is the structure of the ``src/dataflows`` directory that is flexible and can be organised in the way that best suits your standards and ways of working. The Framework will:
 
-* Read all the Data Flow Spec files under the ``src/dataflows`` directory, regardless of the folder structure. Filtering of the Dataflows is done when defining your Pipeline and is discussed in the :doc:`build_pipeline_bundle_steps` section. 
+* Read all the Data Flow Spec files under the ``src/dataflows`` directory, regardless of the folder structure. Filtering of the data flows is done when defining your Pipeline and is discussed in the :doc:`build_pipeline_bundle_steps` section.
 * Expect that the schemas, transforms and expectations related to a Data Flow Spec are located in their respective ``schemas``, ``dml`` and ``expectations`` sub-directories within the Data Flow Spec's home directory.
 
 The most common ways to organize your ``src/dataflows`` directory are:

docs/source/concepts.rst

@@ -11,9 +11,9 @@ The below diagram illustrates some of the key concepts of the Framework, which a
 
 .. _concepts_dabs:
 
-Databricks Asset Bundles (DABs)
-===============================
-Databricks Asset Bundles (DABs) are a way to package and deploy Databricks assets such as source code, Spark Declarative Pipelines notebooks and libraries. 
+Declarative Automation Bundles (DABs)
+=====================================
+Declarative Automation Bundles (DABs) are a way to package and deploy Databricks assets such as source code, Spark Declarative Pipelines notebooks and libraries. 
 This concept is core to how the Lakeflow Framework has been designed and implemented.
 
 Detailed documentation on DABs can be found at: https://docs.databricks.com/en/dev-tools/bundles/index.html
@@ -256,12 +256,12 @@ In the Lakeflow Framework the following types of data flows can be defined in a
 
 2. **Flow**
 
-  Flows data flows allow you to create simple or complex data flows, using the different components of a flow as building blocks. They implement the :doc:`feature_multi_source_streaming` feature of DLT.
+  Flows data flows allow you to create simple or complex data flows, using the different components of a flow as building blocks. They implement the :doc:`feature_multi_source_streaming` feature of SDP.
   Flows are useful for Silver and Gold scenarios, and where multiple sources and transformations are required.
 
 3. **Materialized Views**
 
-  Materialized Views are the precomputed results of a query stored in a Table. They are useful for Gold scenarios, and where complex transformations are required.
+  Materialized Views are the pre-computed results of a query stored in a Table. They are useful for Gold scenarios, and where complex transformations are required.
 
 .. important::
 
@@ -363,10 +363,10 @@ Data Flow Spec Components:
    * - :ref:`Table Migration Details (optional) <dataflow-spec-flows-table-migration-configuration>`
      - The details of the table being migrated from.
    * - :ref:`Flow Groups <dataflow-spec-flows-flow-groups-configuration>`
-     - Contains the flow groups for the dataflow.
+     - Contains the flow groups for the data flow.
 
        * A flow group can contain one or more flows.
-       * flows implements the :doc:`feature_multi_source_streaming` feature of DLT.'
+       * flows implements the :doc:`feature_multi_source_streaming` feature of SDP.
 
 
 Flow Groups Explained
@@ -400,7 +400,7 @@ Some key points to note:
 
   * Staging tables are optional.
   * Staging tables can be referenced as a source or target in any of the flows defined in the flow group.
-  * In some cases for very large and complex data flows, you may want to decompose your dataflow into a smaller more manageable data flows. In this instance staging tables may in fact become target tables in smaller more manageable data flows. In these cases they can only be used as a source in downstream Pipelines. This however really depend on the design practices you choose to follow.
+  * In some cases for very large and complex data flows, you may want to decompose your data flow into smaller, more manageable data flows. In this instance staging tables may in fact become target tables in smaller, more manageable data flows. In these cases they can only be used as a source in downstream Pipelines. This, however, really depends on the design practices you choose to follow.
 
 When defining a staging table, you can specify the following:
 
@@ -413,7 +413,7 @@ When defining a staging table, you can specify the following:
 Flows Explained
 ~~~~~~~~~~~~~~~
 
-Flows are the building blocks of a Data Flow and they implement the :doc:`feature_multi_source_streaming` feature of DLT.
+Flows are the building blocks of a Data Flow and they implement the :doc:`feature_multi_source_streaming` feature of SDP.
 
 Flows can be defined in one of two ways:
 
@@ -436,14 +436,14 @@ Flows can be defined in one of two ways:
 
        * **append_view** - Uses a source view to append data to a staging or target table.
        * **append_sql** - Uses a raw SQL statement to append data to a staging or target table.
-       * **merge** - Uses the :ref:`CDC API's <feature_cdc>` to merge data from a source view to a staging or target table.
+       * **merge** - Uses the :doc:`CDC API's <feature_cdc>` to merge data from a source view to a staging or target table.
 
    * - :ref:`Flow Details <dataflow-spec-flows-flow-configuration>`
      - Defines the source and target of the flow and any additional properties required for the flow type.
    * - :ref:`Views <dataflow-spec-flows-flow-configuration>` (optional)
      - Views are used to define the source and any additional transformations for a flow. The different types of views are documented in the following sections:
 
-       * :doc:`feature_source_target_types`
+       * :doc:`feature_source_types`
        * :ref:`dataflow-spec-flows-view-configuration`
 
 .. important::
@@ -457,7 +457,7 @@ Flows can be defined in one of two ways:
 Materialized Views
 -------------------
 
-Materialized Views are the precomputed results of a query stored in a Table. They are typically used for Gold scenarios, and where complex transformations are required.
+Materialized Views are the pre-computed results of a query stored in a Table. They are typically used for Gold scenarios, and where complex transformations are required.
 
 Data Flow Spec Components:
 
@@ -491,7 +491,7 @@ Data Flow Spec Components:
      - Specifies any additional configuration for the target table, its configuration and properties.
    * - :ref:`Data Quality Expectations (optional) <dataflow-spec-materialized-view-data-quality-configuration>`
      - Enable expectations and specify the location of the expectations file(s).
-   * - :ref:`Quarantine Details (optional) <dataflow-spec-materialized-view-quarantine-configuration>`
+   * - :doc:`Quarantine Details (optional) <dataflow_spec_ref_data_quality>`
      - Set the quarantine mode and if the mode is ``table`` the details of the quarantine table.
 
 Patterns

docs/source/contributor_dev_docs.rst

@@ -81,7 +81,7 @@ to install the documentation dependencies separately —
 ``requirements-docs.lock``.
 
 Updating dependencies / regenerating the lockfile
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you add, remove, or bump a documentation dependency, edit
 ``requirements-docs.txt`` (the unhashed source-of-truth file) and then

docs/source/contributor_dev_env.rst

@@ -4,7 +4,7 @@ Development Environment Setup
 The sections below assumes the Lakeflow Framework repository has been cloned from git and you are in the root directory. If not please do so first.
 
 Setting up for development as a contributor to the Lakeflow Framework
-================================================================
+=====================================================================
 
 Once you have cloned the Lakeflow Framework repository, you'll need to follow the steps below to set up the framework.
 
@@ -36,10 +36,10 @@ Once you have cloned the Lakeflow Framework repository, you'll need to follow th
       ./scripts/generate_lockfiles.sh
 
    See :doc:`contributor_dev_docs` for more details on the lockfiles.
-2. **Set up VS Code extentions**
+2. **Set up VS Code extensions**
 
    Once you open the Lakeflow Framework workspace in VS Code for the first time, VS Code will prompt you to install the recommended extensions. 
-   If you missed this prompt, you can review and install the recommended extensions with the Extensions: Show Recommended Extensions command or by clicking on the extentions tab on left side of the window and selecting "Workspace Recommendations".
+   If you missed this prompt, you can review and install the recommended extensions with the Extensions: Show Recommended Extensions command or by clicking on the extensions tab on left side of the window and selecting "Workspace Recommendations".
 
 .. note::
 

docs/source/contributor_dev_git.rst

@@ -4,7 +4,7 @@ GIT
 Branching Strategy
 ------------------
 
-The project follows the Gitflow branching model for version control. This model provides a robust framework for managing larger projects with scheduled releases.
+The project follows the GitFlow branching model for version control. This model provides a robust framework for managing larger projects with scheduled releases.
 
 Our repository maintains the following primary branches:
 
@@ -15,7 +15,7 @@ Our repository maintains the following primary branches:
 * ``fix`` - Short-lived branches for bug fixes
 
 Feature Branch Guidelines
-^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 * Create feature branches from ``develop``
 * Branch naming: ``feature/descriptive-name`` (e.g. ``feature/add-cdc-support``)
@@ -24,7 +24,7 @@ Feature Branch Guidelines
 * Submit pull request to merge back into ``develop`` when complete
 
 Fix Branch Guidelines
-^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^
 
 * Create fix branches from ``develop`` for non-critical bugs
 * Branch naming: ``fix/issue-description`` (e.g. ``fix/logging-format``)
@@ -33,7 +33,7 @@ Fix Branch Guidelines
 * Submit pull request to merge back into ``develop`` when complete
 
 Release Strategy
----------------
+----------------
 
 The project follows a structured release process aligned with the GitFlow branching model:
 

docs/source/contributor_dev_steps.rst

@@ -26,7 +26,7 @@ Development Process
 1. Local Development
 
    - Follow coding standards and style guides
-       - Ensure the yapf extention is installed and enabled in VS Code (refer to step 2 of :doc:`contributor_dev_env`)
+       - Ensure the yapf extension is installed and enabled in VS Code (refer to step 2 of :doc:`contributor_dev_env`)
        - Use yapf to format your python code (right click and select 'Format Document With' then select yapf)
    - Stick to solid principles and object oriented design patterns
    - Deploy updated framework to Databricks to ensure it is working as expected
@@ -35,10 +35,7 @@ Development Process
 
 2. Unit Testing
 
-   - Write unit tests per :doc:`contributor_unit_test`
-   - Test both success and failure scenarios
-   - Ensure test coverage meets requirements
-   - Run existing test suite to check for regressions
+   - Currently being redeveloped and will be added back in soon.
 
 3. Integration Testing  / Samples