You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The framework previously used a single `extensions/` tree for both importable Python
12
+
modules and lifecycle scripts. This caused:
13
+
14
+
- No separation between "code that gets installed on the cluster" and "code that spec
15
+
references by module path".
16
+
- No clear, fork-safe area for customer customisation.
17
+
- Ambiguity between `extensions/` (sys.path) and `extensions/libraries/` (a variant
18
+
that was never released to `main`).
19
+
20
+
OSS consumers also expected a predictable, documented `src/` layout rather than a
21
+
single catch-all directory.
22
+
23
+
## Decision
24
+
25
+
Introduce a canonical `src/` layout for pipeline bundles:
26
+
27
+
| Path | Role |
28
+
|------|------|
29
+
|`src/libraries/`|**Optional** location for wheel files bundled with the pipeline. The directory is also added to `sys.path` for loose `.py` / packages (secondary role). |
30
+
|`src/python/`| All customer Python referenced by Data Flow Specs (`pythonModule`, `pythonTransform.module`). Added to `sys.path`. |
31
+
|`src/init/pre/`| Lifecycle scripts run **before** SDP declarations inside `initialize_pipeline()`. |
32
+
|`src/init/post/`| Lifecycle scripts run **after** SDP declarations. |
33
+
34
+
The **framework bundle** (`framework.sourcePath`) follows a different layout — custom
Copy file name to clipboardExpand all lines: docs/source/build_pipeline_bundle_steps.rst
+46-5Lines changed: 46 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -55,8 +55,13 @@ A new Pipeline Bundle can be created using the following methods.
55
55
│ └── my_first_pipeline.yml
56
56
├── scratch/
57
57
├── src/
58
-
│ ├── dataflows
59
-
│ └── pipeline_configs
58
+
│ ├── dataflows/
59
+
│ ├── init/
60
+
│ │ ├── pre/
61
+
│ │ └── post/
62
+
│ ├── libraries/
63
+
│ ├── pipeline_configs/
64
+
│ └── python/
60
65
├── databricks.yml
61
66
└── README.md
62
67
@@ -78,8 +83,8 @@ A new Pipeline Bundle can be created using the following methods.
78
83
79
84
You can always copy an existing Pipeline Bundle to use as a starting point for a new Pipeline Bundle. If doing this bear in mind that you may need to:
80
85
81
-
- Reset the targets and parameter in the ``databricks.yml`` file
82
-
- Clean out the following folders: ``resources``, ``src/dataflows`` and ``src/pipeline_configs``
86
+
- Reset the targets and parameters in the ``databricks.yml`` file
87
+
- Clear out the following folders: ``resources/``, ``src/dataflows/``, ``src/pipeline_configs/``, ``src/python/``, ``src/libraries/``, ``src/init/pre/``, and ``src/init/post/``
83
88
84
89
2. Update the ``databricks.yml`` File
85
90
-------------------------------------
@@ -173,7 +178,21 @@ Iterate over the following steps to create each individual Data Flow:
173
178
174
179
If necessary add any substitutions required for your Data Flow to the substitutions file.
175
180
176
-
3. **Build the Data Flow Spec:**
181
+
3. **Add Init Scripts** *(optional)*:
182
+
183
+
If your pipeline requires Spark configuration, event hook registration, or any one-time setup that must run outside of Data Flow logic, add ``.py`` scripts to:
184
+
185
+
- ``src/init/pre/`` — run **before** SDP dataflow declarations
186
+
- ``src/init/post/`` — run **after** SDP dataflow declarations
187
+
188
+
Scripts are executed in sorted filename order. Files whose names begin with ``_`` are skipped. Use a numeric prefix (e.g. ``01_setup.py``) to control execution order.
189
+
190
+
Refer to the :doc:`feature_python_extensions` section for full details.
191
+
192
+
.. note::
193
+
This step is optional and only required when pipeline-level lifecycle setup is needed.
194
+
195
+
4. **Build the Data Flow Spec:**
177
196
178
197
a. Create a sub-directory per you selected bundle structure:
179
198
@@ -207,6 +226,17 @@ Iterate over the following steps to create each individual Data Flow:
207
226
208
227
Refer to the :doc:`feature_data_quality_expectations` section for guidance on how to create an expectations file.
209
228
229
+
f. **Add Pipeline Logic Modules** *(optional)*:
230
+
231
+
If your Data Flow Spec references custom Python code — e.g. ``pythonModule``, ``pythonTransform.module``, or a custom sink — add the corresponding ``.py`` modules or packages to ``src/python/``.
232
+
233
+
The framework adds ``src/python/`` to ``sys.path`` at pipeline initialisation so spec strings such as ``"module": "my_transforms"`` resolve without any extra configuration.
234
+
235
+
Refer to the :doc:`feature_python_extensions` section for details on flat vs. package layout options.
236
+
237
+
.. note::
238
+
This step is optional and only required when your Data Flow Spec references a custom Python module.
239
+
210
240
7. Create your Pipeline Definitions
211
241
-----------------------------------
212
242
@@ -303,3 +333,14 @@ To create a single Pipeline definition, follow these steps:
303
333
pipeline.dataFlowIdFilter: <value_flow_id>
304
334
pipeline.flowGroupIdFilter: <value_flow_group_id>
305
335
pipeline.fileFilter: <value_file_path>
336
+
337
+
4. **Add Cluster Libraries** *(optional)*:
338
+
339
+
If your pipeline requires third-party or in-house Python packages installed on the cluster, add them via ``environment.dependencies`` in your pipeline resource YAML. Refer to the :doc:`feature_python_extensions` section for full details. Sources include:
- **Bundle wheel** (wheel stored with the pipeline code) — ``- /Workspace/${workspace.file_path}/src/libraries/my_package.whl``
345
+
346
+
For the last case, place the ``.whl`` file in ``src/libraries/``. You may also place loose ``.py`` files or packages there if they need to be on ``sys.path`` without being spec-referenced (e.g. a shared utility imported indirectly).
│ ├── pipeline_configs/ # Global and env-specific pipeline config (required)
74
+
│ └── python/ # Spec-referenced Python modules and packages (optional)
70
75
├── databricks.yml
71
76
└── README.md
72
77
73
78
.. note::
74
79
75
80
Refer to the :doc:`concepts` section for more details on the different components of a Pipeline Bundle.
76
81
82
+
The ``src/`` directories serve distinct purposes:
83
+
84
+
.. list-table::
85
+
:header-rows: 1
86
+
:widths: 25 75
87
+
88
+
* - Directory
89
+
- Purpose
90
+
* - ``src/dataflows/``
91
+
- All Data Flow Spec files. The framework reads every spec file recursively regardless
92
+
of sub-folder structure. See below for organisation options.
93
+
* - ``src/init/pre/`` and ``src/init/post/``
94
+
- **Optional.** Lifecycle ``.py`` scripts executed before and after SDP declarations
95
+
inside ``initialize_pipeline()``. Run in sorted filename order; files starting
96
+
with ``_`` are skipped.
97
+
* - ``src/libraries/``
98
+
- **Optional.** Wheels bundled with the pipeline and referenced in the DAB
99
+
``libraries:`` YAML, plus any loose ``.py`` modules that need to be on
100
+
``sys.path`` without being spec-referenced. Libraries may equally be sourced from
101
+
PyPI, UC Volumes, or artifact repositories — ``src/libraries/`` is only needed
102
+
when the wheel travels with the bundle.
103
+
* - ``src/pipeline_configs/``
104
+
- Global and environment-specific pipeline configuration (``global.json``,
105
+
substitutions, secrets).
106
+
* - ``src/python/``
107
+
- **Optional.** All customer Python referenced by Data Flow Specs —
108
+
``pythonModule``, ``pythonTransform.module``, and custom sinks. Added to
109
+
``sys.path`` at pipeline initialisation.
110
+
111
+
.. seealso::
112
+
113
+
:doc:`feature_python_extensions` — full reference for ``src/libraries/``,
114
+
``src/python/``, ``src/init/``, and ``src/local/config/``, including examples,
115
+
deprecation notices, and the cluster library installation options.
116
+
77
117
It is the structure of the ``src/dataflows`` directory that is flexible and can be organised in the way that best suits your standards and ways of working. The Framework will:
78
118
79
119
* Read all the Data Flow Spec files under the ``src/dataflows`` directory, regardless of the folder structure. Filtering of the Dataflows is done when defining your Pipeline and is discussed in the :doc:`build_pipeline_bundle_steps` section.
This separation allows the framework to maintain its core dependencies independently while enabling pipeline developers to add custom packages for their specific use cases.
21
21
22
+
.. note::
23
+
24
+
This page covers installing Python packages onto the cluster via the DAB **pipeline environment** (``pip install`` style). If you need to install a wheel that travels with your pipeline code or add loose ``.py`` files to ``sys.path``, see :doc:`feature_python_extensions` for the ``src/libraries/`` approach.
25
+
22
26
.. important::
23
27
24
28
Databricks recommends using the **pipeline environment settings** to manage Python dependencies.
0 commit comments