Skip to content

Commit af84ba2

Browse files
andrewnesterandrewnester
committed
Added an example how to deploy resources only for specific targets
1 parent 51b8714 commit af84ba2

8 files changed

Lines changed: 252 additions & 0 deletions

File tree

‎knowledge_base/target_includes/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
2+
.databricks
3+
*.jar
4+
*.class

‎knowledge_base/target_includes/README.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Target Includes Example
2+
3+
This example demonstrates the concept of using `target_includes` (or similar include mechanisms) in Databricks Asset Bundles to organize job configurations across different environments without duplication.
4+
5+
## Overview
6+
7+
This example implements the use case described in [GitHub Issue #2878](https://github.com/databricks/cli/issues/2878), which requests the ability to include specific resource files based on target configurations.
8+
9+
## Directory Structure
10+
11+
```
12+
target_includes/
13+
├── databricks.yml # Main bundle configuration with 3 targets
14+
├── resources/
15+
│ ├── set1/ # Jobs for dev and staging environments
16+
│ │ ├── job_1.yml
17+
│ │ └── job_2.yml
18+
│ └── set2/ # Jobs for staging and prod environments
19+
│ ├── job_1.yml
20+
│ └── job_2.yml
21+
└── README.md
22+
```
23+
24+
## Target Configuration
25+
26+
The bundle defines three targets:
27+
28+
1. **dev**: Includes only `resources/set1/*.yml`
29+
- Contains: set1-job-1, set1-job-2
30+
- Environment: development
31+
32+
2. **staging**: Includes both `resources/set1/*.yml` and `resources/set2/*.yml`
33+
- Contains: set1-job-1, set1-job-2, set2-job-1, set2-job-2
34+
- Environment: staging
35+
36+
3. **prod**: Includes only `resources/set2/*.yml`
37+
- Contains: set2-job-1, set2-job-2
38+
- Environment: production
39+
40+
## Usage
41+
42+
```bash
43+
# Summary of the bundle resources for dev
44+
databricks bundle summary -p u2m -t dev
45+
46+
Name: target-includes-example
47+
Target: dev
48+
Workspace:
49+
User: ***
50+
Path: ***
51+
Resources:
52+
Jobs:
53+
set1-job-1:
54+
Name: Set1 Job 1 - dev
55+
URL: (not deployed)
56+
set1-job-2:
57+
Name: Set1 Job 2 - dev
58+
URL: (not deployed)
59+
60+
# Summary of the bundle resources for staging
61+
databricks bundle summary -p u2m -t staging
62+
63+
Name: target-includes-example
64+
Target: staging
65+
Workspace:
66+
User: ***
67+
Path: ***
68+
Resources:
69+
Jobs:
70+
set1-job-1:
71+
Name: Set1 Job 1 - staging
72+
URL: (not deployed)
73+
set1-job-2:
74+
Name: Set1 Job 2 - staging
75+
URL: (not deployed)
76+
set2-job-1:
77+
Name: Set2 Job 1 - staging
78+
URL: (not deployed)
79+
set2-job-2:
80+
Name: Set2 Job 2 - staging
81+
URL: (not deployed)
82+
83+
# Summary of the bundle resources for prod
84+
databricks bundle summary -p u2m -t prod
85+
86+
Name: target-includes-example
87+
Target: prod
88+
Workspace:
89+
User: ***
90+
Path: ***
91+
Resources:
92+
Jobs:
93+
set2-job-1:
94+
Name: Set2 Job 1 - prod
95+
URL: (not deployed)
96+
set2-job-2:
97+
Name: Set2 Job 2 - prod
98+
URL: (not deployed)
99+
```
100+
101+
## Notes
102+
103+
There are some key aspects in this implementation
104+
1. In `databricks.yml` file we include (see `include` section) all configuration files for all targets. This does not impact which resources will be deployed for which target.
105+
2. For each job in corresponding configuration file like `resources/set1/job_1.yml` we define in which targets this job should be deployed. We use YAML anchors to avoid duplications between targets.

‎knowledge_base/target_includes/databricks.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
bundle:
2+
name: target-includes-example
3+
4+
include:
5+
- resources/set1/*.yml
6+
- resources/set2/*.yml
7+
8+
variables:
9+
environment:
10+
description: "Define the environment where the job will run"
11+
12+
targets:
13+
dev:
14+
default: true
15+
variables:
16+
environment: dev
17+
18+
staging:
19+
variables:
20+
environment: staging
21+
22+
prod:
23+
variables:
24+
environment: prod

‎knowledge_base/target_includes/resources/set1/job_1.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Job 1 for set1 - used in dev and staging targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set1-job-1:
6+
name: "Set1 Job 1 - ${var.environment}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
new_cluster:
12+
spark_version: "13.3.x-scala2.12"
13+
node_type_id: "Standard_DS3_v2"
14+
num_workers: 1
15+
max_concurrent_runs: 1
16+
timeout_seconds: 3600
17+
schedule:
18+
quartz_cron_expression: "0 0 12 * * ?"
19+
timezone_id: "UTC"
20+
21+
targets:
22+
dev:
23+
resources:
24+
jobs:
25+
<<: *job-config
26+
staging:
27+
resources:
28+
jobs:
29+
<<: *job-config

‎knowledge_base/target_includes/resources/set1/job_2.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Job 2 for set1 - used in dev and staging targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set1-job-2:
6+
name: "Set1 Job 2 - ${var.environment}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
new_cluster:
12+
spark_version: "13.3.x-scala2.12"
13+
node_type_id: "Standard_DS3_v2"
14+
num_workers: 1
15+
max_concurrent_runs: 1
16+
timeout_seconds: 3600
17+
schedule:
18+
quartz_cron_expression: "0 0 12 * * ?"
19+
timezone_id: "UTC"
20+
21+
targets:
22+
dev:
23+
resources:
24+
jobs:
25+
<<: *job-config
26+
staging:
27+
resources:
28+
jobs:
29+
<<: *job-config

‎knowledge_base/target_includes/resources/set2/job_1.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Job 1 for set2 - used in staging and prod targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set2-job-1:
6+
name: "Set2 Job 1 - ${var.environment}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
new_cluster:
12+
spark_version: "13.3.x-scala2.12"
13+
node_type_id: "Standard_DS3_v2"
14+
num_workers: 1
15+
max_concurrent_runs: 1
16+
timeout_seconds: 3600
17+
schedule:
18+
quartz_cron_expression: "0 0 12 * * ?"
19+
timezone_id: "UTC"
20+
21+
targets:
22+
staging:
23+
resources:
24+
jobs:
25+
<<: *job-config
26+
prod:
27+
resources:
28+
jobs:
29+
<<: *job-config

‎knowledge_base/target_includes/resources/set2/job_2.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Job 2 for set2 - used in staging and prod targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set2-job-2:
6+
name: "Set2 Job 2 - ${var.environment}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
new_cluster:
12+
spark_version: "13.3.x-scala2.12"
13+
node_type_id: "Standard_DS3_v2"
14+
num_workers: 1
15+
max_concurrent_runs: 1
16+
timeout_seconds: 3600
17+
schedule:
18+
quartz_cron_expression: "0 0 12 * * ?"
19+
timezone_id: "UTC"
20+
21+
targets:
22+
staging:
23+
resources:
24+
jobs:
25+
<<: *job-config
26+
prod:
27+
resources:
28+
jobs:
29+
<<: *job-config

‎knowledge_base/target_includes/src/notebook.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Databricks notebook source
2+
3+
print("Hello From Notebook!")

0 commit comments

Comments
 (0)