Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions knowledge_base/target_includes/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.databricks
*.jar
*.class
105 changes: 105 additions & 0 deletions knowledge_base/target_includes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Target Includes Example

This example demonstrates the concept of using `target_includes` (or similar include mechanisms) in Databricks Asset Bundles to organize job configurations across different environments without duplication.

## Overview

This example implements the use case described in [GitHub Issue #2878](https://github.com/databricks/cli/issues/2878), which requests the ability to include specific resource files based on target configurations.
Comment thread
andrewnester marked this conversation as resolved.
Outdated

## Directory Structure

```
target_includes/
├── databricks.yml # Main bundle configuration with 3 targets
├── resources/
│ ├── set1/ # Jobs for dev and staging environments
│ │ ├── job_1.yml
│ │ └── job_2.yml
│ └── set2/ # Jobs for staging and prod environments
│ ├── job_1.yml
│ └── job_2.yml
└── README.md
```

## Target Configuration

The bundle defines three targets:

1. **dev**: Includes only `resources/set1/*.yml`
- Contains: set1-job-1, set1-job-2
- Environment: development

2. **staging**: Includes both `resources/set1/*.yml` and `resources/set2/*.yml`
- Contains: set1-job-1, set1-job-2, set2-job-1, set2-job-2
- Environment: staging

3. **prod**: Includes only `resources/set2/*.yml`
- Contains: set2-job-1, set2-job-2
- Environment: production
Comment thread
andrewnester marked this conversation as resolved.
Outdated

## Usage

```bash
# Summary of the bundle resources for dev
databricks bundle summary -p u2m -t dev

Name: target-includes-example
Comment thread
andrewnester marked this conversation as resolved.
Target: dev
Workspace:
User: ***
Path: ***
Resources:
Jobs:
set1-job-1:
Name: Set1 Job 1 - foo
URL: (not deployed)
set1-job-2:
Name: Set1 Job 2 - foo
URL: (not deployed)

# Summary of the bundle resources for staging
databricks bundle summary -p u2m -t staging

Name: target-includes-example
Target: staging
Workspace:
User: ***
Path: ***
Resources:
Jobs:
set1-job-1:
Name: Set1 Job 1 - bar
URL: (not deployed)
set1-job-2:
Name: Set1 Job 2 - bar
URL: (not deployed)
set2-job-1:
Name: Set2 Job 1 - bar
URL: (not deployed)
set2-job-2:
Name: Set2 Job 2 - bar
URL: (not deployed)

# Summary of the bundle resources for prod
databricks bundle summary -p u2m -t prod

Name: target-includes-example
Target: prod
Workspace:
User: ***
Path: ***
Resources:
Jobs:
set2-job-1:
Name: Set2 Job 1 - baz
URL: (not deployed)
set2-job-2:
Name: Set2 Job 2 - baz
URL: (not deployed)
```

## Notes

There are some key aspects in this implementation
1. In `databricks.yml` file we include (see `include` section) all configuration files for all targets. This does not impact which resources will be deployed for which target.
2. For each job in corresponding configuration file like `resources/set1/job_1.yml` we define in which targets this job should be deployed. We use YAML anchors to avoid duplications between targets.
Comment thread
andrewnester marked this conversation as resolved.
Outdated
24 changes: 24 additions & 0 deletions knowledge_base/target_includes/databricks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
bundle:
name: target-includes-example

include:
- resources/set1/*.yml
- resources/set2/*.yml

variables:
name_suffix:
description: "Target specific suffix for the job name"

targets:
dev:
default: true
variables:
name_suffix: foo

staging:
variables:
name_suffix: bar

prod:
variables:
name_suffix: baz
22 changes: 22 additions & 0 deletions knowledge_base/target_includes/resources/set1/job_1.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Job 1 for set1 - used in dev and staging targets
# Using YAML anchors to avoid duplication

job-config: &job-config
set1-job-1:
name: "Set1 Job 1 - ${var.environment}"
Comment thread
andrewnester marked this conversation as resolved.
tasks:
- task_key: "process_data"
notebook_task:
notebook_path: ../../src/notebook.py
max_concurrent_runs: 1
timeout_seconds: 3600

targets:
dev:
resources:
jobs:
<<: *job-config
staging:
resources:
jobs:
<<: *job-config
27 changes: 27 additions & 0 deletions knowledge_base/target_includes/resources/set1/job_2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Job 2 for set1 - used in dev and staging targets
# Using YAML anchors to avoid duplication

job-config: &job-config
set1-job-2:
name: "Set1 Job 2 - ${bundle.target}"
tasks:
- task_key: "process_data"
notebook_task:
notebook_path: ../../src/notebook.py
max_concurrent_runs: 1
timeout_seconds: 3600
trigger:
type: "cron"
cron:
quartz_cron_expression: "0 0 12 * * ?"
timezone_id: "UTC"

targets:
dev:
resources:
jobs:
<<: *job-config
staging:
resources:
jobs:
<<: *job-config
27 changes: 27 additions & 0 deletions knowledge_base/target_includes/resources/set2/job_1.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Job 1 for set2 - used in staging and prod targets
# Using YAML anchors to avoid duplication

job-config: &job-config
set2-job-1:
name: "Set2 Job 1 - ${bundle.target}"
tasks:
- task_key: "process_data"
notebook_task:
notebook_path: ../../src/notebook.py
max_concurrent_runs: 1
timeout_seconds: 3600
trigger:
type: "cron"
cron:
quartz_cron_expression: "0 0 12 * * ?"
timezone_id: "UTC"

targets:
staging:
resources:
jobs:
<<: *job-config
prod:
resources:
jobs:
<<: *job-config
27 changes: 27 additions & 0 deletions knowledge_base/target_includes/resources/set2/job_2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Job 2 for set2 - used in staging and prod targets
# Using YAML anchors to avoid duplication

job-config: &job-config
set2-job-2:
name: "Set2 Job 2 - ${bundle.target}"
tasks:
- task_key: "process_data"
notebook_task:
notebook_path: ../../src/notebook.py
max_concurrent_runs: 1
timeout_seconds: 3600
trigger:
type: "cron"
cron:
quartz_cron_expression: "0 0 12 * * ?"
timezone_id: "UTC"

targets:
staging:
resources:
jobs:
<<: *job-config
prod:
resources:
jobs:
<<: *job-config
3 changes: 3 additions & 0 deletions knowledge_base/target_includes/src/notebook.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Databricks notebook source

print("Hello From Notebook!")