Skip to content

Commit 04eca4d

Browse files
radakamradakam
authored
Add instance_pools resource type to bundles (direct engine only) (#5934)
## Changes Add support for a new `instance_pools` resource type in DABs. This includes: - New `resources.InstancePool` config type (wrapping `compute.CreateInstancePool`) with permissions support. - Direct-engine resource implementation (`ResourceInstancePool`) covering create/read/update/delete, state remapping, and field behaviors (ignored remote changes, recreate-on-change, backend defaults). - Wiring into resource enumerations: supported resources, bundle permissions, presets (prefix + tags), target mode, bind support, and workspace URL patterns. - Regenerated JSON schema, validation (enum/required fields), reference schema, and annotations. - Testserver handlers for the `/api/2.0/instance-pools/*` endpoints. Instance pools are **only supported in direct deployment mode**. ## Why Users want to manage instance pools—pre-provisioned, idle cloud instances that reduce cluster start and auto-scaling times—declaratively alongside their other bundle resources, instead of provisioning them out-of-band. Closes #5641. ## Tests - New acceptance test (`acceptance/bundle/resources/instance_pools`) exercising validate, summary, deploy, update, and destroy, asserting the exact create/edit/delete API requests. - Added `instance_pools` coverage to invariant acceptance suites (no-drift, migrate, continue), with the resource excluded from Terraform-mode migration since it's direct-only. - Unit tests for permissions, presets/target-mode prefixing, bind support, and state-load round-tripping (create/modify/delete).
1 parent 82b62c4 commit 04eca4d

40 files changed

Lines changed: 1095 additions & 3 deletions

‎.nextchanges/bundles/instance-pools.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* Add support for the `instance_pools` resource type in Declarative Automation Bundles. Instance pools are only supported in direct deployment mode.

‎acceptance/bundle/invariant/configs/instance_pool.yml.tmpl‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
resources:
5+
instance_pools:
6+
foo:
7+
instance_pool_name: test-instance-pool-$UNIQUE_NAME
8+
node_type_id: $NODE_TYPE_ID
9+
permissions:
10+
- level: CAN_ATTACH_TO
11+
group_name: users

‎acceptance/bundle/invariant/continue_293/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/migrate/test.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ EnvMatrixExclude.no_catalog = ["INPUT_CONFIG=catalog.yml.tmpl"]
1111
EnvMatrixExclude.no_external_location = ["INPUT_CONFIG=external_location.yml.tmpl"]
1212
# Genie spaces are direct-only too; the terraform deploy that seeds the migration fails for them.
1313
EnvMatrixExclude.no_genie_space = ["INPUT_CONFIG=genie_space.yml.tmpl"]
14+
# Instance pools are direct-only; the terraform deploy that seeds the migration fails for them.
15+
EnvMatrixExclude.no_instance_pool = ["INPUT_CONFIG=instance_pool.yml.tmpl"]
1416

1517
# Cross-resource permission references (e.g. ${resources.jobs.job_b.permissions[0].level})
1618
# don't work in terraform mode: the terraform interpolator converts the path to

‎acceptance/bundle/invariant/no_drift/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/test.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ EnvMatrix.INPUT_CONFIG = [
4242
"experiment.yml.tmpl",
4343
"external_location.yml.tmpl",
4444
"genie_space.yml.tmpl",
45+
"instance_pool.yml.tmpl",
4546
"job.yml.tmpl",
4647
"job_pydabs_10_tasks.yml.tmpl",
4748
"job_pydabs_1000_tasks.yml.tmpl",

‎acceptance/bundle/refschema/out.fields.txt‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -761,6 +761,73 @@ resources.genie_spaces.*.permissions[*].group_name string ALL
761761
resources.genie_spaces.*.permissions[*].level iam.PermissionLevel ALL
762762
resources.genie_spaces.*.permissions[*].service_principal_name string ALL
763763
resources.genie_spaces.*.permissions[*].user_name string ALL
764+
resources.instance_pools.*.aws_attributes *compute.InstancePoolAwsAttributes ALL
765+
resources.instance_pools.*.aws_attributes.availability compute.InstancePoolAwsAttributesAvailability ALL
766+
resources.instance_pools.*.aws_attributes.instance_profile_arn string ALL
767+
resources.instance_pools.*.aws_attributes.spot_bid_price_percent int ALL
768+
resources.instance_pools.*.aws_attributes.zone_id string ALL
769+
resources.instance_pools.*.azure_attributes *compute.InstancePoolAzureAttributes ALL
770+
resources.instance_pools.*.azure_attributes.availability compute.InstancePoolAzureAttributesAvailability ALL
771+
resources.instance_pools.*.azure_attributes.capacity_reservation_group string ALL
772+
resources.instance_pools.*.azure_attributes.spot_bid_max_price float64 ALL
773+
resources.instance_pools.*.custom_tags map[string]string ALL
774+
resources.instance_pools.*.custom_tags.* string ALL
775+
resources.instance_pools.*.default_tags map[string]string REMOTE
776+
resources.instance_pools.*.default_tags.* string REMOTE
777+
resources.instance_pools.*.disk_spec *compute.DiskSpec ALL
778+
resources.instance_pools.*.disk_spec.disk_count int ALL
779+
resources.instance_pools.*.disk_spec.disk_iops int ALL
780+
resources.instance_pools.*.disk_spec.disk_size int ALL
781+
resources.instance_pools.*.disk_spec.disk_throughput int ALL
782+
resources.instance_pools.*.disk_spec.disk_type *compute.DiskType ALL
783+
resources.instance_pools.*.disk_spec.disk_type.azure_disk_volume_type compute.DiskTypeAzureDiskVolumeType ALL
784+
resources.instance_pools.*.disk_spec.disk_type.ebs_volume_type compute.DiskTypeEbsVolumeType ALL
785+
resources.instance_pools.*.enable_elastic_disk bool ALL
786+
resources.instance_pools.*.gcp_attributes *compute.InstancePoolGcpAttributes ALL
787+
resources.instance_pools.*.gcp_attributes.gcp_availability compute.GcpAvailability ALL
788+
resources.instance_pools.*.gcp_attributes.local_ssd_count int ALL
789+
resources.instance_pools.*.gcp_attributes.zone_id string ALL
790+
resources.instance_pools.*.id string INPUT
791+
resources.instance_pools.*.idle_instance_autotermination_minutes int ALL
792+
resources.instance_pools.*.instance_pool_id string REMOTE
793+
resources.instance_pools.*.instance_pool_name string ALL
794+
resources.instance_pools.*.lifecycle resources.Lifecycle INPUT
795+
resources.instance_pools.*.lifecycle.prevent_destroy bool INPUT
796+
resources.instance_pools.*.max_capacity int ALL
797+
resources.instance_pools.*.min_idle_instances int ALL
798+
resources.instance_pools.*.modified_status string INPUT
799+
resources.instance_pools.*.node_type_flexibility *compute.NodeTypeFlexibility ALL
800+
resources.instance_pools.*.node_type_flexibility.alternate_node_type_ids []string ALL
801+
resources.instance_pools.*.node_type_flexibility.alternate_node_type_ids[*] string ALL
802+
resources.instance_pools.*.node_type_id string ALL
803+
resources.instance_pools.*.preloaded_docker_images []compute.DockerImage ALL
804+
resources.instance_pools.*.preloaded_docker_images[*] compute.DockerImage ALL
805+
resources.instance_pools.*.preloaded_docker_images[*].basic_auth *compute.DockerBasicAuth ALL
806+
resources.instance_pools.*.preloaded_docker_images[*].basic_auth.password string ALL
807+
resources.instance_pools.*.preloaded_docker_images[*].basic_auth.username string ALL
808+
resources.instance_pools.*.preloaded_docker_images[*].url string ALL
809+
resources.instance_pools.*.preloaded_spark_versions []string ALL
810+
resources.instance_pools.*.preloaded_spark_versions[*] string ALL
811+
resources.instance_pools.*.remote_disk_throughput int ALL
812+
resources.instance_pools.*.state compute.InstancePoolState REMOTE
813+
resources.instance_pools.*.stats *compute.InstancePoolStats REMOTE
814+
resources.instance_pools.*.stats.idle_count int REMOTE
815+
resources.instance_pools.*.stats.pending_idle_count int REMOTE
816+
resources.instance_pools.*.stats.pending_used_count int REMOTE
817+
resources.instance_pools.*.stats.used_count int REMOTE
818+
resources.instance_pools.*.status *compute.InstancePoolStatus REMOTE
819+
resources.instance_pools.*.status.pending_instance_errors []compute.PendingInstanceError REMOTE
820+
resources.instance_pools.*.status.pending_instance_errors[*] compute.PendingInstanceError REMOTE
821+
resources.instance_pools.*.status.pending_instance_errors[*].instance_id string REMOTE
822+
resources.instance_pools.*.status.pending_instance_errors[*].message string REMOTE
823+
resources.instance_pools.*.total_initial_remote_disk_size int ALL
824+
resources.instance_pools.*.url string INPUT
825+
resources.instance_pools.*.permissions.object_id string ALL
826+
resources.instance_pools.*.permissions[*] dresources.StatePermission ALL
827+
resources.instance_pools.*.permissions[*].group_name string ALL
828+
resources.instance_pools.*.permissions[*].level iam.PermissionLevel ALL
829+
resources.instance_pools.*.permissions[*].service_principal_name string ALL
830+
resources.instance_pools.*.permissions[*].user_name string ALL
764831
resources.job_runs.*.dbt_commands []string ALL
765832
resources.job_runs.*.dbt_commands[*] string ALL
766833
resources.job_runs.*.id string INPUT

‎acceptance/bundle/resources/instance_pools/databricks.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
bundle:
2+
name: test_instance_pool
3+
4+
resources:
5+
instance_pools:
6+
test_instance_pool:
7+
instance_pool_name: my_instance_pool
8+
node_type_id: Standard_DS3_v2
9+
min_idle_instances: 0
10+
max_capacity: 5
11+
idle_instance_autotermination_minutes: 60

‎acceptance/bundle/resources/instance_pools/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/resources/instance_pools/output.txt‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
2+
>>> [CLI] bundle validate
3+
Name: test_instance_pool
4+
Target: default
5+
Workspace:
6+
User: [USERNAME]
7+
Path: /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default
8+
9+
Validation OK!
10+
11+
>>> [CLI] bundle validate -o json
12+
{
13+
"test_instance_pool": {
14+
"idle_instance_autotermination_minutes": 60,
15+
"instance_pool_name": "my_instance_pool",
16+
"max_capacity": 5,
17+
"min_idle_instances": 0,
18+
"node_type_id": "Standard_DS3_v2"
19+
}
20+
}
21+
22+
>>> [CLI] bundle summary
23+
Name: test_instance_pool
24+
Target: default
25+
Workspace:
26+
User: [USERNAME]
27+
Path: /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default
28+
Resources:
29+
Instance Pools:
30+
test_instance_pool:
31+
Name: my_instance_pool
32+
URL: (not deployed)
33+
34+
>>> [CLI] bundle deploy
35+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default/files...
36+
Deploying resources...
37+
Updating deployment state...
38+
Deployment complete!
39+
40+
=== Verify the create request
41+
>>> jq select(.method == "POST" and (.path | contains("/instance-pools/create"))) out.requests.txt
42+
{
43+
"method": "POST",
44+
"path": "/api/2.0/instance-pools/create",
45+
"body": {
46+
"idle_instance_autotermination_minutes": 60,
47+
"instance_pool_name": "my_instance_pool",
48+
"max_capacity": 5,
49+
"min_idle_instances": 0,
50+
"node_type_id": "Standard_DS3_v2"
51+
}
52+
}
53+
54+
>>> [CLI] bundle summary
55+
Name: test_instance_pool
56+
Target: default
57+
Workspace:
58+
User: [USERNAME]
59+
Path: /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default
60+
Resources:
61+
Instance Pools:
62+
test_instance_pool:
63+
Name: my_instance_pool
64+
URL: [DATABRICKS_URL]/compute/instance-pools/[UUID]?w=[NUMID]
65+
66+
=== Update the instance pool name
67+
>>> update_file.py databricks.yml my_instance_pool my_instance_pool_2
68+
69+
>>> [CLI] bundle deploy
70+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default/files...
71+
Deploying resources...
72+
Updating deployment state...
73+
Deployment complete!
74+
75+
=== Verify the update request
76+
>>> jq select(.method == "POST" and (.path | contains("/instance-pools/edit"))) out.requests.txt
77+
{
78+
"method": "POST",
79+
"path": "/api/2.0/instance-pools/edit",
80+
"body": {
81+
"idle_instance_autotermination_minutes": 60,
82+
"instance_pool_id": "[UUID]",
83+
"instance_pool_name": "my_instance_pool_2",
84+
"max_capacity": 5,
85+
"min_idle_instances": 0,
86+
"node_type_id": "Standard_DS3_v2"
87+
}
88+
}
89+
90+
>>> [CLI] bundle summary
91+
Name: test_instance_pool
92+
Target: default
93+
Workspace:
94+
User: [USERNAME]
95+
Path: /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default
96+
Resources:
97+
Instance Pools:
98+
test_instance_pool:
99+
Name: my_instance_pool_2
100+
URL: [DATABRICKS_URL]/compute/instance-pools/[UUID]?w=[NUMID]
101+
102+
=== Destroy the instance pool
103+
>>> [CLI] bundle destroy --auto-approve
104+
The following resources will be deleted:
105+
delete resources.instance_pools.test_instance_pool
106+
107+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default
108+
109+
Deleting files...
110+
Destroy complete!
111+
112+
=== Verify the destroy request
113+
>>> jq select(.method == "POST" and (.path | contains("/instance-pools/delete"))) out.requests.txt
114+
{
115+
"method": "POST",
116+
"path": "/api/2.0/instance-pools/delete",
117+
"body": {
118+
"instance_pool_id": "[UUID]"
119+
}
120+
}
121+
122+
>>> [CLI] bundle summary
123+
Name: test_instance_pool
124+
Target: default
125+
Workspace:
126+
User: [USERNAME]
127+
Path: /Workspace/Users/[USERNAME]/.bundle/test_instance_pool/default
128+
Resources:
129+
Instance Pools:
130+
test_instance_pool:
131+
Name: my_instance_pool_2
132+
URL: (not deployed)
133+
134+
>>> [CLI] bundle destroy --auto-approve
135+
No active deployment found to destroy!

0 commit comments

Comments
 (0)