Skip to content

internal: Add support for 'sensitive' JSON tag for bundle config fields#5896

Merged
andrewnester merged 14 commits into
mainfrom
feat/sensitive-fields
Jul 22, 2026
Merged

internal: Add support for 'sensitive' JSON tag for bundle config fields#5896
andrewnester merged 14 commits into
mainfrom
feat/sensitive-fields

Conversation

@andrewnester

@andrewnester andrewnester commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Changes

Add support for 'sensitive' JSON tag for bundle config fields

Why

This allows us to mark certain fields (for example, secrets.value) as sensitive and let CLI handle masking it in command outputs.

Used in #5861

Tests

Added unit tests. Will be tests end to end with unit tests with UC secrets

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 49ea5f6

Run: 29924738401

Env 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 314 1057 12:59
💚​ aws windows 4 4 316 1055 6:15
🔄​ azure linux 1 4 4 313 1056 13:41
💚​ azure windows 4 4 316 1054 6:48
🟨​ gcp linux 3 1 4 313 1058 11:34
🟨​ gcp windows 3 1 4 315 1056 10:09
9 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED, 1 flaky
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestSyncIncrementalFileOverwritesFolder ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🟨​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
Top 8 slowest tests (at least 2 minutes):
duration env testname
3:31 gcp windows TestAccept
3:26 azure windows TestAccept
3:22 aws windows TestAccept
2:23 aws linux TestFilerRecursiveDelete/workspace_files
2:23 aws linux TestLock
2:21 azure windows TestFilerWorkspaceFilesExtensionsReadDir
2:11 aws linux TestFilerWorkspaceFilesExtensionsReadDir
2:09 gcp linux TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct

shreyas-goenka
shreyas-goenka reviewed Jul 13, 2026
View reviewed changes
Comment thread bundle/config/mask.go Outdated
Comment thread bundle/config/mask.go Outdated
Comment thread cmd/bundle/validate.go Outdated
pietern
pietern reviewed Jul 14, 2026
View reviewed changes
Comment thread libs/dyn/convert/struct_info.go
Comment thread libs/dyn/convert/struct_info.go Outdated
Comment thread libs/dyn/convert/struct_info.go
}
return si.Sensitive
}

@pietern pietern Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a few unit tests here in dyn/convert that demonstrates that this works in cases with embedded structs, anonymous structs, etc. Would also be good to include a user of these fields in the same package.

Comment thread cmd/bundle/validate.go Outdated
Comment thread bundle/config/mask_test.go Outdated
Comment thread bundle/config/validate/no_reference_to_sensitive_fields.go Outdated
@andrewnester
andrewnester requested a review from pietern July 14, 2026 13:29
@andrewnester andrewnester mentioned this pull request Jul 14, 2026
Open
pietern
pietern reviewed Jul 15, 2026
View reviewed changes
Comment thread libs/dyn/yamlsaver/saver.go Outdated
Comment thread libs/dyn/value.go Outdated
Comment thread libs/dyn/convert/from_typed.go
@andrewnester
andrewnester requested a review from pietern July 15, 2026 15:32
pietern
pietern approved these changes Jul 21, 2026
View reviewed changes
Comment thread libs/dyn/convert/from_typed.go Outdated
Comment thread libs/dyn/convert/from_typed_test.go
Comment thread libs/dyn/convert/normalize.go
case dyn.KindString:
// Preserve sensitive strings: NewValue would strip the secretString wrapper.
if src.IsSensitive() {
return src, nil

@pietern pietern Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means it is not possible to update the secret value in the typed tree.

If expected, please include this expectation in the comment.

@andrewnester
andrewnester enabled auto-merge July 22, 2026 12:24
@andrewnester
andrewnester added this pull request to the merge queue Jul 22, 2026
Merged via the queue into main with commit 40ee185 Jul 22, 2026
23 checks passed
@andrewnester
andrewnester deleted the feat/sensitive-fields branch July 22, 2026 15:08
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 40ee185

Run: 29932021784

Env ❌​FAIL 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
❌​ aws linux 21 3 3 2 972 879 186:24
❌​ aws windows 20 2 4 2 912 898 171:38
❌​ azure linux 4 2 4 4 2 880 915 156:06
❌​ azure windows 4 2 4 2 823 934 141:52
❌​ gcp linux 20 5 1 2 856 921 143:01
❌​ gcp windows 20 5 1 2 795 940 132:39
37 interesting tests: 27 FAIL, 6 KNOWN, 2 flaky, 2 SKIP
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/invariant/continue_293 ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/invariant/continue_293/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=instance_pool.yml.tmpl ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
🟨​ TestAccept/bundle/invariant/no_drift 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN= ✅​p ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1 ✅​p ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=instance_pool.yml.tmpl/READPLAN= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=instance_pool.yml.tmpl/READPLAN=1 ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/catalogs/basic ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/catalogs/basic/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/catalogs/with-schemas ✅​p ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/catalogs/with-schemas/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/clusters/lifecycle-started ❌​F ✅​p ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ✅​p ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/dashboards/change-name ❌​F ✅​p 🔄​f ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ✅​p 🔄​f ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/grants/catalogs ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/catalogs/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/schemas/change_privilege ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/schemas/change_privilege/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/schemas/duplicate_principals ❌​F ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/schemas/duplicate_principals/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/schemas/out_of_band_principal ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/schemas/out_of_band_principal/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/volumes ✅​p ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/volumes/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ❌​F ✅​p ✅​p ❌​F ❌​F
🔄​ TestAccept/bundle/resources/permissions/jobs/delete_one ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/permissions/jobs/delete_one/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🟨​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🟨​K 💚​R 💚​R 💚​R 💚​R 💚​R
❌​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ✅​p ✅​p ✅​p ✅​p ✅​p
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
❌​ TestAccept/bundle/resources/volumes/uppercase-name ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/resources/volumes/uppercase-name/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ✅​p ✅​p ❌​F ❌​F
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
Top 50 slowest tests (at least 2 minutes):
duration env testname
14:45 aws linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
13:43 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
13:34 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
12:58 gcp linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
12:53 aws windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
12:47 gcp windows TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct
12:37 gcp windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
10:59 gcp windows TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
10:58 aws windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
10:58 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
10:45 aws windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
10:27 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
9:43 aws linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
9:38 azure linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
9:29 gcp linux TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
9:23 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
9:11 aws linux TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct
9:07 gcp linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
9:05 gcp linux TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
8:52 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
8:51 azure windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:46 azure linux TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:44 azure windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:39 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
8:35 aws windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:34 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct
8:16 azure linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
8:15 gcp windows TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
8:13 azure linux TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:04 gcp linux TestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct
8:04 aws linux TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
7:52 gcp windows TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
7:24 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
7:18 azure linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
7:11 aws linux TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
7:08 aws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
6:40 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:37 aws windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:23 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
6:22 aws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
6:15 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:54 gcp linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:52 azure windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
5:52 azure linux TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:45 aws linux TestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct
5:39 azure windows TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
5:19 aws windows TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=schema.yml.tmpl
5:14 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:12 gcp windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:04 aws windows TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pietern pietern pietern approved these changes
@denik denik Awaiting requested review from denik
@janniklasrose janniklasrose Awaiting requested review from janniklasrose
@shreyas-goenka shreyas-goenka Awaiting requested review from shreyas-goenka

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andrewnester @eng-dev-ecosystem-bot @pietern @shreyas-goenka