Release/0.5.0 (#250)

Key Updates:
- Fully revamped Lakeview SAT dashboards
- New accounts console IP allow list check
- Brand new Secret Scanning feature for Databricks Notebooks
- Updated documentation and clarification instructions
- Bug fixes

Author: dleiva04

.gitignore

@@ -143,8 +143,6 @@ dmypy.json
 # Pyre type checker
 .pyre/
 
-# DABs Generated Template
-dabs/dabs_template/template/tmp
 
 **/.terraform*
 **/terraform.tfvars

README.md

@@ -1,4 +1,4 @@
-# SAT: Monitor the Security Health of your Workspaces
+# SAT: Monitor the Security Health of Databricks Workspaces
 
 The **Security Analysis Tool (SAT)** analyzes your Databricks account and workspace configurations, providing recommendations to help you follow Databricks' security best practices.
 

configs/security_best_practices.csv

@@ -51,7 +51,7 @@ id,check_id,category,check,evaluation_value,severity,recommendation,aws,azure,gc
 50,GOV-15,Governance,"Enable verbose audit logs (on Azure, diagnostic logs)",-1,Medium,"Enable verbose audit logs (on Azure, diagnostic logs)",1,1,1,1,0,Check workspace-conf for enableVerboseAuditLogs setting,curl -n -X GET 'https://<workspace_url>/api/2.0/preview/workspace-conf?keys=enableVerboseAuditLogs',https://docs.databricks.com/en/admin/account-settings/verbose-logs.html,https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/verbose-logs,https://docs.gcp.databricks.com/en/admin/account-settings/verbose-logs.html
 51,DP-9,Data Protection,FileStore endpoint for HTTPS file serving,-1,Medium,Review and disable FileStore endpoint in admin console workspace settings,1,1,1,1,0,Check workspace-conf for enableFileStoreEndpoint setting,curl -n -X GET 'https://<workspace_url>/api/2.0/preview/workspace-conf?keys=enableFileStoreEndpoint',https://docs.databricks.com/dbfs/filestore.html#filestore,https://learn.microsoft.com/en-us/azure/databricks/dbfs/filestore,https://docs.gcp.databricks.com/dbfs/filestore.html#filestore
 53,GOV-16,Governance,Workspace Unity Catalog metastore assignment,-1,Medium,Enable a workspace for Unity Catalog by assigning a Unity Catalog metastore,1,1,1,1,0,Check if current-metastore-assignment has the workspace assigned to metastore_id,curl --netrc -X GET \ https://<workspace_url>/api/2.1/unity-catalog/current-metastore-assignment,https://docs.databricks.com/data-governance/unity-catalog/enable-workspaces.html,https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/enable-workspaces,https://docs.gcp.databricks.com/data-governance/unity-catalog/enable-workspaces.html
-54,GOV-17,Governance,Limit the lifetime (expiration) of metastore Delta Sharing recipient token,-1,High,Set the lifetime of Delta Sharing recipient tokens,1,1,1,1,0,Check if delta_sharing_recipient_token_lifetime_in_seconds is set less than 90 days where delta_sharing_scope is INTERNAL_AND_EXTERNAL,curl --netrc -X GET \ https://<workspace_url>/api/2.1/unity-catalog/metastore_summary,https://docs.databricks.com/data-sharing/create-recipient.html#modify-the-recipient-token-lifetime,https://learn.microsoft.com/en-us/azure/databricks/data-sharing/create-recipient#modify-recipient-token-lifetime,https://docs.gcp.databricks.com/data-sharing/create-recipient.html#modify-the-recipient-token-lifetime
+54,GOV-17,Governance,Limit the lifetime (expiration) of metastore Delta Sharing recipient token,90,High,Set the lifetime of Delta Sharing recipient tokens,1,1,1,1,0,Check if delta_sharing_recipient_token_lifetime_in_seconds is set to less than configured days where delta_sharing_scope is INTERNAL_AND_EXTERNAL,curl --netrc -X GET \ https://<workspace_url>/api/2.1/unity-catalog/metastore_summary,https://docs.databricks.com/data-sharing/create-recipient.html#modify-the-recipient-token-lifetime,https://learn.microsoft.com/en-us/azure/databricks/data-sharing/create-recipient#modify-recipient-token-lifetime,https://docs.gcp.databricks.com/data-sharing/create-recipient.html#modify-the-recipient-token-lifetime
 55,GOV-18,Governance,Delta Sharing IP access lists,-1,Medium,Configure Delta Sharing IP access lists to restrict recipient access to trusted IP addresses,1,1,1,1,0,"Check if ip_access_list is present on share recipients for authentication_type ""TOKEN""",curl --netrc -X GET \ https://<workspace_url>/api/2.1/unity-catalog/recipients,https://docs.databricks.com/data-sharing/access-list.html#use-ip-access-lists-to-restrict-delta-sharing-recipient-access-open-sharing,https://learn.microsoft.com/en-gb/azure/databricks/data-sharing/access-list,https://docs.gcp.databricks.com/data-sharing/create-recipient.html#security-considerations-for-tokens
 56,GOV-19,Governance,Delta Sharing token expiration,-1,Medium,Establish a process for rotating credentials Delta sharing token,1,1,1,1,0,"Check if expiration_time on share recipients for tokens for share with authentication_type ""TOKEN""",curl --netrc -X GET \ https://<workspace_url>/api/2.1/unity-catalog/recipients,https://docs.databricks.com/data-sharing/create-recipient.html#security-considerations-for-tokens,https://learn.microsoft.com/en-us/azure/databricks/data-sharing/create-recipient#--security-considerations-for-tokens,https://docs.gcp.databricks.com/data-sharing/access-list.html#security-considerations-for-tokens
 57,GOV-20,Governance,Existence of Unity Sharing metastores,-1,Low,Create a Unity Catalog metastore,1,1,1,1,0,Check if securable_type = 'METASTORE' exists in metasores,curl --netrc -X GET \ https://<workspace_url>/api/2.1/unity-catalog/metastores,https://docs.databricks.com/data-governance/unity-catalog/create-metastore.html,https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/create-metastore,https://docs.gcp.databricks.com/data-governance/unity-catalog/create-metastore.html
@@ -74,3 +74,4 @@ id,check_id,category,check,evaluation_value,severity,recommendation,aws,azure,gc
 107,GOV-36,Governance,Automatic cluster update,-1,Medium,Ensure that all the clusters in a workspace are periodically updated to the latest host OS image and security updates,1,1,0,1,0,Get the automatic cluster update setting and check the value is set to true,curl --netrc -X GET \ https://<workspace_url>/api/2.0/settings/types/automatic_cluster_update/names/default \ | jq,https://docs.databricks.com/en/admin/clusters/automatic-cluster-update.html,https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/automatic-cluster-update,N/A
 108,INFO-39,Informational,Compliance security profile for the workspace,-1,Low,Validate and deploy on a platform that has put in place controls to meet the unique compliance needs of highly regulated industries,1,1,0,1,0,Check if compliance security profile for new workspaces is enabled,curl -n -X GET 'https:///<workspace_url>/api/2.0/settings/types/shield_csp_enablement_ws_db/names/default',https://docs.databricks.com/en/security/privacy/security-profile.html,https://learn.microsoft.com/en-us/azure/databricks/security/privacy/security-profile,N/A
 109,INFO-40,Informational,Enhanced security monitoring for the workspace,-1,Low,Validate and deploy on a platform that has put in place controls to meet the unique compliance needs of highly regulated industries,1,1,0,1,0,Check if compliance security profile for new workspaces is enabled,curl -n -X GET 'https:///<workspace_url>api/2.0/settings/types/shield_esm_enablement_ws_db/names/default',https://docs.databricks.com/en/security/privacy/enhanced-security-monitoring.html,https://learn.microsoft.com/en-us/azure/databricks/security/privacy/enhanced-security-monitoring,N/A
+110,NS-8,Network Security,IP access lists for account console access,-1,High,Configure IP access lists for the account console to allow users to connect to the account console UI and account-level REST APIs only through a set of approved IP addresses to reduce the risk of account takeover,1,1,1,1,0,Check if ip-access-lists are configured and enabled,curl --netrc -X GET \ https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/ip-access-lists,https://docs.databricks.com/security/network/ip-access-list.html#add-an-ip-access-list,https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/ip-access-list,https://docs.gcp.databricks.com/security/network/ip-access-list.html

configs/trufflehog_detectors.yaml

@@ -0,0 +1,54 @@
+# TruffleHog Custom Detectors Configuration
+# This file contains custom detector definitions for TruffleHog secret scanning
+# Used by the SAT (Security Analysis Tool) TruffleHog integration
+
+- name: DapiToken
+    description: "Databricks DAPI token detector"
+    keywords:
+      - dapi
+    regex:
+      id: (?i)\b(dapi[a-h0-9]{32})
+  
+- name: DkeaToken
+    description: "Databricks DKEA token detector"
+    keywords:
+      - dkea
+    regex:
+      id: (?i)\b(dkea[a-h0-9]{32})
+
+- name: DsapiToken
+    description: "Databricks Databricks Scoped API Token detector"
+    keywords:
+      - dsapi
+    regex:
+      id: (?i)\b(dsapi[a-h0-9]{32})
+
+- name: DoseToken
+    description: "Databricks DOSE token detector"
+    keywords:
+      - dose
+    regex:
+      id: (?i)\b(dose[a-h0-9]{32})
+
+# Configuration settings for TruffleHog scanning
+settings:
+  excluded_detectors:
+    - DatabricksToken  # Exclude built-in Databricks token detector to avoid false positives
+  
+  scan_options:
+    no_update: true
+    json_output: true
+    
+  rate_limiting:
+    api_sleep_seconds: 10  # Sleep between API calls to prevent rate limiting
+    
+  file_paths:
+    temp_config: "/tmp/trufflehog_config.yaml"
+    temp_notebooks: "/tmp/notebooks"
+    results_log: "/tmp/trufflehog_scan_results.json"
+    
+  search_settings:
+    page_size: 50
+    days_back: 1  # Number of days back to search for modified notebooks
+                  # Set to 0 to scan ALL notebooks (no time filter)
+                  # Can be overridden by TIME environment variable (epoch milliseconds)

dabs/dabs_template/template/tmp/resources/sat_driver_job.yml.tmpl

@@ -5,7 +5,7 @@ resources:
       tags:
         Application: SAT
       schedule:
-        quartz_cron_expression: "0 0 8 ? * Mon,Wed,Fri"
+        quartz_cron_expression: "0 0 7 ? * Mon,Wed,Fri"
         timezone_id: "America/New_York"
       tasks:
         - task_key: "sat_initializer"

dabs/dabs_template/template/tmp/resources/sat_secrets_scanner_job.yml.tmpl

@@ -0,0 +1,33 @@
+resources:
+  jobs:
+    sat_secrets:
+      name: "SAT Secrets Scanner"
+      tags:
+        Application: SAT
+      schedule:
+        quartz_cron_expression: "0 0 8 * * ?"
+        timezone_id: "America/New_York"
+
+      tasks:
+        - task_key: "secrets_scanner"
+        
+          {{- if eq .serverless false }}
+          job_cluster_key: job_cluster
+          
+          libraries:
+            - pypi:
+                package: dbl-sat-sdk
+          {{- end }}
+          notebook_task:
+            notebook_path: "../notebooks/security_analysis_secrets_scanner.py"
+      
+      {{- if eq .serverless false }}
+      job_clusters:
+        - job_cluster_key: job_cluster
+          new_cluster:
+            data_security_mode: SINGLE_USER
+            num_workers: 5
+            spark_version: {{.latest_lts}}
+            runtime_engine: "PHOTON"
+            node_type_id: {{.node_type}}
+      {{- end }}

dabs/sat/config.py

@@ -30,18 +30,10 @@ def form():
                 r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", x
             ),
         ),
-        Confirm(
-            name="enable_uc",
-            message="Use Unity Catalog?",
-            default=lambda x: uc_enabled(client),
-            ignore=lambda x: not uc_enabled(client),
-        ),
         List(
             name="catalog",
             message="Select catalog",
             choices=loading(get_catalogs, client=client),
-            ignore=lambda x: not x["enable_uc"],
-            default="hive_metastore",
         ),
         Text(
             name="security_analysis_schema",

dashboards/SAT_Dashboard_definition.json

@@ -33,6 +33,8 @@ SAT is typically run daily as an automated workflow within the customer’s envi
 
 * **Detailed failure explanations**, enabling admins to pinpoint, isolate, and remediate issues quickly.
 
+* **Secret scanning**, using TruffleHog to detect exposed credentials and sensitive data in workspace notebooks.
+
 ## SAT Insights
 
 Data from all configured workspaces is consolidated and presented through a single-pane SQL Dashboard, which serves as the primary consumption layer for SAT insights. All findings are organized into the following well-defined categories:

docs/sat/docs/functionality.mdx

@@ -66,6 +66,8 @@ terraform apply
 * This must be run successfully once. While it can be run multiple times, a single successful run is sufficient.
 2. Run "SAT Driver Notebook":
 * This notebook can be scheduled to run periodically (e.g., every Monday, Wednesday, and Friday).
+3. Run "SAT Secrets Scanner Notebook" (Optional):
+* This notebook scans workspace notebooks for exposed secrets using TruffleHog. Can be run on-demand or scheduled based on security requirements.
 
 ### Step 7: Access the SAT Dashboard
 1. Navigate to the <b>SQL > Dashboard </b> in the left menu from the Databricks workspace.