Release/0.3.3 (#188)

* serverless compatability check

* update logger methods for serverless

* changes to support serverless - 1

* changes to sat driver to make it serverless compataible

* import time from driver

* fix logging utils to create file , fix to dbsql client to parse the correct response

* fixes related to logger

* fix logger regression issues

* bug fixes on the left over file -part 1

* bug fixes and code refactor

* drop staging database before each run

* move drop intermediate schema to initialize

* Update TERRAFORM_Azure.md

updated Azure terraform instruction with SP requirements

* Added dashboard and corrected the name in the setup notebook

* Update TERRAFORM_Azure.md

more readable format.

* updated sdk ,removed local references and minot refactoring

* Bug Fixes using sdk

* logfile changes

* removed stale secrets and moved to variables

* utility to dynamically infer json schema from rest response

* Terraform changes to support sat run on serverless

* New dashboard and code to create

* fixed edge case for PAT expiry when only one token meets the condition.

* GCP tests completed

* Removed internal workspace link in dashboard

* Fixed removed changes

* Removed logger

* Add serverless configuration options for DABs

* Refactor job cluster configuration for serverless compatibility

* Added detailed instructions to add SP to workspaces

* Fixed serverless compute with TF

* added run on serverless to the provider

* update serverless instructions

---------

Co-authored-by: sudharshanraja-db <sudharshanraja-db>
Co-authored-by: Chris Moon <chiwoong.moon@gmail.com>
Co-authored-by: andres-zuniga <andres.zuniga@databricks.com>
Co-authored-by: ramdas.murali <ramdas.murali@databricks.com>
Co-authored-by: arunpamulapati <arunpamulapati>
Co-authored-by: David Leiva <david.leiva@databricks.com>

Author: 5 people

dabs/dabs_template/databricks_template_schema.json

@@ -20,6 +20,10 @@
         "node_type": {
             "type": "string",
             "description": "Node Type"
+        },
+        "serverless": {
+            "type": "boolean",
+            "description": "Serverless"
         }
     },
     "success_message": ""

dabs/dabs_template/template/tmp/resources/sat_driver_job.yml.tmpl

@@ -7,13 +7,16 @@ resources:
         timezone_id: "America/New_York"
       tasks:
         - task_key: "sat_initializer"
+          {{- if eq .serverless false }}
           job_cluster_key: job_cluster
           libraries:
             - pypi:
                 package: dbl-sat-sdk
+          {{- end }}
           notebook_task:
             notebook_path: "../notebooks/security_analysis_driver.py"
 
+      {{- if eq .serverless false }}
       job_clusters:
         - job_cluster_key: job_cluster
           new_cluster:
@@ -26,3 +29,4 @@ resources:
             gcp_attributes:
               google_service_account: {{.google_service_account}}
             {{- end }}
+      {{- end }}

dabs/dabs_template/template/tmp/resources/sat_initiliazer_job.yml.tmpl

@@ -5,13 +5,18 @@ resources:
 
       tasks:
         - task_key: "sat_initializer"
+        
+          {{- if eq .serverless false }}
           job_cluster_key: job_cluster
+          
           libraries:
             - pypi:
                 package: dbl-sat-sdk
+          {{- end }}
           notebook_task:
             notebook_path: "../notebooks/security_analysis_initializer.py"
-
+      
+      {{- if eq .serverless false }}
       job_clusters:
         - job_cluster_key: job_cluster
           new_cluster:
@@ -23,4 +28,5 @@ resources:
             {{- if eq .cloud "gcp" }}
             gcp_attributes:
               google_service_account: {{.google_service_account}}
-            {{- end }}
+            {{- end }}
+      {{- end }}

dabs/main.py

@@ -25,6 +25,7 @@ def install(client: WorkspaceClient, answers: dict, profile: str):
             photon_driver_capable=True,
             photon_worker_capable=True,
         ),
+        "serverless": answers.get("enable_serverless", False),
     }
 
     config_file = "tmp_config.json"

dabs/sat/config.py

@@ -48,6 +48,11 @@ def form():
             message="Schema name for SAT",
             default="security_analysis",
         ),
+        Confirm(
+            name="enable_serverless",
+            message="Run on serverless?",
+            default=True,
+        ),
         List(
             name="warehouse",
             message="Select warehouse",

dashboards/SAT_Dashboard_definition.json

@@ -1,3 +1,4 @@
+Note: The manual setup is out dates and is deprecated. Please use [Terraform or Standard](https://github.com/databricks-industry-solutions/security-analysis-tool/tree/main?tab=readme-ov-file#security-analysis-tool-sat) setup. 
 ## Checklist to prepare for SAT setup
 
 **Note**: SAT creates a new **security_analysis** databses and Delta tables. 

docs/deprecated_old_setup.md

@@ -2,7 +2,7 @@
 
 > **SAT v0.2.0 or higher** brings full support for Unity Catalog. Now you can pick your catalog instead of hive_metastore. Plus, you get to choose your own schema name.
 
-> **Note**: SAT requires at least one SAT set up in a workspace per **account** in AWS or GCP and at least one SAT set up in a workspace per Azure **subscription**.  
+> **Note**: SAT requires at least one SAT set up in a workspace per **account** in AWS or GCP and at least one SAT set up in a workspace per Azure **subscription**. SAT uses the Service Principal configured with SAT to access and analyze configurations by calling account and workspace APIs. Please make sure to add the Service Principal to the workspaces you wish to analyze as instructed in the setup guides below. SAT running on serverless or classic compute can't analyze account and destination workspaces with IP ACLs configured unless the ACLs are updated to allow access from the SAT workspace. SAT running on Serverless compute can't access other workspaces if the SAT workspace has serverless egress controls configured. In these scenarios, a separate SAT can be configured to analyze the respective target workspace by setting up SAT in that workspace.
 
 >  Please make sure to review the SAT report with your business stakeholders, administrators, security team and auditors about SAT report and assess your organizational security requirements before making any security improvements bases on the report, not all deviations required to be mitigated. Some of the recommendations may have cost implications, some of the security features recommended may have dependency feature limitations, please thoroughly review individual feature documentation before making changes to your security configurations.
 

docs/setup.md

@@ -32,7 +32,7 @@ The first step is to create a Service Principal in Databricks. This will allow S
 
 ![AWS_SP_Workspace](../images/gcp_ws.png)
 
-> The Service Principle requires an [Accounts Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-account-admin-roles-to-a-service-principal), [Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://docs.gcp.databricks.com/en/data-governance/unity-catalog/manage-privileges/admin-privileges.html#who-has-metastore-admin-privileges) is required to analyze many of the APIs
+> The Service Principle requires an [Accounts Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-account-admin-roles-to-a-service-principal), [Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://docs.gcp.databricks.com/en/data-governance/unity-catalog/manage-privileges/admin-privileges.html#who-has-metastore-admin-privileges) is required to analyze many of the APIs. Please [add this Service Princple](https://docs.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) to **each workspace** so that SAT can access the APIs for analysis.  
 
 ## Installation
 

docs/setup/aws.md

@@ -63,7 +63,7 @@ After creating the App Registration and client secret, you will need to add the
 
 See the [Databricks documentation](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#--databricks-and-microsoft-entra-id-formerly-azure-active-directory-service-principals) for more information on adding service principals.
 
-> The Service Principle requires an [Accounts Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#--assign-account-admin-roles-to-a-service-principal), [Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/admin-privileges#who-has-metastore-admin-privileges) is required to analyze many of the APIs.
+> The Service Principle requires an [Accounts Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#--assign-account-admin-roles-to-a-service-principal), [Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/admin-privileges#who-has-metastore-admin-privileges) is required to analyze many of the APIs. Please [add this Service Princple](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals) to **each workspace** so that SAT can access the APIs for analysis. 
 
 ## Installation