Skip to content

Pattern4.md

Latest commit

 

History

History
158 lines (111 loc) · 9.6 KB

Pattern4.md

File metadata and controls

158 lines (111 loc) · 9.6 KB

Tutorial for Pattern 4: Cluster Scoped Service Principal

Summary:

This document provides links to a tutorial on implementing Pattern 4: Cluster Scoped Service Principal

Versions:

Name Title Notes Date
Anil Sener Microsoft Cloud Solution Architect – Data & AI Original 01 December 2021

Contents

Pre-requisites

Tutorial Steps

License/Terms of Use

Pre-requisites

This tutorial requires the completion of the steps in Connecting securely to ADLS from ADB section.

This tutorial requires the completion of the steps in the preparation section.

This tutorial requires a premium Databricks Workspace.

Install Databricks Client.

Tutorial Steps

  1. Navigate to Azure Active Directory > App registrations and create two application registration service principals one for loans and another for iot_devices:

  1. Drill down to app registration created for iot_devices and navigate to Certificates & secrets > Client secrets and create a secret and copy the secret value to be used in KeyVault in the later steps:

  1. Drill down to app registration created for loans and navigate to Certificates & secrets > Client secrets/ and create a secret and copy the secret value to be used in KeyVault in the later steps:

  1. Navigate to the KeyVaults and search for the Keyvault called rr-demo which is created during the setup process, then navigate to Secrets and create two secrets for iot_devices and loans using the client secret values generated in step 2 and 3:

  1. When the ACLs for the container are displayed, add the principals for App registrations created in step 1. Please check only Execute right for these principals and click to Save button:

  1. Click on test_container and then you should be able to see the folders inside the container. Then, click ... icon on the rightern side of the iot_devices folder and select Manage ACLs:

  1. When the ACLs for the iot_devices folder are displayed, add the principals for the app registration service principal created for iot_devices and check Read and Execute and click to Save button:

  1. When the ACLs for the loans folder are displayed, add the principals for the app registration service principal created for iot_devices and check Read and Execute and click to Save button:

  1. Navigate to the premium Azure Databricks Workspace > Overview and click Launch Workspace button, choose and admin user to login. When Azure Databricks Workspace is displayed, navigate to Settings > User Settings and then click the Generate New Token button. Copy the token.

  2. Launch a commandline application in you pc and use Databricks Client to execute the following command to configure the databricks client with the workspace providing your premium workspace url and token generated in the previous step:

databricks configure --token
  1. After the Databricks Client configuration execute the following commands to create 2 Databricks secret scopes for iot_devices and loans and to create one Databricks secret per each to be referred in the subsequent steps in this tutorial. Please remember to replace <subscription-id> with your own in each command below:
databricks secrets create-scope --scope  iot-group-scope --scope-backend-type AZURE_KEYVAULT --resource-id /subscriptions/<subscription-id>/resourceGroups/RR-demo/providers/Microsoft.KeyVault/vaults/rr-demo-kv  --dns-name https://rr-demo-kv.vault.azure.net/

databricks secrets put-acl --scope  iot-group-scope  --principal  IoTDevicesGroup --permission READ

databricks secrets create-scope --scope  loan-group-scope --scope-backend-type AZURE_KEYVAULT --resource-id /subscriptions/<subscription-id>/resourceGroups/RR-demo/providers/Microsoft.KeyVault/vaults/rr-demo-kv  --dns-name https://rr-demo-kv.vault.azure.net/

databricks secrets put-acl --scope loan-group-scope --principal  LoanGroup --permission READ

These commands will limit access of Databricks groups only to the app registration service principals relavant to their group. Ideally, we should have created separate Azure key vaults for each secret scope.

  1. Navigate to the premium Azure Databricks Workspace > Overview and click Launch Workspace button, choose and admin user to login. When Azure Databricks Workspace is displayed, navigate to Compute and then create a cluster to be used by IoTDevices group by setting the Spark Configuration as below. Set the permissions to allow IoTDevicesGroup to attach this cluster. Please remember to replace <iot-devices-group-service-principal-application-id>,<secret name> and <Directory-ID> (your Azure Tenant ID):
fs.azure.account.auth.type OAuth
fs.azure.account.oauth.provider.type org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id <iot-devices-group-service-principal-application-id>
fs.azure.account.oauth2.client.secret {{secrets/iot-group-scope/<secret name>}}
fs.azure.account.oauth2.client.endpoint https://login.microsoftonline.com/<Directory-ID>/oauth2/token

  1. Create another cluster to be used by LoanGroup group by setting the Spark Configuration as below. Set the permissions to allow LoanGroup to attach this cluster. Please remember to replace <loan-group-service-principal-application-id>,<secret name> and <Directory-ID> (your Azure Tenant ID)::
fs.azure.account.auth.type OAuth
fs.azure.account.oauth.provider.type org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id <loan-group-service-principal-application-id>
fs.azure.account.oauth2.client.secret {{secrets/loan-group-scope/<secret name>}}
fs.azure.account.oauth2.client.endpoint https://login.microsoftonline.com/<Directory-ID>/oauth2/token

  1. Navigate to the premium Azure Databricks Workspace > Overview on Azure Portal and click Launch Workspace button, choose and TestUser1 user to login. When Azure Databricks Workspace is displayed, navigate to Workspace, then upload pattern4-Cluster-scoped-principals-iot_devices.ipynb notebook to the Databricks Workspace and open the notebook, attach & start the cluster created in step 12 and then run all cells:

RESULT: Files unders /iot_devices folder are readable meanwhile /loans raise an exception due to lack of privileges to access these files on ADLS Gen2. These permissions are limited thanks to the privileges of app registration service principal created for iot_devices on ADLS Gen2 ACLs.

  1. Navigate to the premium Azure Databricks Workspace > Overview on Azure Portal and click Launch Workspace button, choose and TestUser2 user to login. When Azure Databricks Workspace is displayed, navigate to Workspace, then upload pattern4-Cluster-scoped-principals-loans.ipynb notebook to the Databricks Workspace and open the notebook, attach & start the cluster created in step 13 and then run all cells:

RESULT: Files unders /loans folder are readable meanwhile /iot_devices raise an exception due to lack of privileges to access these files on ADLS Gen2. These permissions are limited thanks to the privileges of app registration service principal created for loans on ADLS Gen2 ACLs.

License/Terms of Use

This is a free white paper released into the public domain.

Anyone is free to use or distribute this white paper, for any purpose, commercial or non-commercial, and by any means.

THE WHITE PAPER IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE WHITE PAPER.