Certain features and elements of the data estate require additional administrator setup after deployment of the Azure environment. This section covers those concerns.

SCIM - Sync Users and Groups with Databricks

System for Cross-Domain Identity Management (SCIM) is an open standard protocol for automating the exchange of user identity information between identity domains and IT systems. SCIM ensures that employees added to your teams automatically have accounts created via Microsoft Entra ID / Active Directory in the data environment. User attributes and profiles are synchronized between your directory and the data plane, allowing users to be linked into groups in Databricks, as well as adding and removing users based on Entry group membership.

You should provision identities to your Databricks workspace using Microsoft Entra ID before wide production use. Please see the Databricks guide on SCIM provisioning to learn how.

Databricks Secret Scope Creation

Create Key Vault Secret Scope 

• After your resource groups have been deployed, enter the Databricks workspace in each of your resource groups 
• Note: You must be able to create managed identities in the databricks workspace's tenant.
• Note: This step unfortunately cannot be automated because it requires a user rather than a service principal to accomplish. This is an issue with how AD interacts with Databricks and for now there is no workaround.

Step 1 - Navigate to your Databricks workspace (you will repeat these steps for each environment).

Step 2 - After you open the workspace, add #secrets/createScope to the end of the URL. This will navigate you to the secret scope creation page.  

• For example: https://adb-6293919441821822.2.azuredatabricks.net/?o=6293919441821822#secrets/createScope

Step 3 - In the new window, name the scope primary-key-vault-scope. Make sure to set the Manage Principal field to All Users.

Step 4 - Finally, add your Azure Keyvault dns and resource id. Keyvault DNA can be found in the properties tab of your Keyvault under the name Keyvault URI. The resource id can be found in the properties tab as well.

Step 5 - After all these fields have been filled out, hit create. 

Repeat for each Databricks workspace.