Enable Serverless Databricks SQL Warehouses

With the serverless compute version of the Databricks platform architecture, the compute layer exists in your Azure Databricks account rather than your Azure subscription. This provides a number of advantages:

• Faster cluster spinup time (instances are pre-warmed at no cost to you)
• All consumption counts towards your Azure Databricks Commitment
• Simplified management of quotas and network resources

Azure Region Rollout

• For the latest availability see: Azure Databricks Region Support

Feature details

• Databricks offers serverless SQL warehouses which spin up significantly faster than classic SQL warehouses. Because this compute is spun up in databricks' environment, it is not compatible with our private networking without additional configuration.
• This configuration is a cloud resource called a Network Connectivity Configuration or NCC for short. It allows the creation of managed private endpoints on storage accounts. These managed private endpoints will be added to our datalakes, thus enabling serverless SQL warehouses.

Installation details

• Currently, this feature is in public preview and requires several steps to enable. If this feature can be delayed for your organization, the setup process will be easier once the feature comes out of preview.
• First, your team will need to contact your databricks' account admin and request enablement of the private endpoint feature for NCC on your account.
• Second, contact the Hitachi team and we will complete the remainder of the NCC installation process in your environment.