Enable Serverless Databricks SQL Warehouses
Serverless is the future, but there are steps you may need to take to be ready.
With the serverless compute version of the Databricks platform architecture, the compute layer exists in your Azure Databricks account rather than your Azure subscription. This provides a number of advantages:
- Faster cluster spinup time (instances are pre-warmed at no cost to you)
- All consumption counts towards your Azure Databricks Commitment
- Simplified management of quotas and network resources
Azure Region Rollout
- For the latest availability see: Azure Databricks Region Support
Feature details
- Databricks offers serverless SQL warehouses which spin up significantly faster than classic SQL warehouses. Because this compute is spun up in databricks' environment, it is not compatible with our private networking without additional configuration.
- This configuration is a cloud resource called a Network Connectivity Configuration or NCC for short. It allows the creation of managed private endpoints on storage accounts. These managed private endpoints will be added to our datalakes, thus enabling serverless SQL warehouses.
Installation details
- To install, follow the Databricks guide on Serverless SQL Warehouse configuration.
- Your team will need to contact your databricks' account admin and request enablement of the private endpoint feature for NCC on your account.
- Second, contact the Hitachi team and we will complete the remainder of the NCC installation process in your environment.
Updated 9 days ago