Jan 11
How to store secrets in Azure Databricks
Background
In Azure Databricks, we can write code to perform data transformation on data stored in various Azure Services, e.g. Azure Blob Storage, Azure Synapse. However, as other programs, sometimes, you want to protect credentials used in Azure Databricks, Azure Databricks provides a solid secret management approach to help you achieve that.
Steps
Prepare Databricks command-line interface (CLI) in Azure Cloud Shell
Open Cloud Shell & make sure you select “Bash” for the Cloud Shell Environment.
Create Virtual Environment with below command.
source databrickscli/bin/activate
Install Databricks CLI with below command.
pip install databricks-cli
Create secret in Azure Databricks
Before you can create a secret, you need to authenticate as a user of the Azure Databricks, which requires your Azure Databrics workspace’s URL and a token
Get your Azure Databricks workspace’s URL
You can navigate to your Azure Databricks workspace and copy its URL.
Launch Databricks workspace
Click ‘User Settings’
Click ‘Generate New Token’
Configure access token & click ‘Generate’
Copy access token
After authentication, you need to first create a secret scope which you may group several secrets.
If your databricks is in Standard plan, you can only create secret scope which will be shared with other users in the same workspace.
databricks secrets create-scope --scope <<scope>>
# Example
databricks secrets create-scope --scope storage --initial-manage-principal users # Standard Plan
databricks secrets create-scope --scope storage # Premium plan
You can use below command to create secret under the specified scope.
databricks secrets put --scope <<scope>> --key <<key name>>
databricks secrets put --scope storage --key blob #Example
You can use secret by below command in notebook.
dbutils.secrets.get(scope=<<scope>>,key=<<key>>)
dbutils.secrets.get(scope=storage,key=blob) #Example
Blog: https://joeho.xyz
