How to store secrets in Azure Databricks

Azure Databricks
Azure Databricks

Background

In Azure Databricks, we can write code to perform data transformation on data stored in various Azure Services, e.g. Azure Blob Storage, Azure Synapse. However, as other programs, sometimes, you want to protect credentials used in Azure Databricks, Azure Databricks provides a solid secret management approach to help you achieve that.

Steps

Prepare Databricks command-line interface (CLI) in Azure Cloud Shell

Open Cloud Shell & make sure you select “Bash” for the Cloud Shell Environment.

Launch bash cloud shell
Launch bash cloud shell

Create Virtual Environment with below command.

source databrickscli/bin/activate
Activate virtual environment
Activate virtual environment

Install Databricks CLI with below command.

pip install databricks-cli
Install Databricks CLI
Install Databricks CLI

Create secret in Azure Databricks

Before you can create a secret, you need to authenticate as a user of the Azure Databricks, which requires your Azure Databrics workspace’s URL and a token

Get your Azure Databricks workspace’s URL

You can navigate to your Azure Databricks workspace and copy its URL.

Get Databricks URL
Get Databricks URL

Launch Databricks workspace

Launch Databricks workspace

Click ‘User Settings’

Click ‘User Settings’

Click ‘Generate New Token’

Click ‘Generate New Token’

Configure access token & click ‘Generate’

Configure access token & click ‘Generate’

Copy access token

Copy access token

After authentication, you need to first create a secret scope which you may group several secrets.

If your databricks is in Standard plan, you can only create secret scope which will be shared with other users in the same workspace.

Check databricks plan
Check databricks plan
databricks secrets create-scope --scope <<scope>>

# Example
databricks secrets create-scope --scope storage --initial-manage-principal users # Standard Plan
databricks secrets create-scope --scope storage # Premium plan
Create secret scope
Create secret scope

You can use below command to create secret under the specified scope.

databricks secrets put --scope <<scope>> --key <<key name>>
databricks secrets put --scope storage --key blob #Example
Type command to launch secret editor
Type command to launch secret editor
Type your secret and save
Type your secret and save

You can use secret by below command in notebook.

dbutils.secrets.get(scope=<<scope>>,key=<<key>>)

dbutils.secrets.get(scope=storage,key=blob) #Example
Use stored secret in notebook
Use stored secret in notebook

Blog: https://joeho.xyz

LinkedIn: https://www.linkedin.com/in/joe-ho-0260758a

Infrastructure on Office 365, Power Platform and Azure Products

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Frontend Insider #2

The simplest way to navigate and operate your microservices

How To Prevent NFT Trait Sniping In Your PFP Project

Add shortcuts to your PWA

Pushing Left, Like a Boss — Part 9: An AppSec Program

Introduction of ERA territory

A proposal for a new RPKI validator: OpenBSD rpki-client(1)

Why all programmers must learn C/C++

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Joe Ho

Joe Ho

Infrastructure on Office 365, Power Platform and Azure Products

More from Medium

How to pass parameters between Data Factory and Databricks

Passing parameters between Data Factory and Databricks by Azure Tutorials

Using Delta Lake on Data Bricks to transform Event Hub events in real time

(Azure) Databricks: accelerating big data analytics with the Spark connector for Azure SQL

How to connect to Azure Synapse in Azure Databricks

Azure Databricks