Private Entry Tokens (PATs) are a handy technique to entry companies like Azure Databricks or Azure DevOps with out logging in together with your password. As we speak, many purchasers use Azure DevOps PAT tokens as Git credentials for distant repositories in Databricks Git folders (previously Repos). Sadly, the usage of PAT tokens comes with some downsides. In Azure DevOps, PAT tokens can’t be issued to service principals and managed identities, which signifies that clients resort to a service account or perhaps a consumer’s id. Moreover, the utmost lifespan of PAT tokens is usually days, weeks, and even months. Whereas their rotation (the method of refreshing the tokens such that older ones can now not be used) will be ruled, because of this a leaked token with a protracted lifespan could pose a big danger. A safer various is to entry Azure DevOps sources utilizing a Microsoft Entra ID (previously Azure Energetic Listing) entry token.
From the Microsoft Docs:
As PATs are merely bearer tokens, which means token strings that signify a consumer’s username and password, they’re extremely dangerous to make use of as they’ll simply fall into the mistaken particular person’s palms. Microsoft Entra tokens expire each hour […], which limits the general danger issue when leaked.[1] When contemplating entry to the Azure DevOps Git repositories linked to your Databricks Git folders, you now not have to depend on PATs. Now, you need to use Microsoft Entra ID entry tokens, which have tighter controls round token rotation and expiry.
On this weblog, we’ll discover ways to use an Entra ID entry token as a Git credential in Databricks Git folders to strengthen the safety posture when pulling repositories hosted in Azure DevOps.
Conditions | Create Service Principal
To begin, you want a managed id or service principal. If you happen to shouldn’t have one, observe this doc: Register a Microsoft Entra app and create a service principal. On the finish of it, you should have a service principal you need to use. Word that on this situation no redirect URI is required, so you’ll be able to go away that kind aspect clean. Make certain to create a secret and word it down, along with the service principal ID. (The next steps present how you employ a service principal because the mechanism for authentication, however the identical steps additionally apply to a managed id.)
This course of assumes that you’ve an Azure DevOps challenge arrange with a Git repository you want to hyperlink to a Databricks Git folder.
Step 1 | Grant your service principal Reader permissions in your challenge
UnderAzure DevOps Venture settings > Permissions > Readers add your service principal.
Make sure the entry degree is enough for the required operation below Group settings > Customers.
Step 2 | Grant service principal required permissions in Databricks
If you happen to use Unity Catalog, open one other browser tab and go to your Databricks account console, after which add the service principal to your account.
Now, generate an OAuth secret to authenticate in opposition to the Databricks API (utilizing the CLI) and replica it down someplace safe.
Lastly, grant the service principal consumer permissions in your workspace.
Step 3 | Use the CLI to create Entra ID Token and retailer it in Databricks Git credential
You’ll use the Azure and Databricks CLI for this step. To authenticate in opposition to Databricks, you want a configuration profile (.databrickscfg) configured with the OAuth token we simply created, your workspace URL, and a service principal ID. Your replace to .databrickscfg ought to look one thing like this:
[DEFAULT]
host = https://<workspace-url>.azuredatabricks.internet/
client_id = <service principal ID>
client_secret = <Databricks OAuth token worth>To log the service principal with the AzureCLI we use the key we have now created earlier. The script requests an Entra ID entry token scoped to Azure DevOps (indicated by the UUID 499b84ac-1321-427f-aa17-267ca6975798), then configures a Git credential with the Databricks CLI and makes use of it to arrange our new Git folder:
#!/bin/bash
# Immediate consumer for required inputs and assign to variables
learn -p "Enter Service Principal ID: " service_principal_id
learn -p "Enter Tenant ID: " tenant_id
learn -p "Enter Service Principal Secret: " service_principal_secret
learn -p "Enter Service Principal Title: " service_principal_name
learn -p "Enter your Azure DevOps Group identify: " devops_organization
learn -p "Enter your Azure DevOps challenge identify: " devops_project
learn -p "Enter your Azure DevOps repository identify: " devops_repo
#Login to Azure because the service principal
az login --allow-no-subscriptions --service-principal -u $service_principal_id -p $service_principal_secret --tenant $tenant_id
#Because the service principal, request an EntraID entry token scoped to Azure DevOps.
ENTRA_ID_TOKEN=$(az account get-access-token --resource "499b84ac-1321-427f-aa17-267ca6975798" --query "accessToken" --output tsv)
#Use the entry token as a substitute of a PAT to create a Git credential in Databricks with the service principal's identify as git username.
#This assumes you may have already setup the Databricks CLI .databrickscfg file with workspace, client_id, and client_secret
databricks git-credentials create azureDevOpsServices --personal-access-token $ENTRA_ID_TOKEN --git-username $service_principal_name
#Create a brand new Databricks repository utilizing the service principal identify because the consumer identify
databricks repos create https://$service_principal_name@dev.azure.com/$devops_organization/$devops_project/_git/$devops_repo
Abstract | What’s subsequent?
You’ve now realized how you can generate Microsoft Entra ID entry tokens scoped to Azure DevOps after which retailer them as a Databricks Git credential as a substitute of as a DevOps PAT token. Because the MS Entra ID entry token is short-lived, your pipeline should replace the Git credential utilizing Databricks git-credentials replace, and might then set off a pull by calling Databricks repos replace. As this course of simply showcases the credential setup, further safety measures are often required in a manufacturing setting, like storing the service principal consumer secret and the Databricks OAuth token in a safe secret retailer like Azure Key Vault.
See Use Azure Key Vault secrets and techniques in Azure Pipelines and Secret scopes for additional particulars.