Spark failed to start: Driver unresponsive using vanilla Databricks cluster about azuretre

[marrobi]

Versus the JSON file see some difference, might be recent changes or errors when JSON file has been created.

metastore:
consolidated-westeurope-prod-metastore.mysql.database.azure.com has been added

artifact:
dbartifactsprodwesteu1.blob.core.windows.net has gone
arprodwesteua1.blob.core.windows.net has been added

Added them, now left with stgdbfssvc4c74.dfs.core.windows.net, this should be a private endpoint connection, but is not. We have one for blob, but not DFS. This is likely linked to the changes to ADLS from Blob in the Azure Databricks service.

Will look to PR a fix.

[niekbrom]

@marrobi we applied the fix and managed to get a Databricks cluster up and running.

Thanks for your quick response!

[marrobi]

Which Azure region are you using?

Can you check the Azure firewall logs for any traffic being denied from the subnet containing Databricks?

Also in the cluster logs, anything of note?

Ill try test it out this end, but might be a couple of days.

[tljjvogten]

I'm using region West Europe. I'll dive into the logs. Thank you.

[tljjvogten]

Only error in the cluster logs is the one mentioned.

[marrobi]

OK, if you can check the firewall logs. My guess is a file is trying to be downloaded but is blocked by the firewall - maybe a new FQDN dependancy. It's a couple of months since I've personally tested this.

If nothing obvious it might be worth opening a support ticket via the Azure portal for Databricks who might be able to provide some guidance.

[marrobi]

It could be related to this: https://learn.microsoft.com/en-us/azure/databricks/release-notes/product/2023/march#for-new-workspaces-that-use-udrs-azure-storage-ip-needs-an-additional-udr

[tljjvogten]

I'll check. But progress is slow. Running from one error into another.

[marrobi]

Just spinning up a cluster. Wil let you know how I get on.

[tljjvogten]

Great thanks. Sent from Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Marcus Robinson ***@***.***> Sent: Wednesday, June 21, 2023 3:28:54 PM To: microsoft/AzureTRE ***@***.***> Cc: Vogten, Tom ***@***.***>; Author ***@***.***> Subject: Re: [microsoft/AzureTRE] Spark failed to start: Driver unresponsive using vanilla Databricks cluster (Issue #3582) Just spinning up a cluster. Wil let you know how I get on. — Reply to this email directly, view it on GitHub<#3582 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZKQBVUS2OPZT2BVD5YZHE3XMLZJNANCNFSM6AAAAAAZNEPDDM>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[marrobi]

In the firewall logs I can see:

HTTPS request from 10.1.5.132:39766 to stgdbfssvc4c74.dfs.core.windows.net:443. Action: Deny. No rule matched. Proceeding with default action

HTTPS request from 10.1.5.134:59862 to md-hdd-kjwr2pc2wdgv.z29.blob.storage.azure.net:443. Action: Deny. No rule matched. Proceeding with default action

HTTPS request from 10.1.5.132:38192 to umsarnnfvjl3bbcrdndk.blob.core.windows.net:443. Action: Deny. No rule matched. Proceeding with default action

Will check the rule collection versus published rules. We recently added a note here - https://microsoft.github.io/AzureTRE/unreleased/tre-templates/workspace-services/databricks/

---

This service uses a JSON file to store the various network endpoints required by Databricks to function.

If you hit networking related issues when deploying or using Databricks, please ensure this file https://github.com/microsoft/AzureTRE/blob/main/templates/workspace_services/databricks/terraform/databricks-udr.json contains the approprate settings for the region you are using.

The required settings for each region can be extracted from this document: https://learn.microsoft.com/azure/databricks/resources/supported-regions.

[marrobi]

Nice breaking change from the Azure Databricks service.

[marrobi]

@tljjvogten If you can test the changes in the PR would be appreciated. Works for me.

Easiest way is to create a copy of the Databricks workspace service from the PR branch into a /templates/workspace_services/databricks directory and run make workspace_service_bundle BUNDLE=databricks .

Thanks for reporting this.