analyticjeremy / azure-databricks-monitoring

Use Azure Monitor to track your Spark jobs in Azure Databricks

Hi there. 👋     🍑

I'm Jeremy, and I'm a data analyst.

I work at   as a Data Analyst. I help our Azure engineering teams sort through massive volumes of virtual machine telemetry data and make better decisions about our cloud offerings

• 🎓 expertise with performance tuning and problem solving
• ☁️ a decade of experience building solutions with Microsoft Azure  
• 🥼 built production-grade ML models with scikit-learn, R, and SparkML
• 💻 proficient with Python, SQL, C#, and Scala
• 💬 ask me how your data science and analytics projects can scale with cloud computing

---

Connect with me!

azure-databricks-monitoring's Issues

Automatically get the Pipeline Run ID from Databricks

The Databricks UI shows the ADF pipeline run ID. Is this information exposed in a place where we can grab it and include it in the telemetry?

Use ".getOption" method for SparkConf instead of try/catch (?)

Don't overwrite Databricks custom listener when loading our custom listener

Cluster scoped init script dbfs:/databricks/appinsights/appinsights_logging_init.sh failed: Script exit status is non-zero

Hi,

I'm trying to configure the Monitoring but when i start the Cluster i get :

Message

Cluster terminated. Reason: Init script failure
Help

Cluster scoped init script dbfs:/databricks/appinsights/appinsights_logging_init.sh failed: Script exit status is non-zero

Any Advise plz

Add support for AppInsights map

Expose some options for setting params for the App Insights App Map feature

How do we get the log for Spark SQL to AppInsight

How do we get the log for Spark SQL to AppInsight? We are trying to capture the queries run in Databricks notebook using Spark SQL and %sql options, the same is not available in traces when we check AppInsight, will those be coming as part of this code ? or do we need to customize this code? Can you please help us on the same.....

The sed command in the appinsights_logging_init.sh could be smarter

The sed command in the appinsights_logging_init.sh could be smarter. It just needs to append versus a full replace.

(originally posted by @AdamPaternostro in the repo's README)

Add Job ID for each stage

When sending dimension information from the "Stage Completed" event, include the Spark job ID

Duplicates errors in Application Insights

Logs are duplicates in application insights. init configured correctly.

Using jar file and ADF to call Spark job. We are not using Notebook.

SCRIPT initialization error

Every time i add this script to init scripts in local dbfs getting this error:
"instance_id": "a49ae08872a14977949bdb02fe9dc364",
"databricks_error_message": "Cluster scoped init script dbfs:/appinsights/appinsights_logging_init.sh failed: Script exit status is non-zero"
i am writing the script using notebook to make sure if formats correctly.

any help is much appreciated.

Running the steps as-is given, causes init script error and the driver log would not even give reason

Ran the steps as is. Except for removing the Log analytics part.

The driver log doesnt even give the exception details

adbxmonitor does not compile

The adbxmonitor does compile and fails with the following message:

[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:9:12: object databricks is not a member of package com                            
[error] import com.databricks.dbutils_v1.DBUtilsHolder.dbutils                                                                                                                                                                         
[error]            ^                                                                                                                                                                                                                   
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:22:39: object databricks is not a member of package com                           
[error]   private def getDatabricksInfo():com.databricks.backend.daemon.driver.DriverConf = {                                                                                                                                          
[error]                                       ^                                                                                                                                                                                        
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:23:16: object databricks is not a member of package com                           
[error]     import com.databricks.backend.common.util.Project                                                                                                                                                                          
[error]                ^                                                                                                                                                                                                               
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:24:16: object databricks is not a member of package com                           
[error]     import com.databricks.conf.trusted.ProjectConf                                                                                                                                                                             
[error]                ^                                                                                                                                                                                                               
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:25:16: object databricks is not a member of package com                           
[error]     import com.databricks.backend.daemon.driver.DriverConf                                                                                                                                                                     
[error]                ^                                                                                                                                                                                                               
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:27:9: not found: type DriverConf                                                  
[error]     new DriverConf(ProjectConf.loadLocalConfig(Project.Driver))                                                                                                                                                                
[error]         ^                                                                                                                                                                                                                      
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:27:20: not found: value ProjectConf                                               
[error]     new DriverConf(ProjectConf.loadLocalConfig(Project.Driver))                                                                                                                                                                
[error]                    ^                                                                                                                                                                                                           
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:27:48: not found: value Project                                                   
[error]     new DriverConf(ProjectConf.loadLocalConfig(Project.Driver))                                                                                                                                                                
[error]                                                ^                                                                                                                                                                               
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:31:47: object databricks is not a member of package com                           
[error]   private def getClusterUsageTag(dbrInfo: com.databricks.backend.daemon.driver.DriverConf, key: String): Option[String] = {                                                                                                    
[error]                                               ^                                                                                                                                                                                
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:163:27: not found: value dbutils
[error]     val notebookContext = dbutils.notebook.getContext
[error]                           ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:177:27: not found: value dbutils
[error]     val notebookContext = dbutils.notebook.getContext
[error]                           ^
[error] 11 errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 25 s, completed Apr 22, 2021, 7:20:23 AM

It would seem that dbutils is mssing. Adding it to the sbt-file still fails, this time due to the DriverConf and Project types not being defined:

[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:22:65: object driver is not a member of package com.databricks.backend.daemon
[error]   private def getDatabricksInfo():com.databricks.backend.daemon.driver.DriverConf = {
[error]                                                                 ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:23:35: object common is not a member of package com.databricks.backend
[error]     import com.databricks.backend.common.util.Project
[error]                                   ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:24:27: object conf is not a member of package com.databricks
[error]     import com.databricks.conf.trusted.ProjectConf
[error]                           ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:25:42: object driver is not a member of package com.databricks.backend.daemon
[error]     import com.databricks.backend.daemon.driver.DriverConf
[error]                                          ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:27:9: not found: type DriverConf
[error]     new DriverConf(ProjectConf.loadLocalConfig(Project.Driver))
[error]         ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:27:20: not found: value ProjectConf
[error]     new DriverConf(ProjectConf.loadLocalConfig(Project.Driver))
[error]                    ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:27:48: not found: value Project
[error]     new DriverConf(ProjectConf.loadLocalConfig(Project.Driver))
[error]                                                ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:31:73: object driver is not a member of package com.databricks.backend.daemon
[error]   private def getClusterUsageTag(dbrInfo: com.databricks.backend.daemon.driver.DriverConf, key: String): Option[String] = {
[error]                                                                         ^
[error] /home/tom/project/odf/Azure-Databricks-Monitoring/code/adbx-monitor/src/main/scala/com/microsoft/adbxmonitor/adbxlistener/AdbxListener.scala:163:27: Symbol 'type com.databricks.backend.common.rpc.CommandContext' is missing from the classpath.
[error] This symbol is required by 'method com.databricks.dbutils_v1.NotebookUtils.getContext'.
[error] Make sure that type CommandContext is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[error] A full rebuild may help if 'NotebookUtils.class' was compiled against an incompatible version of com.databricks.backend.common.rpc.
[error]     val notebookContext = dbutils.notebook.getContext
[error]                           ^
[error] 9 errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 8 s, completed Apr 22, 2021, 7:21:09 AM

How do we get the application log running in HDInsight cluster to AppInsight

How do we get the application log running in HDInsight cluster to AppInsight?

Can not start the Cluster with the init script (Script exit status is non-zero)

Cluster do not start. I get the following error:

• Init script dbfs:/databricks/appinsights/appinsights_logging_init.sh failed: Script exit status is non-zero