open-metadata / openmetadata-spark-agent Goto Github PK

We have openmetadata setup on AKS clsuter using helm chart. we have connected databricks and azure sql database services. when we create table on databricks using a table from azure sql, the data lineage should be azure_sql_table --> databricks_table, but this lineage is not coming, we can see lineage between tables on databricks, like databricls_table1 --> databricks_table2. we tried creating tables with openmetadata-spark-agent as well as openmetadata-spark-agent-1.0-beta, nothing is giving expected result.

when data is transferred from S3 to databricks this lineage is not captured

df1 = spark.sql("select ....");
df.createOrRepalceTempView("view1");
df2 = spark.sql("select c1 from view1");
This program will report an error.
infinite recursion error。
org.apache.spark.sql.catalyst.expressions.AttributeReference->org.apache.spark.sql.catalyst.expressions.AttributeReference["canonicalized"]

Fix the warning Unable to writeValueAsString, this results into infinite recursion and fails to generate the lineage attached the logs.

debug.txt

spark.sql("insert into table1 (col1) select concat(col2, col3) from table2 limit 100").show()

this query only generates column level lineage between col1 and col2, we are missing col3 in this lineage

Kindly assist in providing the jar files for this project

I keep getting java.util.NoSuchElementException: No value present error when trying to to run my etl pipeline
the lineage information is not sent to my openmetadata instance, find code snippet for my spark code and the error message
Error.txt
snippet.py.txt