open-metadata / openmetadata-spark-agent Goto Github PK

The agent does not support glue-backed iceberg catalogs, failing with:

24/04/23 12:17:37 ERROR PlanUtils3: Catalog glue is unsupported
io.openlineage.spark3.agent.lifecycle.plan.catalog.UnsupportedCatalogException: glue
	at io.openlineage.spark3.agent.lifecycle.plan.catalog.IcebergHandler.getDatasetIdentifier(IcebergHandler.java:83)
	at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:61)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)

where glue is a catalog of type glue.

This is supported in openlineage-spark starting from 1.8.0

Description:
When creating a new Impala table using Spark with existing Impala tables, the data lineage is not being added. It seems that the toEntity in the code doesn’t retrieve information for the new table.

Expected Behavior:
in general, if it's a new table, I thought it would be appropriate to add a table to the data service (impala here) inside open metadata and data lineage should be automatically added for the new Impala table created in Spark, ensuring complete metadata lineage.

Actual Behavior:
No data lineage is added for the new table, resulting in incomplete metadata lineage.

Slack Link

• https://openmetadata.slack.com/archives/C02B6955S4S/p1714439693952469?thread_ts=1714439693.952469&cid=C02B6955S4S

Additional Information:

• Environment: YARN
• Openmetadata Configs: 1.3x
• OpenMetadata-spark-agent : 1.0.0-beta
• Spark Version: 3.2.3
• Impala Version: 4.1.1

df1 = spark.sql("select ....");
df.createOrRepalceTempView("view1");
df2 = spark.sql("select c1 from view1");
This program will report an error.
infinite recursion error。
org.apache.spark.sql.catalyst.expressions.AttributeReference->org.apache.spark.sql.catalyst.expressions.AttributeReference["canonicalized"]

We have openmetadata setup on AKS clsuter using helm chart. we have connected databricks and azure sql database services. when we create table on databricks using a table from azure sql, the data lineage should be azure_sql_table --> databricks_table, but this lineage is not coming, we can see lineage between tables on databricks, like databricls_table1 --> databricks_table2. we tried creating tables with openmetadata-spark-agent as well as openmetadata-spark-agent-1.0-beta, nothing is giving expected result.

I keep getting java.util.NoSuchElementException: No value present error when trying to to run my etl pipeline
the lineage information is not sent to my openmetadata instance, find code snippet for my spark code and the error message
Error.txt
snippet.py.txt

Kindly assist in providing the jar files for this project

Fix the warning Unable to writeValueAsString, this results into infinite recursion and fails to generate the lineage attached the logs.

debug.txt

spark.sql("insert into table1 (col1) select concat(col2, col3) from table2 limit 100").show()

this query only generates column level lineage between col1 and col2, we are missing col3 in this lineage

when data is transferred from S3 to databricks this lineage is not captured