AnalysisException: resolved attribute(s) missing for blockEntityLinkage about spark-lucenerdd

[yeikel]

For now I can use a regular link with no blocking , but ideally I'd like to know if it possible to use the blocking method.

[zouzias]

Can you try with version 0.3.5 and report back?

[yeikel]

Seems to be fixed in 0.3.5

[yeikel]

Actually it is not..

I am still seeing this in 0.3.5

org.apache.spark.sql.AnalysisException: resolved attribute(s) cd#594 missing from name#738,address#748,mkt_cd#101,id#2 in operator !Project [name#738, address#748, mkt_cd#101, id#2, concat(mkt_cd#594) AS __PARTITION_COLUMN__#778];;

[zouzias]

Can you share the full exception here?

[zouzias]

I pushed a hotfix here: https://github.com/zouzias/spark-lucenerdd/pull/151/files (feedback is welcome) and I plan to release it tonight under 0.3.6-SNAPSHOT.

[yeikel]

Will this be available with Spark 2.1?

I will be testing as soon as it is in maven central

[zouzias]

I reproduced here. I will try to fix now.

[zouzias]

Released a fix under 0.3.6-SNAPSHOT. Can you look again if the exception appears?

Tests are clean on the CI: https://travis-ci.org/zouzias/spark-lucenerdd/jobs/499546155

[yeikel]

I believe it is fixed now. Thank you

[zouzias]

Glad to hear.

[yeikel]

This is very strange.

When I run it in a cluster (reading from a hive table) , I don't see this error anymore. On the other hand, when I run it in my local (reading parquet files) , I see it. I am not sure how to replicate it and the contents of the files are sensitive , so I can't share them here.

I need to test more , but let's leave it closed for now.