julioasotodv / spark-tree-plotting Goto Github PK

On trying to generate the json representation of a tree model by calling tree_json = generate_tree_json(model), the following error is thrown:

AttributeError: 'DecisionTreeModel' object has no attribute '_java_obj'

Unable to run:

tree_json = generate_tree_json(dtree_model,
                               withNodeIDs=True # each tree with node ID
                               )

265 sc = SparkContext.getOrCreate() 266 --> 267 json_tree = sc._jvm.com.vfive.spark.ml.SparkMLTree(DecisionTreeClassificationModel._java_obj).toJsonPlotFormat() 268 269 if withNodeIDs: TypeError: 'JavaPackage' object is not callable

The online method doesn't work, so I use the offline method to install, I start a session like this:
spark = SparkSession \
.builder \
.appName("Python Spark SQL basic example") \
.config("spark.jars","/Users/timmy/lala/spark/spark-3.0.0-preview-bin-hadoop2.7/my_example/2020spring_datamining/spark-tree-plotting/target/scala-2.11/spark-tree-plotting_0.2.jar")\
.getOrCreate()

then I got an error:
from spark_tree_plotting import plot_tree
ModuleNotFoundError: No module named 'spark_tree_plotting'

Hi,

We were happy to find this lib as we need to visualize the decision tree created using ML, but we are afraid that it cannot support the current Spark version.
We were trying multiple options in terms of setup, but when calling the plot_tree function, we get the following error:

ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 883, in send_command
response = connection.send_command(command)
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1040, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
Py4JNetworkError: Error while receiving

(...)

Py4JError: com.jasoto.spark.ml.SparkMLTree does not exist in the JVM

Code:

SUBMIT_ARGS = "--packages julioasotodv:spark-tree-plotting:0.2 pyspark-shell"
os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS

spark = (
SparkSession
.builder
.appName('jartest')
#.config('spark.jars', '/tmp/UniCredit_T100900/jars/spark-tree-plotting-0.2.jar')
.getOrCreate()
)

We'd appreciate some support on usage of this package.

Regards,
Michal