Spark Support about greykite

[al-bert]

Hi Venkat, we don't have immediate plans to support spark dataframes. If you convert your spark dataframe to a pandas dataframe you could run forecasts on the driver: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.toPandas.html. From there, you could try using joblibspark as the backend for scikit-learn's grid search: https://github.com/joblib/joblib-spark

[venk2k16]

Thanks for the reply Albert.

That is what we are currently doing, converting RDD to pandas dataframe and running it on the drivers. Our issue is that we have like 100 different timeseries that we want to run the forecasts for. Native RDD suppor would be of great help.

[al-bert]

I see! We don't plan to natively support spark dataframes, so your best bet would be to find another way to run this in parallel. I'll close this thread, but if you find a good approach, please comment to benefit other users!