sabrinalameiras / ml-with-apache-spark

A series of Jupyter notebooks that walk you through Machine Learning with Apache Spark ecosystem using Spark MLlib, PyTorch and TensorFlow.

Machine Learning with Apache Spark Notebooks

This project aims at teaching you the Apache Spark MLlib in python. It contains the example code and solutions to the exercises in O'Reilly upcoming book Machine Learning with Apache Spark by Adi Polak.

Quick Start

Want to play with these notebooks online without having to install anything?

TBD as we evaluate free platforms, if you have any recommendations, please reach out on @AdiPolak

At the moment, you can use docker with running the following command. Memory is for providing jupyter environment with more memory, --mount is for mounting the local library, where all you work will be saved. adipolak/ml-with-apache-spark is an image in docker hub. -p are the port used for interacting with jupyter notebook.

docker run -it --memory="28g" --memory-swap="30g"  -p 8888:8888 --mount type=bind,source=$(pwd),target=/home/jovyan adipolak/ml-with-apache-spark

Just want to quickly look at some notebooks, without executing any code?

Brows it on the Juypyter.org notebook viewer

Want to run this project using a Docker image?

Read the Docker instructions - TBD

Want to install this project on your own machine?

TBD - would probably be using Docker.

FAQ

Which Python version should I use?

I get Py4JJavaError: An error occurred while calling o{some number}.parquet. (Reading Parquet file), what to do? From Jupyter enter the terminal and validate your Java version, given that we use Spark 3.1.1 it should be openjdk version "11.0.11". You local Java runtime should be of the same version - local Java runtime - is the Java that installed on the machine where you are running your docker from.

Which Apache Spark version should I use? The apache spark version is 3.1.1

Contributors

I want to thank everyone who contributed to this project by providing helpful feedback, filing issues, or submitting Pull Requests. If you would like to contribute, please feel free to submit a pull request, or reach out on @AdiPolak.