jkthompson / pyspark-pictures

Learn the pyspark API through pictures and simple examples

pyspark-pictures

Learn the pyspark API through pictures and simple examples

View on NBViewer

• RDD API
• DataFrame API

RDD Example:

# flatMap
x = sc.parallelize([1,2,3])
y = x.flatMap(lambda x: (x, 100*x, x**2))
print(x.collect())
print(y.collect())

[1, 2, 3]
[1, 100, 1, 2, 200, 4, 3, 300, 9]

Install (for interactive use)

1. install Spark
2. install IPython notebook

Quick Start

1. start pyspark inside IPython notebook

   IPYTHON_OPTS="notebook" pyspark

2. open browser to notebook link

3. open pyspark-pictures.ipynb or pyspark-pictures-dataframes.ipynb

4. edit example code, press: ctrl + enter to run each cell

References

pyspark API

Contribute

Contributors are welcome
Original images are here, download to pdf, convert to svg with: genSVD (pdf2svg)

pyspark-pictures's Issues

Pictures not visible (from blog comment)

Thanks.
The best cheet sheet of pyspark.
In the notebook the pictures are not visible and complains about the v3 version, instead of the v4.
Python 2.7.6 , IPython 3.1.0 , in a prepackaged VM with Spark (from Edx mooc)