Introduction course to Spark 3, using Scala
- (Optional) Install Spark 3.x locally
- https://spark.apache.org/downloads.html
- or mac:
brew install apache-spark
- Unzip data
Listens on port 8888.
Change in Makefile & part_04.StructuredStreaming
if already in use.
Run make stream-stocks
It will send a stock each 0.7 seconds Each stock record is a window of 5 minutes, and there are 3-4 stocks per window. So about 5 min of stocks per 2.8 seconds.
make build-jar
stores it in the output folder
make run-trainer
Does the spark-submit with all correct properties
First finish the code :-)
make run-predictor
Does the spark-submit with all correct properties
make download-survey
APIKEY=???? make download-stock
get APIKEY from https://www.alphavantage.co/sbt runMain part_03.DatasetForStreaming
to generate json