Labs and Homework for CUSP GX-6002
To start PYSPARK in terminal:
PYSPARK_DRIVER_PYTHON=
which jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' /Applications/spark-2.3.1-bin-hadoop2.6/bin/pyspark
For Hadoop (Log in to HPC):
- mkdir
- check pwd
- move files into dumbo
(scp <file_name> dumbo:<folder>/)
- move files into HDFS in dumbo terminal
(hadoop fs -put <file_name> .
hadoop fs -ls output_<anything_goes>