Parallel version of Apriori, FP-Growth and ECLAT algorithm implemented under Spark RDD platform. The implementations are without Spark MLlib library and able to run on clusters in docker container.
The following steps will make you run codes on your spark cluster's containers. This implementation is based on one master and two workers configuration.
- Docker installed
- Docker compose installed
cd docker
git clone https://github.com/WangZX-0630/spark-fim-2s
cd spark-fim-2s
docker build -t zxwang/spark-hadoop-2s:3.3.2 .
docker-compose up -d
SSH login without password is required. You need to start ssh service in all nodes. Despite this, hadoop services are also required to be started in all nodes.
./start.sh
docker exec -it spark-fim-2s-spark-1 bash
Run a simple test to check whether the cluster is running properly. The test code is stored in the share/test.py
file. The test code will print the number of nodes in the cluster.
root@master:/opt >> spark-submit --master spark://master:7077 share/test.py
The following datasets are used in the test.
chess.dat
kosarak.dat
mushroom.dat
retail.dat
These datasets are from the FIMI website. You could download them from the website and put under the share/data dir.
wget http://fimi.ua.ac.be/data/chess.dat.gz
wget http://fimi.ua.ac.be/data/kosarak.dat.gz
wget http://fimi.ua.ac.be/data/mushroom.dat.gz
wget http://fimi.ua.ac.be/data/retail.dat.gz
gunzip *.gz
mv *.dat share/data
Put data sets into HDFS. The data sets are stored in the share/data
directory. Here put chess.dat into HDFS.
root@master:/opt >> hadoop fs -put share/data/chess.dat /
Run single fp-growth algorithm on one data set chess.dat. The minimum support is 0.3. Time Cost will be printed in the console.
root@master:/opt >> spark-submit --master spark://master:7077 share/fp-growth.py chess.dat 0.3
In the auto-test.py
, you can set the minimum support and the data sets you want to run. The performance results will be stored in the share/result.txt file.
root@master:/opt >> python share/auto_test.py
You can extend the cluster nodes number by following steps:
- Create new directory for the new image. For example,
spark-fim-4s
. Copy all files inspark-fim-2s
tospark-fim-4s
.
cd docker
mkdir spark-fim-4s
cp -r spark-fim-2s/* spark-fim-4s
- In
config/workers
, add more workders.
worker1
worker2
worker3
worker4
- Build a new image with a new tag, such as
zxwang/spark-hadoop-4s:3.3.2
if two more workers added.
docker build -t zxwang/spark-hadoop-4s:3.3.2 .
- Modify
docker-compose.yml
file. Update image name and volumes. Add more spark-workers.
For example, you can add two more worker nodes by adding the following lines in the docker-compose.yml file and update the image and volumes.
spark-worker-3:
image: zxwang/spark-hadoop-4s:3.3.2
hostname: worker3
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://master:7077
- SPARK_WORKER_MEMORY=2G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
volumes:
- ~/docker/spark-fim-4s/share:/opt/share
ports:
- '8083:8081'
spark-worker-4:
image: zxwang/spark-hadoop-4s:3.3.2
hostname: worker4
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://master:7077
- SPARK_WORKER_MEMORY=2G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
volumes:
- ~/docker/spark-fim-4s/share:/opt/share
ports:
- '8084:8081'
- Modify start.sh file as following:
docker exec -it spark-fim-4s-spark-1 sh start-hadoop.sh
docker exec -it spark-fim-4s-spark-worker-1-1 sh start-hadoop.sh
docker exec -it spark-fim-4s-spark-worker-2-1 sh start-hadoop.sh
docker exec -it spark-fim-4s-spark-worker-3-1 sh start-hadoop.sh
docker exec -it spark-fim-4s-spark-worker-4-1 sh start-hadoop.sh
- Run
docker-compose up -d
to start the cluster.
spark-fim-2s
├── Dockerfile
├── config
│ ├── core-site.xml
│ ├── hadoop-env.sh
│ ├── hdfs-site.xml
│ ├── mapred-site.xml
│ ├── ssh_config
│ ├── workers
│ └── yarn-site.xml
├── docker-compose.yml
├── share
│ ├── apriori.py
│ ├── auto_test.py
│ ├── data
│ │ ├── chess.dat
│ │ ├── kosarak.dat
│ │ ├── mushroom.dat
│ │ └── retail.dat
│ ├── fp-growth.py
│ ├── requirements.txt
│ ├── result.txt
│ └── test.py
├── start-hadoop.sh
└── start.sh