- The project setup basic
Hadoop Ecosystem
includes:Apache Hive
,Apache Spark
,JDBC MySQL
,Apache Sqoop
. - Use
Docker
to install the project
- Step 1: CD to
/sqoop_server_hdfs
cd ./sqoop_server_hdfs
./download.sh
-
./download.sh
will download packages from internet and saved in/sqoop_server_hdfs/packages
(this is a folder contain all component). -
Step 2: Execute command will run all container:
docker-compose up --build -d
- Step 3: Create and grant permission for user MySQL:
GRANT ALL PRIVILEGES ON sakila.* TO 'hdfs_part_1'@'%';
CREATE DATABASE sakila;
- Step 4: Import Sample data from
sakila-db.zip
:
curl -Lo ./sakila-db.zip http://downloads.mysql.com/docs/sakila-db.zip
unzip ./sakila-db.zip
cat sakila-db/sakila-schema.sql | docker exec -i mysql_hdfs_part_1 mysql -u hdfs_part_1 --password=hdfs_part_1 sakila
cat sakila-db/sakila-data.sql | docker exec -i mysql_hdfs_part_1 mysql -u hdfs_part_1 --password=hdfs_part_1 sakila
rm -rf unzip sakila-db
# Check sample data
docker exec -i mysql_hdfs_part_1 mysql -u hdfs_part_1 --password=hdfs_part_1 -e "select film_id, title from sakila.film limit 1"
-
Finally, this is easy!
-
NOTE: you can edit project config from files in path
/sqoop_server_hdfs/
like:hdfs-site.xml
(namenode.dir, datanode.dir),hive-site.xml
(ConnectionURL, ConnectionUserName, ConnectionPassword, JDBC Driver) -
import.sh
contains sqoop script import data from MySQL.