Docker file about Hadoop. Start all services via docker-compose.
- JDK 1.8.0_162
- CentOS 6.6
- Hadoop 3.0.0-RC0
docker-compose -f build-images.yml build
docker-compose up -d
And you would get the list of containers:
- hadoop-master
- hadoop-slave1
- hadoop-slave2
-
Get into container of hbase master via below command:
docker exec -it master bash
-
Using Hadoop command:
hadoop fs -ls /
-
Executing mapreduce example pi:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0.jar pi 16 1000
If the program worked correctly, the following should be displayed at the end of the program output stream:
Estimated value of Pi is 3.14250000000000000000
docker-compose start
Hadoop:http://{hostip}:9870
Yarn Application:http://{hostip}:8088
JobHistory Server: http://{hostip}:19888
Note: {hostip} is the ip of your host node. If your host ip is 192.168.1.78, you could access the Hadoop web ui via http://192.168.1.78:9870
The patch updates the HDFS default HTTP/RPC ports to non-ephemeral ports. The changes are listed below:
Namenode ports: 50470 --> 9871, 50070 --> 9870, 8020 --> 9820
Secondary NN ports: 50091 --> 9869, 50090 --> 9868
Datanode ports: 50020 --> 9867, 50010 --> 9866, 50475 --> 9865, 50075 --> 9864
To see HDFS-9427 to know more details.