vagrant-hadoop-2.7.1-spark-1.4.1-hive-1.2.1
Introduction
Vagrant project to spin up a single virtual machine running:
- Hadoop 2.7.1
- Hive 1.2.1
- Spark 1.4.1
The virtual machine will be running the following services:
- HDFS NameNode + NameNode
- YARN ResourceManager + JobHistoryServer + ProxyServer
- Hive metastore and server2
- Spark history server
Getting Started
- Download and install VirtualBox
- Download and install Vagrant.
- Run
vagrant box add centos65 https://github.com/2creatives/vagrant-centos/releases/download/v6.5.1/centos65-x86_64-20131205.box
- Go to releases and download and extract the latest source of this project.
- In your terminal change your directory into the project directory (i.e.
cd vagrant-hadoop-2.7.1-spark-1.4.1-hive-1.2.1-<version>
). - Run
vagrant up
to create the VM. - Execute
vagrant ssh
to login to the VM.
Web user interfaces
Here are some useful links to navigate to various UI's:
- YARN resource manager: (http://10.211.55.101:8088)
- Job history: (http://10.211.55.101:19888/jobhistory/)
- HDFS: (http://10.211.55.101:50070/dfshealth.html)
- Spark history server: (http://10.211.55.101:18080)
- Spark context UI (if a Spark context is running): (http://10.211.55.101:4040)
Validating your virtual machine setup
To test out the virtual machine setup, and for examples of how to run MapReduce, Hive and Spark, head on over to VALIDATING.md.
More advanced setup
If you'd like to learn more about working and optimizing Vagrant then take a look at ADVANCED.md.
Credits
This project is based on the great work carried out at (https://github.com/vangj/vagrant-hadoop-2.4.1-spark-1.0.1).