sathiyarajanm / ansible-role-hadoop

Hadoop stack role

Description

This role is used by me for small hadoop cluster (mostly learning purposes) However with this role you can get simple hadoop cluster up and running in virtually no time This includes HDFS, MR and YARN, and can be installed in distributed mode on any amount of machines If you intend to use it for production... don't. Seriously, just dont. Take a look at ambari/cdh/hortonworks instead.

Usage

To install parts of hadoop stack on your machines, use this vars in playbook/hostvars:

hadoop_hdfs_namenode: true
hadoop_hdfs_secondarynamenode: true
hadoop_hdfs_datanode: true
hadoop_hdfs_nfs_gateway: true
hadoop_yarn_resourcemanager: true
hadoop_yarn_nodemanager: true
hadoop_mapred_historyserver: true

Also make sure to specify hadoop_master (where namenode and nodemanager is located). If you have masters on separate machines, you can override them per-service:

hadoop_hdfs_master: 127.0.0.1
hadoop_yarn_master: 127.0.0.1
hadoop_mapred_master: 127.0.0.1

By default, those variables are aliased to hadoop_master Everything else is optional, you can see that params in defaults/main.yml

Low spec mode

Setting hadoop_low_settings to true will use very low-end settings for hadoop, allowing you to run hadoop cluster on very low-end VPSes or even stuff like raspberrypi/orangepi and similar boards.

Default webui ports

hadoop2:

• 50070 - namenode web
• 50090 - secondarynamenode web
• 19888 - MR1 web
• 8088 - YARN web

hadoop3:

• 9870 - namenode web
• 9868 - secondarynamenode web

TBD:

• HDFS HA
• Multiple disk support for HDFS datanodes

Credits

Thanks to this repo for like half of code here, especially systemd units