Increase HDFS block size 1GB about kylin

[abansal]

Are we increasing the Mapper's input size to 1GB or HDFS block size?
I would assume the task is to increase only Mapper's input, please correct me if I am wrong

[liyang-gmt8]

The intention is to increase HDFS block size only, and retain the current mapper input size. It was suggested that HDFS block size should be bigger to reduce hadoop/hbase overhead. The mapper input size should not be affected, which "mapred.max.split.size" come into play.

We need some in house test after the change to confirm the benefit.

[abansal]

Is this where you intend to change the block size?: https://github.com/abansal/Kylin/blob/master/job/src/main/java/com/kylinolap/job/hadoop/cardinality/HiveColumnCardinalityJob.java#L141-141

[liyang-gmt8]

Hi Ankur

The goal of this is to control the cube data files, which is the output of cube build jobs.

Check out this file: https://github.com/KylinOLAP/Kylin/blob/master/examples/test_case_data/hadoop_job_conf.xml

It is applied to all cube build jobs via com.kylinolap.job.JobInstanceBuilder.appendMapReduceParameters(). Try search where this method is called. You will find how a chain of jobs are created to complete a cube build.

Setting "dfs.block.size" and "mapred.max.split.size" in the hadoop_job_conf.xml shall do the trick.

We can do testing in eBay's hadoop cluster once you are familiar with the code around the job engine.