HiBench on Amazon EMR and S3 about hibench

[adrian-wang]

We didn't test HiBench on S3 before, but I think it should work.
From the log you provided, it seems your hadoop is not configured well for S3. Maybe you should refer to documentation of S3 about how to use S3 with Hadoop for help.

[ZuluPro]

Hello @adrian-wang

By configuring fs.default.name with s3a://, I have the same kind of error: org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient

Do you think Hi-bench could officialy support S3 benchmarks ?

[adrian-wang]

HiBench only runs some big data workloads on cluster, I think you should verify your S3 settings with yarn settings. Make sure your settings work before you begin your benchmark.

[bgaborg]

export DATA_HDFS=s3://

you should use s3a://

[weinick]

@bgaborg : I'm using s3a://, but met a similar error. Did you run successfully before? Can you share your configuration if it works? Thanks.

19/06/12 20:02:25 INFO dfsioe.DfsioeConfig: Setting testRootDir to 's3a://hibenchd
ata/HiBench/Dfsioe/Input'
java.lang.IllegalArgumentException: Wrong FS: s3a://hibenchdata/HiBench/Dfsioe/Inp
ut/io_control, expected: hdfs://hacluster
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:665)
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFil
eSystem.java:218)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileS
ystem.java:839)

[adrian-wang]

@weinick I think you should config your fs.defaultFs in core-site.xml to s3a://....

[weinick]

@adrian-wang Hi Adrian, thanks for your reply. As you said, I tried to update fs.defaultFs on my YARN core-site.xml, it still didn't work but the error message changed. I have added S3 AK/SK into this file also and I can run S3 based wordcount job. I'm pretty sure that the bucket name is correct and the credential is correct. Do you have any other idea?

19/06/12 22:04:48 INFO mapreduce.Job: map 0% reduce 0%
19/06/12 22:04:48 INFO mapreduce.Job: Job job_1560180791815_0030 failed with state FAIL
e FAILED due to: Application application_1560180791815_0030 failed 2 times due tontaine
AM Container for appattempt_1560180791815_0030_000002 exited with exitCode: -10
00
Failing this attempt.Diagnostics: doesBucketExist on hibenchdata: com.amazonaws.Amode :
mazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider
EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider :mode :
com.amazonaws.AmazonClientException: The requested metadata is not found at http
://169.254.169.254/latest/meta-data/iam/security-credentials/: No AWS Credentials
provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider Se FAIL
haredInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: Thntaine
e requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/
security-credentials/mazonC
org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on hibenchdata: conmentV
m.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentws.Ama
ialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredenti/lates
alsProvider : com.amazonaws.AmazonClientException: The requested metadata is not dentia
found at http://169.254.169.254/latest/meta-data/iam/security-credentials/: No AWsProvi
S Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredenti http:
alsProvider SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClient
Exception: The requested metadata is not found at http://169.254.169.254/latest/mm.amaz
eta-data/iam/security-credentials/

[adrian-wang]

@weinick Can you use the same configuration to do a hadoop fs -ls /? Your log appears there's something wrong with credential settings.