smerity / cc-warc-examples Goto Github PK

Hello Stephen,

I tried to execute your program in Amazon EMR, using AMI 3.3.1 (Hadoop 2.4.0)

Also I checked with different libraries of hadoop-core/hadoop-client.

On executing:

hadoop jar cc-warc-examples-0.1-SNAPSHOT-jar-with-dependencies.jar org.commoncrawl.examples.mapreduce.WATServerType

I get the following recurring error:
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

With mapred (old api) I don't get the error. But I will need features from MapReduce library.

Let me know what you recommend or if you have any workaround for that.

Thanks,
Prith

when i am trying to give input path (below path) to hadoop ,i am getting "Error: org.jets3t.service.impl.rest.httpclient.RestS3Service.(Lorg/jets3t/service/security/AWSCredentials;)V
" Error

Input path: s3n://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-48/segments/1386163035819/wet/CC-MAIN-20131204131715-00000-ip-10-33-133-15.ec2.internal.warc.wet.gz