mozilla-metrics / akela Goto Github PK

Apologies for directly opening an issue but I didn't found a way to just drop a question.

I've tried to backup parts of my hdfs with your class and I constantly get several IOException (see below). Just using distcp works fine. Any ideas?

Thanks
Yves

[ec2-user@ip-10-40-211-16 ~]$ sudo /usr/local/hadoop-1.0.0/bin/hadoop jar akela-0.5-SNAPSHOT-job.jar com.mozilla.hadoop.Backup hdfs://ip-10-40-211-16.ec2.internal:8020/hbase/logfile local2
/usr/local/hbase-0.92.0/hbase-0.92.0.jar:/usr/local/hbase-0.92.0/lib/zookeeper-3.3.1.jar:/usr/local/hbase-0.92.0/conf:/usr/local/hbase-0.92.0/lib/guava-r09.jar:/usr/local/hbase-0.92.0/lib/zookeeper-3.4.2.jar
Adding input path: Backup-inputsource0.txt
Adding input path: Backup-inputsource1.txt
12/11/11 10:06:16 INFO input.FileInputFormat: Total input paths to process : 2
12/11/11 10:06:16 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
12/11/11 10:06:16 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev c7d54fffe5a853c437ee23413ba71fc6af23c91d]
12/11/11 10:06:16 INFO mapred.JobClient: Running job: job_201210221953_0008
12/11/11 10:06:17 INFO mapred.JobClient: map 0% reduce 0%
12/11/11 10:06:31 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_0, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:34 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_0, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:37 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_1, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:40 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_1, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:43 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_2, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:46 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_2, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:55 INFO mapred.JobClient: Job complete: job_201210221953_0008
12/11/11 10:06:55 INFO mapred.JobClient: Counters: 7
12/11/11 10:06:55 INFO mapred.JobClient: Job Counters
12/11/11 10:06:55 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=50404
12/11/11 10:06:55 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/11/11 10:06:55 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/11/11 10:06:55 INFO mapred.JobClient: Launched map tasks=8
12/11/11 10:06:55 INFO mapred.JobClient: Data-local map tasks=8
12/11/11 10:06:55 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
12/11/11 10:06:55 INFO mapred.JobClient: Failed map tasks=1`

I have been trying to parse a complex json with com.mozilla.pig.eval.json.JsonTupleMap()

Json Field:
{"Series":[{"DataType":"x","DataValue":"y"},{"DataType":"a","DataValue":"b"},{"DataType":"y","DataValue":"z"}]

Pig code:
raw = LOAD '/user/nidhi/piped-data'
USING PigStorage('|') AS
(Device_Type:chararray,
Event_Id:chararray,
JSON_Series:chararray);

parsed = FOREACH raw GENERATE Device_Type, Event_Id, JsonTupleMap(JSON_Series) as json:map[];

And I encounter this error:
ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 0: java.io.IOException: Deserialization error: could not instantiate 'com.mozilla.pig.eval.json.JsonTupleMap' with arguments 'null'

Can someone help me in identifying the problem here.

Thanks,
Nidhi