Fatal java.lang.OutOfMemoryError thrown while processing document about cermine

[marekhorst]

Again, this looks like regression since I was able to extract metadata out of this document using:

http://cermine.ceon.pl/cermine

where 1.8-SNAPSHOT version is deployed.

[marekhorst]

Stack traces are slightly different for each attempt, here are the two examples:

2016-10-25 11:12:49,612 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
    at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
    at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:308)
    at java.lang.StringCoding.encode(StringCoding.java:344)
    at java.lang.String.getBytes(String.java:918)
    at com.itextpdf.text.pdf.fonts.cmaps.CMapToUnicode.convertToInt(CMapToUnicode.java:131)
    at com.itextpdf.text.pdf.fonts.cmaps.CMapToUnicode.createReverseMapping(CMapToUnicode.java:114)
    at com.itextpdf.text.pdf.CMapAwareDocumentFont.processToUnicode(CMapAwareDocumentFont.java:165)
    at com.itextpdf.text.pdf.CMapAwareDocumentFont.initFont(CMapAwareDocumentFont.java:106)
    at com.itextpdf.text.pdf.CMapAwareDocumentFont.<init>(CMapAwareDocumentFont.java:102)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.getFont(PdfContentStreamProcessor.java:138)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.access$5300(PdfContentStreamProcessor.java:60)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor$SetTextFont.invoke(PdfContentStreamProcessor.java:659)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.invokeOperator(PdfContentStreamProcessor.java:286)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.processContent(PdfContentStreamProcessor.java:425)
    at pl.edu.icm.cermine.structure.ITextCharacterExtractor.extractCharacters(ITextCharacterExtractor.java:107)
    at pl.edu.icm.cermine.ExtractionUtils.extractCharacters(ExtractionUtils.java:55)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:326)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.getContentAsNLM(InternalContentExtractor.java:286)
    at pl.edu.icm.cermine.ContentExtractor.getContentAsNLM(ContentExtractor.java:612)
    at pl.edu.icm.cermine.ContentExtractor.getContentAsNLM(ContentExtractor.java:628)
    at eu.dnetlib.iis.wf.metadataextraction.AbstractMetadataExtractorMapper.handleContent(AbstractMetadataExtractorMapper.java:192)
    at eu.dnetlib.iis.wf.metadataextraction.AbstractMetadataExtractorMapper.processStream(AbstractMetadataExtractorMapper.java:167)
    at eu.dnetlib.iis.wf.metadataextraction.MetadataExtractorMapper.map(MetadataExtractorMapper.java:31)
    at eu.dnetlib.iis.wf.metadataextraction.MetadataExtractorMapper.map(MetadataExtractorMapper.java:19)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)

2016-10-25 11:36:56,910 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.Integer.valueOf(Integer.java:832)
    at com.itextpdf.text.pdf.fonts.cmaps.CMapToUnicode.createReverseMapping(CMapToUnicode.java:114)
    at com.itextpdf.text.pdf.CMapAwareDocumentFont.processToUnicode(CMapAwareDocumentFont.java:165)
    at com.itextpdf.text.pdf.CMapAwareDocumentFont.initFont(CMapAwareDocumentFont.java:106)
    at com.itextpdf.text.pdf.CMapAwareDocumentFont.<init>(CMapAwareDocumentFont.java:102)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.getFont(PdfContentStreamProcessor.java:138)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.access$5300(PdfContentStreamProcessor.java:60)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor$SetTextFont.invoke(PdfContentStreamProcessor.java:659)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.invokeOperator(PdfContentStreamProcessor.java:286)
    at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.processContent(PdfContentStreamProcessor.java:425)
    at pl.edu.icm.cermine.structure.ITextCharacterExtractor.extractCharacters(ITextCharacterExtractor.java:107)
    at pl.edu.icm.cermine.ExtractionUtils.extractCharacters(ExtractionUtils.java:55)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:326)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
    at pl.edu.icm.cermine.InternalContentExtractor.getContentAsNLM(InternalContentExtractor.java:286)
    at pl.edu.icm.cermine.ContentExtractor.getContentAsNLM(ContentExtractor.java:612)
    at pl.edu.icm.cermine.ContentExtractor.getContentAsNLM(ContentExtractor.java:628)
    at eu.dnetlib.iis.wf.metadataextraction.AbstractMetadataExtractorMapper.handleContent(AbstractMetadataExtractorMapper.java:192)
    at eu.dnetlib.iis.wf.metadataextraction.AbstractMetadataExtractorMapper.processStream(AbstractMetadataExtractorMapper.java:167)
    at eu.dnetlib.iis.wf.metadataextraction.MetadataExtractorMapper.map(MetadataExtractorMapper.java:31)
    at eu.dnetlib.iis.wf.metadataextraction.MetadataExtractorMapper.map(MetadataExtractorMapper.java:19)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)

[marekhorst]

After running several tests in my local environment and in CI I can give you more details related to this failure.

It turned out I was able to finally process this file after bumping Xmx memory to 8GB. The problem is IIS metadataextraction mapper is allowed to use 4GB only and we cannot increase it significantly since CDH5 YARN is configured to give us 6GB at most (and we don't want to alter it since this will decrease the number of available mappers).

[marekhorst]

I just found this document:

http://oro.open.ac.uk/35300/1/TELInterdisciplinarity.pdf

among OpenAIRE2020 publications causing similar problem on CERMINE 1.11 with 4g memory assigned:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
	at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
	at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:308)
	at java.lang.StringCoding.encode(StringCoding.java:344)
	at java.lang.String.getBytes(String.java:918)
	at com.itextpdf.text.pdf.fonts.cmaps.CMapToUnicode.convertToInt(CMapToUnicode.java:131)
	at com.itextpdf.text.pdf.fonts.cmaps.CMapToUnicode.createReverseMapping(CMapToUnicode.java:114)
	at com.itextpdf.text.pdf.CMapAwareDocumentFont.processToUnicode(CMapAwareDocumentFont.java:165)
	at com.itextpdf.text.pdf.CMapAwareDocumentFont.initFont(CMapAwareDocumentFont.java:106)
	at com.itextpdf.text.pdf.CMapAwareDocumentFont.<init>(CMapAwareDocumentFont.java:102)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.getFont(PdfContentStreamProcessor.java:138)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.access$5300(PdfContentStreamProcessor.java:60)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor$SetTextFont.invoke(PdfContentStreamProcessor.java:659)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.invokeOperator(PdfContentStreamProcessor.java:286)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.processContent(PdfContentStreamProcessor.java:425)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor$FormXObjectDoHandler.handleXObject(PdfContentStreamProcessor.java:1264)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.displayXObject(PdfContentStreamProcessor.java:352)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.access$6100(PdfContentStreamProcessor.java:60)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor$Do.invoke(PdfContentStreamProcessor.java:1000)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.invokeOperator(PdfContentStreamProcessor.java:286)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.processContent(PdfContentStreamProcessor.java:425)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor$FormXObjectDoHandler.handleXObject(PdfContentStreamProcessor.java:1264)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.displayXObject(PdfContentStreamProcessor.java:352)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.access$6100(PdfContentStreamProcessor.java:60)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor$Do.invoke(PdfContentStreamProcessor.java:1000)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.invokeOperator(PdfContentStreamProcessor.java:286)
	at com.itextpdf.text.pdf.parser.PdfContentStreamProcessor.processContent(PdfContentStreamProcessor.java:425)
	at pl.edu.icm.cermine.structure.ITextCharacterExtractor.extractCharacters(ITextCharacterExtractor.java:107)
	at pl.edu.icm.cermine.ExtractionUtils.extractCharacters(ExtractionUtils.java:55)
	at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:326)
	at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)
	at pl.edu.icm.cermine.InternalContentExtractor.doWork(InternalContentExtractor.java:320)

It worked after bumping memory to 6g although, again, IIS metadataextraction mapper is allowed to use 4GB memory. Assigning more memory to job triggering CERMINE we are significantly decreasing metadataextraction performance because YARN can instantiate less containers at given time.

Could you check whether there is anything we could do to make CERMINE work with such documents with limited amount of xmx memory?

[dtkaczyk]

This issue was solved by upgrading iText version (8e82b5c)