Large portions of text body missing. about cermine

[dtkaczyk]

@pafnucy CERMINE indeed focuses on extracting metadata and references, and currently the default behaviour is to analyze a few first and a few last pages of the file. It would be possible to extract the entire text, but as the system currently stores the structure of the entire file in the memory while processing, there might be performance problems in the case of large files. And of course rewriting it so that the entire file is not kept in the memory is not trivial and would take time.

How are you using CERMINE: from your code, from command line using JAR file?

[pafnucy]

Using JAR

[eichmann]

@pafnucy I too just ran into this issue, and after some digging, came up with this, which overrides the default "first 20 and last 20 page" configuration:

ComponentConfiguration config = new ComponentConfiguration();
ITextCharacterExtractor charExtractor = new ITextCharacterExtractor();
charExtractor.setPagesLimits(1000, 1000);
config.setCharacterExtractor(charExtractor);
    ContentExtractor extractor = new ContentExtractor();
    extractor.setConf(config);