Data preprocessing process about synpg

[ej0cl6]

We follow the preprocessing steps of SCPN. You can refer to this repo https://github.com/miyyer/scpn
It contains a Java command to run StanfordCore NLP parser.

To run your own sentences through the pretrained model, first parse them using Stanford CoreNLP. We used CoreNLP version 3.7.0 for the results in the paper, along with the Shift-Reduce Parser models (https://nlp.stanford.edu/software/stanford-srparser-2014-10-23-models.jar). More concretely, the command we ran to parse the ParaNMT dataset used in our paper is:

java -Xmx12g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLP -threads 1 -annotators tokenize,ssplit,pos,parse -ssplit.eolonly -filelist filenames.txt -outputFormat text -parse.model edu/stanford/nlp/models/srparser/englishSR.ser.gz -outputDirectory /outputdir/

Note that StanfordCore NLP now supports Python (Stanza), but the parsed results are a little bit different from the Java version. So I would recommend you to use the Java parser if you want to directly use the pre-trained SynPG.

[hanyc0619]

Thanks!