prepro.py fails with assert len(lines) %2 == 0 about mt-dnn

[ankmathur96]

same error - commenting to see if there's a followup

[ankmathur96]

As an initial diagnosis, it seems that the QNNLI loader starts by loading the QNLI train TSV file, which has 104744 lines including the header. We remove the header, leaving an odd number of lines which fails the assert. This is true for the dev and test set files as well, which have 5464 lines.

I'm not familiar with what data is supposed to be read in, but it seems only 1 of the pairs of lines in the QNLI data actually don't trigger the if statement here. Not sure if that's expected, but seemed weird.

I tried dropping the assert and breaking if there's no subsequent line (if idx+1 >= len(lines)), but that fails on the test set on the assertion here. I tried adding code that simply continues the loop if block1[1] != block2[1].

That code is still running, but it's possible the preprocessing breaking might screw up later stages of the model and might make fine-tuning worse? @namisan please let us know if there's a stable commit that doesn't experience this problem

[namisan]

Thanks for suggestions. I've fixed the issue and please refer README for more information.

[ankmathur96]

Confirmed this works - thanks @namisan!