Different tokenizers used on training and validation data about adaptive

[yufengm]

Thanks for your note.

I've also spotted this before. But the preprocessing is just following the Karpathy's step, in which he just applied the nltk tokenizer.

Hope this helps.

[KeepingItClassy]

But it's actually skewing the evaluation results because a different tokenizer creates a different vocabulary. For accurate results, both the training and the validation/test captions need to be tokenized in the same way.

[yufengm]

Yeah, you are totally right. That's what most of the recent papers are doing, as they were following Karpathy's standard.

[CherishineNi]

@ KeepingItClassy
Could you share your test code? Code can be achieved on the screen images and captions. Thanks very much.