Handling smartquotes about mitie

[davisking]

The preprocessing and tokenization in ner_stream is pretty basic. However, we are setting up python, R, and java APIs right now that will make it easy for you to use whatever kind of preprocessing and tokenization you want (e.g. like NLTK's tokenizer) since in general different applications require different sorts of preprocessing and tokenization. But yeah, it's annoying that the default tokenizer doesn't handle a smartquote. So I just updated it and if you pull and recompile it should work properly now.

Cheers,
Davis

[geovedi]

Thanks for the update. Will definitely wait for the APIs.

[swadey]

@geovedi the simple ner_stream tokenizer doesn't handle the more general problem of unicode normalization. We should probably do this. If you can file an issue for posterity and discussion, we can keep track of this.

As @davisking mentioned, the solution may not be to put it in ner_stream but into the bindings. The issue there is that it affects training.