Validate model on real text data about dilated-cnn-ner

[marc88]

@sbrugman : I beg your pardon but, can we use the commit you've made, to save predictions for both conll and Ontonotes sets, such that it takes in text and pumps out their labels?
Is PR (Inference and minor improvements #16) complete?

Goal:
Input: Germany lost a soccer game to Spain on Tuesday

Output: ['B-LOC', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O']

Edit:
It would be of great help if someone can suggest the directory where the predictions are saved

Thanks

Regards

[Alaska47]

Hi, I'm also looking for a way to validate the model in a more interactive manner by using real text data. @marc88, were you able to find a way to get the tags given a sentence?

[strubell]

Unfortunately the code doesn't currently support this. Happy to take pull requests, though!

…

On Tue, Jun 25, 2019 at 10:56 AM Aneesh Kotnana ***@***.***> wrote: Hi, I'm also looking for a way to validate the model in a more interactive manner by using real text data. @marc88 <https://github.com/marc88>, were you able to find a way to get the tags given a sentence? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=AAY5TNZ4J3H6NNO5YGW475DP4IWZLA5CNFSM4GPWPID2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYQQXWA#issuecomment-505482200>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAY5TN6AHWJQQA3QSFEPKVDP4IWZLANCNFSM4GPWPIDQ> .

[Alaska47]

I think I might have something that may work but I don't think I'm doing the preprocessing properly.

Essentially, I tokenized the sample sentence and then using code from the make_example function in preprocess.py, created the different feature vectors (tokens, shapes, chars, seq_len, and tok_len). Additionally, I passed in the vocabulary that the model was trained on, and if there were any tokens that were OOV, I just added them to the vocabulary.

However, I'm not sure this is either the right way to do or even a viable way to do it. From what I read in the code, the model uses a version of the embeddings scheme presented in the Lample paper so it takes into account word embeddings and character embeddings. So in the case of a OOV token, the embedding scheme should be able to create a robust embedding for it, right (provided I enabled character embeddings in the conf files)? If not, how should I deal with OOV tokens in the preprocessing stage?

I was printing out some sample feature vectors from the dev set and training set and comparing them to the ones that I generated dynamically and they look reasonably similar (in terms of padding and stuff) so I assumed I didn't mess up the preprocessing. I used code from the run_evaluation method to do model inference. In the feed_dict, I just passed an array of zeroes for model.input_y (is this okay or should I do anything different?) as the model should be predicting the labels. Right now, I'm looking at the predictions and trying to remap them back into the appropriate labels. My understanding is that there is padding at the beginning and end of the token feature vector. When I look at the predictions, the padding has a label for it that isn't O. Should I just disregard that token?

Also, how does the preprocessing treat tokens with numbers? Does it change all the digits to 0? I was looking at the tokens.txt file produced by the preprocessing and I was seeing conflicting behavior (i.e some tokens had their digits changed to all 0s and other tokens had their digits unchanged). For reference, I'm training this model for an NER task that involves detecting numerical data (i.e phone numbers).