freiz / terminator Goto Github PK

Either I don't understand how to drive this lib. Or it's not operating correctly. Probably more "a" hopefully.

So here is what I see when I train a decent size of spam and ham email from my personal folders:

This is the sorted scores of all the spam and inbox email that I trained the filter on. The DB file is 23MB which sounds plausible. There doesn't seem to be a significant difference between the two groups of email. Basically my code parses the email into a list of words, removing all the headers and HTML encoding. Then it passes that as the content std::string to Terminator::Train with the spam bool set. Then I went back and called Predict on the same email to make the graphs in the image.

I did notice that the classifier_weights_ seem to be weird:
classifier_weights_[0]=-1.25549e+65
classifier_weights_[1]=0
classifier_weights_[2]=0
classifier_weights_[3]=0
classifier_weights_[4]=0
classifier_weights_[5]=0
classifier_weights_[6]=0
classifier_weights_[7]=0

That looks wrong... but it's what gets saved and reloaded... maybe that's part of my issue?

I was training a whole bunch of email and one of them has a length of 3 bytes.

In Terminator::Vectorization it uses an unsigned variable for 'len' so 'len - NGRAM' evaluates to -1 which wraps around to max unsigned.

My short term fix is to change len and i to int64_t. So negative numbers exit the loop correctly.

Some examples: embeded, precison, navie, memroy, exsiting, perfomance, persistance

And "those need adaptive model" should be "those that need adaptive models".

"implementation are described" -> "implementation is described"

"The only dependencies" -> "The only dependency".