Comments (4)
The NaN usually occurs if one of the vectors being compared has an L2 norm of 0 which usually means the matrix has no non-zero values. The result of calculating cosine similarity on such a vector is undefined. Have you trained (fitted) the vectoriser prior to using it to transform your document(s)? I suspect at least some of the documents in your test data are 100% comprised of words that were not present in your training corpus. Perhaps try using a larger training data set to make sure the vectoriser learns more words during training.
from nlp.
Let me know how you get on?
from nlp.
Yeah! My training data set is pretty small now. It's true that some of the documents in my test data are comprised of words that were not present in training corpus. And I'll have a new try.
Thanks for answering my question in such a short time.
from nlp.
Not a problem and you are most welcome. You could also try the HashingVectoriser as a direct drop in replacement for the Count vectoriser which doesn't require training.
from nlp.
Related Issues (12)
- Ocr HOT 4
- LDA model persistence HOT 2
- Punctuation and PoS tagging
- [Question] Optimal number of topics in LDA HOT 1
- Out of memory - What are sensible upper bounds?
- License? HOT 1
- Interface for Tokeniser, Allow Custom Tokenisers? HOT 1
- Example fails, possibly due to gonum/matrix being deprecated? HOT 2
- Vectorisers.go only tokenise for a-Z languages. HOT 3
- Online/streaming LDA? HOT 1
- Methods for large corpora? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nlp.