adamsp / adnwc Goto Github PK

Stopwords can't be mixed, and stopwords for only one language won't suffice. Need to identify language of a given post (I've used a library for this in Java previously), and then filter them out into separate 'clouds'.

Currently using a dirty hack and only accepting posts with en_XX as locale - however, just because you have a non-en locale doesn't mean your post is not in English, similarly just because you have an en locale doesn't mean your post is in English.

App.Net offers the ability to request 'annotations' with posts:
http://developers.app.net/docs/resources/post/#general-parameters

The annotations documentation shows a list of 'core' annotations - including one for the post language. Unfortunately, from what I've seen, no clients fill these annotations in - they're optional.

At the moment I'm using a dictionary to track word occurrences - which is fine for a limited number of words. I don't know what the performance will be like after running for an extended period of time - memory usage, specifically, but also whether I'll strike problems with resizing or hash collisions.

Currently considering a simple patricia trie/radix trie.