domnikl / highscore

Find and rank keywords in text

Hey, I am Dominik 🧑‍💻

I live in Germany 🏫, started working as a Full Stack Developer in 2003 👴 and I am currently working as a Senior Data Engineer at YAZIO👨‍💻.

• 🔭 I’m currently working on grooveguessr
• 💻 mainly working with Kotlin, Spring Boot and Python in my day job now
• 🦀 love Rust
• I have a keen interest in aviation 🛫, football 🏈, and 3D printing

👯 Communities

• Better Programming (2023-)
• YAZIO Engineering Medium (2022-)
• dev.to (2019-)
• DesignPatternsPHP (2011-2022)
• Mayflower blog (2011-2013)

highscore's Issues

Use a Whitelist and a Blacklist?

Is it posible to use both whitelist and blacklist?

detect the language of a text

I think language detection could be solved easily with a list of words that a language uses most. E.g. for English: "for with that this", German: "der die das ..." and so on, but could be very hard for short texts.

rate based on amount of vowels and constants in a word (%)

non english words?

Hi!
How could I add others locales support?

add a whitelist

Whitelist should only be considered if one was given, Blacklist should not be used when whitelist was given (= makes no sense)

allow custom blacklist of words to be used

Highscore::Content.new should get an optional parameter that is an array of words that will get ignored

Stemming not working?

Can't seem to get stemming to work. Any ideas?

2.0.0-p648 :065 > x = Highscore::Content.new('fishes dog fishing boat', Highscore::Whitelist.load('fish'))
#<Highscore::Content:0x007fb2beea6e60 @content="fishes dog fishing boat", @blacklist=nil, @whitelist=#<Highscore::Whitelist:0x007fb2beea7158 @words=["fishing"], @bloom_filter=#<BloomFilter::Native:0x007fb2beea7040 @opts={:size=>40, :hashes=>28, :seed=>1466442694, :bucket=>4, :raise=>true}, @bf=#<CBloomFilter:0x007fb2beea6f00 @hash_value={}>>>, @language_wordlists={}, @emphasis={:multiplier=>1.0, :upper_case=>3.0, :long_words=>2.0, :short_words_threshold=>2, :bonus_multiplier=>3.0, :bonus_list=>nil, :long_words_threshold=>15, :vowels=>0, :consonants=>0, :ignore_short_words=>true, :ignore_case=>false, :ignore=>nil, :word_pattern=>/\p{Word}+/, :stemming=>false}>

2.0.0-p648 :065 > x.configure { set :stemming, true }

2.0.0-p648 :065 > x.keywords
#<Highscore::Keywords:0x007fb2bee451b0 @keywords={}>

Stemming?

Any thoughts on stemming (or making it an option) the keywords?

Looks like gems like fast-stemming would make this pretty easy to do.

Use a bonuslist and a blacklist?

Is it possible to use both a bonuslist and a blacklist? Essentially, I want to be able to exclude words while giving others a boost.

rank number should be a property of Keyword

method to merge Keywords only works with unranked data

atm it is only possible to merge Keywords objects that only hold unranked keywords in a Hash.

# @param keyword String
def <<(keyword)
   ...
end

Why is that a string, when this should be a Keyword object instead?

add a settings flag to ignore/don't ignore very short words

At the moment, words with less than or two chars are ignored. This should be default but configurable via a set :ignore_short_words, false call.

random seed error

With rails 4, ruby 2, development environment, highscore 1.2.0 will throw an error of "random seed error", though it works in ten percent of my tries, but it works entirely well in production .In my development environment, I have to change the highscore version to 1.1.0, no error occored.

bin/highscore should provide a simple CLI tool

Parameters should be:

• a text via STDIN
• additional parameters, like path to a blacklist file
• ranking parameters
• print the resulting list of ranked keywords to STDOUT

ability to preseed with previously used keywords

Just like black- and whitelists, there should be a method to preseed the process with a list of keywords that have been used in the past. Those should then get ranked higher than anything else.

encapsulate Keywords in their own class

Keywords are currently handled as Hash and Arrays (ranked Keywords are arrays because Hashes are not sortable). This should be fixed to their own classes that handle sorting and stuff like that.

rank blacklisted words

Just like the actual list of ranked keywords, there should be methods to get a (ranked) list of (explicitely) ignored keywords.

stemming for '0181,' throws error. undefined method `stem' for "0181,":String

I know stemming cannot happen for this. but please handle if possible.

monkey-patch String: add a keywords method that returns keywords object

To simplify Keyword handling, String should have a keywords method that uses the standard blacklist and returns a collection of keywords (Keywords object). Optionally, a blacklist file and configuration could be attached via a fancy block syntax.

Remove compatibility for Ruby 1.8.7

Travis build for 2.0.0 works, so 1.8.7 should no longer be supported. Remove all ugly compatiblity code for 1.8.

Exception when handling keywords that are actually Fixnums

NoMethodError: undefined method upcase’ for 98:Fixnum from /Users/roberthead/.rvm/gems/ree-1.8.7-2011.03@shopdragon/gems/highscore-0.4.0/lib/highscore/content.rb:52:inkeywords’

use bloom filter for blacklist (for performance reasons)

The current approad to just use Array::include? is a serious bottleneck ...
This may be a good entry point: https://github.com/igrigorik/bloomfilter-rb

allow for multiple blacklist/whitelist adapters to be used

Current implementation only works with files, but should also work with

• relational databases
• simple key-value stores like Redis
• Memcache, etc.

Best possible implementation would be to have an adapter infrastructure that reads blacklisted items lazy (?)
Language shouldn't be ignored either!

allow multiple blacklists (per language)

• refactor API, add add_wordlist method for Highscore::Content to add blacklist, Parameters: Blacklist object and language
• add CLI options for every possible language

add a new method merge to Keywords class

to merge in keywords from another piece of text ...

Bloomfilter bucket gets filled up depending on size of list and raises exception

Hi there,

Using the words here as a blacklist, I get the following error when trying to initialize it.

Steps to reproduce

words = YAML.load_file 'stopwords.yml'
Highscore::Blacklist.load(words)

RuntimeError: bucket got filled up
    ./bloomfilter-rb-2.1.1/lib/bloomfilter/native.rb:24:in `insert'
    ./bloomfilter-rb-2.1.1/lib/bloomfilter/native.rb:24:in `insert'
    ./highscore-1.2.0/lib/highscore/wordlist.rb:115:in `block in init_bloom_filter'
    ./highscore-1.2.0/lib/highscore/wordlist.rb:47:in `block in each'
    ./highscore-1.2.0/lib/highscore/wordlist.rb:47:in `each'
    ./highscore-1.2.0/lib/highscore/wordlist.rb:47:in `each'
    ./highscore-1.2.0/lib/highscore/wordlist.rb:115:in `init_bloom_filter'
    ./highscore-1.2.0/lib/highscore/wordlist.rb:41:in `initialize'

Here are the params used for the bloomfilter creation when that list is supplied.

{ :size => 22800, :bucket => 4, :raise => true, :hashes => 28 }

use Ruby 2.0's lazy Enumerators for Keywords

Check if Ruby 2.0's lazy enumerator makes sense to use for the Keywords object.

lambda method for blacklisting

In addition to the normal blacklist functionality, there should be a method, where one can pass a block or lambda into and it evaluatues whether to ignore the word or not.

Use case: all words that contain more than 2 numbers should be ignored:

c = Highscore::Content.new(url)
c.configure do
  set :ignore, lambda { |w| w.gsub(/[^0-9]/, '').length > 2 }
end