darkhelmet / fetch

A full text search engine written in Go

fetch

What is it?

fetch is a full text search engine and server written in go

How?

POST '{ "title": "My Blog Post", "body": "Ruby is the bomb..." }' /blog/posts/5 (/:index/:scope/:id)

Pipeline

JSON parts → Tokenizer → Analyzer → Storage

Tokenzier

String -> TokenChan

Analyzer

Tokens maybe have a type/flag/something and analyzers only work on certain filters?

TokenChan -> email -> superstrip(invalid characters, ascii, strip, lowercase) -> stopword -> stemming -> double_metaphone -> ngram? -> TokenChan

Storage

storage.Index(index, scope, id, field, tokens)

storage.Search(index, scope, query) # -> List of ids

Quick start

• Install go (In the Fetch the repository step, use weekly instead of release)
• Run ./all.bash from the src directory

Bugs

Submit bugs at my FogBugz

Hacking

1. Install gospec
2. Install git-flow
3. Fork the repo, clone, and run ./setup.bash
4. Use git-flow to start a feature: git flow feature start my-feature
5. Write code and tests
6. Stage files (so git ls-files picks them up) and run ./src/format.bash to format your source files according to gotfmt and my settings. DO THIS BEFORE COMMITTING.
7. Commit
8. Repeat steps 5-7 until feature is complete.
9. Use git-flow to finish the feature: git flow feature finish my-feature
10. Send a pull request.

The most important parts are tests and gofmt. If you don’t have tests, or didn’t run the formatting script, I’ll tell you to go do that. If you continually ignore this hacking workflow, I’ll start to continually ignore your pull requests!

License and Authors

fetch is licensed under the terms of the Apache License, Version 2.0. See LICENSE for details.

fetch was created by Daniel Huckstep. Type git shortlog -s for a full list of contributors.

The Porter Stemming algorithm in stemming.h is written by Martin Porter.

The Double Metaphone algorithm in double_metaphone.h is written by Maurice Aubrey and used with permission to “do what [I] want with it”.

fetch's Issues

Convert accented characters to ascii equivalents in superstrip filter

Substring filter

porter stemming filter

http://tartarus.org/~martin/PorterStemmer/

ASCII filter

• remove non-ascii
• convert non-ascii

PostgreSQL storage support

Doesn't index words separated by slashes properly

Indexing http://blog.darkhax.com/2009/08/30/threaded-awesome and searching for "debugging" doesn't work.

File based configuration

JSON

Use git-flow

https://github.com/nvie/gitflow

Lowercase filter

Query parser

Handling:

• AND
• OR
• Field specific items

double metaphone filter

Riak storage support

Reject blank tokens in superstrip filter

superstrip filter

Combine ascii, lowercase, and punctuation since it's probably something you'll do most of the time.

"percolation" like elasticsearch

Notify a web hook when a document is indexed that matches (a) term(s)

Better tokenizer

Use a lexer/scanner or something to actually pull out words, instead of just splitting on whitespace.

Web frontend

https://github.com/hoisie/web.go

Relevance in Redis storage engine

Use a sorted set, each token has a score equal to the number of occurrences in the field.

metaphone3 filter

If it can get it sorted out.

Profiling flags

http://blog.golang.org/2011/06/profiling-go-programs.html

Consider log based algorithm for scoring

Something that tapers off so a million occurrences only count so many times.

Tests

omg tests

Refactor out vector in favor of just using builtin append

Strip useless characters

punctuation, etc.

Snowball based english stemmer

Ignored fields

Fields starting with an underscore should be ignored.

Stemmer tests for agreement and happiness

Agreement should stay the say, happiness should probably go back to happy.

MySQL storage support

Logging

Configurable

email filter

Extract emails and separate them out.

Test to ensure double_metaphone filter actually returns a proper secondary encoding

Because of the weird way I do it.

ngram filter

Requires more research

Stopword filter

Update README to reflect v0.0.1

Levenshtein distance filter

Seems like it could be interesting to handle spelling mistakes and stuff....

Will require research.

Simple tokenizer

Basically just split on whitespace

Redis storage support

Ability to boost relevance for certain fields

Investigate build options

• https://github.com/skelterjohn/go-gb
• http://code.google.com/p/gobuild/