GithubHelp home page GithubHelp logo

baygeldin / tantiny Goto Github PK

View Code? Open in Web Editor NEW
212.0 2.0 7.0 61 KB

Tiny full-text search for Ruby powered by Tantivy

License: MIT License

Ruby 61.12% Shell 0.11% Rust 38.77%
ruby rust search-engine tantivy

tantiny's Introduction

Build workflow Tantiny Maintainability Test Coverage

Warning

The gem is not currently maintained and the development is put on hold. If you're interested in taking over, feel free to reach out to me.

Tantiny

Need a fast full-text search for your Ruby script, but Solr and Elasticsearch are an overkill? 😏

You're in the right place. Tantiny is a minimalistic full-text search library for Ruby based on Tantivy (an awesome alternative to Apache Lucene written in Rust). It's great for cases when your task at hand requires a full-text search, but configuring a full-blown distributed search engine would take more time than the task itself. And even if you already use such an engine in your project (which is highly likely, actually), it still might be easier to just use Tantiny instead because unlike Solr and Elasticsearch it doesn't need anything to work (no separate server or process or whatever), it's purely embeddable. So, when you find yourself in a situation when using your search engine of choice would be tricky/inconvinient or would require additional setup you can always revert back to a quick and dirty solution that is nontheless flexible and fast.

Tantiny is not exactly Ruby bindings to Tantivy, but it tries to be close. The main philosophy is to provide low-level access to Tantivy's inverted index, but with a nice Ruby-esque API, sensible defaults, and additional functionality sprinkled on top.

Take a look at the most basic example:

index = Tantiny::Index.new("/path/to/index") { text :description }

index << { id: 1, description: "Hello World!" }
index << { id: 2, description: "What's up?" }
index << { id: 3, description: "Goodbye World!" }

index.reload

index.search("world") # 1, 3

Installation

Add this line to your application's Gemfile:

gem "tantiny"

And then execute:

$ bundle install

Or install it yourself as:

$ gem install tantiny

You don't have to have Rust installed on your system since Tantiny will try to download the pre-compiled binaries hosted on GitHub releases during the installation. However, if no pre-compiled binaries were found for your system (which is a combination of platform, architecture, and Ruby version) you will need to install Rust first.

Warning

Only Rust versions up to 1.77 are supported. See this issue for more details.

Important

Please, make sure to specify the minor version when declaring dependency on tantiny. The API is a subject to change, and until it reaches 1.0.0 a bump in the minor version will most likely signify a breaking change.

Defining the index

You have to specify a path to where the index would be stored and a block that defines the schema:

Tantiny::Index.new "/tmp/index" do
  id :imdb_id
  facet :category
  string :title
  text :description
  integer :duration
  double :rating
  date :release_date
end

Here are the descriptions for every field type:

Type Description
id Specifies where documents' ids are stored (defaults to :id).
facet Fields with values like /animals/birds (i.e. hierarchial categories).
string Fields with text that are not tokenized.
text Fields with text that are tokenized by the specified tokenizer.
integer Fields with integer values.
double Fields with float values.
date Fields with either DateTime type or something that converts to it.

Managing documents

You can feed the index any kind of object that has methods specified in your schema, but plain hashes also work:

rio_bravo = OpenStruct.new(
  imdb_id: "tt0053221",
  type: '/western/US',
  title: "Rio Bravo",
  description: "A small-town sheriff enlists a drunk, a kid and an old man to help him fight off a ruthless cattle baron.",
  duration: 141,
  rating: 8.0,
  release_date: Date.parse("March 18, 1959")
)

index << rio_bravo

hanabi = {
  imdb_id: "tt0119250",
  type: "/crime/Japan",
  title: "Hana-bi",
  description: "Nishi leaves the police in the face of harrowing personal and professional difficulties. Spiraling into depression, he makes questionable decisions.",
  duration: 103,
  rating: 7.7,
  release_date: Date.parse("December 1, 1998")
}

index << hanabi

brother = {
  imdb_id: "tt0118767",
  type: "/crime/Russia",
  title: "Brother",
  description: "An ex-soldier with a personal honor code enters the family crime business in St. Petersburg, Russia.",
  duration: 99,
  rating: 7.9,
  release_date: Date.parse("December 12, 1997")
}

index << brother

In order to update the document just add it again (as long as the id is the same):

rio_bravo.rating = 10.0
index << rio_bravo

You can also delete it if you want:

index.delete(rio_bravo.imdb_id)

Transactions

If you need to perform multiple writing operations (i.e. more than one) you should always use transaction:

index.transaction do
  index << rio_bravo
  index << hanabi
  index << brother
end

Transactions group changes and commit them to the index in one go. This is dramatically more efficient than performing these changes one by one. In fact, all writing operations (i.e. << and delete) are wrapped in a transaction implicitly when you call them outside of a transaction, so calling << 10 times outside of a transaction is the same thing as performing 10 separate transactions.

Concurrency and thread-safety

Tantiny is thread-safe meaning that you can safely share a single instance of the index between threads. You can also spawn separate processes that could write to and read from the same index. However, while reading from the index should be parallel, writing to it is not. Whenever you call transaction or any other operation that modify the index (i.e. << and delete) it will lock the index for the duration of the operation or wait for another process or thread to release the lock. The only exception to this is when there is another process with an index with an exclusive writer running somewhere in which case the methods that modify the index will fail immediately.

Thus, it's best to have a single writer process and many reader processes if you want to avoid blocking calls. The proper way to do this is to set exclusive_writer to true when initializing the index:

index = Tantiny::Index.new("/path/to/index", exclusive_writer: true) {}

This way the index writer will only be acquired once which means the memory for it and indexing threads will only be allocated once as well. Otherwise a new index writer is acquired every time you perform a writing operation.

Searching

Make sure that your index is up-to-date by reloading it first:

index.reload

And search it (finally!):

index.search("a drunk, a kid, and an old man")

By default it will return ids of 10 best matching documents, but you can customize it:

index.search("a drunk, a kid, and an old man", limit: 100)

You may wonder, how exactly does it conduct the search? Well, the default behavior is to use smart_query search (see below for details) over all text fields defined in your schema. So, you can pass the parameters that the smart_query accepts right here:

index.search("a dlunk, a kib, and an olt mab", fuzzy_distance: 1)

However, you can customize it by composing your own query out of basic building blocks:

popular_movies = index.range_query(:rating, 8.0..10.0)
about_sheriffs = index.term_query(:description, "sheriff")
crime_movies = index.facet_query(:cetegory, "/crime")
long_ass_movies = index.range_query(:duration, 180..9999)
something_flashy = index.smart_query(:description, "bourgeoisie")

index.search((popular_movies & about_sheriffs) | (crime_movies & !long_ass_movies) | something_flashy)

I know, weird taste! But pretty cool, huh? Take a look at all the available queries below.

Supported queries

Query Behavior
all_query Returns all indexed documents.
empty_query Returns exactly nothing (used internally).
term_query Documents that contain the specified term.
fuzzy_term_query Documents that contain the specified term within a Levenshtein distance.
phrase_query Documents that contain the specified sequence of terms.
regex_query Documents that contain a term that matches the specified regex.
prefix_query Documents that contain a term with the specified prefix.
range_query Documents that with an integer, double or date field within the specified range.
facet_query Documents that belong to the specified category.
smart_query A combination of term_query, fuzzy_term_query and prefix_query.

Take a look at the signatures file to see what parameters do queries accept.

Searching on multiple fields

All queries can search on multuple fields (except for facet_query because it doesn't make sense there).

So, the following query:

index.term_query(%i[title description], "hello")

Is equivalent to:

index.term_query(:title, "hello") | index.term_query(:description, "hello")

Boosting queries

All queries support the boost parameter that allows to bump documents position in the search:

about_cowboys = index.term_query(:description, "cowboy", boost: 2.0)
about_samurai = index.term_query(:description, "samurai") # sorry, Musashi...

index.search(about_cowboys | about_samurai)

smart_query behavior

The smart_query search will extract terms from your query string using the respective field tokenizers and search the index for documents that contain those terms via the term_query. If the fuzzy_distance parameter is specified it will use the fuzzy_term_query. Also, it allows the last term to be unfinished by using the prefix_query.

So, the following query:

index.smart_query(%i[en_text ru_text], "dollars рубли eur", fuzzy_distance: 1)

Is equivalent to:

t1_en = index.fuzzy_term_query(:en_text, "dollar")
t2_en = index.fuzzy_term_query(:en_text, "рубли")
t3_en = index.fuzzy_term_query(:en_text, "eur")
t3_prefix_en = index.prefix_query(:en_text, "eur")

t1_ru = index.fuzzy_term_query(:ru_text, "dollars")
t2_ru = index.fuzzy_term_query(:ru_text, "рубл")
t3_ru = index.fuzzy_term_query(:ru_text, "eur")
t3_prefix_ru = index.prefix_query(:ru_text, "eur")

(t1_en & t2_en & (t3_en | t3_prefix_en)) | (t1_ru & t2_ru & (t3_ru | t3_prefix_ru))

Notice how words "dollars" and "рубли" are stemmed differently depending on the field we are searching. This is assuming we have en_text and ru_text fields in our schema that use English and Russian stemmer tokenizers respectively.

About regex_query

The regex_query accepts the regex pattern, but it has to be a Rust regex, not a Ruby Regexp. So, instead of index.regex_query(:description, /hel[lp]/) you need to use index.regex_query(:description, "hel[lp]"). As a side note, the regex_query is pretty fast because it uses the fst crate internally.

Tokenizers

So, we've mentioned tokenizers more than once already. What are they?

Tokenizers is what Tantivy uses to chop your text onto terms to build an inverted index. Then you can search the index by these terms. It's an important concept to understand so that you don't get confused when index.term_query(:description, "Hello") returns nothing because Hello isn't a term, but hello is. You have to extract the terms from the query before searching the index. Currently, only smart_query does that for you. Also, the only field type that is tokenized is text, so for string fields you should use the exact match (i.e. index.term_query(:title, "Hello")).

Specifying the tokenizer

By default the simple tokenizer is used, but you can specify the desired tokenizer globally via index options or locally via field specific options:

en_stemmer = Tantiny::Tokenizer.new(:stemmer)
ru_stemmer = Tantiny::Tokenizer.new(:stemmer, language: :ru)

Tantiny::Index.new "/tmp/index", tokenizer: en_stemmer do
  text :description_en
  text :description_ru, tokenizer: ru_stemmer
end

Simple tokenizer

Simple tokenizer chops the text on punctuation and whitespaces, removes long tokens, and lowercases the text.

tokenizer = Tantiny::Tokenizer.new(:simple)
tokenizer.terms("Hello World!") # ["hello", "world"]

Stemmer tokenizer

Stemmer tokenizers is exactly like simple tokenizer, but with additional stemming according to the specified language (defaults to English).

tokenizer = Tantiny::Tokenizer.new(:stemmer, language: :ru)
tokenizer.terms("Привет миру сему!") # ["привет", "мир", "сем"]

Take a look at the source to see what languages are supported.

Ngram tokenizer

Ngram tokenizer chops your text onto ngrams of specified size.

tokenizer = Tantiny::Tokenizer.new(:ngram, min: 5, max: 10, prefix_only: true)
tokenizer.terms("Morrowind") # ["Morro", "Morrow", "Morrowi", "Morrowin", "Morrowind"]

Retrieving documents

You may have noticed that search method returns only documents ids. This is by design. The documents themselves are not stored in the index. Tantiny is a minimalistic library, so it tries to keep things simple. If you need to retrieve a full document, use a key-value store like Redis alongside.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake build to build native extensions, and then rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

We use conventional commits to automatically generate the CHANGELOG, bump the semantic version, and to publish and release the gem. All you need to do is stick to the convention and CI will take care of everything else for you.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/baygeldin/tantiny.

License

The gem is available as open source under the terms of the MIT License.

tantiny's People

Contributors

baygeldin avatar github-actions[bot] avatar mfkp avatar takahashim avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

tantiny's Issues

How do I release Lock?

I got this error:

Tantiny::TantivyError (Failed to acquire Lockfile: LockBusy. Some("Failed to acquire index lock. If you are using a regular directory, this means there is already an IndexWriter working on this Directory, in this process or in a different process."))

How can I release the lock?

in memory indexing

hello everyone, thanks for your work on this gem.
I'm looking forward to use it but our use-case is a bit different: we need to filter ~100 contents at a time, then we delete the index and go with the other 100 contents, and so on for thousands of times a day.

I was wondering, is it possible to use in memory indexing instead of writing on file?

I saw that tantivy supports this use-case
https://docs.rs/tantivy/latest/tantivy/struct.Index.html#method.create_in_ram

I've never worked with Ruby and Rust extensions but I can dive into a solution if you tell me that's feasible

UPDATE: I've been playing a bit with the code and it seems that this change does most of the job
a-chris@9c47fc0

Is it something you would like to have on tantiny? Maybe making it configurable while creating a new Index object?

Error converting to String

Just pulled the repo and getting this error. Using Ruby 3.3.3

Steps:

bin/setup
rake build
rake spec
=>
TypeError:                                                                                                                                                
        Error converting to String                                                                                                                              
      # ./lib/tantiny/tokenizer.rb:15:in `__new_stemmer_tokenizer'                                                                                              
      # ./lib/tantiny/tokenizer.rb:15:in `new'                         

Failed to build gem

Hello, I can't install the gem (main branch) on mac osx and ubuntu. Both failing in the same ways.

Installing tantiny 0.3.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/teo/.asdf/installs/ruby/3.0.3/lib/ruby/gems/3.0.0/gems/tantiny-0.3.0/ext
/Users/teo/.asdf/installs/ruby/3.0.3/bin/ruby -I/Users/teo/.asdf/installs/ruby/3.0.3/lib/ruby/3.0.0 -rrubygems
/Users/teo/.asdf/installs/ruby/3.0.3/lib/ruby/gems/3.0.0/gems/rake-13.0.6/exe/rake
RUBYARCHDIR\=/Users/teo/.asdf/installs/ruby/3.0.3/lib/ruby/gems/3.0.0/extensions/x86_64-darwin-21/3.0.0/tantiny-0.3.0
RUBYLIBDIR\=/Users/teo/.asdf/installs/ruby/3.0.3/lib/ruby/gems/3.0.0/extensions/x86_64-darwin-21/3.0.0/tantiny-0.3.0
rake aborted!
ArgumentError: wrong number of arguments (given 2, expected 1)

(See full trace by running task with --trace)

Am I miss something?

Aggregations

Hi @baygeldin, this is an awesome project! Do you have any plans to add support for aggregations?

Error loading library

Hello! My attempt to require the gem causes a LoadError:

$ gem install tantiny

Fetching tantiny-0.3.1.gem
Building native extensions. This could take a while...
Successfully installed tantiny-0.3.1
Parsing documentation for tantiny-0.3.1
Installing ri documentation for tantiny-0.3.1
Done installing documentation for tantiny after 0 seconds
1 gem installed

$ irb

irb(main):001:0> require "tantiny"
/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.rb:18:in `require_relative': cannot load such file -- /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.so (LoadError)
	from /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.rb:18:in `<top (required)>'
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from (irb):1:in `<main>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `load'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `<main>'
<internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- tantiny (LoadError)
	from <internal:/Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from (irb):1:in `<main>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `load'
	from /Users/alex/.asdf/installs/ruby/3.1.0/bin/irb:25:in `<main>'

tantiny.so actually exists:

ls -al /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.so
-rwxr-xr-x  1 alex  staff  3706829 Apr 18 21:46 /Users/alex/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/tantiny-0.3.1/lib/tantiny.so

I'm running Ruby on MacBook Pro M1. Rust is installed from Homebrew (brew install rust).

Term types

Are there plans to support term queries for types other than strings?

Custom tokenizer

I want to use Tantiny with Japanese. There are several Tantivy tokenizers for Japanese language. I'm now considering lindera-tantivy which supports not only Japanese but also Chinese and Korean. Is it possible to use these custom tokenizers with Tantivy via Tantiny?

Returning the match indexes along with results?

Hello, first of all, thanks for publishing this, looks to be a very interesting gem.

I'm wondering if it would be possible to return the matched text (or index of matched characters) along with the results? This would be useful in cases where a fuzzy match returns results, and I would want to highlight the matching text (or just show a snippet of text around the matching text for context) in the results.

In the readme it says:

"You may have noticed that search method returns only documents ids. This is by design. The documents themselves are not stored in the index."

So maybe this is not possible, but I figured I would ask anyway:

From your example:

brother = {
  imdb_id: "tt0118767",
  type: "/crime/Russia",
  title: "Brother",
  description: "An ex-soldier with a personal honor code enters the family crime business in St. Petersburg, Russia.",
  duration: 99,
  rating: 7.9,
  release_date: Date.parse("December 12, 1997")
}

Right now, this is how the search returns:

index.search('bersonal coder', fuzzy_distance: 1)
=> ["tt0118767"]

It would be great to return something like:

index.search('bersonal coder', fuzzy_distance: 1)
=> 
[{
  "tt0118767": {
    match_ranges: [[21, 28], [36, 39]]
  }
}]

That way I could display the results in my search listing like:

An ex-soldier with a personal honor code enters the family crime business in St. Petersburg, Russia.

I haven't dug into the source yet to see if this is possible, but I figure you'd know the limitations better and might be able to provide some input on if this is feasible.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.