Index aliases about retire

[karmi]

Hi,

yes, support for index aliases is something I'd very much like to add soon.

Nevertheless, I haven't put any thoughts into how exactly the support should look like on the DSL and feature level.

I don't think Tire will ever support aliases in the Escargot sense.

The immediate reason is that I never really understood why Escargot bakes something like this so deeply into the library. Using and managing aliases is something the application is much better suited then a library such as Tire.
With no offence implied, the whole aliases support in Escargot seems particular to a certain use case, at least to my eyes.

Tire supports importing data into an arbitrary index, via Ruby API or a Rake task (see Readme).

Tire supports arbitrary index name for your model, via the index_name method.

While there should be an interface for managing aliases, I think Tire should stop there.

[rubish]

I have read the readme but it was not clear what exactly does rake task do. Also, if it creates new indexes whenever it is run, how are indexes, which are no longer in use handled.

As far as the support of aliases and DSL is concerned, you would be a much better judge, considering you have developed the gem with some philosophy in mind. The suggestion was something I very much liked in escargot and found missing in tire. I am happy if I can do something similar easily, even if it is not out of the box solution.

[karmi]

This task:

$ rake environment tire:import CLASS='Article' INDEX='articles-2011-05'

imports all your Article records into an index named articles-2001-05. It's your application responsibility, then, to manage these aliases.

It does not create a new index whenever it's run -- this maybe the case with Escargot, but not with Tire.

I'm wondering what are you referring to when saying:

The suggestion was something I very much liked in escargot and found missing in tire.

Maybe I'm simply not getting the support for aliases in Escargot. Based on the information in the Readme, I would consider its approach too complicated and, to be honest, not too practical. I'd be delighted to hear some argument to the contrary.

[rubish]

The approach taken by escargot may be complicated, but its seems practically very useful to me.

Usecase: Lets say I have some some index already created for the model. I change the mapping, which can be quite frequent if the project is in its initial stages and under active development. Much less frequent if the project has already matured. What escargot lets me do is, create a new index, while old index still exists and can be used to fulfill the search queries. Once new index is ready silently and seamlessly transfer over to new index. It also provides the task to delete any unused indexes that might be eating up the disk space.

There might be some more use cases, but this was the simplest which came to my mind.

[karmi]

@rubish: Hi, thanks for the update!

Yes, this a very common scenario. I would argue, however, that the approach taken by Escargot, where all of this is deeply baked into the application/models, is „too much” hand-holding. It's of course very convenient from the end users perspective.

Tire's approach to the scenario would be something like this:

1/ You have a model called Article, with no index_name set, so it defaults to articles
2/ You either not set any mapping, or set one just approximately.
3/ You notice that your mapping should be updated, while still prototyping the app.
4/ You update the mapping within your model, delete the ES articles index manually, and reindex everything (see the section on import in Readme)
5/ You start the app with new mapping in effect.

No magic, no aliases, no backround jobs.

It is, for sure, much more expensive to do stuff like this in production, where you don't want the downtime. There still is a way:

1/ You notice that the mapping for index with millions of documents needs updating.
2/ You manually create the index with correct mapping with a name like articles-2011-07. This is perfectly valid with a Tire::Index.create :mappings => ... call.
3/ You import all of your data into your newly created index.
4/ When the import is done, you either delete the old index, and create an alias articles pointing to articles-2011-07, or just change the index_name for your model.
5/ You restart the application with the new mapping in effect.

Again, no black magic, very straightforward, and under your control.

Now, can I imagine there's a better way around this? Sure! I would very much like a Tire::Index#reindex method, which would use the ElasticSearch's scan type of search, to reimport everything from the old index into new one. But in practice, both of those two approaches outlined above, are „good enough”...

[rubish]

I was talking about the second scenario you just mentioned above and the didn't thought of the alternative you just described, probably because of my preconceived notions that the gem should do everything.

I very much agree with what you suggest, but managing this process can be error prone. Say, i successfully completed the process on my local machine but made a typo while working on production server. That's of-course not your problem at all, but might be quite frustrating for the user.

Above approach would seem perfectly fine to me, if there is a migration system, where I can write a script on my local machine, test it thoroughly and while deploying on production just run the script. I know I can handle such things manually, but think of automated deployments using capistrano.

Please let me know your thoughts.

[karmi]

(...) probably because of my preconceived notions that the gem should do everything.

You're not alone :) Lots of people come to Tire, probably thinking it should shield them from nearly everything. But it won't, and it can't, since the world is a very complex place. Conditions vary widely. (See for instance tire#34.)

I think the proper way to think about all this reindexing, mapping updating stuff is this:

YOU WILL HAVE TO UPDATE THE MAPPING AND REINDEX EVERYTHING, EVENTUALLY*

:) That's just the simple truth. Once you get over it, you can fine tune the process to match your needs exactly, and not bend some library to do what you need to do.

Both of those approaches are valid, we have did both in our app in situations like these...

[jarosan]

This page comes up on top if you search google for "elasticsearch tire reindex". Just wanted to share a small rake task i wrote today for reindexing: https://gist.github.com/3124884

Maybe it would save somebody an hour or two :)

[mahemoff]

@jarosan That's incredibly useful. Thanks. I made a small change to handle multiple classes https://gist.github.com/3751744