imperialwicket / elasticsearch-logstash-index-mgmt Goto Github PK

Hi,

I use elasticsearch and logstash for several website. For each website I have 2 indexes per day. For example for my website www.ZZZZZZ.com there is one index named "logstash-ZZZZZZ_index-YYYY.MM.DD" and another one named "logstash-ZZZZZZ_failure-YYYY.MM.DD".

When I do :
elasticsearch-remove-old-indices.sh -i 1 -g logstash-ZZZZZZ_failure
i haven't any problem but when I do:
elasticsearch-remove-old-indices.sh -i 1 -g logstash-ZZZZZZ_index

I have this error message:
"No indices returned containing 'logstash-ZZZZZZ_index' from http://localhost:9200"
Although there is 3 index open in Elasticsearch:
logstash-ZZZZZZ_index.2013.07.27
logstash-ZZZZZZ_index.2013.07.28
logstash-ZZZZZZ_index.2013.07.29

i don't understand why I have this problem. Can you help me?

Restore script should be aware of both the current directory and the target tmp directory.

Hi,

Thank you for your help with my last problem.

For my different site, I keep open only one index, that of the current day. Every day my crontab closes all indexes of the day before. This part is okay.

I would also remove indexes older than 7 days. But "http://localhost:9200/_status?pretty=true" only gives list of open indexes. How can I have a list of all index?

It would be highly appreciate if someone could help or share any idea on it. Many thanks.

Regards,
Hubert

Looks like there is a problem when restoring. Mapping has changed on ES 2.
MapperParsingException[Failed to parse mapping [testing-2016.03.22]: Root mapping definition has unsupported parameters...

ElasticSearch used is 2.1.1 for testing this script.
As a test scenario:
curl -XPUT 'http://localhost:9200/testing-2016.03.22/' -d '{"settings":{"number_of_shards":5,"number_of_replicas":0},"mappings":{"testing-2016.03.22":{"mappings":{"doc":{"properties":{"date":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"sentence":{"type":"string"},"value":{"type":"long"}}}}}}}'

Some failures during restore result in partial restore states. When failures occur mid-restore, proper clean up should happen. (deleting the new ES-index, for example)

The awk expression in elasticsearch-remove-old-indices.sh does not work on Alpine Linux (musl based, http://wiki.alpinelinux.org/wiki/Awk).

Maybe can be substituted by a more simple `awk '/logstash/{print $3}'?

Should be more clear about optional backup techniques.

Remove 'S3CMD' in favor of something like 'BACKUP_COMMAND', and add a check to ensure that the supplied command is valid on the system.

This way s3cmd, boto, scp, cp, etc. all make more sense and using an alternative to s3cmd feels less hacky.

Need to be sure this works properly in the restoration process, too.

Hello.

The command ./elasticsearch-remove-old-indices.sh works like a charm. Thanks.

But I suggest you to add to your readme that you support too marvel and hekad.

In my case, I just ran

./elasticsearch-remove-old-indices.sh -e http://es.lxc:9200 -g marvel -i 1

And it works perfectly.

The last day of a month will get stored in the wrong location, since the backup is pulled based on yesterday, but then stored in a target directory based on today.

Intentionally supporting only nodes that have all data at the moment.

Discussion:
it looks like
https://github.com/dpippen/elasticsearch-logstash-index-mgmt/blob/master/elasticsearch-backup-index.sh
assumes a 1 or 2 node ES cluster
...
yeah, it would probably be a matter of grabbing routings and then making a worklist to run on each node
...
if you get the routing and filter for primary shard you might have a list of 7 nodes (some nodes end up with 2 primary shards)
to get it running somewhat naturally with the current code, i think i'd want a state entry in redis and strategically offset (via cron) bash scripts.
but i really don't feel good about that
wasn't archiving supposed to make it to ES by 1.0?
you could rely on the es notion of the state
dump the mappings, create a snapshot of node and shard assignments
each node is responsible for making a tar of its shards and copying to S3
so in my example there would be 7 tar files for the index on S3
do you want to be able to restore those via script?
the other 13 nodes would just exit, nothing to do
the restore would be similar, just pull down logstash-$DATE-*.tgz

Howdy!

Lovely tool you're working on! Any possibly channeling effort on this project into the Elasticsearch Curator tool, instead? It has similar goals to this project and is officially maintained by us at Elasticsearch. The project itself is fairly old (2 years? maybe more?) and supports lots of index management activities, with more on the way! We're adding snapshotting, shard routing, and other cool features very soon, and I would love your help in improving curator.

Thoughts?

Any idea when you will write for ES 5. Bocz in es5 it stored folder with uuid name