Current --tail logic is unreliable for multi node clusters or multiple data sources with possible delays about es-utils

[clintongormley]

One correction: the refresh_interval is not an upper bound. It merely determines how often a refresh will be started. While refreshes are usually very fast, it is possible that they can take considerably longer than 1s, for instance when using eager global ordinals on very high cardinality fields.

We are planning to add a changes API which will solve this problem. Unfortunately, this depends on first adding sequence IDs, which is quite a complicated task, so will take a while to make this happen.

[reyjrar]

I'll look into one of those solutions for the next release.

[reyjrar]

I haven't forgotten about this, but really, the only way to cleanly and completely accomplish it would be to have the equivalent of a SERIAL or PRIMARY KEY AUTO_INCREMENT. There's no clean way to ask ElasticSearch "give me all the documents you indexed after this one." It's a free-text search engine, not a traditional RDBMS. All the solutions I've come up with aren't "complete" or would require an in-memory cache of all seen documents and essentially record all ID's of documents seen and not advance to the next minute until all the previous minutes documents had been seen.

Given that, --tail is still interesting for low frequency events or to know "hey, yes, I'm indexing data" so I'm leaving it in the distribution. But there's no easy way to fix this.