Allow Embedded ElasticSearch instance to install plugins like cloud-aws, cloud-gce, etc. about logstash

[colinsurprenant]

thanks for documenting your workaround for this. running embedded elasticseach is typically only meant for local quick & easy prototyping and testing, it wasn't mean for connecting to a cluster or even installing plugins. I suggest you install an external elasticsearch instance and use it to install your plugins and connect to your cluster. Let me know if this makes sense.

[colinsurprenant]

Will go ahead and close this.

[shurane]

So Logstash uses ES to connect to a cluster anyhow. I'm talking about:

output {
    elasticsearch { cluster => "logstash" }
    stdout { codec => rubydebug }
}

That relies on the ES that's bundled with Logstash, as far as I can tell.

I'm not sure how else Logstash is supposed to connect to an ES cluster on AWS or GCE.

Although, I know that @imperialwicket from IRC got it up and running another way, though.

[colinsurprenant]

Ok, there are 2 things here:

• starting/using an embedded ES server. A local ES server will be started locally and data can be indexed in this local ES.
• logstash using the ES libraries (known as the node or transport protocols) to communicate with a remote ES server.

when using:

output {
  elasticsearch {
    host => "some_host"
    cluster => "logstash"
  }
}

logstash will send data to a ES server running on some_host using the node protocol (which is the default, using port 9300). You can also use the http protocol to talk to a remote ES server using the protocol => "http" option.

Does this make sense?

[shurane]

Alright, that does make more sense. The embedded ES server uses the same ES libraries that are used for the node or transport protocols when logstash communicates to an ES cluster, correct? Thanks, clears up a bit.

I think what I was trying to say is that with an ES cluster on AWS EC2, logstash cannot discover the cluster using the ES libraries, as by default, the ES cluster on AWS EC2 discovers each other not with multicast/unicast/IP addresses, but using the cloud-aws plugin.

I hope that clears things up a bit.

[colinsurprenant]

That's right, logstash includes the ES jars which are used either to spin an embedded ES server or for providing the node/transport protocol.

For speaking to your AWS cluster, you should be able to connect with both node/transport or http, given you expose the correct tcp ports (by default 9200 for http and 9300 for node/transport). Unless you have particular reason for using node/transport, I'd suggest you use http when connecting to a remote cluster.

Let me know if that help.

[joekiller]

For anyone trying to run the cloud-aws plugin I solved it via: elastic/elasticsearch-cloud-aws#71 (comment)

[jlintz]

Just ran into this issue as well. Would really love the ability to easily install the cloud-aws plugin to have it connect to my existing ES cluster. Was unable to find any documentation on how to go about doing so, and this github issue was the only reference I've found

[vdbrooks]

+1 for @jlintz comment. I am implementing this today on AWS and also need the ability to install the cloud-aws plugin via Logstash. Have some hope that @joekiller solution will be the answer.

[jlintz]

@vdbrooks I ended up just setting up a non-data ES instance with the cloud-aws plugin configured and then pointed logstash at localhost to workaround this. It was a bit more straight forward to handle with the puppet modules

[joekiller]

The easiest way for me to make this work is to install ElasticSearch 1.1.1 from the binaries, install the cloud-aws plugin via their bin/plugin script and then add to my /etc/sysconfig/logstash entry the following:

LS_JAVA_OPTS="$LS_JAVA_OPTS -Des.path.home=/usr/local/elasticsearch -Des.config=/opt/logstash/elasticsearch.yml"

The LS_JAVA_OPTS is picked up by bin/logstash and appended to the startup. You can modify anything that ElasticSearch is looking for by checking out the following reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-dir-layout.html

I just set my es.path.home to the root of the elasticsearch install because that allows it to resolve the {es.home}/plugins path as well which is easier than copying stuff all over the place.

YOU MUST MATCH ES BINARY VERSIONS FOR THIS TO WORK. IE ES 1.1.1 is what Logstash 1.4.2 uses. This all works because logstash's internal ElasticSearch jar file is referenced first so the only jars that are loaded afterwards are those which are not present at startup.

Side note you can see from elasticsearch's bin/plugin file that it is really just a java call that they are executing (https://github.com/elasticsearch/elasticsearch/blob/master/bin/plugin). Someone could feasibly use the jar files in logstash's vendor/jar/elasticsearch/lib directory to make the proper org.elasticsearch.plugins.PluginManager call but I'm not bothering because I'm installing a mirror ES 1.1.1 binary on the machine along with logstash to get the plugin. Different strokes people but in the end, use the java opts to make logstash do your dance.