[Discussion] multi-instance support about puppet-elasticsearch

[jsnod]

This would be useful even for local development where you'd like to emulate a production cluster environment on localhost.

[electrical]

Hi @afx114 how would you see the instance creation part?
I was thinking of the following:

elasticsearch::instance { 'instance-a':
  config => { }
}

elasticsearch::instance { 'instance-b':
  config => { }
}

These will create their own directory for the config and init files.

[ogavrisevs]

Really don't see why add more than one ES to one VM because in production even in cluster you will operate (1:1) one hardware (VM) = one ES. And with Vagrant already you can set a multi-VM, multi-ES cluster locally, just using small OS install footprints. For example https://github.com/kenny13/elasticsearch/blob/master/Vagrantfile with "vagrant up /localInstance[1-3]/" you will end up with ES cluster of three VM's .

[electrical]

I do know that in some situations people in production have very powerful machines ( 32 cores, 128GB mem or more )
in some of those cases people could run more then 1 instance on 1 hardware machine.

[jsnod]

Plus, it's much easier to just provision using @electrical 's proposal above than having to deal with multiple VMs.

[mrbanzai]

On a support call with Elasticsearch yesterday, it was recommended that we double the RAM on our data nodes (to 128GB) and run two instances per-node. We were thinking of a custom multi-instance initscript (overwriting the RPM-provided one) and separate configs (to provide different data and logs directories, as well as config to ensure the primary and replica dataset aren't both stored on the same host, etc.), but I don't have a good idea of the wider concerns. Perhaps managing the instances as separate services is preferable, but potentially more confusing.

What I'm most concerned with at the moment is trying to work within the RPM-provided skeleton (/etc/sysconfig/elasticsearch, /etc/init.d/elasticsearch) while also adding a second instance. So that would require either being able to update the initscript to support multiple instances with different configs, or configuring one instance the "normal" way and being able to tack on an additional instance with a new initscript + defaults.

I know that systemd can handle multi-instance support moderately easily, but I'm stuck in SysV world for better or worse.

[electrical]

Im currently in the process of making a design document for the multi-instance support.
There is only 1 issue with the current plan: it will be a non backwards compatible release.

The idea is to have multi instance support in this form:

class { ‘elasticsearch’:
  manage_repo => true,
  repo_version => ‘1.2’,
  config => { cluster.name’ => ‘clustername’ }
}

elasticsearch::instance { ‘instance_a’:
  config => { ‘node.name’ => ‘instance_a’ }
}

elasticsearch::instance{ ‘instance_b’:
  config => { ‘node.name’ => ‘instance_b’ }
}

The downside of this is that when only a single instance is required, you still need to define an instance since we can't detect automatically if an instance is defined or not by the user.

The only way would be to add an option multi_instance => true/false to the class to create the default instance or not.

What are your opinions about this?

[abaddononion]

As part of my personal preference, I very much don't like making the "simple use cases" more complex. So I'm a fan of the notion of having the multi_instance flag, and having it default to "false". And when it's false, it rolls out the default instance for you.

And then I would say if you set the multi_instance flag to "true", you would then need to define all of your instances at that point.

I mean... I guess in theory with that model you could leave multi_instance set to "false", and then define other elasticsearch::instance objects anyway? It would create a potentially confusing scenario where you SAY you're not doing multi_instance, and then really you are... but I don't think that's a deal-breaker, personally.

And if you want to have multiple instances, it seems to me that having to know to add multi_instance => true; and then define all of your instances by hand, really isn't that unexpected of a thing.

Unless I'm missing something, having the multi_instance flag and defaulting it to "false" seems most elegant way to go here. Keeps the simple use cases simple, and the more complex ones... still pretty intuitive.

[jsnod]

Agreed with @abaddononion -- since multiple instances seems like such an edge-case, the configs for it should probably be edge-case as well. Best to keep the non-edge-case stuff simple.

[jordansissel]

I'm in favor of of requiring one elasticsearch::instance resource per ES instance because:

Objectively (hopefully, anyway)

• it permits use with hiera more easily
• it permits creation of any number of ES nodes using create_resource()

Given the above, I am in favor of this being the minimal declaration:

class { "elasticsearch": ... ; }
elasticsearch::instance { "node-name-here": ; }

Subjectively:

• I prefer explicit declarations rather than implicit. This aids in maintenance and readability later. This means I want you to use elasticsearch::instance for every ES node you want to run.
• If you move from 1 instance to 2 instances, it is less work and more obvious if we require elasticsearch::instance explicitly. "less work" because with this method, you can simply create a 2nd resource:

# Add a 'node2' !
elasticsearch::instance { ["node1", "node2"]: ... ; }

But with the multi_instance world, you have to do two things: First, set multi_instance => false and then create two new elasticsearch::instance resources.

Maintenance feels easier to me if we require an instance for single node deployments.

[neoice]

+1 @ #62 (comment)

at $work, we try to construct our in-house modules such that you instantiate a "named instance" of a service. we've found this pattern to be extremely powerful for creating complex infrastructures and that it has long-term maintenance benefits.

[electrical]

Been busy with the rewrite for the multi instance support.
Progress can be found in the multi_instance_rewrite branch.

The main implementation is working and currently running extensive system tests.
Still need to fix the rspec tests.

Hope to have it released in a few days.

[electrical]

Forgot to update the thread.
The Multi instance support was released in version 0.4.0

Cheers.