elastic / helm-charts

You know, for Kubernetes

Elastic Stack Kubernetes Helm Charts

• Supported Configurations
  • Stack Versions
  • Kubernetes Versions
  • Helm Versions

These Helm charts are designed to be a lightweight way to configure Elastic official Docker images.

Warning When it comes to running the Elastic on Kubernetes infrastructure, we recommend Elastic Cloud on Kubernetes (ECK) as the best way to run and manage the Elastic Stack.

ECK offers many operational benefits for both our basic-tier and our enterprise-tier customers, such as spinning up cluster nodes that were lost on failed infrastructure, seamless upgrades, rolling cluster changes, and much much more.

With the release of the Elastic Stack Helm charts for Elastic version 8.5.1, we are handing over the ongoing maintenance of our Elastic Stack Helm charts to the community and contributors. This repository will finally be archived after 6 months time. Elastic Stacks deployed on Kubernetes through Helm charts will still be fully supported under EOL limitations.

Since we want to provide an even better experience for our customers by running the Elastic Stack on Kubernetes, we will continue maintaining the Helm charts applicable to ECK Custom Resources. These charts can be found in the ECK repository.

Helm charts will currently be maintained for ECK Enterprise-tier customers, however, we encourage the community to engage with the existing Helm charts for the Elastic Stack and continue supporting their ongoing maintenance.

See #1731 for more details.

Supported Configurations

We recommend that the Helm chart version is aligned to the version of the product you want to deploy, when a chart release exists for the given stack version. This will ensure that you are using a chart version that has been tested against the corresponding production version. This will also ensure that the documentation and examples for the chart will work with the version of the product, you are installing.

For example, if you want to deploy an Elasticsearch 7.7.1 cluster, use the corresponding 7.7.1 tag.

However, we don't expect to release new charts versions, so if a chart for the latest patch version doesn't exist, you can use the latest chart with the same MAJOR.MINOR version and override the Docker image tag to the latest patch version with the imageTag value.

For example, if you want to deploy an Elasticsearch 7.17.5 cluster, use the corresponding 7.17.3 tag, with imageTag=7.17.5 value.

Stack Versions

Chart	Latest 8 Version	Latest 7 Version	Latest 6 Version
APM Server	8.5.1 (Beta since 7.7.0)	7.17.3 (Beta since 7.7.0)	6.8.22 (Alpha)
Elasticsearch	8.5.1 (GA since 7.7.0)	7.17.3 (GA since 7.7.0)	6.8.22 (Beta)
Filebeat	8.5.1 (GA since 7.7.0)	7.17.3 (GA since 7.7.0)	6.8.22 (Beta)
Kibana	8.5.1 (GA since 7.7.0)	7.17.3 (GA since 7.7.0)	6.8.22 (Beta)
Logstash	8.5.1 (Beta since 7.5.0)	7.17.3 (Beta since 7.5.0)	6.8.22 (Beta)
Metricbeat	8.5.1 (GA since 7.7.0)	7.17.3 (GA since 7.7.0)	6.8.22 (Beta)

Kubernetes Versions

The charts are currently tested against all GKE versions available. The exact versions are defined under KUBERNETES_VERSIONS in helpers/matrix.yml.

Helm Versions

While we are checking backward compatibility, the charts are only tested with Helm version mentioned in helm-tester Dockerfile (currently 3.10.2).

helm-charts's Issues

[elasticsearch] "not enough master nodes" error with istio

Chart version: 6.6.0-alpha1
Istio version: 1.1.0-snapshot.6

When deploying the chart on a Istio cluster, I get the error "not enough master nodes discovered during pinging"

After some debugging looks like since Elasticsearch is using elasticsearch-master-headless service to discover the master nodes, this service only has defined the port 9200 in its service definition. Adding the transport port of 9300 fixed this error.

Please add your charts to https://hub.helm.sh

Imho it would be cool if the charts would be a bit promoted. Sp please add your charts to https://hub.helm.sh by creating a pr on: https://github.com/helm/hub/blob/master/config/repo-values.yaml

[Elasticsearch][8.0.0] don't find cluster nodes

Seems like it is not possible to find any node in the cluster, even do all are up and running checking for the other nodes.

{"type": "server", "timestamp": "2019-04-17T14:12:01,189+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304] from hosts providers and [{elasticsearch-master-0}{-6TRq956TxeZVWcFddKTKQ}{kLzc0NhoQj6v2LvAiCBd7A}{10.8.0.12}{10.8.0.12:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }

Changing the name to the whole headless name does not work eather elasticsearch-master-{0..3}.elasticsearch-master-headless.default.svc.cluster.local

{"type": "server", "timestamp": "2019-04-17T14:20:53,452+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0.elasticsearch-master-headless.default.svc.cluster.local, elasticsearch-master-1.elasticsearch-master-headless.default.svc.cluster.local, elasticsearch-master-2.elasticsearch-master-headless.default.svc.cluster.local] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304] from hosts providers and [{elasticsearch-master-0}{-6TRq956TxeZVWcFddKTKQ}{y5MQasEnQ2W-geYeRuaHyA}{10.8.0.13}{10.8.0.13:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }

similar to #72

[Bug Report] headless service name missconfigured

when I change the value of cluster name to "test", the headless service name is changed correctly, but the value of env "discovery.zen.ping.unicast.hosts" kept as "elasticsearch-master-headless", and i have to change it manually. here is my value.yaml:

clusterName: test
nodeGroup: master
replicas: 2
minimumMasterNodes: 2
antiAffinity: soft
esMajorVersion: 7
esJavaOpts: "-Xmx8g -Xms8g"

resources:
  requests:
    cpu: 1
    memory: 16Gi
  limits:
    cpu: 1
    memory: 16Gi

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: es-hdd-storage
  resources:
    requests:
      storage: "5Ti"

updateStrategy: RollingUpdate
maxUnavailable: 1

Operator support

Are there plans to implement Operator https://coreos.com/operators/ for deploying and managing ES clusters?

Memory locking is not enabled

Hi,
your doc recommend to enable memory locking at bootstrap - this is completely missing on the helm chart.

It's a bit tricky on kubernetes but it's possible:

1. enable the required capabilities

      securityContext:
        capabilities:
          add:
          - IPC_LOCK
          - SYS_RESOURCE
   

2. set ulimit before starting elasticsearch

      command:
        - /bin/sh
        - -cxe
        - |
          ulimit -l unlimited
          /usr/local/bin/docker-entrypoint.sh eswrapper
   

3. enable via environment

      env:
        - name: bootstrap.memory_lock
          value: "true"
   

This colud be configurable via helm variable.

kibana 6.6.0-alpha1 not matching the tag?

#17, according to GitHub, is in v6.6.0-alpha1:

However, the actual release seems to be missing this PR?

% grep -A2 kibana requirements.yaml 
  - name: kibana
    repository: https://helm.elastic.co
    version: 6.6.0-alpha1

% helm dep update
...
Downloading kibana from repo https://helm.elastic.co
...

% grep -A2 kibana requirements.lock
- name: kibana
  repository: https://helm.elastic.co
  version: 6.6.0-alpha1

% tar -zxf charts/kibana-6.6.0-alpha1.tgz -O kibana/templates/deployment.yaml
...
        readinessProbe:
{{ toYaml .Values.readinessProbe | indent 10 }}
...
                http "/app/kibana"

This doesn't match what was merged in the #17.

Thoughts?

Helm charts for APM

It would be great to have official helm charts for running APM Server on kubernetes.

Kibana: Readiness vs basePath

Hello,

When customising the basePath, Kibana expect according request, which make the readiness failing, and Kibana service never up.

It's here:

helm-charts/kibana/templates/deployment.yaml

Line 94 in 11c2317

http "/app/kibana"

When using this config:

          kibana.yml: |
            server.basePath: "/kibana"
            server.rewriteBasePath: true

We would need to have the readiness probe hitting http '/kibana/app/kibana'.

Thank you.

curl: (7) Failed connect to elasticsearch-master:9200; Connection refused

Hello,

I'm trying to follow Installing, yet running into following scenario:

mbp:~ alexus$ helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
mbp:~ alexus$ helm install --name elasticsearch elastic/elasticsearch --version 6.5.3-alpha1
NAME:   elasticsearch
LAST DEPLOYED: Thu Dec 20 21:32:34 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/StatefulSet
NAME                  DESIRED  CURRENT  AGE
elasticsearch-master  3        3        0s

==> v1/Pod(related)
NAME                    READY  STATUS   RESTARTS  AGE
elasticsearch-master-0  0/1    Pending  0         0s
elasticsearch-master-1  0/1    Pending  0         0s
elasticsearch-master-2  0/1    Pending  0         0s

==> v1beta1/PodDisruptionBudget
NAME                      MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
elasticsearch-master-pdb  N/A            1                0                    0s

==> v1/Service
NAME                           TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)            AGE
elasticsearch-master           ClusterIP  10.110.160.47  <none>       9200/TCP,9300/TCP  0s
elasticsearch-master-headless  ClusterIP  None           <none>       9200/TCP           0s


NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=default -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
  $ helm test elasticsearch

mbp:~ alexus$ 
mbp:~ alexus$ kubectl get pods --namespace=default -l app=elasticsearch-master -w
NAME                     READY     STATUS    RESTARTS   AGE
elasticsearch-master-0   0/1       Pending   0          25s
elasticsearch-master-1   0/1       Pending   0          25s
elasticsearch-master-2   0/1       Pending   0          25s

mbp:~ alexus$ helm test elasticsearch
RUNNING: elasticsearch-rxsef-test
FAILED: elasticsearch-rxsef-test, run `kubectl logs elasticsearch-rxsef-test --namespace default` for more info
Error: 1 test(s) failed
mbp:~ alexus$ kubectl logs elasticsearch-rxsef-test --namespace default
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0curl: (7) Failed connect to elasticsearch-master:9200; Connection refused
mbp:~ alexus$ 
mbp:~ alexus$ helm version
Client: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
mbp:~ alexus$ docker --version
Docker version 18.09.0, build 4d60db4
mbp:~ alexus$ uname -a
Darwin mbp.lan 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
mbp:~ alexus$ 

I'm using latest Desktop Docker (macOS).

Please advise.

Scope PodDisruptionBudget to the namespace?

After deploying the elasticsearch chart to one namespace, attempting to deploy it to a second namespace appears to fail with the following error:

Release "foo" does not exist. Installing it now.                                                                                            
Error: release draal failed: poddisruptionbudgets.policy "foo-data-pdb" already exists                                                      

There seems to be some conflict on the name. Should this resource be named to include the namespace where the chart is installed?

RFC: change extraEnvs to be a map instead of a list.

When using nested charts (parent/child/etc), it's often useful to add additional entries for values to pass into in child charts. However, extraEnvs is a list, and helm cannot merge lists, but it can merge maps! (Citation: https://github.com/jordansissel/experiments/tree/master/helm/subchart-values-from-parent)

By way of example, I might have one child chart which configures Elasticsearch with my favorite default features (enable security, enable tls, etc). And I could create a new chart where I want this, plus creating a SAML realm. Because extraEnvs is a list, I can't just inject these SAML-specific settings. Instead, I have to copy+paste the child chart's extraEnvs (enable security, enable tls, etc) and append my SAML ones.

If extraEnvs were a map, I believe this would be solved:

Example appearance in the child described in above:

extraEnvs:
  xpack.security.enabled: { value: true }
  ...
  ELASTIC_PASSWORD: 
    valueFrom: { secretKeyRef: { name: elastic-credentials, key: password } }

And the parent chart (to add SAML, for example)

extraEnvs:
  xpack.security.authc.realms.saml1.type: { value: true }
  ...

Thoughts? I know this would deviate from Kubernetes "environment variables are defined as an list", so I'm open to discussion.

Alternative approach

Stepping back, one alternative could be to have a general "settings" which could be used to generate a configmap instead of using environment variables for settings.

Helm chart version and docker image tag mismatch for 6.6.0

Seems the Docker imageTag is set to "6.6.0" in the '6.6.0-alpha1' helm chart.

Is the '6.6.0-alpha1' chart a stable '6.6.0' application release?

Timeout expired waiting for volumes to attach or mount for pod

Chart version: 7.0.1-alpha1

Kubernetes version: v1.13.1

Kubernetes provider: Barer metal

Helm Version: v2.11.0

Run the helm chart.

helm install elastic/elasticsearch --namespace efk --name efk-elasticsearch --version 7.0.1-alpha1 --set imageTag=7.0.1 --set replicas=1 --set resources.requests.memory=1Gi --set volumeClaimTemplate.storageClassName=nfs --set volumeClaimTemplate.resources.requests.storage=600Gi --set "nodeSelector.kubernetes\.io/hostname"=node3

Persistent Volume is created, and gets the status "bound".

elasticsearch-master-elasticsearch-master-0 Bound pvc-a23fd9a2-7347-11e9-bdfc-0050563dbb49 600Gi ReadWriteOnce nfs

But the elasticsearch pod does not starts.

pod has unbound immediate PersistentVolumeClaims Unable to mount volumes for pod "elasticsearch-master-0_efk(a240e2f2-7347-11e9-bdfc-0050563dbb49)": timeout expired waiting for volumes to attach or mount for pod "efk"/"elasticsearch-master-0". list of unmounted volumes=[elasticsearch-master]. list of unattached volumes=[elasticsearch-master default-token-q8bks]

Thanks for the help.

[Elasticsearch][8.0.0] deprecated settings avoid Elasticsearch to start

The discovery.zen.ping.unicast.hosts avoid Elasticsearch to start if you use the 8.0.0 Docker image

{"type": "server", "timestamp": "2019-04-17T13:51:31,328+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "uncaught exception in thread [main]" , 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [discovery.zen.ping.unicast.hosts] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"Caused by: java.lang.IllegalArgumentException: unknown setting [discovery.zen.ping.unicast.hosts] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:531) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:148) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.node.Node.<init>(Node.java:343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.node.Node.<init>(Node.java:253) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]",
"... 6 more"] }

Official Logstash Helm chart feature request

Are you working on Logstash Helm chart by chance? I found this one, but it's not working right: https://github.com/helm/charts/tree/master/stable/logstash - it's got some issues....

EKS Storage Volume Support

I see that this Chart is tested against GKE. I work at Nike and we are using EKS.

According to the Helm Chart documentation, I believe the only thing I need to change in the values.yaml file is the storageClassName: "standard" variable to storageClassName: "gp2"

I have verified that the cluster does, in fact, have a storage class set up. However, when I go to deploy the Chart and I describe my master elasticsearch nodes, I am getting this error message:

pod has unbound immediate PersistentVolumeClaims (repeated 3 times)

I believe that I am still missing some configuration setting having to do with Storage. Please let me know if you need any additional information. Any help is appreciated.

[elasticsearch] Allow for configurable health status value in readinessProbe

In the elasticsearch statefulset readinessProbe command, the cluster health status has a hardcoded value of green. It's not always the case that, as with coordinator nodes, green is the minimum acceptable state for cluster health.

Publish new version of elasticsearch chart

@Crazybus any chance to publish the new chart if it's ready ?

I saw you committed all the changes for ES7 and I am hoping to use the new chart asap without cloning the repo.

Thanks in advance.

[elasticsearch] imagePullSecrets not supported?

It does not look like imagePullSecrets tag in values.yaml is being processed. We need this because we pull from an internal Docker store, rather than the public ones.

Steps to reproduce:

1. Modify values.yaml:

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "6.6.0"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: "somesecret"

2. Execute:

helm install elasticsearch --debug --dry-run --name elasticsearch

Output:

image: docker.elastic.co/elasticsearch/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 6.6.1

I tried several permutations (double quotes, no quotes, newline with the ' - somesecret ' notation, etc) to no avail.

The Kibana chart also suffers from the same problem. If someone fixes the Elasticsearch chart, I'll fix the Kibana chart.

Bump chart to version 7.0.0

@Crazybus can you please bump the chart version to 7.0.0?

Cannot mount elasticsearch keystore in pod, device busy

I'm trying to mount the elasticsearch keystore per the documentation but I keep getting the error below. I've verified the the elasticsearch keystore file is valid (by adding it to a test continer and testing the values).

Error:

Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy

Steps taken:

1. Command to create the secret:
   kubectl create secret generic elasticsearch-keystore --from-file=./elasticsearch.keystore

2. SecretMounts in my yaml configuration:

  secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs
  - name: elastic-license
    secretName: elastic-license
    path: /usr/share/elasticsearch/config/license
  - name: elasticsearch-keystore
    secretName: elasticsearch-keystore
    path: /usr/share/elasticsearch/config/elasticsearch.keystore
    subPath: elasticsearch.keystore 

Is there something I am missing?

README.md needs an update

heml install line still has 6.6.2-alpha1

esJavaOpts vs jvm.options

I have set esJavaOpts to -Xmx8g -Xms8g" which correctly sets the env var in the container: ES_JAVA_OPTS=-Xmx8g -Xms8g

There also exists a jvm.options configuration which sets -Xms1g and -Xmx1g. This seems to cause the following settings when starting Elasticsearch:

1000290+      1      0 99 07:36 ?        04:12:54 /opt/jdk-11.0.1/bin/java -Xms1g -Xmx1g 
[...]
-Xmx8g -Xms8g 
[...]

Standardize on Elasticsearch connection value names

As discovered in #22 (review) the values used to connect to Elasticsearch differ between Kibana and Elasticsearch. In the future when charts are added for Logstash/Beats/others this could become even more fragmented and annoying to configure.

Because the Elasticsearch chart needs to be able to access parts of the Elasticsearch URL separately I think it makes most sense to split the URL up into separate variables.

Proposal:

esHost: elasticsearch-master
esProtocol: http
esPort: 9200

elastic/filebeat has not been pushed to the helm repo

helm repo add elastic https://helm.elastic.co
helm repo update
helm install --name filebeat elastic/filebeat --version 7.0.0-alpha1

Error: failed to download "elastic/filebeat"

Discovery timeout: Sometimes the -master-headless resolves to itself.

Logs from foo-master-0 show it timed out searching for other masters:

[2018-12-18T06:49:37,298][INFO ][o.e.p.PluginsService     ] [foo-master-0] loaded plugin [ingest-user-agent]
[2018-12-18T06:49:49,072][INFO ][o.e.x.s.a.s.FileRolesStore] [foo-master-0] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2018-12-18T06:49:50,286][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [foo-master-0] [controller/205] [Main.cc@109] controller (64 bit): Version 6.5.3 (Build f418a701d70c6e) Copyright (c) 2018 Elasticsearch BV
[2018-12-18T06:49:51,588][INFO ][o.e.d.DiscoveryModule    ] [foo-master-0] using discovery type [zen] and host providers [settings]
[2018-12-18T06:49:54,000][INFO ][o.e.n.Node               ] [foo-master-0] initialized
[2018-12-18T06:49:54,000][INFO ][o.e.n.Node               ] [foo-master-0] starting ... 
[2018-12-18T06:49:54,594][INFO ][o.e.t.TransportService   ] [foo-master-0] publish_address {10.0.5.23:9300}, bound_addresses {[::]:9300}
[2018-12-18T06:49:54,608][INFO ][o.e.b.BootstrapChecks    ] [foo-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-12-18T06:50:24,687][WARN ][o.e.n.Node               ] [foo-master-0] timed out while waiting for initial discovery state - timeout: 30s 
[2018-12-18T06:50:24,696][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [foo-master-0] publish_address {10.0.5.23:9200}, bound_addresses {[::]:9200}
[2018-12-18T06:50:24,697][INFO ][o.e.n.Node               ] [foo-master-0] started
[2018-12-18T06:50:29,077][WARN ][r.suppressed             ] [foo-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null

<the above message repeats every 10 seconds>

[root@foo-master-0 elasticsearch]# ping foo-master-headless
PING foo-master-headless.default.svc.cluster.local (10.0.5.23) 56(84) bytes of data.
64 bytes from foo-master-0.foo-master-headless.default.svc.cluster.local (10.0.5.23): icmp_seq=1 ttl=64 time=0.013 ms
...

[root@foo-master-0 elasticsearch]# dig +short foo-master-headless.default.svc.cluster.local
10.0.4.7
10.0.5.23
10.0.3.39

It seems somehow that dns has cached that 'foo-master-headless` resolves to self. Symptoms of this problem are noticed when a node fails to join the cluster:

% kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
foo-data-0                   1/1     Running   0          32m
foo-data-1                   1/1     Running   0          32m
foo-data-2                   0/1     Running   0          20m
foo-master-0                 0/1     Running   0          20m
foo-master-1                 1/1     Running   0          32m
foo-master-2                 1/1     Running   0          32m

Sometimes the cluster forms fine. Other times, certain nodes get stuck and never seem to join the cluster.

Objects Import

Describe the feature:
The ability to import saved objects like dashboards, index pattern, ... would be most useful.

Client/Coordinating nodes with the ES helm chart?

Hi, the elasticsearch chart in the community repo spins up client nodes as deployment. Looking at elastic's ES chart, if all 3 roles are set to false and cross cluster search is disabled through esConfig, the corresponding statefulset will compose of coordinating nodes(?)
Can you please explain why a statefulset is preferred over deployment for coordinating nodes and why does it need a persistent volume claim? Or is this chart not supposed to be used for client-only node? I looked through the examples and readme but didn't see any mention of coordinating only nodes. TIA

Kibana chart should allow custom annotations

The Kibana chart should provide the podAnnotations value to add annotations to the created pod.

This feature is essential to using the new "autodiscovery hints" feature in Filebeat and Metricbeat.

Openshift support

Hello there,

I was looking for the best way to deploy elasticsearch / kibana on openshift and it looks like I could use some help. : )

I tried adapting these charts and kept some notes with the problems that came up.

I would welcome suggestions (even for a completely different method of deploying into openshift - e.g. openshift templates / operator) and am happy to help with any questions you might have so we can figure something out (I have an openshift cluster standing by).

Thanks!

Can't up a multi node config failed to resolve host [master-headless]

Hello,

When trying to set up a multi-role based cluster I encounter the following issue after setting up the master nodes :

 {"type": "server", "timestamp": "2019-05-02T01:16:08,994+0000", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elasticsearch", "node.name": "elasticsearch-data-0",  "message": "failed to resolve host [master-headless]" , 
"stacktrace": ["java.net.UnknownHostException: master-headless",...

The configs of the masters :

clusterName: "elasticsearch"
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.0.0"
imagePullPolicy: "IfNotPresent"
esMajorVersion: 7
protocol: "http"

nodeSelector:
    app: elasticsearch
  
replicas: 3

nodeGroup: "master"

roles:
  master: "true"
  ingest: "false"
  data: "false"

ressources: 
  requests.cpu: 500m
  requests.memory: 2Gi
  limits.cpu: 1500m
  limits.memory: 2Gi

esJavaOpts: "-Xmx1g -Xms1g"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 10Gi
  storageClassName: do-block-storage

The configs of the data nodes :

clusterName: "elasticsearch"
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.0.0"
imagePullPolicy: "IfNotPresent"
esMajorVersion: 7
protocol: "http"

nodeSelector:
    app: elasticsearch

masterService: "master"
replicas: 3

nodeGroup: "data"

roles:
  master: "false"
  ingest: "false"
  data: "true"

ressources: 
  requests.cpu: 500m
  requests.memory: 2Gi
  limits.cpu: 1500m
  limits.memory: 2Gi

esJavaOpts: "-Xmx1g -Xms1g"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 250Gi
  storageClassName: do-block-storage

EDIT :
Its look like there is an issue wirth the name generation (?) because the concerned service should be elasticsearch-master-headless but only master-headless is mentioned in the log of the data nodes

We can see as well that the DNS receive the actual request :

2019-05-02T03:07:07.093Z [INFO] xx.xx.xx.133:60301 - 35027 "A IN master-headless.svc.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000196715s
2019-05-02T03:07:07.093Z [INFO] xx.xx.xx.133:60301 - 4324 "AAAA IN master-headless.svc.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000291045s
2019-05-02T03:07:07.094Z [INFO] xx.xx.xx.133:59951 - 47699 "A IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000130911s
2019-05-02T03:07:07.094Z [INFO] xx.xx.xx.133:59951 - 26206 "AAAA IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000127551s
2019-05-02T03:07:07.095Z [INFO] xx.xx.xx.133:43284 - 19072 "A IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000137498s
2019-05-02T03:07:07.096Z [INFO] xx.xx.xx.133:43284 - 62598 "AAAA IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000205652s
2019-05-02T03:07:09.616Z [INFO] xx.xx.xx.157:57068 - 64127 "AAAA IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000086844s
2019-05-02T03:07:09.616Z [INFO] xx.xx.xx.157:57068 - 1916 "A IN master-headless. udp 33 false 512" NXDOMAIN qr,rd,ra 108 0.000160398s
2019-05-02T03:07:10.021Z [INFO] xx.xx.xx.176:33201 - 15919 "A IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,rd 140 0.000085146s
2019-05-02T03:07:10.022Z [INFO] xx.xx.xx.176:33201 - 60982 "AAAA IN master-headless.cluster.local. udp 47 false 512" NXDOMAIN qr,rd 140 0.000082036s

Dockerize helper scripts for release process

Request came from: #111 (comment)

• Dockerize bumper.py
• Dockerize release.py

Error: could not find tiller

I'm trying to follow Installing, yet getting following error:

toor@suey:~$ helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
toor@suey:~$ helm repo list
NAME   	URL                                             
stable 	https://kubernetes-charts.storage.googleapis.com
local  	http://127.0.0.1:8879/charts                    
elastic	https://helm.elastic.co                         
toor@suey:~$ helm install --name elasticsearch elastic/elasticsearch --version 6.5.3-alpha1
Error: could not find tiller
toor@suey:~$ 

toor@suey:~$ tiller -version
v2.12.0
toor@suey:~$ 

Please advise.

Why not support official helm charts ?

Hi Guys,
I really want to say that it's very nice that we have yet another version of helm charts for Elastic and Kibana,
But the question is why you don't participate in creating charts from helm/charts repo ?
They are more mature and well tested. And for sure they have many more options.

WDYT guys ?

Regards,
Maciek

Kibana readinessProbe fails when server.ssl.enabled=true

The readinessProbe only appears to target HTTP, not HTTPS.

Make sure all examples can be run by everyone

As pointed out in #26 (comment) most of the examples are pulling secrets from our vault service. Apart from the license we use for testing everything else can be auto generated for testing purposes. For the license it is also possible to use the trial API for testing. As part of automated testing we will continue to use the real license because some rules are relaxed when running a trial (such as no enforced node to node encryption).

The end goal of this issue is to make sure that all users and contributors can run any of the examples without needing to first figure out how to create and generate their own secrets.

multi volumeClaimTemplate support on elasticsearch chart

With elasticsearch , in some cases , we need some extra volumes with your own properties to store snapshots or backups (for example). Current version , only supports one VolumeClaimTemplate. Could be very useful for us, support for multi volumes on elasticsearch group nodes.
Thanks

Use already existing PVC for data

Hey,

we already deployed elasticsearch 6.6.0 and would like to use helm for the next update. Is it possible to use the "old" pvc containting our data without having copy it somewhere?

Right know, after reading the docs, I thought of:

• mounting it as an additional volume
• setting the data path in elasticsearch.yml
• setting the size of existing data volume from 30Gi to 0Gi to prevent wasting storage

Is there any easier way?

helm test fails when security is enabled

% helm ls elasticsearch-data
NAME                    REVISION        UPDATED                         STATUS          CHART                           APP VERSION     NAMESPACE
elasticsearch-data      3               Sat Dec 15 21:30:24 2018        DEPLOYED        elasticsearch-6.5.3-alpha1      6.5.3           default  

Running helm test:

% helm test elasticsearch-data
RUNNING: elasticsearch-data-rwrbf-test
FAILED: elasticsearch-data-rwrbf-test, run `kubectl logs elasticsearch-data-rwrbf-test --namespace default` for more info
Error: 1 test(s) failed           

Helm test logs:

% kubectl logs elasticsearch-data-rwrbf-test --namespace default
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server                  

The curl command doesn't use https (nor configured w/ credentials and certificates)

% kubectl get pod elasticsearch-data-rwrbf-test -o yaml | grep curl
      curl -XGET --fail 'redacted:9200/_cluster/health?wait_for_status=green&timeout=1s'

https enabled in the pods

% kubectl get pod redacted-0 -o yaml | grep -A1 security.http.ssl.enabled
    - name: xpack.security.http.ssl.enabled
      value: "true"

[Elasticsearch] run 7.0.0-SNAPSHOT versions on 6.6.0-alpha1 chart

I have tried to use the Helm chart 6.6.0-alpha1 with the Docker image 7.0.0-SNAPSHOT, but it did not find a minimum masters number to start (2), I know it probably does not work but I report it only in case.

{"type": "server", "timestamp": "2019-03-07T15:52:32,007+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{elasticsearch-master-2}{3augAIonSmya_3Nh8ejlVQ}{9dnV86RNT_OElXMEifJqGg}{10.8.8.97}{10.8.8.97:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{vOIg6cnCQSiAefXM_d0jpA}{DLZ_IGVRQW6tgE2EfidHfg}{10.8.6.34}{10.8.6.34:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [10.8.8.97:9300, 10.8.6.34:9300] from hosts providers and [{elasticsearch-master-0}{EhVP4Rl-Qv-ZnTRpyZ7lHQ}{TF2RPZB6Rw6_EFKuJfAU4g}{10.8.5.126}{10.8.5.126:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
{"type": "server", "timestamp": "2019-03-07T15:52:32,314+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])"  }
{"type": "server", "timestamp": "2019-03-07T15:52:32,314+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}" , 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:549) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",

This is the configuration used

clusterName: elasticsearch
nodeGroup: "master"
masterService: "elasticsearch-master"
roles:
  master: "true"
  ingest: "true"
  data: "true"
replicas: 3
#minimumMasterNodes: 2
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.0.0-SNAPSHOT"
imagePullPolicy: "Always"
esJavaOpts: "-Xmx1g -Xms1g"
resources:
  requests:
    cpu: "100m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "2Gi"
volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "standard"
  resources:
    requests:
      storage: 60Gi
protocol: http
httpPort: 9200
transportPort: 9300
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: elastic-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
  path: /
  hosts:
    - elasticsearch.example.com
  tls:
    - hosts:
      - elasticsearch.example.com
      secretName: ingress-tls

Kibana deployment name vs Elasticsearch statefulset name.

For Elasticsearch, the statefulset name seems to follow {clusterName}-{nodeGroup}

Kibana's deployment, however, seems to be kibana-kibana. If I set nameOverride for Kibana, the deployment becomes kibana-{nameOverride}.

I kinda want to see things grouped by cluster name, and while Kibana has no such concept, I would wish for the kibana deployment to be called {nameOverride}-kibana.

Given this is a naming proposal, I'm almost willing to close it immediately myself because naming is hard and maybe nobody will like any one proposal ;)

Thoughts?

Why `masterService` in values.yaml ?

Hi all,

I read how this chart works and I don't understand why is there masterService in values.yaml.
Because service is defined with {{ template "uname" . }}-headless which is exactly the concatenation of clusterName and nodeGroup {{ .Values.clusterName }}-{{ .Values.nodeGroup }}.

In helpers :

{{- define "uname" -}}
{{ .Values.clusterName }}-{{ .Values.nodeGroup }}
{{- end -}}

In service.yaml :

kind: Service
apiVersion: v1
metadata:
  name: {{ template "uname" . }}-headless

But in statefulset.yaml :

          - name: discovery.zen.ping.unicast.hosts
            value: "{{ .Values.masterService }}-headless"

Slow re-election when elected master pod is deleted

First of all - thank you guys for the chart!

I was playing around with the multi-node example and experienced some odd behavior. Here's how I'm reproducing the issue.

After the multi example is deployed, open up the multi-data service to your local in one terminal:

$ kubectl port-forward service/multi-data 9200

Watch the call to /_cat/master in another terminal:

$ watch -n1 time curl -s http://localhost:9200/_cat/master?v

In a third terminal, whoever the elected master is, delete them:

$ kubectl delete pod multi-master-0

The API call in the second terminal will now hang. After 30 seconds, the request will timeout and we might see the following error for a split second:

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Soon after, the cluster recovers and the API call from the second window starts responding again. Here are the logs off another master node before and after the re-election:

[2019-02-17T01:49:03,736][INFO ][o.e.d.z.ZenDiscovery     ] [multi-master-1] master_left [{multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [shut_down]
[2019-02-17T01:49:03,736][WARN ][o.e.d.z.ZenDiscovery     ] [multi-master-1] master left (reason = shut_down), current nodes: nodes:
   {multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, master
   {multi-data-0}{Ndz2WGGiSz6Y1XO1tWyWRw}{nPoZagPdRr2Tq45BPAkh_g}{10.40.2.13}{10.40.2.13:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
   {multi-data-2}{MmkHmP6XRTibriDLazv_1A}{iS1mfA0JQ-yURqZ4-ng2zQ}{10.40.1.8}{10.40.1.8:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
   {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
   {multi-master-1}{uetiuhetRbasFRNLqI6ixg}{AwsDmF2aTZS1JesCE0HZ0A}{10.40.2.15}{10.40.2.15:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, local
   {multi-data-1}{Uh9aucMmS6uLRBeG5559_w}{tlxqIDoZRjCLMfMWxU0IyQ}{10.40.0.11}{10.40.0.11:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}

[2019-02-17T01:49:03,830][WARN ][o.e.t.TcpTransport       ] [multi-master-1] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.40.1.14:36958}]
java.nio.channels.ClosedChannelException: null
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2019-02-17T01:49:07,107][INFO ][o.e.c.s.ClusterApplierService] [multi-master-1] detected_master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, reason: apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [95]])
[2019-02-17T01:49:34,832][WARN ][o.e.c.NodeConnectionsService] [multi-master-1] failed to connect to node {multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [multi-master-0][10.40.1.14:9300] connect_timeout[30s]
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1576) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-6.6.0.jar:6.6.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-02-17T01:49:37,200][WARN ][o.e.c.s.ClusterApplierService] [multi-master-1] cluster state applier task [apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [95]])] took [30s] above the warn threshold of 30s
[2019-02-17T01:49:40,315][INFO ][o.e.c.s.ClusterApplierService] [multi-master-1] removed {{multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{Es2wrbyiQdWz5CnT4V5wkA}{10.40.1.14}{10.40.1.14:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [96]])
[2019-02-17T01:49:55,945][WARN ][o.e.t.TransportService   ] [multi-master-1] Received response for a request that has timed out, sent [47893ms] ago, timed out [17870ms] ago, action [internal:discovery/zen/fd/master_ping], node [{multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], id [579]
[2019-02-17T01:50:03,784][INFO ][o.e.c.s.ClusterApplierService] [multi-master-1] added {{multi-master-0}{KZPjmKZtSf2LGV-IyvtfOg}{LqkyAGeTSUW2mtNmt49HIA}{10.40.1.15}{10.40.1.15:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {multi-master-2}{0PQvO9UhT--kvv2mEuJyBg}{WVb0wCALRP6KIt2oHNTmcQ}{10.40.0.20}{10.40.0.20:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [98]])

I figured Kubernetes might be killing the pods too abruptly, so I followed the instructions at https://www.elastic.co/guide/en/elasticsearch/reference/6.6/stopping-elasticsearch.html to stop Elasticsearch. Sure enough, if we kill the process from the elected master pod directly, the re-election will be quick!

Assuming mutli-master-2 is the new master:

$ kubectl exec multi-master-2 -- kill -SIGTERM 1

Notice how the API call from the second terminal only hangs for around 3 seconds this time!

Reading through the docs for the termination of pods (https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods), Kubernetes does in fact send a SIGTERM to the container, so I'm guessing deleting a pod does something more than just send a SIGTERM that Elasticsearch doesn't like.

[elasticsearch] Create or allow setting a serviceAccount to be used

Hopefully most clusters have something along the lines of restricted for the default service account in a namespace. By default, with no serviceAccount specified, the default serviceAccount is used.

Two values are required, one for the initContainer, which is already requiring privileged permissions and root access, and one for the es container, which doesn't require the same level.

please point or document how to get the free version / open source x-pack license

Hi

I'm unable to find a way to get the free / open source x-pack license

can you please guide how to?

Allow hostPath for data

Hello,

we are running kubernetes on-premise where local storage is used. Unfortunately, auto-provisioning for local storage volumes is not yet available on kubernetes. It would be very useful if the chart supports "hostPath" as option alternatively to volumeClaimTemplate.

I added a similar feature to redis-ha a while ago: helm/charts#11815

I you agree to this change, I'd create a PR for elastic.

Best regards,
Michael.

Install plugins?

initContainers doesn't seem to be exposed. Is there a way to install plugins with this chart?

Is podManagementPolicy:OrderedReady supported?

Including podManagementPolicy in values.yaml makes it seem like it may be, but I am not sure it would work, with the way readiness probes are used. Could you please clarify this? (Maybe also in the docs ...)

PVCs in migration guide looks incorrect

In the migration example I can see a couple of issues:

• Client node defines a PVC
• Data node doesn't define a PVC

Also, even there doesn't seem to be a way to disable PVC creation for a Client node. PVC always gets created.