SSL support about kubernetes-elasticsearch-cluster

[devth]

Good point, I hadn't considered K8S secrets. I'll be working on it over the next few days.

[pires]

No. But I am open to participate in a debate around it or review a related PR.

[devth]

I'm working on a custom image that installs and configures it. I'd be happy to PR if it's something you're interested in. I think enabling SSL by default makes sense, but not necessarily all the other SearchGuard features, like authn/z. The SG SSL plugin does only this.

I'm following the SG SSL QuickStart.

One compromise I made was to use the same keystore on every node instead of installing a unique one, as this is difficult with Docker / immutable infra, unless using something like Vault and vaultjks to generate the keystores and truststores at runtime.

I'm also using the default settings, which as the QuickStart notes:

This tutorial shows how to quickly set up SG SSL. Note that the settings and passwords we use here are not safe for production, and only meant to get SG SSL working as quickly as possible!

To do it correctly, I think a user would still need to use a custom image?

[pires]

I think enabling SSL by default makes sense

Agreed. But there will be problems with certificates, Java key/truststores, etc.

One compromise I made was to use the same keystore on every node instead of installing a unique one, as this is difficult with Docker / immutable infra, unless using something like Vault and vaultjks to generate the keystores and truststores at runtime.

Well, if we're to use self-signed certificates, there is a way one can provide a single keystore with a single certificate. Something like:

CA key, certificate and keystore

openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=our-ca"
keytool -importcert -trustcacerts -keystore ca.jks -alias root -storepass <password> -file ca.pem -noprompt

Elasticsearch node key, certificate and keystore

First we need a file named ssl.cnf with the following contents:

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = elasticsearch

Then, execute:

openssl genrsa -out es-key.pem 2048
openssl req -new -key es-key.pem -out es.csr -subj "/CN=our-es" -config ssl.cnf
openssl x509 -req -in es.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out es.pem -days 3650 -extensions v3_req -extfile ssl.cnf

If all goes well, now we just need to generate the keystore:

./generate_jks ca.pem es.pem es-key.pem <keystore_name> <password>

The generate_jks is a shell script with the following usage:

./generate_jks /path/to/ca /path/to/cert /path/to/key keystore_name password

and generates:

1. Full certificate chain in #PKCS12 format
2. Generates keystore by importing .p12 file generated in 1
3. Generates truststore (in the same file as the keystore) with previously generated CA keystore

Does any of this makes sense to you?

[devth]

Makes sense overall, though my ssl is a bit weak so I don't follow the specific commands 100%.

Not sure if you saw but the SG SSL repo provides example scripts that generate the keystore and truststore, similar to what you outlined.

Were you thinking of baking the cert into the image at build time? Can't generate them on startup because all images need to use the same key and trust stores.

We could use ONBUILD and let people optionally build the own images from yours if they want unique certs. Otherwise the certs are exposed to anyone and SSL is near-worthless IIUC.

[pires]

No, I didn't check the repo.

ONBUILD is a nice solution but with Kubernetes one can leave the certificate generation to the user and just ask them to create a secret which exposes the keystore in the pod filesystem. Either way, a password will always be needed as well and we can leverage on secrets as well.

Right now, I'm heads down with something at work so I won't be able to pick it up. Perhaps, in a week's time. Ping me by then if you don't come up with a solution.

[devth]

We ended up deciding to use Elastic's Shield product, so I didn't get very far with this. I pushed my changes here: devth/docker-elasticsearch-kubernetes@a550a29

I was adding the keystore and truststore directly. Next step would be to remove that and obtain them from K8S secrets like you suggested.

[pires]

Leaving this open as a reference for users looking for same thing.