Comments (11)
There is a bit more needed than just providing the extraConfig
section. The following steps working for me:
- Generating certificates and placing them to k8s secrets. There are multiple ways for doing this and I have opted to use own consul helpers:
$ consul tls ca create
$ consul tls cert create -server -additional-dnsname=*.c01-consul-server.devops.svc
$ kubectl create secret generic consul-server-certs \
--from-file=ca.pem=consul-agent-ca.pem \
--from-file=consul.pem=dc1-server-consul-0.pem \
--from-file=consul-key.pem=dc1-server-consul-0-key.pem
$ consul tls cert create -client
$ kubectl create secret generic consul-client-certs \
--from-file=ca.pem=consul-agent-ca.pem \
--from-file=consul.pem=dc1-client-consul-0.pem \
--from-file=consul-key.pem=dc1-client-consul-0-key.pem
- Adjusting health probes. It's possible to get
curl
commands to work either, but using consul own probes is a bit more straightforward thanks to env variables. For example inserver-statefulset.yaml
...
livenessProbe:
exec:
command:
- consul
- operator
- raft
- list-peers
failureThreshold: 2
initialDelaySeconds: 5
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
exec:
command:
- consul
- members
failureThreshold: 2
initialDelaySeconds: 5
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 5
...
and the client-daemonset.yaml
can use a similar check.
- Now, the following additional values will fill the gaps and glue it all together:
global:
enabled: true
domain: consul
datacenter: dc1
gossipEncryption:
secretName: consul
secretKey: gossip-encryption-key
server:
enabled: true
replicas: 3
bootstrapExpect: 3
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: server
topologyKey: failure-domain.beta.kubernetes.io/zone
extraConfig: |
{
"ui": true,
"client_addr": "0.0.0.0",
"enable_script_checks": false,
"disable_remote_exec": true,
"verify_incoming": true,
"verify_outgoing": true,
"verify_server_hostname": true,
"ca_file": "/consul/userconfig/consul-server-certs/ca.pem",
"cert_file": "/consul/userconfig/consul-server-certs/consul.pem",
"key_file": "/consul/userconfig/consul-server-certs/consul-key.pem",
"ports": {
"http": -1,
"https": 8500
}
}
extraEnvironmentVars:
CONSUL_CACERT: /consul/userconfig/consul-server-certs/ca.pem
CONSUL_CLIENT_CERT: /consul/userconfig/consul-server-certs/consul.pem
CONSUL_CLIENT_KEY: /consul/userconfig/consul-server-certs/consul-key.pem
CONSUL_HTTP_SSL: true
# CONSUL_HTTP_ADDR: https://localhost:8501
extraVolumes:
- type: secret
name: consul-server-certs
load: false
client:
enabled: true
grpc: true
extraConfig: |
{
"verify_incoming": true,
"verify_outgoing": true,
"verify_server_hostname": true,
"ca_file": "/consul/userconfig/consul-client-certs/ca.pem",
"cert_file": "/consul/userconfig/consul-client-certs/consul.pem",
"key_file": "/consul/userconfig/consul-client-certs/consul-key.pem",
"ports": {
"http": -1,
"https": 8500
}
}
extraEnvironmentVars:
CONSUL_CACERT: /consul/userconfig/consul-client-certs/ca.pem
CONSUL_CLIENT_CERT: /consul/userconfig/consul-client-certs/consul.pem
CONSUL_CLIENT_KEY: /consul/userconfig/consul-client-certs/consul-key.pem
CONSUL_HTTP_SSL: true
# CONSUL_HTTP_ADDR: https://localhost:8501
extraVolumes:
- type: secret
name: consul-client-certs
load: false
dns:
enabled: true
ui:
enabled: true
service:
enabled: true
type: NodePort
NB: the affinity rules are not essential here, I have had them in place due to multi-zone cluster setup.
Keep in mind that this configuration is running HTTPS on the same 8500 port, instead of HTTP. Normally, in production you would want to use HTTPS, so this adjustment is fine. If for some reason you need to keep the both HTTP and HTTPS ports available, you can use 8501 for HTTPS and uncomment CONSUL_HTTP_ADDR env variable to point all the tools to the proper API endpoint.
from consul-helm.
I know this thread is old and has been closed but this is the only thread in here that deals with TLS for this chart. It seems as if the consul syncCatalog is designed to work without tls. Once tls is enabled on consul server and client (Agent), syncCatalog seems to stop working. Has anyone been able to get it to work correctly?
from consul-helm.
Can you provide an example of the whole helm --set command? It should be sufficient to set that value as a json string wrapped in single quotes.
from consul-helm.
Hi Jeff,
Sure.
There's a couple challenges with this consul-helm chart.
-
Missing an easy way to import / utilize existing TLS certs to join a k8s instance with existing consul cluster outside of k8s (must be created manually)
-
The chart requires significant manual clone + edit operations to get started with anything missing in the chart (passed in via extraConfig + extraVolumes)
Example helm install:
- First manually create Kubernetes Secret (consul-certs) containing the 3 TLS Certs + Gossip Key required for "in-k8s" Consul to speak with external Consul cluster. (not shown)
`
git clone https://github.com/hashicorp/consul-helm.git && cd consul-helm
helm install --name consul .
--namespace consul
--set "global.enabled=false"
--set "global.datacenter=noc"
--set "global.image=consul:1.4.0"
--set "server.enabled=false"
--set "dns.enabled=true"
--set "syncCatalog.enabled=true"
--set "client.enabled=true"
--set 'client.join[0]=192.168.250.102'
--set 'client.join[1]=192.168.250.103'
--set 'client.join[2]=192.168.250.104'
--set 'client.join[3]=192.168.250.105'
--set 'client.join[4]=192.168.250.106'
--set "client.extraVolumes[0].type=secret"
--set "client.extraVolumes[0].name=consul-certs"
--set client.extraConfig='{
"ca_path": "/consul/userconfig/consul-certs",
"ca_file": "/consul/userconfig/consul-certs/ca.pem",
"cert_file": "/consul/userconfig/consul-certs/consul-client.pem",
"key_file": "/consul/userconfig/consul-certs/consul-client-key.pem"
}'
`
Helm fails with:
Error: render error in "consul/templates/client-config-configmap.yaml": template: consul/templates/client-config-configmap.yaml:15:14: executing "consul/templates/client-config-configmap.yaml" at <.Values.client.extra...>: wrong type for value; expected string; got []interface {}
This is probably because Helm (go) insists this value in "--set" parameters must be a raw string (albeit JSON). Single quoting it does not protect Helm from choking on the string. I've tried various quoting and escaping methods to no avail.
Bottom line: There seems to be no way to configure TLS or anything else in extraConfig w/o doing a bunch of smelly manual operations (clone, edit, check, install?)
Could you provide an example where this works via helm install w/o unsightly values.yaml edits?
Thanks,
from consul-helm.
You might try --set-string or --set-file for client.extraConfig instead of --set.
https://github.com/technosophos/k8s-helm/blob/master/docs/using_helm.md
from consul-helm.
--set-string fails with the same error:
Error: render error in "consul/templates/client-config-configmap.yaml": template: consul/templates/client-config-configmap.yaml:15:14: executing "consul/templates/client-config-configmap.yaml" at <.Values.client.extra...>: wrong type for value; expected string; got []interface {}
--set-file requires more custom file operations instead of just tuning --set params
from consul-helm.
Hi @vanderhoofen,
I've just merged a fix for how we interpret extraConfig
which should enable setting it from the command line. From that PR description:
This enables users to specify extraConfig values using Helm's --set CLI
flag while installing the Helm chart. The format for using this command is:
--set 'client.extraConfig="{"log_level": "DEBUG"}"'
Please feel free to open a new issue if this doesn't solve the issue for you.
from consul-helm.
Well, I solved this in a easier way.
You don't need to write down your config with --set client.extraConfig
.
This chart deploys a configmap named consul-client-config
.
Just change the key extra-from-values.json
in consul-client-config
config map with your file and voilá!
my-values.yml:
---
client:
enabled: "true"
extraVolumes:
- name: "consul-secret"
load: "false"
type: "secret"
join:
- "xxx.com"
- "yyy.com"
dns:
enabled: "true"
global:
datacenter: "datacenter1"
domain: "mydomain"
enabled: "false"
config map consul-client-config
key: extra-from-values.json
:
{
"verify_outgoing": true,
"verify_incoming": true,
"verify_server_hostname": true,
"ca_file": "/consul/userconfig/consul-secret/ca",
"key_file": "/consul/userconfig/consul-secret/key",
"cert_file": "/consul/userconfig/consul-secret/cert",
"primary_datacenter": "datacenter1",
"acl_down_policy": "extend-cache",
"acl_default_policy": "allow",
"acl_agent_token": "14253987253985329853298",
"acl_agent_master_token": "14267473457345738"
}
--
from consul-helm.
Hi @vanderhoofen,
I've just merged a fix for how we interpret
extraConfig
which should enable setting it from the command line. From that PR description:This enables users to specify extraConfig values using Helm's --set CLI
flag while installing the Helm chart. The format for using this command is:--set 'client.extraConfig="{"log_level": "DEBUG"}"'
Please feel free to open a new issue if this doesn't solve the issue for you.
This is great. Thanks
from consul-helm.
Is this still the prefered way for consul/vault on k8s with tls?
from consul-helm.
@HofmannZ this ticket is super old so perhaps opening a new one with your specific use-case would be better. Quickly, turning on TLS can be done with:
global:
tls:
enabled: true
from consul-helm.
Related Issues (20)
- Variable Interpolation within Helm chart HOT 7
- [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)" HOT 9
- GKE Ingress requires pathType to be ImplementationSpecific
- flag provided but not defined: -log-json HOT 6
- Mesh-Gateway k8s: Error initializing configuration HOT 6
- Chart v0.32.+ : invalid config key "TransparentProxy" for proxy-defaults.yaml HOT 2
- Could not resolve host: static-server HOT 9
- test issue migration
- consul-consul-webhook-cert-manager flag provided but not defined: -log-json HOT 2
- 0.32.1/1.10.0 WAN Federation consul-server-acl-init job failing to execute resulting in failed helm installation HOT 8
- Kubernetes, ConsulCatalog and Traefik - Changing default Sync Rule between K8s and ConsulCatalog? HOT 2
- Consul connect injected sidecars not permitted by Pod Security Policies HOT 2
- Allow specifying a default policy / role for namespace mirroring HOT 2
- How route traffic to another consul node? HOT 5
- Consul ingress gateways not starting after chart upgrade HOT 6
- Which directory consul kv data is stored HOT 4
- consul connect: 503 after scaling in or pod restarts HOT 15
- feat: Add `-recursor` flag
- Failed helm upgrade leaves behind a job that prevents future upgrades? HOT 7
- Allow Consul client daemonset connectivity via nodePort (instead of only hostPort) HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from consul-helm.