Comments (9)
Gotcha, that definitely makes sense and it looks like the consul
DaemonSet is currently missing the following in the spec:
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
so it's not actually binding to the host ports specified and explains why my setup as well as @popopanda are experiencing the issue.
from consul-helm.
This same issue occurs with Digital Ocean's k8 offering. The only way to resolve is to change the host ip the sync catalog deployment to reflect the internal domain of the consul server service rather than piggy backing off of the clusterIp which is injected via hostip
https://github.com/hashicorp/consul-helm/blob/master/templates/sync-catalog-deployment.yaml#L36
https://github.com/hashicorp/consul-helm/blob/master/templates/sync-catalog-deployment.yaml#L49
-http-addr=consul-server:8500 \
Perhaps this should be optional in the values.yml
For others looking, a present fix is to override the sync-catalog-deployment.yaml file with the change to reflect the consul service name as the hostIp.
from consul-helm.
Also, this is my values.yml I used
global:
enabled: true
# Domain to register the Consul DNS server to listen for.
domain: consul
image: "consul:1.3.0"
imageK8S: "hashicorp/consul-k8s:0.2.0"
datacenter: dc1
server:
enabled: true
image: null
replicas: 3
bootstrapExpect: 3 # Should <= replicas count
storage: 10Gi
storageClass: null
connect: true
resources: {}
updatePartition: 0
disruptionBudget:
enabled: true
maxUnavailable: null
extraConfig: |
{}
extraVolumes: []
# - type: secret (or "configMap")
# name: my-secret
# load: false # if true, will add to `-config-dir` to load by Consul
client:
enabled: "-"
image: null
join: null
grpc: false
resources: {}
extraConfig: |
{}
extraVolumes: []
# - type: secret (or "configMap")
# name: my-secret
# load: false # if true, will add to `-config-dir` to load by Consul
dns:
enabled: "-"
ui:
enabled: true
service:
enabled: true
type: NodePort
syncCatalog:
# True if you want to enable the catalog sync. "-" for default.
enabled: true
image: null
toConsul: true
toK8S: true
k8sPrefix: null
connectInject:
enabled: false
image: null # image for consul-k8s that contains the injector
default: false # true will inject by default, otherwise requires annotation
imageConsul: null
imageEnvoy: null
namespaceSelector: null
certs:
secretName: null
caBundle: ""
certName: tls.crt
keyName: tls.key
from consul-helm.
The issue here is the consul-sync-catalog
deployment is being configured with the hostIP
of the pod when it should just be using the consul-server
Service. I manually changed the deployment to use the following command
:
- /bin/sh
- -ec
- |
consul-k8s sync-catalog \
-http-addr=consul-server:8500 \
-consul-domain=consul \
-k8s-write-namespace=${NAMESPACE}
I'm not sure what the thinking is behind using the hostIP
for the -http-addr
flag.
from consul-helm.
@jipperinbham Consul tooling should use the local agent wherever possible. The servers aren't meant to field cluster-wide API requests on their own (though they can to a fairly high scale). The local agent performs aggressive API batching, pipelining, and TCP multiplexing along with local caching to make Consul a lot more performant. Additionally, in the case of the catalog sync, it provides a local agent to register a service against in some cases which provides proper node health checking.
Please ensure you have the Consul clients installed using the Helm chart. This will expose the agent HTTP API on port 8500 on the host IP.
from consul-helm.
Just for me to refer to later, the issue is that we're using the downward API to inject the HOST_IP environment variable so we can talk to the agent running on our node. In some deployments, ex. EKS with the aws-cni plugin that uses multiple enis, the HOST_IP is only one IP of multiple and so we might not be able to talk to the agent on that specific HOST_IP.
I'm not sure why we're not using hostNetwork and will need to ask the team.
from consul-helm.
Gotcha, that definitely makes sense and it looks like the
consul
DaemonSet is currently missing the following in the spec:dnsPolicy: ClusterFirstWithHostNet hostNetwork: true
so it's not actually binding to the host ports specified and explains why my setup as well as @popopanda are experiencing the issue.
@jipperinbham we are running with hostPort: 8500
so hostNetwork should not be required
consul-helm/templates/client-daemonset.yaml
Line 156 in 24d3115
from consul-helm.
This same issue occurs with Digital Ocean's k8 offering. The only way to resolve is to change the host ip the sync catalog deployment to reflect the internal domain of the consul server service rather than piggy backing off of the clusterIp which is injected via hostip
https://github.com/hashicorp/consul-helm/blob/master/templates/sync-catalog-deployment.yaml#L36
https://github.com/hashicorp/consul-helm/blob/master/templates/sync-catalog-deployment.yaml#L49-http-addr=consul-server:8500 \
Perhaps this should be optional in the values.yml
For others looking, a present fix is to override the sync-catalog-deployment.yaml file with the change to reflect the consul service name as the hostIp.
@ptariche I just tested this chart with Digital Ocean and the sync deployment could reach the local consul agent using its hostPort. Maybe something has changed on DO's side?
from consul-helm.
@popopanda from my testing, this should be working. Since this ticket is 1 year old, it's likely things have changed. It's totally our fault that this has sat here for so long but I'm going to close it now.
If folks are still having issues with the sync catalog pod not being able to talk to the local consul agent please open a new issue.
from consul-helm.
Related Issues (20)
- Variable Interpolation within Helm chart HOT 7
- [ERROR] Unable to get Agent services: error="Unexpected response code: 403 (ACL not found)" HOT 9
- GKE Ingress requires pathType to be ImplementationSpecific
- flag provided but not defined: -log-json HOT 6
- Mesh-Gateway k8s: Error initializing configuration HOT 6
- Chart v0.32.+ : invalid config key "TransparentProxy" for proxy-defaults.yaml HOT 2
- Could not resolve host: static-server HOT 9
- test issue migration
- consul-consul-webhook-cert-manager flag provided but not defined: -log-json HOT 2
- 0.32.1/1.10.0 WAN Federation consul-server-acl-init job failing to execute resulting in failed helm installation HOT 8
- Kubernetes, ConsulCatalog and Traefik - Changing default Sync Rule between K8s and ConsulCatalog? HOT 2
- Consul connect injected sidecars not permitted by Pod Security Policies HOT 2
- Allow specifying a default policy / role for namespace mirroring HOT 2
- How route traffic to another consul node? HOT 5
- Consul ingress gateways not starting after chart upgrade HOT 6
- Which directory consul kv data is stored HOT 4
- consul connect: 503 after scaling in or pod restarts HOT 15
- feat: Add `-recursor` flag
- Failed helm upgrade leaves behind a job that prevents future upgrades? HOT 7
- Allow Consul client daemonset connectivity via nodePort (instead of only hostPort) HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from consul-helm.