Getting error Failed to resolve consul-server-0.consul-server.default.svc: lookup consul-server-0.consul-server.default.svc on 10.96.0.10:53: no such host * Failed to join 10.244.1.78: Remote state is encrypted and encryption is not configured while setting up consul DNS on kubernetes cluster about consul-helm

[shibi1989]

Getting error Failed to resolve consul-server-0.consul-server.default.svc: lookup consul-server-0.consul-server.default.svc on 10.96.0.10:53: no such host * Failed to join 10.244.1.78: Remote state is encrypted and encryption is not configured.

Already having core DNS in kubernetes cluster. Using kubernetes 1.12.2 version.
I have used helm to install the consul.

[deepanvermag3]

Is this resolved for you?

[shibi1989]

No

…

On Tue, Dec 25, 2018, 11:45 AM Deepan Verma ***@***.***> wrote: Is this resolved for you? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#76 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQ0TjfpMTi8AUtmhbXQUKgUHINHW5AcBks5u8cJlgaJpZM4ZAX9l> .

[giovannicandido]

Any update?

[adilyse]

Hi @shibi1989

Could you provide some additional information about your setup and installation? What type of kubernetes cluster are you using (cloud hosted, local setup, etc)? What version of the helm chart did you use? What configuration options did you use?

This information will help me narrow down what might be going wrong.

Thanks,
Rebecca

[erkanerol]

I have the same issue on EKS

helm/charts#10729

[vnbx]

Having the same issue with helm chart and ConsulAgent config.

[giovannicandido]

In my case the error was because no statefull storage could be created because the default was not set. But this message is complete useless in this case.

https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

[dwaiba]

Works fine for me on setting server.storageClass and using external provisioner like nfs-client on private clusters. Also works fine on gke where defaults are set

[lkysow]

I'm going to close this issue since it's 6 months old. If anyone is still having this issue please comment here and I can re-open.

[yueru-mylove]

I also got this problem. Though the consul agent join each member correctly, it still display some errors in the log below, almost the same with the error above.

    2019/12/26 03:47:48 [WARN] memberlist: Failed to resolve hashicorp-consul-server-1.hashicorp-consul-server.default.svc: lookup hashicorp-consul-server-1.hashicorp-consul-server.default.svc on 10.10.0.10:53: no such host
    2019/12/26 03:47:48 [WARN] memberlist: Failed to resolve hashicorp-consul-server-2.hashicorp-consul-server.default.svc: lookup hashicorp-consul-server-2.hashicorp-consul-server.default.svc on 10.10.0.10:53: no such host
    2019/12/26 03:47:48 [WARN] agent: (LAN) couldn't join: 0 Err: 3 errors occurred:
	* Failed to join 10.244.0.83: dial tcp 10.244.0.83:8301: connect: no route to host
	* Failed to resolve hashicorp-consul-server-1.hashicorp-consul-server.default.svc: lookup hashicorp-consul-server-1.hashicorp-consul-server.default.svc on 10.10.0.10:53: no such host
	* Failed to resolve hashicorp-consul-server-2.hashicorp-consul-server.default.svc: lookup hashicorp-consul-server-2.hashicorp-consul-server.default.svc on 10.10.0.10:53: no such host

    2019/12/26 03:47:48 [WARN] agent: Join LAN failed: <nil>, retrying in 30s
    2019/12/26 03:47:52 [WARN]  raft: Failed to get previous log: 1 log not found (last: 0)
    2019/12/26 03:47:52 [INFO] consul: New leader elected: hashicorp-consul-server-1
    2019/12/26 03:47:52 [INFO] agent: Synced node info
    2019/12/26 03:47:59 [INFO] serf: EventMemberJoin: master 10.244.0.84
    2019/12/26 03:48:18 [INFO] agent: (LAN) joining: [hashicorp-consul-server-0.hashicorp-consul-server.default.svc hashicorp-consul-server-1.hashicorp-consul-server.default.svc hashicorp-consul-server-2.hashicorp-consul-server.default.svc]
    2019/12/26 03:48:18 [INFO] agent: (LAN) joined: 3
    2019/12/26 03:48:18 [INFO] agent: Join LAN completed. Synced with 3 initial agents

And the kube version is 1.16.2. bootstrapped by kubeadm , stand-alone. Need help.

[lkysow]

@yueru-mylove it looks like in the end of your logs the consul cluster comes up correctly. Sometimes KubeDNS takes a while to start working so the initial errors are expectes. Are you noticing some functionality not working? Please open up a new issue if so because I don't see the encryption errors in your logs which was the reason for this issue.

[yueru-mylove]

@Ikysow Thankyou so much, sir. It working normaly. And I just have a question if a consul dns is must when deploying consul-cluster with k8s as Ｉ ever got a failed deploying without consul dns using customized sts, svc, sa, cm yaml file while each part is working normally.
And the error log is as belows:

2020/01/02 04:48:06 [ERR] agent: failed to sync remote state: No cluster leader
2020/01/02 04:48:18 [ERR] agent: Coordinate update error: No cluster leader
2020/01/02 04:48:41 [ERR] agent: failed to sync remote state: No cluster leader
2020/01/02 04:48:51 [ERR] agent: Coordinate update error: No cluster leader

Thank a lot.

[lkysow]

Consul DNS is not required. Those errors happen when the servers can't elect a leader. I'd look at the logs on each server to try and figure out what's going on.

What exactly do you mean you got failed deployment without Consul DNS? How are you turning it on/off? With:

dns:
  enabled: false

? This won't affect leader election.

[yueru-mylove]

The consul-0 starting logs is as belows:

==> Log data will now stream in as it occurs:

2020/01/03 01:57:14 [INFO]  raft: Initial configuration (index=0): []
2020/01/03 01:57:14 [INFO] serf: EventMemberJoin: consul-0.dc1 10.244.0.102
2020/01/03 01:57:14 [INFO]  raft: Node at 10.244.0.102:8300 [Follower] entering Follower state (Leader: "")
2020/01/03 01:57:14 [INFO] serf: EventMemberJoin: consul-0 10.244.0.102
2020/01/03 01:57:14 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2020/01/03 01:57:14 [INFO] serf: Attempting re-join to previously known node: consul-2: 10.244.0.101:8301
2020/01/03 01:57:14 [INFO] serf: Attempting re-join to previously known node: consul-2.dc1: 10.244.0.101:8302
2020/01/03 01:57:14 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2020/01/03 01:57:14 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2020/01/03 01:57:14 [INFO] consul: Adding LAN server consul-0 (Addr: tcp/10.244.0.102:8300) (DC: dc1)
2020/01/03 01:57:14 [INFO] consul: Handled member-join event for server "consul-0.dc1" in area "wan"
2020/01/03 01:57:14 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s mdns os packet scaleway softlayer triton vsphere
2020/01/03 01:57:14 [INFO] agent: Joining LAN cluster...
2020/01/03 01:57:14 [INFO] agent: (LAN) joining: [consul-0.consul.default.svc.cluster.local consul-1.consul.default.svc.cluster.local consul-2.consul.default.svc.cluster.local]
2020/01/03 01:57:14 [INFO] agent: started state syncer

==> Consul agent running!
2020/01/03 01:57:14 [WARN] memberlist: Failed to resolve consul-1.consul.default.svc.cluster.local: lookup consul-1.consul.default.svc.cluster.local on 10.10.0.10:53: no such host
2020/01/03 01:57:14 [WARN] memberlist: Failed to resolve consul-2.consul.default.svc.cluster.local: lookup consul-2.consul.default.svc.cluster.local on 10.10.0.10:53: no such host
2020/01/03 01:57:14 [INFO] agent: (LAN) joined: 1
2020/01/03 01:57:14 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2020/01/03 01:57:17 [WARN] serf: Failed to re-join any previously known node
2020/01/03 01:57:17 [WARN] serf: Failed to re-join any previously known node
2020/01/03 01:57:21 [ERR] agent: failed to sync remote state: No cluster leader
2020/01/03 01:57:22 [WARN] raft: no known peers, aborting election
2020/01/03 01:57:42 [ERR] agent: Coordinate update error: No cluster leader
2020/01/03 01:57:54 [ERR] agent: failed to sync remote state: No cluster leader

And consul-1 and consul-2 only has log:

==> Log data will now stream in as it occurs:

So I really don't know why I got this. Consul service:

apiVersion: v1
kind: Service
metadata:
   name: consul
   labels:
     name: consul
spec:
   type: ClusterIP
   clusterIP: None
   ports:
    - name: http
      port: 8500
      targetPort: 8500
    - name: https
      port: 8443
      targetPort: 8443
    - name: rpc
      port: 8400
      targetPort: 8400
    - name: serflan-tcp
      protocol: "TCP"
      port: 8301
      targetPort: 8301
    - name: serflan-udp
      protocol: "UDP"
      port: 8301
      targetPort: 8301
    - name: serfwan-tcp
      protocol: "TCP"
      port: 8302
      targetPort: 8302
    - name: serfwan-udp
      protocol: "UDP"
      port: 8302
      targetPort: 8302
    - name: server
      port: 8300
      targetPort: 8300
    - name: consuldns
      port: 8600
      targetPort: 8600
   selector:
    app: consul

sts file:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
spec:
  serviceName: consul
  replicas: 3
  selector:
    matchLabels:
      app: consul
  template:
    metadata:
      labels:
        app: consul
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: consul
          image: consul:latest
          args:
            - "agent"
            - "-server"
            - "-bootstrap-expect=3"
            - "-ui"
            - "-data-dir=/consul/data"
            - "-bind=0.0.0.0"
            - "-client=0.0.0.0"
            - "-advertise=$(PODIP)"
            - "-retry-join=consul-0.consul.$(NAMESPACE).svc.cluster.local"
            - "-retry-join=consul-1.consul.$(NAMESPACE).svc.cluster.local"
            - "-retry-join=consul-2.consul.$(NAMESPACE).svc.cluster.local"
            - "-domain=cluster.local"
            - "-disable-host-node-id"
          volumeMounts:
            - name: data
              mountPath: /consul/data
          env:
            - name: PODIP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8500
              name: ui-port
            - containerPort: 8400
              name: alt-port
            - containerPort: 53
              name: udp-port
            - containerPort: 8443
              name: https-port
            - containerPort: 8080
              name: http-port
            - containerPort: 8301
              name: serflan
            - containerPort: 8302
              name: serfwan
            - containerPort: 8600
              name: consuldns
            - containerPort: 8300
              name: server
      volumes:
        - name: data
          hostPath:
            path: /home/data

Thank you so much.

[lkysow]

@yueru-mylove I'm moving this to #323. Please follow up there.

[yueru-mylove]

@yueru-mylove I'm moving this to #323. Please follow up there.

Thank you so much. sir.

[zonArt]

Hi,

Don't know if I should open a new issue or can re-use this one but I have the exact same issue (well.... maybe it's not exactly the same). I'm trying to deploy consul clients within kubernetes and got the servers outside. When I do so I receive this error message: Failed to join <server IPs>: Remote state is encrypted and encryption is not configured. I agree I haven't setup gossip encryption as I'm using mTLS so I supposed setting tls was enough but it doesn't seem to. Here is my values.yaml file:

server:
  enabled: false

externalServers:
  enabled: true
  hosts: ["<node1.my.domain>", "<node2.my.domain>", "<node3.my.domain>"]
  useSystemRoots: true

client:
  image: consul:1.7.0
  join:  ["<node1.my.domain>", "<node2.my.domain>", "<node3.my.domain>"]

global:
  domain: <my.domain>
  tls:
    enabled: true
    caCert:
      secretName: consul-agent
      secretKey: tls.crt
    caKey:
      secretName: consul-agent
      secretKey: tls.key

Might not be fully complete so that's why I'm asking for help

Additional information:
I'm using kubernetes 1.18 on a self hosted cluster, consul is also running internally on three VMs with version 1.7.0 thus tried to deploy 1.7.0 agents, deployed with helm 2 and version 0.24.1 of the helm chart

[lkysow]

@zonArt can you please open a new issue and we'll respond there. Thanks!

[zonArt]

@zonArt can you please open a new issue and we'll respond there. Thanks!

Hi @lkysow, thanks, I've just opened it here (with same content)

[Jorgevillada]

if you change default storage in eks(gp2) or configure another storageclass(EFS, nfs, rook). you must guarantee that they are not the same folder for the replicas. the files

checkpoint-signature  
node-id  
raft  
serf

will be overwritten and you won't be able to find the another servers.

[unittolabs]

I faced the same problem at AKS with the given config:

  global:
    enabled: false
    image: consul:1.8.5
    imageK8S: hashicorp/consul-k8s:0.19.0
    datacenter: azure-dev

    acls:
#      manageSystemACLs: true

      bootstrapToken:
        secretName: consul-acl-token
        secretKey: vault

  server:
    enabled: true
    replicas: 3
    bootstrapExpect: 3
    connect: false

    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: 200m
        memory: 256Mi

    affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: {{ template "consul.name" . }}
                release: "{{ .Release.Name }}"
                component: server
            topologyKey: kubernetes.io/hostname

  ui:
    enabled: true

but what is interesting, it that I don't face it when I try to deploy the same config in GKE.

logs from AKS:

2020-11-13T10:57:28.801Z [WARN]  agent: Join cluster failed, will retry: cluster=LAN retry_interval=30s error=<nil>
2020-11-13T10:57:30.044Z [ERROR] agent: Coordinate update error: error="No cluster leader"
2020-11-13T10:57:37.912Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
2020-11-13T10:57:58.802Z [INFO]  agent: (LAN) joining: lan_addresses=[vault-consul-server-0.vault-consul-server.vault.svc, vault-consul-server-1.vault-consul-server.vault.svc, vault-consul-server-2.vault-consul-server.vault.svc]
2020-11-13T10:57:58.813Z [WARN]  agent.server.memberlist.lan: memberlist: Failed to resolve vault-consul-server-0.vault-consul-server.vault.svc: lookup vault-consul-server-0.vault-consul-server.vault.svc on 10.0.0.10:53: no such host
2020-11-13T10:57:59.057Z [WARN]  agent.server.memberlist.lan: memberlist: Failed to resolve vault-consul-server-1.vault-consul-server.vault.svc: lookup vault-consul-server-1.vault-consul-server.vault.svc on 10.0.0.10:53: no such host
2020-11-13T10:57:59.392Z [WARN]  agent.server.memberlist.lan: memberlist: Failed to resolve vault-consul-server-2.vault-consul-server.vault.svc: lookup vault-consul-server-2.vault-consul-server.vault.svc on 10.0.0.10:53: no such host
2020-11-13T10:57:59.392Z [WARN]  agent: (LAN) couldn't join: number_of_nodes=0 error="3 errors occurred:
* Failed to resolve vault-consul-server-0.vault-consul-server.vault.svc: lookup vault-consul-server-0.vault-consul-server.vault.svc on 10.0.0.10:53: no such host
* Failed to resolve vault-consul-server-1.vault-consul-server.vault.svc: lookup vault-consul-server-1.vault-consul-server.vault.svc on 10.0.0.10:53: no such host
* Failed to resolve vault-consul-server-2.vault-consul-server.vault.svc: lookup vault-consul-server-2.vault-consul-server.vault.svc on 10.0.0.10:53: no such host

[lkysow]

@unitto1 can you open up a new issue please?

[unittolabs]

@unitto1 can you open up a new issue please?

In my case, it was fixed via moving to the new AKS cluster, so create the new issue will be not reasonable.
Seems it's some weird error linked to the misconfiguration of Kubenetes Storage Class but I'm not sure.