GithubHelp home page GithubHelp logo

consul's Introduction

Consul with the Autopilot Pattern

Consul in Docker, designed to be self-operating according to the autopilot pattern. This application demonstrates support for configuring the Consul raft so it can be used as a highly-available discovery catalog for other applications using the Autopilot pattern.

DockerPulls DockerStars

Run consul in a virtualbox swarm cluster

  1. If you don't have it, install the Docker for Windows, docker for Mac, or Docker Toolbox(including docker and docker-compose) on your laptop or other environment.
  2. Create a swarm mode cluster

The option --engine-opt experimental is not mandatory but on docker v1.13 you can run docker service logs consul, to view the logs from all the cluster service.

for i in 1 2 3; do
    docker-machine create -d virtualbox --engine-opt experimental swarm-$i
done

eval $(docker-machine env swarm-1)

docker swarm init --advertise-addr $(docker-machine ip swarm-1)

TOKEN=$(docker swarm join-token -q manager)

for i in 2 3; do
  eval $(docker-machine env swarm-$i)
  docker swarm join --token $TOKEN --advertise-addr $(docker-machine ip swarm-$i) $(docker-machine ip swarm-1):2377
done

  1. Create a network for consul
docker network create consul-net -d overlay --subnet=172.20.0.0/24

The option --subnet is not mandatory, is just I want to be on different network that the usually 10.x.x.x that docker assign by default.

  1. Create a service for the swarm

1.1 Using docker-compose v1.10+

docker deploy -c docker-compose.yml consul

1.2 Manually creating the service

docker service create --network=consul-net --name=consul \
    -e 'CONSUL_LOCAL_CONFIG={"skip_leave_on_interrupt": true}' \
    -e CONSUL_BIND_INTERFACE='eth0' \
    -e CONSUL=consul \
    -e CONSUL_CHECK_LEADER=true \
    --replicas 3 \
    --update-delay 10s \
    --update-parallelism 1 \
    -p 8500:8500 sdelrio/consul

Environment vars

The image use the official consul as base image, so all environment vars can be used (like CONSUL_LOCAL_CONFIG, CONSUL_BIND_INTERFACE).

  • CONSUL: Name of the service to ask for other consul peers. The image will use docker DNS to find the other peers. Usually this will be the name of the swarm service.
  • CONSUL_CHECK_LEADER: If is true the logs will show each health check interval if the container is the leader and or container's IP and the leader's IP:
consul_consul.3.l0e0zr114x50@swarm-1    | 2017/01/26 00:11:22     [CP] I'm leader (172.20.0.6)
consul_consul.2.qwx39safki82@swarm-2    | 2017/01/26 00:11:26     [CP] Leader is 172.20.0.6, I'm 172.20.0.3
consul_consul.4.qws5isxw6gpm@swarm-3    | 2017/01/26 00:11:27     [CP] Leader is 172.20.0.6, I'm 172.20.0.4
consul_consul.3.l0e0zr114x50@swarm-1    | 2017/01/26 00:11:32     [CP] I'm leader (172.20.0.6)
consul_consul.2.qwx39safki82@swarm-2    | 2017/01/26 00:11:36     [CP] Leader is 172.20.0.6, I'm 172.20.0.3
consul_consul.4.qws5isxw6gpm@swarm-3    | 2017/01/26 00:11:37     [CP] Leader is 172.20.0.6, I'm 172.20.0.4
consul_consul.3.l0e0zr114x50@swarm-1    | 2017/01/26 00:11:42     [CP] I'm leader (172.20.0.6)
  • PROTO: Default value to http, for leader election, we could set this env var to https. So the election will resolve to ${PROTO}://${CONSUL}:8500/v1/status/leader.

Docker Entry Point

The entrypoint will execute consul with containerpilot, you can use the command to set your own parameters, by default the command will need 3 replicas:

agent -server -bootstrap-expect 3 -ui -client=0.0.0.0 -retry-interval 5s --log-level warn -disable-host-node-id

Consul generates node IDs from hostname and my conflict when you run two or more consul containers on the same host. Consul (v0.8.1+) has -disable-host-node-id flag to avoid this issue.

References

consul's People

Contributors

sdelrio avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

consul's Issues

info

how to mount a volume for saving kv data in a specific path?

Updating

Hy,

i've the following config:

  consul:
    image: sdelrio/consul
    labels: [app=consul]
    environment:
      - 'CONSUL_LOCAL_CONFIG={"skip_leave_on_interrupt": true}'
      - CONSUL_BIND_INTERFACE=eth0
      - CONSUL=consul
      - CONSUL_CHECK_LEADER=false
    deploy:
      replicas: 3
      placement:
        constraints: [node.role == manager]
      update_config:
        parallelism: 1
        delay: 30s
        monitor: 10s
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 10
        window: 60s

The Problem is only while updating the stack, it throws about 200 errors, some of them:

"agent failed to sync no cluster leader"
"REQUEST PUT /v1/sessions/renew.."

May there is a way to suppress those messages while updating?

Might be a good idea to add -disable-host-node-id to startup command

Consul generates node ids from hostname and may conflict when you run two or more consul containers in the same host (might happen in swarm mode). Consul 8.1 has -disable-host-node-id flag to avoid this issue. Workarround in docker-compose until you change Dockerfile:

entrypoint: ["/usr/local/bin/containerpilot", "/usr/local/bin/docker-entrypoint.sh" ]
command: ["agent", "-server", "-bootstrap-expect", "3", "-ui", "-client=0.0.0.0", "-retry-interval", "5s", "--log-level", "warn", "-disable-host-node-id" ]

Can't use with persistent storage

I'm trying to use your image with persistent storage, but it is not possible. To simulate the problem, please do the following steps:

How to simulate

1 - Create 3 swarm nodes (could be the managers...)
2 - Create 3 directories inside the nodes (/consul/data for example)
3 - Deploy this swarm stack:

#docker-compose-infra.yml
version: "3.2"

networks:
  out:
    driver: overlay
    ipam:
      driver: default
      config:
        - subnet: 10.253.0.0/16
    attachable: true

services:
  consul:
    image: sdelrio/consul
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      resources:
        reservations:
          memory: 100m
      update_config:
        parallelism: 1
        delay: 10s
    environment:
      - 'CONSUL_LOCAL_CONFIG={"skip_leave_on_interrupt": true}'
      - CONSUL_BIND_INTERFACE=eth0
      - CONSUL=consul
      - CONSUL_CHECK_LEADER=true
    networks:
      - out
    volumes:
      - /consul/data:/consul/data

4 - Wait for service to be up:

infra_consul.0.psuzvybt0ff1@swarm-manager-003    | 2017/07/11 11:11:25     [CP] consul join 10.254.0.32 10.254.0.30
infra_consul.0.psuzvybt0ff1@swarm-manager-003    | 2017/07/11 11:11:25 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.vbhav29dqio9@swarm-manager-001    | 2017/07/11 11:11:25     [CP] consul join 10.254.0.23 10.254.0.32
infra_consul.0.vbhav29dqio9@swarm-manager-001    | 2017/07/11 11:11:25 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.psuzvybt0ff1@swarm-manager-003    | 2017/07/11 11:11:27     [CP] Leader is 10.254.0.30, I'm 10.254.0.23
infra_consul.0.vbhav29dqio9@swarm-manager-001    | 2017/07/11 11:11:28     [CP] I'm leader (10.254.0.30)
infra_consul.0.luufz7r4t1bj@swarm-manager-002    | 2017/07/11 11:11:28     [CP] Leader is 10.254.0.30, I'm 10.254.0.32
infra_consul.0.psuzvybt0ff1@swarm-manager-003    | 2017/07/11 11:11:37     [CP] Leader is 10.254.0.30, I'm 10.254.0.23
infra_consul.0.vbhav29dqio9@swarm-manager-001    | 2017/07/11 11:11:38     [CP] I'm leader (10.254.0.30)
infra_consul.0.luufz7r4t1bj@swarm-manager-002    | 2017/07/11 11:11:38     [CP] Leader is 10.254.0.30, I'm 10.254.0.32

5 - Remove the entire swarm stack or just the service

docker service rm infra_consul
#### OR ####
docker stack rm infra

6 - Deploy the stack again
7 - You will see the bellow errors infinitely and consul will never get started:

infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:09 [ERR] http: Request PUT /v1/agent/check/pass/consul-ui-19f851f8e6fe?note=ok, error: CheckID "consul-ui-19f851f8e6fe" does not have associated TTL from=127.0.0.1:34174
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:09 Unexpected response code: 500 (CheckID "consul-ui-19f851f8e6fe" does not have associated TTL)
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | Service not registered, registering...
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.21:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:11 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:12 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:13 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.21:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:13 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.23:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:16 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.30:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:16 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:17 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:19 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:21 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.21:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:22 [WARN] raft: Heartbeat timeout from "" reached, starting election
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:23 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:24 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:24 [ERR] agent: coordinate update error: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.30:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:26 [WARN] raft: Heartbeat timeout from "" reached, starting election
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:26 [ERR] agent: coordinate update error: No cluster leader
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:28 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:28 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:29 [ERR] agent: coordinate update error: No cluster leader
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.23:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:30 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:30     [CP] No peers connected from 40ee4b0139a9. Joining peers on DNS tasks.consul
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:31 [ERR] agent: failed to sync remote state: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:31 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:31 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:31     [CP] No peers connected from 19f851f8e6fe. Joining peers on DNS tasks.consul
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:31 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.21:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:32 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.23:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:33     [CP] consul join 10.254.0.23 10.254.0.30
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:33 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.30:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:34     [CP] consul join 10.254.0.21 10.254.0.23
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:34 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:35 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:35 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:35 [ERR] agent: failed to sync remote state: No cluster leader
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:37 [ERR] agent: failed to sync remote state: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.21:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:38 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:40 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:40     [CP] No peers connected from 19f851f8e6fe. Joining peers on DNS tasks.consul
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:40     [CP] consul join 10.254.0.23 10.254.0.30
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:40 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:41 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.30:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:41 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:41     [CP] No peers connected from 40ee4b0139a9. Joining peers on DNS tasks.consul
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:42 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:43 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:43     [CP] consul join 10.254.0.21 10.254.0.23
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:43 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.23:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:45 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.21:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:47 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:47     [CP] No peers connected from 19f851f8e6fe. Joining peers on DNS tasks.consul
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:47     [CP] consul join 10.254.0.23 10.254.0.30
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:47 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.30:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:50 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:50 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:50     [CP] No peers connected from 40ee4b0139a9. Joining peers on DNS tasks.consul
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:50     [CP] consul join 10.254.0.23 10.254.0.21
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:50 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:52 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:52 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.23:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:53 [ERR] agent: coordinate update error: No cluster leader
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:53 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:55 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    |     2017/07/11 11:22:55 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.21:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:55 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.30:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:55 [WARN] raft: Election timeout reached, restarting election
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:55 [ERR] agent: coordinate update error: No cluster leader
infra_consul.0.xndxnsidf4gp@swarm-manager-003    |     2017/07/11 11:22:56 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.254.0.32:8300 10.254.0.32:8300}: dial tcp 10.254.0.23:0->10.254.0.32:8300: getsockopt: no route to host
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:57     [CP] No peers connected from 19f851f8e6fe. Joining peers on DNS tasks.consul
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:57     [CP] consul join 10.254.0.30 10.254.0.23
infra_consul.0.y5b94v8hhlp8@swarm-manager-002    | 2017/07/11 11:22:57 Successfully joined cluster by contacting 2 nodes.
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:57 [ERR] agent: failed to sync changes: No cluster leader
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    | 2017/07/11 11:22:57     [CP] No peers connected from 40ee4b0139a9. Joining peers on DNS tasks.consul
infra_consul.0.rte5uwvdyfdv@swarm-manager-001    |     2017/07/11 11:22:58 [WARN] raft: Election timeout reached, restarting election

How to recover

1 - Delete the entire stack or just the service
2 - Remove all contents from /consul/data/ from all of the 3 nodes
3 - Deploy the stack again

This recovery procedure works to make service available again, but with the cost of wipe the entire consul data.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.