kelseyhightower / kubernetes-redis-cluster

Kubernetes Redis Cluster configs and tutorial

Kubernetes Redis Cluster

Create Disks

gcloud compute disks create --size=10GB \
  'redis-1' 'redis-2' 'redis-3' \
  'redis-4' 'redis-5' 'redis-6'

Create Redis Cluster Configuration

kubectl create configmap redis-conf --from-file=redis.conf

Create Redis Nodes

kubectl create -f replicasets

Create Redis Services

kubectl create -f services

Connect Nodes

kubectl run -i --tty ubuntu --image=ubuntu \
  --restart=Never /bin/bash

apt-get update
apt-get install ruby vim wget redis-tools
wget http://download.redis.io/redis-stable/src/redis-trib.rb

./redis-trib.rb create --replicas 1 \
  10.131.242.1:6379 \
  10.131.242.2:6379 \
  10.131.242.3:6379 \
  10.131.242.4:6379 \
  10.131.242.5:6379 \
  10.131.242.6:6379

Add a new node

gcloud compute disks create --size=10GB 'redis-7'

kubectl create -f replicaset/redis-7.yaml

kubectl create -f services/redis-7.yaml

CLUSTER MEET 10.131.242.7 6379

kubernetes-redis-cluster's Issues

hard coded clusterIP in service definition

I see hard-coded clusterIP values e.g. clusterIP: "10.131.242.1"
I tried to assign clusterIP based on the pods ip kubectl get pods -o wide
But service create fails
Error from server: error when creating "services/redis-1.yaml": Service "redis-1" is invalid: spec.clusterIP: Invalid value: "10.0.1.4": provided IP is not in the valid range...

I also tried to use CLUSTER-IP from kubectl get svc -o wide
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes 10.3.240.1 443/TCP 46m
redis 10.3.246.70 6379/TCP,16379/TCP 43m app=redis

but that also fails as invalid.

What will be a good clusterIP to define in the instance service?

error redis cluster

Hello after applying the yaml, we have this error on cluster AKS
pod/redis-cluster-0 0/1 CrashLoopBackOff 1 33s

$ kubectl logs redis-cluster-0 -n ope
standard_init_linux.go:228: exec user process caused: exec format error

What is the probleme pls

GCE access to GKE redis cluster?

This is great for redis traffic inside your GKE cluster. Is there any way to make the redis cluster on GKE so it is also available to GCE instances in the same project, but not the entire world? Either a private L4 load balancer (GCP project network address 10.240.x.x) or firewall rules so GCE instances can access the redis cluster but its blocked outside your GCP project?

Simple Redis setup on GKE?

Hey Kelsey - there are Redis examples in Kubernetes/examples (all slightly different) and this one. but basically if I start simple with one deployment with 1 replica of a Redis pod, with Redis in append only mode sitting on GCE persistent disk, then for whatever reason if it fails won't the deployment just recreate the pod and I am back in business? Oh, and a service of course.

Not sure why I need all the complexity of master/slave/sentinel or clustering? If I really want to go high availability then wouldn't using the recommended solution, e.g. "Redis Sentinel is the official high availability solution for Redis."?

Basically I am just looking for guidance on how to set up a "reasonably" high availability redis service on GKE. Any thing you want to point me at?

infinite Waiting for the cluster to join...............................................

HI, i tried to run a redis cluster in openshift which bases on k8s following the steps describes in this repo, but there is a problem of infinite Waiting for the cluster to join..... when the redis.rb was executed.

the six pods are running correctly. and here is the log of redis.rb:

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.30.122.99:6379
172.30.122.98:6379
172.30.122.97:6379
Adding replica 172.30.122.96:6379 to 172.30.122.99:6379
Adding replica 172.30.122.95:6379 to 172.30.122.98:6379
Adding replica 172.30.122.94:6379 to 172.30.122.97:6379
M: 7e42913aabc8f55dd412a959da9b0bd58a4133c9 172.30.122.99:6379
   slots:0-5460 (5461 slots) master
M: c2e738fa25345b98ca14d3d8a240980c7ffbda1b 172.30.122.98:6379
   slots:5461-10922 (5462 slots) master
M: 81078cb5c548c036277e70781901e027186c01d6 172.30.122.97:6379
   slots:10923-16383 (5461 slots) master
S: faed0f1ce212fc1e25434ca278b082c52d3668be 172.30.122.96:6379
   replicates 7e42913aabc8f55dd412a959da9b0bd58a4133c9
S: 47147b7aef64511d7fbd33be11593eee0f852c29 172.30.122.95:6379
   replicates c2e738fa25345b98ca14d3d8a240980c7ffbda1b
S: 2c95e9e47cfc0697acaf3f0c233788f11a4f17bd 172.30.122.94:6379
   replicates 81078cb5c548c036277e70781901e027186c01d6
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join......................................................................................................................................

and here is the logs of the six pods:
pod1:

Server started, Redis version 3.2.0
1:M 19 Jul 06:56:32.611 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 19 Jul 06:56:32.611 * The server is now ready to accept connections on port 6379
1:M 19 Jul 06:56:55.376 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
1:M 19 Jul 06:56:55.400 # IP address for this node updated to 10.1.19.36

pod2:

1:M 19 Jul 06:56:29.264 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 19 Jul 06:56:29.264 # Server started, Redis version 3.2.0
1:M 19 Jul 06:56:29.265 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 19 Jul 06:56:29.265 * The server is now ready to accept connections on port 6379
1:M 19 Jul 06:56:57.672 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH

pod3:

1:M 19 Jul 06:56:22.636 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 19 Jul 06:56:22.636 # Server started, Redis version 3.2.0
1:M 19 Jul 06:56:22.636 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 19 Jul 06:56:22.636 * The server is now ready to accept connections on port 6379
1:M 19 Jul 06:56:51.286 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH

pod 4 pod 5 and pod6 are the same as pod 3, the only difference is "configEpoch set to 4,5,6 via CLUSTER SET-CONFIG-EPOCH"

anyone could help~~~~~