This is a Docker project to bring up a local Riak CS cluster.
Follow the instructions on Docker's website to install Docker 0.10.0+.
From there, ensure that your DOCKER_HOST
environmental variable is set
correctly:
$ export DOCKER_HOST="tcp://192.168.59.103:2375"
Note: If you're using boot2docker
ensure that you forward the virtual machine port range (49000-49900
, 8080
,
8888
). This
will allow you to interact with the containers as if they were running
locally:
$ for i in {49000..49900}; do
VBoxManage modifyvm "boot2docker-vm" --natpf1 "tcp-port$i,tcp,,$i,,$i";
VBoxManage modifyvm "boot2docker-vm" --natpf1 "udp-port$i,udp,,$i,,$i";
done
$ for i in {8080,8888}; do
VBoxManage modifyvm "boot2docker-vm" --natpf1 "tcp-port$i,tcp,,$i,,$i";
VBoxManage modifyvm "boot2docker-vm" --natpf1 "udp-port$i,udp,,$i,,$i";
done
In order to tune the Docker host housing Riak containers, consider applying
the following sysctl
settings:
vm.swappiness = 0
net.ipv4.tcp_max_syn_backlog = 40000
net.core.somaxconn = 40000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_moderate_rcvbuf = 1
In order to interact with the entire cluster through one endpoint, the
DOCKER_RIAK_CS_HAPROXY
environmental variable makes use of the
hectcastro/riak-cs-haproxy
image. This image automatically load balances incoming HTTP requests to port
8080
against linked Riak CS containers.
$ git clone https://github.com/hectcastro/docker-riak-cs.git
$ cd docker-riak-cs
$ make build
DOCKER_RIAK_CS_CLUSTER_SIZE
– The number of nodes in your Riak CS cluster (default:5
)DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING
– A flag to automatically cluster Riak (default:false
)DOCKER_RIAK_CS_HAPROXY
- Enable an HAProxy container to load balance requests against all Riak CS nodes (default:false
)DOCKER_RIAK_CS_DEBUG
– A flag toset -x
on the cluster management scripts (default:false
)
$ DOCKER_RIAK_CS_HAPROXY=1 DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=1 DOCKER_RIAK_CS_CLUSTER_SIZE=5 make start-cluster
./bin/start-cluster.sh
Bringing up cluster nodes:
Successfully brought up [riak-cs01]
Successfully brought up [riak-cs02]
Successfully brought up [riak-cs03]
Successfully brought up [riak-cs04]
Successfully brought up [riak-cs05]
Successfully brought up [riak-cs-haproxy]
Riak CS credentials:
admin_key: VA4H7GSPO1J0NKMYT-TJ
admin_secret: GvaJALz20W4-Xb330SBft8kPK3d-KKgG4fAMdA==
Please wait approximately 30 seconds for the cluster to stabilize.
From outside the container, we can interact with the HTTP interfaces of Riak and Riak CS. Additionally, the Riak CS HTTP interface supports an Amazon S3 or OpenStack Swift compatible API.
Riak's HTTP interface has an endpoint called /stats
that emits Riak
statistics. The test-cluster
Makefile
target hits a random container's
/stats
endpoint and pretty-prints its output to the console.
The most interesting attributes for testing cluster membership are
ring_members
:
$ make test-cluster | egrep -A6 "ring_members"
"ring_members": [
"[email protected]",
"[email protected]",
"[email protected]",
"[email protected]",
"[email protected]"
],
And ring_ownership
:
$ make test-cluster | egrep "ring_ownership"
"ring_ownership": "[{'[email protected]',3},\n {'[email protected]',4},\n {'[email protected]',3},\n {'[email protected]',4},\n {'[email protected]',3},\n {'[email protected]',4},\n {'[email protected]',3},\n {'[email protected]',4},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3},\n {'[email protected]',3}]",
Together, these attributes let us know that this particular Riak node knows about all of the other Riak instances.
s3cmd
is convenient command-line tool to test the Riak CS Amazon S3
compatible API. Unfortunately, there is no easy way to extract the admin_key
and admin_secret
needed for s3cmd
to connect to the cluster.
First, we have to SSH into one of the Riak CS containers (see SSH section below for details):
$ ssh -i insecure_key [email protected]
Next, we extract the admin_key
and admin_secret
from the Riak CS
configuration file:
root@90caa115f34f:~# egrep "admin_key" /etc/riak-cs/app.config | cut -d'"' -f2
AU4RL35KFK4N1EFTA0LO
root@90caa115f34f:~# egrep "admin_secret" /etc/riak-cs/app.config | cut -d'"' -f2
9EXxoSTLzrJFkwBDk2lijWiQiSeSa3o7eZOQ-w==
Then, we need to the port mappings for 8080
. For example, here's how to get
the port mapping for riak-cs01
:
root@90caa115f34f:~# exit
$ docker port riak-cs01 8080 | cut -d":" -f2
49158
Now we have everything needed to connect to the cluster with s3cmd
:
$ s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Access Key: AU4RL35KFK4N1EFTA0LO
Secret Key: 9EXxoSTLzrJFkwBDk2lijWiQiSeSa3o7eZOQ-w==
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/local/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name: localhost
HTTP Proxy server port [3128]: 49158
New settings:
Access Key: AU4RL35KFK4N1EFTA0LO
Secret Key: 9EXxoSTLzrJFkwBDk2lijWiQiSeSa3o7eZOQ-w==
Encryption password:
Path to GPG program: /usr/local/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name: localhost
HTTP Proxy server port: 49158
Test access with supplied credentials? [Y/n] y
Please wait...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y
Configuration saved to '/Users/hector/.s3cfg'
The phusion/baseimage-docker
image has the ability to enable an insecure key for conveniently logging
into a container via SSH. It is enabled in the Dockerfile
by default here:
RUN /usr/sbin/enable_insecure_key
In order to login to the container via SSH using the insecure key, follow the steps below.
Use docker inspect
to determine the container IP address:
$ docker inspect $CONTAINER_ID | egrep IPAddress
"IPAddress": "172.17.0.2",
Download the insecure key, alter its permissions, and use it to SSH into the container via its IP address:
$ curl -o insecure_key -fSL https://github.com/phusion/baseimage-docker/raw/master/image/insecure_key
$ chmod 600 insecure_key
$ ssh -i insecure_key [email protected]
Note: If you're using
boot2docker, ensure that you're
issuing the SSH command from within the virtual machine running boot2docker
.
$ make stop-cluster
./bin/stop-cluster.sh
Stopped the cluster and cleared all of the running containers.