apache / accumulo-docker

Apache Accumulo Docker

Apache Accumulo Docker Image

This is the first release of this project. Eventually, this project will create an apache/accumulo image at DockerHub. Until then, you will need to build your own image. The main branch of this repo creates a Docker image for Accumulo 2.0+. If you want to create a Docker image for Accumulo 1.9, there is a 1.9 branch for that.

Obtain the Docker image

To obtain the docker image created by this project, you can either pull it from DockerHub at apache/accumulo or build it yourself. To pull the image from DockerHub, run the command below:

docker pull apache/accumulo

While it is easier to pull from DockerHub, the image will default to the software versions below:

Software	Version
Accumulo	2.1.2
Hadoop	3.3.6
ZooKeeper	3.8.2

If these versions do not match what is running on your cluster, you should consider building your own image with matching versions. However, Accumulo must be 2.0.0+. Below are instructions for building an image:

1. Clone the Accumulo docker repo

    git clone [email protected]:apache/accumulo-docker.git
   

2. Build the default Accumulo docker image using the command below.

    cd /path/to/accumulo-docker
    docker build -t accumulo .
   

   Or build the Accumulo docker image with specific released versions of Hadoop, Zookeeper, etc that will downloaded from Apache using the command below:

    docker build --build-arg ZOOKEEPER_VERSION=3.4.8 --build-arg HADOOP_VERSION=2.7.0 -t accumulo .
   

   Or build with an Accumulo tarball (located in same directory as DockerFile) using the command below:

    docker build --build-arg ACCUMULO_FILE=accumulo-<version>-bin.tar.gz -t accumulo .
   

Image basics

The entrypoint for the Accumulo docker image is the accumulo script. While the primary use case for this image is to start Accumulo processes (i.e tserver, master, etc), you can run other commands in the accumulo script to test out the image:

# No arguments prints Accumulo command usage
docker run accumulo
# Print Accumulo version
docker run accumulo version
# Print Accumulo classpath
docker run accumulo classpath

Run Accumulo using Docker

Before you can run Accumulo services in Docker, you will need to install Accumulo, configure accumulo.properties, and initialize your instance with --upload-accumulo-props. This will upload configuration to Zookeeper and limit how much configuration needs to be set on the command line.

$ accumulo init --upload-accumulo-props
...
Uploading properties in accumulo.properties to Zookeeper. Properties that cannot be set in Zookeeper will be skipped:
Skipped - instance.secret = <hidden>
Skipped - instance.volumes = hdfs://localhost:8020/accumulo
Skipped - instance.zookeeper.host = localhost:2181
Uploaded - table.durability = flush
Uploaded - tserver.memory.maps.native.enabled = false
Uploaded - tserver.readahead.concurrent.max = 64
Uploaded - tserver.server.threads.minimum = 64
Uploaded - tserver.walog.max.size = 512M

Any configuration that is skipped above will need to be passed in as a command line option to Accumulo services running in Docker containers. These options can be set in an environment variable which is used in later commands.

export ACCUMULO_CL_OPTS="-o instance.secret=mysecret -o instance.volumes=hdfs://localhost:8020/accumulo -o instance.zookeeper.host=localhost:2181"

The Accumulo docker image expects that the HDFS path set by instance.volumes is owned by the accumulo user. This can be accomplished by running the command below (replace the HDFS path with yours):

hdfs dfs -chown -R accumulo hdfs://localhost:8020/accumulo

Docker engine

Use the docker command to start local docker containers. The commands below will start a local Accumulo cluster with two tablet servers.

docker run -d --network="host" accumulo monitor $ACCUMULO_CL_OPTS
docker run -d --network="host" accumulo tserver $ACCUMULO_CL_OPTS
docker run -d --network="host" accumulo tserver $ACCUMULO_CL_OPTS
docker run -d --network="host" accumulo master $ACCUMULO_CL_OPTS
docker run -d --network="host" accumulo gc $ACCUMULO_CL_OPTS

If you would like to set Java heap size inside the Docker container, start the local docker container using the command below:

docker run -e ACCUMULO_JAVA_OPTS='-Xmx1g' -d --network="host" accumulo tserver $ACCUMULO_CL_OPTS

Marathon

Using the Marathon UI, you can start Accumulo services using the following JSON configuration template. The template is configured to start an Accumulo monitor but it can be modified to start other Accumulo services such as master, tserver and gc. For tablet servers, set instances to the number of tablet servers that you want to run.

{
  "id": "accumulo-monitor",
  "cmd": "accumulo monitor -o instance.secret=mysecret -o instance.volumes=hdfs://localhost:8020/accumulo -o instance.zookeeper.host=localhost:2181",
  "cpus": 1,
  "mem": 512,
  "disk": 0,
  "instances": 1,
  "container": {
    "docker": {
      "image": "apache/accumulo",
      "network": "HOST"
    },
    "type": "DOCKER"
  }
}

accumulo-docker's Issues

Ports exposed by different components of Accumulo

Any reason why no documentation exists for ports that are used and exposed when running this image?

Verify the accumulo-docker is registered under Apache OSS program

Docker may be changing registration requirements or process relating to organizations with publishing artifacts in DockerHub. Accumulo docker is likely covered under the Apache umbrella. Check that accumulo-docker is properly registered as an Apache project, and if not, complete the necessary registration process for DockerHub.

Accumulo 2.0.0 is out...

Hi,

Ref. the README; will you publish an official image soon?

Following the readme to get running fails

I was following the README to get accumulo running with docker but I was unable to do so without some changes to the Dockerfile. When I ran the Docker build, it would fail to complete the move of hadoop into /opt/hadoop. I was able to get around the issue by switching from moving those dirs to copying them, but that isn't a great solution in the long term. I got into the container at the point just before the move and the file permissions for some the directories under /tmp/hadoop/ were all question marks (???????) and the user and group were just listed as an id.

I think we probably want to add a umask and --no-same-permissions to the tar extract commands to correct that. I will try that out tonight and if that works I will put in a PR.

Instructions are a bit confusing

In the README.md file, I see the below text

Before you can run Accumulo services in Docker, you will need to install Accumulo, configure accumulo.properties, and initialize your instance with --upload-accumulo-props. This will upload configuration to Zookeeper and limit how much configuration needs to be set on the command line.

```bash
$ accumulo init --upload-accumulo-props
...

So, where is this supposed to be invoked? Are we to run this as docker run command or am I missing something here?

Unable to run accumulo

I built the image and ran docker run accumulo and I got:

HADOOP_HOME=/path/to/hadoop is not set to a valid directory in accumulo-env.sh

I added HADOOP_HOME to Dockerfile and then got:

2018-09-10 21:25:49,371 [start.Main] ERROR: Unable to find Hadoop Configuration class on classpath, check configuration.
java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at org.apache.accumulo.start.classloader.AccumuloClassLoader$1.loadClass(AccumuloClassLoader.java:198)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.accumulo.start.Main.main(Main.java:52)

I added ENV HADOOP_CONF_DIR $HADOOP_HOME/etc/hadoop and I still get Unable to find Hadoop Configuration class on classpath, check configuration..

accumulo init

Running a new image,
$> docker run accumulo init . Hangs, warns to be sure Zookeeper is running.
$> docker run accumulo sookeeper . Problem reading instance id out of hdfs at hdfs://localhost:8020/accumulo/instance_id

Do we have a catch 22?