Extending docker-compose file to add volumes, point to logstash config file about elk-docker

[spujadas]

@Analect Great to hear that the image is (mostly!) working out for you, here are a few suggestions that may help.

Volumes and permissions

The VOLUME directive just makes the directory persistent across restarts (/var/lib/elasticsearch is where Elasticsearch stores its data), and if you want to map it to a directory on your host, then the way you went about binding volumes is the right one.

The most likely reason it's failing is indeed because of permissions… and this is where it gets a tad ugly (see here for Docker permission-related nightmares).

The Elasticsearch process in the container runs as the elasticsearch user (with UID 103, belonging to group elasticsearch, GID 107), so that user or group must be allowed to write to /home/mccoole/Development/Tools/data/elastic/data ($HOST_DIR hereafter for the sake of readability).

Here are three options that should work (in increasing order of cleanliness… and complexity):

• Do a chmod a+w on $HOST_DIR. Quick, but dirty and insecure.
• Do a chgrp 107 (107 is elasticsearch's GID in the image) and a chmod g+w on $HOST_DIR. Slightly better, but hardly portable if elasticsearch's GID changes in a future version of the image, and still not secure if 107 corresponds to an "untrusted" GID on your host.
• Do a chmod g+w on $HOST_DIR, and rework the launch script (start.sh) in the image to dynamically add elasticsearch to $HOST_DIR owner group when the container is started. See here for details on how to do that.

Logstash config file(s)

You want to put config files for Logstash in /etc/logstash/conf.d (here's where the default ones get added to the image in the Dockerfile). They'll be loaded in alphabetical order when Logstash is started, as per the options that are passed to logstash.

tl;dr – logstash's use of the -f option in /etc/init.d/logstash is how the config files' location is declared.

What you did for Kafka in your Dockerfile looks fine to me, not sure what issue you're having there.

Referencing another container in Docker Compose

I assume that you have entry like this in your docker-compose.yml file:

zk:
  - ...

You need to add a links directive pointing to zk to your elk entry:

elk:
  ...
  links:
    - zk

In your configuration file, instead of the ZooKeeper container's IP address, you can now use the hostname zk:

zk_connect => "zk:2181"

Bonus: Logstash web interface

Had added it to the image, but later realised – just as you did – that I shouldn't have so ended up removing it.

Let me know how it goes.

[spujadas]

No news is good news? Closing, but feel free to reopen if needed.

[Analect]

@spujadas
Sorry for the delayed feedback.
I have to admit, the whole permissioning around attached volumes is something I haven't been able to properly resolve. Part of the reason I'm mounting back to the host is to be able to drop in new elasticsearch databases, but often elasticsearch isn't then able to open up these on a container restart. I worked through your suggestions above ...

... but the only 'fix' that has worked for me is to 'chown -R elasticsearch:elasticsearch' the /var/lib/elasticsearch folder in the container. This seems to somehow resolve problems with elasticsearch failing to start. I note from the dockerfile (linked below) in the docker-elasticsearch repo that this is what they seem to do.

https://github.com/airdock-io/docker-elasticsearch/blob/master/Dockerfile

As regards the zk container references within the logstash config file, I've been able to integrate that into the compose file, so thanks for your guidance on that.

Next big step will be getting everything working under ver 2.0 of ES!

[spujadas]

Thanks for the feedback. I've had quite a few headaches of my own with both volume mounting and permissions, and have yet to find a way to make everything work consistently. 😕

Anyway, good to hear that you found a solution that works out for you.
/var/lib/elasticsearch is actually chowned to elasticsearch:elasticsearch in the image when Elasticsearch is installed, but this gets overridden when a volume is mounted, so what you're saying is if you re-chown it after it's mounted then it works… which makes sense, even though having to do this by in the container each time is hardly user-friendly: I'll do a few tests to see if I can put a chown in the start.sh script to automate this in the container without breaking permissions on the host.

And yeah, I also definitely need to get cracking on ES 2 and Logstash 2 RCs and betas!

[LoungeFlyZ]

I'm finding when running docker on OSX that even chown'ing the directory isn't then allowing me to restart the service and have it run correctly. I am mounting a local dir into the contain in my docker compose file. It seems i can run the chown command over and over and keep getting:

changed ownership of '/var/lib/elasticsearch/elasticsearch/' from 1000:staff to elasticsearch:elasticsearch

[spujadas]

@LoungeFlyZ Disclaimer: haven't got an OS X handy and not familiar with the intricacies of HFS+, so I'll do my best for this one.

First of all some questions for the purposes of situational awareness.
Have you tried the three options from my original comment (#7 (comment))?
What are the permissions on the directory you're mounting?
What user are your running docker under?
When chowning from inside the container and getting changed ownership…, what permissions are showing up on /var/lib/elasticsearch?
(By the way couldn't help but notice that you have two elasticsearch/s in /var/lib/elasticsearch/elasticsearch/ in your message – would have expected only one.)
Also, could you post your docker-compose.yml?

[LoungeFlyZ]

@spujadas, thanks for replying. I didn't try the first two as those would mean our team need to do setup steps on each machine when they want to spin up our project.

Permissions on the directory on the host: drwxr-xr-x 4 Chris staff
Docker is running on VirtualBox with boot2docker VM.
When chown'ing the permissions on the dir in the container dont change from 1000 staff no matter how many times i run it.
Sorry about the dir path typo. I can confirm this is the same for the /var/lib/elasticsearch.

Here is the docker compose config i was using at the time:

elk: image: sebp/elk ports: - "5601:5601" - "9200:9200" - "5044:5044" - "5000:5000" volumes: - ./tmp/elasticsearch:/var/lib/elasticsearch

I gave up and moved to mounting a volume from a data container using volumes_from and its working well. From what i read this isn't a problem with your image but an issue many are facing with docker volumes in general. I don't see how you could solve it other than moving to running the app as root or a user that is shared between container and host.

[spujadas]

Unfortunately, you're right: a) there's indeed not much I can do about it, and b) it's a general issue with Docker volumes, which are (still…) a bit of a nightmare. 😕
So indeed if you can't change the permissions of the volumes on your host to match the ones in the container (which admittedly isn't practical), then your volumes_from approach sounds like the best plan B!

[wesleymusgrove]

I was facing a similar issue as @Analect's OP, where the elasticsearch service couldn't start because of volume permissions with /var/lib/elasticsearch being unable to write to a mounted VirtualBox shared folder:

curl: (7) Failed to connect to localhost port 9200: Connection refused
elk_1 | 
elk_1 |   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
elk_1 |   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
elk_1 | Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch
elk_1 |   at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
elk_1 |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
elk_1 |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
elk_1 |   at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308)
elk_1 |   at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:333)
elk_1 |   at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
elk_1 |   ... 7 more
elk_1 | [2016-02-16 20:36:01,782][ERROR][bootstrap                ] Exception
elk_1 | java.lang.IllegalStateException: Unable to access 'path.data' (/var/lib/elasticsearch)
elk_1 |   at org.elasticsearch.bootstrap.Security.addPath(Security.java:316)
elk_1 |   at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:256)
elk_1 |   at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:212)
elk_1 |   at org.elasticsearch.bootstrap.Security.configure(Security.java:118)
elk_1 |   at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:196)
elk_1 |   at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167)
elk_1 |   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
elk_1 |   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
elk_1 | Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch
elk_1 |   at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
elk_1 |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
elk_1 |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
elk_1 |   at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308)
elk_1 |   at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:333)
elk_1 |   at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)

My setup is a Windows 7 host running VirtualBox with an Ubuntu 14.04 guest with Docker running inside Ubuntu. My Ubuntu VM was running out of disk space when I allowed elasticsearch to store it's data there, which is why I was wanting to store it outside Ubuntu in a VirtualBox shared folder instead, where my Windows host has plenty of disk space.

Here's my docker-compose.yml:

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5000:5000"
    - "5044:5044"
  volumes:
    # this is the mounted Vbox share folder
    - /media/sf_Desktop/elastic_data:/var/lib/elasticsearch

The problem was that the ownership of the VirtualBox shared folder is root:vboxsf. So on my Ubuntu VM, I ran vim /etc/group to learn that the GID of the vboxsf group was 999. Then I ran docker exec -it [elk_container_id] bash and then vim /etc/group to see if there was an existing group already using GID 999. It turned out that the logstash and kibana groups created by the Dockerfile were being assigned GIDs of 998 and 999, respectively. So I made the following changes to the Dockerfile to manually assign those groups different GIDs:

&& groupadd -r logstash -g 997 \

&& groupadd -r kibana -g 998 \

Then I created a "fake" vboxsf group and assigned it the GID of 999, so it would match the real vboxsf outside of the docker container in Ubuntu world. And then I added the elasticsearch user to this newly created vboxsf group.

RUN groupadd -r vboxsf -g 999 \
 && usermod -aG vboxsf elasticsearch

Now the elasticsearch user can write to the mounted VirtualBox share folder /media/sf_Desktop/elastic_data.

I realize this kind of group GID hardcoding is a hacky workaround and that the GID of the vboxsf group in Ubuntu can possibly change, rendering a GID mismatch between the docker container's vboxsf group and Ubuntu's vboxsf group, but I'll cross that bridge when I come to it. :)

[spujadas]

Thanks for sharing that, @wesleymusgrove
From what your experience and what I've read here and there, it would seem that "Docker volume permissions" and "hacky" often go hand in hand! 😄