Too many open files about elk-docker

[spujadas]

Hi,

As a preliminary note, I haven't faced your issue myself, but I was able to reproduce it by using Docker's --ulimit option (or, similarly, Docker Compose's ulimits option) to force a much lower value of nofile (specifically by setting the soft and hard limits to 120 open files).

I tried to fix it by trying multiple ways to increase the allowed open files count but nothing seems to work.

Could you please elaborate on what you already tried?

From what I've seen, increasing the maximum number of open files is actually the way to go.

Over here, docker exec-ing in the running container shows that the default ulimit values for the maximum number of open file descriptors are comfortably high:

root@8ca84d162215:/# ulimit -n
524288
root@8ca84d162215:/# ulimit -Hn
1048576
root@8ca84d162215:/# ulimit -Sn
524288

However, the maximum number of open files assigned to Elasticsearch is reduced to 65,535 by Elasticsearch's init file:

# Maximum number of open files
MAX_OPEN_FILES=65535

As confirmed by:

root@8ca84d162215:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  18056  1584 ?        Ss   17:57   0:00 /bin/bash /usr/local/bin/start.sh
elastic+    34  4.5 15.1 3549124 310320 ?      Sl   17:57   0:16 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcM
...

root@8ca84d162215:/# cat /proc/34/limits
Limit                     Soft Limit           Hard Limit           Units
...
Max open files            65535                65535                files

The documentation for Elasticsearch suggests that this is actually a sufficiently high value, so as far as I can tell:

• Either you're hitting this 65k limit, and in that case you'd need to override the MAX_OPEN_FILES variable in ES's init script and increase it to a much larger value (say 200000).

  Could you docker exec in a running container, edit /etc/init.d/elastisearch, change the line that sets MAX_OPEN_FILES to set it to a much larger value (say 200000), do a service elasticsearch restart, and see how it goes?

  If that works, then I'll update the image so that MAX_OPEN_FILES can be overridden at runtime when starting the container using an env var.

• Or your container was created with low ulimits, e.g. lower than the value of MAX_OPEN_FILES, which would show up in the container's logs as:

   * Starting Elasticsearch Server
  /etc/init.d/elasticsearch: 158: ulimit: error setting limit (Operation not permitted)
  

  In that case, could you start the container with Docker's --ulimit option to set the nofile soft and hard values to, say, 65535, and see how it goes?

  If that works, then the image is fine, but I'll add some words on this in the documentation.

If none of these work, then could you report the values of the ulimit and cat /proc/... commands listed above, and could you also docker exec inside the container, do an apt-get install lsof, and report the output of lsof -p <pid of elasticsearch> | wc -l (which will show the number of currently open file descriptors used by Elasticsearch)?

[tallavi]

Thanks for the quick and detailed response. I have a much lower number on my machine (a AWS EC2 instance):

[ec2-user@ip-10-0-0-186 data]$ ulimit -n
1024
[ec2-user@ip-10-0-0-186 data]$ ulimit -Hn
4096
[ec2-user@ip-10-0-0-186 data]$ ulimit -Sn
1024

Same numbers also appear when I exec into the running elk container.

I did follow the solution here: http://stackoverflow.com/questions/11342167/how-to-increase-ulimit-on-amazon-ec2-instance

And changed the number to 200000 in my host machine, but it still didn't change the one inside the container. I figured I need to restart the docker service for the change to take effect, but now the stopping of the service fails:

[ec2-user@ip-10-0-0-186 data]$ service docker stop
/etc/init.d/docker: line 35: ulimit: open files: cannot modify limit: Operation not permitted
Stopping docker:

It seems that the docker script wants to change the value back but it can't?

Since the values are very small on the host machine, and not just inside the container, it makes me think it's not related to setting ulimit when I start the container. Do you tend to agree?

[spujadas]

Hmmm... on my host (Ubuntu 14.04 on VirtualBox/Windows) I have the same figures as you (1024 and 4096), but for both the Docker daemon (cat /proc/<docker daemon process id>/limits) and inside the container I have the much higher values shown above.

I'm unfamiliar with EC2's management of ulimits, but my /etc/init.d/docker sets the ulimit for open files to 1048576, so could you use that value instead of 200000 in your /etc/security/limits.conf? I'm guessing that that's the reason you can't restart the daemon (attempting to increase the ulimit above the maximum allowed 200k).

Then restart the Docker daemon and the container as usual, and have a look at the ulimits for both.
If the daemon is showing a high value and the container isn't, then you can try to start the container with docker's --ulimit option (see https://docs.docker.com/engine/reference/commandline/run/#/set-ulimits-in-container-ulimit).

[spujadas]

Housekeeping: closing this issue, please feel free to reopen if needed.