RabbitMQ fails to start, permission issue? about docker-documentserver

[agolybev]

Hi @poVoq, You could find docker-compose project with traefik proxy and document server here.

[JustasStankunas]

@thomass4t, i think i have just found the problem. it looks to be related to already logged issue #68 regards not being able to run onlyoffice without interactive shell. if adding stdin_open: true to docker-compose file (which corresponds to docker -i) problem is gone.

though now it is interesting that @ShockwaveNN @agolybev were able to run this container without stdin_open: true as it looks that docker-compose is starting container without interactive (at least for me).

[snorre-k]

The problem is related to the way, the container is stopped (only tested on Ubuntu).

On the first start, the directory /var/run/rabbitmq does not exist and is created by the Rabbitmq startup script (/etc/init.d/rabbitmq-server).

When stopping (on Ubuntu), there is no graceful shutdown of the container. Instead the container gets killed. You can test this with a huge stop timeout (e.g.: time docker stop -t 180 onlyoffice-document-server - this ends after 180 seconds with sending a kill -9 to the start process).

After this kill, the directory is left and this prevents the next start of Rabbitmq.

The directory /var/run/rabbitmq is removed by the Rabbitmq start script at the failing start.
Therefore the next (second) start goes well.

I have probably fixed this problem by deleting the directory in the Documentserver start script /app/onlyoffice/run-document-server.sh around line 301:

  if [ ${RABBITMQ_SERVER_HOST} != "localhost" ]; then
    update_rabbitmq_setting
  else
    LOCAL_SERVICES+=("rabbitmq-server")
    # allow Rabbitmq startup after container kill
    rm -rf /var/run/rabbitmq
  fi

With this little code a docker restart onlyoffice-document-server works every time. Also a restart of docker itself (systemctl restart docker.service) and a reboot of the host do work.

Maybe you could add this two lines to the container startup file.

[ShockwaveNN]

but then it tries to restart rabbitmq and fails:

But why it try to restart rabbitmq server in first place?
Normal docker start process looks like this:

 * Starting PostgreSQL 9.5 database server                                                                                                                [ OK ] 
 * Starting RabbitMQ Messaging Server rabbitmq-server                                                                                                     [ OK ] 
Starting redis-server: redis-server.
Starting supervisor: supervisord.
 * Starting nginx nginx                                                                                                                                   [ OK ] 
Generating AllFonts.js, please wait...Done
onlyoffice-documentserver:docservice: stopped
onlyoffice-documentserver:docservice: started
onlyoffice-documentserver:converter: stopped
onlyoffice-documentserver:converter: started
 * Reloading nginx configuration nginx 

[poVoq]

Yeah, exactly my question :) There seems to be nothing that I am doing specifically that would cause that.

[ShockwaveNN]

@poVoq How did you start container? Via portainer (of which I'm not familiar)

[poVoq]

Yes, but Portainer is just a GUI for regular Docker commands. Hmm, but I could try again with a docker-compose file (as I need to pass certain labels to the Traefik reverse-proxy) and see if that makes a difference. Would be surprising though.

[JustasStankunas]

i am having same issue, with docker-compose..

  onlyoffice:
    container_name: onlyoffice
    image: onlyoffice/documentserver:latest
    restart: unless-stopped
    expose:
      - "443"
    volumes:
      - onlyoffice-data:/var/www/onlyoffice/Data
      - onlyoffice-log:/var/log/

[agolybev]

Hi @JustasStankunas, could you provide entire 'docker-compose.yml' file?

[JustasStankunas]

(removed sensitive data)

version: '3.4'

services:
  nextcloud-db:
    image: mariadb
    container_name: nextcloud-db
    restart: unless-stopped
    volumes:
      - nextcloud_db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=
    env_file:
      - nextcloud_db.env

  nextcloud-app:
    image: nextcloud:fpm
    container_name: nextcloud-app
    restart: unless-stopped
    volumes:
      - nextcloud_app:/var/www/html
    environment:
      - MYSQL_HOST=nextcloud-db
    env_file:
      - nextcloud_db.env
    depends_on:
      - nextcloud-db

  nextcloud-web:
    build: ./nextcloud-web
    container_name: nextcloud-web
    restart: unless-stopped
    volumes:
      - nextcloud_app:/var/www/html:ro
    environment:
      - VIRTUAL_HOST=
      - VIRTUAL_NETWORK=public_proxy-tier
      - VIRTUAL_PORT=80
      - LETSENCRYPT_HOST=
      - LETSENCRYPT_EMAIL=
    depends_on:
      - nextcloud-app
    networks:
      - proxy-tier
      - default

  onlyoffice:
    container_name: onlyoffice
    image: onlyoffice/documentserver:latest
    restart: unless-stopped
    expose:
      - "443"
    volumes:
      - onlyoffice-data:/var/www/onlyoffice/Data
      - onlyoffice-log:/var/log/

volumes:
  nextcloud_app:
  nextcloud_db:
  onlyoffice-data:
  onlyoffice-log:

networks:
  proxy-tier:
    external:
      name: public_proxy-tier

though not sure what could be causing this issue outside from its container..

[agolybev]

OK @JustasStankunas, I'll test it.

[JustasStankunas]

@agolybev, by the way, nextcloud is working just fine. i am using it already for more then half a year. i just tried to add onlyoffice today.

[agolybev]

@JustasStankunas , unfortunally it work for me.
I've simplified your docker-compose.yml file to this:

version: '2.0'

services:
  onlyoffice:
    container_name: onlyoffice
    image: onlyoffice/documentserver:latest
    restart: unless-stopped
    expose:
      - "443"
    volumes:
      - onlyoffice-data:/var/www/onlyoffice/Data
      - onlyoffice-log:/var/log/

volumes:
  onlyoffice-data:
  onlyoffice-log:

networks:
  proxy-tier:
    external:
      name: public_proxy-tier

And documentserver ran successfully:

$ sudo docker-compose up
WARNING: Some networks were defined but are not used by any service: proxy-tier
Starting onlyoffice ... 
Starting onlyoffice ... done
Attaching to onlyoffice
onlyoffice    |  * Starting PostgreSQL 9.5 database server
onlyoffice    |    ...done.
onlyoffice    |  * Starting RabbitMQ Messaging Server rabbitmq-server
onlyoffice    |    ...done.
onlyoffice    | Starting redis-server: redis-server.
onlyoffice    | Starting supervisor: supervisord.
onlyoffice    |  * Starting nginx nginx
onlyoffice    |    ...done.
onlyoffice    | Generating AllFonts.js, please wait...Done
onlyoffice    | onlyoffice-documentserver:docservice: stopped
onlyoffice    | onlyoffice-documentserver:docservice: started
onlyoffice    | onlyoffice-documentserver:converter: stopped
onlyoffice    | onlyoffice-documentserver:converter: started
onlyoffice    |  * Reloading nginx configuration nginx
onlyoffice    |    ...done.

We'll watch for this issue.

[JustasStankunas]

@agolybev. so it looks there are two different views.. in the attached mode it does not show up anything . but if checked logs it shows up these errors.

then i thought that maybe exposing ports instead of mapping them could be the issue, but changeing this part did not gave any affect.

i have tried running this container without docker-compose and it worked.
docker run -i -t -p 192.168.11.103:80:80 onlyoffice/documentserver

so it definitely looks like something with compose.. i have tried other versions and that resulted into just infinite loop of continues restarting (though now permissions issue).

[ShockwaveNN]

@JustasStankunas Try to check permission on host for onlyoffice-log:/var/log/, (or /var/www/onlyoffice/Data just in case), cause user onlyoffice inside container need write privilege to those dirs

Or run
docker run -i -t -p 192.168.11.103:80:80 -v onlyoffice-data:/var/www/onlyoffice/Data -v onlyoffice-log:/var/log/ onlyoffice/documentserver
and see if it working

[JustasStankunas]

@ShockwaveNN,

docker run -i -t -p 192.168.11.103:80:80 -v onlyoffice-data:/var/www/onlyoffice/Data -v onlyoffice-log:/var/log/ onlyoffice/documentserver
works just fine.

[ShockwaveNN]

@JustasStankunas I'm not 100% sure of that, but seems this is still actual. moby/moby#7198

[JustasStankunas]

though looking now into volumes, it looks that running without docker-compose. this container creates more volumes, then with docker-compose.
for example running it with command:
docker run -i -t -p 192.168.11.103:80:80 -v onlyoffice-data:/var/www/onlyoffice/Data -v onlyoffice-log:/var/log/ onlyoffice/documentserver
i can see additional volumes for these dirs in addition what is defined above:

• /etc/onlyoffice
• /usr/share/fonts/truetype/custom
• /var/lib/postgresql
• /var/lib/onlyoffice
• /var/log/onlyoffice

[ShockwaveNN]

@JustasStankunas Yeah, they mentioned in https://github.com/ONLYOFFICE/Docker-DocumentServer/blob/master/README.md#storing-data but they are not required to mount
At least they are not empty inside container?

[JustasStankunas]

i must take back my comment about persistent volumes. it looks all to be ok..
and by the way if i run onlyoffice with compose it is working (i can open it via web browser) for very short amount of time - until it tries to restart itself.

[ShockwaveNN]

@JustasStankunas So if documentserver trying to restart itself it mean somewhere in logs (/var/log/onlyoffice) there is an error by which it try to restart

[JustasStankunas]

hmm. maybe they are more saying to you then me..

root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat converter/err.log
root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat converter/out.log

[2018-04-04 09:26:05.771] [WARN] nodeJS - update cluster with 1 workers
[2018-04-04 09:26:05.779] [WARN] nodeJS - worker 628 started.
[2018-04-04 09:26:05.780] [WARN] nodeJS - update cluster with 1 workers
[2018-04-04 09:26:13.034] [WARN] nodeJS - update cluster with 1 workers
[2018-04-04 09:26:13.041] [WARN] nodeJS - worker 687 started.
[2018-04-04 09:26:13.041] [WARN] nodeJS - update cluster with 1 workers

root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat docservice/err.log
root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat docservice/out.log

[2018-04-04 09:26:05.696] [WARN] nodeJS - Plugins watch exception (https://nodejs.org/docs/latest/api/fs.html#fs_availability).
[2018-04-04 09:26:05.709] [WARN] nodeJS - update cluster with 1 workers
[2018-04-04 09:26:05.719] [WARN] nodeJS - worker 618 started.
[2018-04-04 09:26:05.720] [WARN] nodeJS - update cluster with 1 workers
[2018-04-04 09:26:06.147] [WARN] nodeJS - Express server starting...
[2018-04-04 09:26:08.801] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode
[2018-04-04 09:26:11.690] [WARN] nodeJS - Plugins watch exception (https://nodejs.org/docs/latest/api/fs.html#fs_availability).
[2018-04-04 09:26:11.699] [WARN] nodeJS - update cluster with 1 workers
[2018-04-04 09:26:11.704] [WARN] nodeJS - worker 664 started.
[2018-04-04 09:26:11.705] [WARN] nodeJS - update cluster with 1 workers
[2018-04-04 09:26:12.105] [WARN] nodeJS - Express server starting...
[2018-04-04 09:26:12.486] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode

root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat gc/err.log
root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat gc/out.log
root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat metrics/err.log
(node:544) DeprecationWarning: process.EventEmitter is deprecated. Use require('events') instead.
root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat metrics/out.log

4 Apr 09:26:05 - [544] reading config file: ../../config/config.js
4 Apr 09:26:05 - server is up INFO

root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat nginx.error.log
root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat spellchecker/err.log
root@ca1b94d32e6c:/var/log/onlyoffice/documentserver# cat spellchecker/out.log

[2018-04-04 09:26:05.435] [WARN] nodeJS - start cluster with 1 workers
[2018-04-04 09:26:05.452] [WARN] nodeJS - worker 604 started.
[2018-04-04 09:26:06.028] [WARN] nodeJS - Express server starting...
[2018-04-04 09:26:06.081] [WARN] nodeJS - Express server listening on port 8080 in production-linux mode

[ShockwaveNN]

@JustasStankunas docservice/out.log seem weird, starting of workers should performed only once. But nevertheless I see no error :(
Ok, and what it RabbitMQ service log? (AFAIK log should be at default location)

[JustasStankunas]

here are RabbitMQ logs. (other are empty):
startup_err.log
[email protected]
[email protected]
shutdown_log.log

interestingly there looks to be some old (from March) logs:

root@9606a2735689:/var/log/rabbitmq# ls -l
total 16
-rw-r--r-- 1 rabbitmq rabbitmq 2294 Mar 26 13:53 [email protected]
-rw-r--r-- 1 rabbitmq rabbitmq    0 Mar 26 13:53 [email protected]
-rw-r--r-- 1 rabbitmq rabbitmq 3123 Apr  4 10:59 [email protected]
-rw-r--r-- 1 rabbitmq rabbitmq    0 Apr  4 10:59 [email protected]
-rw-r--r-- 1 root     root        0 Mar 26 13:53 shutdown_err
-rw-r--r-- 1 root     root       50 Mar 26 13:53 shutdown_log
-rw-r--r-- 1 rabbitmq rabbitmq   74 Apr  4 10:59 startup_err
-rw-r--r-- 1 rabbitmq rabbitmq    0 Apr  4 10:59 startup_log

[ShockwaveNN]

@JustasStankunas Could you try this solution inside container and see if Rabbit restart works?
https://stackoverflow.com/a/9013572/820501
chown -R rabbitmq:rabbitmq /var/lib/rabbitmq/

[JustasStankunas]

@ShockwaveNN just tried that, restarted container and nothing changed.
i checked before doing this (since there are not a lot directories and files in this location) and everything were owned by rabbitmq:rabbitmq anyway..

just a curious question, can you replicate this issue on your end, or it is just on my end?

[ShockwaveNN]

@JustasStankunas I tried simplified version from #92 (comment) and everything seems fine for me too.
So we cannot replicate your issue on our end, sorry.
I've never seen any troubles with rabbit inside docker before.

[JustasStankunas]

just tried creating new compose file but still same issue.. :( only thing i could think of now is different versions of docker compose and docker itself?

docker-compose version 1.20.1, build 5d8c71b
Docker version 18.03.0-ce, build 0520e24

[ShockwaveNN]

@JustasStankunas
Same here on Kubuntu 17.10
To be clear - simplified version from #92 (comment) also give you Rabbit error?
If so I'm alsmost sure it is permission problem, try chmod -R 777 in all mounts (I know it's a bad practice and overkill but I have no better ideas :()

[JustasStankunas]

@ShockwaveNN do you mean in docker itself chmod everything to 777 or on my host?

[ShockwaveNN]

@JustasStankunas on host, but be carefull and have backups of all files just in case :)

[thomass4t]

Hello,
I had the same issue. I created the folder /var/run/rabbitmq manually within the container and made a chown rabbitmy:rabbitmy rabbitmy
Afterwards I stopped and restarted the container successfully.

Short addendum:
The issue seems to happen from time to time. When I start a container via plain docker, sometimes this error message shows up, sometimes not. Up to now I couldn't figure out the root cause.

[JustasStankunas]

@thomass4t this looks interesting. i checked what worked for you and in my case it looks little different. when ever i start fresh onlyoffice container and if i am quick enough i can see /var/run/rabbitmq and it is owned by rabbitmq. But when it restarts it self, this folder is gone, and then rabbitmq is not starting anymore. if i create it manually and chown it to rabbitmq:rabbitmq and restart container, same thing happens again.

[thomass4t]

Hello Justas,
yes... i can confirm this observation. When I restart the container the folder is gone and problem pops up again.

[ShockwaveNN]

@JustasStankunas yeap, thanks to #99 we run documentserver without interactive mode, but currently it's not properly tested, so there may be troubles.
But it current stable version stdin_open: true is present in our example docker-compose.yml, so you should use it too.

[mhoffmann75]

I can confirm that stdin_open: true fixes the problem. Otherwise rabbitmq fails to create its directory /var/run/rabbitmq in non-interactive mode (docker-compose up -d) . Strange enough in interactive mode it works ...
So from my understanding stdin_open: true is enough to make it work interactive and non-interactive and you might close this issue.

[poVoq]

Thus to confirm, selecting interactive mode and TTY (-i -t) in Portainer when creating the container make it work for me as well. Thanks a lot for the help!

Edit: the strange thing is though that when I "restart" the container through Portainer it again exhibits the same error... but if I "recreate" it it works fine.

[danielepercivaldi]

I tested versions from 5.0.7.38 to 5.1.3.35.
The issue was injected in version 5.1.0.115.
In version 5.0.7.38 it works like a charm.
Starting fron version 5.1.0.115 you need restarting it twice in order to make RabbitMQ to start normally.

1. The first time it cannot create its folder in /var/run due to denied permissions.
2. The second time it works.

So, I think, something went wrong when version 5.1 was released and that this issue should be reopened.

[ShockwaveNN]

@danielepercivaldi Hi, could you show exact parameters of your docker run command? I try to inspect this issue

[danielepercivaldi]

Hi @ShockwaveNN .
Sorry for the delay.
Below what you requested:
docker run --restart=always -i -t -d -p 8081:80 --name onlyoffice --env-file ./onlyoffice.env onlyoffice/documentserver

onlyoffice.env:

JWT_ENABLED=true
JWT_SECRET=*************
JWT_HEADER=AuthorizationJwt

Thanks.

[ShockwaveNN]

@danielepercivaldi I see nothing wrong in running documentserver by your command.
I think trouble persist only if there some docker volumes mounts, but your command has none

[danielepercivaldi]

@ShockwaveNN Are you able to reproduce the issue?

[ShockwaveNN]

@danielepercivaldi No, I'm not able to reproduce issue.
RabbitMQ started normally via your command from first time on 5.1.0.115
Which docker and Host OS you are using? Maybe it somehow related.
Mine is
Ubuntu 18.04 LTS
Docker version 18.03.1-ce, build 9ee9f40

[danielepercivaldi]

@ShockwaveNN Mine is
Ubuntu 16.04 LTS
Docker version 18.03.1-ce, build 9ee9f40

[ShockwaveNN]

@danielepercivaldi so to make sure
you run
docker run --restart=always -i -t -d -p 8081:80 --name onlyoffice --env-file ./onlyoffice.env onlyoffice/documentserver:5.1.0.115
and have errors in starting RabbitMQ?

[danielepercivaldi]

@ShockwaveNN No.
Problem occurs when you "docker restart onlyoffice".
You need restarting it twice in order to make RabbitMQ start right.

[ShockwaveNN]

@danielepercivaldi Oh, thanks. I can confirm, it's some bug.

Works fine:

docker run -itd --name onlyoffice onlyoffice/documentserver:5.0.7.38
sleep 1m # for init of services
docker restart onlyoffice
sleep 1m # for init of services
docker exec onlyoffice cat /var/log/rabbitmq/startup_err

out mkdir: cannot create directory ‘/var/run/rabbitmq’: Permission denied :

docker run -itd --name onlyoffice onlyoffice/documentserver:5.1.0.115
sleep 1m # for init of services
docker restart onlyoffice
sleep 1m # for init of services
docker exec onlyoffice cat /var/log/rabbitmq/startup_err

[ShockwaveNN]

Just a note - problem seems actual for host with Ubuntu 16.04 and 18.04
Problem didn't reproduces on host with Ubuntu 14.04 (travis.ci latest image and DigitalOcean image) and Centos 7. Strange.

[DrRSatzteil]

I also had the same problem on Ubuntu 17.10. And it is still there after the 18.04 update.

[Nine-Tailed]

Similar error in virtualbox with ubuntu 16.04.
docker run -dit -p 88:80 --restart always --name oods onlyoffice/documentserver

[Daniel-Beardsmore]

What would you change in hostconfig.json and config.v2.json to apply this setting?

[ShockwaveNN]

@Daniel-Beardsmore sorry, can you explain in more details?

[Daniel-Beardsmore]

"stdin_open: true" only works if you have docker-compose.yml. What do you add instead into hostconfig.json and config.v2.json to apply the same change afterwards to a container that demonstrates this fault?

[ShockwaveNN]

@Daniel-Beardsmore next stable version of Docker-Documentserver (5.2) will not require stdin_open: true
Since https://github.com/ONLYOFFICE/Docker-DocumentServer/pull/99/files
Son I don't think we should discuss it if I understand you correctly

[danielepercivaldi]

@agolybev @ShockwaveNN Any news? Need waiting for V.5.2?
Thanks.

[ShockwaveNN]

@danielepercivaldi problem with mkdir: cannot create directory ‘/var/run/rabbitmq’: Permission denied is still actual in 5.2. We post any updates here.

[svalo]

I was hit by this as well, running a docker exec <name-of-container> service rabbitmq-server restart makes rabbit start and onlyoffice work. It seems rabbit doesn't like the way docker handles the container restart.
I do not have any problem issuing docker stop <name-of-container> followed by docker start <name-of-container>.

If it's not possible to handle the way docker restarts a container would it be possible to insert a timeout in the 'wait_for_connection' function so it returns an error and the container exits?

In this way the container wouldn't wait indefinitely for rabbitmq and a 'restart unless-stopped' policy would restart the second time the container bringing it up.

If this solution isn't the way to go maybe issuing a restart of the rabbitmq-server service inside the container after a certain timeout in the 'wait_for_rabbitmq' function would be another option

thanks

[unclegaara]

I have the same issue with docker-compose version 1.22.0 on ubuntu 18.04

[pgera]

Seeing this too with systemd. It doesn't come up on boot due to the error described in this thread. However, restarting the service makes it work on the second attempt

docker-compose file:

version: '3'

services:
  onlyoffice-document-server:
    container_name: onlyoffice-document-server
    image: onlyoffice/documentserver
    restart: always
    stdin_open: true
    tty: true
    ports:
      - 9980:80
    env_file:
      - onlyoffice.env

systemd file:

[Unit]
Description=%i service with docker compose
After=docker.service network-online.target
Requires=docker.service network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes

WorkingDirectory=/etc/docker/compose/%i

ExecStartPre=/usr/bin/docker-compose build --pull
ExecStart=/usr/bin/docker-compose up -d

ExecStop=/usr/bin/docker-compose down

ExecReload=/usr/bin/docker-compose pull --quiet
ExecReload=/usr/bin/docker-compose up -d

[Install]
WantedBy=multi-user.target

[ShockwaveNN]

@snorre-k Thank you for your investigation and fix, we added those lines at #131
And checked by

docker run -itd --name onlyoffice onlyoffice/4testing-documentserver-ie:5.2.3.49
sleep 1m # for init of services
docker restart onlyoffice
sleep 1m # for init of services
docker exec onlyoffice cat /var/log/rabbitmq/startup_err

Everything seems good, Rabbitmq starts correctly.
This will be released as 5.2.3 soon.

[ShockwaveNN]

@exodus3318 please create separate issue for your problem and describe your problem in more details

[ShockwaveNN]

DocumentServer 5.2.3.64 is released by now.
Issue should be fixed.