Can not view retained messages in admin dashboard after updating to 5.6.0 from 5.5.1 about emqx

[zmstone]

eacces hints filesystem permission issue.
could you check if the dir /opt/emqx/fs_emqx (mounted to docker?) can be read by the emqx user?

[jandillmann]

Yes that was my first thought, too – but the directories were created by the Docker image of emqx 5.5.1 and are accessible by UID/GID 1000 (emqx inside the container).

Interestingly, after switching from Retained messages on disk to RAM and back to disk the error message changed:

500 INTERNAL_ERROR: exit, {noproc,{gen_server,call,[emqx_retainer,{page_read,undefined,1,20},infinity]}}, [{gen_server,call,3,[{file,"gen_server.erl"},{line,419}]},{emqx_retainer_api,lookup_retained,2,[{file,"emqx_retainer_api.erl"},{line,188}]},{minirest_handler,apply_callback,3,[{file,"minirest_handler.erl"},{line,152}]},{minirest_handler,handle,2,[{file,"minirest_handler.erl"},{line,56}]},{minirest_handler,init,2,[{file,"minirest_handler.erl"},{line,27}]},{cowboy_handler,execute,2,[{file,"cowboy_handler.erl"},{line,41}]},{cowboy_stream_h,execute,3,[{file,"cowboy_stream_h.erl"},{line,318}]},{cowboy_stream_h,request_process,3,[{file,"cowboy_stream_h.erl"},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]

[keynslug]

Hi @jandillmann,

after switching from Retained messages on disk to RAM and back to disk the error message changed

I'm also curious why this happens.

1. Could you please describe what steps did you perform to make a switch?
2. Were there any warnings / errors in the EMQX service logs coinciding with this switch?

{file_error,"/opt/emqx/fs_emqx@mydomain_1_10195.1",eacces}

I suspect that this is an unintended consequence of 58d0f04. If so, forcing the container's workdir to /opt/emqx/data instead supposedly might help.

[jandillmann]

Hi @keynslug,

In the dashboard, I went to "MQTT Settings" -> "Retainer", switched "Storage Method" from disc to ram, saved, and back. I did not see any error messages in the log when switching between ram and disc.

In the compose.yaml, the volumes are configured as this:

    volumes:
      - "./emqx/data:/opt/emqx/data"
      - "./emqx/log:/opt/emqx/log"

Could this be the issue, even if both directories and their sub-directories and files are owned by UID/GID 1000 and rwx?

[keynslug]

Could this be the issue, even if both directories and their sub-directories and files are owned by UID/GID 1000 and rwx?

No this should be fine I guess. It's the fact that /opt/emqx is not writable by UID 1000 in the container file system anymore that (supposedly) causes these weird errors. That's why I think you might try setting working_dir: /opt/emqx/data in compose.yaml, so that those temporary files EMQX want to write from time to time will reside there instead.

[jandillmann]

Sadly this did not help. I set working_dir: /opt/emqx/data in compose.yaml and restarted the stack with docker compose down && docker compose up -d, but the same error as in the original issue still shows.

[keynslug]

Sadly this did not help.

Oh, that was unexpected. Thanks anyway for trying out and for the original report. Hopefully #12875 fixes this properly, which should land in the next release.