"No space left on device" even though external drive is connected about umbrel-os

[johnpc]

I determined the cause. It's a 32GB SD card and 7.7GB were taken up by each of /sd-root/var/log/syslog, /sd-root/var/log/messages and /sd-root/var/log/user.log for a total of 23.1GB in log files. The rest of the SD storage contains various caches and the OS itself. Is it safe to remove those three log files? Why did they get so bloated?

[banneord-puzzle]

Following advice in: https://askubuntu.com/questions/746535/var-log-syslog-growing-indefinitely-in-size it might be a good idea to hunt down the offending service/daemon that is constantly filling up the logs with errors.

Start by looking at the log entries themselves.

Does any error repeat?
Is it related to the Umbrel itself?

[lukechilds]

@johnpc thanks for reporting this. Did you find anything in the logfiles that pointed to what the problem was?

Sounds like you are but just to clarify, you're running Umbrel OS on a Raspberry Pi, right?

Also when did this problem start? Was it immediately after doing an update or seemingly randomly? If after an update do you remember which version you were update from/to?

Thanks!

[IanMichaelHarper]

I have just experienced this exact same issue. My syslog is 10G and has the following message written 1382210 times:

Dec  2 00:41:31 umbrel dockerd[856]: time="2023-12-02T00:41:31.842107093Z" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/1170aa1573e586fc60e782aaed295e4c66af8372910a40ea368429541397ab72/1170aa1573e586fc60e782aaed295e4c66af8372910a40ea368429541397ab72-json.log: no space left on device"

My daemon.log.1 is 13G and has the same message written over and over.

I was not using my umbrel around the 2nd December when this log is dated, but it was not an issue until a few days ago when I began installing more apps and transferring other data files to my umbrel. Then suddenly my Umbrel stopped working and I began getting this error on the command line. Before that it seemed to work fine via the web interface.

I did not update the Umbrel OS at all in this time but I did update a few apps.

Removing these log files fixed the issue.