EMFILE: too many open files about bree

[r00b]

Specifically I believe the issue was that as Node worker threads exited they did not necessarily flush their file streams, thus when the number of open file streams exceeded ulimit, the parent process crashed. I think.

[shadowgate15]

@alvin30595 I would suggest using the workerMessageHandler and do the logging work on the main process. Worker threads and logging is pretty difficult.

[ErisDS]

@r00b thank you so much!

[r00b]

I've seen solutions ranging from installing watchman to just setting the ulimit -n to a higher int (although mine is already 10240) but I'm unsure if these are correct since they are machine-based solutions, and when my application runs in a different machine I don't want it to have the same problem.

[niftylettuce]

Not sure what the redis-service contents are? But yes, it's common practice to even increase this on servers.

We do this on Forward Email out of the box with our ansible config:

https://github.com/forwardemail/forwardemail.net/blob/2108d0b2db10295748996431263fbde16584a615/ansible/playbooks/security.yml#L169-L175

[r00b]

Turns out this was an issue regarding how winstonjs/winston-daily-rotate-file works within worker threads. Spoiler alert, it doesn't work very well. Thanks for putting up with all of my non-issue issues.

[ErisDS]

@robertsteilberg Think we may have hit the same issue but with bunyan - how did you resolve it? Was it purely not using logging in the worker thread or something else?

[ErisDS]

@r00b would be really helpful if you could share some more details please

[r00b]

Hi @ErisDS, apologies for late reply, I am on vacation this month. Unfortunately I found no way to resolve this without moving away from winstonjs/winston-daily-rotate-file and avoiding a logging service that involves writing files to disk--this seems to be an issue with Node worker threads, not Bree, I think. It has been awhile since I was deep in this though. In the end I found that maintaining logs by writing to files was not the best option for me anyway, seems that using services to capture logs and keep them in a platform somewhere was a better option.

[alvinlys]

if that the case, what logger you used to replace winston? Mind to share your alternative? Thanks

Hi @ErisDS, apologies for late reply, I am on vacation this month. Unfortunately I found no way to resolve this without moving away from winstonjs/winston-daily-rotate-file and avoiding a logging service that involves writing files to disk--this seems to be an issue with Node worker threads, not Bree, I think. It has been awhile since I was deep in this though. In the end I found that maintaining logs by writing to files was not the best option for me anyway, seems that using services to capture logs and keep them in a platform somewhere was a better option.