Comments (3)
Okay, so far, I see the following:
docker logs -f nostream-cache
1:C 29 Dec 2023 10:20:16.148 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 29 Dec 2023 10:20:16.148 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 29 Dec 2023 10:20:16.148 # Configuration loaded
1:M 29 Dec 2023 10:20:16.156 # Server initialized
1:C 29 Dec 2023 10:48:02.237 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 29 Dec 2023 10:48:02.237 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 29 Dec 2023 10:48:02.237 # Configuration loaded
1:M 29 Dec 2023 10:48:02.244 # Server initialized
1:C 29 Dec 2023 12:15:14.797 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 29 Dec 2023 12:15:14.797 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 29 Dec 2023 12:15:14.797 # Configuration loaded
1:M 29 Dec 2023 12:15:14.800 # Server initialized
Redis seems to restart now and then because of a out-of-memory error:
Dec 29 10:32:36 at kernel: [62794.438636] out_of_memory+0x106/0x2e0
Dec 29 10:32:36 at kernel: [62794.438930] Tasks state (memory values in pages):
Dec 29 10:32:36 at kernel: [62794.439403] Out of memory: Killed process 1142634 (redis-server) total-vm:27478268kB, anon-rss:11599084kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:47208kB oom_score_adj:0
Dec 29 10:32:40 at start[1117079]: nostream | uncaught error: [ErrorReply: LOADING Redis is loading the dataset in memory]
Dec 29 10:32:41 at start[1117079]: nostream | web-socket-adapter: unable to handle message: [ErrorReply: LOADING Redis is loading the dataset in memory]
Dec 29 10:32:42 at start[1117079]: nostream | web-socket-adauncaught error: [ErrorReply: LOADING Redis is loading the dataset in memory]
Dec 29 10:32:42 at start[1117079]: nostream | at TCP.uncaught error: [ErrorReply: LOADING Redis is loading the dataset in memory]
Dec 29 10:32:42 at start[1117079]: nostream | at TCP.onStreamRead (node:internal/stream_base_commons:web-socket-adapter: unable to handle message: [ErrorReply: LOADING Redis is loading the dataset in memory]
Dec 29 10:32:42 at start[1117079]: nostream | web-socket-adapter: unable to handle message: [ErrorReply: LOADING Redis is loading the dataset in memory]
Memory is fine, most of the time, this is a typical value:
total used free shared buff/cache available
Mem: 15989 1283 13958 154 747 14264
Swap: 16383 797 15586
But sometimes it looks like this:
total used free shared buff/cache available
Mem: 15989 15499 159 174 330 68
Swap: 16383 2054 14329
This is the output of top, sorted by memory shortly after the above output:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1263844 lxd 20 0 18.7g 12.6g 1064 R 95.7 80.5 5:08.09 redis-server
1252985 xxx 20 0 1466820 1.1g 3924 S 13.0 6.8 13:21.33 node
1279048 lxd 20 0 4416636 138436 126904 S 0.0 0.8 0:01.27 postgres
So it seems redis is sometimes allocating so much memory that the process is killed. How can I limit the memory use?
Redis has a max-memory setting, but how do I set it in the nostream docker environment. What this actually help?
from nostream.
@pgerstbach We had a conversation off Github where we addressed the OOM issue with Redis as well as a connection issue due to NGINX. Do you mind updating here what was the solution?
from nostream.
@pgerstbach We had a conversation off Github where we addressed the OOM issue with Redis as well as a connection issue due to NGINX. Do you mind updating here what was the solution?
Certainly, I just waited long enough to see the issue gone!
So, here is a documentation how my issue was resolved. This would not be possible without @cameri, so thanks a lot for your help!
Solution Part One: Limit memory use
In my syslog I found multiple out of memory errors like this:
out_of_memory+0x106/0x2e0
Tasks state (memory values in pages):
Out of memory: Killed process 640637 (redis-server) total-vm:8235260kB, anon-rss:7209804kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:14252kB oom_score_adj:0
So redis reserved more memory than available, even an additional 16GB swap file was not enough in some cases.
The solution of this memory problem is to reduce "rate limiting" in settings.yaml
. The redis cache is currently used only to implement those rate limits. There are limits on invoices, connections, events and messages possible (and enabled by default).
The longer the period used, the larger the memory consumption. I still had some ratelimits with periods of days in my config. After removing them the memory consumption was down to a normal level. If needed also ratelimits with periods of hour(s) should be removed.
After removing the ratelimits with longest period my out-of-memory erros where gone.
But still I had outages, so I found one more configuration issue:
Solution Part Two: Configuring nginx accordingly
I realized that my nginx instance was not configered properly. It was running out of file and connection limits.
In my nginx.conf I added the following lines:
worker_rlimit_nofile 8192;
events {
worker_connections 4096;
}
From the nginx docs: https://nginx.org/en/docs/ngx_core_module.html
- worker_rlimit_nofile: Changes the limit on the maximum number of open files (RLIMIT_NOFILE) for worker processes.
- worker_connections: Sets the maximum number of simultaneous connections that can be opened by a worker process.
Of course the values need to be aligned to your server resources and requirements: YMMV
So my issue was not really related to nostream, but maybe you have similar issues and hopefuly my solution will also help others! 🤙
from nostream.
Related Issues (20)
- PostgreSQL DB CPU maxing out HOT 1
- [BUG] docker fails on ubuntu HOT 5
- [BUG] sh: husky: not found
- [BUG] changing port doesn't work HOT 3
- [BUG] what public key do i put in settings? HOT 1
- [BUG] HOT 1
- Mkdir error
- [BUG] Endless checking of an unpaid invoice (processor LNBits) HOT 11
- Unable to easily verify if the npub has paid the invoice HOT 1
- Work to change database HOT 1
- Unexpected Token when starting relay HOT 2
- Feature request: NIP-45 Event Count
- [BUG] - Payments taking a long time to clear, or don't clear HOT 1
- [BUG] Throws error when receiving a replaceable event after its delete event HOT 3
- [REQUEST] A more streamlined approach to customising a Railway deployment HOT 1
- I can't run the relay nostream-db is unhealthy [BUG] HOT 4
- [REQUEST] REQ tag search separated from JSONB @> to event_tags table to speed up the process. HOT 2
- [REQUEST] Purging old events
- [BUG] Installation permission problem
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nostream.