Program killed after a few hours runnning. about prysm

[Fantasim]

I've managed to decrease the memory consumption by passing these parameters:

--disable-broadcast-slashings --max-goroutines 3000 --backfill-worker-count 16 blob-batch-limit 32 block-batch-limit 32 --p2p-max-peers 35

So far, after 2 days it didn't get killed anymore.

[Fantasim]

Here is more logs about the killing reason:

Apr  3 11:47:51 louis kernel: [1441781.832771] geth invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr  3 11:47:51 louis kernel: [1441781.832776] CPU: 6 PID: 300109 Comm: geth Not tainted 5.15.0-100-generic #110-Ubuntu
Apr  3 11:47:51 louis kernel: [1441781.832778] Hardware name: Dell Inc. XPS 15 9500/05XYW7, BIOS 1.6.1 12/24/2020
Apr  3 11:47:51 louis kernel: [1441781.832780] Call Trace:
Apr  3 11:47:51 louis kernel: [1441781.832781]  <TASK>
Apr  3 11:47:51 louis kernel: [1441781.832783]  show_stack+0x52/0x5c
Apr  3 11:47:51 louis kernel: [1441781.832788]  dump_stack_lvl+0x4a/0x63
Apr  3 11:47:51 louis kernel: [1441781.832791]  dump_stack+0x10/0x16
Apr  3 11:47:51 louis kernel: [1441781.832794]  dump_header+0x53/0x228
Apr  3 11:47:51 louis kernel: [1441781.832796]  oom_kill_process.cold+0xb/0x10
Apr  3 11:47:51 louis kernel: [1441781.832798]  out_of_memory+0x106/0x2e0
Apr  3 11:47:51 louis kernel: [1441781.832801]  __alloc_pages_slowpath.constprop.0+0x9b7/0xa80
Apr  3 11:47:51 louis kernel: [1441781.832805]  __alloc_pages+0x311/0x330
Apr  3 11:47:51 louis kernel: [1441781.832808]  alloc_pages+0x9e/0x1e0
Apr  3 11:47:51 louis kernel: [1441781.832810]  __page_cache_alloc+0x7e/0x90
Apr  3 11:47:51 louis kernel: [1441781.832813]  pagecache_get_page+0x152/0x590
Apr  3 11:47:51 louis kernel: [1441781.832815]  ? page_cache_ra_unbounded+0x166/0x210
Apr  3 11:47:51 louis kernel: [1441781.832818]  filemap_fault+0x488/0xab0
Apr  3 11:47:51 louis kernel: [1441781.832821]  ? filemap_map_pages+0x309/0x400
Apr  3 11:47:51 louis kernel: [1441781.832823]  __do_fault+0x39/0x120
Apr  3 11:47:51 louis kernel: [1441781.832825]  do_read_fault+0xeb/0x160
Apr  3 11:47:51 louis kernel: [1441781.832827]  do_fault+0xa0/0x2e0
Apr  3 11:47:51 louis kernel: [1441781.832828]  handle_pte_fault+0x1cd/0x240
Apr  3 11:47:51 louis kernel: [1441781.832829]  __handle_mm_fault+0x405/0x6f0
Apr  3 11:47:51 louis kernel: [1441781.832832]  handle_mm_fault+0xd8/0x2c0
Apr  3 11:47:51 louis kernel: [1441781.832833]  do_user_addr_fault+0x1c9/0x670
Apr  3 11:47:51 louis kernel: [1441781.832836]  exc_page_fault+0x77/0x170
Apr  3 11:47:51 louis kernel: [1441781.832840]  asm_exc_page_fault+0x27/0x30
Apr  3 11:47:51 louis kernel: [1441781.832842] RIP: 0033:0x41d01b
Apr  3 11:47:51 louis kernel: [1441781.832847] Code: Unable to access opcode bytes at RIP 0x41cff1.
Apr  3 11:47:51 louis kernel: [1441781.832847] RSP: 002b:00007fa6f0ff8c68 EFLAGS: 00010206
Apr  3 11:47:51 louis kernel: [1441781.832850] RAX: 00007fa6a5719818 RBX: 000000c0704fe388 RCX: 000000c0704fe380
Apr  3 11:47:51 louis kernel: [1441781.832851] RDX: 00000000019089e0 RSI: 0000000001e9e3a8 RDI: 000000000000003e
Apr  3 11:47:51 louis kernel: [1441781.832852] RBP: 00007fa6f0ff8c78 R08: 000000c000102a80 R09: 000000c00005e508
Apr  3 11:47:51 louis kernel: [1441781.832854] R10: 000000c021217000 R11: 0000000000000077 R12: 00007fa6f0ff8d08
Apr  3 11:47:51 louis kernel: [1441781.832855] R13: 000000c0000b7530 R14: 000000c000702540 R15: 0000000000000001
Apr  3 11:47:51 louis kernel: [1441781.832857]  </TASK>
Apr  3 11:47:51 louis kernel: [1441781.832858] Mem-Info:
Apr  3 11:47:51 louis kernel: [1441781.832859] active_anon:2522342 inactive_anon:1104385 isolated_anon:62
Apr  3 11:47:51 louis kernel: [1441781.832859]  active_file:195 inactive_file:316 isolated_file:0
Apr  3 11:47:51 louis kernel: [1441781.832859]  unevictable:6963 dirty:1 writeback:0
Apr  3 11:47:51 louis kernel: [1441781.832859]  slab_reclaimable:220845 slab_unreclaimable:50325
Apr  3 11:47:51 louis kernel: [1441781.832859]  mapped:2414 shmem:55 pagetables:37795 bounce:0
Apr  3 11:47:51 louis kernel: [1441781.832859]  kernel_misc_reclaimable:0
Apr  3 11:47:51 louis kernel: [1441781.832859]  free:55425 free_pcp:2213 free_cma:0
Apr  3 11:47:51 louis kernel: [1441781.832864] Node 0 active_anon:10089368kB inactive_anon:4417540kB active_file:780kB inactive_file:1264kB unevictable:27852kB isolated(anon):248kB isolated(file):0kB mapped:9656kB dirty:4kB writeback:0kB shmem:220kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4096kB writeback_tmp:0kB kernel_stack:6032kB pagetables:151180kB all_unreclaimable? no
Apr  3 11:47:51 louis kernel: [1441781.832868] Node 0 DMA free:13312kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr  3 11:47:51 louis kernel: [1441781.832872] lowmem_reserve[]: 0 1387 15601 15601 15601
Apr  3 11:47:51 louis kernel: [1441781.832875] Node 0 DMA32 free:62632kB min:6004kB low:7504kB high:9004kB reserved_highatomic:0KB active_anon:670308kB inactive_anon:612684kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:1593488kB managed:1527716kB mlocked:0kB bounce:0kB free_pcp:7612kB local_pcp:24kB free_cma:0kB
Apr  3 11:47:51 louis kernel: [1441781.832880] lowmem_reserve[]: 0 0 14214 14214 14214
Apr  3 11:47:51 louis kernel: [1441781.832883] Node 0 Normal free:145756kB min:199908kB low:215284kB high:230660kB reserved_highatomic:30720KB active_anon:9419168kB inactive_anon:3805424kB active_file:1680kB inactive_file:72kB unevictable:27852kB writepending:4kB present:14901248kB managed:14564004kB mlocked:27728kB bounce:0kB free_pcp:1240kB local_pcp:0kB free_cma:0kB
Apr  3 11:47:51 louis kernel: [1441781.832888] lowmem_reserve[]: 0 0 0 0 0
Apr  3 11:47:51 louis kernel: [1441781.832891] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
Apr  3 11:47:51 louis kernel: [1441781.832901] Node 0 DMA32: 1089*4kB (UME) 832*8kB (UME) 1016*16kB (UME) 381*32kB (UME) 102*64kB (UE) 34*128kB (UME) 5*256kB (U) 4*512kB (UM) 1*1024kB (U) 0*2048kB 2*4096kB (UM) = 62884kB
Apr  3 11:47:51 louis kernel: [1441781.832914] Node 0 Normal: 13876*4kB (UME) 2422*8kB (UME) 3652*16kB (UME) 418*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 146688kB
Apr  3 11:47:51 louis kernel: [1441781.832926] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr  3 11:47:51 louis kernel: [1441781.832927] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr  3 11:47:51 louis kernel: [1441781.832929] 19071 total pagecache pages
Apr  3 11:47:51 louis kernel: [1441781.832930] 16185 pages in swap cache
Apr  3 11:47:51 louis kernel: [1441781.832930] Swap cache stats: add 100106425, delete 100091384, find 25297157/53872443
Apr  3 11:47:51 louis kernel: [1441781.832932] Free swap  = 0kB
Apr  3 11:47:51 louis kernel: [1441781.832932] Total swap = 4194300kB
Apr  3 11:47:51 louis kernel: [1441781.832933] 4127682 pages RAM
Apr  3 11:47:51 louis kernel: [1441781.832934] 0 pages HighMem/MovableOnly
Apr  3 11:47:51 louis kernel: [1441781.832934] 100912 pages reserved
Apr  3 11:47:51 louis kernel: [1441781.832935] 0 pages hwpoisoned
Apr  3 11:47:51 louis kernel: [1441781.832935] Tasks state (memory values in pages):
Apr  3 11:47:51 louis kernel: [1441781.832936] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr  3 11:47:51 louis kernel: [1441781.832944] [    552]     0   552    72356     6802   118784        0         -1000 multipathd
Apr  3 11:47:51 louis kernel: [1441781.832948] [    880]   103   880     2254      718    61440       97          -900 dbus-daemon
Apr  3 11:47:51 louis kernel: [1441781.832951] [    885]     0   885    20711      460    57344       65             0 irqbalance
Apr  3 11:47:51 louis kernel: [1441781.832953] [    886]     0   886     8183      798    98304     1945             0 networkd-dispat
Apr  3 11:47:51 louis kernel: [1441781.832955] [    895]     0   895   514774     2271   344064      996          -900 snapd
Apr  3 11:47:51 louis kernel: [1441781.832957] [    896]     0   896    11998      450   102400      228             0 systemd-logind
Apr  3 11:47:51 louis kernel: [1441781.832959] [    904]     0   904     4126      477    77824      213             0 wpa_supplicant
Apr  3 11:47:51 louis kernel: [1441781.832961] [    960]     0   960     1724      545    57344       37             0 cron
Apr  3 11:47:51 louis kernel: [1441781.832963] [   1006]     0  1006    27438      794   118784     2064             0 unattended-upgr
Apr  3 11:47:51 louis kernel: [1441781.832965] [   1016]     0  1016     1544      187    49152       32             0 agetty
Apr  3 11:47:51 louis kernel: [1441781.832967] [   1999]     0  1999     4440      530    69632      502             0 wpa_supplicant
Apr  3 11:47:51 louis kernel: [1441781.832969] [   2123]  1000  2123     2032      516    49152       67             0 screen
Apr  3 11:47:51 louis kernel: [1441781.832971] [   2124]  1000  2124     3483      629    65536      596             0 zsh
Apr  3 11:47:51 louis kernel: [1441781.832973] [   2204]  1000  2204     1998      421    49152       55             0 screen
Apr  3 11:47:51 louis kernel: [1441781.832974] [   2205]  1000  2205     3530      607    65536      649             0 zsh
Apr  3 11:47:51 louis kernel: [1441781.832976] [  41136]  1000 41136     2026      270    57344      304             0 ssh-agent
Apr  3 11:47:51 louis kernel: [1441781.832978] [ 263182]     0 263182    73897      739   159744      719             0 packagekitd
Apr  3 11:47:51 louis kernel: [1441781.832980] [ 263188]   107 263188    55601      307    86016      244             0 rsyslogd
Apr  3 11:47:51 louis kernel: [1441781.832982] [ 263197]     0 263197    24216      425   212992      190          -250 systemd-journal
Apr  3 11:47:51 louis kernel: [1441781.832984] [ 263204]   104 263204    22341      201    73728      198             0 systemd-timesyn
Apr  3 11:47:51 louis kernel: [1441781.832987] [ 263210]     0 263210    59945      338   102400      237             0 upowerd
Apr  3 11:47:51 louis kernel: [1441781.832988] [ 263217]     0 263217    98145      675   131072      461             0 udisksd
Apr  3 11:47:51 louis kernel: [1441781.832990] [ 263229]     0 263229    59097      785    94208      404             0 polkitd
Apr  3 11:47:51 louis kernel: [1441781.832992] [ 263234]     0 263234    79490      723   118784      491             0 ModemManager
Apr  3 11:47:51 louis kernel: [1441781.832994] [ 263235]     0 263235    59412      568   102400      181             0 boltd
Apr  3 11:47:51 louis kernel: [1441781.832996] [ 263240]     0 263240   113699      434   323584    16516             0 fwupd
Apr  3 11:47:51 louis kernel: [1441781.832998] [ 263252]     0 263252     6408      659    77824      342         -1000 systemd-udevd
Apr  3 11:47:51 louis kernel: [1441781.833000] [ 263254]   101 263254     4065      476    65536      191             0 systemd-network
Apr  3 11:47:51 louis kernel: [1441781.833003] [ 263260]   102 263260     6418      681    90112     1026             0 systemd-resolve
Apr  3 11:47:51 louis kernel: [1441781.833005] [ 263275]     0 263275    68408      524   110592      209             0 thermald
Apr  3 11:47:51 louis kernel: [1441781.833007] [ 295947]     0 295947     3859      708    73728      359         -1000 sshd
Apr  3 11:47:51 louis kernel: [1441781.833009] [ 298273]  1000 298273 76171458  2062447 132521984   313309             0 beacon-chain-v5
Apr  3 11:47:51 louis kernel: [1441781.833012] [ 300094]  1000 300094  3381260  1540881 19103744   705035             0 geth
Apr  3 11:47:51 louis kernel: [1441781.833014] [ 308083]     0 308083     3937     1075    73728       47             0 sshd
Apr  3 11:47:51 louis kernel: [1441781.833016] [ 308085]  1000 308085     4038      657    77824       57             0 sshd
Apr  3 11:47:51 louis kernel: [1441781.833018] [ 308086]  1000 308086     3456     1139    61440       45             0 zsh
Apr  3 11:47:51 louis kernel: [1441781.833021] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=ssh.service,mems_allowed=0,global_oom,task_memcg=/system.slice/ssh.service,task=beacon-chain-v5,pid=298273,uid=1000
Apr  3 11:47:51 louis kernel: [1441781.833066] Out of memory: Killed process 298273 (beacon-chain-v5) total-vm:304685832kB, anon-rss:8249788kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:129416kB oom_score_adj:0
Apr  3 11:47:51 louis systemd[1]: ssh.service: A process of this unit has been killed by the OOM killer.
Apr  3 11:47:51 louis systemd[1]: ssh.service: Failed with result 'oom-kill'.

[james-prysm]

it said

geth invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

so you ran out of memory after a few hours, have you tried killing other processes or increasing memory for your node.

[Fantasim]

it said

geth invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

so you ran out of memory after a few hours, have you tried killing other processes or increasing memory for your node.

I have no other processes except geth running. It's everywhere said that 16gb of memory should be enough. I am not able to increase the memory of my node.

Is there a way to decrease the memory consumption of Prysm so it never go over the limit?

[prestonvanloon]

@Fantasim could you please post all of the flags you are using when the problem was occurring? Thanks for posting your workaround. We may be able to identify a memory leak if we could understand how you were running the process.

[Fantasim]

Update: the program did get killed anyway with these flags:

--disable-broadcast-slashings
--max-goroutines 3000
--backfill-worker-count 16

[prestonvanloon]

@Fantasim Could you post all of your flags? We cannot reproduce if we do not understand how you are running the program.

[prestonvanloon]

16Gb of RAM is going to be tough too. We can look for memory issues with Prysm but really need to know how you are running it.