Redis memory issue (Kali Linux 2021.2) about emba

[Humberto-Davila]

Confirmed , after hugepage disable , it running well.

[Humberto-Davila]

I know , I was procrastinating about doing the GUI to manage binwalk and some tools, virtualization in my experience must be done according to the binaries to emulate, 32/64 bit, also realtek has in the latest Kernel virtualization and emulation detection and the process starts crashing and dumping memory, so it creates a crash on host. I did this work manually and just can say that the report contains the data I found and of course dozens more. Thank you for your work on this. If you need a hand or are stuck on something just drop an email. Regards La informacion contenida en este correo electronico y cualquier documento adjunto es confidencial y puede tener derechos reservados o ser propiedad intelectual. Si usted no es el destinatario usted no esta autorizado a usar o discutir esta informacion, por lo tanto se le pide notificarnos via correo electronico o por telefono y borrar el mensaje de su sistema de correo. The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.

…

On Thu, 8 Jul 2021 at 02:38, Michael Messner ***@***.***> wrote: 2GB of memory is not a lot for emba. Probably we should add a wiki section with hardware requirements. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUILWT4PV6NFUY7USYWSYWTTWVIXLANCNFSM46FY5HCQ> .

[Humberto-Davila]

Thank you for your feedback. I will add a link to this thread in our wiki

Just another parameter needs to be changed in redis-server,

save <seconds> <changes>
#
#   Will save the DB if both the given number of seconds and the given
#   number of write operations against the DB occurred.
#
#   In the example below the behavior will be to save:
#   after 900 sec (15 min) if at least 1 key changed
#   after 300 sec (5 min) if at least 10 keys changed
#   after 60 sec if at least 10000 keys changed
#
#   Note: you can disable saving completely by commenting out all "save" lines.
#
#   It is also possible to remove all the previously configured save
#   points by adding a save directive with a single empty string argument
#   like in the following example:
#
#   save ""

save 900 1
save 300 10
save 60 10000

this is ok if you have > 8GB ram, but I suggest 100 seconds per GB / RAM 
eX:  4GB
save 400 100

with 2GB I put:
save 120 1

because just mongo and docker took 90 % of RAM

The Kali Distro just came with a 1GB swap partition, same amount of GB on RAM for swap space will help a lot .

EDIT: Added markdown tags for better readability.

[D4SH0x01]

cve-search has a problem getting the number of available cores in virtual machines, bypass needs to be set manually set to avoid crashing.

sudo env "WORKER_SIZE=1" ./sbin/db_updater.py -c
or
export WORKER_SIZE=1
or save WORKER_SIZE=1 in a file in /etc/enviroment.d/

this way the queue will be sent to all cores in sync.

Regards

[m-1-k-3]

In my first installation I had some issues with the cve-database. It needs a complete drop and repopulation:

└─$ sudo ./external/cve-search/sbin/db_updater.py -f

afterwards it was running as expected. Could someone verify this?

I have included this update to our installer: 2c26077

[Humberto-Davila]

It last just 20 minutes , after crashing multiple times.

redis-server has a maxmemory value of 0, also is not auto started.

'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled'
'To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf / reboot

supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.

[p4cx]

Thank you for your feedback and your solution. Can you please give us some information about your setup? I also had issues with Redis once and solved it by increasing the available storage for my Kali VM from 4GB to 8GB. But I don't know if it's the same problem, so I need additional information from you.

[Humberto-Davila]

Thank you for your feedback and your solution. Can you please give us some information about your setup? I also had issues with Redis once and solved it by increasing the available storage for my Kali VM from 4GB to 8GB. But I don't know if it's the same problem, so I need additional information from you.

The issue is that redis-server crashes all other process because of the huge memory pages, you cannot limit it because you will broke a dependency method, I tried 5, 8 , 12 GB ram and it is worst 2GB Ram is enough, just set transparent_hugepage to
'madvise' or better to 'never' , do not give permission to make a giant RAM DRIVE to redis.

last time I used transparent_hugepage was in red hat 5

Kali Rolling (2021.2) x64 2021-05-31 on WorkStation 16.x VM

Just disable permanently by creating a service.

$sudo nano /etc/systemd/system/disable-thp.service

[Unit]
Description=Disable Transparent Huge Pages (THP)

[Service]
Type=simple
ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl start disable-thp
$ sudo systemctl enable disable-thp

Both Mongo and Redis has the issue with hugepages. so be sure that DBs start after disabling the pages and its defrag.

Kind Regards.

[m-1-k-3]

As far as I can see the redis server is started during the installation process (see

emba/installer.sh

Line 530 in 849eba6

/etc/init.d/redis-server start

). Additionally the update script is also able to start the redis server (see

emba/config/cve_database_updater.init

Line 8 in 849eba6

/etc/init.d/redis-server start

). During normal operation redis server is not needed and does not need to be started:

[m-1-k-3]

2GB of memory is not a lot for emba. Probably we should add a wiki section with hardware requirements.

[Humberto-Davila]

pmd_set_huge: Cannot satisfy [mem 0xf0000000-0xf0200000] with a huge-page mapping due to MTRR override. Out of memory: Killed process 1032607 (cwmpClient) total-vm:2513456kB, anon-rss:871180kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:1828kB oom_score_adj:0 Out of memory: Killed process 640 (mongod) total-vm:2275988kB, ****/ redis-server invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE) /***** *oom*-*killer* is a process that the kernel will call when a system is over committed on memory. This will only be *invoked* if your system has exhausted both physical and swap memory. Xorg invoked oom-killer Out of memory: Killed process 1035790.... and so on....... after the service install is running smooth [image: image.png] Kind regards

…

On Thu, 8 Jul 2021 at 02:37, Michael Messner ***@***.***> wrote: As far as I can see the redis server is started during the installation process (see https://github.com/e-m-b-a/emba/blob/849eba667a52db7e769d978658b66aa4e75c203c/installer.sh#L530). Additionally the update script is also able to start the redis server (see https://github.com/e-m-b-a/emba/blob/849eba667a52db7e769d978658b66aa4e75c203c/config/cve_database_updater.init#L8). During normal operation redis server is not needed and does not need to be started: [image: image] <https://user-images.githubusercontent.com/497520/124881707-f49baf00-dfcf-11eb-9903-adc52f0131a8.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUILWT2I3INWIKWX7FFQIDDTWVITFANCNFSM46FY5HCQ> .

[m-1-k-3]

I have added a quick prerequisits section to our wiki: https://github.com/e-m-b-a/emba/wiki/Installation#prerequisites

[m-1-k-3]

Thank you for your feedback. I will add a link to this thread in our wiki

[m-1-k-3]

Thank you for your feedback and your manual on getting everything up and running on less powered hosts.
I have created the following entry in our wiki: https://github.com/e-m-b-a/emba/wiki/Installation#prerequisites
Additionally I have linked your manual in the cve installation area: https://github.com/e-m-b-a/emba/wiki/Installation#cve-search-installation-just-in-case-the-installer-fails

Thanks and keep on using and improving emba :)

[floyd-fuh]

Just an fyi, this entire thing is still an issue. I also set stop-writes-on-bgsave-error no in /etc/redis/redis.conf to prevent some errors. After the installation fails you can do the following to clean out (maybe add to the installation page on the wiki for us redis noobs?):

# redis-cli
127.0.0.1:6379> FLUSHDB 
OK
127.0.0.1:6379> FLUSHALL 
OK
127.0.0.1:6379> exit
# /etc/init.d/redis-server restart
Restarting redis-server (via systemctl): redis-server.service.
# ./external/cve-search/sbin/db_updater.py -f

And then suddenly the installation ran smoothly.

Off-topic fyi: Your tool depends on netstat, so it is necessary to sudo apt install net-tools on a fresh Ubuntu, maybe make that a dependency

[m-1-k-3]

Thank you for your ongoing effort and updates on this redis issue. I will add some notes to the wiki.

Regarding the net-tools: They should get installed:

emba/installer.sh

Line 357 in 5bc2275

print_tool_info "net-tools" 1

Could you do a quick check if your EMBA installation is up to date and if the installation of this dep is failing?

[floyd-fuh]

I'm running in a fresh Ubuntu VM and cloned EMBA from git then ran the installer and yes, it seems to have failed to install the net-tools.

I could (untested!) imagine that the way you handle parameters with a dash (-) in it, that it somehow breaks because the dash is interpreted as a command line parameter switch. But that's just wild speculation.