emba installation with proxy about emba

[torabi12]

Now I have this:
$ pip3 list | grep nested
nested-lookup 0.2.23

[torabi12]

I finished the manul CPE DB download and I could start an emba scan with -p /scan_profiles/full_scan.emba and the cve-search part is working fine!!! This is the first time when it is working with kali user and PROXY installation.
Thank you very much for your support and advices.

In the future it would be nice to fine tune the installation process because the above way is not the best for sure but it works.

[m-1-k-3]

I have currently not tried to install EMBA with a proxy. So let's try it ;)

First, could you try to update/populate the cve-database manually:
$ sudo ./external/cve-search/sbin/db_updater.py -v
I would also recommend using the -f parameter to ensure to get a clean database.

[torabi12]

I tried it but I get an error:

$ sudo ./external/cve-search/sbin/db_updater.py -v -f
[sudo] password for kali:
Traceback (most recent call last):
File "/home/kali/emba/./external/cve-search/sbin/db_updater.py", line 21, in
from lib.DatabaseSchemaChecker import SchemaChecker
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseSchemaChecker.py", line 7, in
from lib.DatabaseHandler import DatabaseHandler
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseHandler.py", line 1, in
from lib.ApiRequests import JSONApiRequest
File "/home/kali/emba/external/cve-search/sbin/../lib/ApiRequests.py", line 5, in
from nested_lookup import nested_lookup, nested_update
ModuleNotFoundError: No module named 'nested_lookup'

I did the same installation without proxy on a different machine and it worked well, but I would like to use the emba in a powerful company workstation and I must use proxy to get any internet access.

[m-1-k-3]

Is your installation corrupt? Could you do a quick check if the rest of cve-search is correctly installed.
The following command should show nested-lookup:
└─$ pip3 list | grep nested nested-lookup 0.2.23
If there is no output, then you probably need to re-install the complete cve-search environment as described here: https://cve-search.github.io/cve-search/getting_started/installation.html#dependencies

[torabi12]

Right now it gives this:
$ pip3 list | grep nested nested-lookup 0.2.23
grep: nested-lookup: No such file or directory
grep: 0.2.23: No such file or directory
ERROR: Pipe to stdout was broken
Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

What I have done before:
Took a a new kali linux:
https://kali.download/virtual-images/kali-2021.4/kali-linux-2021.4-vmware-amd64.7z

Opened in VMware Workstation Pro.
Change shell to bash. for root and kali users:

chsh -s /bin/bash

chsh -s /bin/bash kali

Set the proxy settings as described in the original message above. Insatlled docker and set the proxy as well for docker. Then:
$ sudp apt upgrade
reboot
$ git clone https://github.com/e-m-b-a/emba.git
$ cd ~emba
$ sudo ./install.sh -d

and the installer was not able to install pip requirements in this case. If I do the same without proxy on a different machine it will works fine.
Now I start the whole process from the beginning and I will manually install cve-search before staring emba installer:

$ sudo xargs apt-get install -y < requirements.system
$ pip3 install --proxy http://A.B.C.D:1234 -r requirements.txt

I don't know how to set up proxy for pip to get the emba installer use it automatically.

[m-1-k-3]

Is the python error now gone and can you update your cve-search database now?

[torabi12]

I tried it but I couldn't update the database:

$ pip3 list | grep nested
nested-lookup 0.2.23

┌──(kali㉿kali)-[~/emba]
└─$ sudo ./external/cve-search/sbin/db_updater.py -v -f
[sudo] password for kali:
Traceback (most recent call last):
File "/home/kali/emba/./external/cve-search/sbin/db_updater.py", line 21, in
from lib.DatabaseSchemaChecker import SchemaChecker
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseSchemaChecker.py", line 7, in
from lib.DatabaseHandler import DatabaseHandler
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseHandler.py", line 1, in
from lib.ApiRequests import JSONApiRequest
File "/home/kali/emba/external/cve-search/sbin/../lib/ApiRequests.py", line 5, in
from nested_lookup import nested_lookup, nested_update
ModuleNotFoundError: No module named 'nested_lookup'

I repeated the whole installation process and the setup just stopped again the same point with the error above.

If I do the same as root user it is working but as kali I am not able to install emba.

[torabi12]

I also tried this command:

$ ./sbin/db_mgmt_cpe_dictionary.py -p
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_cpe_dictionary.py", line 47, in
cpd = CPEDownloads()
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 51, in init
super().init(self.feed_type, self.prefix)
File "/home/kali/emba/external/cve-search/sbin/../lib/JSONFileHandler.py", line 10, in init
super().init(feed_type)
File "/home/kali/emba/external/cve-search/sbin/../lib/DownloadHandler.py", line 52, in init
self.file_queue.clear()
File "/home/kali/emba/external/cve-search/sbin/../lib/redis_q.py", line 45, in clear
self.__db.delete(self.key)
File "/home/kali/.local/lib/python3.9/site-packages/redis/commands/core.py", line 972, in delete
return self.execute_command('DEL', *names)
File "/home/kali/.local/lib/python3.9/site-packages/redis/client.py", line 1068, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 1173, in get_connection
connection.connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 575, in connect
self.on_connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 653, in on_connect
auth_response = self.read_response()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 765, in read_response
raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair

but I got an error again:(

[BenediktMKuehne]

pip packages are not global

[torabi12]

I see. I installed the packages as root & kali:

(root💀kali)-[/home/kali/emba/cve-search]
└─# pip3 freeze | grep pymongo
pymongo==3.12.1

┌──(root💀kali)-[/home/kali/emba/cve-search]
└─# exit
logout

┌──(kali㉿kali)-[~/emba/external/cve-search]
└─$ pip3 freeze | grep pymongo
pymongo==3.12.1

┌──(kali㉿kali)-[~/emba/external/cve-search]
└─$ ./sbin/db_mgmt_cpe_dictionary.py -p
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_cpe_dictionary.py", line 47, in
cpd = CPEDownloads()
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 51, in init
super().init(self.feed_type, self.prefix)
File "/home/kali/emba/external/cve-search/sbin/../lib/JSONFileHandler.py", line 10, in init
super().init(feed_type)
File "/home/kali/emba/external/cve-search/sbin/../lib/DownloadHandler.py", line 52, in init
self.file_queue.clear()
File "/home/kali/emba/external/cve-search/sbin/../lib/redis_q.py", line 45, in clear
self.__db.delete(self.key)
File "/home/kali/.local/lib/python3.9/site-packages/redis/commands/core.py", line 972, in delete
return self.execute_command('DEL', *names)
File "/home/kali/.local/lib/python3.9/site-packages/redis/client.py", line 1068, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 1173, in get_connection
connection.connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 575, in connect
self.on_connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 653, in on_connect
auth_response = self.read_response()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 765, in read_response
raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair

The issue is that if I don't use PROXY the default installation process works perfectly:

$ git clone emba
$ cd emba
$ sudo ./installer.sh -d

If I set proxy settings the installer fails. I tried it many times. A few days ago I was install emba with kali user, the installer failed at the same point but I was able to execute these:

$ cd ~/emba/external/cve-search/
$ ./sbin/db_mgmt_cpe_dictionary.py -p
$ ./sbin/db_mgmt_json.py -p
$ ./sbin/db_updater.py -c
$ sudo /etc/init.d/redis-server restart
$ ./sbin/db_updater.py -v

and I started a firmware scan which looked fine, but the CVE-search part did not work at all.

Would it be possible to install emba with PROXY on your side and compare the results?
I am not an expert in Linux, I thought my script would cover all the proxy related things but it looks like incomplete somehow.
Or give me any hint and I will try it!

Thank you very much.

[m-1-k-3]

But the error is changing :) So, there is some progress.

Now we have a Redis auth error. Please check the credentials in external/cve-search/etc/configuration.ini and compare it with the /etc/redis/redis.conf credentials. They need to match.

Currently I have no proxy environment but I put this on my todo list.

[torabi12]

Good morning,

In the configuration.ini file I have these parameters:

[Redis]
Host: localhost
Port: 6379
Password: 9EkREV4Bk
redisQ: 9
VendorsDB: 10
NotificationsDB: 11
RefDB: 12

[Database]
Host: 172.36.0.1
Port: 27017
DB: cvedb
PluginName: mongodb

[dbmgt]
Tmpdir: ./tmp/

[FulltextIndex]
Indexdir: ./indexdir/

[Webserver]
Host: 127.0.0.1
Port: 5000
Debug: False
PageLength: 50
LoginRequired: True
OIDC: False
CLIENT_ID: xx
CLIENT_SECRET: xx
IDP_DISCOVERY_URL: xx
SSL_VERIFY: False
SSL: True
Certificate: ssl/cve-search.crt
Key: ssl/cve-search.key
WebInterface: Full
MountPath: /MOUNTY/MC/MOUNT

[Logging]
Logging: True
Logfile: log/cve-search.log
Updatelogfile: log/update_populate.log
MaxSize: 150MB
Backlog: 5

[Proxy]
http:
IgnoreCerts: False

[CVE]
StartYear: 2002

and in the redis.conf is this:

bind 127.0.0.1 ::1
protected-mode yes
port 6379
requirepass 9EkREV4Bk

In the configuration.ini file's [Proxy] section is it ok with blank parameter?

[torabi12]

Hello Michael, Hello Benedikt,

I know it is long but I did yesterday night a brand new installation with emba with proxy and I would like to share with you all the steps what I did. Some of them can be unnecessary but behind proxy the default very easy installer way is not working for me.

VMware Workstation Pro 16.2.2
Kali-Linux-2021.4-vmware-amd64.vmx
https://kali.download/virtual-images/kali-2021.4/kali-linux-2021.4-vmware-amd64.7z

As root:

chsh -s /bin/bash

chsh -s /bin/bash kali

logout / login
$ echo $0
/bin/bash

Proxy Setup
$ sudo ./proxy.sh
proxy.txt

logout / login

Docker Install & Proxy Setup
$ sudo ./docker.sh
docker.txt

$ sudp apt upgrade
reboot

As kali user:
$ git clone https://github.com/e-m-b-a/emba.git
$ git clone https://github.com/cve-search/cve-search.git

I clone the cve-search by manually because I have to install the prerequisites, let's do it:

$ cd cve-search
$ xargs sudo apt-get install -y < requirements.system
$ pip3 install -r requirements.txt
$ sudo su -

pip3 install -r requirements.txt

Verify:

pip freeze | grep pymongo

exit
$ pip freeze | grep pymongo

All of them work fine in this way.

MongoDB installation
$ cd ..
$ wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
$ sudo touch /etc/apt/sources.list.d/mongodb-org-4.4.list
$ echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/4.4 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
$ sudo apt update
$ sudo apt install -y mongodb-org
$ sudo systemctl daemon-reload
$ sudo systemctl start mongod
$ sudo systemctl status mongod
$ sudo systemctl enable mongod

MongoDB is running fine.

and now emba installation:

$ cd ~/emba
$ sudo ./installer.sh -d

but the installer fails at the CPE Downloads section:

and now I am doing manual download with these commands:

$ cd ~/emba/external/cve-search/
$ ./sbin/db_mgmt_cpe_dictionary.py -p
$ ./sbin/db_mgmt_json.py -p
$ ./sbin/db_updater.py -c
$ sudo /etc/init.d/redis-server restart

I am doing right now this, it takes some time to get all the DB content to be ready but if I add the command one by one it is working as kali user:

I would like to use EMBArk in this proxy environment but without successful emba installation I have no chance to install EMBArk.

[torabi12]

During the installation I have 3 warnings:
WARNING: The FIRMWARE variable is not set. Defaulting to a blank string.
WARNING: The LOG variable is not set. Defaulting to a blank string.
WARNING: The EMBA variable is not set. Defaulting to a blank string.

I would like to mount an external filesystem in /mnt/emba_logs/ folder to have enough space for the tests, would these environment variables be fine to ignore the warning messages?

EMBA='/home/kali/emba/'
FIRMWARE='/home/kali/firmwares/'
LOG='/mnt/emba_logs/'

[m-1-k-3]

Great to hear that you have managed the installation. Thank you very much for detailed explanation and the howto. Currently I work on a redesign of the installer. As soon as we are ready with this update I will check the proxy installation topic again.

You do not need to set the variables. Just configure the emba startup command with the following parameters:
-l paramater for the log directory,
-f for the firmware
Both can be on a different media.

Hope that helps.

[torabi12]

Thanks, I just wanted to skip the warnings during the install but it does not count.

Have a nice day!
Regards,
Torabi

[m-1-k-3]

Linked this issue from the wiki https://github.com/e-m-b-a/emba/wiki/Installation#random-installation-topics

I will close this for now and come back to it later

[m-1-k-3]

FAQ is also updated: https://github.com/e-m-b-a/emba/wiki/FAQ