Comments (19)
Now I have this:
$ pip3 list | grep nested
nested-lookup 0.2.23
from emba.
I finished the manul CPE DB download and I could start an emba scan with -p /scan_profiles/full_scan.emba and the cve-search part is working fine!!! This is the first time when it is working with kali user and PROXY installation.
Thank you very much for your support and advices.
In the future it would be nice to fine tune the installation process because the above way is not the best for sure but it works.
from emba.
I have currently not tried to install EMBA with a proxy. So let's try it ;)
First, could you try to update/populate the cve-database manually:
$ sudo ./external/cve-search/sbin/db_updater.py -v
I would also recommend using the -f parameter to ensure to get a clean database.
from emba.
I tried it but I get an error:
$ sudo ./external/cve-search/sbin/db_updater.py -v -f
[sudo] password for kali:
Traceback (most recent call last):
File "/home/kali/emba/./external/cve-search/sbin/db_updater.py", line 21, in
from lib.DatabaseSchemaChecker import SchemaChecker
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseSchemaChecker.py", line 7, in
from lib.DatabaseHandler import DatabaseHandler
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseHandler.py", line 1, in
from lib.ApiRequests import JSONApiRequest
File "/home/kali/emba/external/cve-search/sbin/../lib/ApiRequests.py", line 5, in
from nested_lookup import nested_lookup, nested_update
ModuleNotFoundError: No module named 'nested_lookup'
I did the same installation without proxy on a different machine and it worked well, but I would like to use the emba in a powerful company workstation and I must use proxy to get any internet access.
from emba.
Is your installation corrupt? Could you do a quick check if the rest of cve-search is correctly installed.
The following command should show nested-lookup:
└─$ pip3 list | grep nested nested-lookup 0.2.23
If there is no output, then you probably need to re-install the complete cve-search environment as described here: https://cve-search.github.io/cve-search/getting_started/installation.html#dependencies
from emba.
Right now it gives this:
$ pip3 list | grep nested nested-lookup 0.2.23
grep: nested-lookup: No such file or directory
grep: 0.2.23: No such file or directory
ERROR: Pipe to stdout was broken
Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
What I have done before:
Took a a new kali linux:
https://kali.download/virtual-images/kali-2021.4/kali-linux-2021.4-vmware-amd64.7z
Opened in VMware Workstation Pro.
Change shell to bash. for root and kali users:
chsh -s /bin/bash
chsh -s /bin/bash kali
Set the proxy settings as described in the original message above. Insatlled docker and set the proxy as well for docker. Then:
$ sudp apt upgrade
reboot
$ git clone https://github.com/e-m-b-a/emba.git
$ cd ~emba
$ sudo ./install.sh -d
and the installer was not able to install pip requirements in this case. If I do the same without proxy on a different machine it will works fine.
Now I start the whole process from the beginning and I will manually install cve-search before staring emba installer:
$ sudo xargs apt-get install -y < requirements.system
$ pip3 install --proxy http://A.B.C.D:1234 -r requirements.txt
I don't know how to set up proxy for pip to get the emba installer use it automatically.
from emba.
Is the python error now gone and can you update your cve-search database now?
from emba.
I tried it but I couldn't update the database:
$ pip3 list | grep nested
nested-lookup 0.2.23
┌──(kali㉿kali)-[~/emba]
└─$ sudo ./external/cve-search/sbin/db_updater.py -v -f
[sudo] password for kali:
Traceback (most recent call last):
File "/home/kali/emba/./external/cve-search/sbin/db_updater.py", line 21, in
from lib.DatabaseSchemaChecker import SchemaChecker
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseSchemaChecker.py", line 7, in
from lib.DatabaseHandler import DatabaseHandler
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseHandler.py", line 1, in
from lib.ApiRequests import JSONApiRequest
File "/home/kali/emba/external/cve-search/sbin/../lib/ApiRequests.py", line 5, in
from nested_lookup import nested_lookup, nested_update
ModuleNotFoundError: No module named 'nested_lookup'
I repeated the whole installation process and the setup just stopped again the same point with the error above.
If I do the same as root user it is working but as kali I am not able to install emba.
from emba.
I also tried this command:
$ ./sbin/db_mgmt_cpe_dictionary.py -p
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_cpe_dictionary.py", line 47, in
cpd = CPEDownloads()
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 51, in init
super().init(self.feed_type, self.prefix)
File "/home/kali/emba/external/cve-search/sbin/../lib/JSONFileHandler.py", line 10, in init
super().init(feed_type)
File "/home/kali/emba/external/cve-search/sbin/../lib/DownloadHandler.py", line 52, in init
self.file_queue.clear()
File "/home/kali/emba/external/cve-search/sbin/../lib/redis_q.py", line 45, in clear
self.__db.delete(self.key)
File "/home/kali/.local/lib/python3.9/site-packages/redis/commands/core.py", line 972, in delete
return self.execute_command('DEL', *names)
File "/home/kali/.local/lib/python3.9/site-packages/redis/client.py", line 1068, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 1173, in get_connection
connection.connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 575, in connect
self.on_connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 653, in on_connect
auth_response = self.read_response()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 765, in read_response
raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair
but I got an error again:(
from emba.
from emba.
I see. I installed the packages as root & kali:
(root💀kali)-[/home/kali/emba/cve-search]
└─# pip3 freeze | grep pymongo
pymongo==3.12.1
┌──(root💀kali)-[/home/kali/emba/cve-search]
└─# exit
logout
┌──(kali㉿kali)-[~/emba/external/cve-search]
└─$ pip3 freeze | grep pymongo
pymongo==3.12.1
┌──(kali㉿kali)-[~/emba/external/cve-search]
└─$ ./sbin/db_mgmt_cpe_dictionary.py -p
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_cpe_dictionary.py", line 47, in
cpd = CPEDownloads()
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 51, in init
super().init(self.feed_type, self.prefix)
File "/home/kali/emba/external/cve-search/sbin/../lib/JSONFileHandler.py", line 10, in init
super().init(feed_type)
File "/home/kali/emba/external/cve-search/sbin/../lib/DownloadHandler.py", line 52, in init
self.file_queue.clear()
File "/home/kali/emba/external/cve-search/sbin/../lib/redis_q.py", line 45, in clear
self.__db.delete(self.key)
File "/home/kali/.local/lib/python3.9/site-packages/redis/commands/core.py", line 972, in delete
return self.execute_command('DEL', *names)
File "/home/kali/.local/lib/python3.9/site-packages/redis/client.py", line 1068, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 1173, in get_connection
connection.connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 575, in connect
self.on_connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 653, in on_connect
auth_response = self.read_response()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 765, in read_response
raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair
The issue is that if I don't use PROXY the default installation process works perfectly:
$ git clone emba
$ cd emba
$ sudo ./installer.sh -d
If I set proxy settings the installer fails. I tried it many times. A few days ago I was install emba with kali user, the installer failed at the same point but I was able to execute these:
$ cd ~/emba/external/cve-search/
$ ./sbin/db_mgmt_cpe_dictionary.py -p
$ ./sbin/db_mgmt_json.py -p
$ ./sbin/db_updater.py -c
$ sudo /etc/init.d/redis-server restart
$ ./sbin/db_updater.py -v
and I started a firmware scan which looked fine, but the CVE-search part did not work at all.
Would it be possible to install emba with PROXY on your side and compare the results?
I am not an expert in Linux, I thought my script would cover all the proxy related things but it looks like incomplete somehow.
Or give me any hint and I will try it!
Thank you very much.
from emba.
But the error is changing :) So, there is some progress.
Now we have a Redis auth error. Please check the credentials in external/cve-search/etc/configuration.ini and compare it with the /etc/redis/redis.conf credentials. They need to match.
Currently I have no proxy environment but I put this on my todo list.
from emba.
Good morning,
In the configuration.ini file I have these parameters:
[Redis]
Host: localhost
Port: 6379
Password: 9EkREV4Bk
redisQ: 9
VendorsDB: 10
NotificationsDB: 11
RefDB: 12
[Database]
Host: 172.36.0.1
Port: 27017
DB: cvedb
PluginName: mongodb
[dbmgt]
Tmpdir: ./tmp/
[FulltextIndex]
Indexdir: ./indexdir/
[Webserver]
Host: 127.0.0.1
Port: 5000
Debug: False
PageLength: 50
LoginRequired: True
OIDC: False
CLIENT_ID: xx
CLIENT_SECRET: xx
IDP_DISCOVERY_URL: xx
SSL_VERIFY: False
SSL: True
Certificate: ssl/cve-search.crt
Key: ssl/cve-search.key
WebInterface: Full
MountPath: /MOUNTY/MC/MOUNT
[Logging]
Logging: True
Logfile: log/cve-search.log
Updatelogfile: log/update_populate.log
MaxSize: 150MB
Backlog: 5
[Proxy]
http:
IgnoreCerts: False
[CVE]
StartYear: 2002
and in the redis.conf is this:
bind 127.0.0.1 ::1
protected-mode yes
port 6379
requirepass 9EkREV4Bk
In the configuration.ini file's [Proxy] section is it ok with blank parameter?
from emba.
Hello Michael, Hello Benedikt,
I know it is long but I did yesterday night a brand new installation with emba with proxy and I would like to share with you all the steps what I did. Some of them can be unnecessary but behind proxy the default very easy installer way is not working for me.
VMware Workstation Pro 16.2.2
Kali-Linux-2021.4-vmware-amd64.vmx
https://kali.download/virtual-images/kali-2021.4/kali-linux-2021.4-vmware-amd64.7z
As root:
chsh -s /bin/bash
chsh -s /bin/bash kali
logout / login
$ echo $0
/bin/bash
Proxy Setup
$ sudo ./proxy.sh
proxy.txt
logout / login
Docker Install & Proxy Setup
$ sudo ./docker.sh
docker.txt
$ sudp apt upgrade
reboot
As kali user:
$ git clone https://github.com/e-m-b-a/emba.git
$ git clone https://github.com/cve-search/cve-search.git
I clone the cve-search by manually because I have to install the prerequisites, let's do it:
$ cd cve-search
$ xargs sudo apt-get install -y < requirements.system
$ pip3 install -r requirements.txt
$ sudo su -
pip3 install -r requirements.txt
Verify:
pip freeze | grep pymongo
exit
$ pip freeze | grep pymongo
All of them work fine in this way.
MongoDB installation
$ cd ..
$ wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
$ sudo touch /etc/apt/sources.list.d/mongodb-org-4.4.list
$ echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/4.4 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
$ sudo apt update
$ sudo apt install -y mongodb-org
$ sudo systemctl daemon-reload
$ sudo systemctl start mongod
$ sudo systemctl status mongod
$ sudo systemctl enable mongod
MongoDB is running fine.
and now emba installation:
$ cd ~/emba
$ sudo ./installer.sh -d
but the installer fails at the CPE Downloads section:
and now I am doing manual download with these commands:
$ cd ~/emba/external/cve-search/
$ ./sbin/db_mgmt_cpe_dictionary.py -p
$ ./sbin/db_mgmt_json.py -p
$ ./sbin/db_updater.py -c
$ sudo /etc/init.d/redis-server restart
I am doing right now this, it takes some time to get all the DB content to be ready but if I add the command one by one it is working as kali user:
I would like to use EMBArk in this proxy environment but without successful emba installation I have no chance to install EMBArk.
from emba.
During the installation I have 3 warnings:
WARNING: The FIRMWARE variable is not set. Defaulting to a blank string.
WARNING: The LOG variable is not set. Defaulting to a blank string.
WARNING: The EMBA variable is not set. Defaulting to a blank string.
I would like to mount an external filesystem in /mnt/emba_logs/ folder to have enough space for the tests, would these environment variables be fine to ignore the warning messages?
EMBA='/home/kali/emba/'
FIRMWARE='/home/kali/firmwares/'
LOG='/mnt/emba_logs/'
from emba.
Great to hear that you have managed the installation. Thank you very much for detailed explanation and the howto. Currently I work on a redesign of the installer. As soon as we are ready with this update I will check the proxy installation topic again.
You do not need to set the variables. Just configure the emba startup command with the following parameters:
-l paramater for the log directory,
-f for the firmware
Both can be on a different media.
Hope that helps.
from emba.
Thanks, I just wanted to skip the warnings during the install but it does not count.
Have a nice day!
Regards,
Torabi
from emba.
Linked this issue from the wiki https://github.com/e-m-b-a/emba/wiki/Installation#random-installation-topics
I will close this for now and come back to it later
from emba.
FAQ is also updated: https://github.com/e-m-b-a/emba/wiki/FAQ
from emba.
Related Issues (20)
- NVD CVE database missing HOT 3
- f10 + p99 csv files question HOT 3
- Specify file system instead of firmware HOT 4
- Internet connection - not OK HOT 6
- Improve "ISC DHCP" detection and CVE search HOT 13
- installer.sh failing with non-english locale HOT 10
- module_blacklist.txt gets ignored with default-scan.emba profile HOT 2
- Are we able to get the file system in the firmware? HOT 7
- The s110 module cannot be executed normally HOT 4
- New Kali Linux 2024-2 available HOT 8
- docker image question HOT 9
- emba install error << when i start build the emba in docker with this command "sudo docker-compose build emba " the install stop in 7/9 step << please help me HOT 6
- unsupported manifest format HOT 5
- installer issue HOT 3
- running issue HOT 9
- cannot finish the installation HOT 7
- "./installer.sh -d " failed HOT 3
- Report generation after a (default) scan HOT 8
- Run sudo ./installer.sh in a gitpod.io instance and receive an error HOT 8
- Installation stopped HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emba.