unable to start dagda about dagda

[eliasgranderubio]

Hi,

@3lcarry , my last commit should fix the ValueError you had when the init db process was run.

@ntart23 , if you are using the same machine for the MongoDB and the Dagda Server, you need at least 6GB RAM because MongoDB uses a lot of RAM for the indexes creation and the init db process uses RAM in the same way.

Regards.

[eliasgranderubio]

Hi @ntart23 ,

Have you review the Dagda documentation about its CLI usage?

On the other hand, be careful with the sudo because you look like set the environment variable with a non-sudo user and then you use the CLI with a sudo user. Please, review this issue for more information because it was some similar to your description.

Regards.

[ntart23]

Thanks for quickly response .
I referenced your answer and reinstalled ubuntu , docker, dagda with root(without sudo) and solved that problem.

but now i have a new problem. after start dagda, i typed [export DAGDA_HOST='127.0.0.1' and export DAGDA_PORT=5000 and python3 dagda.py vuln --init ] in other terminal. but when i typed init_status , it is printed status:initializing over 4hours.

and i referenced #32 , so i typed start dagda -d , but forever initializing and not became updated too.

root@ubuntu:/home/joon/Downloads/dagda-master/dagda# python3 dagda.py vuln --init
{
"msg": "Accepted the init db request"
}
root@ubuntu:/home/joon/Downloads/dagda-master/dagda# python3 dagda.py vuln --init_status
{
"status": "Initializing",
"timestamp": "2019-05-28 02:53:58.304500"
}
root@ubuntu:/home/joon/Downloads/dagda-master/dagda# python3 dagda.py vuln --init_status
{
"status": "Initializing",
"timestamp": "2019-05-28 02:53:58.304500"
}

Serving on http://localhost:5000
<2019-05-28 11:53:58,363> <db_composer> <compose_vuln_db:53>
<2019-05-28 11:53:58,364> <db_composer> <compose_vuln_db:58> <Updating CVE collection ...>
<2019-05-28 11:53:58,365> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2002>
<2019-05-28 11:53:58,368> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2003>
<2019-05-28 11:53:58,370> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2004>
<2019-05-28 11:53:58,371> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2005>
<2019-05-28 11:53:58,373> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2006>
<2019-05-28 11:53:58,381> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2007>
<2019-05-28 11:53:58,383> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2008>
<2019-05-28 11:53:58,386> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2009>
<2019-05-28 11:53:58,388> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2010>
<2019-05-28 11:53:58,394> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2011>
<2019-05-28 11:53:58,397> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2012>
<2019-05-28 11:53:58,400> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2013>
<2019-05-28 11:53:58,403> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2014>
<2019-05-28 11:53:58,405> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2015>
<2019-05-28 11:53:58,407> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2016>
<2019-05-28 11:53:58,408> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2017>
<2019-05-28 11:53:58,410> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2018>
<2019-05-28 11:53:58,412> <db_composer> <_threaded_cve_gathering:148> <... Including CVEs - 2019>

root@ubuntu:/home/joon/Downloads/dagda-master/dagda# netstat -an | grep 5000
tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:36200 127.0.0.1:5000 TIME_WAIT
tcp 0 0 127.0.0.1:36198 127.0.0.1:5000 TIME_WAIT

in this situation, i excuted python3 dagda.py vuln --product openldap --product_version 2.2.20,
then these error came out
{
"err": 500,
"msg": "Internal Server Error"
}
in start dagda terminal
<2019-05-28 16:03:59,356> <log_exception:1761> <Exception on /v1/vuln/products/openldap/2.2.20 [GET]>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 35, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/joon/Downloads/dagda-master/dagda/api/service/vuln.py", line 57, in get_vulns_by_product_and_version
vulns = InternalServer.get_mongodb_driver().get_vulnerabilities(product, version)
File "/home/joon/Downloads/dagda-master/dagda/driver/mongodb_driver.py", line 325, in get_vulnerabilities
for bid in bid_cursor:
File "/usr/local/lib/python3.6/dist-packages/pymongo/cursor.py", line 1189, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python3.6/dist-packages/pymongo/cursor.py", line 1104, in _refresh
self.__send_message(q)
File "/usr/local/lib/python3.6/dist-packages/pymongo/cursor.py", line 982, in __send_message
helpers._check_command_response(first)
File "/usr/local/lib/python3.6/dist-packages/pymongo/helpers.py", line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: text index required for $text query (no such collection 'vuln_database.bid')

everytime i try, i erased and reinstalled mongodb to clean every collections, and my ubuntu RAM is 4GB.
i can't find out what's wrong in my environment settings again. help me please !

[3lcarry]

Hi,
I'm running the same issue, additional after some time the vuln --init_status give me the following error:
{
"status": "Unexpected exception of type ValueError occurred: ('substring not found',)",
"timestamp": "2019-06-09 11:05:03.552788"
}

I've tried with mongo 3.6 and 4.

Additional on the server I've seen the following error:

<2019-06-09 13:04:12,478> <db_composer> <compose_vuln_db:72>
<2019-06-09 13:04:12,479> <db_composer> <compose_vuln_db:77> <Updating Exploit DB collection ...>
<2019-06-09 13:04:18,904> <db_composer> <compose_vuln_db:88>
<2019-06-09 13:04:18,904> <db_composer> <compose_vuln_db:93> <Updating RHSA & RHBA collections ...>
<2019-06-09 13:05:03,544> <dagda_server> <_init_or_update_db:155> <Unexpected exception of type ValueError occurred: ('substring not found',)>
Traceback (most recent call last):
File "/home/carrillo/tfm/dagda/dagda/api/dagda_server.py", line 150, in _init_or_update_db
db_composer.compose_vuln_db()
File "/home/carrillo/tfm/dagda/dagda/vulnDB/db_composer.py", line 100, in compose_vuln_db
rhsa_list, rhba_list, rhsa_info_list, rhba_info_list = get_rhsa_and_rhba_lists_from_file(bz2_file)
File "/home/carrillo/tfm/dagda/dagda/vulnDB/ext_source_util.py", line 271, in get_rhsa_and_rhba_lists_from_file
rhsa_id = rhsa_id[:rhsa_id.index("-", 5)]
ValueError: substring not found

[3lcarry]

Thank you @eliasgranderubio, now is working fine

@ntart23 I was having the same error but after initial sync disappeared (but I have 8GB of RAM)