vuln first sync error about dagda

[eliasgranderubio]

Normally, when a BugTraq Id doesn't exist, the Security Focus web page return the 404 HTTP status code. However, your exception looks that Security Focus web page didn't work fine in that moment.

Anyway, I have added a try-catch for logging this error if it occurs again.

Please, try to initialize your database and give me feedback if the error persists.

[xinity]

sounds better, still initialization is taking too much time :(

it lacks an ending message, so we know initialization is complete

[eliasgranderubio]

I agree with you. At the moment, I'm trying fixing the initialization time and improving the information about BugTraqs in the reports as I have done with the CVEs and the Exploits.

If you want to know if initialization is completed, you have the next endpoint for checking it.

• init-status

[xinity]

At the moment, I'm trying fixing the initialization time and improving the information about BugTraqs in the reports as I have done with the CVEs and the Exploits.

Been working on dagda this week-end, init fails many times, perhaps because you scrap bugtracks and CVE database.

Perhaps a better solution would be to generate a mongodb pre initialize that would reduce sync time for users.
I can help on this topic around Docker integration if you like :)

[eliasgranderubio]

I'm still working for improving the data model and fixing the issues when the bugTracks are scrapped from Security Focus web pages.

I think that your idea could be fine but I have some doubts about:

• If the mongoDB is generated with a pre initialization, I must force a concrete MongoDB version because I think all BSON stored by MongoDB are not full compatible through its versions.
• Following with the previous reasoning, I think it would be better generate a MongoDB export but the export should be hosted in a different place than GitHub, I guess.

Anyway, only the first init take several minutes for populating the MongoDB because Dagda must to create the whole database. The next runs of the init method, the population is incremental so it should not take more than one or two minutes.

Has the initmethod failed yet after the commit that I did 6 days ago?

[xinity]

Has the initmethod failed yet after the commit that I did 6 days ago?

It did because securityfocus wasn't apparently unavailable

[eliasgranderubio]

I've just modified the data model for including the BugTraq ID details. That is way, I have another github repository as my personal PoC about Security Focus web scrapping:

• bidDB_downloader

Anyway, I had not any problem with the web scrapping and I uploaded the result in the previous github repo.

Please, remove your current Dagda data model in your MongoDB, upgrade to the last version and try again. I hope my last changes fixing your issue.

[xinity]

will do as advised and let you know ASAP :)

[eliasgranderubio]

Could you give me some feedback about this issue? Else I suppose the issue was fixed with the last comment that I wrote.