kvasirsecurity / kvasir Goto Github PK

When importing a very large nmap file (100MB), the import will timeout if it is imported as a background process (I guess due to the amount of data required). However it works fine if the import as background is NOT selected.

When using Safari the loading video plays in the background but the controls do not appear. whut?

Error Print 👍
Ticket ID

127.0.0.1.2014-03-20.13-25-35.a13b9990-7855-4854-a106-382a640799df
<type 'exceptions.IOError'> [Errno 21] Is a directory: '/opt/web2py/applications/kvasir/data/scanfiles/'
Version
web2py™ Version 2.9.5-stable+timestamp.2014.03.19.16.24.08
Python Python 2.7.3: /usr/bin/python (prefix: /usr)
Traceback

Traceback (most recent call last):
File "/opt/web2py/gluon/restricted.py", line 220, in restricted
exec ccode in environment
File "/opt/web2py/applications/kvasir/controllers/nmap.py", line 245, in
File "/opt/web2py/gluon/globals.py", line 385, in
self._caller = lambda f: f()
File "/opt/web2py/gluon/tools.py", line 3287, in f
return action(_a, *_b)
File "/opt/web2py/applications/kvasir/controllers/nmap.py", line 145, in import_xml_scan
update_hosts=form.vars.f_update_hosts,
File "applications/kvasir/modules/skaldship/nmap.py", line 88, in process_xml
nmap_parsed.parse_file(filename)
File "applications/kvasir/modules/zenmapCore_Kvasir/NmapParser.py", line 817, in parse_file
f = open(filename, "r")
IOError: [Errno 21] Is a directory: '/opt/web2py/applications/kvasir/data/scanfiles/'

Error snapshot help

<type 'exceptions.IOError'>([Errno 21] Is a directory: '/opt/web2py/applications/kvasir/data/scanfiles/')

Any can help me ? Where is the problem ?

Thanks

fails to parse all information from more recent nmap XML files, i think it's due to zenmapCore_Kvasir was 'forked' before nmap 6.40 came out

the following is a simple nmap -sV xml output to local box scan with openssh only, after a few fixes it does import but it completely disregards the banner information...

http://pastebin.com/raw.php?i=tzwgWHxU

i've done a lame hack since i need this working for now and i updated parts of zenmapCore_Kvasir to fix it

It has been reported that Nessus authentication no longer works so you're unable to pull reports directly from it.

The logo and processing video should be customizable. Right now you have to manually change the files which causes problems with updates as files become out of sync.

Some settings for kvasir.yaml:

• logo_image: images/Kvasir_portrait.png
• loading_video: videos/loading.m4v

If loading_video is blank / none then a overloading loading gif will be used. Files must be located in the static/ directory.

Users should be able to select a list of IP addresses from host/list or a specific IP from host/detail and send available information to Metasploit Pro to import.

After I created an initial account on my local Kali install, when I try to log in, it says the user needs verification. How do I fix this?

Hello,

Importing an XML from nmap v6.00 give me the following error (only pasting the begining of the ticket):

<type 'exceptions.AttributeError'> 'NmapParserSAX' object has no attribute 'port_service_cpe'

Versión
web2py™ Version 2.6.4-stable+timestamp.2013.09.22.01.43.37
Python Python 2.7.3: /usr/bin/python (prefix: /usr)

Traceback

Traceback (most recent call last):
File "/opt/web2py/gluon/restricted.py", line 217, in restricted
exec ccode in environment
File "/opt/web2py/applications/kvasir/controllers/nmap.py", line 146, in
File "/opt/web2py/gluon/globals.py", line 371, in
self._caller = lambda f: f()
File "/opt/web2py/gluon/tools.py", line 3207, in f
return action(_a, *_b)
File "/opt/web2py/applications/kvasir/controllers/nmap.py", line 139, in import_xml_scan
update_hosts=form.vars.f_update_hosts,
File "applications/kvasir/modules/skaldship/nmap.py", line 89, in process_xml
nmap_parsed.parse_file(filename)
File "applications/kvasir/modules/zenmapCore_Kvasir/NmapParser.py", line 818, in parse_file
self.parse(f)
File "applications/kvasir/modules/zenmapCore_Kvasir/NmapParser.py", line 812, in parse
self.parser.parse(f)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 207, in feed
self._parser.Parse(data, isFinal)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 304, in end_element
self._cont_handler.endElement(name)
File "applications/kvasir/modules/zenmapCore_Kvasir/NmapParser.py", line 1083, in endElement
self.dic_port["service_cpe"] = self.port_service_cpe
AttributeError: 'NmapParserSAX' object has no attribute 'port_service_cpe'

Error snapshot help
<type 'exceptions.AttributeError'>('NmapParserSAX' object has no attribute 'port_service_cpe')

nmap XML file header information:

Thank you for this cool tool.

Following your instructions I get:

IOError: [Errno 2] No such file or directory: 'applications/test/databases/sql.log'

A simple mkdir fixes it but for sure other users will encour in this error!

Regards,
Francesco Ongaro

The chosen extension should execute on boxes but is apparently missing within modal dialogs called from AddModal.

No need to separate IPv4 and IPv6 addresses so merge both into a single field, f_ipaddr. This must be a unique field.

Quiet a disruptive change: UI, controllers, modules, functions and API

I have followed the instruction to a T and watched the YouTube video. There has been changes since the last installation video. I cannot get this tool to work. I'd love to see what it is capable of. Could you help me get through the installation properly?

This is the ticket
Ticket ID

127.0.0.1.2014-04-20.00-21-26.c14e333a-2ace-463c-bfc2-407dc32aa1dc
<type 'exceptions.RuntimeError'> Failure to connect, tried 5 times: Traceback (most recent call last): File "/opt/web2py/gluon/dal.py", line 7865, in init self._adapter = ADAPTERSself._dbname File "/opt/web2py/gluon/dal.py", line 694, in call obj = super(AdapterMeta, cls).call(_args, *_kwargs) File "/opt/web2py/gluon/dal.py", line 2870, in init if do_connect: self.reconnect() File "/opt/web2py/gluon/dal.py", line 675, in reconnect self.connection = f() File "/opt/web2py/gluon/dal.py", line 2868, in connector return self.driver.connect(msg,**driver_args) File "/usr/lib/python2.7/dist-packages/psycopg2/init.py", line 179, in connect connection_factory=connection_factory, async=async) OperationalError: FATAL: password authentication failed for user "pguser" FATAL: password authentication failed for user "pguser"

Need to dive into the CANVAS RPC daemon features more. Is this possible?

When executing

./web2py.py -R applications/scan/private/user.py -S scan -M -A -u foo -p bar

An error is returned:

ValueError: invalid literal for int() with base 10: 'SELECT'

The last line of the backtrace is:

File "/home/web2py/gluon/contrib/pg8000/protocol.py", line 783, in createFromData
args['rows'] = int(values[-1])

This seems a well known error, see for reference: https://groups.google.com/forum/#!topic/web2py/nNN1mMk1OgY

"I was just using the pg8000 that comes with web2py, however someone informed me that web2py is broken as shipped (!) and sure enough, after installing psycopg2 my code worked. I can't believe they would ship a product that does not work...and this is such basic functionality! If this is true, they really need to a) Not include pg8000 with their next release and b) Put some kind of blaring warning message with the current release that you NEED psycopg2 to be able to work with postgres."

So not directly related to Kvasir but could confuse the random Debian user!

Regards,
Francesco

Support for launching Metasploit exploits from the vulndata tables would entail the following:

• Allow for selection of targets or single target
• Load module parameters from MSF API
• Load available payloads from MSF API
• Accept user-defined edits of parameters/payloads
• Submit via API

From Metasploit Pro this should all end up as a Task so redirect to the Pro Task list. For Framework not sure of the scenario after submission. Will msfconsole receive the session? Is the session in some sort of RPC limbo? Needs more analysis.

Hi,

I followed the installation instructions presented here https://github.com/KvasirSecurity/Kvasir/wiki/Installation and everything works fine, except for the last step. I see the kvasir login page and I can enter the username and password, but I can't add user/pass combination.

I get the following error when trying to add a new user:

# ./web2py.py -R applications/kvasir/private/user.py -S Kvasir -M -A -u kvasir -p kvasir                                                               
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2014
Version 2.9.5-trunk+timestamp.2014.05.09.15.41.38
Database drivers available: SQLite(sqlite3), MySQL(pymysql), MySQL(MySQLdb), PostgreSQL(psycopg2), PostgreSQL(pg8000), MSSQL(pyodbc), DB2(pyodbc), Teradata(pyodbc), Ingres(pyodbc), IMAP(imaplib)

Kvasir User Add/Modify Management

Traceback (most recent call last):
  File "/opt/web2py/gluon/shell.py", line 262, in run
    execfile(startfile, _env)
  File "applications/kvasir/private/user.py", line 53, in <module>
    user_row = db(db.auth_user.username == user).select().first()
  File "/opt/web2py/gluon/dal.py", line 9017, in __getitem__
    return ogetattr(self, str(key))
AttributeError: 'Table' object has no attribute 'username'

Any ideas about the error and why it happens?

We don't have CORE Impact but from those who do I have heard it may be possible to export the exploit / CVE mapping from their database. Someone should do this, document how they do it and send me the result to build an import routine.

Or someone can take total ownership of this and submit a PR!

This happens because at line 59 there is a

if not password or password == '':
sys.exit("Password cannot be blank\n")

And "password" has not been assigned.

While should be handled in a similar way "user" is assigned in the program to work from command line options (non-interactive/prompt).

My fix was adding

if options.password and not options.prompt:
password = options.password

before

if not password or password == '':

Regards,
Francesco

Sometimes a service will have a vulnerability reference id for a record that doesn't exist. This can synchronization can happen for a couple of reasons but it plays havoc with reporting and statistics.

Possible solution is to generate a private script that will build a list of vulndata ids and cycle through the t_service_vulns records, removing anything with a f_vulndata_id that doesn't exist anymore.

Internal error in http://127.0.0.1:8000/Kvasir/default/index ,

Error ticket for "Kvasir"
Ticket ID
127.0.0.1.2013-10-10.22-49-26.64bd5121-6fec-4052-ab20-0cd738f795c5

<type 'exceptions.NameError'> name 'auth' is not defined

Versión
web2py™ Version 2.6.3-stable+timestamp.2013.09.15.17.01.20
Python Python 2.7.3: /usr/bin/python (prefix: /usr)
Traceback

Traceback (most recent call last):
File "/home/python/Público/web2py/web2py/gluon/restricted.py", line 217, in restricted
exec ccode in environment
File "/home/python/Público/web2py/web2py/applications/Kvasir/models/db_01_static.py", line 33, in
Field('f_nessus_pw', 'password', default='password', label=T('Nessus Password')),
NameError: name 'auth' is not defined

Hi, for the past few months I've been working on an improved module cache for metasploit-framework. Within the next 4 to 6 weeks, this module caching branch (https://github.com/limhoff-r7/metasploit-framework/tree/feature/module-caching) will under go a Public Alpha for Metasploit Framework contributors and then be merged into rapid7/metasploit-framework/tree/master once the community is comfortable with the changes and Metasploit Pro has adjusted too. I tell you this, because the database schema will be changed and you may need to change Kvasir's database schema to be able to read the new format. The new schema can be seen by checking out metasploiit_data_models's entity-relationship diagram.

git clone [email protected]:rapid7/metasploit_data_models.git
git checkout feature/module-caching
bundle install
rake yard
open doc/Mdm.html.

To assist our SQL/Report expert, @shuckins-r7, I've also added a page to cover translating different queries from the old (current) schema to the new schema:

open doc/file.mdm_module_sql_translation.html

If you'd like to be included in the public alpha please contact @todb-r7 to be added to the list. I'm sure I'll be busy during the public alpha stomping bugs, but I do have prior experience with Python, so I may be able to help specifically with any issues Kvasir has.

Data is sometimes put into t_errata and never into t_event_log. This should be better utilized for instance Scan filenames could be put in here. Should probably figure out some key names before going too far.

The Metasploit XML report file includes vulnerability data. It does some normalization for its own use so Kvasir may need to re-normalize it to make sure to limit duplications.

IP addresses don't need to be separated between v4 and v6. Merge into a single f_ipaddr field. This will break some third-party tools as the API will change but it's necessary.

Commit 8094b91 works around this but need to track down unicode issues with lxml again:

[!] Error parsing test data: 'ascii' codec can't encode characters in position 446-448: ordinal not in range(128)

Currently when adding entries (vulns, services, etc) Select boxes use content that is generated by the 01_form_modal and placed into HTML. This is easy but since content may change in other windows it requires a page refresh to see that new data.

Example: http://ivaynberg.github.io/select2/#ajax

The index charts look funny when the vulnerability data does not use a 0-10 scoring. By default we show f_severity but allow the user to view as CVSS (or both!)

CPE elements that begin with C seem to have problems. From a nessus report:

The remote operating system matched the following CPE : 

  cpe:/o:cisco:ios:12.4 -> Cisco IOS 12.4
</plugin_output>

Returns: isco:ios:12.4 -> Cisco IOS 12.4

You've collected hashes, Kvasir compares them to already known cleartext combinations, how about running what's left through John the Ripper?

• JTR directory run directory should be configured in yaml
• Support JTR option selection
• Set scheduler timeout length or run time?
• Q: use std JTR or magnum/bleeding?
• Supported hash types? (LM, NTLM, Oracle, DES, MDx, SHAx, etc)

(Pwnwiki)[https://github.com/pwnwiki/pwnwiki.github.io] is a great static wiki. Kvasir should easily be able to serve up the files given a path in the kvasir.yaml config file.

Kvasir is a little difficult to configure and as more features are added it'll only get worse. There should be a central YAML-formatted configuration file that can be pulled from the user's $HOME/.kvasir directory with a fallback as the Kvasir root.

A few gotchas:

• A failure to find the file can't be easily caught in the WebUI as the load will occur within web2py's code and not Kvasir.
• Default configuration items will need to be set during loading. This may cause some things to appear working when they really do not (e.g. typo in the YAML configs)
• YAML loading should happen before critical DB functions are executed.

Hi all
I have installed the application as given in wiki. But when I try to log in I get error " Registration needs Verification " How do I verify it. Please help

Currently zenmapCore does not process NSE results. This needs to be done so Kvasir can load script results and parse them into useful database tables!

Add support to communicate with more than one Nessus server to pull reports.

Currently the WebShot valkyrie runs down a list of hosts one-at-a-time. This is annoying for large lists of HTTP services (and those mis-identified).

One solution is to do multiprocessing. Another is to set up multiple scheduler jobs in groups of 100.

When 'Update Hosts' is selected in the Nessus Import data, the import will crash in the class NessusHosts.parse() method.

Under self.update_hosts it tries to refer a variable localdb that does not exists. The import worsk fine if 'Update Hosts' is not selected.

On first run the following tables are created:

auth_cas
auth_event
auth_group
auth_membership
auth_permission
auth_user
scheduler_run
scheduler_task
scheduler_worker
t_accounts
t_apps
t_cpe_apps
t_cpe_hardware
t_cpe_os
t_errata
t_event_log
t_evidence
t_exploit_references
t_exploits
t_host_notes
t_host_os_refs
t_hosts
t_netbios
t_os
t_service_info
t_service_vulns
t_services
t_services_apps_refs
t_snmp
t_vuln_references
t_vuln_refs
t_vulndata

With a 0 row count.

This way it's impossible to login and the scheduler output errors like:

ERROR:web2py.scheduler.checker#25967: error popping tasks
ERROR:web2py.scheduler.checker#25967:Error retrieving status

Is this intentional?

Regards,
Francesco Ongaro

The WebAPI query / IP lookup feature has not been tested so it's disabled in 1.0.0. Should devote some time to test before 1.0.1 release.

i got this error

Error ticket for "kvasir"
Ticket ID

192.168.100.1.2013-10-19.14-03-59.4a580b8a-914e-4e42-b8cb-a8ff6ad917c8
<type 'exceptions.RuntimeError'> Failure to connect, tried 5 times: Traceback (most recent call last): File "/opt/web2py/gluon/dal.py", line 7562, in init self._adapter = ADAPTERSself._dbname File "/opt/web2py/gluon/dal.py", line 2713, in init if do_connect: self.reconnect() File "/opt/web2py/gluon/dal.py", line 627, in reconnect self.connection = f() File "/opt/web2py/gluon/dal.py", line 2711, in connector return self.driver.connect(msg,**driver_args) File "/usr/lib/python2.7/dist-packages/psycopg2/init.py", line 179, in connect connection_factory=connection_factory, async=async) OperationalError: could not connect to server: Connection refused Is the server running on host "localhost" (127.0.0.1) and accepting TCP/IP connections on port 5432?

I am trying to import nmap xml file but the import is in queue for a long time. How do I execute it.

QualysGuard was originally used by the SPA team so there are internal libraries that can be used to parse the data into python objects. Need to then turn those objects into Kvasir database entries and connections.

I'm not a big fan of response.flash for longer and possibly more important update information. Things like URLs are not clickable in the flash box. Long content is not really viewable. HTML is stripped, etc.

We should write our own response header processor to place content in an alert div instead.

Where the user has the option of adding processed files to "evidence" they should now go to "tool output" instead. Evidence should be for:

• screenshot / images
• documents
• actual evidence files you want to share

Schiffman's already done the heavy lifting: https://github.com/mschiffm/cvrfparse/blob/master/cvrfparse/cvrfparse.py

Since exploitdb is now synced on github (https://github.com/offensive-security/exploit-database) we should be able to point directly to and installation directory and allow updates.

In Linux when using the chromium browser on LOAD()ed objects, Chosen doesn't set the correct width:

Need to complete the newer task scheduler interface as the current one is buggy:

• task status output does not work correctly
• scheduler_run only picks up the first, not the last
• cannot force a task to stop if multiple scheduler_runs

Currently the os/by_host html lists out all the known OS references for the host. There is a "select all" but there should be a "select highest" or "select all but highest" or whatever it can be called.

This would select all OS ref records EXCEPT the highest certainty.

Support to parse Nessus and OpenVAS XML report formats into our database structure.