fidius / cvedb

A ruby gem for Rails to fetch, parse, store and search CVE entries provided by the National Vulnerability Database.

FIDIUS fidius-cvedb

The FIDIUS CVE-DB Gem is used to run your own vulnerability database, based on the National Vulnerability Database (NVD nvd.nist.gov). It can be used for projects that need access to vulnerability information, such as the Common Vulnerabilities and Exposures (CVE) identifiers.

It includes rake tasks to download and parse XML files provided by the NVD, to store and update them in your personal database. Furthermore it includes ActiveRecord models, migrations and example database configuration to store Vulnerabilities easily.

This gem is being developed in the context of the student project "FIDIUS" at the Universitaet Bremen; for more information about FIDIUS visit fidius.me.

Installation

Simply install this package with Rubygems:

$ gem install fidius-cvedb

Then add it to your gemfile (Rails 3)

gem 'fidius-cvedb'
gem 'mysql2' # only required when you use mysql db like in the example below

or environment.rb (prior Rails 3)

require 'rubygems'
require 'active_record'
require 'fidius-cvedb'

Please note: The CVE-DB Gem has only been tested with Linux systems and might not work with Windows.

Configuration

The setup depends on the context you want to use the gem. It can be used in the context of the FIDIUS Command&Control Server, or in your own Rails app. It might access an already existing database or migrate a new one.

0. Go to your Rails-app folder and run fidius-cvedb --standalone or fidius-cvedb --fidius, depending on the context you are using it. For Rails versions prior 3 this will create symlinks for the Rake tasks.

1. Set up a new CVE Database if you need to or configure an existing one, add the CVE Database to your database.yml accordingly. Note that it must be named "cve_db":

      cve_db:
        adapter: mysql2
        encoding: utf8
        database: my_cve_database
        pool: 5
        username: my_username
        password: my_password
        host: localhost
   

2. When you created a new database, run rake nvd:migrate to create the tables needed.

3. When you set up your own database initialize it (note that it needs to be migrated before). Go to your Rails-app folder and run rake nvd:initialize. This will download all available information from the NVD, parse and store it in your database. This takes about 3 hours, depending on your machine. To keep your database up-to-date run rake nvd:update regularly, e.g. as a daily cron job.

4. Now you should be able to use the NVD entries, to test this go to your console (rails console | ruby script/console) and get an entry:

    $ FIDIUS::CveDb::NvdEntry.first
   

Synopsis

This package comes with an executable script. You may invoke it as

$ fidius-cvedb <option>

where option may be:

• -f | --fidius Initialize CVE-DB for Usage in FIDIUS C&C-Server
• -s | --standalone Initialize CVE-DB standalone version
• -h | --help Show help message
• -v | --version Shows the gem version

Authors and Contact

fidius-cvedb was written by

• FIDIUS Intrusion Detection with Intelligent User Support [email protected], http://fidius.me
• in particular:
• Andreas Bender [email protected]
• Jens Färber [email protected]

If you have any questions, remarks, suggestion, improvements, etc. feel free to drop a line at the addresses given above. You might also join #fidius on Freenode or use the contact form on our website.

License

Simplified BSD License and GNU GPLv2. See also the file LICENSE.

cvedb's Issues

cvedb does not parse XMLs with PostgreSQL

Hi there,

I tried using fidius-cvedb with a postgresql database in rails and when I try and parse an xml I get the following:

rake RAILS_ENV=cve_db nvd:parse["nvdcve-2.0-modified.xml"]
...
rails runner /usr/local/ruby/lib/ruby/gems/1.9.1/gems/fidius-cvedb-0.0.7/lib/cveparser/main.rb -p  /home/tomek/cve/cveparser/xml/nvdcve-2.0-modified.xml
You did not specify how you would like Rails to report deprecation notices for your cve_db environment, please set config.active_support.deprecation to :log, :notify or :stderr at config/environments/cve_db.rb
[*] Start parsing "/home/tomek/cve/cveparser/xml/nvdcve-2.0-modified.xml"
Parsed 100 CVE Entries.
[*] Finished parsing, parsed 116 entries in 1 seconds.
[*] Storing the CVE-Entries in DB
Store: CVE-2003-0497 [1/116]
Store: CVE-2003-0498 [2/116]
Store: CVE-2007-6750 [3/116]
Store: CVE-2009-0052 [4/116]
/usr/local/ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:980:in `get_last_result': PGError: ERROR:  value too long for type character varying(255) (ActiveRecord::StatementInvalid)
: INSERT INTO "nvd_entries" ("created_at", "cve", "cwe", "last_modified", "published", "summary", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7) RETURNING "id"

I believe its because in your migrations your using strings for the database datatypes and i believe the postgresql adapter in rails maps that to a varying(255) datatype. I believe its the summaries that are that long and causing this error so maybe the solution would be to change the summary column in nvd_entries from string to text?

I am running rails 3.1.0 with ruby 1.9.3

Thanks,
Tomek

EDIT: it also seems like there may be something in the vulnerability_references table thats causing the same error:

...
Store: CVE-2003-1228 [1147/1515]
Store: CVE-2003-1229 [1148/1515]
Store: CVE-2003-1230 [1149/1515]
Store: CVE-2003-1231 [1150/1515]
Store: CVE-2003-1232 [1151/1515]
/usr/local/ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:980:in `get_last_result': PGError: ERROR:  value too long for type character varying(255) (ActiveRecord::StatementInvalid)
: INSERT INTO "vulnerability_references" ("created_at", "link", "name", "nvd_entry_id", "source", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"

looking at CVE-2003-1232| the name and link seem to be 303 characters:

Name: http://groups.google.com/group/gnu.emacs.bug/browse_frm/thread/9424ec1b2fdae321/c691a2da8904db0f?hl=en&lr=&ie=UTF-8&oe=UTF-8&rnum=1&prev=/groups%3Fq%3Dguninski%2Bemacs%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3Dmailman.763.1041357806.19936.bug-gnu-emacs%2540gnu.org%26rnum%3D1#c691a2da8904db0f
Hyperlink:http://groups.google.com/group/gnu.emacs.bug/browse_frm/thread/9424ec1b2fdae321/c691a2da8904db0f?hl=en&lr=&ie=UTF-8&oe=UTF-8&rnum=1&prev=/groups%3Fq%3Dguninski%2Bemacs%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3Dmailman.763.1041357806.19936.bug-gnu-emacs%2540gnu.org%26rnum%3D1#c691a2da8904db0f