GithubHelp home page GithubHelp logo

pycvesearch's Introduction

cve-search

Join the chat at https://gitter.im/cve-search/cve-search Build & Test Black formatting CodeQL

cve-search logo

cve-search is a tool to import CVE (Common Vulnerabilities and Exposures) and CPE (Common Platform Enumeration) into a MongoDB to facilitate search and processing of CVEs.

The main objective of the software is to avoid doing direct and public lookups into the public CVE databases. Local lookups are usually faster and you can limit your sensitive queries via the Internet.

cve-search includes a back-end to store vulnerabilities and related information, an intuitive web interface for search and managing vulnerabilities, a series of tools to query the system and a web API interface.

cve-search is used by many organizations including the public CVE services of CIRCL.

This document gives you basic information how to start with cve-search. For more information please refer to the documentation in the /doc folder of this project.

Getting started

Check the documentation to get you started

Usage

You can search the database using search.py.

usage: search.py [-h] [-q Q] [-p P [P ...]] [--only-if-vulnerable] [--strict_vendor_product] [--lax] [-f F] [-c C] [-o O]
                 [-l] [-n] [-r] [-a] [-v V] [-s S] [-t T] [-i I]

Search for vulnerabilities in the National Vulnerability DB. Data from http://nvd.nist.org.

options:
  -h, --help            show this help message and exit
  -q Q                  Q = search pip requirements file for CVEs, e.g. dep/myreq.txt
  -p P [P ...]          S = search one or more products, e.g. o:microsoft:windows_7 or o:cisco:ios:12.1 or
                        o:microsoft:windows_7 o:cisco:ios:12.1. Add --only-if-vulnerable if only vulnerabilities that
                        directly affect the product are wanted.
  --only-if-vulnerable  With this option, "-p" will only return vulnerabilities directly assigned to the product. I.e.
                        it will not consider "windows_7" if it is only mentioned as affected OS in an adobe:reader
                        vulnerability.
  --strict_vendor_product
                        With this option, a strict vendor product search is executed. The values in "-p" should be
                        formatted as vendor:product, e.g. microsoft:windows_7
  --lax                 Strict search for software version is disabled. Likely gives false positives for earlier
                        versions that were not yet vulnerable. Note that version comparison for non-numeric values
                        is done with simplifications.
  -f F                  F = free text search in vulnerability summary
  -c C                  search one or more CVE-ID
  -o O                  O = output format [csv|html|json|xml|cveid]
  -l                    sort in descending mode
  -n                    lookup complete cpe (Common Platform Enumeration) name for vulnerable configuration
  -r                    lookup ranking of vulnerable configuration
  -a                    Lookup CAPEC for related CWE weaknesses
  -v V                  vendor name to lookup in reference URLs
  -s S                  search in summary text
  -t T                  search in last n day
  -i I                  Limit output to n elements (default: unlimited)

Examples:

./bin/search.py -p cisco:ios:12.4
./bin/search.py -p cisco:ios:12.4 -o json
./bin/search.py -f nagios -n
./bin/search.py -p microsoft:windows_7 -o html

If you want to search all the WebEx vulnerabilities and only printing the official references from the supplier.

./bin/search.py -p webex: -o csv  -v "cisco"

You can also dump the JSON for a specific CVE ID.

./bin/search.py -c CVE-2010-3333 -o json

Or dump the last 2 CVE entries in RSS or Atom format.

./bin/dump_last.py -f atom -l 2

Or you can use the webinterface.

./web/index.py

Usage of the ranking database

There is a ranking database allowing to rank software vulnerabilities based on their common platform enumeration name. The ranking can be done per organization or department within your organization or any meaningful name for you.

As an example, you can add a partial CPE name like "sap:netweaver" which is very critical for your accounting department.

./sbin/db_ranking.py  -c "sap:netweaver" -g "accounting" -r 3

and then you can lookup the ranking (-r option) for a specific CVE-ID:

./bin/search.py -c CVE-2012-4341  -r  -n

Advanced usage

As cve-search is based on a set of tools, it can be used and combined with standard Unix tools. If you ever wonder what are the top vendors using the term "unknown" for their vulnerabilities:

python3 bin/search_fulltext.py -q unknown -f \
    | jq -c '. | .vulnerable_configuration[0]' \
    | cut -f5 -d: | sort  | uniq -c  | sort -nr | head -10

1500 oracle
381 sun
372 hp
232 google
208 ibm
126 mozilla
103 microsoft
100 adobe
 78 apple
 68 linux

You can compare CVSS (Common Vulnerability Scoring System ) values of some products based on their CPE name. Like comparing oracle:java versus sun:jre and using R to make some statistics about their CVSS values:

python3 bin/search.py -p oracle:java -o json \
  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.800   5.350   9.300   7.832  10.000  10.000
python3 bin/search.py -p sun:jre -o json \
  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.000   5.000   7.500   7.333  10.000  10.000

Fulltext indexing

If you want to index all the CVEs from your current MongoDB collection:

./sbin/db_fulltext.py -l 0

and you query the fulltext index (to get a list of matching CVE-ID):

./bin/search_fulltext.py -q NFS -q Linux

or to query the fulltext index and output the JSON object for each CVE-ID:

./bin/search_fulltext.py -q NFS -q Linux -f

Fulltext visualization

The fulltext indexer visualization is using the fulltext indexes to build a list of the most common keywords used in CVE. NLTK is required to generate the keywords with the most common English stopwords and lemmatize the output. NTLK for Python 3 exists but you need to use the alpha version of NLTK.

./bin/search_fulltext.py  -g -s >cve.json

cve-search visualization

You can see a visualization on the demo site.

Web interface

The web interface is a minimal interface to see the last CVE entries and query a specific CVE. You'll need flask in order to run the website and Flask-PyMongo. To start the web interface:

cd ./web
./index.py

Then you can connect on http://127.0.0.1:5000/ to browser the last CVE.

Web API interface

The web interface includes a minimal JSON API to get CVE by ID, by vendor or product. A public version of the API is also accessible on cve.circl.lu.

List the know vendors in JSON

curl "http://127.0.0.1:5000/api/browse/"

Dump the product of a specific vendor in JSON

curl "http://127.0.0.1:5000/api/browse/zyxel"
{
  "product": [
    "n300_netusb_nbg-419n",
    "n300_netusb_nbg-419n_firmware",
    "p-660h-61",
    "p-660h-63",
    "p-660h-67",
    "p-660h-d1",
    "p-660h-d3",
    "p-660h-t1",
    "p-660h-t3",
    "p-660hw",
    "p-660hw_d1",
    "p-660hw_d3",
    "p-660hw_t3"
  ],
  "vendor": "zyxel"
}

Find the associated vulnerabilities to a vendor and a product.

curl "http://127.0.0.1:5000/api/search/zyxel/p-660hw" | jq .
[
  {
    "cwe": "CWE-352",
    "references": [
      "http://www.exploit-db.com/exploits/33518",
      "http://secunia.com/advisories/58513",
      "http://packetstormsecurity.com/files/126812/Zyxel-P-660HW-T1-Cross-Site-Request-Forgery.html",
      "http://osvdb.org/show/osvdb/107449"
    ],
    "vulnerable_configuration": [
      "cpe:/h:zyxel:p-660hw:_t1:v3"
    ],
    "Published": "2014-06-16T14:55:09.713-04:00",
    "id": "CVE-2014-4162",
    "Modified": "2014-07-17T01:07:29.683-04:00",
    "cvss": 6.8,
    "summary": "Multiple cross-site request forgery (CSRF) vulnerabilities in the Zyxel P-660HW-T1 (v3) wireless router allow remote attackers to hijack the authentication of administrators for requests that change the (1) wifi password or (2) SSID via a request to Forms/WLAN_General_1."
  },
  {
    "cwe": "CWE-20",
    "references": [
      "http://www.kb.cert.org/vuls/id/893726"
    ],
    "vulnerable_configuration": [
      "cpe:/h:zyxel:p-660h-63:-",
      "cpe:/h:zyxel:p-660h-t1:-",
      "cpe:/h:zyxel:p-660h-d3:-",
      "cpe:/h:zyxel:p-660h-t3:v2",
      "cpe:/h:zyxel:p-660h-t1:v2",
      "cpe:/h:zyxel:p-660h-d1:-",
      "cpe:/h:zyxel:p-660h-67:-",
      "cpe:/h:zyxel:p-660h-61:-",
      "cpe:/h:zyxel:p-660hw_t3:v2",
      "cpe:/h:zyxel:p-660hw_t3:-",
      "cpe:/h:zyxel:p-660hw_d3:-",
      "cpe:/h:zyxel:p-660hw_d1:v2",
      "cpe:/h:zyxel:p-660hw_d1:-",
      "cpe:/h:zyxel:p-660hw:_t1:v2",
      "cpe:/h:zyxel:p-660hw:_t1:-"
    ],

Software using cve-search

Docker versions

Official dockerized version of cve-search:

CVE-Search-Docker

There are some unofficial dockerized versions of cve-search (which are not maintained by us):

Changelog

You can find the changelog on GitHub Releases (legacy changelog).

License

cve-search is free software released under the "GNU Affero General Public License v3.0"

Copyright (c) 2012 Wim Remes - https://github.com/wimremes/
Copyright (c) 2012-2024 Alexandre Dulaunoy - https://github.com/adulau/
Copyright (c) 2015-2019 Pieter-Jan Moreels - https://github.com/pidgeyl/
Copyright (c) 2020-2024 Paul Tikken - https://github.com/P-T-I

pycvesearch's People

Contributors

adulau avatar codepros avatar likema avatar rafiot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pycvesearch's Issues

Type Error

Error

Traceback (most recent call last):
  File "\path\To\File\<filename>.py", line 2, in <module>
    cve = CVESearch()
TypeError: __init__() missing 1 required positional argument: 'base_url'

Code :

from pycvesearch import CVESearch
cve = CVESearch()
cve.id('CVE-2014-0160')

How I Installed it :

pip install pycvesearch
  • Its a Example Code From The README.MD , But Its not working?
  • My System Is : Windows 10

JSONDecodeError doing "cve.search('microsoft/office')"

python 3.9 - running example gives exception

Traceback (most recent call last):
  File "/Users/drace/opt/miniconda3/envs/mloperator/lib/python3.9/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Users/drace/opt/miniconda3/envs/mloperator/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Users/drace/opt/miniconda3/envs/mloperator/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/drace/opt/miniconda3/envs/mloperator/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/drace/opt/miniconda3/envs/mloperator/lib/python3.9/site-packages/pycvesearch/core.py", line 41, in search
    return data.json()
  File "/Users/drace/opt/miniconda3/envs/mloperator/lib/python3.9/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>> ```

search function is broken

al other function's are working but search is crashing because a json decoding fault
raise JSONDecodeError("Expecting value", s, err.value) from None

Socket networking problem with updated version of cvefor method

The fix Alexandre provided for the previous issue with the same method worked just for a while. Now my output is a new error which you can find below. I'm using the Python wrapper of cve-search (PyCVESearch) for remote requests on cve.circl.lu, shall I update any module in Python3 or through pip?

EDIT: now Python APIs (specially cvefor method) work, but response to cvefor requests is very very very slow (cve.cvefor('cpe:2.3:o:canonical:ubuntu_linux:16.04:*') lasts something like 25/30 minutes to output a JSON object). Quite difficult to handle and I'm using them into my master thesis.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 137, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 81, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 217, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 146, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f6dd95f6320>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 283, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cve.circl.lu', port=443): Max retries exceeded with url: /api/cvefor/cpe:/a:microsoft:office:2011::mac (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f6dd95f6320>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/pycvesearch/core.py", line 76, in cvefor
    data = self._http_get('cvefor', query=param)
  File "/usr/local/lib/python3.6/dist-packages/pycvesearch/core.py", line 26, in _http_get
    return self.session.get(url, timeout=self.timeout)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 492, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 480, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 588, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='cve.circl.lu', port=443): Max retries exceeded with url: /api/cvefor/cpe:/a:microsoft:office:2011::mac (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f6dd95f6320>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

returns nothing for 'CVE-2024-23328'

from pycvesearch import CVESearch

cve = CVESearch('https://cve.circl.lu')
mycve = cve.id('CVE-2024-23328')
print(mycve)

The above code returns nothing. It shows information on most of the CVEs. However, it does not return any information on some of the CVEs. Could you have a look at where the problem is in the above code or a bug in your repo?

404 Error

I have used the following setup:

from pycvesearch import CVESearch

cve = CVESearch()
cvelist = cve.search("microsoft/office")
print(cvelist)

And the command line shows

<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache Server at cve.circl.lu Port 443</address>
</body></html>

cvefor API does not work

Is there any kind of problem related to cvefor APIs of the pycvesearch Python module? If I forward a request using the cvefor method, the response is blank. Why? Until last week the module worked perfectly.

JSONDecodeError while browsing the vendor

Code

from pycvesearch import CVESearch
cve = CVESearch()
cve.search('microsoft/office')

Complete Error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/terabyte/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pycvesearch/core.py", line 40, in search
    return data.json()
  File "/home/terabyte/.pyenv/versions/3.7.5/lib/python3.7/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/terabyte/.pyenv/versions/3.7.5/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/terabyte/.pyenv/versions/3.7.5/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/terabyte/.pyenv/versions/3.7.5/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I can't import it from python.

Hi!

When I try import from python (from pycvesearch import CVESearch
) I get the following error:

"Unresolved import: CVESearch"

Can you help me please?

Thankss!!!! :)

JC

Unicode error during setup.py install

Hello,

I'm getting the following error when executing:

python3 setup.py install

error:
Traceback (most recent call last): File "setup.py", line 15, in <module> long_description=open('README.md', 'r').read(), File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1315: ordinal not in range(128)

It looks like the offending character is in this author's name found in README.md:

Copyright 2016 Raphaël Vinot

Locally removing that line resolves the installation issue for me

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.