foospidy / grepbugs

A regex based source code scanner.

GrepBugs

A regex based source code scanner.

Usage

python grepbugs.py -d <source directory>
python grepbugs.py -r github -a <account>
python grepbugs.py -r github -a <account> -f

The latest regular expressions will be pulled from https://www.grepbugs.com You can now sign-in at https://grepbugs.com/login to contribute regex rules.

A basic HTML report will be generated in the out/ directory. A tab-delimited file with a subset of the information is also created.

Example reports: https://www.grepbugs.com/reports

Offline Usage

If you need to run grepbugs when there is not Internet connection then you should, before going offline, download the rules file from https://grepbugs.com/rules and save it to GrepBugs/data/grepbugs.json.

Configuration

The etc/grepbugs.cfg file can be used to configure:

• MySQL database (for storing scan results)
• Path to grep binary

Dependencies

• GNU grep (http://www.gnu.org/software/grep/)

  • On Debian run: apt-get install grep
  • On OSX, you will need to install gnu grep (see http://www.heystephenwood.com/2013/09/install-gnu-grep-on-mac-osx.html)
  • On Windows, download the installer package from http://gnuwin32.sourceforge.net/packages/grep.htm

• cloc (http://cloc.sourceforge.net/)

  • On Debian run: apt-get install cloc
  • On OSX run: brew install cloc
  • On Windows, download the binary from http://sourceforge.net/projects/cloc/files/cloc/v1.64/

• git (http://git-scm.com/)

  • Only required if you want to do repo scanning
  • On Debian run: apt-get install git
  • On OSX, configure Xcode command line tools

• svn (https://subversion.apache.org/)

  • Only required if you want to do repo scanning
  • On Debian run: apt-get install subversion
  • On OSX, configure Xcode command line tools

• MySQL support

  • On Debian run: apt-get install python-mysqldb and if this does not work then try one of these:
    • apt-get install libmysqlclient-dev
    • pip install MySQL-python

• requests (http://docs.python-requests.org/en/latest/)

  • On Debian run one of these two options:
    • apt-get install python-requests
    • pip install requests
  • On OSX run: pip install requests
  • On Windows,
    1. Download tarball: https://github.com/kennethreitz/requests/zipball/master
    2. Manually install requests via python setup.py install

Using MySQL Database

Create a database and run the following create statements.

CREATE TABLE `projects` (
  `project_id` varchar(36) NOT NULL,
  `repo` varchar(50) NOT NULL,
  `account` varchar(50) NOT NULL,
  `project` varchar(100) DEFAULT NULL,
  `default_branch` varchar(50) DEFAULT NULL,
  `last_scan` datetime DEFAULT NULL,
  PRIMARY KEY (`project_id`),
  KEY `idx_account` (`account`)
);

CREATE TABLE `results` (
  `result_id` varchar(36) NOT NULL,
  `scan_id` varchar(36) NOT NULL,
  `language` varchar(50) DEFAULT NULL,
  `regex_id` int(11) DEFAULT NULL,
  `regex_text` text,
  `description` text,
  PRIMARY KEY (`result_id`),
  KEY `idx_scan_id` (`scan_id`)
);

CREATE TABLE `results_detail` (
  `result_detail_id` varchar(36) NOT NULL,
  `result_id` varchar(36) NOT NULL,
  `file` text,
  `line` int(11) DEFAULT NULL,
  `code` text,
  PRIMARY KEY (`result_detail_id`),
  KEY `idx_result_id` (`result_id`)
);

CREATE TABLE `scans` (
  `scan_id` varchar(36) NOT NULL,
  `project_id` varchar(36) DEFAULT NULL,
  `date_time` datetime DEFAULT NULL,
  `cloc_out` text,
  PRIMARY KEY (`scan_id`),
  KEY `idx_project_id` (`project_id`)
);

Using on Windows

The Windows instructions are beta (we've done it once!) and we welcome suggestions from users. Install python on Windows and make sure requests is installed too. Install grep and cloc as needed, then modify the configuration file with the full path to the binaries if they are not on the path. We are unsure if you use a single \ or a double one in the PATH or if you can specify drives. Modify the tmpdir setting to a location which exists.

Then, run grepbugs as normal. It should work correctly.

grepbugs's Issues

Need a fully "offline" version of GB

GB does not have a fully offline mode. You need to install the requests library even if you are not using the product on a network. You must somehow make a request to download the rule file. If you have no network access, GB cannot even run.

I suggest a manual bit of instructions for boot-strapping GB. I'm not sure if removing the 'requests' requirement is worth the trouble, but a simple "how to use with zero network access" would be quite handy.

format string attack false positive?

Why would GrepBugs flag this as "Susceptible to format string attacks. "?

fprintf(stderr, "Other helper already running, refusing to start.\n");`

Is it just triggering on the fprintf or something else?

Error when running the script

I am getting the following error:

Error executing cloc sql! Aborting scan! See log file for details.

Error handling for full disk

Need to handle this error better:

Traceback (most recent call last):
File "grepbugs.py", line 855, in
scan_id = repo_scan(args.r, args.a, args.f, args.no_reports)
File "grepbugs.py", line 407, in repo_scan
checkout_code(cmd, checkout_url, account, project_name)
File "grepbugs.py", line 567, in checkout_code
os.makedirs(account_folder)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 28] No space left on device: '/home/grepbugs/opt/GrepBugs/remotesrc/...

SSLV3 HANDSHAKE FAILURE

Logfile:

2015-10-12 14:59:45 INFO Using grep binary grep
2015-10-12 14:59:45 INFO Starting local scan with scan id 1cd8ed0c-70e1-11e5-9e79-10ddb1b3b89f
2015-10-12 14:59:45 INFO Retreiving rules from http://grepbugs.com/json
2015-10-12 14:59:45 INFO Starting new HTTP connection (1): grepbugs.com
2015-10-12 14:59:45 DEBUG "GET /json HTTP/1.1" 301 None
2015-10-12 14:59:45 INFO Starting new HTTPS connection (1): grepbugs.com
2015-10-12 14:59:46 WARNING Error retreiving grep rules: ConnectionError (attempt 1 of 3): [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)
2015-10-12 14:59:49 INFO Starting new HTTP connection (1): grepbugs.com
2015-10-12 14:59:49 DEBUG "GET /json HTTP/1.1" 301 None
2015-10-12 14:59:49 INFO Starting new HTTPS connection (1): grepbugs.com
2015-10-12 14:59:49 WARNING Error retreiving grep rules: ConnectionError (attempt 2 of 3): [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)
2015-10-12 14:59:52 INFO Starting new HTTP connection (1): grepbugs.com
2015-10-12 14:59:52 DEBUG "GET /json HTTP/1.1" 301 None
2015-10-12 14:59:52 INFO Starting new HTTPS connection (1): grepbugs.com
2015-10-12 14:59:52 WARNING Error retreiving grep rules: ConnectionError (attempt 3 of 3): [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)
2015-10-12 14:59:55 DEBUG Error retreiving grep rules (no more tries left. could be using old grep rules.): [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)
2015-10-12 14:59:55 INFO Running cloc for sql output.
2015-10-12 15:01:35 INFO Reading grep rules from json file.
2015-10-12 15:01:35 CRITICAL Unhandled exception: [Errno 2] No such file or directory: '/Users/xxxx/Desktop/grepbugs/data/grepbugs.json'

uname -a

Darwin xxxxxx.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64

other details

OpenSSL 1.0.2d 9 Jul 2015
cloc 1.64
Python 2.7.10