owtf / ptp

Pentester's Tools Parser (PTP) provides an unified way to retrieve the information from all (final goal) automated pentesting tools and assign an automated ranking for each finding.

What is PTP?

The primary goal of ptp (Pentester's Tools Parser) is to enhance OWASP - OWTF project in order to provide an automated ranking for each plugin. This will allow the user to focus attention on the most likely weak areas of a web application or network first, which will be valuable to efficiently use the remaining time in a penetration assessment.

Instead of evaluating every plugins run by OWASP - OWTF and defining the rankings for each of them, thanks to ptp, the user will be able to focus on the ones that have been ranked with the highest risks. The user is then able to confirm or override the automated rankings since we estimate that she/he is the only one that can accurately detect the false positives.

When developing the automated ranking system, ptp's main goal was joined with a secondary one.

Apart from its main feature which is ranking the results from security tools reports, it also provides an unified way to reuse these reports directly in your python code, without having to deal with complex parsing.

Installation

Using pip

The ptp library is available on PyPI at the following address: https://pypi.python.org/pypi/ptp.

The easiest way to install it is using pip.

$ pip install ptp

Note: If an error occurs during the installation process, check your permissions. It might be required to run pip as root.

From scratch

It is also possible to install the library from its repository. You will then be able to use the latest possible version or even try the develop branch.

The first step is to clone the repository of the project:

$ git clone https://github.com/owtf/ptp.git

Then use the Makefile command:

$ make install

Usage

from __future__ import print_function
from ptp import PTP


if __name__ == '__main__':
    ptp = PTP()
    ptp.parse('path/to/the/report/directory')
    print('Highest severity:', ptp.highest_ranking)

Documentation

The documentation is available online at the following address: https://owtf.github.io/ptp/.

It explains how to use the library and even how to contribute. Plus it contains the technical documentation of the project.

Current support

• arachni (0.4.x to 1.x) (XML and JSON report)
• burpsuite (1.x.x) (XML report)
• dirbuster (1.0-RC1)
• hoppy (1.x.x)
• metasploit
• owasp
• robots.txt
• skipfish (2.10b)
• w3af (1.x.x) (XML report)
• wapiti (2.x.x) (XML report)

Warning: Since v0.4, PTP relies on the fact that the supported tools are following semantic version (except observed otherwise). In other words, as long as the tool doesn't update its MAJOR version, PTP will assume that it can parse its report, reducing the maintenance cost on our side.

ptp's Issues

[metasploit] The snmp_enumshares signature is broken

The snmp_enumshares signature is ' - ' in order to match the enumeration of the shares.

But when the module fails because the connection is refused, this signature also matches:

[*] Initializing modules...
^[[0m^[[0mRHOSTS => 127.0.0.1
^[[0mRPORT => 161
^[[0m^[[1m^[[34m[*]^[[0m Error: 127.0.0.1: Errno::ECONNREFUSED Connection refused - recvfrom(2)
^[[1m^[[34m[*]^[[0m Scanned 1 of 1 hosts (100% complete)
^[[1m^[[34m[*]^[[0m Auxiliary module execution completed
^[[0m

For now, 448187c removes the broken signature (which by the way breaks one unit test case).

New release (adds new http parser, more improvements)

@DePierre time for a new release?

libptp._recursive_find does not properly handle single file in directory

With the following configuration:

./foo
    ./file.txt

The _recursive_find produces the following output: ['f', 'i', 'l', 'e', '.', 't', 'x', t'].

Create unit test with sample report for Wapiti

Would be nice to create at least one tests for Wapiti parser. What I would be looking for is to create a sample report from Wapiti like the other tests (for Arachni for instance).

@DoomTaper do you think that you could kindly take care of this as well? Let me know if you can't, since the release deadline is approaching very quickly.

Use regexes when matching the version numbers

For now, PTP is not advanced enough for that to be a problem. But from a long term point of view, PTP should use regexes when matching version numbers.

For instance, if MyParser is able to parse the reports of a tool from the version 0.1 to 0.9 because the format is stable. Then it would mean that MyParser should have the following __version__ list:

__version__ = ['0.1', '0.2', '0.3', ..., '0.8', '0.9']

This is cucumbersome and would look like the following using a regex:

__version__ = [r"^0\.[1-9]$"]

Which is way better IMO.

Create unit test with sample report for BurpSuite

Would be nice to create at least one tests for BurpSuite newly implemented parser. What I would be looking for is to create a sample report from BurpSuite like the other tests (for Arachni for instance).

@DoomTaper could you take care of this?

Skipfish problems

Hello, when i use the tool i get this error:

Traceback (most recent call last):
File "i.py", line 6, in
ptp.parse('/home/ml/02/propios/skipfish/5874/unix2/')
File "/root/PTP-0.2.0/ptp/ptp.py", line 120, in parse
self.vulns = self.parser.parse_report()
File "/root/PTP-0.2.0/ptp/tools/skipfish/parser.py", line 177, in parse_report
for vuln in ast.literal_eval(report[REPORT_VAR_NAME])]
File "/usr/lib/python2.7/ast.py", line 49, in literal_eval
node_or_string = parse(node_or_string, mode='eval')
File "/usr/lib/python2.7/ast.py", line 37, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "", line 834
{ 'url': 'http://congeladostiendadelsol.com/icons/small/?C=9876sfi
^
SyntaxError: EOL while scanning string literal

im using this source code:

---

from future import print_function
from ptp import PTP

if name == 'main':
ptp = PTP()
ptp.parse('/home/ml/02/propios/skipfish/5874/unix2/')
print('Highest severity:', ptp.get_highest_ranking())

-0------

Tnx!

[wapiti] Parsing the JSON report instead of the XML report

I see that (in Wapiti code) if you choose to generate a HTML report, it will automatically generate a JSON report with it. Maybe PTP can be changed to parse the JSON report rather than the XML report for automatic rankings?
This way we can get the HTML report as well as the automatic rankings.

Arachni not automatically moving to json parsing

Ideally if arachni parser didn't find the xml parser in the report folder then it should automatically move to search for json file. Previously it was working as shown in below screenshot

but now instead of searching for json file it justs exits with an error as shown in screenshot

The unit tests of metasploit are not complete.

Only the scanner ones are written for now.
It could be nice to complete the unit tests in order to cover every metasploit supported modules.

[wapiti] PTP does not correctly parse wapiti XML reports

Due to a misunderstanding of wapiti's XML report, the parsing is meaningless.

For now PTP only checks if the XML report has <vulnerability> tags and then matches the name tag.

But Wapiti's reports always have these tags. PTP should check if the vulnerability sections have <entry> sub-sections that represent the real discoveries.

Imprecise exception display

Although ptp handles precise exceptions but it doesn't show that
e.g.
ptp=PTP('w3af')
ptp.parse('some/random/non-existing/directory')

Below error is shown
Traceback (most recent call last): File "testing.py", line 3, in <module> c = ptp.parse('/root/Desktop/w3af_repo') File "/root/Desktop/ptp/ptp/ptp.py", line 121, in parse raise NotSupportedToolError('This tool is not supported by PTP.') ptp.libptp.exceptions.NotSupportedToolError: This tool is not supported by PTP.

while it should display why the error occurred like above it should show directory doesn't exists or something related to it, helps in correcting error for user itself.

HTTP traffic parser

since it is able to parse rankings so we should extend it to parse http traffic also because supported tools also provides their captured http traffic + we can use that traffic to run grep plugins with OWTF.

Create unit test with sample report for Hoppy

Would be nice to create at least one tests for Hoppy newly implemented parser. What I would be looking for is to create a sample report from Hoppy like the other tests (for Arachni for instance).

@DoomTaper could you take care of this?

Exception thrown while running PTES-001 plugin in OWTF

How to reproduce:
Run PTES-001 in OWTF

Command run by owtf:
cd owtf_review/targets/http__google.com/partial/Ftp/active;/usr/share/metasploit-framework/msfconsole -q -x "use auxiliary/scanner/ftp/anonymous; set RHOSTS 216.58.220.46; set RPORT 21; set FTPUSER \"anonymous\"; set FTPPASS \"[email protected]\"; run; exit"

Error Message: [!] Unexpected exception when running PTP: __init__() got an unexpected keyword argument 'light'

Re-enable Hoppy for next release

Hoppy support has been disabled for v4.0. In order to re-enable it, 880d94a has to be reverted.

But first #20 has to be fixed.

Parsing HTTP requests should link to finding

@DoomTaper has implemented new HTTP parsers (via 713a8fb), where PTP now supports new tool reports that embed the HTTP requests sent to the target.

However, the HTTP requests are being parsed in a bulk-mode. The parser creates a new key transactions in the vulns dictionary returned by ptp.parse() which is not very useful currently since there is no way to find what HTTP request(s) led to the discovery of the finding.

Instead, the parser should for each finding specify the corresponding HTTP requests (as a PoC provided by the tool.

For instance, instead of having an output like:

>>> myptp.parse(pathname='my/other/directory')
[{'ranking': 1}, {'ranking': 2}, {'transactions': [{. . .}, {. . .}, ...]}]

PTP should output:

>>> myptp.parse(pathname='my/other/directory')
[{'ranking': 1, 'transaction': [{. . .}, . . .]}, {'ranking': 2, 'transactions': [{. . .}, . . .]}]

@DoomTaper If you are willing to have a look, feel free.

[arachni] latest versions are not supported because not tested yet.

Didn't have time these past weeks to keep PTP updated for all supported tools.

For sure PTP is late for a couple of subversions of Arachni (https://github.com/Arachni/arachni/releases). This is an easy fix because Arachni doesn't not break its report XML format from a version to the next one.

PTP auto-detection feature broken

As @DoomTaper pointed out, the fix for #14 broke PTP's auto-detection feature:

Python 2.7.12+ (default, Aug  4 2016, 20:04:34) 
[GCC 6.1.1 20160724] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ptp import PTP
>>> a = PTP()
>>> a.parse('/tmp/test/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "ptp/ptp.py", line 119, in parse
    self._init_parser(*args, **kwargs)
  File "ptp/ptp.py", line 94, in _init_parser
    if parser.is_mine(*args, **kwargs):
  File "ptp/tools/wapiti/parser.py", line 52, in is_mine
    stream = cls.handle_file(pathname, filename, first=first)
  File "ptp/libptp/parser.py", line 178, in handle_file
    raise IOError("Report matching '%s' cannot be found." % os.path.join(pathname, filename))
IOError: Report matching '/tmp/test/*.xml' cannot be found.
>>> 

[w3af] Add support to latest versions

PTP currently supports w3af 1.6.0.2 and 1.6.0.3 but the latest version is 1.6.0.5.

• PTP's w3af parser should be checked in order to see if the format has not been changed and therefore if it supports it.
• It would be nice to support all current versions from 1.6 to 1.6.0.5 (need some testing as well).

Update documentation

Current documentation starts being out-of-date, regarding ptp features and the parsers newly implemented. Would need some update to reflect the current development status.

ptp can't parse more than once

After initialising ptp if I run ptp.parse correctly for first time then after that if I run ptp.parse with any arbitrary pathname, it will be a success and will again be parsing taking the pathname initially provided.
Steps to reproduce
ptp = PTP() or ptp = PTP(<tool_name>)

doing parsing with a valid folder such that ptp.parse is a success
vulns = ptp.parse(<pathname>)

Now use ptp.parse with any arbitrary pathname (even invalid) and any number if times without initialising again
vulns2 = ptp.parse('asdads')
vulns3 = ptp.parse('xcvxvxv')

You will see that it's a success.

Problem: This happens because when we do it correct for the first time self.parser is initialised and later when we again use it since self.parser is already initialised self._init_parser doesn't run and hence self.stream is not updated.

How ptp can be more useful by solving this issue?

1. If say I have 10 reports of different tools then I will not initialise ptp with any tool and just do ptp.parse 10 times to get ranking of each report.
2. If I have 10 reports of a single tool say w3af then I will initialise ptp with w3af and will do ptp.parse 10 times.

[arachni] JSON support

It seems that Arachni latest version can generate json reports.
Maybe ptp should support it.