ymv / warcmitmproxy

HTTP(S) proxy that saves traffic to a WARC file, using libmitmproxy.

WarcMITMProxy

WarcMITMProxy is an HTTP(S) proxy that saves all web traffic to a file. The file format used is the Web ARChive (WARC) format (ISO 28500). WarcMITMProxy uses mitmproxy. This lets you browse the web using a regular browser, and anything you view in the browser is archived in the WARC file.

Prerequisites

• mitmproxy: Won't work with latest version, call pip install -r frozen to install correct version

Usage

To use WarcMITMProxy, open a web browser and change the proxy settings to connect to 127.0.0.1:8000. Then from a console, run python warcmitmproxy.py. Any browsing activity is passed through the proxy and saved into a WARC file. I recommend browsing with QupZilla, which is a portable, lightweight browser based WebKit.

SSL Certificate

WarcMITMProxy requires an SSL certificate support reading HTTPS sessions. Although the program comes with a pre-generated certificate that can be imported into Windows, the certificate is publicly available, so a personal certificate should be generated by following the steps below.

Generating an SSL Certificate on Windows 7

Install OpenSSL, then use the following commands. They will create four files: ca.key, ca.crt, ca.pem, and ca.p12.

cd C:\OpenSSL-Win32\bin
set OPENSSL_CONF=C:\OpenSSL-Win32\bin\openssl.cfg
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -out ca.crt
openssl pkcs12 -export -out cert.p12 -inkey ca.key -in ca.crt
copy /B ca.key+ca.crt ca.pem

Using an SSL Certificate

After creating the files, the certificate must be imported into the Windows Certificate manager. Click start and type certmgr.msc and open the application. Right-click the Trusted Root Certification Authorities folder and go to All Tasks > Import. Click Next and then Browse for a certificate. In the filter in the bottom right, change it to "Personal Information Exchange (*.pfx, *.p12)" and import cert.p12.

To use the certificate with WarcMITMProxy, copy cert.pem into WarcMITMProxy's folder.