The goal of this project is to allow for accurate detection of HTTPS interception and robust TLS fingerprinting. This project is based off of The Security Impact of HTTPS Interception, and started as a port to Go of their processing scripts and fingerprints.
More context about MITMEngine is available in this Cloudflare blog post. Quick Links:
- Signatures and Fingerprints: Core Definitions
- MITM Detection Methodology
- API
- Example Usage
- Building and Testing
- How to Contribute
- mergeDB utility
- Go
- Wireshark 3.0.0 (
wireshark -vto check)
Detailed documentation lives with the code (copy package to $(GOPATH)/src/github.com/cloudflare/mitmengine first).
godoc -http=:6060
http://localhost:6060/pkg/github.com/cloudflare/mitmengine
In this project, fingerprints map to concrete instantiations of an object, while signatures can represent multiple objects. We use this convention because a fingerprint is usually an inherent property of an object, while a signature can be chosen. In the same way, an actual client request seen by a server would have a fingerprint, while the software generating the request can choose it's own signature (e.g., by choosing which cipher suites it supports).
A client request fingerprint is derived from a client request to a server, and contains both TLS and HTTP features. A client request signature represents all of the possible fingerprints that a piece of software can generate. The aim is to make each signature specific enough that it can uniquely identify a piece of software.
A User Agent signature represent a set of User Agents generated by a browser. A User Agent signature for a browser allows for a range of browser versions, and allows for specifying the OS name, OS platform, OS version range, and device type for creating more fine-grained signatures.
A browser signature contains both a User Agent signature and a client request signature. This allows for a signature to represent all of the possible fingerprints generated by Chrome 31-38 on Windows 10, for example.
A MITM signature contains a client request signature along with additional details about the MITM software, including a security grade which can be affected by factors outside of the client request, such as whether or not the software validates certificates.
We consider an HTTPS connection to be intercepted when there is a mismatch between the expected client request signature corresponding to the browser identified by the User Agent, and the actual client request fingerprint of the request.
If a signature is inaccurate or outdated for a given piece of client software, it is possible that the signature will falsely flag a connection as being intercepted.
If a proxy closely mimics the request of the client, then we may not expect to detect a mismatch. If the browser signatures are overly broad, we will also fail to detect interception.
The reference browser and MITM software fingerprints used in MALCOLM can be found in reference_fingerprints/mitmengine/.
This set of fingerprints is a combination of what is pulled from the TLS Client Hello pcaps in reference_fingerprints/pcaps/, as well as the top 500 User Agents + TLS Client Hello
pairs observed on Cloudflare's network and labeled with a high trustworthiness rating (that is, traffic corresponding to human and friendly bot activity).
Ideally, we don't have to rely on reference fingerprints sampled from Cloudflare's network; instead, we would have a comprehensive set of pcaps to build our set of reference TLS Client Hellos. Interested in helping us build out our dataset? See how you can contribute!
First, a user must create a mitmengine.Config struct to pass into mitmengine.NewProcessor. A mitmengine.Config
struct can specify filenames of files containing browser fingerprints, MITM fingerprints, and MITM
headers. Alternatively, it can also specify a configuration file for reading the previously mentioned files from any
other source; right now, MITMEngine supports reading these files from Amazon S3 client-compatible databases (including
Amazon S3 and Ceph). Additional file readers for databases (which we call "loaders") can be defined in the loaders
package, and as long as new loaders implement the Loader interface, they should work with the rest of MITMEngine out of the
box.
The intended entrypoint to the MITMEngine package is through the Processor.Check function, which takes a User Agent and client request fingerprint, and returns a mitm detection report. Additional API functions will be added in the future to allow for adding new signatures to a running process, for example.
An example use of the API is below. A more complete application is available at cmd/demo/main.go, and can be built by running make bin/demo.
rawUa := "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"
requestFingerprintString := "303:dada,1301,1302,1303,c02b,c02f,c02c,c030,cca9,cca8,c013,c014,9c,9d,2f,35,a:aaaa,0,17,ff01,a,b,23,10,5,d,12,33,2d,2b,1b,dada,15:9a9a,1d,17,18:0::"
uaFingerprintString := "1:72.0.3626:2:3:10.14.3:1:"
requestFingerprint, _ := fp.NewRequestFingerprint(requestFingerprintString)
uaFingerprint, _ := fp.NewUAFingerprint(uaFingerprintString)
report := mitmProcessor.Check(uaFingerprint, rawUa, requestFingerprint)
The TLS requestFingerprintString has the following format:
<tls_version>:<cipher_suites>:<extension_names>:<curves>:<ec_point_fmts>:<http_headers>:<quirks>
The uaFingerprint has the following format:
# <browser_name>:<browser_version>:<os_platform>:<os_name>:<os_version>:<device_type>:<quirks>
An example of how to parse User Agents into the format for uaFingerprint is in the cmd/demo/main.go file.
To use MITMEngine, remember to pull in its dependencies. You'll likely want to run vendoring or gomod logic before running tests on MITMEngine.
To test, run make test and to see code coverage, run make cover.
As browser and mitm fingerprints quickly become outdated, we are actively seeking to update the fingerprint repository with new samples. To contribute fingerprint samples, please follow these steps:
(tested on macOS Mojave 10.14.3)
Create server RSA certificate and key pair:
openssl req -new -x509 -sha256 -out server.crt -nodes -keyout server.pem -subj /CN=localhost
Start server on port 4433:
openssl s_server -www -cipher AES256-SHA -key server.pem -cert server.crt
Start TShark capture to decrypt HTTP headers (TShark >= 3.0.0):
tshark -i loopback -o tls.keys_list:"127.0.0.1,4433,http,server.pem" -Tjson -e http.request.line -Y http > header.json
Start TShark capture of TLS Client Hello:
tshark -i loopback -f "tcp port 4433" -w handshake.pcap
Visit https://localhost:4433 from the TLS client you wish to fingerprint. For example,
echo -e "GET /test HTTP/1.1\r\nHost:example.com\r\n\r\n" | openssl s_client -connect localhost:4433
-
Generate a fingerprint sample (
header.json,handshake.pcap) as described above, and place in the directoryreference_fingerprints/pcaps/<desc>, where<desc>is a unique and descriptive name. -
Add a line to
reference_fingerprints/fingerprint_metadata.jsonlwith the below fields. Recognized options for theos,device,platform, andbrowserfields are those defined in theuasurferpackage. Recognized options formitm_fingerprint.typeare listed below. Seereference_fingerprints/fingerprint_metadata.jsonlfor examples; any unknown fields can be left blank or omitted.{ "desc": "", "comment": "", "handshake_pcap": "", "header_json": "<(optional) path to file containing the client HTTP request", "ua_fingerprint": {"raw_ua": "", "os": "<WindowsPhone|Windows|MacOSX|iOS|Android|...>", "os_version": "..", "device": "<Windows|Mac|Linux|...>", "platform": "<Computer|Tablet|Phone|...>", "browser": "<Chrome|IE|Safari|Firefox|...>", "browser_version": ".."}, "mitm_fingerprint": { "name": "", "type": "<Antivirus|FakeBrowser|Malware|Parental|Proxy>" }}
-
Submit a pull request with above changes.
Other PRs and feature requests are welcome!
mergeDB (in cmd/mergedb) is a utility for merging similar TLS ClientHello fingerprints across multiple close versions of browsers or MITM software.
Use mergeDB to consolidate large lists of User Agent / Client Hello fingerprints:
go run cmd/mergedb/main.go
By default, mergeDB will run on the fingerprints in the reference_fingerprints/mitmengine directory.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent