lemonsqueeze / urlgulp Goto Github PK

-------------------------------=:[ urlgulp ]:=-------------------------------


--------------=:[ http traffic overview for fun and privacy ]:=--------------

urlgulp started as a quick hack to keep an eye on network activity from
such moderately trusted entities as browsers. It is basically an http
sniffer with some dns and tcp capabilities added. 

It focuses on giving a quick overview of what's happening, and tries hard not
to flood you with unnecessary information. Used together with full fledged
traffic analysis tools like wireshark it can be quite powerfull: you use
urlgulp to get a big picture of what's happening, before diving in with
wireshark for really detailed stuff. 

It's quite possible to leave it running in the background as network data
is processed on the fly and needs not be saved.

Requests are grouped by host to give a structured outlook. To achieve this
requests are first buffered for the duration of an interval, after which a
summary of contacted hosts and associated requests is shown. Some potential
privacy issues and unwanted traffic are also highlighted:
 - third party domains involved
 - referer leaked to outside domain
 - dns aliases to other domains (trying to hide use of third parties)
 - non http connections, for completeness (host:port is shown)

features include:
 - folding of types of requests (images, css, js) to keep things short
 - pretty printing of urls (coloring based on file types, truncating)
 - summary of dns requests
 - user agent display
 - and other goodies ...


---=:[ limitations ]:=-------------------------------------------------------

It's currently well suited for use on a single computer where traffic can be
kept to a minimum, so that requests from different sources don't get
mixed up together. Support for filtering / monitoring different sources may
come one day.

Keep in mind that content of ssl traffic isn't known, so obviously no urls
there (connection will show up as host:https).

urlgulp uses passive dns resolution to display hostnames for (non http) tcp
connections: it listens to dns traffic, and so doesn't have to make its own
queries. unresolved IPs will show up if it doesn't get a chance to see the
request (happened before it started, dns cache, plain IP is used ...)

And lastly its view of http traffic is based on what urlsnarf can feed it,
so there's stuff it can't know about, like cookies for instance.


---=:[ installation ]:=------------------------------------------------------

linux:
- make sure dependencies are installed:
    perl, tcpdump, urlsnarf (dsniff package)
- run as root.


---=:[ usage ]:=-------------------------------------------------------------

Usage: urlgulp [options] [-i interface]

display mode:
  -k            wait for key press to display (default).
  -f            display every interval seconds.
  -w            display after interval seconds of silence.

other options:
  -d <interval> set display interval for -f and -w (default: 5s).
  -u            show user agent used for each request.
  -l            show legend.
  -q            quiet mode. Don't log urls, only domains.
  -r            raw url mode. just dump it all, no url coloring, grouping or truncating.
  -v            be verbose. (use twice for more).
  -g js,img,css specify the set of url types affected by grouping.
  -g all        shortcut for -g img,js,css.
  -g none       disable grouping.
  --tcp-only    track tcp connections only (no urls)
                will show which IP addresses are actually being contacted,
                useful to check whether a remote proxy is really being used or not.
  --warn-ok     allow grouping of urls with warnings (ref, post)
  --req-order   order hosts as they were first requested.
                (default: hosts in domain first, third party last)
  --version     show version.


---=:[ examples ]:=----------------------------------------------------------

soon.