cosimo / vlogdump

Small tool to filter/convert/match raw varnishlog output

NAME

vlogdump

DESCRIPTION

Tool to parse varnishlog output and keep track of transactions. Match those that we want to debug and show detailed information about them.

This tool is meant to help when setting up or troubleshooting a varnish configuration.

It's confirmed working with Varnish 2.x, 3.x, 4.x and 5.x. For some versions, it might be necessary to use a specific development branch instead of master.

EXAMPLES

Here's some examples of usage.

Show all requests that are cache misses

$ varnishlog | vlogdump -v only_misses=1

172.22.0.15 => GET /assets/tn/m/mq/e85ed0a7b1b87120a0a2bfa025531c6733a48802 HTTP/1.0 MISS
            <= 200 OK                                                          28.432 ms
172.22.0.18 => GET /assets/tn/m/mq/5a9e9440c5c85e8dc5d65e03e15c95e390901fa7 HTTP/1.0 MISS
            <= 200 OK                                                          36.905 ms
172.22.0.18 => GET /assets//icons/categories/te/icon32x32-technology.png HTTP/1.0 MISS
            <= 304 Not Modified                                                 0.589 ms
172.22.0.15 => GET /api/fetch/article-preview/?client=2&language=en-GB HTTP/1.1 MISS
            <= 301 MOVED PERMANENTLY                                            8.381 ms
172.22.0.18 => GET /assets/tn/m/mq/c3830e95b717761005e26ce49ebab253e0ccb40b HTTP/1.0 MISS
            <= 200 OK                                                         291.354 ms
172.22.0.18 => GET /api/1.0/category/list?client=2&language=en-GB HTTP/1.1 MISS
            <= 200 OK                                                          58.025 ms
...

Show only requests that are cache hits

Same as above, but -v only_hits=1.

Show only requests (not) for certain URLs

$ varnishlog | vlogdump -v url_match=assets

To negate the pattern, prepend a ! character before the regular expression. You will get only requests for URLs that DO NOT match assets:

$ varnishlog | vlogdump -v url_match=!assets

For each transaction, show the request headers

$ varnishlog | vlogdump -v show_req_headers=1

172.22.0.15 => GET /assets/3a626ed2f029fc5802c244673c6e168914568080 HTTP/1.0 MISS
            <= 200 OK                                                          51.483 ms
    req.http.Host = discovery.opera.com
    req.http.User-Agent = Amazon CloudFront
    req.http.X-Forwarded-For = 111.12.34.56
    req.http.X-Amz-Cf-Id = NS3KqU_AKnueKyQcJWQXbLyVAYR7jvPJ5QGl9raDg0AZZaPkt87avA==
    req.http.Connection = keep-alive
    ...

Only show transactions where the request headers match X

Where X can be a string or a regular expression:

$ varnishlog | vlogdump -v show_req_headers=1 -v req_headers_match=X

...

and there's more. Check the full list of options below.

OPTIONS

This being an awk script, all options need to be supplied in the form -v <option>=<value>.

client_match=<regexp>

Shows only transactions from clients (IP addresses) matching a given regular expression.

show_req_headers=0|1

Turn on or off the output of HTTP request headers

show_resp_headers=0|1

Turn on or off the output of HTTP response headers

req_headers_match=<regexp>

Shows only transactions with request headers (any header) matching the given regular expression

Prepend '!' to the pattern to negate the meaning, to display only transactions that *do not* match the pattern.

resp_headers_match=<regexp>

Shows only transactions with response headers (any header) matching the given regular expression

Prepend '!' to the pattern to negate the meaning, to display only transactions that *do not* match the pattern.

req_method=<regexp>

Shows only transactions with the given HTTP request method, or a method that matches the given regexp.

Prepend '!' to the pattern to negate the meaning, to display only transactions that *do not* match the pattern.

only_hits=0|1

Shows only transactions that were served from the cache

only_misses=0|1

Shows only transactions that were served by going to the backend server.

only_slow=1

only_slow=<ms>

Shows only requests that are considered slow. The time used is the backend response time. If you use only_slow=1, the default threshold is 1s. You can also specify your own threshold, using for example only_slow=500. Value is in milliseconds.

url_match=<regexp>

Show only the transactions for URLs matching the given regular expression.

Prepend '!' to the pattern to negate the meaning, to display only transactions where the URL *does not* match the given regular expression.

Example:

varnishlog | vlogdump -v url_match='!/assets'

AUTHOR

Cosimo Streppone, [email protected]