cosimo / vlogdump

Small tool to filter/convert/match raw varnishlog output

NAME

vlogdump

DESCRIPTION

Tool to parse varnishlog output and keep track of transactions. Match those that we want to debug and show detailed information about them.

This tool is meant to help when setting up or troubleshooting a varnish configuration.

It's confirmed working with Varnish 2.x, 3.x, 4.x and 5.x. For some versions, it might be necessary to use a specific development branch instead of master.

EXAMPLES

Here's some examples of usage.

Show all requests that are cache misses

$ varnishlog | vlogdump -v only_misses=1

172.22.0.15 => GET /assets/tn/m/mq/e85ed0a7b1b87120a0a2bfa025531c6733a48802 HTTP/1.0 MISS
            <= 200 OK                                                          28.432 ms
172.22.0.18 => GET /assets/tn/m/mq/5a9e9440c5c85e8dc5d65e03e15c95e390901fa7 HTTP/1.0 MISS
            <= 200 OK                                                          36.905 ms
172.22.0.18 => GET /assets//icons/categories/te/icon32x32-technology.png HTTP/1.0 MISS
            <= 304 Not Modified                                                 0.589 ms
172.22.0.15 => GET /api/fetch/article-preview/?client=2&language=en-GB HTTP/1.1 MISS
            <= 301 MOVED PERMANENTLY                                            8.381 ms
172.22.0.18 => GET /assets/tn/m/mq/c3830e95b717761005e26ce49ebab253e0ccb40b HTTP/1.0 MISS
            <= 200 OK                                                         291.354 ms
172.22.0.18 => GET /api/1.0/category/list?client=2&language=en-GB HTTP/1.1 MISS
            <= 200 OK                                                          58.025 ms
...

Show only requests that are cache hits

Same as above, but -v only_hits=1.

Show only requests (not) for certain URLs

$ varnishlog | vlogdump -v url_match=assets

To negate the pattern, prepend a ! character before the regular expression. You will get only requests for URLs that DO NOT match assets:

$ varnishlog | vlogdump -v url_match=!assets

For each transaction, show the request headers

$ varnishlog | vlogdump -v show_req_headers=1

172.22.0.15 => GET /assets/3a626ed2f029fc5802c244673c6e168914568080 HTTP/1.0 MISS
            <= 200 OK                                                          51.483 ms
    req.http.Host = discovery.opera.com
    req.http.User-Agent = Amazon CloudFront
    req.http.X-Forwarded-For = 111.12.34.56
    req.http.X-Amz-Cf-Id = NS3KqU_AKnueKyQcJWQXbLyVAYR7jvPJ5QGl9raDg0AZZaPkt87avA==
    req.http.Connection = keep-alive
    ...

Only show transactions where the request headers match X

Where X can be a string or a regular expression:

$ varnishlog | vlogdump -v show_req_headers=1 -v req_headers_match=X

...

and there's more. Check the full list of options below.

OPTIONS

This being an awk script, all options need to be supplied in the form -v <option>=<value>.

client_match=<regexp>

Shows only transactions from clients (IP addresses) matching a given regular expression.

show_req_headers=0|1

Turn on or off the output of HTTP request headers

show_resp_headers=0|1

Turn on or off the output of HTTP response headers

req_headers_match=<regexp>

Shows only transactions with request headers (any header) matching the given regular expression

Prepend '!' to the pattern to negate the meaning, to display only transactions that *do not* match the pattern.

resp_headers_match=<regexp>

Shows only transactions with response headers (any header) matching the given regular expression

Prepend '!' to the pattern to negate the meaning, to display only transactions that *do not* match the pattern.

req_method=<regexp>

Shows only transactions with the given HTTP request method, or a method that matches the given regexp.

Prepend '!' to the pattern to negate the meaning, to display only transactions that *do not* match the pattern.

only_hits=0|1

Shows only transactions that were served from the cache

only_misses=0|1

Shows only transactions that were served by going to the backend server.

only_slow=1

only_slow=<ms>

Shows only requests that are considered slow. The time used is the backend response time. If you use only_slow=1, the default threshold is 1s. You can also specify your own threshold, using for example only_slow=500. Value is in milliseconds.

url_match=<regexp>

Show only the transactions for URLs matching the given regular expression.

Prepend '!' to the pattern to negate the meaning, to display only transactions where the URL *does not* match the given regular expression.

Example:

varnishlog | vlogdump -v url_match='!/assets'

AUTHOR

Cosimo Streppone, [email protected]

vlogdump's Issues

Add to the Utility directory

Hi Cosimo,

I think this script might be useful for quite a few users. Feel free to add it to the Utility Directory:
http://varnish.org/utilities

And if you have any other VMODs or utilities (I bet you do) make sure you share them on the varnish.org site as well :-)

Best, Rubén.

vlogdump doesn't understand varnish 3.0 vlog call+return

In Varnish 2.x, the VCL_call and VCL_return were always on different log lines.

   29 VCL_call     c recv
   29 VCL_return   c lookup
   29 VCL_call     c hash
   29 VCL_return   c hash
   29 Hit          c 1008399193
   29 VCL_call     c hit
   29 VCL_return   c deliver
   29 VCL_call     c deliver
   29 VCL_return   c deliver
   29 TxProtocol   c HTTP/1.1
   29 TxStatus     c 200
   29 TxResponse   c OK

In Varnish 3.x, it's possible to receive logs that look like these:

   32 VCL_call    c   recv lookup
   ...
   32 VCL_call    c   hit deliver
   32 VCL_call    c   deliver deliver

vlogdump currently doesn't understand this.

Use with v4 of Varnish

Hi,

The varnishlog of Varnish v4 no longer operates in the same manner.
An update is planned ?

Best regards !

VlogDump stating a MISS when it is a HIT

This tool shows MISS in front of the request, even if I asked only for HITS ?? And request is a HIT too....

what does this MISS signify ??

$ varnishlog | ./vlogdump -v only_hits=1

127.0.0.1 => GET /public/stylesheets/main.css HTTP/1.1 MISS
<= 304 Not Modified 0.141 ms
127.0.0.1 => GET /public/javascripts/jquery-1.6.4.min.js HTTP/1.1 MISS
<= 304 Not Modified 0.157 ms

Log dump can be viewed at
https://docs.google.com/document/d/1Go_LLmBG94AieirKgnWWMmQcjrONiz5qM_JHbcHmGM4/edit?usp=sharing