isabella232 / hutils

A collection of command line utilities for working with logfmt.

hutils

A small collection of utilies for logfmt processing.

Installation

gem install hutils

Utilities

lcut

lcut extracts values from a logfmt trace based on some field name.

$ ltap 'instrumentation app=api earliest=-1m at=finish' | lcut method path
GET     /providers/users/search
GET     /vendor/resources/6307854
GET     /health
GET     /vendor/resources/6007506
GET     /vendor/resources/7117492

lfmt

lfmt prettifies logfmt lines as they emerge from a stream, and highlights their key sections.

(Note that the example below doesn't demonstrate color, which is one of the more important features of logfmt.)

$ ltap 'instrumentation app=api earliest=-1m at=finish' | lfmt
[email protected] app: api at: finish component: manager_apiauthorized elapsed: 0.008 instance_name: api.108081 instrumentation length: 339 method: GET path: /providers/users/search request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513 status: 200 user: [email protected] user_id: 105750 version: 1
[email protected] app: api at: finish elapsed: 0.001 instance_name: api.136540 instrumentation method: GET path: /vendor/resources/6307854 request_id: 055df716-fc62-4554-b976-e2fe2472e107 status: 200 user: [email protected] user_id: 97546 version: 2
[email protected] app: api at: finish elapsed: 0.000 instance_name: api.93579 instrumentation method: GET path: /health request_id: 6af07088-82af-4f50-87c1-8b5d248807f0 status: 200 user: [email protected] user_id: 105750 version: 1
[email protected] app: api at: finish elapsed: 0.174 instance_name: api.108081 instrumentation method: GET path: /vendor/resources/6007506 request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513 status: 200 user: [email protected] user_id: 105750 version: 1
[email protected] app: api at: finish elapsed: 0.162 instance_name: api.108081 instrumentation method: GET path: /vendor/resources/7117492 request_id: 7480424d-5a8a-488a-a32a-55812fde5f4b status: 200 user: [email protected] user_id: 105750 version: 1

ltap

ltap accesses messages from popular log providers in a consistent way so that it can easily be parsed by other utilities that operate on logfmt traces. Currently supported providers are Papertrail and Splunk.

$ ltap 'instrumentation app=api earliest=-1m at=finish'
[email protected] instrumentation method=GET path=/providers/users/search request_id=d5c373fd-d1ec-4986-bc43-2617431116f2 at=finish elapsed=0.008 length=339 status=200 app=api instance_name=api.108081 version=1 component=manager_apiauthorized app=api instance_name=api.108081 request_id=ef82825d-4c10-41f3-89ed-6bf805aa4513 version=1 [email protected] user_id=105750
[email protected] instrumentation method=GET path=/vendor/resources/6307854 request_id=d2f25032-9aaa-41e9-8aaf-9a46a44523d1 at=finish elapsed=0.110 status=200 app=api instance_name=api.136540 version=1 [email protected] user_id=105750step=check_oauth_scope! request_id=055df716-fc62-4554-b976-e2fe2472e107 version=2 [email protected] user_id=97546 app=api instance_name=api.136540 at=finish elapsed=0.001
[email protected] instrumentation method=GET path=/health request_id=6af07088-82af-4f50-87c1-8b5d248807f0 at=finish elapsed=0.000 status=200 app=api instance_name=api.93579 version=1 [email protected] user_id=105750
[email protected] instrumentation method=GET path=/vendor/resources/6007506 request_id=ef82825d-4c10-41f3-89ed-6bf805aa4513 at=finish elapsed=0.174 status=200 app=api instance_name=api.108081 version=1 [email protected] user_id=105750
[email protected] instrumentation method=GET path=/vendor/resources/7117492 request_id=7480424d-5a8a-488a-a32a-55812fde5f4b at=finish elapsed=0.162 status=200 app=api instance_name=api.108081 version=1 [email protected] user_id=105750

ltap can be configured using ~/.ltap. A sample Papertrail configuration looks like the following:

[global]
profile = my_papertrail

[my_papertrail]
key = an-api-key
type = papertrail

A sample Splunk configuration:

[global]
profile = my_splunk

[my_splunk]
earliest = -24h
type = splunk
url = https://brandur:[email protected]:8089

lviz

lviz helps to visualize logfmt output by building a tree out of some set of data by combining common sets of key/value pairs into shared parent nodes. Messages remain ordered by time, which removes some potential for commonality, but in many cases a disproportionate number of attributes can be moved up to nodes close to the top of the tree. Output is colorized and important keys are highlighted to make traces more easily digestible.

$ ltap 'instrumentation app=api earliest=-1m at=finish' | lviz
+ app: api
  instrumentation
  method: GET
  status: 200

        + [email protected]
          component: manager_apiauthorized
          elapsed: 0.008
          instance_name: api.108081
          length: 339
          path: /providers/users/search
          request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513
          user: [email protected]
          user_id: 105750
          version: 1

        + [email protected]
          elapsed: 0.001
          instance_name: api.136540
          path: /vendor/resources/6307854
          request_id: 055df716-fc62-4554-b976-e2fe2472e107
          user: [email protected]
          user_id: 97546
          version: 2

        + user: [email protected]
          user_id: 105750
          version: 1

                + [email protected]
                  elapsed: 0.000
                  instance_name: api.93579
                  path: /health
                  request_id: 6af07088-82af-4f50-87c1-8b5d248807f0

                + [email protected]
                  instance_name: api.108081

                        + elapsed: 0.174
                          path: /vendor/resources/6007506
                          request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513

                        + elapsed: 0.162
                          path: /vendor/resources/7117492
                          request_id: 7480424d-5a8a-488a-a32a-55812fde5f4b

lviz can be configured with ~/.lviz. For example:

[global]
highlights = path,user
ignore = at

lviz can also produce a compact mode of output using -c or --compact.

Testing

bundle install
bundle exec rake