pombredanne / swailing

An opinionated logging library for applications that produce a debilitating amount of log output.

Swailing

An opinionated logging library for applications that produce a debilitating amount of log output.

Swailing uses the token bucket algorithm and provides facilities for PostgreSQL style error logs.

Token Bucket

The token bucket algorithm is a simple mechanism for throttling burstable events. There are two parameters: fill rate and capacity. The fill rate is the number of tokens that a bucket accumulates per second. The capacity is the largest number of tokens that the bucket holds. Each time your application emits log output, Swailing consumes one of the tokens in the bucket. Thus, we allow outputting a steady rate of messages less than or equal to the fill rate, with the capability of bursting up to the capacity.

PostgreSQL Style Error Logs

In addition to plain log lines (called primary), Swailing allows the program to output detail and hint lines as well. See the PostgreSQL guide for what the semantic differences are.

Usage

logger = swailing.Logger(logging.getLogger())

with logger.info() as L:
    L.primary('this is the primary message: %d', 12345)
    L.detail('this is an optional detail: %s failed', 'syscall')
    L.hint('also an optional hint to do yadda yadda')

That outputs to the supplied logger:

this is the primary message: 12345
DETAIL: this is an optional detail: syscall failed
HINT: also an optional hint to do yadda yadda

But you can always fallback to regular style logging:

logger.info('this is a regular log call %d', 12345)

You can silence details and hints at runtime by setting the verbosity:

logger.set_verbosity(swailing.PRIMARY)

with logger.info() as L:
    L.primary('this is the primary message: %d', 12345)
    L.detail('this is an optional detail: %s failed', 'syscall')
    L.hint('also an optional hint to do yadda yadda')

Outputs:

this is the primary message: 12345

You can also disable the "DETAIL: ..." and "HINT: ..." prefix, e.g.

logger = swailing.Logger(logging.getLogger(), with_prefix=False)

It might make sense to pass in a dictionary of information to .detail and have the logger json dump the dict. In that case you can do:

logger = swailing.Logger(logging.getLogger(), structured_detail=True)

with logger.info() as L:
	L.primary('this is the primary message %d', 12345)
	L.detail({'request_id': 12345, 'request_ts': 1454026366})
	L.hint('some hinty stuff')

Outputs:

this is the primary message: 12345
DETAIL: {"request_ts": 1454026366, "request_id": 12345}
HINT: some hinty stuff

Throttling:

logger = swailing.Logger(logging.getLogger(), fill_rate=100, capacity=1000)

# A burst of 1000+ calls, exceeding 100 / second...
# Then a pause of at least 1/100 second...

logger.info('then what happens?')

Outputs the first 1000 calls, and then:

(...throttled 2342 log lines...)

then what happens?

Authors

• Wes Chow
• Adrian Kramer