go-logfmt / logfmt Goto Github PK

I was wondering why the string "null" gets quoted. I would have expected it to be rendered as k=null.

I'm using go-kit/log and need to redact log entries, i.e. remove usernames, passwords etc. For this purpose I'd like to pass the keyVals through a redact function. Since loggers are stacked, the removal must happen at the time log data is produced, not when it's written (i.e. can't do what in the io.Writer). Since keyVals can be any interface, it would be great to reuse logfmts encoding to []byte/string before passing the value to redaction.

It handles a panic arising from attempting to print a null string, but if the stringer panics for some other reason it will re-throw the panic. For my purposes it would be better to note the panic in the log message and carry on.

I guess tastes vary, but it surprised me that there is a panic handler but only for one specific case.

Have the logger produce a record even if the key does not follow rules so at least the value is logged lost. Some possible solutions:

1. change the key to "invalid_key" or something similar
2. remove all invalid characters from key so "the key" becomes "thekey"
3. replace all invalid characters
   with underscores so "the key" become "the_key"
   with hex values with surrounding underscores so "the:key" becomes "the_3A_key"
   or include an x "the_x3A_key".

At the moment this package does not support logging values of type array, chan, func, map, slice, or struct. Of these, I cannot think of a reason to support chan or func. But I think we need a good story for array, map, slice, and struct.

Precedents

Brandur Leach's logfmt blog post does not explicitly explain how to represent compound types in logfmt, nor does the defacto logfmt definition at github.com/kr/logfmt. There are some ideas in an old Gist by Kieth Rarick. The Gist shows this JSON data:

{"dyno":"web", "metadata":{"stack":["line 1", "line 2"], "msg":"null"}}

converted to logfmt as follows.

dyno=web metadata={stack=["line 1" "line 2"] msg="null"}

But the parser at github.com/kr/logfmt does not handle this syntax.

Logrus relies on Go's %v format in the fmt package and writes:

dyno=web metadata=map[stack:[line 1 line 2] msg:null]

log15 uses Go's %+v format and writes:

dyno=web metadata="map[msg:null stack:[line 1 line 2]]"

Neither of these logging packages produce output that can be easily parsed back into the original data structure.

I have not investigated what the non-Go logfmt libraries do.

Ways Forward

Given that there is not a clear best practice for encoding compound data structures in logfmt, this package must either refuse to handle them (as it does now) or define a robust encoding for these data types.

@kr, @bmizerany, @inconshreveable, @sirupsen: I would appreciate it if any of you would like to comment on this issue.

either I'm misunderstanding the use of logfmt.MarshalKeyvals, or it doesn't work as expected. I'm doing the following:

keyvals := []string{"app", "myapp", "env", "prod", "error", "errormsg"}
fmt.Println("keyvals:", keyvals)
out, err := logfmt.MarshalKeyvals(keyvals)
fmt.Println(string(out), err)

but instead of getting formatted output, I get:

keyvals: [app myapp env prod error errormsg]
 <nil>

so no output, and no error.

this is go version 1.14.5.

Error: bufio.Scanner: token too long occurs when scanning large data.

Fix is to set token size to maximum buffer size:
#10