go-chi / httplog Goto Github PK

Hello, everyone!
I noticed that there's a limitation when logging the response body. In my use case, response body has the error stack trace and, due to its size, many characters are not being logged, making it difficult to properly understand error root cause. Would it be possible to increase the character limitation?

Currently, the response body will only be logged if the returned status code is >= 400.

https://github.com/go-chi/httplog/blob/master/httplog.go#L90C4-L96C7.
https://github.com/go-chi/httplog/blob/master/httplog.go#L146C3-L149C4.

I have a use case for httplog to act as an audit capability that logs all requests and responses to/from the system. With the current implementation, the response body is only "auditable" if the server processes the request with an error.

My suggestion is to include a new configurable option that would be used for the condition rather than status >= 400. The option can default to the existing logic but allow consumers to define if they want response bodies to be logged in different situations.

if l.Options.ResponseBody {
	body, _ := extra.([]byte)
	responseLog = append(responseLog, slog.Attr{Key: "body", Value: slog.StringValue(string(body))})
}

Health checks can be pretty noisey for logs and just clog up the log backend without much value. We could completely silence certain paths, but I think that is a bit too aggressive. Instead, lets add QuietDownRoutes: []string which takes a list of paths, like []string{"/", "/ping", "/status"} and for these routes, the logging will only sample/emit a log every 5 minutes. We can also add another option QuietDownPeriod with the default of time.Duration of 5 * time.Minute. It should work, so on boot, the first log will always render, then won't log until the 5 min period is over, etc.

Code:

package main

import (
	"net/http"

	"github.com/go-chi/chi/v5"
	"github.com/go-chi/httplog"
	"github.com/rs/zerolog/log"
)

func main() {
	r := chi.NewRouter()
	r.Use(httplog.RequestLogger(log.With().Str("service", "http").Logger()))
	r.Get("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte(`oi`))
	})
	log.Info().Msg("Running on :3000")
	http.ListenAndServe(":3000", r)
}

Log when I request GET / using xh :3000:

❯ go run ./main.go 
{"level":"info","time":"2022-01-10T08:43:25-03:00","message":"Running on :3000"}
{"level":"info","service":"http","httpRequest":{"proto":"HTTP/1.1","remoteIP":"[::1]:59744","requestID":"dell-rrsp/SaPMv1pZaJ-000001","requestMethod":"GET","requestPath":"/","requestURL":"http://localhost:3000/"},"httpRequest":{"header":{"accept":"*/*","accept-encoding":"gzip, deflate, br","connection":"keep-alive","user-agent":"xh/0.14.1"},"proto":"HTTP/1.1","remoteIP":"[::1]:59744","requestID":"dell-rrsp/SaPMv1pZaJ-000001","requestMethod":"GET","requestPath":"/","requestURL":"http://localhost:3000/","scheme":"http"},"time":"2022-01-10T08:43:28-03:00","message":"Request: GET /"}
{"level":"info","service":"http","httpRequest":{"proto":"HTTP/1.1","remoteIP":"[::1]:59744","requestID":"dell-rrsp/SaPMv1pZaJ-000001","requestMethod":"GET","requestPath":"/","requestURL":"http://localhost:3000/"},"httpResponse":{"bytes":2,"elapsed":0.011784,"status":200},"time":"2022-01-10T08:43:28-03:00","message":"Response: 200 OK"}

The go.mod file

module bug

go 1.17

require (
	github.com/go-chi/chi/v5 v5.0.7
	github.com/go-chi/httplog v0.2.1
	github.com/rs/zerolog v1.26.1
)

httplog.LogEntry should return *slog.Logger instead of slog.Logger because then, the returned value implements interfaces for the slog API. Reason is that slog’s Logger methods need pointer receivers.

Note that *slog.Logger is used in slog’s own code all over the place, and even httplog’s LogEntry needs to de-reference in order to return a plain slog.Logger.

When an error with level panic occurs, it does not print the stack trace. It still appears in the terminal output, but is not written in the file.

{"timestamp":"2023-12-09T13:56:15.436815815+02:00","level":"ERROR","message":"Response: 500 Server Error - interface conversion: error is *fmt.wrapError, not *app_error.ApplicationError","service":"app-logger","httpRequest":{"url":"http://localhost:8000/api/v1/hotels","method":"GET","path":"/api/v1/hotels","remoteIP":"[::1]:50340","proto":"HTTP/1.1","requestID":"stepan-ThinkPad-L15-Gen-2/f8pOxWi02A-000002"},"stacktrace":"#","panic":"interface conversion: error is *fmt.wrapError, not *app_error.ApplicationError","httpResponse":{"status":500,"bytes":0,"elapsed":1.042674,"body":""}}

const YYYYMMDD = "2006-01-02"

func newLogger() *httplog.Logger {
	return httplog.NewLogger("app-logger", httplog.Options{
		JSON:             true,
		LogLevel:         slog.LevelError,
		Concise:          true,
		MessageFieldName: "message",
		Tags: map[string]string{
			"version": "v1.0-81aa4244d9fc8076a",
			"env":     "dev",
		},
		QuietDownRoutes: []string{
			"/",
			"/ping",
		},
		QuietDownPeriod: 10 * time.Second,
		Writer:          newWriter(),
	})
}

func newWriter() io.Writer {
	fileNamePath := fmt.Sprintf("logs/%s.log", time.Now().UTC().Format(YYYYMMDD))
	errorLogFile, err := os.OpenFile(fileNamePath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
	if err != nil {
		log.Fatal("Error opening file: ", err)
	}

	return io.MultiWriter(os.Stdout, errorLogFile)
}

In httplog.go file there is this function:

func (l *RequestLoggerEntry) Panic(v interface{}, stack []byte) {
	stacktrace := "#"
	if l.Options.JSON {
		stacktrace = string(stack)
	}
	l.Logger = *l.Logger.With(
		slog.Attr{
			Key:   "stacktrace",
			Value: slog.StringValue(stacktrace)},
		slog.Attr{
			Key:   "panic",
			Value: slog.StringValue(fmt.Sprintf("%+v", v)),
		})

	l.msg = fmt.Sprintf("%+v", v)

	if !l.Options.JSON {
		middleware.PrintPrettyStack(v)
	}
}

I examined a bit and it seems that the l.Options are not the same that we passed to httplog.NewLogger function. They come from RequestLoggerEntry struct, not from Logger struct.

Is there a way to enable it that I don't know?

Hi people,

So this is a feature request, and I'd be happy to submit a PR for it if it's deemed ok. In my use case I'd like to exclude some endpoints from being logged, e.g. the "healthz" and "readyz" endpoints used by k8, which are completely useless to me in terms of logging.

Hi there,

I've encountered an issue while running the example code provided in this repository. Specifically, I noticed that the log output does not include the expected error message 'err: err here'.

Steps to reproduce:

1. Run the example code.
2. Access http://localhost:8000/err in browser
3. Observe the log output.

Expected Behavior:
I expected to see 'err: err here' in the log output.

Actual Behavior:
The 'err: err here' message is missing from the log output.

Environment:

• github.com/go-chi/chi/v5 v5.0.10
• github.com/go-chi/httplog/v2 v2.0.1
• go1.21.1 darwin/arm64
• OS: macOS 14.0

Screenshot:

I intend to use your module for our application. It was fine until I need to set the writer to os.Stdout.
I can see the writer is set on config.go like below
log.Logger = log.Output(zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: time.RFC3339})
Would you please show me the way to set it to os.Stdout?
And I would be happy if I can configure the TimeFormat too.

Thank you.

requestLogFields will print duplicated url:

func requestLogFields(r *http.Request, concise bool) map[string]interface{} {
	scheme := "http"
	if r.TLS != nil {
		scheme = "https"
	}
	requestURL := fmt.Sprintf("%s://%s%s", scheme, r.Host, r.RequestURI)
        ...
}

Currently this does not print Response Headers in the logs:

	logger := httplog.NewLogger("http-relay", httplog.Options{
		JSON:     false,
		LogLevel: slog.LevelDebug,
		Concise: false,
		ResponseHeaders: true,
	})
	r.Use(httplog.RequestLogger(logger))

This is fixed by either #34 or #39 : the problem is that the Options are not being passed to the RequestLoggerEntry in the function NewLogEntry.

How do write an file?

need to have an option to append to replaceAttrs function

Hi there,

I spent several hours trying to figure an issue with zerolog, which i reported here - rs/zerolog#434, and which I eventually traced to the fact that httplog.NewLogger, via httplog.Configure sets global zerolog configurations.

This is a problem - in my use case go-chi is used to expose some webhooks from an application that is not a rest API primarily, but also in other cases the user may not want to use the httplog logger throughout the application but rather import directly from github.com/rs/zerolog/log or create an application specific logger in a logger package. Furthermore, its completely obfuscated that this is happening, leading to a very annoying and complex debug process.

I would therefore like to request / suggest that httplog removes the global configuration, which from my PoV are completely unnecessary there.

Using httplog in combination with chi compress middleware logs compressed response body. Has anyone encountered similar issues before? I wrote a custom implementation of httplog where I manually decompress response, but I'm wondering if there is a cleaner solution?

To reproduce, add compress middleware to example and set Content-Type header to text/plain:

// main.go, 24-27
r.Use(httplog.RequestLogger(logger, []string{"/ping"}))
r.Use(middleware.Compress(5))
r.Use(middleware.SetHeader("Content-Type", "text/plain"))
r.Use(middleware.Heartbeat("/ping"))

Example of console output when calling /warn endpoint with JSON: false and Concise: false:

where you can see response body logged as "\u001f�\u0008\u0000\u0000\u0000\u0000\u0000\u0000�*O,�S�H-J\u0005\u0004\u0000\u0000��?�\u001dv\t\u0000\u0000\u0000"

currently the private requestLogger is hardcoded in the Handler. As far as I can tell this means that you cannot customize the logger request fields.

Would it be useful to have the same interface as the chi middleware logger and have the Handler take in a LogFormatter. That way a user could customize the request fields.

I'm pretty new to this package so I could be wrong but I'd be happy to make an MR with this change.

When JSON: true, Concise: false, the first log entry have two fields with the same key: httpRequest. One of these fields is the concise version, the other is the full version.

Minimum reproducible code

package main

import (
	"net/http"

	"github.com/go-chi/chi/v5"
	"github.com/go-chi/httplog"
)

func main() {
	r := chi.NewRouter()

	logger := httplog.NewLogger("test", httplog.Options{JSON: true, Concise: false})
	r.Use(httplog.Handler(logger))

	r.Get("/", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
	_ = http.ListenAndServe(":8000", r)
}

Generated Log

{"level":"info","service":"test","httpRequest":{"proto":"HTTP/1.1","remoteIP":"127.0.0.1:49473","requestMethod":"GET","requestPath":"/","requestURL":"http://localhost:8000/"},"httpRequest":{"header":{"accept":"*/*","user-agent":"curl/8.0.1"},"proto":"HTTP/1.1","remoteIP":"127.0.0.1:49473","requestMethod":"GET","requestPath":"/","requestURL":"http://localhost:8000/","scheme":"http"},"timestamp":"2023-07-19T23:06:38.1838116+06:00","message":"Request: GET /"}

Here the first httpRequest field is the compact version and is redundant, since the second httpRequest field contains the same info and some more.

Expected Log

{"level":"info","service":"test","httpRequest":{"header":{"accept":"*/*","user-agent":"curl/8.0.1"},"proto":"HTTP/1.1","remoteIP":"127.0.0.1:49413","requestMethod":"GET","requestPath":"/","requestURL":"http://localhost:8000/","scheme":"http"},"timestamp":"2023-07-19T23:05:40.7004855+06:00","message":"Request: GET /"}

I am submitting a PR fixing this issue.

Go 1.20 will ship with new "slog" package. We can use it now via "golang.org/x/exp/slog"

Let's upgrade httplog to use it instead :)

It is likely that an application using slog will want to use its own pre-configured *slog.Logger or provide a slog.Handler that a logger should be created from.

The design of the package makes it confusing to do this.

Sure, I can manually create a httplog.Logger, but the options mix what is required for "configuring the logger" and what is used in logging.

l := &httplog.Logger{
  Logger: myExistingSlogLogger,
  Options: httplog.Options{
    JSON: true, // I'm guessing this does nothing
  },
}

I would suggest that the package asks only for a slog.Handler and only uses options to configure what to log:

func Handler(h slog.Handler, opts Options) func(next http.Handler) http.Handler {
	if h == nil {
		h = slog.Default().Handler()
	}
	// implementation
}

type Options struct {
	// Level determines what level to log request details at
	Level slog.Level 

	// Concise mode includes fewer log details during the request flow. For example
	// excluding details like request content length, user-agent and other details.
	// This is useful if during development your console is too noisy.
	Concise bool

	// RequestHeaders enables logging of all request headers, however sensitive
	// headers like authorization, cookie and set-cookie are hidden.
	RequestHeaders bool

	// SkipRequestHeaders are additional requests headers which are redacted from the logs
	SkipRequestHeaders []string

	// QuietDownRoutes are routes which are temporarily excluded from logging for a QuietDownPeriod after it occurs
	// for the first time
	// to cancel noise from logging for routes that are known to be noisy.
	QuietDownRoutes []string

	// QuietDownPeriod is the duration for which a route is excluded from logging after it occurs for the first time
	// if the route is in QuietDownRoutes
	QuietDownPeriod time.Duration
}

I understand that my changes would be breaking (so that means a v3 so soon after a v2), but I think it becomes easier to use with other slog compatible packages.

Error:

panic serving [::1]:50476: runtime error: index out of range [8] with length 8
goroutine 42 [running]:
net/http.(*conn).serve.func1()
....
/.asdf/installs/golang/1.21.3/packages/pkg/mod/github.com/go-chi/httplog/[email protected]/httplog.go:262 +0x4a8
.....

This is the problematic line: