GithubHelp home page GithubHelp logo

samber / slog-loki Goto Github PK

View Code? Open in Web Editor NEW
34.0 2.0 4.0 449 KB

๐Ÿšจ slog: Loki handler

Home Page: https://pkg.go.dev/github.com/samber/slog-loki/v2

License: MIT License

Makefile 16.20% Go 83.80%
attribute error errors go golang grafana handler log log-level logger

slog-loki's Introduction

slog: Loki handler

tag Go Version GoDoc Build Status Go report Coverage Contributors License

A Loki Handler for slog Go library.

See also:

HTTP middlewares:

Loggers:

Log sinks:

๐Ÿš€ Install

go get github.com/samber/slog-loki/v3

Compatibility: go >= 1.21

No breaking changes will be made to exported APIs before v4.0.0.

๐Ÿ’ก Usage

GoDoc: https://pkg.go.dev/github.com/samber/slog-loki/v3

Handler options

type Option struct {
	// log level (default: debug)
	Level slog.Leveler

	// loki
	Client *loki.Client

	// optional: customize webhook event builder
	Converter Converter
	// optional: fetch attributes from context
	AttrFromContext []func(ctx context.Context) []slog.Attr

	// optional: see slog.HandlerOptions
	AddSource   bool
	ReplaceAttr func(groups []string, a slog.Attr) slog.Attr
}

Attributes will be injected in log payload.

Other global parameters:

slogloki.SourceKey = "source"
slogloki.ErrorKeys = []string{"error", "err"}
slogloki.SubAttributeSeparator = "__"
slogloki.AttributeKeyInvalidCharReplacement = "_"

Example

import (
	"github.com/grafana/loki-client-go/loki"
	slogloki "github.com/samber/slog-loki/v3"
	"log/slog"
)

func main() {
	// setup loki client
	config, _ := loki.NewDefaultConfig("http://localhost:3100/loki/api/v1/push")
	config.TenantID = "xyz"
	client, _ := loki.New(config)

	logger := slog.New(slogloki.Option{Level: slog.LevelDebug, Client: client}.NewLokiHandler())
	logger = logger.
		With("environment", "dev").
		With("release", "v1.0.0")

	// log error
	logger.Error("caramba!")

	// log user signup
	logger.Info("user registration")

	// stop loki client and purge buffers
	client.Stop()
}

Tracing

Import the samber/slog-otel library.

import (
	slogloki "github.com/samber/slog-loki"
	slogotel "github.com/samber/slog-otel"
	"go.opentelemetry.io/otel/sdk/trace"
)

func main() {
	tp := trace.NewTracerProvider(
		trace.WithSampler(trace.AlwaysSample()),
	)
	tracer := tp.Tracer("hello/world")

	ctx, span := tracer.Start(context.Background(), "foo")
	defer span.End()

	span.AddEvent("bar")

	logger := slog.New(
		slogloki.Option{
			// ...
			AttrFromContext: []func(ctx context.Context) []slog.Attr{
				slogotel.ExtractOtelAttrFromContext([]string{"tracing"}, "trace_id", "span_id"),
			},
		}.NewLokiHandler(),
	)

	logger.ErrorContext(ctx, "a message")
}

๐Ÿค Contributing

Don't hesitate ;)

# Install some dev dependencies
make tools

# Run tests
make test
# or
make watch-test

๐Ÿ‘ค Contributors

Contributors

๐Ÿ’ซ Show your support

Give a โญ๏ธ if this project helped you!

GitHub Sponsors

๐Ÿ“ License

Copyright ยฉ 2023 Samuel Berthe.

This project is MIT licensed.

slog-loki's People

Contributors

carlohamalainen avatar dependabot[bot] avatar samber avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

slog-loki's Issues

Confused by how to use this package

Hi @samber!

I seem to not understand how to use this package.
When I configure the endpoint as in your example (just with the hostname of the actual server), I get the following output:

time=2023-11-10T12:16:08.199+01:00 level=INFO msg="promtail.ClientJson: Unexpected HTTP status code: 200, message: OK"

These entries are not logged when using the https://{YOUR_LOKI_ENDPOINT}/loki/api/v1/push endpoint, which is the usual one accepted by Promtail.
So I guess I have configured the correct endpoint?

However, I cannot find the logs anywhere in my Grafana instance with the Loki server being configured as the data source.
No labels are generated (should those be the keys in my log entries?) and thus I cannot query any.

This is my configuration (importing github.com/samber/slog-loki/v2 and github.com/samber/slog-multi):

logger := slog.New(
	slogmulti.Fanout(
		slogloki.Option{Level: slog.LevelDebug, Endpoint: lokiEndpoint}.NewLokiHandler(),
		slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelDebug}),
		slog.NewJSONHandler(f, &slog.HandlerOptions{Level: slog.LevelDebug})))
host, err := os.Hostname()
if err != nil {
	logger.Error("Failed to get hostname!", "error", err)
	os.Exit(1)
}
logger = logger.With("tool", toolName)
logger = logger.With("hostname", host)

Any help would be great!

x-scope-orgid

Hey ๐Ÿ‘‹๐Ÿผ โ€” I was looking through this code. I saw you used the same promtail client we use internally. Since the development stalled, we updated it in a fork to e.g. support x-scope-orgid in promtail. I wanted to ask if you'd be up for pulling our fork in, so we can support this in your handler?

Here's a diff of changes:
afiskon/promtail-client@master...hostwithquantum:promtail-client:pnmain

Till

Rename branch to `main`

It'd be nice if the main branch was renamed to main instead of the legacy / old term.

Unless a large number of users have active clones and are are sending PRs it's usually enough to just follow the GitHub instructions to rename the branch.

endpoint example

with the endpoint example of

endpoint := "localhost:3100"

the slog-loki (promclient really) gives an error of

time=2023-07-29T13:28:02.365-07:00 level=INFO msg="promtail.ClientJson: unable to send an HTTP request: Post \"localhost:3100\": unsupported protocol scheme \"localhost\""

Omitting client.Stop() prevents logs being pushed to loki server

Hello

I managed to get logs pushed to my loki grafana setup (with some trial and error) and I was wondering if it is necessary to call client.Stop() in order for the logs to be pushed to the loki server?

`func SetDefaultLogger() {
url := "http://localhost:3100/loki/api/v1/push"

config, lokiErr := loki.NewDefaultConfig(url)
if lokiErr != nil {
	panic(lokiErr)
}

client, lokiErr := loki.New(config)
if lokiErr != nil {
	panic(lokiErr)
}

// Is this necessary?
defer client.Stop()

option := slogloki.Option{Level: slog.LevelDebug, Client: client}
handler := option.NewLokiHandler()
logger := slog.New(handler).With(slog.String("job", "audit-log"))

slog.SetDefault(logger)

log.Print("log.Print")
logger.Debug("logger.Debug")
logger.Info("logger.Info")
logger.Warn("logger.Warn")
logger.Error("logger.Error")

}`

If I remove the line defer client.Stop() I don't see the messages in Grafana.

The problem for me is that my application will hang if the client is stopped and I try and call the default logger elsewhere in the code.

Is there a way I can set this default logger without having to call client.Stop()

Do not use `.` in labels in the flatten function of converter.go

I'm using slog-loki just for my personal project. I've tried the newer version with updating the client of loki before merging #5 .

I found slog-loki might need to convert label names in different way.

slog-loki/converter.go

Lines 39 to 55 in 7da5375

func flatten(prefix string, src map[string]any, dest model.LabelSet) {
if len(prefix) > 0 {
prefix += "."
}
for k, v := range src {
switch child := v.(type) {
case map[string]any:
flatten(prefix+k, child, dest)
case []any:
for i := 0; i < len(child); i++ {
dest[model.LabelName(prefix+k+"."+strconv.Itoa(i))] = model.LabelValue(fmt.Sprintf("%v", child[i]))
}
default:
dest[model.LabelName(prefix+k)] = model.LabelValue(fmt.Sprintf("%v", v))
}
}
}

The function is using . to express enclosed slog.Attrs. But this seems not to be allowed in loki[1].

I tweaked that to use _ in my usecase and it's easy to send a PR for this fix. But this solution might be not the ideal that the auhor of this library think of.

[1]https://grafana.com/docs/loki/latest/get-started/labels/#format

Failure to send log message can block the process

Hi,

I've encountered the following problem: When using slog-loki, if the Loki server is not reachable when sending the first message, the process may halt. This only seems to occur if there is a significant (>= 1 second?) delay between the first two messages. Consider the following code:

package main

import (
	"time"

	"github.com/grafana/loki-client-go/loki"
	slogloki "github.com/samber/slog-loki/v3"

	"log/slog"
)

func main() {
	config, _ := loki.NewDefaultConfig("http://localhost:3100/loki/api/v1/push")
	config.TenantID = "xyz"
	client, _ := loki.New(config)

	lokiLogHandler := slogloki.Option{Level: slog.LevelDebug, Client: client}.NewLokiHandler()
	defer func() {
		client.Stop()
	}()
	lokiLogger := slog.New(lokiLogHandler)

	lokiLogger.Error("Message 1")
	slog.Info("Message 1 logged")

	time.Sleep(1 * time.Second)

	lokiLogger.Error("Message 2")
	slog.Info("Message 2 logged")
	lokiLogger.Error("Message 3")
	slog.Info("Message 3 logged")

	time.Sleep(3 * time.Second)
	slog.Info("Post sleep")
}

When running this code without an available Loki instance, I'd expect the application to output the three "Message x logged" info logs to stdout, as well as a lot of "error sending batch, will retry" messages to stderr. However, the application halts for about seven minutes and then continues to run as expected:

$ go run main.go             
2024/07/22 22:22:28 INFO Message 1 logged
level=warn component=client host=localhost:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
[...]
level=error component=client host=localhost:3100 msg="final error sending batch" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
2024/07/22 22:29:33 INFO Message 2 logged
2024/07/22 22:29:33 INFO Message 3 logged
level=warn component=client host=localhost:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
level=warn component=client host=localhost:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
2024/07/22 22:29:36 INFO Post sleep
level=warn component=client host=localhost:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
[...]

This error only occurs if there is a significant delay between the first two messages. When I remove the first sleep, the expected behaviour occurs.

[...]
	lokiLogger.Error("Message 1")
	slog.Info("Message 1 logged")
	lokiLogger.Error("Message 2")
	slog.Info("Message 2 logged")
	lokiLogger.Error("Message 3")
	slog.Info("Message 3 logged")
[...]

Output:

$ go run main.go
2024/07/22 22:33:45 INFO Message 1 logged
2024/07/22 22:33:45 INFO Message 2 logged
2024/07/22 22:33:45 INFO Message 3 logged
level=warn component=client host=localhost:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
level=warn component=client host=localhost:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
2024/07/22 22:33:48 INFO Post sleep
level=warn component=client host=localhost:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://localhost:3100/loki/api/v1/push\": dial tcp 127.0.0.1:3100: connect: connection refused"
[...]

I haven't yet had time to investigate further edge cases or to look into how grafana/loki-client-go handles message batching, which I think could be related to this issue.

Any ideas on what the cause of this issue may be and how to solve it?

Best regards,
Daniel

`slog-loki` may need to convert `-` in labels to another character

This is quite relevant to another issue #10.

That was about the character used in this library itself, thus it should be fixed on this library side.
But this issue is about the other characters especially about -.

slog-echo is using - in the label name for example.

https://github.com/samber/slog-echo/blob/bbb3b8d417a6a6afeb828047829eccbc2ec46a56/middleware.go#L180-L187

This is not the character supported in Loki[1]. slog-loki may need to escape them to another character?

[1]https://grafana.com/docs/loki/latest/get-started/labels/#format

Log in JSON style

How can i log messages in json format?

Currently my log messages in loki look like this:
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.