riandyrn / otelchi

OpenTelemetry instrumentation for go-chi/chi

otelchi

OpenTelemetry instrumentation for go-chi/chi.

Essentialy this is adaptation from otelmux but instead using gorilla/mux, we use go-chi/chi.

Currently it could only instrument traces.

Contributions are welcomed!

Install

$ go get github.com/riandyrn/otelchi

Examples

See examples for details.

Why Port This?

I was planning to make this project as part of Open Telemetry Go instrumentation project. However based on this comment they no longer accept new instrumentation. This is why I maintain this project here.

otelchi's Issues

Use method in span name

Currently, the span name is set to the matching RoutePattern. If GET/POST/DELETE methods all apply to the same route, these all get lumped under the same name. Would it work to adjust this to include the method here or have you not ran into this issue?

question: should the middleware inject the trace context into the http.Request?

I'm implementing a simple http reverse proxy in Go using chi. The proxy routes requests to microservices, and handles service to service requests as well as the initial "front door" request. I added otelchi.Middleware in hopes of getting a trace tree of all requests (eg: initial/parent request and child spans for called services) , but all spans were created as root spans. I eventually figured out that the trace context was not injected into the http request in func (tw traceware) ServeHTTP()

I worked around this by creating a middleware that injects the context, called after the otelchi middleware:

type Injector struct {
	next http.Handler
}

func (injector Injector) ServeHTTP(w http.ResponseWriter, req *http.Request) {

	jaegerPropagator.Inject(req.Context(), propagation.HeaderCarrier(req.Header))
	injector.next.ServeHTTP(w, req)
}

note that jaegerPropagator is a file scoped variable, just a hack to get this working.

Is there any reason why the otelchi middleware does not inject the trace context from the span it creates? It seems to me that is probably always the right thing to do, but I am pretty new to all this. Thanks.

Metric support

Hi there!

Would you be interested in adding optional support for some basic metrics? I'm thinking a single histogram to track request latency tagged with http method, route and response code.

Since Histograms also provide a count metric that should be enough to build your typical RED dashboard.

Happy to submit a PR for it!

panic doesn't mark span as error

the middleware doesn't call recover anywhere, pressumably because otel already does that according to the trace

go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
	/home/aep/.go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/span.go:396 +0x2a
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc00033e480, {0x0, 0x0, 0xc0000b4780?})
	/home/aep/.go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/span.go:435 +0x8ee

this records an event, but does not set the span as error.

confusingly, a 404 is set as error, but i don't understand where that's happening either. because otelchi calls span.SetStatus . should it also do that for panic, or is that a problem in otel?

Injecting trace headers?

Hey, sorry in advance for the newbie question 🙈

I've just been testing otelchi with some of our services but I think the trace headers are not injected on the outgoing request. Do I need to this manually? I couldn't find anything in the regarding injection in either this code base or the otel docs.

Cheers!

are you open to adding support for logging query values?

If you are open to a PR, I'd like to add support for logging query values to this middleware. What do you think?

(I maintain a New Relic middleware and doing that has been extremely useful.)

panic in net/http

Met this issue today which makes net/http panic. After opening an issue and snooping a lot around, it seems disabling otelchi was the solution.

golang/go#60862

There is something otelchi does wrong if it makes net/http panic like this. There is no feedback as to where the error is coming from, and I'm unable to debug this further.

Even if it was a developer error, the panic shouldn't come from net/http and come before, with an explaination.

Empty status code in span when handler is not sending data

If you create a HTTP Handler that is not writing anything and not setting the status code (WriteHeader) the status in the span is 0. The HTTP Server returns a 200 is this expected? It is because the status value is set to 0 as an initial value.

• https://github.com/riandyrn/otelchi/blob/master/middleware.go#L78

I can create a PR that changes this to 200?

Add filter for ignoring certain endpoints

Thanks for this project: I've been using it for a couple months now.

Would it be possible to ignore a healthcheck endpoint due to the frequency of calls?

Elsewhere it seems the pattern is to accept function filters that return a boolean: open-telemetry/opentelemetry-go-contrib#896

I'd be happy to open a pull request if you think the idea has merit.

bug: tracing with empty `spanName`

I was using v5.0.8 of chi when this error started happening on my service:

2023/02/16 10:23:22 failed to export to Google Cloud Trace: rpc error: code = InvalidArgument desc = Missing span display name!

I propose a fix in #18

New Development Strategy

At first, we thought that the OpenTelemetry project would eventually slow its development pace and become more backward compatible like many Go libraries.

We intentionally waited for this moment to continue otelchi development. However, we were wrong; instead of slowing down, its evolution is becoming much faster. This makes otelchi quickly obsolete in the OpenTelemetry ecosystem. 😅

On the other hand, we cannot just make otelchi comply with the latest version of the OpenTelemetry Go library because we need to set the minimum Go requirement to at least 1.20, which will make this library inaccessible for many Go versions (1.16 - 1.19).

Yet, people usually start to implement tracing when their system is already proven to be stable. So changing the minimum Go version of the system to use the latest version of the otelchi could lead to many unseen (and possibly catastrophic) consequences.

With these constraints in mind, we would like to share the new development strategy for this library:

base otelchi version	min go version	semconv version	otel version
0.6.0 ✅	1.15-1.17	1.10.0 1.12.0	1.7.0 1.10.0
0.7.0	1.17	1.12.0	1.10.0
0.7.0 ✅	1.18	1.17.0	1.14.0
0.8.0 ✅	1.19	1.17.0 1.20.0	1.19.0 1.24.0
0.9.0	1.20	1.20.0	1.24.0

So, first, we will develop version 0.6.0 by adjusting the otelchi implementation with the versions of semconv and otel shown in the table. After version 0.6.0 seems stable, we will develop 0.7.0. This cycle repeats until 0.10.0.

So what happens if there is a bug in version 0.6.0? We will patch it and release version 0.6.1. Then, we will merge this patch into all versions after 0.6.1 whenever applicable.

After these versions are stable enough (no bugs), we will start following the compatibility policy chosen by OpenTelemetry, which is mentioned here.

Update:

• March 24, 2024 - 16:05 UTC+7 => When I was working with #29, I discovered that it is possible to use otel v1.10.0 &
  semconv v1.12.0 without setting the go version in go.mod to go 1.17. This is why I decided that otelchi v0.6.0 will cover go 1.15-1.17.
• April 29, 2024 - 09:50 UTC+7 => In #41, @ProtozoaJr found that by setting minimum go version to 1.19, otelchi can use otel v1.24.0 & semconv v1.20.0.

Trace ID in header

Would you be open to adding a configuration option so we can set the trace id as a response header? Currently we do this by adding another middleware after otelchi in the stack, but it'd be nice to have it combined.