Comments (11)
@iamemilio I see a few ways of resolving the issue.
-
Documenting logcontext approach.
Currently it's not clear about if it's support or not. If you use it, it silently does not do anything. Documentation for what should be used in serverless context is unclear. These are the issues that could be fixed by documenting, improving error handling (or configuration handling) on the agent side. What I also don't like is there being two logcontext standards and not knowing what's current and what should be used. Should not be the case, one should be deprecated. -
Treating serverless applications as first-class citizens in New Relic. Currently it's quite tricky to get it right. There is some development on the topic, I agree, but in the end there are issues in the current approach taken where it's not really production ready (timeouts) or some of the features don't work (logcontext). In the end, serverless applications are usually very similar to APM applications, apart from there being multiple dynamic backends. Therefore I don't see that there should be big differences in integrations.
I also understand that there are some complexities in terms of mapping things, knowing EntityGUIDs. However that's also a problem for short-lived processes (e.g. take console application or examples where you wait for connection to NewRelic to get EntityGUID).
What I would love to see is NewRelic taking more strategic direction:
- Making serverless applications first-class citizens of NewRelic
- Making NewRelic work with short-lived processes
I believe for both there are some ideas that could be pursued like for example enhancing Ingest APIs where those would infer Entity from Ingest Key as an example. It could also lead to simplifications on how NewRelic works. As an example for why it would be a simplification is look at the query NewRelic makes for Logs when it involves Serverless application. While technically it could just be a filter for trace.id and or EntityGuid, it now requires AWS RequestID, etc which does not work in more complicated cases.
Sorry for a long text, but looking forward for a more strategical approach. I will also bring this up to our Account Executive.
from go-agent.
To give more context (pun intended), the whole setup is correct, and behind the scenes, it works as expected, meaning:
- Transaction is correctly created a propagated when the lambda is invoked.
- Distributed Tracing in UI shows all components - Logs are correctly displayed under the Trace Details UI.
- When I'd extract the data manually it's available just fine, and I can push it to logs:
log.WithField("trace_metadata", txn.GetTraceMetadata()).Debug("testing NR trace data")
Additionally, I'm adding here a sample code showing how we did the integration:
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/events"
"github.com/newrelic/go-agent/v3/integrations/logcontext-v2/nrlogrus"
"github.com/newrelic/go-agent/v3/integrations/nrlambda"
"github.com/newrelic/go-agent/v3/newrelic"
"github.com/sirupsen/logrus"
)
func main() {
app, err := newrelic.NewApplication(
nrlambda.ConfigOption(),
newrelic.ConfigAppLogDecoratingEnabled(true),
)
if nil != err {
fmt.Println("error creating app (invalid config):", err)
}
logrus.SetFormatter(nrlogrus.NewFormatter(app, &logrus.TextFormatter{}))
logrus.SetLevel(logrus.TraceLevel)
nrlambda.Start(handle, app)
}
func handle(ctx context.Context, snsEvent events.SNSEvent) {
log := logrus.WithContext(ctx)
log.Debug("testing logging with context")
}
from go-agent.
Yeah, so this is a bit of a documentation gap IMO. When the agent is running in server less mode, logs in context is not supported in the agent. We expect users to instead collect logs through either the newrelic-lambda-extension, or the cloud watch-log-ingestion lambda function. All agents disable their internal logs in context when running in server less mode.
from go-agent.
Thanks for really digging into the root cause of this though, this is a very well reported issue.
from go-agent.
@iamemilio so what are our possibilities now? What's the preferred way (if you could share the links). And does it mean that by design we can't have a correlation between Logs and Transactions in case of serverless integrations?
from go-agent.
From what I understand, I don't think logs in context is fully supported in serverless environments right now, unfortunately. It's a bit more of a complicated situation than it seem on the face of it because there is information that is essential to how logs get linked "in context" that is missing in a serverless environment. You can use the newrelic-lambda-extension, or the cloud watch ingestion function to gather logs from AWS and send them to newrelic. That does add some additional metadata, but it's not equivalent to the logs in context feature. This is something that impacts all agents, and if it's important to you, I would recommend getting in touch with New Relic Support and filing a feature request.
from go-agent.
cc @ak-war
from go-agent.
I mean, I found a way to do it, but it requires a manual injection of Transaction metadata to the logs based on the logs in context v1. I've added those two functions to our codebase:
func WithContext(ctx context.Context) *logrus.Entry {
log := logrus.WithContext(ctx)
txn := newrelic.FromContext(ctx)
return appendNewRelicFields(txn, log)
}
func appendNewRelicFields(txn *newrelic.Transaction, log *logrus.Entry) *logrus.Entry {
md := txn.GetLinkingMetadata()
log = log.WithFields(
logrus.Fields{
logcontext.KeyTraceID: md.TraceID,
logcontext.KeySpanID: md.SpanID,
logcontext.KeyEntityName: md.EntityName,
logcontext.KeyEntityType: md.EntityType,
logcontext.KeyEntityGUID: md.EntityGUID,
logcontext.KeyHostname: md.Hostname,
},
)
return log
}
And that gets me wondering, why it was disabled in v2? Because I feel we're missing some background to understand the reasoning behind it 🤔
from go-agent.
Yeah, in theory we could enable local log decoration in serverless environments, which would always append any linking transaction, entity, and span data. But when the agent is running in serverless mode, the entity is unknown to the agent. The linking to the New Relic entity is normally done during the connect process in agents, but serverless apps skip that step to save time. Instead there is a special backend pipeline that they send data to that links data and logs to New Relic entities. We would probably have to modify this backend pipeline to be able to handle decorated log lines properly. While it might be possible, it's not something the agents are allowed to do and it's pretty much 100% untested on our end. If you wanted to try it and see if the backend can accept decorated logs, you're welcome to fork the agent at your own risk, give it a go, and let us know what your findings are. I don't think it would be dangerous, but it may not work the way you want it to. Otherwise, I pinged our PM here and we can track this as a potential roadmap item in the future and hope to come back with an agent wide solution if its possible.
from go-agent.
These are all good points. I can see if we can issue a deprecation of the outdated log context standard/packaging. In terms of company strategy on short lived processes and lambda, that is firmly out of my control. Please do bring it up with your representatives at New Relic. That helps us gauge interest, and makes issues more visible to our leadership who ultimately decide how to steer the product. That might be something @ak-war could help you with as well.
from go-agent.
Otherwise, I pinged our PM here and we can track this as a potential roadmap item in the future and hope to come back with an agent wide solution if its possible.
It's been quite some time. Any updates?
from go-agent.
Related Issues (20)
- External Services(AWS sdk go v2 ) api calls are not logged / visible when using github.com/newrelic/go-agent/v3/integrations/nrawssdk-v HOT 3
- Remove fasthttp dependency from main module HOT 2
- Update google.golang.org/grpc from 1.54.0 to 1.56.3 HOT 2
- transaction.go reqBody(req *http.Request) turn nil body into non Nil body for request.Body == http.NoBody HOT 1
- Add support to CopyFrom method in pgx integration HOT 1
- fmt.Stringer support for user attributes (especially for SQL query parameters)
- UNKNOWN log levels when using the native log package HOT 2
- `.Ignore()` not ignoring transaction (go-agent + nrgin) HOT 6
- Integration for gqlgen HOT 2
- nrpgx5 data-race HOT 5
- Changelog Go agent EOL Policy link is broken HOT 1
- agent with nrzap is not forwarding logs to newrelic HOT 3
- WrapBackgroundCore background logger is not sending logs to new relic but transaction logger is HOT 12
- zap logger.With log message properties are not sent to new relic HOT 4
- Attributes are not forwarded to NewRelic for logs HOT 16
- go agent , but not see span, please help me ,thanks HOT 3
- Specify service version during initial newrelic configuration HOT 6
- Possible memory leak in Go agent HOT 11
- How to use AWS Lambda Web Adapter in golang app
- uber/zap WithField(...) not showing up in NR APM logs HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-agent.