Comments (12)
If it can be of any help, I am still not experiencing this issue after downgrading to v3.19.2.
It might be that the new code added after that is totally safe and unrelated and this is just luck. But if I had to start somewhere, the newly changed code is where I'd start looking into.
from go-agent.
Thanks for reporting, we will fix this ASAP
from go-agent.
This is going to be a tricky one. The library that the panic triggered on has not changed for 7 years though, so I doubt the version matters. It must just be an edge case that flew under the radar for quite some time.
from go-agent.
Yeah, putting a recover would be a good stopgap. Hmm, let me double check everything. That is a good point about the nil array, that might be a good lead for us, but I am not sure that we use the unsafe library for anything. I'll have to double check that.
from go-agent.
No problem, this is one of the most bizarre bugs I have seen in a long time haha. I hope it gets resolved and thanks for all the help! There will be a release this week with the patch, guaranteed.
from go-agent.
@nicoroy2561 Would it be possible to provide me with a reproducer?
from go-agent.
@nicoroy2561 Would it be possible to provide me with a reproducer?
Sorry, the stacktrace I posted is the whole I got from the crash. It's not even long enough for me to tell where it originates from.
It's also impossible to reproduce it because it only happened in live, I never encountered it when testing.
from go-agent.
I was not able to reproduce this, no matter what I tried. I did get some nice additional coverage out of it though 😄. #610. I am totally stumped right now. How often does this error occur in your environment? If it happens again, can you try to capture any additional information that could give us a clue as to why this might be happening?
from go-agent.
Hi,
sorry for the late response.
Answering your questions first, I got this error only once on my live server. If what you're saying is true and the library has never changed over different versions, that makes me fear that it might just randomly happen again.
I took some time to properly look both at the code causing the panic and the stack trace, and I noticed something that seems quite weird to me:
github.com/newrelic/go-agent/v3/internal/jsonx.AppendString(0xc000b5baf0?, {0x0, 0x3})
The very first line of the stacktrace is just... weird. Leaving aside the '?' which apparently is normal, the next values are for the string's underlying struct. And they show a 0x3
length value, but a nil pointer 0x0
.
This makes absolutely no sense as it shouldn't be possible to have a string object that has a length bigger than the underlying []byte (array?).
And that's likely what's causing the issue, because I kinda expect len() to only look at the length int and not check the actual pointer.
Since the problematic string originates from
func agentAttributesJSON(a *attributes, buf *bytes.Buffer, d destinationSet)
which takes the string from a.Agent, of type map[string]agentAttributeValue
, it makes me think that there may be something somewhere either using Unsafe or using c code to set one such erroneous value within that map.
from go-agent.
Perhaps a recover could be added to func (app *app) doHarvest(h *harvest, harvestStart time.Time, run *appRun)
so that when it's spawned by func (app *app) process()
it doesn't crash the whole application, but instead skips harvesting data that single time.
I'd much rather get the stacktrace and a message like "New Relic data harvest failed" than a app-wide panic, especially if it can then potentially go on harvesting data later (skipping the problematic one?).
from go-agent.
We have not made any changes to that code path for quite a while. I am not sure if your library is doing any unsafe pointer modification, but I have not been able to find a culprit in the go agent yet. We will release a panic recover that will protect the harvest process from crashing fatally in this weeks release, and we recommend you upgrade to it. I'll keep this issue in the back of my mind, but I have to time box this investigation for the time being... If you get a log message reporting a panic in the new release, please @ me here and we will re-asses things. Harvest panics should no longer crash your application, but will result in that harvest's data getting dropped.
Leaving this open: mitigated, not resolved.
from go-agent.
I just checked and couldn't find any unsafe usage in my application (and dependecies). I'm sure there's gotta be something somewhere.
Either that, or I have zero idea on how it's possible to generate one such weird data structure.
I'll let you know if I get any other stack traces. Will update the lib as soon as I see the commit :)
Thank you for the help!
from go-agent.
Related Issues (20)
- Add slog integration HOT 8
- Missing release notes for v3.25.1 HOT 3
- Consumes a lot of memory during large body size requests HOT 2
- External Services(AWS sdk go v2 ) api calls are not logged / visible when using github.com/newrelic/go-agent/v3/integrations/nrawssdk-v HOT 3
- Remove fasthttp dependency from main module HOT 2
- Update google.golang.org/grpc from 1.54.0 to 1.56.3 HOT 2
- transaction.go reqBody(req *http.Request) turn nil body into non Nil body for request.Body == http.NoBody HOT 1
- Add support to CopyFrom method in pgx integration HOT 1
- fmt.Stringer support for user attributes (especially for SQL query parameters)
- UNKNOWN log levels when using the native log package HOT 2
- `.Ignore()` not ignoring transaction (go-agent + nrgin) HOT 6
- Integration for gqlgen HOT 2
- nrpgx5 data-race HOT 5
- Changelog Go agent EOL Policy link is broken HOT 1
- agent with nrzap is not forwarding logs to newrelic HOT 3
- WrapBackgroundCore background logger is not sending logs to new relic but transaction logger is HOT 12
- zap logger.With log message properties are not sent to new relic HOT 4
- Attributes are not forwarded to NewRelic for logs HOT 16
- go agent , but not see span, please help me ,thanks HOT 3
- Specify service version during initial newrelic configuration HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-agent.