"invalid byte sequence in us-ascii" on certain pages about newrelic-ruby-agent

[jasonrclark]

Hey there @kliuless. Sorry again for the delay on getting back to you. We've tried to replicate the issue here, and from that investigation had the following questions to help us track it down:

1. What Ruby version are you running?
2. What's the expected encoding of the page contents itself (specifically is the page UTF-8 or something else)?
3. What's the exact error message that it's giving when it fails?
4. Would it be possible for you to patch the agent to call beginning_of_source.encoding and beginning_of_source.valid_encoding? We'd be happy to help you figure out how to do that if you're in a position where that'd be a possibility.

Additionally, a possible workaround would be to disable the auto-instrumentation provided by NewRelic::Rack::BrowserMonitoring and instead manually inject the javascript per this doc: https://docs.newrelic.com/docs/ruby/page-load-timing-in-ruby

[kliuless]

@jasonclark - let me try to reproduce this in our dev environment.

[kliuless]

@jasonrclark Well, for some reason the response object passed to BrowserMonitoring#call is yielding strings with US-ASCII encoding. I also tried with a new rails (3.2) app with a simple pass-thru middleware, and the middleware received a UTF-8 response. Could you please use a test app with the BrowserMonitoring middleware to see what's the encoding of a fragment in the gather_source method? If you get UTF-8, it's probably something in our particular app. If you get US-ASCII, then it seems like we have a bigger problem.

I'll keep trying to track this down. It's possible we have some middleware that's changing the encoding.

[jasonrclark]

@kliuless Ran a few of our test apps, and much like your new rails 3.2 app we're getting UTF-8 through consistently.

I was running this on Ruby 2.1. What version are you working with?

It does seem likely to be something in your app, but you're likely not the only one with different encodings, so I'd love to get a repro on this so we can address it in the agent.

Beyond other middlewares, I'd also be curious whether any of your template files are stored with a different encoding. I haven't managed to get my test app to duplicate passing through the altered encoding yet, but it seems like a likely way that might happen.

Appreciate your continued searching on this!

[kliuless]

@jasonrclark Turns out that this probably isn't a bug in the newrelic gem. We're doing some post-processing on some of our rendered content that's changing the encoding to US-ASCII.

[jasonrclark]

@kliuless Good to hear you're making progress. We've definitely encountered similar situations where a string is forced to an encoding where it's invalid and code downstream reacts badly.

From your description that sounds like the case here (a "Jürgen" getting forced to US-ASCII where it isn't valid). Things other than the agent might be unhappy with that, so you'll want to deal with it in your app, but the agent should be more resilient. I'm adding better error handling in that region thanks to your report, so these type of errors won't be a crashing situation due to the agent anymore.

I will report back here when we've got an agent released that has the fix, although it sounds like you might fix up the problem in your app more directly before that happens.

Thanks again for the report and investigation!

[kliuless]

@jasonrclark I found a minimal test case to reproduce this in Rails 3.2.17. Put this in an erb view:

Hello world <%= "\u2014".force_encoding(Encoding::US_ASCII).html_safe %>

The critical elements:

• A UTF-8 character outside the US-ASCII set is forced to US-ASCII.
• The forced part of the string is marked as html_safe. If this is omitted, Rails does some regexp matching which would raise the encoding error instead of letting the string pass through to the rest of the stack.

Our app is doing something much more complex than this contrived example, but believe it or not, the "US-ASCII" never explicitly appears anywhere in our code, any of our gems, or even the ruby standard lib. Net::HTTP creates an internal string for use as a buffer, and because ruby 1.9's default source file encoding is US-ASCII, that becomes the buffer's encoding too. We're inserting a string we've fetched from an http service, causing the problem above.
See this stackoverflow answer for more the Net::HTTP issue.

[jasonrclark]

@kliuless Great detective work! That's a fascinating problem.

I'll take that case and make sure that the fix I was applying will handle this nicely for the next agent release. Thanks again for all your work on this.

[benweint]

Hi @kliuless - thanks again for reporting this, and for your help in tracking it down! The latest version of the Ruby agent (3.9.0, just released today) should handle this scenario in a more graceful way. Please give it a try, and let us know if you run into any trouble!