Browser monitoring causes truncated responses via nginx and unicorn about newrelic-ruby-agent

[benweint]

Hi @dipth - thanks for writing to us about this issue!

This is not something we've encountered before, and I'm not able to reproduce it locally, so I have a few more questions for you to better understand your setup:

1. Can you describe how the page in question is being rendered? Is it rendered from a template using a standard Rails controller action, or are you calling send_data or send_file?
2. Is the connection from nginx -> unicorn over a TCP socket or a UNIX domain socket?
3. Are you using automatic instrumentation (the default) or manual instrumentation?

Thanks!

[dipth]

@benweint have you tried creating a Engineyard account and testing on their service?

To answer your questions:

1. The page is rendered from a template via a controller as normal.
2. I am unable to answer this question as I am not completely sure how this works internally on the Engineyard platform.
3. The bug is only present when using automatic instrumentation.

[nj]

A few additional comments:

1. Engine Yard has a test account with 500 Free Hours, if you want to see/test out the configuration in subject.
2. Using manual instrumentation, doesn't have any issues. This works completely as expected.
   We implemented this (= manual instrumentation) afterwards, when we got aware of the injection issue.

[kaspergrubbe]

I do believe EngineYard have this setup:

HAproxy (loadbalancer) -> Nginx (Appserver) -> Unicorn (App)

Nginx connects to Unicorn through a unix socket:

upstream upstream_APPNAME {
  server unix:/var/run/engineyard/unicorn_APPNAME.sock fail_timeout=0;
}

[benweint]

Hey folks - thanks for the details!

I'm still trying to get setup to reproduce this (having some trouble today deploying a test application to EY which is slowing me down), and I've not been able to reproduce by running HAproxy -> nginx -> unicorn locally using a modified copy of EngineYard's haproxy configuration file.

While I'm trying to get a test app running on EY, we might be able to make progress by having you just try some experiments while SSH'd into one of your EY instances. What I'd like to check first is whether nginx is serving back a correct content-length header to haproxy. This should be pretty easy to check: SSH in to one of your instances, and manually curl the running nginx instance that's listening on port 81, like this:

curl -iv http://localhost:81

If you are able to try this test with auto-instrumentation enabled in a staging environment, that would be very helpful in narrowing down where exactly the truncation is happening (e.g. between unicorn and nginx or nginx and haproxy).

Thanks!

[dipth]

Here is the result:

deploy@ip-10-228-43-208 ~ $ curl -iv -H 'Host: downtown.dk'  http://localhost:81/upgrade
* About to connect() to localhost port 81 (#0)
*   Trying ::1...
* Connection refused
*   Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 81 (#0)
> GET /upgrade HTTP/1.1
> User-Agent: curl/7.26.0
> Accept: */*
> Host: downtown.dk
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.2.9
Server: nginx/1.2.9
< Date: Wed, 09 Jul 2014 08:11:49 GMT
Date: Wed, 09 Jul 2014 08:11:49 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Content-Length: 2892
Content-Length: 2892
< Connection: keep-alive
Connection: keep-alive
< Status: 200 OK
Status: 200 OK
< X-UA-Compatible: IE=Edge,chrome=1
X-UA-Compatible: IE=Edge,chrome=1
< ETag: "de06277ac327fe3cde8046ce10aa1431"
ETag: "de06277ac327fe3cde8046ce10aa1431"
< Cache-Control: max-age=0, private, must-revalidate
Cache-Control: max-age=0, private, must-revalidate
< Set-Cookie: organization_id=2; path=/; expires=Thu, 09-Jul-2015 08:11:49 GMT
Set-Cookie: organization_id=2; path=/; expires=Thu, 09-Jul-2015 08:11:49 GMT
< Set-Cookie: pio_uid=3668eef9829961a836bcb550d7c6ab31; path=/
Set-Cookie: pio_uid=3668eef9829961a836bcb550d7c6ab31; path=/
< Set-Cookie: _naoshi_session=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJTA3ZThmNjY3ZDJjYjY1YmMyNjIzNWMzYmExZjhjY2E2BjsAVEkiHnRyYWNrX3VuaXF1ZV92aXNpdF9jYWxsZWQGOwBGVEkiFG9yZ2FuaXphdGlvbl9pZAY7AEZpBw%3D%3D--a8c2bda4b92646ecbde2ebb9513b65faa091a254; domain=.downtown.dk; path=/
Set-Cookie: _naoshi_session=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJTA3ZThmNjY3ZDJjYjY1YmMyNjIzNWMzYmExZjhjY2E2BjsAVEkiHnRyYWNrX3VuaXF1ZV92aXNpdF9jYWxsZWQGOwBGVEkiFG9yZ2FuaXphdGlvbl9pZAY7AEZpBw%3D%3D--a8c2bda4b92646ecbde2ebb9513b65faa091a254; domain=.downtown.dk; path=/
< X-Runtime: 0.066354
X-Runtime: 0.066354
< X-Rack-Cache: miss
X-Rack-Cache: miss

<
<!DOCTYPE html>
<html lang="da">
<head>
<meta charset="utf-8">
<script type="text/javascript">window.NREUM||(NREUM={});NREUM.info={"beacon":"beacon-6.newrelic.com","errorBeacon":"bam.nr-data.net","licenseKey":"068869f06c","applicationID":"3061755","transactionName":"e11fQEcND1hVF04WFl9AUFBQTQpaVAAZ","queueTime":1,"applicationTime":66,"ttGuid":"","agentToken":null,"agent":"js-agent.newrelic.com/nr-411.min.js"}</script>
<script type="text/javascript">window.NREUM||(NREUM={}),__nr_require=function(t,n,e){function r(e){if(!n[e]){var o=n[e]={exports:{}};t[e][0].call(o.exports,function(n){var o=t[e][1][n];return r(o?o:n)},o,o.exports)}return n[e].exports}if("function"==typeof __nr_require)return __nr_require;for(var o=0;o<e.length;o++)r(e[o]);return r}({D5DuLP:[function(t,n){function e(t,n){var e=r[t];return e?e.apply(this,n):(o[t]||(o[t]=[]),void o[t].push(n))}var r={},o={};n.exports=e,e.queues=o,e.handlers=r},{}],handle:[function(t,n){n.exports=t("D5DuLP")},{}],G9z0Bl:[function(t,n){function e(){var t=l.info=NREUM.info;if(t&&t.agent&&t.licenseKey&&t.applicationID&&p&&p.body){l.proto="https"===f.split(":")[0]||t.sslForHttp?"https://":"http://",i("mark",["onload",a()]);var n=p.createElement("script");n.src=l.proto+t.agent,p.body.appendChild(n)}}function r(){"complete"===p.readyState&&o()}function o(){i("mark",["domContent",a()])}function a(){return(new Date).getTime()}var i=t("handle"),u=window,p=u.document,s="addEventListener",c="attachEvent",f=(""+location).split("?")[0],l=n.exports={offset:a(),origin:f,features:[]};p[s]?(p[s]("DOMContentLoaded",o,!1),u[s]("load",e,!1)):(p[c]("onreadystatechange",r),u[c]("onload",e)),i("mark",["firstbyte",a()])},{handle:"D5DuLP"}],loader:[function(t,n){n.exports=t("G9z0Bl")},{}]},{},["G9z0Bl"]);</script>
<meta content='&lt;meta content="" name="description" /&gt;' name="description">
<meta content="" name="author">
<title>Downtown</title>
<link href="https://fonts.googleapis.com/css?family=PT+Sans:400,700" media="screen" rel="stylesheet" type="text/css" />
<style>
  /*<![CDATA[*/
    body {
      background-color: #F2F2F0;
      font-family: 'PT Sans', sans-serif;
      text-align: center;
    }
  /*]]>*/
</style>
</head>
<body>
<img alt="Upgrade" src="https://ddgvwerrov012.cloudfront.net/assets/upgrade-fccf7d70173522254d7fe010f20b12ad.gif" />
<h1>Hov! Det ser ud til, at du bruger en forældet browser.</h1>
<p>
For at kunne gennemføre køb på siden, er du nødt til at opgradere din browser.
<br>
Det kan du gøre ved at klikke <a href="http://windows.microsoft.com/da-dk/internet-explorer/download-ie">her</a>.
</p>
<p>
Vil du se vores lækre kampagner, kan du klikke <a href="/upgrade?ignore=true">her</a>
</p>
<p>
<strong>Med venlig hilsen:</strong>
<br>
<img alt="Logo_upgrade" src="https://ddgvwerrov012.cloudfront.net/assets/downtown.dk/logo_upgrade-582b99b2e13169ffcd8fdbd0a99e5edd.gif" />
</p>
</body>
</html>
* Connection #0 to host localhost left intact
* Closing connection #0

[benweint]

Thanks for trying that test @dipth!

It looks like at least in that case, RUM injection was performed, the response body was not truncated, and the content-length header coming out of nginx was correct (it matches the actual response body length). That would suggest that the problem lies somehow with haproxy, but a few things to confirm before we reach that conclusion:

1. Can you confirm whether you did this test with manual RUM instrumentation or auto-instrumentation? We're most interested in the result when using auto-instrumentation, which might mean that you'll have to do it in a staging environment.
2. When the problem was exhibiting previously, did it happen consistently on every request, or only on a subset of requests?

Thanks!

[dipth]

I did in fact test it with auto-instrumentation enabled.
I was however unable to reproduce the problem on the staging server, so I actually ran the test on our production servers, which leads me to believe that the problem is only present in some versions of the EY instance framework.

The problem does happen consistently if you for instance use curl to perform the request, but the funny thing is that when you use a browser to perform requests, it is only the first request that exhibits the problem. I have tried including cookies in my curl requests and still saw the problem so I don't think the problem is cookie or session-based.

I did however notice that if I pass these flags to curl --header "Accept-Encoding:gzip, deflate" --compressed, the request will not be truncated.

EY support did tell me that Unicorn was reporting incorrect response length to nginx, but I haven't been able to verify this myself.

[samg]

@dipth - I work on New Relic's Browser App Monitoring team. Typically that kind of truncation occurs when the Content-Length response header is set incorrectly. The Ruby agent does update this header after it's modified the response body (at https://github.com/newrelic/rpm/blob/master/lib/new_relic/rack/browser_monitoring.rb#L79-L81) but it sounds likely that something about your production setup is either interfering with or bypassing this functionality in some cases.

I'd suggest inspecting the Content-Length headers as they pass through your infrastructure to try to pin down if and where they're incorrect. Also if you open a support ticket at support.newrelic.com it will be easier for us to collect the pertinent (and potentially private) details like logs, responses, etc., to help you debug it.

Thanks!