Comments (9)
Hi @dipth - thanks for writing to us about this issue!
This is not something we've encountered before, and I'm not able to reproduce it locally, so I have a few more questions for you to better understand your setup:
- Can you describe how the page in question is being rendered? Is it rendered from a template using a standard Rails controller action, or are you calling
send_data
orsend_file
? - Is the connection from nginx -> unicorn over a TCP socket or a UNIX domain socket?
- Are you using automatic instrumentation (the default) or manual instrumentation?
Thanks!
from newrelic-ruby-agent.
@benweint have you tried creating a Engineyard account and testing on their service?
To answer your questions:
- The page is rendered from a template via a controller as normal.
- I am unable to answer this question as I am not completely sure how this works internally on the Engineyard platform.
- The bug is only present when using automatic instrumentation.
from newrelic-ruby-agent.
A few additional comments:
- Engine Yard has a test account with 500 Free Hours, if you want to see/test out the configuration in subject.
- Using manual instrumentation, doesn't have any issues. This works completely as expected.
We implemented this (= manual instrumentation) afterwards, when we got aware of the injection issue.
from newrelic-ruby-agent.
I do believe EngineYard have this setup:
HAproxy (loadbalancer) -> Nginx (Appserver) -> Unicorn (App)
Nginx connects to Unicorn through a unix socket:
upstream upstream_APPNAME {
server unix:/var/run/engineyard/unicorn_APPNAME.sock fail_timeout=0;
}
from newrelic-ruby-agent.
Hey folks - thanks for the details!
I'm still trying to get setup to reproduce this (having some trouble today deploying a test application to EY which is slowing me down), and I've not been able to reproduce by running HAproxy -> nginx -> unicorn locally using a modified copy of EngineYard's haproxy configuration file.
While I'm trying to get a test app running on EY, we might be able to make progress by having you just try some experiments while SSH'd into one of your EY instances. What I'd like to check first is whether nginx is serving back a correct content-length header to haproxy. This should be pretty easy to check: SSH in to one of your instances, and manually curl the running nginx instance that's listening on port 81, like this:
curl -iv http://localhost:81
If you are able to try this test with auto-instrumentation enabled in a staging environment, that would be very helpful in narrowing down where exactly the truncation is happening (e.g. between unicorn and nginx or nginx and haproxy).
Thanks!
from newrelic-ruby-agent.
Here is the result:
deploy@ip-10-228-43-208 ~ $ curl -iv -H 'Host: downtown.dk' http://localhost:81/upgrade
* About to connect() to localhost port 81 (#0)
* Trying ::1...
* Connection refused
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 81 (#0)
> GET /upgrade HTTP/1.1
> User-Agent: curl/7.26.0
> Accept: */*
> Host: downtown.dk
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.2.9
Server: nginx/1.2.9
< Date: Wed, 09 Jul 2014 08:11:49 GMT
Date: Wed, 09 Jul 2014 08:11:49 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Content-Length: 2892
Content-Length: 2892
< Connection: keep-alive
Connection: keep-alive
< Status: 200 OK
Status: 200 OK
< X-UA-Compatible: IE=Edge,chrome=1
X-UA-Compatible: IE=Edge,chrome=1
< ETag: "de06277ac327fe3cde8046ce10aa1431"
ETag: "de06277ac327fe3cde8046ce10aa1431"
< Cache-Control: max-age=0, private, must-revalidate
Cache-Control: max-age=0, private, must-revalidate
< Set-Cookie: organization_id=2; path=/; expires=Thu, 09-Jul-2015 08:11:49 GMT
Set-Cookie: organization_id=2; path=/; expires=Thu, 09-Jul-2015 08:11:49 GMT
< Set-Cookie: pio_uid=3668eef9829961a836bcb550d7c6ab31; path=/
Set-Cookie: pio_uid=3668eef9829961a836bcb550d7c6ab31; path=/
< Set-Cookie: _naoshi_session=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJTA3ZThmNjY3ZDJjYjY1YmMyNjIzNWMzYmExZjhjY2E2BjsAVEkiHnRyYWNrX3VuaXF1ZV92aXNpdF9jYWxsZWQGOwBGVEkiFG9yZ2FuaXphdGlvbl9pZAY7AEZpBw%3D%3D--a8c2bda4b92646ecbde2ebb9513b65faa091a254; domain=.downtown.dk; path=/
Set-Cookie: _naoshi_session=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJTA3ZThmNjY3ZDJjYjY1YmMyNjIzNWMzYmExZjhjY2E2BjsAVEkiHnRyYWNrX3VuaXF1ZV92aXNpdF9jYWxsZWQGOwBGVEkiFG9yZ2FuaXphdGlvbl9pZAY7AEZpBw%3D%3D--a8c2bda4b92646ecbde2ebb9513b65faa091a254; domain=.downtown.dk; path=/
< X-Runtime: 0.066354
X-Runtime: 0.066354
< X-Rack-Cache: miss
X-Rack-Cache: miss
<
<!DOCTYPE html>
<html lang="da">
<head>
<meta charset="utf-8">
<script type="text/javascript">window.NREUM||(NREUM={});NREUM.info={"beacon":"beacon-6.newrelic.com","errorBeacon":"bam.nr-data.net","licenseKey":"068869f06c","applicationID":"3061755","transactionName":"e11fQEcND1hVF04WFl9AUFBQTQpaVAAZ","queueTime":1,"applicationTime":66,"ttGuid":"","agentToken":null,"agent":"js-agent.newrelic.com/nr-411.min.js"}</script>
<script type="text/javascript">window.NREUM||(NREUM={}),__nr_require=function(t,n,e){function r(e){if(!n[e]){var o=n[e]={exports:{}};t[e][0].call(o.exports,function(n){var o=t[e][1][n];return r(o?o:n)},o,o.exports)}return n[e].exports}if("function"==typeof __nr_require)return __nr_require;for(var o=0;o<e.length;o++)r(e[o]);return r}({D5DuLP:[function(t,n){function e(t,n){var e=r[t];return e?e.apply(this,n):(o[t]||(o[t]=[]),void o[t].push(n))}var r={},o={};n.exports=e,e.queues=o,e.handlers=r},{}],handle:[function(t,n){n.exports=t("D5DuLP")},{}],G9z0Bl:[function(t,n){function e(){var t=l.info=NREUM.info;if(t&&t.agent&&t.licenseKey&&t.applicationID&&p&&p.body){l.proto="https"===f.split(":")[0]||t.sslForHttp?"https://":"http://",i("mark",["onload",a()]);var n=p.createElement("script");n.src=l.proto+t.agent,p.body.appendChild(n)}}function r(){"complete"===p.readyState&&o()}function o(){i("mark",["domContent",a()])}function a(){return(new Date).getTime()}var i=t("handle"),u=window,p=u.document,s="addEventListener",c="attachEvent",f=(""+location).split("?")[0],l=n.exports={offset:a(),origin:f,features:[]};p[s]?(p[s]("DOMContentLoaded",o,!1),u[s]("load",e,!1)):(p[c]("onreadystatechange",r),u[c]("onload",e)),i("mark",["firstbyte",a()])},{handle:"D5DuLP"}],loader:[function(t,n){n.exports=t("G9z0Bl")},{}]},{},["G9z0Bl"]);</script>
<meta content='<meta content="" name="description" />' name="description">
<meta content="" name="author">
<title>Downtown</title>
<link href="https://fonts.googleapis.com/css?family=PT+Sans:400,700" media="screen" rel="stylesheet" type="text/css" />
<style>
/*<![CDATA[*/
body {
background-color: #F2F2F0;
font-family: 'PT Sans', sans-serif;
text-align: center;
}
/*]]>*/
</style>
</head>
<body>
<img alt="Upgrade" src="https://ddgvwerrov012.cloudfront.net/assets/upgrade-fccf7d70173522254d7fe010f20b12ad.gif" />
<h1>Hov! Det ser ud til, at du bruger en forældet browser.</h1>
<p>
For at kunne gennemføre køb på siden, er du nødt til at opgradere din browser.
<br>
Det kan du gøre ved at klikke <a href="http://windows.microsoft.com/da-dk/internet-explorer/download-ie">her</a>.
</p>
<p>
Vil du se vores lækre kampagner, kan du klikke <a href="/upgrade?ignore=true">her</a>
</p>
<p>
<strong>Med venlig hilsen:</strong>
<br>
<img alt="Logo_upgrade" src="https://ddgvwerrov012.cloudfront.net/assets/downtown.dk/logo_upgrade-582b99b2e13169ffcd8fdbd0a99e5edd.gif" />
</p>
</body>
</html>
* Connection #0 to host localhost left intact
* Closing connection #0
from newrelic-ruby-agent.
Thanks for trying that test @dipth!
It looks like at least in that case, RUM injection was performed, the response body was not truncated, and the content-length header coming out of nginx was correct (it matches the actual response body length). That would suggest that the problem lies somehow with haproxy, but a few things to confirm before we reach that conclusion:
- Can you confirm whether you did this test with manual RUM instrumentation or auto-instrumentation? We're most interested in the result when using auto-instrumentation, which might mean that you'll have to do it in a staging environment.
- When the problem was exhibiting previously, did it happen consistently on every request, or only on a subset of requests?
Thanks!
from newrelic-ruby-agent.
I did in fact test it with auto-instrumentation enabled.
I was however unable to reproduce the problem on the staging server, so I actually ran the test on our production servers, which leads me to believe that the problem is only present in some versions of the EY instance framework.
The problem does happen consistently if you for instance use curl to perform the request, but the funny thing is that when you use a browser to perform requests, it is only the first request that exhibits the problem. I have tried including cookies in my curl requests and still saw the problem so I don't think the problem is cookie or session-based.
I did however notice that if I pass these flags to curl --header "Accept-Encoding:gzip, deflate" --compressed
, the request will not be truncated.
EY support did tell me that Unicorn was reporting incorrect response length to nginx, but I haven't been able to verify this myself.
from newrelic-ruby-agent.
@dipth - I work on New Relic's Browser App Monitoring team. Typically that kind of truncation occurs when the Content-Length response header is set incorrectly. The Ruby agent does update this header after it's modified the response body (at https://github.com/newrelic/rpm/blob/master/lib/new_relic/rack/browser_monitoring.rb#L79-L81) but it sounds likely that something about your production setup is either interfering with or bypassing this functionality in some cases.
I'd suggest inspecting the Content-Length headers as they pass through your infrastructure to try to pin down if and where they're incorrect. Also if you open a support ticket at support.newrelic.com it will be easier for us to collect the pertinent (and potentially private) details like logs, responses, etc., to help you debug it.
Thanks!
from newrelic-ruby-agent.
Related Issues (20)
- Support Rails 8 structured logging HOT 1
- Support Rails 8+ Solid Cache HOT 1
- Resque data is being sent as Web data. HOT 4
- Try integrating Napkin Math with our performance tests HOT 1
- Update `darwin` environment report output HOT 4
- Add `windows` OS detection in environment report HOT 1
- Fix `padrino` and `grape` framework detection HOT 1
- Fix `falcon` dispatcher detection HOT 5
- Remove support for ActiveMerchant versions below 1.65.0 HOT 1
- Remove support for DelayedJob versions below 4.1.0 HOT 1
- Remove support for Excon versions below 0.56.0 HOT 1
- Remove support for Sidekiq versions below 5.0.0 HOT 1
- Remove support for Sinatra versions below 2.0.0 HOT 1
- Unit tests coverage report artifacts are not stored in github actions HOT 4
- Create local `technote-space/workflow-conclusion-action` gha HOT 1
- Add aws gems to BMDS HOT 1
- Add OpenSearch testing to CI HOT 1
- Add support warnings to DependencyDetection HOT 2
- Reduce default arg object allocations in instrumentation HOT 1
- Redis Cluster causes instrumentation error HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from newrelic-ruby-agent.