zombocom / rack-timeout Goto Github PK

There are certain configurations where Timeout::Error does not get thrown. This is possible due to certain Rails middlewares. We are currently investigating this issue.

Long ago, I rewrote timeout.rb so that it didn't require a 1-to-1 relationship between a thread and timeout thread. This meant that no matter the number of threads using timeouts, there was only one timeout thread. Given that rack-timeout uses that same 1-to-1 relationship, it is in danger of easily ballooning the number of threads used.

Here is the code for my timeout.rb you could adapt: https://github.com/rubysl/rubysl-timeout/blob/2.0/lib/rubysl/timeout/timeout.rb

You'd need remove the Rubinius::Channel stuff by using a ConditionVariable, but other than that, it should be easy to use.

Timeout still happening in Heroku after using rack-timeout.
Using rack-timeout (0.0.4).
In config / initializers / timeout.rb

Rack::Timeout.timeout = 19 # seconds

On Heroku I'm getting random errors like this:

ActionView::Template::Error (Timeout::Error: execution expired: SELECT blah blah)

and:

ActionView::Template::Error (PGError: SSL SYSCALL error: Connection timed out)

and

ActionView::Template::Error (PGError: SSL SYSCALL error: EOF detected)

I'm not sure the 3 are related.

I'm starting to think Rack::Timeout is involved. I'm thinking that it times out during a database query and then doesn't properly close the database connection.

What is the recommended way to make sure Rack::Timeout properly closes the database connection when a timeout is reached?

These errors can leave my site in a "down" state (all 16 of my unicorn processes stop functioning, I believe because they can no longer connect to the database). Restarting the app fixes this problem.

I'm using Heroku Postgres.

Thanks,

Ben

Hi guys, any reason to not release 0.1.0? Just curious if there was an ongoing discussion I didn't see. I've been using tag: 'v0.1.0beta3' in production for a few months...

Platform: Heroku
App: Rails
Expectation: UUID formatted value for rack-timeout log lines
Actual: 32 bytes of random hex

Is it possible that this lib is not reading the correct header value? This devcenter article states the header named X-Request-ID should be used. However, this lib only reads HTTP_HEROKU_REQUEST_ID and HTTP_X_REQUEST_ID.

Could that be changed ?
It generates a lot of useless noise I feel

Where is that undefined method being called?

We're using Rack::Timeout on Heroku to handle slow responses. However it appears to inject the timeout error into the executed SQL on some slow requests, giving us errors like:

ActiveRecord::StatementInvalid: Timeout::Error: execution expired: SELECT COUNT(*) FROM `discussions` WHERE `discussions`.`lesson_id` = 402991 AND `discussions`.`excluded` = 0 AND `discussions`.`style_cd` = 1

What is the correct way to avoid this, or recover from this?

I was recently bitten by an issue that occurred when rack-timeout raises Rack::Timeout::RequestTimeoutError during the middle of an ActiveRecord query. It interrupts the processing of the request but the query is still being processed by the database. Since Rails is running in production mode, ActiveRecord::Base's singleton connection isn't reset before the next web request.

Since the original request was cancelled and the results of the query are never read, the database eventually closes the connection, raising "Mysql2::Error: closed MySQL connection" during the middle of the next request.

I feel like this is probably a pretty common gotcha, and it's mentioned would be worth giving an example in the README to save other developers the same headaches.

This is what I'm doing:

# controllers/application_controller.rb
class ApplicationController < ActionController::Base

  rescue_from Rack::Timeout::RequestTimeoutError, :with => :handle_timeout
  #...
  protected

  def handle_timeout(exception)
    # Send exception report to service for tracking

    # If the timeout occurs during the middle of a MySQL query, we need to cancel the
    # query so that "Mysql2::Error: closed MySQL connection" isn't raised in the middle
    # of a subsequent request
    ActiveRecord::Base.connection.reset!

    # Render error page
    respond_with_error_status(503)
  end

rack.logger is a good default; rails's is a bit more of a shitshow to work with and I don't really wanna special-case it.

After trying the 0.1.0 beta 2, I noticed this strange log, that I thought not possible.

I simulated the timeout by adding a 'sleep 5' to a rails controller action, which took already 5 or 6 seconds.

I use Foreman and here are the logs

2:09:25 web.1  | source=rack-timeout id=604f1e4270dc2af45d95afed93aaef71 timeout=10000ms state=ready at=info
12:09:37 web.1  | source=rack-timeout id=604f1e4270dc2af45d95afed93aaef71 timeout=10000ms duration=11387ms state=completed at=info
12:09:37 web.1  | 127.0.0.1 - - [13/May/2013 12:09:37] "GET /url HTTP/1.1" 500 - 11.6862

development.log showed this

Completed 500 Internal Server Error in 9004ms

NoMethodError - undefined method `result' for #<Rack::Timeout::RequestTimeoutError: execution expired>:

Hi,

I've been seeing some strange timeouts on Heroku ever since I upgraded to the latest rack-timeout.

Here is a screenshot from New Relic that shows Rack::Timeout#call being responsible for the high response time:

And here is the app: https://github.com/smcgov/ohana-api-smc

I'm deleting the gem for now.

Hello,

Heroku advised me to update my gem, but to do so I have to read through all your recent commits, which can lead to interpretation mistakes on my end.

It would be really great if you maintained a changelog for this gem. It would be really helpful.

Thanks

Watching help app, the age metric seems to imply there's a lot of queuing - anywhere between 100ms and 1000ms, but that seems unlikely. I think there might be a rounding/truncation bug in rack-timeout.

Note that I'm using 0a6dc7c (0.1.0beta2) due to the Rack::Lint issue, so perhaps this is already addressed.

After upgrading to rack-timeout 0.1.1 we get a bunch of log output like this in development when loading a single page:

https://gist.github.com/barmstrong/56481bcf7de9c29838fd

What does this output mean and any idea why so much of it? Thanks!

Hi!

I'm using rack-timeout 0.2.0 on Heroku with Rails.I get this error from time to time:

2015-03-12T08:30:11.110453+00:00 app[web.2]: E, [2015-03-12T08:30:11.110420 #12] ERROR -- : /app/vendor/bundle/ruby/2.2.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:670:in `worker_loop'
2015-03-12T08:30:11.110484+00:00 app[web.2]: E, [2015-03-12T08:30:11.110450 #12] ERROR -- : /app/vendor/bundle/ruby/2.2.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:525:in `spawn_missing_workers'
2015-03-12T08:30:11.110514+00:00 app[web.2]: E, [2015-03-12T08:30:11.110481 #12] ERROR -- : /app/vendor/bundle/ruby/2.2.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:140:in `start'
2015-03-12T08:30:11.110577+00:00 app[web.2]: E, [2015-03-12T08:30:11.110512 #12] ERROR -- : /app/vendor/bundle/ruby/2.2.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>'
2015-03-12T08:30:11.110611+00:00 app[web.2]: E, [2015-03-12T08:30:11.110576 #12] ERROR -- : /app/vendor/bundle/ruby/2.2.0/bin/unicorn:23:in `load'
2015-03-12T08:30:11.110640+00:00 app[web.2]: E, [2015-03-12T08:30:11.110608 #12] ERROR -- : /app/vendor/bundle/ruby/2.2.0/bin/unicorn:23:in `<main>'
2015-03-12T08:30:11.111314+00:00 app[web.2]: source=rack-timeout id=84784c36-ceb1-41cb-a6c4-d0a8fbfb7761 wait=30696ms timeout=30000ms state=expired
2015-03-12T08:30:11.160040+00:00 app[web.2]:
2015-03-12T08:30:11.160044+00:00 app[web.2]: ActionView::Template::Error (Request ran for longer than 1.8885111324057604 seconds.):

So the request ran for 30696ms when the timeout is 30000ms, but the error says it ran for longer than 1.8 seconds, which is wrong.

Also, a weird part is that I have this in an initializer:

Rack::Timeout.timeout = 22 if Rails.env.production?

Am I doing something wrong, or did I catch a bug?

Thanks!

I'm running a Rails 4.1.6 app on Heroku, developing locally on OSX Yosemite, and I found an issue that might be a regression:

Upon updating my app to use v0.1.0 from v0.0.4, my test suite started hanging on some of my Capybara feature tests. If I roll back to using v0.0.4, my test suite takes under 45 seconds to run. If I re-update to v0.1.0, my test suite never seems to finish (I cut it off after 11 minutes). The slow/delayed tests are all Capybara features. If I mix up the order it doesn't seem to make a difference, but the issue only affects tests that include loading some content by Ajax.

The issue seems to have no effect on my application in staging (running on Heroku), it only manifests itself during tests.

I'm using Capybara 2.4.4 and Capybara-webkit 1.3.0 with RSpec 3.1.7.

per interagent/pliny#142 (comment)

cc @brandur

When the application times out, should a 504 Gateway Timeout be returned to the client?

Hi,

I am trying to telling user the user-friendly message if the request takes long time. However, when I try to resuce it from rescue_from method, I never get the Rack::Timeout::RequestTimeoutError.

I tried to debug it, I see that the error raised is ActiveRecord::StatementInvalid instead of Rack::Timeout::RequestTimeoutError and in the exception message, there is Rack::Timeout::RequestTimeoutError.

   150    def render_500(ex)
   151      debugger
=> 152      logger.error { "System Error catched: #{ex.message}\n\t#{ex.backtrace.join("\n\t")}" }
   153      @error_url = request.url
   154      @ex = ex
   155      @remind_msg = "<p>You may want to go to <a href='/'>Home</a> and try again.</p>"
   156      render "500", :layout => 'error_page'
(rdb:1) e ex.class
ActiveRecord::StatementInvalid
(rdb:1) e ex.message
"Rack::Timeout::RequestTimeoutError: Request ran for longer than 10 seconds.: xxx

rack-timeout-0.2.3 on Heroku with jRuby 1.7.19

Rack app error: #<NoMethodError: undefined method `info' for true:TrueClass>
/app/vendor/bundle/jruby/1.9/gems/rack-timeout-0.2.3/lib/rack/timeout/logger.rb:46:in `log_state_change'
org/jruby/RubyProc.java:271:in `call'
/app/vendor/bundle/jruby/1.9/gems/rack-timeout-0.2.3/lib/rack/timeout.rb:176:in `notify_state_change_observers'
org/jruby/RubyArray.java:1613:in `each'
/app/vendor/bundle/jruby/1.9/gems/rack-timeout-0.2.3/lib/rack/timeout.rb:176:in `notify_state_change_observers'
/app/vendor/bundle/jruby/1.9/gems/rack-timeout-0.2.3/lib/rack/timeout.rb:153:in `_set_state!'
/app/vendor/bundle/jruby/1.9/gems/rack-timeout-0.2.3/lib/rack/timeout.rb:115:in `call'
/app/vendor/bundle/jruby/1.9/gems/newrelic_rpm-3.11.1.284/lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
/app/vendor/bundle/jruby/1.9/gems/newrelic_rpm-3.11.1.284/lib/new_relic/agent/instrumentation/middleware_tracing.rb:63:in `call'
/app/vendor/bundle/jruby/1.9/gems/puma-2.11.1-java/lib/puma/configuration.rb:82:in `call'
/app/vendor/bundle/jruby/1.9/gems/puma-2.11.1-java/lib/puma/server.rb:507:in `handle_request'
/app/vendor/bundle/jruby/1.9/gems/puma-2.11.1-java/lib/puma/server.rb:505:in `handle_request'
/app/vendor/bundle/jruby/1.9/gems/puma-2.11.1-java/lib/puma/server.rb:375:in `process_client'
/app/vendor/bundle/jruby/1.9/gems/puma-2.11.1-java/lib/puma/server.rb:371:in `process_client'
/app/vendor/bundle/jruby/1.9/gems/puma-2.11.1-java/lib/puma/server.rb:262:in `run'
org/jruby/RubyProc.java:271:in `call'
/app/vendor/bundle/jruby/1.9/gems/puma-2.11.1-java/lib/puma/thread_pool.rb:104:in `spawn_thread'

I'm getting the following error:

using 0.1.0 on JRuby 1.7.16

NoMethodError: undefined method `state=' for nil:NilClass
  rack/timeout.rb:148:in `_set_state!'
    env[ENV_INFO_KEY].state = state
  rack/timeout.rb:98:in `call'
    RT._set_state! env, :active
  org/jruby/RubyKernel.java:1501:in `loop'
  rack/timeout.rb:94:in `call'
    loop do

Hi, I see you recently changed the error message to add more info. I think that's a step in the right direction, but it introduced a new problem. Specifically, for anyone that uses error tracking systems, they typically group the errors by name. So for example, the following are treated as different, even if they are both more or less the same:

a. Rack::Timeout::RequestTimeoutError: Request waited 21ms seconds, then ran for longer than 25000ms
b. Rack::Timeout::RequestTimeoutError: Request waited 22ms seconds, then ran for longer than 25000ms

I have two proposals that would avoid this:

1. Change the error message to round to the nearest second, so that you get things like:

a. Rack::Timeout::RequestTimeoutError: Request waited less than 1 seconds, then ran for longer than 25s
b. Rack::Timeout::RequestTimeoutError: Request waited 1 second, then ran for longer than 24s
c. Rack::Timeout::RequestTimeoutError: Request waited 20 seconds, then ran for longer than 5s

1. Let people define their own customizable method for how the error message should appear, with those waiting times as variables. Then when you call the error, use that (or the default).

Let me know what you think.

Here is my problem:

I want to provide great customer service. When a customer sends my server a request, and for whatever reason the request is halted by the rack-timeout app, I would like to print the user portion of their basic authorization header in the rack-timeout log line. This way I can follow up with the customer to resolve any potential issues.

It was not immediately clear to me how I might go about this. Any suggestions?

Here is My current way of handling
But if the app thread has not entered rails stack yet, I cannot rescue and return 503
Which is not expected

I have payment system in my application and sometimes bank request take too much time and application throws RequestTimeOutError. Is it possible to extend timeout for a specific requests or urls like in this comment http://stackoverflow.com/a/17574585/2232794

I have method that sometimes runs long, and I'm trying to rescue it from a timeout. I can't for the life of me get it to rescue. Here's my rescue block.

rescue Rack::Timeout::Error => e
DebugLogger.instance.log("*********************** Rack Timeout Rescue")
raise e

rescue Rack::Timeout::RequestTimeoutError => e
DebugLogger.instance.log("*********************** Rack Request Timeout Rescue")
raise e

rescue Timeout::Error => e
DebugLogger.instance.log("*********************** Timeout Rescue")
raise e

rescue => e
DebugLogger.instance.log(sprintf("got error get objects %s - %s", e.class, e.message))
return request.objects.all;
end

I'm trying to rescue everything there. I eventually get rescued by the last block with this message. "got error get objects Excon::Errors::SocketError - execution expired (Rack::Timeout::RequestTimeoutError)"

Should I be rescuing that?

Spin off from #55

cc @abevoelker, @vasilakisfil, @seanknox

These errors might be worth looking into separately from the capybara issue. Were you all using capybara?

Please list your your server, framework, versions, dependencies here. A full trace on that ThreadError would be great too if you can capture one.

I'm trying to observe timeouts so I can extend a request. The observer block I have will print my test statement, but it doesn't do it until after the request is done. Shouldn't it print every time the timeout happens? I have my timeout set for 5 seconds for testing and the request I just ran took about a minute. Here is my observer block.

Rack::Timeout.register_state_change_observer(:timeout_check) { |env|
puts "*********************** Got timeout error in block"
}

Also, given a 5 second timeout, would this block be accessed every 5 seconds?

Hello,

I was wondering if it was possible to set a different timeout per request ? For example I have a rails app with a /api namespace. I would like the timeout to be around 20 seconds for api requests. On the other hand, on all other requests I'd like it to be around 3 seconds. It this possible ?

I am investigating thi gem because at the moment we're using unicorn a timeout are not really smartly handled since it involves killing the rails process and forking another time whenever it happens.

Thanks !

Hi - I've been running into a problem recently with Rack::Timeout's errors killing Puma workers (since they now descend from Exception rather than StandardError and are not caught by most rescue blocks).

I've worked around this with a quick rack middleware to catch Rack::Timeout's errors and wrap them in StandardErrors:

class TimeoutRecovery
  def initialize(app, options = {})
    @app, @options = app, options
  end

  # Rack::Timeout, as of 0.2.4, raises errors that descend from the Exception
  # class when a request takes too long - however, this is a very strong
  # exception level and will kill Puma's worker if we let it continue down the
  # stack. So instead, catch it and wrap it in a recoverable StandardError.

  class Error < StandardError; end

  def call(env)
    @app.call(env)
  rescue Rack::Timeout::Error => e
    raise Error, "Original Error: #{e.class.inspect}, #{e.message.inspect}, #{e.backtrace.inspect}"
  end
end

It seems like this is something that Rack::Timeout should be doing, though. Unless I've missed something and this is actually a terrible idea. Sure, we should be avoiding long request times where possible, but when they come up they're leaving Puma's workers crashed and unable to serve requests at all until a restart.

Hi @kch!

I'm trying to add a test in a small rails app to ensure that a Timeout::Error is raised for requests taking longer than 30 seconds. It works in production and development mode when tested manually, but I'm not having any luck in Rails' test environment:

I have the following set in config/initializiers/timeout.rb:

Rack::Timeout.timeout = 30

Simple test:

  test "requests should timeout" do
    Rack::Timeout.stubs(:timeout).returns(1)

    assert_raises TimeoutError do
      get :overview # set to sleep(2)
    end

    Rack::Timeout.unstub
  end

$ rake test
.........FFFUUUUU
Timeout::Error expected but nothing was raised

Do have any thoughts?

Cheers,
Lee 🍻

I'm running a Rails 4.1 application that's fairly simple, connecting to Postgres. In both development and production environments I'm having problems with requests timing out. When I'm serving with Unicorn, the unicorn timeout can kill a process appropriaetly (though I don't get the exception sent out to logs or NewRelic or anything). I installed rack-timout, and it's "working" but it's unable to kill the requests.

user@1dde41d2aee9:/rails# rake middleware
use Rack::Timeout
use Librato::Rack
use Rack::Sendfile
use #<ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x007f34cd513ac0>
use Rack::Runtime
...

#config/initializers/rack_timeout.rb
Rack::Timeout.timeout = 1

#merged unicorn_rails & application logs - stdout
source=rack-timeout id=075e816b5792ba35125ac51f55b3b61f timeout=1000ms state=ready at=info
I, [2014-05-03T16:53:11.425984 #1063]  INFO -- : Started GET "/markets/3" for 70.114.154.225 at 2014-05-03 16:53:11 +0000
I, [2014-05-03T16:53:11.427974 #1063]  INFO -- : Processing by MarketsController#show as HTML
I, [2014-05-03T16:53:11.428099 #1063]  INFO -- :   Parameters: {"id"=>"3"}
source=rack-timeout id=075e816b5792ba35125ac51f55b3b61f timeout=1000ms duration=1000ms state=timed_out at=error
E, [2014-05-03T16:54:12.188483 #1060] ERROR -- : worker=0 PID:1063 timeout (61s > 60s), killing
E, [2014-05-03T16:54:12.216382 #1060] ERROR -- : reaped #<Process::Status: pid 1063 SIGKILL (signal 9)> worker=0

I suspect it's a postgres IO issue that I'm hunting (rails/rails#12867), but just opening this because it seems like rack-timeout should be able to kill this.

The following code from our initializer is causing my cucumber features to fail:

Rack::Timeout.logger.level = ::Logger::WARN

The error message is:

$ cucumber
undefined method `level=' for nil:NilClass (NoMethodError)
myapp/config/initializers/timeout.rb:1:in `<top (required)>'

Anyone know how to fix this? Commenting out the initializer code resolves the problem.

Hi,
I added rack-timeout (v. 0.2.1) and saw extreme slowdown (ie 2s to almsost 20s) of requests on development.

Some notable gems I am using: Rails 4, Unicorn, Pry.

Here is my complete Gemfile.lock
https://gist.github.com/hakunin/aa57c0576f6489ff0a90

Removing rack-timeout solves the issue.

I should also mention that the page is computation intensive, is there anything listening on ruby stack?

Off latest master:

NoMethodError at /
undefined method `size' for #<Rack::Lint::InputWrapper:0x00000006376fa8>
Ruby    help/.bundle/ruby/2.0.0/bundler/gems/rack-timeout-631f3af1715d/lib/rack/timeout.rb: in call, line 30 

Due to http://bugs.ruby-lang.org/issues/show/4266 [1], this library doesn't work with downstream multithreaded code that relies on Mutex#synchronize on MRI 1.9.X. Eventually you'll get ThreadError: deadlock; recursive locking exceptions.

It's a bit ironic, cause the README explicitly calls out that your app must be thread-safe, yet it's this library that isn't 😃

[1] bugs.ruby-lang.org is down, but you can read about it at https://groups.google.com/forum/?fromgroups=#!topic/ruby-core-google/bnkrqGTCODw

rack-timeout is big and scary now, interacts with many ruby versions and multiple frameworks, and follows many different code paths depending on its various settings. That means we're in dire need of testing.

We got some good stuff from @chadbailey59 in the cb-appraisals branch, which I've synced with current master in the kch-testing-from-cb-appraisals branch. This is great because it tests with various frameworks/versions, but I find it a tad convoluted and involves a ton of dependencies ATM. Would love to find a way to trim it down.

We have a much simpler but also very good approach in #48 from @JackDanger, synced to current master in the kch-testing-from-jackdanger-patch branch. This only tests within rack itself, bypassing all the http and framework stuff. I like this for most testing, but in addition to that I want integration tests like chad's, or something even more external, maybe using libcurl and static test apps per framework.

So both are good starts but a lot more work is needed. After 0.1.0 is out (#38), this is our top priority.

Especially, aborting mid sql operation can hose sql connections that need to be reset. We'll only provide the means to perform cleanup actions, but no specific cleanup code.

That said, because the SQL case is so common, documentation should provide an example for Rails/ActiveRecord. Such sample code is welcome in the comments.

Something around observers is probably the best option here. See #40 for discussion.

Our app: Rails 4.1.10, Ruby 2.1.2, Heroku's cedar-14 stack.

We've been getting a few (sometimes dozens) of these per day since we added rack-timeout (with a timeout of 28 seconds):

Request ran for longer than 0.05695550888818346 seconds
Request ran for longer than 1.414391502891359 seconds
Request ran for longer than 0.5001389309041748 seconds.
...and others like that.

And there doesn't seem to be a pattern on what action causes this. It seems to happen randomly.

Very few times we do get the expected 28 secs timeout which is what we put rack-timeout for.

Any clues about what could be causing this?

The Gem's readme instructions for Rails apps, aren't working properly for rails apps.

After adding the rack-timeout gem to application's Gemfile and configuring the initializer, I tested a sample application on Heroku, forcing a timeout, and no exception was logged or thrown.

The state of sample app is:

• the request to site#index took 15s
• the rack-timeout is set to 10s
• the request is served, no error is thrown or logged;

The only way to get it to work as expected, is adding use Rack::Timeout on application config.ru file.

But middlewares should go into config/application.rb for Rails apps.

And this should be happening here: https://github.com/kch/rack-timeout/blob/master/lib/rack-timeout.rb#L7

Obs.:

• just to complement, the application server is Puma, 2.8.2. Can this be the cause, since Puma does not have a timeout configuration?
• puma/puma#160
• puma/puma#265

Looking here, I don't think so.

Something is missing?

Is it possible to enforce/skip timeouts only for specific request urls using rack-timeout?

probably should for compatibility

Hi -- first of all thanks for this project. It's nice to see activity pick up again, looks like some really nice new features are on the way! Here's some odd behavior I've seen, perhaps someone can help me figure out what's going on.

I've been running 0.0.3 for many months with no problems in particular. A few weeks ago I upgraded to 0.0.4 and my app started to experience periodic downtime. I did a diff between the two versions, and the only difference of substance was that 0.0.4 injects the middleware at the very beginning of the stack. My development middleware stack under each version is illustrated at the bottom of this ticket. (I know that there have been a ton of changes since 0.0.4, but since this is such a small change that had a big difference with my app, we can still discuss it to see if it causes a problem -- unless of course a recent change actually addresses this).

Reverting to 0.0.3 solved the problem immediately.

Here's the bad behavior: some requests (no pattern than I could discern) take a long time, and then render my app's general 500 page. Now, I have a rescue_from in my application controller which catches Exception, reports it to Airbrake/errbit, and then renders a 500. So, when the exception being injected into the rack-timeout thread raises up to the controller, it should be reported and then the user is shown 500. Indeed, this is the sort of thing that happened under 0.0.3 from time to time. Reqest times out, is reported to Airbrake/errbit, dyno continues serving as normal.

However, in these mysterious cases under 0.0.4, the error was never reported to Airbrake/errbit. What is extra strange about this is that the only time the app renders our custom 500 page (in public/500.html) is after reporting the error to Airbrake/errbit. And when I tested a manually-invoked error, it did indeed have this behavior.

So, summing it all up: for some reason, due to the middleware position change, some requests result in a 500 response with all of these characteristics:

• outside of my controller rescue_from logic, where my manual handling can't catch it
• outside of the Airbrake exception handling logic, which @kch said would still report all errors in the comments here: 2888a5e
• results in rails automatically rending my 500 page
• results in my dyno show the 500 page for all requests, until it is restarted. (not 100% sure on what the pattern here is, but that seems to be the case. fyi, i'm using thin)

phew!

Any ideas?

use Rack::MiniProfiler
>>> Rack::Timeout 0.0.4 <<<
use Airbrake::UserInformer
use Rack::NoWWW
use ActionDispatch::Static
use Rack::Lock
use #<ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x007fae98f1f970>
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use BetterErrors::Middleware
use Airbrake::Rails::Middleware
use ActionDispatch::RemoteIp
use ActionDispatch::Reloader
use ActionDispatch::Callbacks
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use ActionDispatch::Cookies
use ActiveRecord::SessionStore
use ActionDispatch::Flash
use ActionDispatch::ParamsParser
use ActionDispatch::Head
use Rack::ConditionalGet
use Rack::ETag
use ActionDispatch::BestStandardsSupport
>>> Rack::Timeout 0.0.3 <<<
use HireFireApp::Middleware
use RequestStore::Middleware
use Bullet::Rack
use MetaRequest::Middlewares::MetaRequestHandler
use MetaRequest::Middlewares::Headers
use MetaRequest::Middlewares::AppRequestHandler
use OmniAuth::Builder
run Main::Application.routes

I ran into this bug rather frequently on a high traffic site with a somewhat high incidence of timeouts, using Timeout.timeout, very similarly to rack-timeout. It was happening almost daily - a request would time out and raise an exception at a place in ActiveRecord that left a model class in an unclean state; all subsequent requests in the same Unicorn worker that used the corrupted class would crash until the worker was restarted, and who knows what else was corrupted in there but wasn't raising an error.

I've taken to doing the following in an around_filter instead. This might be a good thing to do in rack-timeout, or at least have a note in the README about this possibility, since with enough timeouts you're bound to run into something like it sooner or later. Note that this code is not rescuing from Timeout::Error because if the timeout happens inside a Rails view, it will be caught and re-raised as a ActionView::Template::Error.

  def exit_process_on_timeout(timeout, &block)
    # Ruby's Timeout can leave the process in a bad state since libraries tend
    # not to anticipate having an exception raised arbitrarily.
    # So we assume this process is corrupted and exit it after letting the request finish.
    Timeout.timeout timeout, Timeout::Error, &block
  rescue => e
    if e.message == 'execution expired'
      ::NewRelic::Agent.notice_error e, request: request
      ::NewRelic::Agent::Transaction.stop
      ::NewRelic::Agent.shutdown
      Process.kill 'QUIT', Process.pid
    end

    raise e
  end

I noticed there's a lot of this in the log:

source=rack-timeout id=874bf8 timeout=29000ms state=ready
source=rack-timeout id=874bf8 timeout=29000ms service=0ms state=active
source=rack-timeout id=874bf8 timeout=29000ms service=1ms state=completed
source=rack-timeout id=3bbc38 timeout=29000ms state=ready
source=rack-timeout id=3bbc38 timeout=29000ms service=0ms state=active
source=rack-timeout id=3bbc38 timeout=29000ms service=1ms state=completed

Is there a way to silence rack-timeout, except errors, without changing my log level and affecting other log data?

https://devcenter.heroku.com/admin/articles/feedback/1244

Your last three versions are not using semantic rules. This means that by default, version 0.0.4 is loaded by default. Having to explicitly specify a version in a Gemfile locks the application to that specific version.

Any edge features not considered ready for public consumption should be made available in a separate branch instead of a tag.

Thanks!
George