Notifications not delivered about rapns

[ileitch]

That doesn't sound good. What Ruby version are you using? Can you see if the daemon is still checking for notifications? You should see the SQL queries in your log.

[simo2409]

I'm using it on Engine Yard cloud, I was using ruby 1.9.3p125, yesterday I upgraded to ruby 1.9.3p194 and by now problem didn't happen anymore (but I think it will happen)

I will let you know

[ileitch]

Anything to report?

[simo2409]

We have another kind of problem now, sometimes daemon get "crazy" and starts to send continuously 1 notification (until we kill it).

I will let you know (but the original problem looks passed away)

[macteo]

I'm a teammate of simo2409.

We tracked down the initial issue and it seems to be related with the number of rapns applications we need to use.

The error we get is the following

ActiveRecord::ConnectionTimeoutError, could not obtain a database connection within 5 seconds (waited 5.000238102 seconds). The max pool size is currently 5; consider increasing it.
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:252:in `block (2 levels) in checkout'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:236:in `loop'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:236:in `block in checkout'
/usr/lib64/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:233:in `checkout'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:96:in `block in connection'
/usr/lib64/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:95:in `connection'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:404:in `retrieve_connection'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_specification.rb:170:in `retrieve_connection'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_specification.rb:144:in `connection'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/newrelic_rpm-3.4.2/lib/new_relic/agent/method_tracer.rb:493:in `block in transaction_with_trace_ActiveRecord_self_name_transaction'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/newrelic_rpm-3.4.2/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/newrelic_rpm-3.4.2/lib/new_relic/agent/method_tracer.rb:488:in `transaction_with_trace_ActiveRecord_self_name_transaction'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `save!'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/rapns-2.0.4/lib/rapns/daemon/delivery_handler.rb:54:in `block in deliver'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/rapns-2.0.4/lib/rapns/daemon/database_reconnectable.rb:15:in `with_database_reconnect_and_retry'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/rapns-2.0.4/lib/rapns/daemon/delivery_handler.rb:51:in `deliver'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/rapns-2.0.4/lib/rapns/daemon/delivery_handler.rb:107:in `handle_next_notification'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/rapns-2.0.4/lib/rapns/daemon/delivery_handler.rb:34:in `block (2 levels) in start'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/rapns-2.0.4/lib/rapns/daemon/delivery_handler.rb:32:in `loop'
/data/manage_github/shared/bundled_gems/ruby/1.9.1/gems/rapns-2.0.4/lib/rapns/daemon/delivery_handler.rb:32:in `block in start' 

The kind people at Engineyard told us that it could be related to use of threading in rapns and ActiveRecord connection pool fairness.

Actually we don't have many notifications to deliver, but we do have many applications with few messages for each one. But we do expect to have many applications and many messages.

This is the reply on Engineyard support

Do you have lots of messages that are attempted to be delivered? This can cause the SQL requests to wait for connection availability and result in these timeouts that you.

You can modify the pool size with a custom chef recipe, but you may want to keep this down on the application servers themselves, as can max the available connections on your database server (This is 300 connections by default.)

Alternatively, it may be worth adding in some kind of limitation on the number of concurrent threads that can be delivering messages at the same time. I noticed that there is a DeliveryHandlerPool, but I'm not familiar with rapns in detail, so i'm not sure on the usage of this.

Do you have any suggestion?

Best regards,
Matteo

[ileitch]

Correct. rapns maintains persistent connections to the APNs, each of these consumes a single thread.

It is not possible currently to share database connections among workers, this would need to be a new feature.

As a possible temporary work around, you can increase the pool parameter in database.yml. However this will also increase the number of threads available for your app processes, causing many postgres processes.

This is a feature I will look into implementing.

[ileitch]

This was implemented in 2.0.5.

[masterkain]

So, I'm using rapns 3.4.1.

What just happened is that I kept getting a push message in my app, I went to investigate to find rapns erroring out with this:

==> apps/myapp-rapns/current/log/rapns.log <==
[2013-09-10 00:54:08] [ERROR] ActiveRecord::ConnectionTimeoutError, could not obtain a database connection within 5 seconds (waited 5.000286526 seconds). The max pool size is currently 5; consider increasing it.
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:258:in `block (2 levels) in checkout'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `loop'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `block in checkout'
/home/ubuntu/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:239:in `checkout'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:102:in `block in connection'
/home/ubuntu/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:101:in `connection'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/activerecord-3.2.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:129:in `with_connection'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/store/active_record/reconnectable.rb:17:in `with_database_reconnect_and_retry'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/store/active_record.rb:45:in `mark_batch_delivered'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/batch.rb:81:in `complete_delivered'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/batch.rb:69:in `block in complete'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/batch.rb:67:in `each'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/batch.rb:67:in `complete'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/batch.rb:52:in `block in notification_processed'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/batch.rb:50:in `synchronize'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/batch.rb:50:in `notification_processed'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/delivery_handler.rb:47:in `handle_next_notification'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/delivery_handler.rb:13:in `block (2 levels) in start'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/delivery_handler.rb:12:in `loop'
/home/ubuntu/apps/myapp-rapns/shared/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/delivery_handler.rb:12:in `block in start'

The pool size defined in the database.yml is 70. This is the first time I'm seeing this happening.

Is this related?

[ileitch]

70? Is it defined correctly? The error suggests it's still 5.

How many apps do you have, and how many connections in total?

[danpe]

Are you using Heroku? if so, Heroku ignore your database.yml and sets the pool to 5 automaticly.
To fix that create a new initializer:

Rails.application.config.after_initialize do
  ActiveRecord::Base.connection_pool.disconnect!

  ActiveSupport.on_load(:active_record) do
    config = Rails.application.config.database_configuration[Rails.env]
    config['pool']              = ENV['DB_POOL']      || 70
    ActiveRecord::Base.establish_connection(config)
  end
end

[masterkain]

Hello,
@danpe no, this is an Ubuntu staging server.
@ileitch I have two apps in the rapns tables, one for GCM and one for APN. I'm not sure about the number of connections since I rebooted rapns and it started working again.

The pool is 70 because I reused my usual database.yml for my main rails app. It gets copied during a normal Capistrano deploy.

I don't know either why the pool is being shown as 5. What I did in order to deploy rapns in staging (production is on Heroku) is to create a blank rails app and configure rapns on there, but as I already mentioned the database.yml is a copy of the main Rails app. The database.yml contains only one "staging" entry, so if it was picking some other setting beside staging I expect an exception, so I'm quite sure it's using it.

Not sure where to start looking.

Edit: the database (postgresql) is running on localhost (same host). No extra initializers are present beside the default rails ones and the rapns one.

[chenghung]

rapns 3.4.1
I got the same problem, too
could not obtain a database connection within 5 seconds ...