DelayedJob Monitoring Infinite Loop in development environment about appsignal-ruby

[thijsc]

Are you running DJ with something like script/rails delayed_job start? If so this is probably caused by a compatibility issue with the Daemons gem that DJ uses under the hood we're currently in the process of debugging and fixing.

There's some more information here: thuehlinger/daemons#47

[cec]

Hi @thijsc,

in production I start DJ from capistrano using this gem which calls
bundle exec script/delayed_job -n 1 --pool=.... --pool=.... restart

in development I start DJ with the rake task jobs:work

The problem occurred only once in production, but is consistent in development.

[cec]

Hi @thijsc ,

I got more information about the problem, specifically, I found out why it happens in development and not in production.

I had an error in my Gemfile:

group :development, :test do
  gem 'factory_girl_rails'
  gem 'rspec'
  gem 'rspec-rails'
  gem 'rspec-collection_matchers'
  gem 'shoulda-matchers'
  gem 'database_cleaner'
  gem 'timecop'
  gem 'sunspot-rails-tester
  gem 'capybara'
  gem 'launchy'
  gem 'capybara-screenshot'
  gem 'capybara-webkit'
end

Probably because of a copy-paste, I was requiring spec gems in development.
This resulted in Appsignal::Transaction::GenericRequest to have its ancestors chain become

[
"Appsignal::Transaction::GenericRequest", 
"Object", 
"ActionDispatch::TestProcess", 
"Delayed::MessageSending", 
"ActiveSupport::Dependencies::Loadable", 
"FriendlyId::ObjectUtils", 
"JSON::Ext::Generator::GeneratorMethods::Object", 
"PP::ObjectMixin", 
"Kernel", 
"BasicObject"
]

instead of this:

[
"Appsignal::Transaction", 
"Object", 
"Delayed::MessageSending", 
"ActiveSupport::Dependencies::Loadable", 
"FriendlyId::ObjectUtils", 
"JSON::Ext::Generator::GeneratorMethods::Object", 
"PP::ObjectMixin", 
"Kernel", 
"BasicObject"
]

This led Appsignal's code to expect GenericRequest to have a session, which resulted in an infinite loop (for some reason I cannot fathom).

While running specs, Appsignal is not meant to be active.
I'm monitoring my dev environment for a bunch of reasons, but most people won't face this issue at all.

From my point of view this issue is closed.
In case people make my same mistake, this could save quite a headache cause for me it was hard to track down.
For those facing the same issue, keep in mind to stop spring between trials!

Cheers and kudos for your awesome work!

[thijsc]

I will take a look at that infinite loop. That shouldn't be possible, even in weird circumstances. Leaving this issue open for that.

Thanks for this great debugging session!

[cec]

I see, glad it helps you guys too :)

[thijsc]

I've had trouble reproducing this. I think it might be relevant only in this specific scenario when using Spring. Closing this issue.

[conarro]

Just ran into the same underlying issue (Stack level too deep related to request.session), but not related to DJ.

Environment

• Rails 5.2
• Ruby 2.3
• Appsignal 2.10.7

Reproduction steps

• In appsignal.yml: set Appsignal active: true in the test environment, and ensure skip_session_data is true (this is the default, so if it isn't defined in your config that should work)
• Add include ActionDispatch::TestProcess globally in specs (e.g. via rails_helper) -- we had this in place to make fixture_file_upload available to use in factories
• If you don't have one, write a test to ensure Appsignal works. For us, we had a class reporting a handled error (via Appsignal.send_error), and we had a spec for that code to ensure the errors got sent to Appsignal. E.g. for RSpec: expect(Appsignal).to receive(:send_error).with(StandardError)
• Run the spec, see Stack level too deep errors (with lib/action_dispatch/testing/test_process.rb in the stack trace)

How we fixed it

• Some Googling (query: actionpack request session test stacklevel) brought me to this SO post, which indicates a similar problem, the underlying cause being the use of include ActionDispatch::TestProcess
• A link from that SO post goes here activeadmin/activeadmin#512, which has this comment indicating a similar thing
• With this info, removed our include ActionDispatch::TestProcess in rails_helper.rb and replaced our use of fixture_file_upload in factories to use Rack::Test::UploadedFile.new

---

This doesn't feel like an Appsignal-specific issue, but I wanted to post here for future reference.