Helpers autoloading about turbo-rails

[fxn]

Update

I have been able to understand why this is happening, my intuition is confirmed. Don't have the solution yet, but let me document the cause.

Fundamental pure Ruby gotcha

Let me first explain the gotcha in dynamic Ruby programming which is the root problem we are facing.

Let's imagine we have this:

class C
  include M
end

Cool. Since method lookup is conceptually dynamic in every call site, if we reopen M and add a method

module M
  def new_method
  end
end

C objects respond to new_method from then on. Lookup is dynamic.

OK. This mental model does not work if you add new ancestors to M. So, if you define new_method and instead of reopening M to add a method you reopen M to include a module:

module N
  def new_method
  end
end

module M
  include N
end

now C.new.new_method does not work. This is inconsistent, in my view. I discovered this many years ago precisely debugging some weird issue in Rails, and after hitting my head against a wall a few times.

I asked Matz back in the day about the reason for this, and he said it was performance. Ancestor chains are linearized, and if you add a new ancestor to a module, Ruby does not update the linearized cached ancestor chains of the affected existing classes or modules. Ruby does not keep backreferences registering "where was this module included". (On the other hand, Charles Nutter said this could be easily possible in JRuby.)

How's that gotcha happening in our situation?

Well, we have that situation in this issue. This is the order in which things happen:

1. A controller is loaded in an application initializer (we already said this is wrong, but we are understanding beyond that).
2. AC::Base::HelperMethods gets included in ApplicationController::HelperMethods.
3. turbo-rails includes the engines modules into AC::Base::HelperMethods (here).

If application initialization is correct, the order is different:

1. turbo-rails includes the engines modules into AC::Base::HelperMethods.
2. AC::Base::HelperMethods gets included in ApplicationController::HelperMethods.

See the difference? Due to the gotcha in Ruby explained above, in the first sequence ApplicationController::HelperMethods got included AC::Base::HelperMethods before that module was decorated by turbo-rails, so the decoration arrives too late.

Why is that not happening in development mode?

This is not seen in development mode because, as the warning says, any reloadable class autoloaded during initialization is unloded by Rails. The motivation for this is that during development you realize there was a latent unseen problem (eg, you set a class-level attribute that will be gone on reload), and you are that way forced to fix it. This has been that way since Rails 6 was released.

So, when you use the application in development mode, ApplicationController is reloaded, which gets AC::Base::HelperMethods included, and at this point the Turbo includes are in place, so you get them in the ancestor chain. This is a fresh class now, so you get the last thing.

What to do?

In Rails 7 the problem is solved, the application won't boot and you need to address the issue, it is no longer a warning you are not forced to address (not yet in main, will be).

For Rails < 7 we'll see in the next episode.

[rguiscard]

I have the exact issue execept I only need to include Turbo::StreamsHelper ( #73 ). So I follow the suggestions:

1. move TurboController to app/controllers
2. remove include Turbo::StreamsHelper from ApplicationHelper

And it works !!

Just to note that the reason to have TurboController in config/initializers/devise.rb is because it is used in this video GoRails. I assume many people will follow it and have similar issues later in production environment.

[fxn]

Heads up! I have been able to reproduce without Active Admin this weekend. We're a bit closer.

[fxn]

@kwent excellent, thanks! And wow, that app autoloads quite a bunch of reloadable stuff on boot. Regardless of this ticket, there should be really nothing autoloaded as the warning explains.

This is unrelated to Zeitwerk, BTW, this has been an unseen issue for years, people noticed when reloads didn't reflect changes in weird ways. In recent versions of Rails we are trying to raise awareness upfront of the potential problem there, and gradually go towards having clear and enforced rules about when you can refer to reloadable application code.

Alright. So, as I said above, regardless of the warning and the transition phase we are in, applications run fine even with warnings. This behavior was introduced a year and a half ago with Rails 6.

We are hitting an edge combination of things here in which the setup of this gem together with autoloading too early triggers that discrepancy in available helpers. I'll try to remove Active Admin from the equation and get a really minimal application that nails down the cause of it.

However, the real fix for you people, and the recommended thing to do towards Rails 7, is not to autoload reloadable application code during boot. Let the application run without warnings.

[kwent]

@fxn I have lot of production dependencies. Unsure i'm going to be able to do that.

[fxn]

@lpradovera I have seen something off in your application, but need more information to connect the dots.

There is an initializer autoloading reloadable constants, which has been deprecated and issuing a noisy warning since Rails 6.0, and it is going to be an error condition in Rails 7.0.

The file config/initializers/devise.rb has this controller definition:

class TurboController < ApplicationController
  class Responder < ActionController::Responder
    def to_turbo_stream
      controller.render(options.merge(formats: :html))
    rescue ActionView::MissingTemplate => error
      if get?
        raise error
      elsif has_errors? && default_action
        render rendering_options.merge(formats: :html, status: :unprocessable_entity)
      else
        redirect_to navigation_location
      end
    end
  end

  self.responder = Responder
  respond_to :html, :turbo_stream
end

which is autoloading the reloadable constant ApplicationController. If you edit ApplicationController, changes won't be reflected in TurboController.

Note that Devise asks you for a controller class name as a string:

config.parent_controller = 'TurboController'

precisely to instantiate the controller in a way that works well with reloading.

However, while this warning has been out for a year and a half, people do not get applications failing in production even if the warning is issued. Something remains to be understood. I'd suggest we do 2 things:

1. Please move your controller to app/controllers, remove the includes from ApplicationHelper, make sure the warning is not issued in development mode, and try again. Ideally in a pre-production environment if you have one.
2. On the other hand, could you please tell me how to reproduce the error locally using the current version of the code? Can we change config/environments/development.rb in a way that triggers the error visiting some page? I tried enabling cache_classes and eager loading, and the root path was a 200.

@kwent Do you see such warning in development mode? It looks like this.

[lpradovera]

@fxn I can confirm moving the controller fixes the issue, thanks for catching that. I haven't been able to make it happen outside of production, though.

[fxn]

🙌 thanks so much! I'm a bit busy, but I'll investigate as soon as I can.

[fxn]

Thinking about a fix for this took some reflection, because you have to do something improperly in order to be defensive. Question was, what do you sacrifice, which are the consequences? I think I have it and I'll submit a patch soon.

[jmadkins]

I'm running Rails 6.0.3.4 and I also had to add include Turbo::IncludesHelper to ApplicationHelper. Otherwise, I was getting the error below when running rspec.

ActionView::Template::Error: undefined local variable or method `turbo_include_tags' for #<#<Class>

[dhh]

I can't replicate this. Is this happening on a new app or on an existing one? Could you try to reproduce in a rails new myapp setup? Thanks.

[lpradovera]

I am running into a similar issue in production only, on an application deployed to Heroku that is pretty much a new application plus Devise and Hotwire. Exact same Gemfile lock as the above, Rails 6.1.

Running in rails c locally does not have the same issue.

Happy to give access or otherwise help.

[dhh]

I can't replicate this when I set eager_load = true. Maybe @fxn can help us see if we're missing something.

[fxn]

Let me throw one observation just quickly before I can dig into this.

This engine lazy loads the helpers. Specifically, it loads them in an initializer, and it waits for ActionController::Base to be loaded.

Does that ring any bell? Are you guys hitting ApplicationHelper in a way that uses the missing methods before ActionController::Base is loaded?

[fxn]

To be clear, Turbo::Engine.helpers.ancestors yields

[
  #<Module:0x00007fd10f0c1060>,
  Turbo::Streams::ActionHelper,
  Turbo::StreamsHelper,
  Turbo::IncludesHelper,
  Turbo::FramesHelper,
  Turbo::DriveHelper
]

The classes are autoloaded just fine, the problem here, I suppose, is one of order of execution.

[kwent]

I’m sorry I forgot to mention I was seeing the issue in production mode only. Regards

…

On Tue, Jan 5, 2021 at 8:18 AM Xavier Noria ***@***.***> wrote: To be clear, Engines.helpers yields [#<Module:0x00007fd10f0c1060>, Turbo::Streams::ActionHelper, Turbo::StreamsHelper, Turbo::IncludesHelper, Turbo::FramesHelper, Turbo::DriveHelper] The classes are autoloaded just fine, the problem here, I suppose, is one of order of execution. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGWZJA5FYHDEGGSFT5OIHLSYM3UBANCNFSM4VUH46VA> .

[fxn]

I guess we'd need a way to reproduce. @lpradovera do you think you could generate a minimal app + spec?

[kwent]

Not seeing the issue when i run https://github.com/kwent/turbo-rails-issue-64 with rails -s -e production locally 🤔

[fxn]

@kwent can you reproduce in your app locally in production mode?

[fxn]

@kwent So your app eager loads without errors, the server launches without errors, and you get the exception when you hit a template?

[kwent]

@fxn Correct. I hit the exception at runtime.

[fxn]

Let's see if @lpradovera can help us reproduce, either access to that app or minimal one would be super. Otherwise we'll need to get creative with that production deployment :).

[lpradovera]

@fxn I think it's easier to give you access, it's a private project but it is nothing special. Should I just invite you?

[fxn]

@lpradovera that'd be super.

[kwent]

Interesting for my part. Moving that piece of code into app/controllers/turbo_controller.rb didn't fix it at all. Same exception raised.

class TurboController < ApplicationController
  class Responder < ActionController::Responder
    def to_turbo_stream
      controller.render(options.merge(formats: :html))
    rescue ActionView::MissingTemplate => e
      if get?
        raise e
      elsif has_errors? && default_action
        render rendering_options.merge(formats: :html, status: :unprocessable_entity)
      else
        redirect_to navigation_location
      end
    end
  end

  self.responder = Responder
  respond_to :html, :turbo_stream
end

[fxn]

@kwent Are you setting the name of the Devise controller as a string?

config.parent_controller = 'TurboController'

[fxn]

@rguiscard @lpradovera would you mind sharing your config/environments/production.rb with any sensible value rewritten? (But keeping the config setting, in case the line is important).

Maybe we spot something that helps us explain this.

[kwent]

config.parent_controller = 'TurboController'

I do.

[fxn]

@rguiscard @lpradovera BTW, feel free to send it to me at [email protected] if that is better.

@kwent Do you you know if your application autoloads anything else during boot? It would issue a warning if it does.

[kwent]

@fxn I'll give your access to my app as well so you can dig.

[fxn]

@kwent that's super, thank you!

[eugeneotto]

I am seeing something similar:

ActionView::Template::Error (undefined method `turbo_frame_tag' for #<ActionView::Base:0x00000000015720>):

As some others have mentioned, this only appears in production mode; development works fine.

I've narrowed my case down to an initializer I use to patch ActiveAdmin. Here's a minimal project that reproduces the behavior: https://github.com/eugeneotto/activeadmin-hotwire

Please let me know if you'd like me to try anything on my end. Very excited about the work you all have been doing :)

[fxn]

Hi @eugeneotto that is fantastic, it is being hard to reproduce and that app is going to be helpful. What do we do with that app to trigger the error?

[eugeneotto]

@fxn Whoops, I guess that would help!

• Run rails s and visit http://localhost:3000 – no issues
• Run rails s -e production and visit http://localhost:3000 – ActionView::Template::Error (undefined method turbo_frame_tag for #<ActionView::Base:0x0000000000b6a8>)

[kwent]

@fxn my app also have active admin but not this monkeypatch FYI.

[fxn]

I see a pattern: the minimal application is also loading ApplicationController and ApplicationHelper too early. That is a side-effect of reopening ActiveAdmin::ResourceController during initialization.

@kwent Could you please boot your app in development mode (bin/rails r 1), and see in log/development.log if you get a warning about those constants being autoloaded during initialization?

[kwent]

@fxn

DEPRECATION WARNING: Initialization autoloaded the constants Sso, Sso::IdpSettingsAdapter, Sso::SamlFailedCallbackHandler, ActionText::ContentHelper, ActionText::TagHelper, ApplicationHelper, Account, Account::BootstrapHelper, Account::ColorsHelper, Account::DatesHelper, Account::FormsHelper, Account::InvitationsHelper, Account::LocaleHelper, Account::MarkdownHelper, Account::MembershipsHelper, Account::MessagesHelper, Account::SearchHelper, Account::SubscriptionsHelper, Account::TeamsHelper, Account::UsersHelper, BreadcrumbsHelper, EmailHelper, Fields, Fields::CloudinaryImageHelper, Fields::HtmlEditorHelper, Fields::PhoneFieldHelper, Fields::TrixEditorHelper, ImagesHelper, Slack::ParamsHelper, FontAwesome::Rails::IconHelper, DeviseHelper, Stimulus::StimulusHelper, ApplicationController, TurboController, DeviseController, Devise::SessionsController, Devise::SamlSessionsController, and Koudoku::ApplicationHelper.

Being able to do this is deprecated. Autoloading during initialization is going
to be an error condition in future versions of Rails.

Reloading does not reboot the application, and therefore code executed during
initialization does not run again. So, if you reload Sso, for example,
the expected changes won't be reflected in that stale Module object.

These autoloaded constants have been unloaded.

In order to autoload safely at boot time, please wrap your code in a reloader
callback this way:

    Rails.application.reloader.to_prepare do
      # Autoload classes and modules needed at boot time here.
    end

That block runs when the application boots, and every time there is a reload.
For historical reasons, it may run twice, so it has to be idempotent.

Check the "Autoloading and Reloading Constants" guide to learn more about how
Rails autoloads and reloads.
 (called from <main> at /Users/me/Projects/my-project/config/environment.rb:5)
DEPRECATION WARNING: The sanitized_allowed_protocols= option is deprecated and has no effect. Until Rails 5 the old behavior can still be installed. To do this add the `rails-deprecated-sanitizer` to your Gemfile. Consult the Rails 4.2 upgrade guide for more information. (called from <main> at /Users/me/Projects/my-project/config/environment.rb:5)

[fxn]

Yo! Got sidetracked by other things. Let me summarize were we are for anyone following this thread:

1. Autoloading is working fine, look at the includes, they work. We are missing methods/ancestors, really. The root cause of this issue is somewhere else.
2. We seem to have concluded that this happens when some controllers or helpers are loaded before the application has booted.

The fix in applications is to not do (2). As with any reloadable class, you need to wrap that in a to_prepare block like the printed warning and autoloading guide says. (It really depends on the case, but that is the most common and correct way to autoload reloadable constants during initialization).

In particular, if your application is affected by this, you're not blocked. Just do that and remove the includes.

Now, we have several levels of gem/framework work to do here, and that is why the ticket remains open.

1. Development mode and production mode should be consistent. Generally speaking, they should fail and succeed the same way unless they obviously cannot because you're not eager loading in development or whatever.
2. In Rails < 7, I need to understand why is the divergence happening and see if something can be done about it. I have a hypothesis, but need to connect the dots.
3. In Rails 7 this is going to be fixed out of the box. In Rails 7, trying to autoload a reloadable class while the application boots won't work anymore. You couldn't do it now or ever, but the framework doesn't err and you have the illusion that you can. In Rails 7 you'd get NameError during initialization and development and production will act the same even in this strange edge case.

[fxn]

Autoloading is working fine, look at the includes, they work.

In case you wonder "well, in production mode we eager load anyway, so it is normal that the constants are found". Precisely to guarantee a consistent success/failure behavior, Zeitwerk eager loads autoloading, it is a recursive file tree traversal with const_get, not with require.

Autoloading is working here, problem is somewhere else.