Upgrade to 4.1.0 breaks javascript version of a feature test about cucumber-rails

[bunnymatic]

Sadly I spent some time trying to reproduce in a clean repo that i can share (using what I think should be a similar setup) and can't seem to reproduce it. So... if you have any clues on why having the @javascript turned on could change anything, I'd love to hear it. Related to that, I found it behaves the same with firefox and chrome drivers - breaks the same way in both.

I'll continue to try to reproduce in a repository I can share.

[luke-hill]

This isn't a cucumber ruby issue, it'll be a cucumber rails cucumber ruby interop issue. As the tag functionality is purely covered in cucumber-rails.

The way to prove this is to strip down your repo to just cucumber ruby. And I posit you won't find the issue. Your capybara version may also have an effect on this, although given you're using a late version I doubt that.

I would also look into if the underlying method failure is fill_in (As you've not provided a stacktrace). If it is, then that's pure capybara.

What driver are you using?
Is it purely switching from @javascript to without that causes the issue?

[luke-hill]

I've recently added some tests into this repo to test the JS Drivers more rigorously. Could you see if you get any failures in our repo? Or better yet, push a failing test to this repo?

Let me know how you get on @bunnymatic

[bunnymatic]

Yes - just removing @javascript causes the failure.
I'm using chrome with Selenium

  Capybara::Selenium::Driver.new(app,
                                 browser: :chrome,
                                 args: ['--window-size=1400,1200'])

But I seem to recall trying with Firefox and having the same issue.

I'll give it a try and will look more closely. I still haven't be able to repro this outside of my large project repo. I like your suggestions of trying to pare things down in hopes of finding the culprit.

I'll let you know if I make any progress here.

[bunnymatic]

@luke-hill I just came back to check out what you might have pushed and I can't find a reference to any repo. Were you going to add a link to some repo that I could poke around with?

I've recently added some tests into this repo to test the JS Drivers more rigorously. Could you see if you get any failures in our repo? Or better yet, push a failing test to this repo?

[bunnymatic]

I think your suspicion is correct - that this is likely not cucumber-rails directly. I added logging to my sessions controller and at the ruby tier - in the server - when javascript is on, it fails and when it's not, it succeeds.

The controller does this

  def create
    logger.info("Creating session #{user_session_params}")
    @user_session = UserSession.new(user_session_params)
    logger.info("valid? #{@user_session.valid?}")
    if @user_session.save
      flash[:notice] = "You're in!"
      redirect_back_or_default root_url
    else
      logger.info("fail #{@user_session.errors.full_messages}")
      render :new
    end
  end

Excerpt from @javascript on test

method=GET path=/login format=html controller=UserSessionsController action=new status=200 duration=183.39 view=135.95 db=4.99
  CmsDocument Load (0.4ms)  SELECT `cms_documents`.* FROM `cms_documents` WHERE `cms_documents`.`page` = 'signup' AND `cms_documents`.`section` = 'signup' LIMIT 1
Creating session {"login"=>"bmatic", "password"=>"***********"}
  User Load (0.5ms)  SELECT `users`.* FROM `users` WHERE `users`.`login` = 'bmatic' LIMIT 1
valid? false
fail ["That username/password combination is wrong.  Try again."]

Excerpt from no @javascript test

method=GET path=/login format=html controller=UserSessionsController action=new status=200 duration=34.23 view=29.07 db=3.92
  CmsDocument Load (0.3ms)  SELECT `cms_documents`.* FROM `cms_documents` WHERE `cms_documents`.`page` = 'signup' AND `cms_documents`.`section` = 'signup' LIMIT 1
Creating session {"login"=>"bmatic", "password"=>"***********"}
  User Load (0.3ms)  SELECT `users`.* FROM `users` WHERE `users`.`login` = 'bmatic' LIMIT 1
valid? true

So this is something in user_session.save that is failing. It showed up after upgrading cucumber-rails (which for this run is now cucumber = 5.3.0, cucumber-rails = 2.2.0

In any case, I'm pretty sure this is not your issue. Also I'm reading my notes and I think I meant to file this against cucumber root because it appeared after upgrading cucumber (not cucumber-rails necesarily).

I appreciate your digging in a bit and sorry to have mis filed. I'm going to close this ticket and see if I can't figure out what's going on. If I do, i'll definitely come back here and leave some notes.

[bunnymatic]

I found the problem. It looks like the DatabaseCleaner strategy is not being honored. I'm going to try to repro in a small repo. But in my env i have

DatabaseCleaner.strategy = :transaction
Cucumber::Rails::Database.javascript_strategy = :truncation

which apparently has been working fine up to now.
But with upgraded cucumber = 5.3.0, cucumber-rails = 2.2.0 it seems that it's no longer working. With this same set of configs, if I look at the log/test.log output it's clear that with the old libraries, it's truncating, and with the new ones it looks like it's using transaction for the Background and then maybe truncation for the Scenario which means my background setup data is locked up in the transaction.

Here are snippets from the log files:

With the older gems (the working scenario):

• cucumber (3.1.2)
• cucumber-core (3.2.1)
• cucumber-expressions (6.0.1)
• cucumber-rails (2.0.0)
• cucumber-tag_expressions (1.1.1)
• cucumber-wire (0.0.1)
• database_cleaner (1.8.5)

# excerpt from log/test.log
   (0.3ms)  SELECT `ar_internal_metadata`.`value` FROM `ar_internal_metadata` WHERE `ar_internal_metadata`.`key` = 'schema_sha1'
   (0.3ms)  SELECT `schema_migrations`.`version` FROM `schema_migrations` ORDER BY `schema_migrations`.`version` ASC
Dalli::Server#connect 127.0.0.1:11211
   (0.4ms)  SELECT @@FOREIGN_KEY_CHECKS
   (0.2ms)  SET FOREIGN_KEY_CHECKS = 0
   (0.5ms)  select table_name from information_schema.views where table_schema = 'mau_test'
   (2.8ms)  TRUNCATE TABLE `application_events`;
   (1.9ms)  TRUNCATE TABLE `art_piece_tags`;
   (2.4ms)  TRUNCATE TABLE `art_pieces`;
   (2.4ms)  TRUNCATE TABLE `art_pieces_tags`;
 ...

# the background step is here where it creates an `Artist` record and commits it

   (0.2ms)  SET FOREIGN_KEY_CHECKS = 1
   (0.3ms)  SELECT `ar_internal_metadata`.`value` FROM `ar_internal_metadata` WHERE `ar_internal_metadata`.`key` = 'schema_sha1'
   (0.3ms)  SELECT `schema_migrations`.`version` FROM `schema_migrations` ORDER BY `schema_migrations`.`version` ASC
Dalli::Server#connect 127.0.0.1:11211
  Artist Load (0.3ms)  SELECT `users`.* FROM `users` WHERE `users`.`type` = 'Artist' AND `users`.`login` = 'bmatic' ORDER BY `users`.`id` ASC LIMIT 1
  TRANSACTION (0.3ms)  BEGIN
  User Exists? (0.3ms)  SELECT 1 AS one FROM `users` WHERE `users`.`id` IS NOT NULL AND `users`.`slug` = 'bmatic' LIMIT 1
  User Exists? (0.2ms)  SELECT 1 AS one FROM `users` WHERE `users`.`login` = 'bmatic' LIMIT 1
  User Exists? (0.2ms)  SELECT 1 AS one FROM `users` WHERE `users`.`email` = '[email protected]' LIMIT 1
  BlacklistDomain Load (0.2ms)  SELECT `blacklist_domains`.* FROM `blacklist_domains` WHERE `blacklist_domains`.`domain` = 'example.com' LIMIT 1
  User Exists? (0.2ms)  SELECT 1 AS one FROM `users` WHERE `users`.`persistence_token` = BINARY '7348d486bd53b59bddb1651fca461a51b29040c5c618e5562e45e48a65728ed282be19c72951d8dd730472914297d28a3f2cc4c1b73d0bcf6ac839d24952ee2c' LIMIT 1
  Artist Create (0.4ms)  INSERT INTO `users` (`login`, `email`, `crypted_password`, `password_salt`, `created_at`, `updated_at`, `firstname`, `lastname`, `nomdeplume`, `activation_code`, `activated_at`, `state`, `persistence_token`, `slug`, `links`) VALUES ('bmatic', '[email protected]', '400$8$42$a345beb83ecc9165$36616f3f40330d3d3e4bfd6750ddc3229e12ed6fe9df51290e5ede692d98e4e6', '44_nXxzim1bq9WWyjNqn', '2021-02-06 21:08:14', '2021-02-06 21:08:14', 'Irwin', 'Sherman', 'Williamson-Jones', '7aa4c5434b507fd820f236fef2642cdedc4bd299', '2021-02-06 20:08:14', 'active', '7348d486bd53b59bddb1651fca461a51b29040c5c618e5562e45e48a65728ed282be19c72951d8dd730472914297d28a3f2cc4c1b73d0bcf6ac839d24952ee2c', 'bmatic', '--- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\nwebsite: http://hodkiewicz.co/luther\n')
  TRANSACTION (1.3ms)  COMMIT

...

And from here the test runs fine.

After upgrading to

• cucumber (5.3.0)
• cucumber-core (8.0.1)
• cucumber-create-meta (2.0.4)
• cucumber-cucumber-expressions (10.3.0)
• cucumber-gherkin (15.0.2)
• cucumber-html-formatter (9.0.0)
• cucumber-messages (13.2.1)
• cucumber-rails (2.2.0)
• cucumber-tag-expressions (2.0.4)
• cucumber-wire (4.0.1)
• protobuf-cucumber (3.10.8)
• database_cleaner (1.8.5)

# excerpt from log/test.log
   (0.3ms)  SELECT `ar_internal_metadata`.`value` FROM `ar_internal_metadata` WHERE `ar_internal_metadata`.`key` = 'schema_sha1'
   (0.3ms)  SELECT `schema_migrations`.`version` FROM `schema_migrations` ORDER BY `schema_migrations`.`version` ASC
Dalli::Server#connect 127.0.0.1:11211
  TRANSACTION (0.4ms)  BEGIN
  Artist Load (0.3ms)  SELECT `users`.* FROM `users` WHERE `users`.`type` = 'Artist' AND `users`.`login` = 'bmatic' ORDER BY `users`.`id` ASC LIMIT 1
  TRANSACTION (0.3ms)  SAVEPOINT active_record_1
  User Exists? (0.3ms)  SELECT 1 AS one FROM `users` WHERE `users`.`id` IS NOT NULL AND `users`.`slug` = 'bmatic' LIMIT 1
  User Exists? (0.2ms)  SELECT 1 AS one FROM `users` WHERE `users`.`login` = 'bmatic' LIMIT 1
  User Exists? (0.2ms)  SELECT 1 AS one FROM `users` WHERE `users`.`email` = '[email protected]' LIMIT 1
  BlacklistDomain Load (0.3ms)  SELECT `blacklist_domains`.* FROM `blacklist_domains` WHERE `blacklist_domains`.`domain` = 'example.com' LIMIT 1
  User Exists? (0.3ms)  SELECT 1 AS one FROM `users` WHERE `users`.`persistence_token` = BINARY '05f882952bf9e9dd3b7577c35f8bd5efa5d66adf5c299ba0b656f0ec993c3d9ff9b4dde5b4ccb225e839d527889f6723e8b6d9e680090a975487775345313932' LIMIT 1
  Artist Create (0.4ms)  INSERT INTO `users` (`login`, `email`, `crypted_password`, `password_salt`, `created_at`, `updated_at`, `firstname`, `lastname`, `nomdeplume`, `activation_code`, `activated_at`, `state`, `persistence_token`, `slug`, `links`) VALUES ('bmatic', '[email protected]', '400$8$42$b7903f60b7fdf3d4$09f0c0ca3e41a5b21c0c5f4c432460e08863959f1c2763463d2d7512192e29b4', 'R6jUkc3BsNXnWpJZTaCV', '2021-02-06 21:18:08', '2021-02-06 21:18:08', 'Jin', 'Mercy', 'Ebert-Walsh', '079333e61c4593a853479aafc72e5482510f30b9', '2021-02-06 20:18:08', 'active', '05f882952bf9e9dd3b7577c35f8bd5efa5d66adf5c299ba0b656f0ec993c3d9ff9b4dde5b4ccb225e839d527889f6723e8b6d9e680090a975487775345313932', 'bmatic', '--- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\nwebsite: http://oberbrunner-schmeler.biz/toby\n')
  TRANSACTION (0.2ms)  RELEASE SAVEPOINT active_record_1
  ArtPiece Load (0.4ms)  SELECT `art_pieces`.* FROM `art_pieces` WHERE `art_pieces`.`artist_id` = 1 ORDER BY `art_pieces`.`position` ASC, `art_pieces`.`created_at` DESC LIMIT 1

# Notice the `RELEASE SAVEPOINT` above here... which means the `Artist Create` line which is 
# setting up my data for the test gets lost.
#
# And the fact that this does *not* start out with a pile of `TRUNCATE` calls

...

# Then the `TRUNCATE` step happens only at the very end of the test (i presume in an After cleanup)

   (0.3ms)  SELECT @@FOREIGN_KEY_CHECKS
   (0.2ms)  SET FOREIGN_KEY_CHECKS = 0
   (0.6ms)  select table_name from information_schema.views where table_schema = 'mau_test'
   (3.5ms)  TRUNCATE TABLE `application_events`;
   (1.8ms)  TRUNCATE TABLE `art_piece_tags`;
   (2.2ms)  TRUNCATE TABLE `art_pieces`;
   ...
   (3.0ms)  TRUNCATE TABLE `users`;
   (0.2ms)  SET FOREIGN_KEY_CHECKS = 1
  TRANSACTION (0.2ms)  ROLLBACK

Pretty weird, but now that I can see exactly what's happening, I might have more chance of reproducing in a smaller narrow repo that I can share.

If I can, I'll likely re-open this issue, unless you think from this you can tell me this belongs on someone else's list of issues.

[bunnymatic]

I'm unable to repro in a small separate rails repo. Not sure what this app is doing that's different. Pretty frustrating.

I did find that if I just give up on :transaction for everything and set the strategy in env.rb to :truncation all the time, things work just fine. So I'm just going to roll with that and give up on this.

Again, thanks for your input and time. And keep up the good work on cucumber-rails.

[marnen]

@bunnymatic That makes sense. The JavaScript browser client won't have access to the same transaction in which the Cucumber Ruby process created the user account.

[bunnymatic]

I agree @marnen . Totally makes sense. What doesn't make sense is that before the upgrade the very same tests were working just fine. So something about some of the bundle updates changed where/when :transaction was getting handled. Not a big deal. I certainly had fun (sort of) digging around in the codes. :D