Comments (29)
zhangaaron
commented on July 20, 2024
Can you clarify what the issue here is? The number of specs here passed is 13/13, number of cucumber scenarios passed is 3/3.
from rag.
tansaku
commented on July 20, 2024
@zhangaaron: @rosewcs345 is saying some other students (in her SUNY Binghamton class) don't get this information. (@mtc2013 filed this on her behalf)
Ideally we need a screen shot of the other kind of output they are seeing in order to understand this better ...
from rag.
mtc2013
commented on July 20, 2024
@tansaku could you post the picture of the results returned from the grader when submitting this student's hw through @rosewcs345 spoc? I don't have access.
from rag.
tansaku
commented on July 20, 2024
I think this is it:
from rag.
rosewcs345
commented on July 20, 2024
rspec passes locally, but not accepted by autograder.
Students with this issue used FactoryGirl
from rag.
tansaku
commented on July 20, 2024
so I am essentially replicating this in #93 - I get exactly the same message in the local acceptance test as I do in the grader:
Submission message after grade: Your submission was recorded at 2010-06-19 18:52:28 +0100 : submission is on time.
Score: 40.0/100.0
----------------------------------------
Running student tests found in features/ spec/:
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -S bundle exec cucumber --profile default
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed
----------------------------------------
----------------------------------------
Checking coverage for:
controllers >= 90.00%
models >= 90.00%
----------------------------------------
all files: 84.21%% coverage
controllers: 80.43%% coverage
models: 100.00%% coverage
mailers: 100.00%% coverage
helpers: 100.00%% coverage
libraries: 100.00%% coverage
Failed coverage test (controllers coverage too low).
Score: 0/20.0
----------------------------------------
----------------------------------------
Running reference Cucumber scenarios:
3 out of 3 scenarios passed
Score: 40/40
----------------------------------------
Total score: 40.0 / 100.0
Completed in 19.305187 seconds.
from rag.
tansaku
commented on July 20, 2024
and if I open the zipped file from the student all the specs pass and the coverage is good:
[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/features/support/hw4 (92_factorygirl_in_hw4)]$
→ be rspec
...................
Finished in 0.20686 seconds (files took 2.21 seconds to load)
19 examples, 0 failures
Coverage report generated for RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/features/support/hw4/coverage. 0.0 / 0.0 LOC (100.0%) covered.
from rag.
tansaku
commented on July 20, 2024
however if I go to the submission folder where the grader has unpackaged the student code (and done various things with it) then we get the following:
[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42 (92_factorygirl_in_hw4)]$
→ be rspec
...................
Finished in 0.20125 seconds (files took 1.51 seconds to load)
19 examples, 0 failures
Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/coverage. 57 / 57 LOC (100.0%) covered.
which is very similar ...
from rag.
tansaku
commented on July 20, 2024
now the grader doesn't run bundle exec rspec, it runs a special rake task like so ....
[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42 (92_factorygirl_in_hw4)]$
→ bundle exec rake saas:run_student_tests
----BEGIN CUCUMBER----
--------------------------------------------------------------------------------
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -S bundle exec cucumber --profile default
Using the default profile...
Feature: search for movies by director
As a movie buff
So that I can find movies with my favorite director
I want to include and serach on director information in movies I enter
Background: movies in database # features/add_director.feature:7
Given the following movies exist: # features/step_definitions/director_steps.rb:1
| title | rating | director | release_date |
| Star Wars | PG | George Lucas | 1977-05-25 |
| Blade Runner | PG | Ridley Scott | 1982-06-25 |
| Alien | R | | 1979-05-25 |
| THX-1138 | R | George Lucas | 1971-03-11 |
Scenario: add director to existing movie # features/add_director.feature:16
When I go to the edit page for "Alien" # features/step_definitions/web_steps.rb:48
And I fill in "Director" with "Ridley Scott" # features/step_definitions/web_steps.rb:60
And I press "Update Movie Info" # features/step_definitions/web_steps.rb:52
DEPRECATION: Using `should` from rspec-expectations' old `:should` syntax without explicitly enabling the syntax is deprecated. Use the new `:expect` syntax or explicitly enable `:should` with `config.expect_with(:rspec) { |c| c.syntax = :should }` instead. Called from /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/features/step_definitions/web_steps.rb:107:in `block in <top (required)>'.
Then the director of "Alien" should be "Ridley Scott" # features/step_definitions/director_steps.rb:7
Scenario: find movie with same director # features/add_director.feature:22
Given I am on the details page for "Star Wars" # features/step_definitions/web_steps.rb:44
When I follow "Find Movies With Same Director" # features/step_definitions/web_steps.rb:56
Then I should be on the Similar Movies page for "Star Wars" # features/step_definitions/web_steps.rb:230
And I should see "THX-1138" # features/step_definitions/web_steps.rb:105
But I should not see "Blade Runner" # features/step_definitions/web_steps.rb:123
Scenario: can't find similar movies if we don't know director (sad path) # features/add_director.feature:29
Given I am on the details page for "Alien" # features/step_definitions/web_steps.rb:44
Then I should not see "Ridley Scott" # features/step_definitions/web_steps.rb:123
When I follow "Find Movies With Same Director" # features/step_definitions/web_steps.rb:56
Then I should be on the home page # features/step_definitions/web_steps.rb:230
And I should see "'Alien' has no director info" # features/step_definitions/web_steps.rb:105
3 scenarios (3 passed)
17 steps (17 passed)
0m0.414s
Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/coverage. 57 / 57 LOC (100.0%) covered.
--------------------------------------------------------------------------------
----END CUCUMBER----
----BEGIN RSPEC----
--------------------------------------------------------------------------------
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...................
Finished in 0.19556 seconds (files took 1.46 seconds to load)
19 examples, 0 failures
Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/coverage. 57 / 57 LOC (100.0%) covered.
--------------------------------------------------------------------------------
----END RSPEC----
but running that in the submission directory above also generates a 100% test coverage ...
from rag.
tansaku
commented on July 20, 2024
so stepping through the HW4 grader in the debugger is not helping me very much, but if I look in the coverage directory in the submission directory I can see that a <100% coverage is being reported there:
from rag.
tansaku
commented on July 20, 2024
showing gaps in the movies controller like so:
from rag.
tansaku
commented on July 20, 2024
and if I run bundle exec rake saas:run_student_tests again in that submission folder then the coverage updates to 100%, indicating that somehow on the first run of that task, that perhaps not everything is in place ...?
from rag.
tansaku
commented on July 20, 2024
so I hooked the grader with the debugger to stop just before it starts parsing the test coverage files. The coverage files then show another different story
Is there some race condition here where the parsing of the coverage files starts before they have finished being generated? That doesn't make sense ...
from rag.
tansaku
commented on July 20, 2024
so with the grader hooked on the debugger I ran the rake task and got the failure in the terminal:
Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-39-08/coverage. 48 / 57 LOC (84.21%) covered.
--------------------------------------------------------------------------------
----END CUCUMBER----
----BEGIN RSPEC----
--------------------------------------------------------------------------------
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
....FF.............
Failures:
1) MoviesController GET #index should sort movies by title
Failure/Error: expect(assigns(:movies)).to match_array(Movie.order(:title))
expected collection contained: [#<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
actual collection contained: [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
the extra elements were: [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>]
# ./spec/controllers/movies_controller_spec.rb:108:in `block (3 levels) in <top (required)>'
2) MoviesController GET #index should sort movies by rating
Failure/Error: expect(assigns(:movies)).to match_array(Movie.order(:release_date))
expected collection contained: [#<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
actual collection contained: [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
the extra elements were: [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>]
# ./spec/controllers/movies_controller_spec.rb:115:in `block (3 levels) in <top (required)>'
Finished in 0.21762 seconds (files took 1.59 seconds to load)
19 examples, 2 failures
Failed examples:
rspec ./spec/controllers/movies_controller_spec.rb:104 # MoviesController GET #index should sort movies by title
rspec ./spec/controllers/movies_controller_spec.rb:112 # MoviesController GET #index should sort movies by rating
Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-39-08/coverage. 57 / 57 LOC (100.0%) covered.
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed
from rag.
tansaku
commented on July 20, 2024
what I would like to do now is leave the grader to finish running, and then compare what, if any, files are changed over the completion of the grader activity - I would do that with git, but the submissions directory is nested inside the rag git space ...
from rag.
tansaku
commented on July 20, 2024
so I tarred the directory up (to take a snapshot of it) and then let the grader carry on - the acceptance test now failed because this time around the different coverage report gives us 60% instead of 40% (go figure), but when I test in the submission directory again I get all tests passing and 100% coverage implying that something has changed through the completion of the grader run ...
from rag.
tansaku
commented on July 20, 2024
so the only differences that appear are in the timestamps in the coverage directory and the contents of the test.log
[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4 (92_factorygirl_in_hw4)]$
→ diff -bur 2016-03-24-12-39-08 ~/tmp/
.ssh/ 2016-03-24-12-39-08/ ssh.tar
[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4 (92_factorygirl_in_hw4)]$
→ diff -bur 2016-03-24-12-39-08 ~/tmp/2016-03-24-12-39-08/
diff -bur 2016-03-24-12-39-08/coverage/.resultset.json /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/.resultset.json
--- 2016-03-24-12-39-08/coverage/.resultset.json 2016-03-24 12:52:50.000000000 +0000
+++ /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/.resultset.json 2016-03-24 12:50:05.000000000 +0000
@@ -227,6 +227,6 @@
null
]
},
- "timestamp": 1458823970
+ "timestamp": 1458823805
}
}
diff -bur 2016-03-24-12-39-08/coverage/index.html /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/index.html
--- 2016-03-24-12-39-08/coverage/index.html 2016-03-24 12:52:50.000000000 +0000
+++ /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/index.html 2016-03-24 12:50:05.000000000 +0000
@@ -14,7 +14,7 @@
<img src="./assets/0.10.0/loading.gif" alt="loading"/>
</div>
<div id="wrapper" style="display:none;">
- <div class="timestamp">Generated <abbr class="timeago" title="2016-03-24T12:52:50+00:00">2016-03-24T12:52:50+00:00</abbr></div>
+ <div class="timestamp">Generated <abbr class="timeago" title="2016-03-24T12:50:05+00:00">2016-03-24T12:50:05+00:00</abbr></div>
<ul class="group_tabs"></ul>
<div id="content">
Binary files 2016-03-24-12-39-08/db/test.sqlite3 and /Users/tansaku/tmp/2016-03-24-12-39-08/db/test.sqlite3 differ
diff -bur 2016-03-24-12-39-08/log/test.log /Users/tansaku/tmp/2016-03-24-12-39-08/log/test.log
--- 2016-03-24-12-39-08/log/test.log 2016-03-24 12:52:50.000000000 +0000
+++ /Users/tansaku/tmp/2016-03-24-12-39-08/log/test.log 2016-03-24 12:50:05.000000000 +0000
@@ -759,355 +759,3 @@
(0.5ms) rollback transaction
(0.1ms) begin transaction
(0.0ms) rollback transaction
- (0.8ms) CREATE TABLE "movies" ("id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "title" varchar, "rating" varchar, "description" text, "release_date" datetime, "created_at" datetime, "updated_at" datetime, "director" varchar)
- (0.8ms) CREATE TABLE "schema_migrations" ("version" varchar NOT NULL)
- (0.1ms) select sqlite_version(*)
- (0.8ms) CREATE UNIQUE INDEX "unique_schema_migrations" ON "schema_migrations" ("version")
- (0.1ms) SELECT version FROM "schema_migrations"
- (0.6ms) INSERT INTO "schema_migrations" (version) VALUES ('20160310215150')
- (0.6ms) INSERT INTO "schema_migrations" (version) VALUES ('20111119180638')
- (0.5ms) INSERT INTO "schema_migrations" (version) VALUES ('20120130161449')
from rag.
tansaku
commented on July 20, 2024
which makes me think this must be some sort of database issue - that something to do with FactoryGirl is running in a thread or something? That some process isn't completing before the coverage is being assessed ... or something ... or am I totally on the wrong track?
from rag.
tansaku
commented on July 20, 2024
is the problem here that the tests are being run more than once and that the test database is not being cleared between runs ...?
from rag.
tansaku
commented on July 20, 2024
running repeatedly in the submission directory does not cause a problem, but looking at the failure we got, it appears to be that the movies table is holding more movies that we expect ...
from rag.
tansaku
commented on July 20, 2024
ah, is it that the cucumber and specs are being run together, and elements from the cucumber test are hanging around in the database and not being cleaned before the specs are run or something ...?
from rag.
tansaku
commented on July 20, 2024
hmm, looks like we have the movie Aladdin being added in both a migration and also the seeds file ...
from rag.
tansaku
commented on July 20, 2024
and that this is a migration added by the grader?
from rag.
tansaku
commented on July 20, 2024
so the grader gets the test db set up with some data in the movies table:
[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-13-32-11 (92_factorygirl_in_hw4)]$
→ sqlite3 db/test.sqlite3
SQLite version 3.8.10.2 2015-05-20 18:17:19
Enter ".help" for usage hints.
sqlite> .tables
movies schema_migrations
sqlite> select * from movies;
1|Aladdin|G||1992-11-25 00:00:00.000000|2016-03-24 13:32:14.898606|2016-03-24 13:32:14.898606|
2|The Terminator|R||1984-10-26 00:00:00.000000|2016-03-24 13:32:14.901964|2016-03-24 13:32:14.901964|
3|When Harry Met Sally|R||1989-07-21 00:00:00.000000|2016-03-24 13:32:14.902735|2016-03-24 13:32:14.902735|
4|The Help|PG-13||2011-08-10 00:00:00.000000|2016-03-24 13:32:14.903478|2016-03-24 13:32:14.903478|
5|Chocolat|PG-13||2001-01-05 00:00:00.000000|2016-03-24 13:32:14.904258|2016-03-24 13:32:14.904258|
6|Amelie|R||2001-04-25 00:00:00.000000|2016-03-24 13:32:14.905083|2016-03-24 13:32:14.905083|
7|2001: A Space Odyssey|G||1968-04-06 00:00:00.000000|2016-03-24 13:32:14.906002|2016-03-24 13:32:14.906002|
8|The Incredibles|PG||2004-11-05 00:00:00.000000|2016-03-24 13:32:14.907292|2016-03-24 13:32:14.907292|
9|Raiders of the Lost Ark|PG||1981-06-12 00:00:00.000000|2016-03-24 13:32:14.908173|2016-03-24 13:32:14.908173|
10|Chicken Run|G||2000-06-21 00:00:00.000000|2016-03-24 13:32:14.908977|2016-03-24 13:32:14.908977|
from rag.
tansaku
commented on July 20, 2024
which I tried deleting, but that has no impact ...
from rag.
tansaku
commented on July 20, 2024
however if I drop the db after and then run the tests straight after running the migrations then I get the failures ...
from rag.
tansaku
commented on July 20, 2024
hmm, but if I remove that extra migration from the hw4 grader template - I still get the same overall failures, and checking the submission directory that migration is not present ... what?
I can no longer replicate the error with by running turning over the db in the submission directory ... which makes me think there might be some caching complexity - gonna push this into the branch and see what the CI thinks ...
from rag.
mtc2013
commented on July 20, 2024
@tansaku placed factory_girls_rails into the root autograder gem file as a quick fix for some of the issues relating to FactoryGirl that were silently failing. It appears that the way the AG launches the rake task does not make use of the bundle installed gems from the student submission.
from rag.
tansaku
commented on July 20, 2024
Thanks @mtc2013 - just to follow up on that with some more details for @zhangaaron and @rosewcs345 it looks like the hw4grader runs the two commands bundle install --without production and bundle exec rake saas:run_student_tests in two separate processes.
The grader runs the tests in the correct directory (a generated submission directory) where the Gemfile and Gemfile.lock are correctly set up (and now have factory_girl because I added it to the hw4 skeleton). However when the hw4 grader runs the following code:
Open3.popen3(env, "bundle exec rake saas:run_student_tests")
the gem environment that's available appears to be that of the grader itself and not that of the merged student/hw4-template directory (this might be due to the bundle install having been run in a separate process, but then again ...).
We've rectified this temporarily by adding the factory_girl_rails gem to the grader Gemfile, and set that up on staging (which @rosewcs345 's students are using), which seems to make factory_girl_rails available at the point when the grader runs the test on the student assignment.
It's possible that we are still mis-interpreting things. The hw4 grader template also includes an additional migration that dumps extra movies in the database that can interfere with tests that pass on student machines. I'm not clear if this extra movie migration is actually necessary for the hw4 grader (I've removed it in the PR), however it can definitely cause failures that are then not reported back to the student (as well as making it difficult for developers to debug).
And of course anytime any a student uses any new gem in their submission things will break silently. It seems odd actually that we are even using Open3.popen3 since we are already running the hw4grader in a fork, but I guess that's just a way of running for and waiting for a series of threads to finish. I had thought that running a bundle exec prefixed process in a directory with a particular Gemfile and Gemfile.lock would guarantee the gems in that file to be available, but perhaps it is different for threads ...? (ruby generated threads somehow inherit the parent thread's gemset?) That's something we should carefully experiment with, although there's the lower hanging fruit of adjusting the grader to make these errors visible rather than failing silently ...
Anyway, I'm off for the next 10 days. I've added the factory_girl_rails gem to the grader in the pull request, and at some point before the next run of the MOOC we will have to get all this stuff out onto the master branch and into production, but things are a bit jury-rigged at the moment so I'd be cautious about merging a lot of stuff in ...
from rag.
Related Issues (20)
- tell learner whether failure was a timeout or an error
- tr count issue HOT 6
- Ruby Calisthenics support
- error on grader HOT 4
- running out of space on grader
- A locally correct answer that can't pass the auto-grader for HW4 HOT 10
- Locally passed test cases fail on auto-grader for HW5
- suppress output to stdout during tests
- add acceptance test for Sinatra Hangperson
- could mutation grader output indicate that mutations are taking place? HOT 1
- bugs on master HOT 4
- Github Grader
- unit acceptance test grader fails in strange way
- acceptance unit test assignment (feature grader) appears to give full marks for rspec when no rspec tests are present
- acceptance unit test grader has data being added in a migration
- Student feedback on controller coverage is different from local feedback
- Legacy HW1 grader reported spinning HOT 6
- can timeout be made more robust?
- grader down (reported for HW4) HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rag.