saasbook / rag Goto Github PK
View Code? Open in Web Editor NEWRuby Auto-grader
Home Page: http://saas-class.org
License: Other
Ruby Auto-grader
Home Page: http://saas-class.org
License: Other
We have some students reporting that answers that work locally can't pass the auto-grader. An example can be found in this file. Another example would be calling check method with ratings[#{rating}]. We believe this is due to the version of Cucumber and Cypabara of the auto-grader.
it would be great if we could tell the learner whether a failure was a timeout or an error explicitly, i.e. not one or the other, but which
Sometimes assignment specs will require new gems. At the moment it appears we need to add those gems to the grader Gemfile, which thus prevents the grader from correctly grading any assignment that introduces new gems.
Ideally the grader would automatically pull in the necessary gems.
We barraged with output from rspec and cucumber when running the tests - it would be much easier to manoeuvre if that output could be suppressed.
It looks like it's supposed to be but the suppression mechanism is not working in rspec 3.3, but it is in rspec 3.6
this breaks ruby style guidelines that modules or classes should be in a file of the same name:
https://github.com/bbatsov/ruby-style-guide#one-class-per-file
The immediate problem is that it makes navigating the codebase difficult
Just checked the running grader and saw lots of these:
/home/ubuntu/.rvm/gems/ruby-2.2.2/bundler/gems/xqueue-ruby-ed709d6b395a/lib/xqueue_ruby/xqueue_submission.rb:69:in `fetch_files!': undefined method `inject' for "":String (NoMethodError)
from /home/ubuntu/.rvm/gems/ruby-2.2.2/bundler/gems/xqueue-ruby-ed709d6b395a/lib/xqueue_ruby/xqueue_ruby.rb:135:in `get_submission'
from /home/ubuntu/rag/lib/submission/xqueue.rb:19:in `next_submission_with_assignment'
from /home/ubuntu/rag/lib/submission/polling.rb:18:in `block in run'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `loop'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `run'
from run_autograder.rb:27:in `<main>'
run_autograder.rb:10: warning: class variable access from toplevel
run_autograder exited.
/home/ubuntu/.rvm/gems/ruby-2.2.2/bundler/gems/xqueue-ruby-ed709d6b395a/lib/xqueue_ruby/xqueue_submission.rb:69:in `fetch_files!': undefined method `inject' for "":String (NoMethodError)
from /home/ubuntu/.rvm/gems/ruby-2.2.2/bundler/gems/xqueue-ruby-ed709d6b395a/lib/xqueue_ruby/xqueue_ruby.rb:135:in `get_submission'
from /home/ubuntu/rag/lib/submission/xqueue.rb:19:in `next_submission_with_assignment'
from /home/ubuntu/rag/lib/submission/polling.rb:18:in `block in run'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `loop'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `run'
from run_autograder.rb:27:in `<main>'
Not working in live deployment, need cucumber tests to check we can grade this, maybe ...
Students receive 60/100, autograder does not tell them what failed. Possibly due to the fact that they have extra .feature files?
Zip submission of a student that exhibits this problem:
hw4_bug0.zip
just updating to the latest code, specs all pass, but features fail:
cucumber
Feature: Autograder configured to accept student submissions from edX and grade them
As an instructor
So that I can give feedback to my students on their code responses
I want to be able to create a code submission page in edX and grade it using rag
@require_net_connect
Scenario: simple one file submission against one spec file RSpecGrader # features/autograder_xqueue_stubbed.feature:6
Given an XQueue that has submission "simple_rspec_xqueue.json" in queue # features/step_definitions/adapter_x_queue_steps.rb:28
["http://www.fakedownload.com/hw1.rb"]
register_uri for : http://www.fakedownload.com/hw1.rb
And has been setup with the config file "conf.yml" # features/step_definitions/adapter_x_queue_steps.rb:44
No such file or directory @ rb_sysopen - logs/log-2015-08-27 10:59:36 +0100 (Errno::ENOENT)
./lib/rag_logger.rb:8:in `initialize'
./lib/rag_logger.rb:8:in `open'
./lib/rag_logger.rb:8:in `configure_logger'
./lib/submission/base.rb:10:in `initialize'
./lib/submission/polling.rb:8:in `initialize'
./lib/submission/xqueue.rb:13:in `initialize'
./lib/adapter.rb:14:in `new'
./lib/adapter.rb:14:in `load'
./features/step_definitions/adapter_x_queue_steps.rb:45:in `/^has been setup with the config file "(.*?)"$/'
features/autograder_xqueue_stubbed.feature:8:in `And has been setup with the config file "conf.yml"'
Then I should receive a grade of "30" for my assignment # features/step_definitions/adapter_x_queue_steps.rb:49
Running the tests on fresh OSX 10.11.2 (El Capitan) I got the following trying to run the cucumber tests
The authenticity of host 'github.com (192.30.252.129)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.252.129' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Then I should receive a grade of "100" for my assignment # features/step_definitions/adapter_x_queue_steps.rb:49
expected PutResultException, got #<IOError: Fatal error: Retrieving spec files from [email protected]:saasbook/hw-ruby-intro-ci.git repository failed.> with backtrace:
# ./lib/assignment/xqueue.rb:58:in `fetch_spec_file'
# ./lib/assignment/xqueue.rb:34:in `initialize'
# ./lib/submission/xqueue.rb:21:in `new'
# ./lib/submission/xqueue.rb:21:in `next_submission_with_assignment'
# ./lib/submission/polling.rb:18:in `block in run'
# ./lib/submission/polling.rb:17:in `loop'
# ./lib/submission/polling.rb:17:in `run'
# ./features/step_definitions/adapter_x_queue_steps.rb:53:in `block (3 levels) in <top (required)>' (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/adapter_x_queue_steps.rb:50:in `/^I should receive a grade of "(.*?)" for my assignment$/'
features/autograder_xqueue_stubbed.feature:20:in `Then I should receive a grade of "100" for my assignment'
cucumber features/autograder_xqueue_stubbed.feature:16 # Scenario: student submits RSpecGrader assignment containing multiple files with specs hosted on Github
See: https://www.pivotaltracker.com/story/show/112748271
"Invalid grader reply. Please contact the course staff."
"There was a fatal error with your submission. It either timed out or caused an exception."
No obvious errors in the log. Have restarted the grader - terminal was not showing output - restarted without being able to see text on command prompt ...
I wonder if this might clean up the code a little ... e.g.
RSpec::Core::Runner.run(['-r', submission, spec], output, errs)
We need a process for managing ongoing grader development, which we should add to CONTRIBUTING.md
we should adjust the README to indicate that one can start testing on C9 or on your own dev laptop
would be good for consistency
[tansaku@Samuels-MBP:~/Documents/Github/saasbook/rag (master)]$
→ bundle exec rspec
.......F....*.*
Pending: (Failures listed here are expected and do not affect your suite's status)
1) Submission::Xqueue it can create an assignment from a submission and grade it should pass assignment and submission to the autograder
# run in a separate thread to avoid infinite loop
Failure/Error: expect(true).to be_false
expected true to respond to `false?` or perhaps you meant `be false` or `be_falsey`
# ./spec/submission/xqueue_spec.rb:39:in `block (3 levels) in <top (required)>'
2) Submission::Xqueue when there is no submission in queue it should sleep when
# run in seperate thread to avoid infinite loop
Failure/Error: expect(true).to be_false
expected true to respond to `false?` or perhaps you meant `be false` or `be_falsey`
# ./spec/submission/xqueue_spec.rb:56:in `block (3 levels) in <top (required)>'
Failures:
1) Graders::RspecGrader should be able to grade a simple homework gives points to a hw1 solution
Failure/Error: expect(b[:raw_score]).to be == 30
expected: == 30
got: 0
# ./spec/graders/rspec_grader_spec.rb:20:in `block (3 levels) in <top (required)>'
Finished in 0.57171 seconds (files took 1.13 seconds to load)
15 examples, 1 failure, 2 pending
Failed examples:
rspec ./spec/graders/rspec_grader_spec.rb:18 # Graders::RspecGrader should be able to grade a simple homework gives points to a hw1 solution
Coverage report generated for RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/coverage. 356 / 942 LOC (37.79%) covered.
[tansaku@Samuels-MBP:~/Documents/Github/saasbook/rag (master)]$
→ bundle exec cucumber
..
Deprecation Warnings:
Using `should` from rspec-expectations' old `:should` syntax without explicitly enabling the syntax is deprecated. Use the new `:expect` syntax or explicitly enable `:should` with `config.expect_with(:rspec) { |c| c.syntax = :should }` instead. Called from /var/folders/2z/kh7rq7jn5l3gpbldyfg1tb4c0000gn/T/spec_file20170927-45695-1ivzqz1:26:in `block (3 levels) in <top (required)>'.
If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.
........F..................................
(::) failed steps (::)
expected: == 100.0
got: 0.0 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/adapter_x_queue_steps.rb:76:in `/^I should receive a grade of "(.*?)" for my assignment$/'
features/autograder_xqueue_stubbed.feature:17:in `Then I should receive a grade of "100" for my assignment'
Failing Scenarios:
cucumber features/autograder_xqueue_stubbed.feature:13 # Scenario: student submits a HerokuGrader homework
11 scenarios (1 failed, 10 passed)
45 steps (1 failed, 44 passed)
2m38.224s
I was trying to add a new grader: https://github.com/saasbook/rag/tree/62_support_legacy_assignment but it wasn't automatically included. The grader choked saying that HerokuWithLoginRspecGrader was unknown.
I guess that the new grader just needs to be required somewhere? Perhaps we should set it so that all rb files in the grader directory are automatically pulled in?
The older had a GitHub grader that could inspect code on GitHub - we need it for the AV102 course and maybe for future conversions of the grader to work more closely with GitHub
since we don't have any at present.
Interesting question about whether we rely on some heroku endpoint, or try and run our own server locally to hit based on the solution code?
We should not assume that a grader admin understands the ins and outs of tmux or screen.
Also note that tmux suddenly becomes much more usable for a grader admin if they have normal xterm scrolling set up:
http://superuser.com/questions/209437/how-do-i-scroll-in-tmux/326592#326592
It's unclear where to put spec files in the assignment spec repo, and there's currently no mention of the access issues that one may encounter
Hello all.
I get this message since a week now trying to upload hw4. I removed factory_girl completely in my homework now using fixtures as it was discussed in the forum.
Your submission was recorded at 2015-12-08 20:51:01 +0000 : submission is late and scaled by 0.75
Score: 0/100
There was a fatal error with your submission. It either timed out or caused an exception.
I just get that message without any other backtrace to identify the error.
Any help would be greatly appreciated!
Reviewing the code for the HW3 grader, it looks like it also doesn't inherit the @timeout instance variable correctly. I'm not aware of an instance of the HW3 grader getting stuck, but we should review the code; and at the very least add a test case for some other HW3 cases other than a perfect pass.
This seems lower priority, so perhaps this test should be set up with a new approach that avoids exposing the solutions as discussed here: #70
since cs169dev slack instance not being used anymore?
@zhangaaron @armandofox - we should probably review how much of the assignment solutions are being exposed here:
https://github.com/saasbook/rag/tree/master/features/support
During the MOOC of fall 2015 HW4 submissions appeared to cause the grader to crash hard such that it required restarting.
It would be good to address this before the current round of SPOCs get to that assignment.
It seems like we would need a load testing harness to simulate the conditions that caused the problem
@rosewcs345 reported a student's submission failing with the message "unable to deliver your submission":
Rose added:
Unpacked their submission, removed extraneous files, re-zipped and uploaded. Still won't even accept. Last night it was being graded, but all the rspec tests were ignored. Now it gives the same error msg I showed you earlier (Unable to deliver your submission to grader (Reason: cannot connect to server). Please try again later.) , but when I submit a different student's submission, it is accepted and graded.
I noticed that they are using newer rspec syntax. Can the autograder handle it? (although it doesn't seem likely that it would produce the above error)
@rilson17 reports the grader is down. I see the following in the grader direct output in tmux:
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in `require'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in `block in require'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:240:in `load_dependency'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in `require'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:360:in `require_or_load'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:494:in `load_missing_constant'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:184:in `const_missing'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:526:in `load_missing_constant'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:184:in `const_missing'
/home/ubuntu/rag/submissions/a55eec8354504b64f3d70a7448f3e045/assignment4/2016-05-03-18-26-57/db/migrate/20120130161449_add_more_movies.rb:16:in `block in up'
/home/ubuntu/rag/submissions/a55eec8354504b64f3d70a7448f3e045/assignment4/2016-05-03-18-26-57/db/migrate/20120130161449_add_more_movies.rb:15:in `each'
/home/ubuntu/rag/submissions/a55eec8354504b64f3d70a7448f3e045/assignment4/2016-05-03-18-26-57/db/migrate/20120130161449_add_more_movies.rb:15:in `up'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:608:in `exec_migration'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:589:in `block (2 levels) in migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:588:in `block in migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:587:in `migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:764:in `migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:994:in `block in execute_migration_in_transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:1040:in `block in ddl_transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/transactions.rb:220:in `transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:1040:in `ddl_transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:993:in `execute_migration_in_transaction'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:955:in `block in migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:951:in `each'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:951:in `migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:819:in `up'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/migration.rb:797:in `migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/tasks/database_tasks.rb:137:in `migrate'
/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.3/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
from /home/ubuntu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open3.rb:199:in `popen_run'
from /home/ubuntu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open3.rb:93:in `popen3'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:152:in `block in setup_rails_app'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:151:in `each'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:151:in `setup_rails_app'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:82:in `block (2 levels) in runner_block'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:336:in `time_operation'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:81:in `block in runner_block'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:76:in `chdir'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:76:in `runner_block'
from /home/ubuntu/rag/lib/auto_grader.rb:73:in `call'
from /home/ubuntu/rag/lib/auto_grader.rb:73:in `block in run_in_subprocess'
from /home/ubuntu/rag/lib/auto_grader.rb:69:in `fork'
from /home/ubuntu/rag/lib/auto_grader.rb:69:in `run_in_subprocess'
from /home/ubuntu/rag/lib/graders/feature_grader/hw4_grader.rb:55:in `grade'
from /home/ubuntu/rag/lib/submission/base.rb:20:in `handle_submission'
from /home/ubuntu/rag/lib/submission/polling.rb:19:in `block in run'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `loop'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `run'
from run_autograder.rb:27:in `<main>'
Submission message after grade: Your submission was recorded at 2016-05-03 18:21:38 +0000 : submission is on time.
Score: 0/100
There was a fatal error with your submission. It either timed out or caused an exception.
logged in to the grader and saw this error:
Submission message after grade: Your submission was recorded at 2016-04-06 04:41:05 +0000 : submission is on time.
Score: 1.0/1.0
--------------------------------------------------------------------------------
Test passed. (+0.2)
--------------------------------------------------------------------------------
Test passed. (+0.075)
--------------------------------------------------------------------------------
Test passed. (+0.075)
--------------------------------------------------------------------------------
Test passed. (+0.075)
--------------------------------------------------------------------------------
Test passed. (+0.075)
--------------------------------------------------------------------------------
Test passed. (+0.25)
--------------------------------------------------------------------------------
Test passed. (+0.25)
Total score: 1.0 / 1.0
Completed in 35.455482645 seconds.
Cloning into 'temp_repo'...
remote: Counting objects: 10, done.
remote: Total 10 (delta 0), reused 0 (delta 0), pack-reused 10
Receiving objects: 100% (10/10), done.
Checking connectivity... done.
run_autograder.rb:10: warning: class variable access from toplevel
run_autograder exited.
/home/ubuntu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/time.rb:300:in `local': argument out of range (ArgumentError)
from /home/ubuntu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/time.rb:300:in `make_time'
from /home/ubuntu/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/time.rb:364:in `parse'
from /home/ubuntu/rag/lib/assignment/xqueue.rb:14:in `create_time_bracket'
from /home/ubuntu/rag/lib/assignment/xqueue.rb:36:in `block in initialize'
from /home/ubuntu/rag/lib/assignment/xqueue.rb:36:in `each'
from /home/ubuntu/rag/lib/assignment/xqueue.rb:36:in `map'
from /home/ubuntu/rag/lib/assignment/xqueue.rb:36:in `initialize'
from /home/ubuntu/rag/lib/submission/xqueue.rb:21:in `new'
from /home/ubuntu/rag/lib/submission/xqueue.rb:21:in `next_submission_with_assignment'
from /home/ubuntu/rag/lib/submission/polling.rb:18:in `block in run'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `loop'
from /home/ubuntu/rag/lib/submission/polling.rb:17:in `run'
from run_autograder.rb:27:in `<main>'
The grader currently caches the assignment spec the first time it operates. Great for network efficiency, but it means there is no external mechanism for indicating to the grader that a homework specification has changed.
This issue should be clearly flagged in documentation, and possibly addressed with some way to indicate a spec is 'dirty' or to have the grader make a lightweight check of the status of the file on github ...
Just tried submitting reference solution to staging grader and got this response:
Your submission was recorded at 2016-01-19 12:02:29 +0000 : submission is on time.
Score: 0/200
cannot load such file -- ruby-debugcannot load such file -- ruby-debug
The grader currently downloads every file in an assignments 'autograder' (or 'spec'?) folder. However if a spec needs to load files relative to itself it has a problem in as much as the grader will try to load files relative to it's root, and so currently it's not clear how to allow 3rd parties to develop specs that require associated files.
Berkeley SPOC TAs and MOOC students reported the grader was down.
SSHing into the instance I got a filespace warning, and couldn't re-start the grader. This is the filespace stats I saw.
ubuntu@ip-10-182-65-242:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 4.2G 3.2G 57% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 3.7G 8.0K 3.7G 1% /dev
tmpfs 752M 208K 752M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.7G 0 3.7G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/xvdb 30G 45M 28G 1% /mnt
I followed the solution here
http://stackoverflow.com/a/9728026/316729
Creating a 160Gig volume based on the existing snapshot - one wrinkle was that I had to make sure the volume was in us-east-1c
Here are the stats afterwards and the grader seems to be working
ubuntu@ip-10-229-30-162:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 158G 4.2G 147G 3% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 3.7G 8.0K 3.7G 1% /dev
/dev/xvdb 30G 45M 28G 1% /mnt
Ideally we need monitoring on this and some form of log/submission archiving or deleting
MigrationGrader is in the code base but not tested
in many other projects I use develop rather than master, and we used to use it in rag, but I've updated the CONTRIBUTING.md to say master, and maybe we stick with that for the time being ... so should probably delete develop branch ...
at the moment, the output does not indicate that the app is being mutated under the hood, and students often miss that the failure is caused by the mutation - perhaps we could emphasize further?
see https://github.com/saasbook/rag/blob/master/lib/graders/feature_grader/hw4_grader.rb#L204 where it looks like we should have the rspec_score set to 0 if the rspec_max is zero
do large log files cause the grader trouble?
This is our current file system usage on the production grader:
ubuntu@ip-10-182-65-242:~$ df -ah
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 3.5G 3.9G 48% /
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 0 0 0 - /sys/fs/fuse/connections
none 0 0 0 - /sys/kernel/debug
none 0 0 0 - /sys/kernel/security
udev 3.7G 8.0K 3.7G 1% /dev
devpts 0 0 0 - /dev/pts
tmpfs 752M 208K 752M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.7G 0 3.7G 0% /run/shm
none 100M 0 100M 0% /run/user
none 0 0 0 - /sys/fs/pstore
systemd 0 0 0 - /sys/fs/cgroup/systemd
/dev/xvdb 30G 45M 28G 1% /mnt
we have 70Mb of log files:
ubuntu@ip-10-182-65-242:~/rag/logs$ du -m
70 .
and we have about 2 gigs in the submissions directory:
ubuntu@ip-10-182-65-242:~/rag/submissions$ du -sh
2.0G .
In situations where a live grader is not available it's important that users can run the grader locally from the command line to check a possible solution.
One can do that directly with rspec like so:
bundle exec rspec -r ../hw-oracle-of-bacon/lib/oracle_of_bacon.rb autograder/oracle_of_bacon_spec.rb
but this won't generate a grade per se. We should check we can grade from the command line and document the process
Appears that the HW4 cucumber test is not working as intended. I'm unsure when this happened, or if this is an issue with the fact that we are running a Cucumber test suite within a Cucumber test suite and some environment variables are being reset during the test run. I seem to recall having some of these issues over the summer but I don't recall how I solved it, if I did. It hasn't been showing up on Travis because Travis doesn't run these tests, and the other tests don't have the problem since it doesn't get to running the Cucumber features. @tansaku, have you run into this issue during running the test suite?
bundle exec cucumber features/autograder_xqueue_stubbed.feature:47
Feature: Autograder configured to accept student submissions from edX and grade them
As an instructor
So that I can give feedback to my students on their code responses
I want to be able to create a code submission page in edX and grade it using rag
@require_net_connect
Scenario: student submits a HW4 # features/autograder_xqueue_stubbed.feature:47
Given I set up a test that requires internet connection # features/step_definitions/adapter_x_queue_steps.rb:24
Given an XQueue that has submission "hw4_submission.json" in queue # features/step_definitions/adapter_x_queue_steps.rb:28
["http://www.fakedownload.com/hw4.zip"]
register_uri for : http://www.fakedownload.com/hw4.zip
And has been setup with the config file "conf.yml" # features/step_definitions/adapter_x_queue_steps.rb:44
/media/aaron/why/sum2015/rag/lib/graders/feature_grader/lib/HW4feature.rb:151:in `rescue in run!': invalid cucumber results["Disabling profiles...\n", "\n", " *** ERROR: #<SystemExit: exit> *** \n", "Run options: --seed 35036\n", "\n", "# Running:\n", "\n", "\n", "\n", "Finished in 0.001925s, 0.0000 runs/s, 0.0000 assertions/s.\n", "\n", "0 runs, 0 assertions, 0 failures, 0 errors, 0 skips\n"] (StandardError)
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/lib/HW4feature.rb:161:in `run!'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/lib/HW4feature.rb:45:in `block in total'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/lib/HW4feature.rb:42:in `each'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/lib/HW4feature.rb:42:in `total'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/hw4_grader.rb:242:in `check_ref_cucumber'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/hw4_grader.rb:109:in `block (2 levels) in runner_block'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/hw4_grader.rb:332:in `time_operation'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/hw4_grader.rb:105:in `block in runner_block'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/hw4_grader.rb:76:in `chdir'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/hw4_grader.rb:76:in `runner_block'
from /media/aaron/why/sum2015/rag/lib/auto_grader.rb:73:in `call'
from /media/aaron/why/sum2015/rag/lib/auto_grader.rb:73:in `block in run_in_subprocess'
from /media/aaron/why/sum2015/rag/lib/auto_grader.rb:69:in `fork'
from /media/aaron/why/sum2015/rag/lib/auto_grader.rb:69:in `run_in_subprocess'
from /media/aaron/why/sum2015/rag/lib/graders/feature_grader/hw4_grader.rb:55:in `grade'
from /media/aaron/why/sum2015/rag/lib/submission/base.rb:20:in `handle_submission'
from /media/aaron/why/sum2015/rag/lib/submission/polling.rb:19:in `block in run'
from /media/aaron/why/sum2015/rag/lib/submission/polling.rb:17:in `loop'
from /media/aaron/why/sum2015/rag/lib/submission/polling.rb:17:in `run'
from /media/aaron/why/sum2015/rag/features/step_definitions/adapter_x_queue_steps.rb:53:in `block (3 levels) in <top (required)>'
Submission message after grade: Your submission was recorded at 2010-06-19 18:52:28 -0700 : submission is on time.
Score: 0/100
There was a fatal error with your submission. It either timed out or caused an exception.
Then I should receive a grade of "100" for my assignment # features/step_definitions/adapter_x_queue_steps.rb:49
expected: == 100.0
got: 0.0 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/adapter_x_queue_steps.rb:57:in `/^I should receive a grade of "(.*?)" for my assignment$/'
features/autograder_xqueue_stubbed.feature:51:in `Then I should receive a grade of "100" for my assignment'
Failing Scenarios:
cucumber features/autograder_xqueue_stubbed.feature:47 # Scenario: student submits a HW4
1 scenario (1 failed)
4 steps (1 failed, 3 passed)
0m16.693s
Can we enforce a default timeout to avoid the issue of a nil timeout being set if the graders initialize method does not call super, e.g. use class or class-instance vars?
@rilson17 is interested in having the Calisthenics assignment ported to the new grader, or at least made available
For HW5 (https://github.com/saasbook/hw-acceptance-unit-test-cycle), some locally passed rspec tests failed on auto-grader.
Specifically, they are
it 'should set variable properly' do
get :related_movies, id: 1
expect(assigns(:movies).length).to eq(2)
end
it 'should render the related_movies template' do
get :related_movies, id: 1
expect(response).to render_template('related_movies')
end
The output of the auto-grader is
----BEGIN RSPEC----
--------------------------------------------------------------------------------
/home/ubuntu/.rvm/rubies/ruby-2.2.2/bin/ruby -I/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /home/ubuntu/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
..FF
Failures:
1) MoviesController GET Related Movies should set variable properly
Failure/Error: expect(assigns(:movies).length).to eq(2)
NoMethodError:
undefined method `length' for nil:NilClass
# ./spec/controllers/search_movie_by_director_spec.rb:29:in `block (3 levels) in '
2) MoviesController GET Related Movies should render the related_movies template
Failure/Error: expect(response).to render_template('related_movies')
expecting <"related_movies"> but rendering with <[]>
# ./spec/controllers/search_movie_by_director_spec.rb:34:in `block (3 levels) in '
Finished in 0.07296 seconds (files took 2.93 seconds to load)
4 examples, 2 failures
Failed examples:
rspec ./spec/controllers/search_movie_by_director_spec.rb:27 # MoviesController GET Related Movies should set variable properly
rspec ./spec/controllers/search_movie_by_director_spec.rb:32 # MoviesController GET Related Movies should render the related_movies template
Coverage report generated for Cucumber Features, RSpec to /home/ubuntu/rag/submissions/d8ca89a1d3e9682e1efc16a6334af2ba/assignment4/2017-07-11-21-25-45/coverage. 48 / 53 LOC (90.57%) covered.
/home/ubuntu/.rvm/rubies/ruby-2.2.2/bin/ruby -S bundle exec cucumber --profile default
/home/ubuntu/.rvm/rubies/ruby-2.2.2/bin/ruby -I/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/home/ubuntu/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /home/ubuntu/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed
----------------------------------------
we have code that causes some paths to end up going to the "non-existent" rails home page, e.g. this link when hit in cucumber scenario takes us to index.html
link_to 'Movie Title', movies_path(:sort => 'title', :ratings => @selected_ratings)
We have done case A
a) take boiler plate
b) add reference sort_filtu.feature and steps and paths
case A is okay
Case B
a) take boiler plate
b) sort_filter.feature is added
c) other changes (routes, paths, add_director)
I think this class name change will make it much clearer what the class actually does - HW5 is not very descriptive and also somewhat out of date as the HW in the MOOC is now in two parts and called LEGACY-HW-1 and LEGACY-HW-2
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.